Local LLM Deployment for Air-Gapped Development Environments

Air-gapped LLM deployment isn't a config setting you flip on a normal AI stack. It's a different stack, built from the ground up so that nothing inside it ever needs to phone home. That distinction matters most in three places: defense, healthcare, and financial services, and not because people in those industries are nervous by nature. Their constraints are written into law. A broker-dealer sending client portfolio data through a vendor's inference API isn't taking a risk, it's running directly against SEC Regulation S-P's requirements for safeguarding client data. A hospital routing patient records through a third-party model endpoint isn't cutting a corner, it's creating serious HIPAA exposure. And in classified defense work, a cloud API isn't a compliance question to weigh. It disqualifies the architecture before anyone even asks about model quality.

Los Alamos National Laboratory gives the clearest real-world marker here. Early in 2025, the lab moved its LLM stack fully on-premises to handle Controlled Unclassified Information, ITAR-flagged material, and Unclassified Controlled Nuclear Information. That's not a lab experimenting with local models for fun. That's a facility whose data categories make an outbound API call a non-starter, full stop.

So what does it actually take to run a coding-capable LLM stack with zero path to the outside internet? Every layer, from inference serving to embeddings to vector storage to how you even get model updates in the door, has to be rebuilt around one rule: nothing assumes it can reach out. Below is what that rebuild actually looks like.

What "air-gapped" actually means architecturally, and why most "isolated" stacks are not

Here's where a lot of teams trip up before they've even started: they call something "air-gapped" when it's really just "isolated," and those are not the same claim.

An isolated stack usually means a private VPC, a NAT gateway, and an egress allowlist. Traffic is restricted, sure, but there's still a route out. A truly air-gapped environment has no NAT, no DNS resolution to external hostnames, no public certificate authority chain, and no path by which a packet could leave even if something inside tried. Air-gapped isn't a setting you toggle. It's an architecture where every single thing the system needs while it's running is already sitting inside the enclave before the perimeter closes.

That gap between "isolated" and "air-gapped" splits further into physical and logical versions, and the difference isn't academic:

Physical air-gapping means there's no network cable, no wireless card, nothing connecting the inference server to anything outside. Administration happens at a local console, full stop. This removes entire attack categories, remote code execution and credential theft included, because there's no wire for an attacker to walk in on. Logical air-gapping relies on software-defined boundaries, like VPCs and micro-segmentation, that depend on the hypervisor doing its job correctly. That's a weaker guarantee than a cut wire, and it's not what defense classification levels like IL5 or IL6, or FedRAMP High, are designed around environments where.

The one-way data diode is one middle-ground option to know about. It lets telemetry flow out for monitoring purposes without opening anything resembling a two-way network connection. It functions as a valve rather than a network path, and it often gives teams some visibility into an otherwise sealed system.

The egress audit: finding every dependency before racking a single GPU

Most teams want to jump straight to picking hardware and a model. That's backwards. Before any GPU gets racked, the actual first step is figuring out everything the current stack quietly reaches out to the internet for, because you can't design a sealed box around a system you don't fully understand yet.

An egress audit usually turns up more than the obvious API call to a model provider. It surfaces things like:

• A hosted LLM used as a judge for evaluation pipelines
• Weight downloads pulled straight from Hugging Face at runtime
• Prompt-monitoring SaaS tools quietly logging every request
• Vector database telemetry pings
• OpenTelemetry collectors shipping traces to a managed backend
• Slack webhooks fired off from eval pipelines
• npm postinstall scripts buried inside SDKs, reaching out during install

One practical way to find all of this: run the existing stack inside a network namespace with a default-deny egress rule, then just watch what breaks. Every failure maps directly to something the air-gapped version has to replace with a local equivalent. Most teams doing this for the first time find somewhere between five and fifteen distinct egress dependencies they had no idea existed. That's not a small number, and it's usually the point where a project's timeline estimate quietly doubles.

The full pre-staged stack: what every air-gapped enclave must contain at runtime

Picking the model is, honestly, the easy part of this whole project. The operational surface around it, everything that has to be present and working before the network perimeter seals shut, is far wider than most people expect going in.

Here's what has to be pre-staged, meaning fully in place before the enclave closes off:

A signed local model registry. Not something the system downloads when it needs it. A versioned artifact with chain-of-custody documentation attached. GPU inference workers: vLLM is the common choice for high-throughput serving; it uses PagedAttention to manage VRAM efficiently and can handle concurrent requests from multiple isolated workstations hitting one GPU. Ollama is the simpler option for teams without dedicated ML engineers on staff, since it manages quantized GGUF files and keeps local serving straightforward. A local embedding model, running inside the enclave. This is the single most common gap teams miss. A "complete" air-gapped RAG setup that quietly calls an external embedding API isn't air-gapped. It's isolated with a hole in it. A local, self-hosted vector database. No managed vector DB service survives behind the perimeter, by definition. A container registry mirror. Every image pre-pulled and mirrored internally. No runtime pulls from Docker Hub or any external registry. OS, language runtime, and Python package mirrors. A pip install at runtime normally calls out to PyPI. Inside the enclave, that call has to resolve to an internal mirror instead. An on-premises observability stack. OpenTelemetry collectors need to ship traces to something internal, not a managed SaaS dashboard. Internal PKI. No public certificate authority chain exists inside the perimeter, so TLS termination and certificate issuance both have to happen locally. Local MCP gateways for legacy data systems. This lets an LLM query internal databases without ever reaching out externally, which matters a lot for defense agencies running infrastructure that's decades old.

On the container runtime side, Ubuntu 22.04 LTS and RHEL are the common OS choices, with Docker or Podman handling containers (Podman is often the pick where a Docker daemon isn't allowed to run at all).

NVIDIA NIM is purpose-built for exactly this problem. NIM for LLMs supports air-gapped deployment by preloading models into a local repository, keeping all model artifacts inside the enclave. Its create-model-store command builds that local repository directly, so there's no call out to the NGC registry or Hugging Face Hub once the system is running.

Model artifact provenance: why the weight file is a regulated artifact, not a download

Treating a model's weight file as just another file to download is a mistake, and not a theoretical one. Researchers have shown that Hugging Face's Safetensors conversion service could be compromised in ways that hijack a submitted model. OWASP's LLM Top 10 lists supply chain risk as a primary category for a reason.

The SafeTensors format itself solves one real problem, pickle-style code execution during load, but it doesn't solve provenance. There's still no widely adopted way to cryptographically sign a set of weights and verify that signature before loading it. That gap doesn't close itself. Teams have to build it: hash-pin every model artifact, sign it against an internal key during ingestion, and refuse to load anything whose signature doesn't check out.

This is where the idea of an MLBOM (an AI/ML Bill of Materials, increasingly published in CycloneDX format) comes in. A regular software SBOM tracks libraries and their versions. An MLBOM has to track the model itself, the tokenizer, the safety classifier, any LoRA adapters layered on top, merged checkpoints, the quantization tooling used to produce the deployed weights, the eval suite that gated the release, and every license attached anywhere in that chain.

A fine-tune built on top of another fine-tune can drag along a license clause from a base model nobody on the team ever actually agreed to, and this can happen without anyone noticing until it surfaces as a legal or compliance problem. A fine-tune built on top of another fine-tune can drag along a license clause from a base model nobody on the team ever actually agreed to. The MLBOM is the only artifact that makes that whole chain visible and auditable after the fact.

Choosing a model for an offline coding environment: VRAM, context, and task fit

Here's the constraint that actually decides which model a team ends up running: available VRAM, not benchmark rank. A model sitting at the top of SWE-Bench is worth exactly nothing if it doesn't fit on the hardware sitting in the rack.

Mistral 7B is a useful sizing anchor here, since its VRAM footprint scales predictably with quantization:

• Full precision: 16.3 GB, which needs a dedicated workstation GPU
• 8-bit quantization: 9.7 GB, fits a standard 12 GB workstation card
• 4-bit quantization: 5.8 GB, workable even on older or edge hardware

Past that baseline, the right model depends heavily on the actual task:

Best overall for coding and agentic work, on a 24 to 32 GB GPU or 32 GB Mac: qwen3-coder:30b. It's a 30B mixture-of-experts model with only 3.3B active parameters, runs at 19 GB under Q4_K_M quantization, and supports a 256K context window. Best quality per gigabyte at that hardware tier. Strongest agentic SWE-Bench score at accessible VRAM: devstral:24b, hitting 46.8% on SWE-Bench Verified at just 14 GB. It's the only local coding model in this range with a hard published agentic benchmark number. Best dense single-GPU model at 18 GB VRAM: Qwen 3.5-27B, scoring 49.2% on SWE-Bench with broad programming language support. Top HumanEval scores among local weights: Qwen 3.5 Coder 32B at 92.1%, which actually beats Claude Sonnet 4.6's 89.4% and closes in on Opus 4.6's 94.2%. DeepSeek's 236B variant goes further, at 95.7%, but needs multiple GPUs to run. Best dense model on overall benchmarks: Qwen 3.6 27B, at 77.2% on SWE-bench, running on 22 GB of VRAM. Laguna XS 2.1 from Poolside is the newest agentic entrant, scoring 70.9% on SWE-Bench Verified with a 256K context window, released under the OpenMDW-1.1 license. Best overall coding performance, high SWE-Bench score: Kimi K2.6, at 58.6 on SWE-Bench Pro. It's a mixture-of-experts model (32B active out of 1T total parameters) under a Modified MIT license. Fast inline autocomplete: Codestral, a strong option for real-time completion inside an IDE. Best quality per gigabyte on tight hardware: DeepSeek-Coder V3 (Distilled), landing at 40.5% on SWE-Bench Verified while fitting in 12 GB. Reasoning-heavy and algorithmic work: DeepSeek's larger variants, solid picks for complex problem-solving tasks. Largest context window among local models covered here: Llama 4 Scout, with a context window far larger than the others, enough to load an entire mid-sized codebase into one prompt. Fastest tokens per second: Gemma 4 26B A4B, trading some benchmark ranking for raw speed. Open-weights option at 128K context on 16 GB RAM: gpt-oss:20b, OpenAI's open-weight release running at MXFP4. Safest license for multilingual and commercial use: Qwen3 235B-A22B, under Apache 2.0, a mixture-of-experts model (235B total, 22B active) with a wide ecosystem of fine-tunes and quantized versions already built around it.

Context window size matters far more for an agent than it does for autocomplete. Why? Because a coding agent reloads files, test output, and its own prior reasoning at every single turn, not just once. A model with a genuinely usable long context window pays off disproportionately more in an agentic workflow than it ever would for someone just wanting the next line of code finished. That's not a spec sheet detail. It's the difference between an agent that can hold an entire codebase's context and one that starts forgetting what it was doing three files ago.