Air-gapped RAG: a reference architecture

Retrieval-Augmented Generation is the most practical way to make a language model useful on private data. But most RAG tutorials assume you can call OpenAI, pin a vector database to a managed cloud, and move on. In regulated industries — defense, healthcare, finance, critical infrastructure — that assumption is a non-starter.

This article lays out a reference architecture for RAG that runs entirely air-gapped: no data leaves your network, no model phones home, no third-party service sees your queries.

Why air-gapped matters

The argument for air-gapping is not theoretical. It addresses three concrete risks:

Data exfiltration. Every API call to an external model sends your prompt — and your retrieved context — over the wire. Even with contractual guarantees, the data transits infrastructure you do not control.
Compliance. Regulations like GDPR, HIPAA, SecNumCloud, and various defense classification frameworks impose strict boundaries on where data can be processed. "The model provider promises not to log it" does not satisfy an auditor.
Availability. An air-gapped system works when the internet does not. For operational environments — field deployments, ships, remote facilities — this is not optional.

The four components

A RAG pipeline has four moving parts. Each one needs an air-gapped equivalent.

1. The embedding model

Embeddings convert documents and queries into vectors. In a cloud setup, you call an API. Air-gapped, you run the model locally.

Good choices today:

Sentence-Transformers (e.g., all-MiniLM-L6-v2) — small, fast, runs on CPU. Good enough for most document retrieval tasks.
BGE or E5 family — stronger retrieval performance, still manageable on a single GPU.
Instructor models — if you need task-specific instructions in your embeddings.

The key constraint: the embedding model must be downloaded and verified before deployment. Pin the model hash. Do not rely on pulling from Hugging Face at runtime.

2. The vector store

The vector store indexes your embeddings and handles similarity search. Several options work fully offline:

Qdrant — Rust-based, performant, supports filtering and metadata. Runs as a single binary with no external dependencies.
Chroma — Python-native, simple API, embeds directly in your application process.
pgvector — if you already run PostgreSQL, this extension adds vector similarity search without another service.
FAISS — Facebook's library, no server process, just a library call. Best for batch workloads.

Avoid managed vector databases (Pinecone, Weaviate Cloud) — they require network access by design.

3. The language model

This is the core of the pipeline. Air-gapped means self-hosted. The practical options:

For GPU-equipped servers:

Mistral, Llama 3, or Qwen families in 7B–70B sizes, served via vLLM or TGI.
Quantized variants (GPTQ, AWQ, or GGUF) to maximize throughput per GPU.

For CPU-only environments:

llama.cpp with GGUF models — runs on commodity hardware, surprisingly capable at 7B–13B sizes.
Smaller models (3B–7B) for constrained edge devices.

Key decisions:

Model size vs. hardware budget. A quantized 13B on one GPU often outperforms a full 7B.
Context window. RAG injects retrieved passages into the prompt — you need enough context to fit them. 8K tokens minimum; 32K+ preferred.
Licensing. Llama 3 and Mistral have permissive licenses. Some models have restrictions on commercial or government use — verify before deployment.

4. The orchestrator

The orchestrator ties everything together: takes a user query, embeds it, retrieves relevant passages, builds a prompt, and sends it to the model.

In a cloud setup, this is often LangChain or LlamaIndex calling external services. Air-gapped, you need the same logic but with all calls staying local.

Options:

LangChain / LlamaIndex — both support local models and local vector stores. Strip out any cloud-dependent plugins.
Custom Python — for simple pipelines, 50 lines of code with requests (to a local vLLM endpoint) and the vector store client is often more maintainable than a framework.
Haystack — modular, supports local components natively.

Network architecture

The reference deployment looks like this:

┌─────────────────────────────────────────────────┐
│                 Secure enclave                   │
│                                                  │
│  ┌──────────┐   ┌──────────┐   ┌──────────────┐ │
│  │  Ingestion│   │  Vector  │   │   LLM        │ │
│  │  pipeline │──▶│  store   │   │   (vLLM)     │ │
│  └──────────┘   └────┬─────┘   └──────┬───────┘ │
│                      │                │          │
│                 ┌────▼────────────────▼───────┐  │
│                 │     Orchestrator            │  │
│                 │     (API server)            │  │
│                 └────────────┬────────────────┘  │
│                              │                   │
│  ┌───────────────────────────▼────────────────┐  │
│  │          Internal network only             │  │
│  │          (no egress to internet)           │  │
│  └────────────────────────────────────────────┘  │
│                              │                   │
│                 ┌────────────▼────────────────┐  │
│                 │   User interface / app      │  │
│                 └────────────────────────────-┘  │
└─────────────────────────────────────────────────┘

Key principles:

No egress. The enclave has no route to the internet. Models, embeddings, and data are loaded via a secure transfer process (USB, dedicated transfer network, or similar).
Minimal ingress. Users access the system through an internal network or VPN. The API surface is small and auditable.
Separation of concerns. The ingestion pipeline (document processing, chunking, embedding) runs separately from the query pipeline. Ingestion can be batch; queries are real-time.

Document ingestion pipeline

Getting documents into the system requires a pipeline:

Format handling. PDF, DOCX, HTML, plain text. Use Apache Tika or unstructured for extraction. Both run locally.
Chunking. Split documents into passages of 256–512 tokens. Overlap chunks by 10–20% to preserve context at boundaries.
Embedding. Run each chunk through the embedding model. Store the vector alongside the chunk text and metadata (source document, page number, classification level).
Indexing. Insert into the vector store with appropriate metadata filters.

For classified environments, the ingestion pipeline must handle document markings and ensure that retrieval respects access controls — a user with SECRET clearance should not retrieve TOP SECRET passages.

What to watch out for

Model drift is not a problem. Unlike cloud APIs that update without warning, your local model is frozen. This is actually an advantage — reproducibility is guaranteed.

Hardware sizing is the hard part. Undersized GPUs mean slow inference; oversized means wasted budget. Start with one A100 or equivalent, measure actual throughput on your workload, then scale.

Updates require a process. When a better model comes out, you need a secure path to evaluate it offline, validate it, and deploy it. Build this process early — do not wait until the current model is outdated.

Evaluation is non-negotiable. Without an evaluation framework, you cannot know if your RAG pipeline is actually answering correctly. Build a test set of questions with known answers from your corpus. Run it after every change.

Getting started

Pick a model size that fits your hardware. When in doubt, start with a quantized 13B.
Choose a vector store that matches your ops team's skills. If they know PostgreSQL, use pgvector.
Build the simplest possible pipeline first — embed, retrieve, generate. No reranking, no query expansion, no hybrid search. Add complexity only when evaluation shows you need it.
Lock down the network. No egress, minimal ingress, audit everything.
Write evaluation tests before you write features.

We design and deploy air-gapped RAG systems for organizations that cannot compromise on data sovereignty.

Get in touch to discuss your architecture.