Build a Private AI Assistant With Local Models + RAG

A RAG assistant built the usual way sends two things off your machine for every single question: the document chunks it embeds, and the retrieved context it hands to a model for the final answer. That's a reasonable trade for most use cases, and a nonstarter for others, HR files, contracts, medical notes, anything with a compliance requirement that the document never left a specific boundary in the first place. "Ask an AI assistant about these documents" and "these documents can never leave this machine" aren't in conflict. They just require the entire pipeline, embedding, storage, retrieval, and generation, to actually run locally, not just the last step.

RAG From Scratch built the retrieval pipeline itself, by hand, using cloud embeddings and a cloud model, specifically to make the mechanics visible. This is the same shape of pipeline, chunk, embed, store, retrieve, generate, rebuilt so that no step in it ever makes a network call at all.

The Architecture

What "Local" Actually Has to Mean Here

The broader question of when local models make sense at all, hardware, cost, and capability tradeoffs, is covered in full in AI Offline vs Online Models, and getting Ollama installed and running a first local model is covered in How to Build Your Own Offline AI Application. This article assumes both of those are already true and focuses specifically on what changes when the goal is a document-aware assistant instead of a plain chat window.

Choosing Local Models for This Specific Job

A model that's fine for casual chat isn't automatically the right choice for grounded question-answering over private documents, and embedding and generation are two separate decisions.

For generation, the real constraint is instruction-following discipline, staying inside the retrieved context instead of filling gaps from training data, more than raw capability. A mid-sized instruction-tuned model, llama3.1:8b or similar, run through Ollama, is usually enough for this specific job. The honest tradeoff: smaller local models are measurably weaker than frontier cloud models at complex reasoning, which matters more for some document sets than others, covered in the decision framework in AI Offline vs Online Models.

Chunking and Embedding, Entirely Local

import requests
from pathlib import Path
 
OLLAMA_URL = "http://localhost:11434"
EMBED_MODEL = "nomic-embed-text"
 
def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunks.append(" ".join(words[start:end]))
        start = end - overlap
    return chunks
 
def embed(text: str) -> list[float]:
    response = requests.post(
        f"{OLLAMA_URL}/api/embeddings",
        json={"model": EMBED_MODEL, "prompt": text},
    )
    return response.json()["embedding"]
 
def load_documents(folder: str) -> list[dict]:
    records = []
    for path in Path(folder).glob("*.txt"):
        for chunk in chunk_text(path.read_text()):
            records.append({"source": path.name, "chunk": chunk, "embedding": embed(chunk)})
    return records

A Persistent Local Vector Store

The educational version of this step keeps vectors in memory for the length of one script. A real assistant needs to index a document set once and answer questions against it across many separate runs, which means the store has to survive the process exiting.

import sqlite3
import numpy as np
import json
 
def init_store(db_path: str = "assistant.db") -> sqlite3.Connection:
    conn = sqlite3.connect(db_path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS chunks (
            id INTEGER PRIMARY KEY,
            source TEXT,
            chunk TEXT,
            embedding TEXT
        )
    """)
    return conn
 
def save_records(conn: sqlite3.Connection, records: list[dict]) -> None:
    conn.executemany(
        "INSERT INTO chunks (source, chunk, embedding) VALUES (?, ?, ?)",
        [(r["source"], r["chunk"], json.dumps(r["embedding"])) for r in records],
    )
    conn.commit()
 
def search(conn: sqlite3.Connection, query_embedding: list[float], top_k: int = 4) -> list[dict]:
    rows = conn.execute("SELECT source, chunk, embedding FROM chunks").fetchall()
    query_vec = np.array(query_embedding)
    scored = []
    for source, chunk, embedding_json in rows:
        vec = np.array(json.loads(embedding_json))
        score = np.dot(query_vec, vec) / (np.linalg.norm(query_vec) * np.linalg.norm(vec))
        scored.append({"source": source, "chunk": chunk, "score": float(score)})
    scored.sort(key=lambda r: r["score"], reverse=True)
    return scored[:top_k]

Retrieval and Grounded Generation, Fully Local

GENERATE_MODEL = "llama3.1:8b"
 
def build_prompt(query: str, retrieved: list[dict]) -> str:
    if not retrieved:
        context = "No relevant documents were found for this question."
    else:
        context = "\n\n".join(f"[Source: {r['source']}]\n{r['chunk']}" for r in retrieved)
    return (
        f"Answer the question using only the context below. "
        f"If the context doesn't contain enough information, say so explicitly.\n\n"
        f"Context:\n{context}\n\nQuestion: {query}"
    )
 
def ask(conn: sqlite3.Connection, query: str) -> str:
    query_embedding = embed(query)
    retrieved = search(conn, query_embedding)
    prompt = build_prompt(query, retrieved)
 
    response = requests.post(
        f"{OLLAMA_URL}/api/generate",
        json={"model": GENERATE_MODEL, "prompt": prompt, "stream": False},
    )
    return response.json()["response"]

Verifying Nothing Actually Leaves the Machine

An architecture diagram claiming something is private is not the same as confirming it. The difference matters enough to actually check, not just assume.

A Worked Trace

Indexing a small folder of internal policy documents, then asking two questions.

When Local Isn't the Right Call

The Bottom Line

A private AI assistant isn't a local LLM with a nice prompt, it's a pipeline where every component, chunking, embedding, storage, retrieval, and generation, runs on the machine the documents already live on, with nothing assumed private that wasn't actually verified. The mechanics are the same RAG pipeline covered from first principles in RAG From Scratch. What changes is where every one of those mechanics actually executes, and whether anyone bothered to check.

The documents never had to leave the machine. Building the assistant so that's actually true, not just architecturally implied, is the entire point of doing this locally at all.