AI Agent Memory: Short-Term, Long-Term, and Persistent Memory Explained

Two failures look identical from the outside and come from opposite causes. In the first, an agent is told early in a long conversation to always use PowerShell instead of bash, and forty turns later suggests a bash script anyway, not because it disagreed, but because that instruction scrolled out of what it could actually attend to. In the second, an agent correctly remembers a user's timezone across an entire session, gets redeployed for an unrelated bug fix, and asks for the timezone again the next morning, because the "memory" holding it was never anything more durable than a Python dictionary in a process that no longer exists.

Both are memory bugs. They require completely different fixes, because "memory" in an agent system isn't one thing, it's at least three architecturally distinct mechanisms with different lifetimes, different storage, and different failure modes, and conflating them is exactly how both bugs above get built by accident.

Three Layers, Three Different Problems

Short-Term Memory: The Context Window Itself

This is the least architecturally interesting layer and the one every agent has by default, because it's just the running list of messages sent to the model on every call. It requires no design decision to exist. It requires a real one to not silently fail as a conversation grows:

class ShortTermMemory:
    def __init__(self, max_messages: int = 40):
        self.messages: list[dict] = []
        self.max_messages = max_messages
 
    def add(self, role: str, content: str):
        self.messages.append({"role": role, "content": content})
        if len(self.messages) > self.max_messages:
            self._consolidate()
 
    def _consolidate(self):
        # Summarize the oldest half instead of silently dropping it.
        oldest = self.messages[: self.max_messages // 2]
        summary = summarize(oldest)
        remainder = self.messages[self.max_messages // 2 :]
        self.messages = [{"role": "system", "content": f"Earlier context: {summary}"}] + remainder

Short-term memory never survives past the run it belongs to, and that's correct, not a limitation to work around. It exists to give the current reasoning step everything it needs right now. What happens to anything worth keeping after the run ends is a different layer's job entirely.

Long-Term Memory: Deciding What Actually Survives

Long-term memory holds structured, retrievable facts that should carry into future sessions, a user's stated preference, a correction the agent got wrong once and shouldn't repeat, a project's current state. The two real design questions here are how something gets in, and how it gets found again.

import time
 
class LongTermMemory:
    def __init__(self, store: "PersistentStore"):
        self.store = store
 
    def remember(self, user_id: str, key: str, value: str):
        self.store.upsert(user_id, key, value, timestamp=time.time())
 
    def recall(self, user_id: str, query: str, top_k: int = 5) -> list[dict]:
        # Semantic retrieval over this user's own remembered facts,
        # the same cosine-similarity mechanism covered in
        # RAG From Scratch, applied to self-generated memory instead
        # of externally authored documents.
        return self.store.search(user_id, query, top_k=top_k)

Two genuinely different ways something ends up in long-term memory, and most real systems need both:

Persistent Memory: Making Long-Term Memory Actually Survive

Here's the distinction that the opening timezone bug depends on entirely: "long-term" describes how long something should conceptually matter. It says nothing about whether the thing holding it will still exist after a crash. LongTermMemory above just delegates to a PersistentStore, and that's deliberate, because durability is a separate concern with its own separate failure mode:

import sqlite3
import json
 
class PersistentStore:
    def __init__(self, db_path: str = "agent_memory.db"):
        self.conn = sqlite3.connect(db_path)
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS memories (
                user_id TEXT NOT NULL,
                key TEXT NOT NULL,
                value TEXT NOT NULL,
                embedding TEXT NOT NULL,
                timestamp REAL NOT NULL,
                PRIMARY KEY (user_id, key)
            )
        """)
        self.conn.commit()
 
    def upsert(self, user_id: str, key: str, value: str, timestamp: float):
        vector = embed_query(value)  # reuses the embedding step from RAG From Scratch
        self.conn.execute(
            "INSERT OR REPLACE INTO memories VALUES (?, ?, ?, ?, ?)",
            (user_id, key, value, json.dumps(vector), timestamp),
        )
        self.conn.commit()
 
    def search(self, user_id: str, query: str, top_k: int = 5) -> list[dict]:
        rows = self.conn.execute(
            "SELECT key, value, embedding FROM memories WHERE user_id = ?", (user_id,)
        ).fetchall()
        query_vector = embed_query(query)
        scored = [
            {"key": k, "value": v, "score": cosine_similarity(json.loads(e), query_vector)}
            for k, v, e in rows
        ]
        return sorted(scored, key=lambda r: r["score"], reverse=True)[:top_k]

The Concurrency Problem Persistence Alone Doesn't Solve

Moving to SQLite fixes the restart problem. It doesn't automatically fix a second one: if the agent runs as multiple concurrent instances behind a load balancer, each instance's own local SQLite file diverges from the others the moment two instances handle the same user. A fact remembered by instance A is invisible to instance B until both are reading from the same shared database, not one file per instance. At real production scale, this typically means moving from local SQLite to a shared, networked store, Postgres with pgvector for the same similarity search shown above, or a managed vector database, but the underlying requirement doesn't change: every instance that might serve a given user needs to be reading and writing the exact same persistent store, not a copy of it.

How This Differs From RAG

RAG From Scratch builds nearly the same retrieval mechanism, chunk, embed, store, search by cosine similarity, and it's worth being precise about what's actually different here, because the code above deliberately reuses it. RAG retrieves from a corpus of documents someone else authored: runbooks, policies, product docs. Long-term memory retrieves from state the agent itself generated about its own interactions: what this specific user said, what the agent decided, what turned out to be true. The retrieval math is the same. What's being retrieved, and who authored it, is not, and that distinction matters most exactly where it seems least important: when deciding how much to trust what comes back. A document from an approved runbook and a memory the agent wrote about itself six weeks ago deserve different default levels of confidence, one is verified organizational knowledge, the other is the system's own possibly-outdated belief about the world.

Memory Isolation Is a Security Boundary

Forgetting Is a Feature, Not a Bug

Long-term memory that only ever grows eventually becomes its own failure mode: retrieval gets slower, results get noisier as old, no-longer-true facts compete with current ones, and, for anything involving real personal data, indefinite retention becomes a genuine privacy liability rather than just an engineering inconvenience. Three practical answers, usually combined:

One Full Trace, Across All Three Layers

An agent handling an incident, restarted mid-week for an unrelated deploy, illustrates every layer at once:

The Bottom Line

"Give the agent memory" sounds like one feature and is actually three separate engineering decisions: how much of the current run to keep in context without drowning the model in irrelevant history, what's actually worth carrying forward once the run ends, and whether the thing storing that is durable enough to survive the process restarting without anyone noticing it happened. Most agent memory bugs trace back to exactly one of those three being treated as if it were one of the other two, an instruction that should have been consolidated but wasn't, or a fact that was faithfully remembered by code that was never actually built to survive a restart.