Agentic Memory: How to Give AI Agents the Ability to Remember

Large Language Models are stateless: every API call starts from a blank slate. An "agent" that forgets the user's name between sessions, repeats mistakes it already made, and re-derives the same facts over and over is not much of an agent. Agentic memory is the set of design patterns that gives an LLM-powered agent persistent, useful state.

This article walks through the full memory stack with runnable code: the statelessness problem, the memory taxonomy, context-window management, semantic and episodic stores, procedural rules, retrieval scoring, and a memory tool the agent drives itself. Every implementation here is available as a complete, end-to-end notebook in the companion repository on GitHub.

The Statelessness Problem

The chat completions API has no server-side conversation state. Each request is independent, and the model only knows what is in the current request's messages array.

Throughout this article, two small helpers wrap the OpenAI SDK: llm for a plain chat call, and parse_llm for structured output validated against a Pydantic schema.

import json, math, time
from openai import OpenAI
 
client = OpenAI()  # reads OPENAI_API_KEY from the environment
MODEL = "gpt-4o"
 
def llm(messages: list[dict], system: str | None = None) -> str:
    msgs = ([{"role": "system", "content": system}] if system else []) + messages
    r = client.chat.completions.create(model=MODEL, messages=msgs)
    return r.choices[0].message.content
 
def parse_llm(user_content: str, system: str, schema):
    r = client.chat.completions.parse(
        model=MODEL,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user_content},
        ],
        response_format=schema,
    )
    return r.choices[0].message.parsed

Now watch what happens across two separate calls:

# Call 1: introduce ourselves
llm([{"role": "user", "content": "Hi! My name is Aadhil and my favorite language is Python."}])
# "Hello Aadhil! It's great to meet you..."
 
# Call 2: a brand-new request. The model has no idea who we are.
llm([{"role": "user", "content": "What's my name and favorite language?"}])
# "I'm sorry, but I don't have access to personal data about individuals..."

The second call cannot answer, because the model genuinely does not know. Everything an agent "remembers" is something your application put back into the prompt. Agentic memory is the engineering discipline of deciding what to store, where to store it, when to write it, and how to get the right pieces back into context at the right time.

The Memory Taxonomy

Agent memory design borrows heavily from cognitive science. The standard taxonomy:

                        AGENT MEMORY
                             │
            ┌────────────────┴────────────────┐
            │                                 │
     SHORT-TERM MEMORY                 LONG-TERM MEMORY
     (working memory)                  (persistent stores)
            │                                 │
   ┌────────┴────────┐          ┌─────────────┼──────────────┐
   │                 │          │             │              │
  Conversation   Scratchpad  EPISODIC      SEMANTIC      PROCEDURAL
  history        (current    "what          "what           "how
  (this turn's   reasoning,   happened"      is true"        to act"
  context)       tool state)

Two axes cut across this taxonomy and drive every design decision:

1. Scope: in-session (dies with the conversation) versus cross-session (survives restarts).
2. Who writes it: the application (deterministic pipelines) versus the agent itself (the model decides what is worth remembering via a memory tool).

The strongest agents use both: deterministic pipelines for reliability, agent-driven writes for judgment.

Architecture of a Memory-Augmented Agent

A production memory system has four moving parts around the LLM call:

 user input ──▶ 1. RETRIEVE     pull relevant memories
                     │          (semantic + episodic + procedural)
                     ▼
                2. ASSEMBLE     system = persona + rules + recalled memories
                     │          messages = working memory
                     ▼
                ┌─────────┐
                │   LLM   │──── may call a memory TOOL (read/write long-term)
                └────┬────┘
                     ▼
                3. RESPOND      answer goes to the user
                     │
                     ▼
                4. CONSOLIDATE  append turn to working memory
                                extract new facts ──▶ semantic store
                                on session end ──▶ episodic store
                                on feedback ──▶ procedural rules

Retrieve, assemble, respond, consolidate: that is the whole architecture. And it points at the key insight of this article. Memory is a prompt-assembly problem, not a database problem. The hard parts are the write policy (what is worth keeping) and the read policy (what is worth injecting into a limited context window). Get that selection logic right and a flat JSON file is enough. Get it wrong and no vector database will save you.

Short-Term Memory and the Context Window

The simplest memory is the conversation history itself, accumulated by the application and resent on every call:

class ConversationMemory:
    """Working memory: accumulates the current conversation's turns."""
 
    def __init__(self, system: str = "You are a helpful assistant."):
        self.system = system
        self.messages: list[dict] = []
 
    def send(self, user_message: str) -> str:
        self.messages.append({"role": "user", "content": user_message})
        reply = llm(self.messages, system=self.system)
        self.messages.append({"role": "assistant", "content": reply})
        return reply

This works until the conversation outgrows the context window or your budget. Three standard management strategies, in increasing sophistication:

The sliding window is trivial (drop the oldest user-assistant pairs until the history fits a token budget, counted with tiktoken), but it forgets: the user's name from turn one is simply gone. Compaction is better. When the history gets long, replace old turns with a dense summary and keep the recent turns verbatim:

def compact(messages: list[dict], keep_recent: int = 4) -> list[dict]:
    """Summarize everything except the last `keep_recent` messages."""
    if len(messages) <= keep_recent:
        return messages
 
    old, recent = messages[:-keep_recent], messages[-keep_recent:]
    transcript = "\n".join(f"{m['role'].upper()}: {m['content']}" for m in old)
 
    summary = llm(
        [{"role": "user", "content": transcript}],
        system=(
            "You compress conversation history for an AI agent's context window. "
            "Preserve user identity and preferences, decisions made, open questions, "
            "and any facts the agent will need later. Omit pleasantries."
        ),
    )
 
    return [
        {"role": "user", "content": f"[Summary of the conversation so far]\n{summary}"},
        {"role": "assistant", "content": "Understood. I have the context. Continuing."},
        *recent,
    ]

Providers also offer hosted working memory (for example, chaining turns by a previous response ID so the server recalls the thread). That is convenient, but note what it is: hosted short-term state for one thread. It does not extract facts, learn rules, or recall across conversations. The layers below do.

Semantic Memory: Durable Facts

Semantic memory stores what is true, independent of when you learned it. The pipeline: extract candidate facts from a conversation with structured outputs, deduplicate against the existing store, persist, and recall relevant facts into future system prompts.

from pydantic import BaseModel
 
class Fact(BaseModel):
    fact: str          # one atomic statement, e.g. "User's timezone is IST"
    category: str      # "identity" | "preference" | "project" | "constraint"
 
class FactList(BaseModel):
    facts: list[Fact]
 
def extract_facts(transcript: str) -> list[Fact]:
    """Pull out durable, atomic facts worth remembering long-term."""
    return parse_llm(
        transcript,
        system=(
            "Extract durable facts about the user or their work that would be "
            "useful in FUTURE conversations. One atomic fact per entry. "
            "Skip small talk, one-off requests, and anything session-specific. "
            "Return an empty list if nothing qualifies."
        ),
        schema=FactList,
    ).facts

Run this on a transcript where the user mentions they are a data engineer building a legal CRM side project with React and Tailwind, prefers code without semicolons, and also asks about the weather, and the extractor captures those four facts while skipping the weather question. A good write policy is as much about what you do not store as what you do.

The store itself is a JSON file with dedup on write and a recall method that ranks facts by keyword overlap with the current query, then formats them as a system-prompt block:

Known facts about the user:
- Aadhil is a data engineer.
- Aadhil is working on a legal CRM side project.
- Aadhil is using React with Tailwind for his project.
- Aadhil prefers code examples without semicolons.

The payoff is immediate: a brand-new session with an empty history still tailors its answers (React and Tailwind ecosystem, no semicolons) because the facts crossed the session boundary through the store.

Episodic Memory: Remembering Past Sessions

Semantic memory stores facts; episodic memory stores events. Each episode is a summarized past session with metadata that supports retrieval: a dense summary, a timestamp for recency scoring, an LLM-assigned importance rating from 1 to 10 at write time, and keywords for cheap relevance matching (production systems use embeddings).

class EpisodeRecord(BaseModel):
    summary: str
    importance: int      # 1 (trivial) .. 10 (critical to remember)
    keywords: list[str]
 
def archive_session(messages: list[dict]) -> EpisodeRecord:
    transcript = "\n".join(f"{m['role'].upper()}: {m['content']}" for m in messages)
    return parse_llm(
        transcript,
        system=(
            "Summarize this session for an agent's episodic memory. "
            "Capture what was worked on, decisions, outcomes, and unresolved items. "
            "Rate importance 1-10: how likely is this session to matter later?"
        ),
        schema=EpisodeRecord,
    )

Procedural Memory: Learning How to Behave

Procedural memory is the agent's learned behavior: rules distilled from feedback, injected into every future system prompt. This is how an agent stops repeating mistakes. When the user says "stop giving me three options every time, just pick one," that becomes a durable rule.

The store is a rule list. The interesting part is the update step, where an LLM reconciles new feedback with existing rules, merging and revising rather than blindly appending:

class RuleUpdate(BaseModel):
    rules: list[str]        # the complete NEW rule list (replaces the old one)
    reasoning: str
 
def learn(current_rules: list[str], feedback: str) -> RuleUpdate:
    return parse_llm(
        f"Current rules:\n{json.dumps(current_rules)}\n\nNew feedback:\n{feedback}",
        system=(
            "You maintain an AI assistant's behavioral rule list. "
            "Given the current rules and new user feedback, return the complete "
            "updated list: merge related rules, revise contradicted ones, keep each "
            "rule one imperative sentence, and keep the list under 10 rules. "
            "Only encode durable behavioral preferences, not one-off requests."
        ),
        schema=RuleUpdate,
    )

Retrieval Scoring: Which Memories Make the Cut?

You cannot inject everything. Context is finite, and irrelevant memories actively hurt by distracting the model. The classic scoring function comes from the Generative Agents paper (Park et al., 2023):

score = w_rel * relevance  +  w_rec * recency  +  w_imp * importance

• Relevance: similarity between the memory and the current query (embeddings in production, keyword overlap in the notebook).
• Recency: exponential decay by age, so newer memories score higher.
• Importance: the write-time 1 to 10 rating.

def score_episode(episode: dict, query: str, *, half_life_days: float = 7.0,
                  w_rel: float = 1.0, w_rec: float = 0.5, w_imp: float = 0.3) -> float:
    q_words = set(query.lower().split())
    ep_words = set(episode["keywords"]) | set(episode["summary"].lower().split())
    relevance = len(q_words & ep_words) / max(len(q_words), 1)
 
    age_days = (time.time() - episode["timestamp"]) / 86400
    recency = math.exp(-math.log(2) * age_days / half_life_days)
 
    return w_rel * relevance + w_rec * recency + w_imp * (episode["importance"] / 10)
 
def retrieve_episodes(memory: EpisodicMemory, query: str, top_k: int = 3) -> list[dict]:
    ranked = sorted(memory.episodes, key=lambda e: score_episode(e, query), reverse=True)
    return ranked[:top_k]

In practice this ranks a three-day-old "debugged a CORS error in the CRM's API layer" episode above yesterday's vacation chat when the query is about an API bug: relevance and importance beat pure recency. The weights are a product decision. A personal assistant weights recency higher (this morning's request matters most); a technical copilot weights relevance (last month's debugging session on this exact file beats yesterday's small talk).

A nice bonus: the formula gives you forgetting for free. Memories whose scores decay below a threshold simply stop being retrieved, fading out the way human memories do.

Agent-Driven Memory: A Memory Tool via Function Calling

Everything so far was application-driven: the pipeline decided what to extract and when. The complementary pattern is agent-driven memory. Give the model a tool and let it decide what is worth writing down. Anthropic ships a memory tool like this natively; with OpenAI you build it yourself with function calling, which is exactly what the notebook does.

The notebook exposes a file-based memory as a single function tool with six commands (view, create, str_replace, insert, delete, rename) operating on a /memories directory. The model issues commands; the backend executes them. Ask the agent to "remember that my standup is at 9:30 and my sprint goal is shipping the invoice module," and it writes the files itself. A completely fresh session can then recover that state by reading its own notes.

One security note that matters: every path the model sends is untrusted output. Resolve it to canonical form and verify it stays inside your memory root, rejecting traversal and absolute escapes:

def _resolve(self, path: str) -> Path:
    """Map a model-supplied /memories/... path to disk, rejecting escapes."""
    candidate = (self.root / path.lstrip("/")).resolve()
    if not candidate.is_relative_to(self.root):
        raise ValueError(f"Path escapes memory root: {path}")
    return candidate

When to use which pattern:

Putting It All Together

The final MemoryAgent composes every layer into the architecture from earlier: retrieve semantic facts, top-scored episodes, and procedural rules into the system prompt; run the conversation in working memory with client-side compaction; and on session end, consolidate by extracting facts and archiving the episode.

class MemoryAgent:
    """An agent wiring together working, semantic, episodic, and procedural memory."""
 
    def __init__(self, persona: str = "You are a helpful personal assistant."):
        self.persona = persona
        self.semantic = SemanticMemory()
        self.episodic = EpisodicMemory()
        self.procedural = ProceduralMemory()
        self.messages: list[dict] = []          # working memory
 
    def _build_system(self, query: str) -> str:
        parts = [self.persona]
        if block := self.procedural.as_system_block():
            parts.append(block)
        if block := self.semantic.as_system_block(query):
            parts.append(block)
        episodes = retrieve_episodes(self.episodic, query, top_k=2)
        if episodes:
            parts.append("Relevant past sessions:\n" +
                         "\n".join(f"- {e['summary']}" for e in episodes))
        return "\n\n".join(parts)
 
    def chat(self, user_message: str) -> str:
        self.messages.append({"role": "user", "content": user_message})
        if len(self.messages) > 20:
            self.messages = compact(self.messages, keep_recent=8)
        reply = llm(self.messages, system=self._build_system(user_message))
        self.messages.append({"role": "assistant", "content": reply})
        return reply
 
    def end_session(self):
        if not self.messages:
            return
        transcript = "\n".join(f"{m['role'].upper()}: {m['content']}"
                               for m in self.messages)
        self.semantic.add(extract_facts(transcript))   # facts -> semantic store
        self.episodic.archive_session(self.messages)   # session -> episodic store
        self.messages = []                             # clear working memory

Ask it about a PDF library for invoice export in session one, end the session, and in a fresh session two ask "what was your recommendation again?" The agent answers correctly with an empty messages array: the episode came back through retrieval, the user facts through the semantic store, and the answer is short and decisive because of the learned procedural rules.

Best Practices and Pitfalls

What consistently works:

1. Design the write policy first. "Store everything, filter at read time" drowns retrieval in noise. Be selective at write time.
2. Atomic facts beat blobs. One fact per record makes dedup, supersession, and deletion tractable.
3. Layer your memories. Working, episodic, semantic, and procedural each solve a different problem; most real agents need at least three of the four.
4. Score retrieval on relevance, recency, and importance, and tune the weights per domain.
5. Consolidate on a schedule. Periodic reflection passes merge duplicates, abstract patterns, and fix stale facts.
6. Validate every model-supplied path in memory-tool backends.
7. Never store secrets in memory files, and mind PII regulations before persisting user data.

And the traps that catch teams in production:

• Context stuffing. Injecting 50 memories "just in case" degrades answers and costs tokens. Use top-k with a score floor.
• Stale facts. Appending without reconciling means the prompt eventually contains both "lives in Colombo" and "lives in Berlin."
• Cache-busting injection. Providers cache prompt prefixes; keep the stable persona and rules first, and inject volatile recalled memories after the stable prefix so they do not break prefix reuse.
• Shared stores. One memory store per user, always. Key every store by authenticated user ID; one user's memories must never leak into another user's prompt.
• Deletion that does not delete. "Forget what I said about X" must actually remove the record. Under GDPR and CCPA that is a legal requirement, not a nice-to-have.
• Memory as a crutch for bad prompts. If the agent needs the same instruction every session, that is a system-prompt fix, not a memory.

Conclusion

Memory turns a stateless LLM into something that behaves like an agent: it accumulates facts, recalls past work, and adapts its behavior from feedback. The architecture is simpler than it sounds: four stores, a write policy, a read policy, and a scoring function, all of which fit in a few hundred lines of Python before you ever need a vector database.

The full runnable notebook contains every implementation from this article end to end, plus the advanced topics: forgetting and decay, consolidation, swapping keyword matching for embeddings, and multi-user isolation. Run the cells, inspect the memory files it writes, and swap the JSON stores for your database of choice when you are ready for production.

For further reading, the two papers that shaped this field: Generative Agents (Park et al., 2023) for retrieval scoring and reflection, and MemGPT (Packer et al., 2023) for paged, tiered memory management.

An agent without memory answers questions. An agent with memory builds a relationship: with the user, with the codebase, and with its own past work. The difference is a few hundred lines of prompt-assembly logic, and now you know how to write them.