Agent memory: context windows, RAG, and long-term state
An agent's working memory is the model's context window — the text it can see right now. It is limited, and it starts empty every session. Longer memory is built on top: retrieval (RAG) fetches knowledge when needed, and outside stores the agent can write to hold things for later.
AI models forget everything between calls. Whatever an agent “knows” right now is either in its context window or was fetched into it. So memory is something builders add — not something the model has.
Start with the limit. The context window is all the text a model can see while writing a reply — its working memory. The instructions, the conversation, every tool result, and the reply itself all take up space in it. And more is not always better: as the window fills up, the model gets less accurate at recalling what is in it. So choosing what goes in matters as much as how much fits [1].
A common way to describe this borrows from human memory. Short-term memory is whatever sits inside the window. Long-term memory is an outside store the agent searches when it needs something [2]. RAG (retrieval-augmented generation — fetching useful documents into the window on demand) is the read-only half. You split your documents into small chunks, then pull in the relevant ones for each task. But splitting carelessly loses meaning. That is why Anthropic's contextual retrieval, which adds a short note of context to each chunk before indexing it, cut failed retrievals by 49% [3].
The context window is working memory; everything longer-lived is a tool.
Can an agent write to its own memory?
The read-write half is newer. Here the agent gets a store it can choose to write to — notes, preferences, task progress — and read again in later sessions. A research project called MemGPT compared this to how a computer's operating system works: treat the context window like fast-but-small main memory, and let the model move information in and out of bigger outside storage by itself [4]. The same idea now ships as memory tools in real agent products.
When is each layer worth it?
Add them in this order. A well-chosen context window alone covers most tasks that fit in one session. Add retrieval when the agent needs more knowledge than the window should carry. Add writable long-term memory only when tasks truly stretch across sessions. It brings a new way to fail — stale or wrong memories pile up — so add it after evaluation exists, not before.
Sources
- Claude docs — Context windows (platform.claude.com)
- Lilian Weng — LLM Powered Autonomous Agents, 23 Jun 2023
- Anthropic — Introducing Contextual Retrieval, 19 Sep 2024
- Packer et al. — MemGPT: Towards LLMs as Operating Systems, arXiv 2310.08560, Oct 2023
Frequently asked questions
How do AI agents remember things?
Within one session, everything lives in the model's context window — its working memory. Across sessions, agents use retrieval (fetching helpful documents into the window) and outside stores the agent writes to itself. The model keeps nothing between calls. Memory is built around the model, not into it.
What is the difference between RAG and agent memory?
RAG is read-only: it looks things up in a fixed set of documents and pulls the useful parts into the window when a question comes in. Agent memory is read-write: the agent chooses what to save — notes, preferences, task progress — and reads it back in later sessions. Most real-world agents use both.
What is a context window?
All the text a model can see while writing a reply: its instructions, the conversation, tool results, and the reply itself. It has a fixed size, and the model gets less accurate as it fills up. So what you put in matters as much as how big the window is.