.svg)
An AI agent can remember every message in the current conversation and still be almost useless when you come back tomorrow. AI agent memory is not a storage problem. A useful memory system has to decide what deserves to survive, how long it should survive, where it belongs, when it should come back, and whether the agent is allowed to use it in the current task. Remembering more is not automatically better memory; in many cases, it just means giving the model more noise to sort through.
Underneath the architecture there are three decisions: what from an interaction is worth writing down, what of it comes back in a later session, and who the agent may show it to. This guide follows all three, from what AI agent memory means through storage and retrieval to permissions, with a practical AI agent memory architecture diagram and examples from Mercury Agent's public documentation.
π― Quick answer
AI agent memory architecture is the system around an AI model that decides what information should be kept, how it is stored, how it survives beyond the current session, and which parts should be retrieved into a future task.
- Working context. What the model can use right now, before any of the decisions below apply.
- Persistent memory. The first decision: what is worth keeping beyond the current session, and with what scope, source and timestamp.
- Retrieval and filtering. The second decision: what comes back into a later task, a small subset rather than everything stored.
- Permissions and scope. The third decision: which memory, files, tools, users or services the agent may reach at all.
A useful memory system turns conversations, actions, preferences, and outcomes into a smaller set of reusable information, then brings back only what is relevant. Memory is generally implemented by the agent system around the model rather than by assuming the base model permanently carries personal state between independent runs.
What is AI agent memory?
At the simplest level, agent memory is information an AI agent can use beyond the immediate prompt it is answering. That sounds straightforward until you separate the different systems involved.
A language model has a context window: the tokens it can see while producing its next response. An agent fills that window with system instructions, recent messages, tool results, retrieved documents and memory items. Once those tokens are gone, the base model carries nothing into a separate future session.
Persistent memory therefore lives outside the model itself, in a store the agent system controls. When a later task arrives, that system decides which of it should be loaded back into the model's working context.
So when someone asks, "What is agent memory?", the practical answer is not that the AI simply remembers things. The agent system stores selected information outside the model and makes relevant parts available again when they are useful.
This is also why a chat transcript is not memory. A transcript is a record, memory is a selection. An hour of debugging CI runs to hundreds of lines, and the part worth carrying into next week might be one sentence: "Project convention: tests live in /tests." Context-engineering guidance for long-running agents pushes the same way, towards selective retrieval, compaction and structured notes, because irrelevant context dilutes what actually matters. Storing everything and retrieving everything is the architecture that ignores both ends of that.
How AI agent memory architecture works
A practical memory architecture has two separate flows, and they carry the first two decisions: a write path, where information becomes memory, and a read path, where stored information is brought back for a later task.
On the write side, an interaction produces potential memory candidates. That could be a preference, a decision, an important event, a project convention, a relationship, a successful procedure, or another piece of durable information. The system then needs a memory decision: should this be ignored, kept only for the current task, persisted, merged with an existing memory, or used to replace a stale or conflicting one?
Whatever is kept gets stored with more than its text: scope, source and a timestamp decide later whether it still applies.
The read path starts when a new task arrives. The agent derives what it needs, searches the appropriate memory scope, retrieves candidates, filters or ranks them, and puts only the useful subset into the current context. The goal is not to maximize how much memory reaches the model. It is to maximize how useful the context is for the task in front of it.
AI agent memory architecture diagram
A useful AI agent memory architecture diagram shows more than a database sitting next to an LLM. Three things have to be visible: memory is created selectively, the store sits outside the model's context, and retrieval is filtered before anything comes back.
Read the diagram as two columns over one store. On the left, an interaction narrows down to a single decision: ignore it, keep it for this task, or persist it with scope, source and a timestamp. On the right, a later task opens back up, but everything passes the permission gate first, and only the set that survives filtering reaches the model. The dotted line is the read itself, and the store at the bottom is the only thing the two columns share. Different agent systems implement these layers differently, so read it as a shape rather than any one product's internals.
AI agent memory systems: the five types
Agents rarely have one memory. The types of agent memory below each have a different lifetime and a different way of going wrong, and the table is the short version of what to expect from each. The sections after it explain what goes into every layer.
The risk column is the one to read twice. Most complaints about agent memory, that it forgot something, that it dragged up something old, that it saw something it should not have, come from one of those five lines rather than from the size of the store. Picking a bigger database does not touch any of them.
This is a useful taxonomy, not a universal standard. Semantic, episodic, and procedural memory are common ways to describe long-term agent memory, while short-term or working memory usually refers to the state needed for the task in front of the agent.
Working memory
Working memory is the information needed for the task happening now: the user's current instruction, recent conversation turns, tool results, intermediate calculations, or the active plan. It can be critical to the next action while having almost no value two weeks later.
Episodic memory
Episodic memory captures things that happened. A deployment failed. A user rejected one approach. The agent previously tried a particular fix. A project moved from one library to another. Episodic memories help answer questions such as, "What happened last time?"
Semantic memory
Semantic memory stores reusable facts or knowledge: a user prefers concise reports, a repository uses pnpm, a client belongs to a specific project, or a team avoids editing generated files. This is often the layer people mean when they say an agent "knows" the user or project.
Procedural memory
Procedural memory is about how to do something. A release flow might be "run tests, build, bump the version, then tag." A recurring support task may have an established sequence that worked well before. In agent systems, procedures can be represented through instructions, skills, policies, examples, or other reusable behavior rules.
External knowledge is adjacent to memory
A Google Drive folder, code repository, CRM, internal wiki, or database may be information the agent can retrieve, but that does not necessarily make it agent memory. The external source has its own lifecycle and may remain authoritative outside the agent.
"Remember that our test folder is /tests" is memory. "Read the current test configuration from the repository" is retrieval from an external source. Both can enter context, but they are not the same system.
How to persist memory across agent sessions
The first decision, what is worth keeping, becomes concrete here. To persist memory across agent sessions, the agent needs durable state that survives when the current model context disappears. During or after an interaction, the agent identifies information worth retaining, saves it with enough metadata to find and interpret later, and retrieves relevant items when a future session needs them.
candidate = extract_memory(current_interaction)
if should_persist(candidate):
memory_store.save(
value=candidate.value,
scope=candidate.scope,
source=candidate.source,
timestamp=now()
)
# A future session
candidates = memory_store.search(current_task)
memories = filter_by_scope_and_relevance(
candidates,
user=current_user,
project=current_project
)
context.add(memories)This is conceptual pseudocode rather than any product's implementation. The storage layer could be SQL, a document store, files, a search index, a vector system, a graph, or a mixture of these. What matters is that the state exists independently of the current context window.
Long-running agent systems use persistent notes for the same reason: a later context window needs something durable to bridge the gap from earlier work. Persistence alone does not solve memory. A system that saves 50,000 items and cannot decide which five matter now has built an archive.
Memory is state on disk, and it sits next to the process that wrote it. Whatever design you pick, the notes from Tuesday have to still be on that machine on Friday, and the machine has to be up when the next session opens. On Atomic Bot, Mercury Agent runs on its own instance around the clock, and what it wrote during earlier work comes back when you start the next session.
Where to store agent memory
The store is an engineering choice, not a definition. Five shapes cover most systems.
Most working systems use two: something durable and structured for the memories themselves, and an index for finding them. Adding the index first is the common mistake, because search over a store with no scope, no provenance and no update rules returns confident nonsense faster.
How retrieval works
Retrieval carries the second decision, what comes back. A new task creates some form of retrieval signal. The system may use the user's message, current project, active goal, referenced people, recent events, or other task state to look for likely matches.
Different systems may rank candidates using semantic similarity, keyword or metadata matches, recency, importance, relationship links, explicit user or project scope, or combinations of these.
Once candidates are found, the work is not over. An old preference may conflict with a newer one, two memories may be duplicates, a fact may belong to another project, and the task may have budget for only a handful of items.
Read it top down: every stage below the gate removes candidates rather than adding them. Authorization sits first on purpose, so a memory that ranks highly but belongs to another user or project never reaches the ranking stage at all. What comes out at the bottom is a handful of items, and the stages above it are where the rest were dropped.
Why AI agent permissions matter in memory architecture
The third decision is who the memory may be shown to, and it gets heavier as the other two work better. Memory becomes more useful as an agent learns more about the user. It also becomes more sensitive. An agent may eventually know project details, people you work with, personal preferences, internal decisions, file locations, workflows, and the services you use. If the same agent can also read files, execute commands, call APIs, or send messages, memory and permissions become part of the same trust problem.
They should not, however, be treated as the same thing. Memory scope answers questions such as: whose memory is this, and does it belong to this user, project, agent, or shared team? Tool permission answers a different question: what is the agent allowed to do now?
A system can get one right and the other wrong. A testing sub-agent may need project conventions but have no reason to see personal notes. An agent might correctly remember that a production credential exists without ever putting the credential itself into model context.
This is why AI agent permissions belong in the architecture rather than in a filter bolted on after retrieval: unauthorized information should never reach model context just because it ranked highly.
One worked example of the action side: Mercury Agent documents an Ask Me mode that prompts before file writes, shell commands and scope changes, an Allow All mode that auto-approves for the session, an allowlist for safe read-only commands, and skills that declare the tools they need. That covers what the agent may do. Whether the same system also scopes memory per user or project is a separate question, and one worth asking of any product you evaluate.
Remembering something and being allowed to act on something are different capabilities.
Agent memory vs. a "second brain"
"Second brain" captures the experience users want: tell the agent something once, and it stays useful later. Storage alone does not get you there. The system also has to form memories, retrieve them, handle contradictions, and let the user control what survives.
Mercury Agent is one worked example. Its documentation describes a Conscious Mind for active working memory and a Subconscious Mind for persistent memories retrieved when contextually relevant, conflict resolution where a higher-confidence memory supersedes a contradictory one, hourly consolidation of profile summaries, person tracking, and SQLite storage on the machine where the agent runs. Users can browse, search, edit, promote or prune the stored items and inspect related people, goals and a relationship graph. That last part is what most systems skip, and memory you cannot inspect is memory you cannot correct.
One deployment detail is worth taking as a question rather than an answer. The store sits with the agent, while a shared pool can be searched across several agents after that first retrieval. Any shared pool raises the same thing to check before you turn it on: which agent is allowed to read whose memory?
The metaphor sells the experience. The engineering questions underneath it stay the same.
Practical example: remembering a project rule across sessions
Consider a developer working on a repository called Atlas. During one session, the developer tells the agent: "For Atlas, never edit generated files under /src/generated, and run pnpm test before opening a PR."
That statement contains two pieces of project knowledge with clear future value. A memory system might represent them conceptually as:
Project: Atlas
Constraint: Do not modify /src/generated
Validation: Run pnpm test before opening a PR
Source: User instruction
Created: 2026-09-24The rest of that conversation is disposable, those two constraints are not. Two weeks later the user opens a new session and asks, "Fix the login regression in Atlas." The project name triggers retrieval of Atlas-scoped memories, and both constraints change how the task gets done, so they belong in working context.
Notice what memory does not decide. It grants no permission to edit files, it does not decide whether shell commands may run, and it does not override a new explicit instruction. Those are separate concerns.
If the user now says, "Generated output is broken too. Update it this time," the current instruction should take precedence over the older stored constraint, or at minimum trigger conflict handling instead of silently following stale memory.
Common agent memory architecture mistakes
The same failures turn up across very different stacks, and most of them are a skipped decision rather than a bug in the tooling.
Useful memory requires forgetting, filtering, updating, and control as much as it requires storage.
FAQ
What is agent memory?
Agent memory is information an AI agent can retain or retrieve beyond the immediate prompt it is answering. It can include recent task state, past events, user preferences, project facts, relationships, or reusable procedures. Short-term memory usually supports the current thread, while persistent or long-term memory survives across sessions.
How do you persist memory across agent sessions?
To persist memory across agent sessions, identify which information should survive the current interaction, store it in durable storage with useful metadata, then retrieve and filter it when a future task makes it relevant. Persistence alone is not enough; the system also needs scope, update rules, retrieval, and a way to handle stale or conflicting information. The store also has to stay reachable: on Atomic Bot each agent gets its own instance that stays on, so Mercury Agent reads back its earlier notes in the next session with no export and no import step.
What is the difference between agent memory and context?
Context is the information currently available to the language model during inference. Agent memory is information stored or managed outside that immediate context so it can be reused later. Retrieved memory becomes context only when the agent system loads it into the current task.
Why are permissions important for AI agent memory?
Permissions matter because remembered information can be sensitive or scoped to a particular user, project, or organization. A strong design prevents unauthorized memory from entering the wrong context and separately controls what tools or external systems the agent can use. Memory access and tool permissions are related security concerns, but they are not the same mechanism.
Is agent memory the same as a vector database?
No. A vector database can be one component of an agent memory system because it can help retrieve semantically similar information. But memory architecture also includes deciding what to store, attaching metadata and scope, managing updates and conflicts, applying permissions, selecting relevant memories, and deciding what enters context. An agent can implement useful memory without using a vector database at all.
Get started
You do not need a complicated stack. Answer the three decisions and the rest follows: write down what will still matter next week, bring back only what changes the current task, and keep the scope of both under control.
Mercury Agent is one example of a system designed around this idea, with a documented Second Brain for persistent context alongside explicit tool-permission controls. It runs on Atomic Bot from $19 a month: 4 vCPU, 8 GB RAM, 80 GB SSD and $5 of model usage included, with 100+ models through OpenRouter or your own API key. Start an instance, pick Mercury Agent from the list, and it runs in a browser tab. No terminal.
β
