All video notes
Video notes · Agent memory

You Can Learn AI Agent Memory System In 12 Min | Semantic & Episodic Memory, RAG, Vector Database

Video by Sean‘s AI Stories on YouTube · 12 min · uploaded June 2026

Notes and analysis by BrainTube, written from the video. Not affiliated with Sean‘s AI Stories. Watch on YouTube

In short

The video draws one diagram of an agent memory system: a working context fed by procedural memory in skill files, semantic memory of durable facts, and episodic memory of dated events, with a summariser that distils episodes into facts every so many conversations. The layering is a good mental model for beginners. It stops before the hard parts: updating or deleting facts that change, deciding what to store, and checking that retrieval works.

The argument

Sean's argument is that a chat call on its own is an ephemeral session. The model sees a system prompt, the current chat history and your question, and nothing survives when the session ends. To make an agent remember, you feed its working context from three stores. Procedural memory says how to behave and lives in text files such as a skill file. Semantic memory holds durable facts, such as what a company sells or who a customer is, and is fetched with RAG from a vector store. Episodic memory logs dated events and past conversations, also in a vector store. Because searching the full event log every time is slow and expensive, a gate triggers a summariser after a set number of chats, and it distils recent episodes into compact facts. He presents this as the pattern behind the memory features in ChatGPT and Claude.

Key ideas, with timestamps

  1. 0:41

    Working memory sits before the model

    A question does not go straight to the LLM. It first lands in a working memory, or current context, built from the user prompt, the chat history and the system prompt. Everything the model knows about you in that moment has to be placed here.

  2. 2:44

    A session is ephemeral

    Using an e-commerce support bot as the example, he points out that the session knows nothing about stock levels, past purchases, taste or old complaints. Without a database, the chat history is the only memory, and it is gone when the conversation ends.

  3. 3:24

    Procedural memory as skill files

    Procedural memory is how the agent should act, for example always apologising and staying polite with an angry customer. He compares it to teaching an employee a habit and says it is usually stored as text, such as a skill markdown file loaded into context.

  4. 4:45

    Semantic memory for durable facts

    A new online store is unknown to the base model, so its products, branding and regular customers have to be stored and fetched. He explains this as RAG over a vector store with top-K search, and argues for selective fetching because a decade of company data will not fit, or will be slow and less accurate if it does.

  5. 6:48

    Episodic memory as a dated log

    Episodic memory records events with times: who bought what and when, when an item was delivered, when a complaint was filed. The reply path writes each conversation back into this store, so the log keeps growing. Like semantic memory, he puts it in a vector store searched by similarity.

  6. 8:52

    Why assistant memories stay short

    He notes that the editable memories in ChatGPT and Claude are brief but keep changing, for example after a small pivot in your company. His explanation is that assistants summarise activity into compact facts at a modest frequency, which saves tokens and speeds up answers compared with searching the whole history.

  7. 9:34

    The consolidation gate

    Summarising every new message into facts would just store everything twice. So a gate waits for a set amount of activity, such as 20 conversations or 100 events, then passes it to a summariser agent that distils it into semantic memory. This is the most useful design detail in the video.

What it gets right

The split into procedural, semantic and episodic memory is a sound way to reason about agent state, and it maps onto what real systems do. Rules and skills belong in files you can read and version. Facts about users and the business belong in a store you can query. Events belong in a log with timestamps.

The consolidation gate is the right instinct. Most beginner diagrams either save every message or save nothing. Batching episodes and distilling them into facts is how you keep the fact store small enough to load cheaply while keeping the full log for when detail matters.

He is also right that selective retrieval beats dumping everything into context, on cost, speed and accuracy.

What it leaves out

  • Updating and forgetting. The summariser only adds facts. When a customer moves house or a company pivots, the old fact has to be replaced or marked stale, or retrieval will return both. Handling conflicts is most of the work in a real memory layer.
  • What not to store. There is no write policy. Without one, the log fills with small talk and the summariser distils noise. Privacy belongs here too: customers' purchase histories and complaints are personal data, and users need a way to see and delete what is kept.
  • Episodic memory is not just a vector store. Questions like "when did this customer last complain?" are about time and identity, which a database filter answers better than similarity search. Most production systems combine vector search with keyword search and metadata filters.
  • Evaluation. Nothing shows how to tell whether the right memory was retrieved. Top-K with K set to five is a starting point, not a design.
  • Context window size. He says most models are roughly 1 million tokens. As of the June 2026 upload, some large models offered that, but many common models had smaller windows. The general point, that more context is slower and less accurate, holds either way.
  • Assistant internals. The claim that ChatGPT and Claude all work this way is a reasonable guess, not something the video shows. Their memory implementations are only partly documented.

How this applies to memory across AI tools

The diagram assumes one agent owns its memory stores. For a person who uses several assistants, that means several separate copies of semantic and episodic memory that never meet. ChatGPT's facts about you are not visible to Claude, and neither sees your notes. See persistent AI memory across tools for that problem, and persistent memory vs the context window for the working-memory side.

One practical answer is to keep shared reference material outside any single assistant and serve it over MCP. BrainTube does this for sources: it saves YouTube videos with their transcripts, articles, PDFs and podcasts into one compiled knowledge base and exposes it as an MCP server, so Claude, ChatGPT, Cursor and other MCP clients can search it. That covers the semantic layer for what you have read and watched. Procedural rules and per-assistant chat memory still live with each tool.

Who should watch it

Watch it if you are new to agent memory and want one picture that separates working context from stored memory. The core is 3:24 to 7:29 (procedural, semantic and episodic memory) and 9:34 to 10:57 (the consolidation gate). Skip the first two minutes if you already know what a system prompt and chat history are. Skip the video if you are building a memory layer and need code, schemas or a way to handle changing facts; it is a conceptual sketch on one diagram, with no implementation.

Frequently asked

What is the difference between semantic and episodic memory in AI agents?
Semantic memory holds durable facts, such as a user's profile or what a company sells, that stay true over time. Episodic memory holds dated events, such as past conversations, purchases or complaints. A common design summarises episodes into semantic facts periodically so the agent can load a small set of facts instead of searching the whole history.
What is procedural memory for an AI agent?
Procedural memory is how the agent should behave: rules, routines and skills. It is usually stored as text, such as a system prompt or a skill file in markdown, and loaded into the context when relevant. It describes behaviour rather than facts about the user.
Do AI agents store memory in a vector database?
Often, but not only there. Vector stores are good for finding text with similar meaning. Questions about dates, identities or exact values are better answered with normal database filters or keyword search, so many systems combine both.

Keep videos like this one searchable

BrainTube saves YouTube videos with their transcripts into one compiled knowledge base that Claude, ChatGPT, Cursor and other MCP clients can search.

Start free

Go deeper

More video notes

Are you the creator and want this page changed or removed? Contact us and we will act within 48 hours.