All video notes
Video notes · Agent memory

AI Agent Memory Architecture: Structured SQL, Semantic Vector Search & Memory Bank

Video by Google Cloud Tech on YouTube · 12 min · uploaded October 2026

Notes and analysis by BrainTube, written from the video. Not affiliated with Google Cloud Tech. Watch on YouTube

In short

Google Cloud's episode argues that agent memory has two kinds: exact facts, which belong in a SQL database, and softer understanding about the user, which belongs in a store you search by meaning, with raw audio and video kept out of memory. It adds a useful timing rule: write facts immediately and extract understanding asynchronously, off the live path. The split is right and widely useful, but the video skips the harder parts: contradictions, stale preferences, user control and facts that are also preferences.

The argument

The video starts from a demo shopping assistant that remembers a customer's wish list and style. Its argument is that short-term session context is like RAM and disappears, so the design question is what to carry into the next session. It sorts long-term memory into facts (exact, with IDs and dates, created and deleted like rows) and understanding (patterns about the person, recalled by meaning). Mixing them in one vector store is the common mistake, because similarity search guesses when the answer must be exact. Raw perception is a separate storage choice. At session start the agent loads both stores; during the session it writes facts with tool calls; after the turn or session it extracts understanding in the background.

Key ideas, with timestamps

  1. 0:20

    The demo

    A voice agent greets a returning user, who asks for an outfit for a customer dinner based on her wish list. The agent names leather ankle boots and a black pump from the list, a red cocktail dress she bought, and a more relaxed alternative. It is a cold open that only works if both kinds of memory loaded before she spoke.

  2. 0:55

    Short-term memory is RAM

    What the agent remembers from five minutes ago lives in the session and vanishes when it ends. The presenter reframes the problem: not whether you can save everything, but what is worth carrying forward. That framing sets up the rest of the episode.

  3. 1:25

    Facts versus understanding

    Facts are exact: a dinner on Thursday, a blue jacket bought. They fit rows and columns with IDs and dates. Understanding is a pattern such as preferring simple over flashy, running warm, or picking dark colors to feel confident. In the video's summary, you look up facts and you recall understanding.

  4. 2:30

    Separate stores

    Facts go in a relational database such as Postgres, MySQL or Cloud SQL, because a question like what is on Thursday needs the exact row. Understanding goes in a vector store, which the presenter stresses is a category: pgvector, AlloyDB or Google's managed Agent Platform Memory Bank. Memory Bank also reads past sessions and extracts and updates memories, which you would otherwise build yourself.

  5. 4:25

    The everything-in-vectors mistake

    Putting purchases and calendar events in a vector store means the agent answers exact questions from similar memories. What did I buy should be a database query; what do I usually like for dinners is a meaning question. This is the clearest rule in the video.

  6. 5:25

    Raw perception is not memory

    Camera frames, audio waveforms and the live stream usually should not be stored as agent memory. What matters is the meaning extracted from them, such as the user calling the dinner formal. You might still store media for replay, audit, debugging or training, but that is a separate decision.

  7. 6:30

    Load at start, write facts now

    At session start the agent reads upcoming occasions, wish list items and profile details from SQL, then searches the vector store for style and comfort preferences. During the session, a request to add a blazer to the wish list is a tool call written immediately, because the user expects it saved now.

  8. 7:55

    Extract understanding off the hot path

    One remark like liking yellow today should not rewrite someone's profile, so understanding needs stronger signals such as a stated reason or a pattern across sessions. Extraction can run per turn, at intervals or at session end, and Memory Bank supports mid-session triggers. Because extraction is slow and voice is latency sensitive, it must run asynchronously and never block audio.

What it gets right

The split is the right first decision. Many agent memory setups start by embedding everything, then struggle when users ask exact questions. Routing by question type (lookup versus recall) is simple to explain and to test: write a few exact questions and a few fuzzy ones, and check each goes to the right store.

Two write speeds. Synchronous writes for facts and asynchronous extraction for understanding is the correct pattern for voice and real-time agents. It also mirrors how other vendors now describe memory: Anthropic's dreaming feature, for example, moves memory consolidation into a background job for the same reason.

Not storing raw media. Keeping frames and audio out of memory saves cost and limits privacy exposure. The extracted meaning is what the agent needs.

What it leaves out

Contradictions and decay. The yellow example covers noise, but not change. If a user preferred dark colors for two years and now wants brighter clothes, which memory wins? A production system needs timestamps on each memory, a rule for newer evidence replacing older, and some way to expire stale patterns.

Facts that are also preferences. An allergy, a shoe size or a dietary restriction is exact and must never be guessed, yet it often arrives in casual speech like a preference. The fact versus understanding line is blurrier than the video suggests, and items that are safety critical should go to the structured store even when they sound soft.

User control. Inferring that someone runs warm or dresses dark to feel confident is personal. The video never mentions showing users what was inferred, letting them correct or delete it, or getting consent. For any consumer agent, that is a requirement, not an extra.

Load-everything at start does not scale. Reading all relevant facts and preferences before the first word works for a wish list. For a large memory, the agent should also retrieve on demand mid-session through a search tool.

One vendor's names. The architecture is general, but the managed option shown is Google's. Equivalent pieces exist elsewhere, and pgvector inside the same Postgres database often keeps both stores in one system.

How this applies to memory across AI tools

This design is for one application remembering its own users. A person using several assistants faces the same split on a different scale: exact things they saved (a specific video, a quote, a document) and a broader sense of what they know about a topic. Search by meaning finds the second; exact lookup by source and date finds the first. Good personal knowledge tools need both.

BrainTube applies this to sources you choose: it saves YouTube videos with their transcripts, articles, PDFs and podcasts into one compiled knowledge base and exposes it as an MCP server, so Claude, ChatGPT, Cursor and other MCP clients can search it.

For the search side, see semantic versus keyword search. For how memory servers plug into assistants, see what an MCP memory server is.

Who should watch it

Watch it if you are designing memory for a customer-facing or voice agent and need a clear rule for where each kind of data goes; the core is 1:25 to 6:30. Watch 7:55 to 10:30 if latency matters, for the timing rules. Skip it if you already separate structured and semantic memory, since there is no code, schema or evaluation here, and skip it if you want personal note-taking advice.

Frequently asked

Should AI agent memory use SQL or a vector database?
Both, for different data. Exact facts like purchases, dates and events belong in a SQL database so the agent gets the precise row. Preferences and patterns belong in a vector store so the agent can recall them by meaning. Putting everything in vectors makes exact answers unreliable.
What is Agent Platform Memory Bank?
In this video, it is Google Cloud's managed option for semantic agent memory. Beyond storing vectors, it reads past sessions, extracts useful memories and updates them over time, and it can be triggered mid-session. A plain vector store such as pgvector needs that extraction logic built yourself.
When should an agent save long-term memories?
Save exact facts immediately with a tool call, because users expect them saved. Extract preferences and patterns asynchronously, after a turn or after the session, so the slow model step never blocks a live conversation. Require repeated or explained signals before changing a stored preference.

Keep videos like this one searchable

BrainTube saves YouTube videos with their transcripts into one compiled knowledge base that Claude, ChatGPT, Cursor and other MCP clients can search.

Start free

Go deeper

More video notes

Are you the creator and want this page changed or removed? Contact us and we will act within 48 hours.