How do you give an AI agent long-term memory?

Add a step that reviews conversations and extracts facts, preferences or lessons into a store such as a vector database. On later turns, search that store with the new query and add the relevant items to the prompt. Use a cheap model to decide when extraction or retrieval is needed, to keep latency down.

MemGPT is a research system that treats the model's prompt like limited memory. It keeps fixed instructions, a working context of current facts, and a larger archive the agent can search, and it lets the agent move information between them with function calls. The idea later carried on in the Letta project.

What are AutoGen teachable agents?

Teachability was an ability you could add to an AutoGen agent so it remembered things across sessions. A text analyzer agent turned messages into task and advice pairs or question and answer pairs and stored them in a local Chroma database, then retrieved them for later queries. AutoGen's API has changed since 2024, so check current docs.

All video notes

Video notes · Agent memory

Unlock AI Agent real power?! Long term memory & Self improving

Video by AI Jason on YouTube · 22 min · uploaded April 2024

Notes and analysis by BrainTube, written from the video. Not affiliated with AI Jason. Watch on YouTube

In short

AI Jason argues that agents start every session from zero and shows a fix: a second knowledge agent that reads each conversation, extracts facts and advice into a vector database, and retrieves them on later turns. He backs it with MemGPT, the CLIN research agent and a short AutoGen teachable-agent demo. The extract-store-retrieve pattern is still the core of agent memory, but the video skips how memories get corrected, scoped per user or checked, and the AutoGen code is dated.

The argument

The video's claim is that memory, not a better model, is what lets an agent improve with use. Without it, a user who says they don't eat fish or dislike a news source must repeat that every time, and every edge case found in production needs a human to patch the prompt. The proposed design copies how people learn: notice what matters, encode it, and recall it later. A separate agent decides what in a conversation is worth keeping, writes it to a vector store as structured knowledge, and the main agent retrieves relevant items like RAG before answering. Cheap models can gate both steps to control latency and cost, and research systems show the same loop can produce general lessons, not just preferences.

Key ideas, with timestamps

0:32
Stateless agents repeat mistakes
His examples are a user's diet preference, a news summary agent told twice to avoid one source, and a meeting scheduler he built that met users in shifting time zones. In each case the agent cannot carry what it learned into the next session, so humans become the bottleneck for improvement.
3:17
A knowledge agent beside the main agent
A background agent asks whether a message holds anything worth saving. If so, it calls a create-knowledge function with a type, such as dietary requirement, and a detail, such as no fish. When the user later asks for a meal plan, the retrieved fact is appended to the query so the main agent answers with it in mind.
4:56
Gate it with a cheaper model
Running extraction and retrieval on every message adds latency. He suggests a fast, cheap model, naming Haiku and Mistral, to decide first whether a message needs saving or a query needs retrieval, and skipping the work when it does not. At 5:29 he adds moving memories unused for six months to cheaper cold storage.
5:29
MemGPT's split context
MemGPT divides the prompt into fixed system instructions, a working context and an archive, with a queue manager deciding what gets loaded. His walk-through: the user mentions a boyfriend named James and a birthday, later mentions Six Flags, and the agent recalls they met there. When the user says they broke up, the agent updates James to ex-boyfriend.
7:09
Learning from human escalation
In a support agent he built, questions it could not answer went to a human manager. The answer went back to the user, and the agent also extracted the new knowledge into its own knowledge base, so the next similar question needed no escalation.
7:41
CLIN: lessons that generalise
CLIN, a continually learning language agent, works in a simulated world that follows real physics. Asked to grow an orange, it fails at first but reflects that the kitchen may hold seeds, then succeeds on a later try. Across tasks it merges lessons such as using a lighter on a pot and turning on a stove into a general rule about heat sources.
13:47
AutoGen teachable agents in five minutes
After a sponsor segment for Gamma (10:29 to 13:14) and a mention of Zep as a hosted option, he adds the teachability ability to an AutoGen agent with a local Chroma database and reset set to false so memory persists. He tells it he doesn't eat fish, restarts, asks for a meal plan, and gets one without fish.
18:46
Inside the teachability class
At the end of a session, a text analyzer agent checks each message. Task messages become task and advice pairs; facts become question and answer pairs, by asking how the user would ask for this information if they forgot it. At 20:59, retrieval rewrites the query into a generalised task title, which improves matching.

What it gets right

The extract, store and retrieve loop he describes is still the shape of most agent memory systems. Separating the job of deciding what to remember from the job of answering is sound, and using a cheaper model as a gate is exactly how teams keep memory from doubling latency.

The most useful detail is in the AutoGen code at 18:46. Storing a fact as the question a user would later ask, not as the raw sentence, makes retrieval far more reliable, because the stored text then looks like future queries. Rewriting a new query into a generalised task title at 20:59 does the same from the other side. Both tricks apply outside AutoGen.

The support escalation example is the strongest real use case in the video. Turning each human answer into a stored item is a cheap way to grow a knowledge base from actual gaps.

What it leaves out

Correcting and forgetting. The MemGPT story at 6:36 shows a memory being updated, but the teachable-agent demo only adds. Without update and delete, old preferences and wrong facts pile up and get retrieved as truth. He even frames never forgetting as an advantage over humans at 2:45, then suggests cold storage at 5:29. Decay and relevance are features you have to design.
Scope per user. The demo uses one local database. A real agent needs memory partitioned by user, or one person's preferences leak into another's answers.
Self-editing prompts need checks. He floats agents that rewrite their own prompts and run A/B tests at 1:39. Without an evaluation set, a self-modified prompt can drift or absorb a bad instruction from one conversation. Stored memories are also a route for injected text to persist.
Privacy. Extracting facts like relationships and diet from chats is personal data. Users should be able to see and delete what was saved.
The code has aged. As of the April 2024 upload, teachability was a contributed ability in the older AutoGen API. AutoGen has been substantially reworked since, so check current docs before copying the setup.

How this applies to memory across AI tools

Everything in the video lives inside one agent framework. The preferences and lessons his knowledge agent saves are only available to that agent, which is the same problem users hit with built-in assistant memory: what one tool learns stays in that tool. For the difference between that and a context window, see persistent memory vs context window.

The portable version puts the store behind a standard interface. BrainTube does this for source material rather than chat facts: it saves YouTube videos with their transcripts, articles, PDFs and podcasts into one compiled knowledge base and exposes it as an MCP server, so Claude, ChatGPT, Cursor and other MCP clients can search it. The retrieval lesson from the video carries over: store things in a shape that matches the questions you will ask later. More options are compared in best MCP memory servers.

Who should watch it

Watch it if you build agents and want a clear mental model of long-term memory plus two research examples in one sitting. The concept section from 3:17 to 5:29, the MemGPT example at 5:29 and the teachability internals from 18:46 to 21:32 are the parts worth your time. Skip 10:29 to 13:14, which is a sponsored product tour unrelated to memory. Skip the whole video if you need current framework code or production concerns such as multi-user isolation and memory editing; the demo is single-user and dates from April 2024.

Frequently asked

How do you give an AI agent long-term memory?: Add a step that reviews conversations and extracts facts, preferences or lessons into a store such as a vector database. On later turns, search that store with the new query and add the relevant items to the prompt. Use a cheap model to decide when extraction or retrieval is needed, to keep latency down.
What is MemGPT?: MemGPT is a research system that treats the model's prompt like limited memory. It keeps fixed instructions, a working context of current facts, and a larger archive the agent can search, and it lets the agent move information between them with function calls. The idea later carried on in the Letta project.
What are AutoGen teachable agents?: Teachability was an ability you could add to an AutoGen agent so it remembered things across sessions. A text analyzer agent turned messages into task and advice pairs or question and answer pairs and stored them in a local Chroma database, then retrieved them for later queries. AutoGen's API has changed since 2024, so check current docs.

Keep videos like this one searchable

BrainTube saves YouTube videos with their transcripts into one compiled knowledge base that Claude, ChatGPT, Cursor and other MCP clients can search.

Start free

Unlock AI Agent real power?! Long term memory & Self improving

In short

The argument

Key ideas, with timestamps

Stateless agents repeat mistakes

A knowledge agent beside the main agent

Gate it with a cheaper model

MemGPT's split context

Learning from human escalation

CLIN: lessons that generalise

AutoGen teachable agents in five minutes

Inside the teachability class