Context Engineering for Agents
Video by LangChain on YouTube · 22 min · uploaded July 2025
Notes and analysis by BrainTube, written from the video. Not affiliated with LangChain. Watch on YouTube
In short
Lance Martin of LangChain sorts context engineering for agents into four moves: write context outside the window, select what to pull back in, compress what stays, and isolate work into separate contexts. The framework is a useful checklist and the examples from Claude Code, Anthropic's research agent, Windsurf and Hugging Face are well chosen. It undersells the costs: every strategy trades tokens for risk, and the talk gives you no way to decide which one to use first beyond running evals.
The argument
The talk starts from Karpathy's analogy that the model is a CPU and the context window is RAM: small, expensive and in need of careful management. Agents make this worse because long tasks and tool calls pile up tokens turn after turn, which leads to the failures Drew Breunig named: poisoning, distraction, confusion and clash. Martin's answer is a taxonomy. Write notes and memories outside the window. Select only the relevant memories, tools and knowledge back in. Compress through summaries and trimming. Isolate through sub-agents, sandboxes and state fields the model does not see. The second half maps each move to LangGraph features, and the closing advice is to measure tokens and run evals before changing anything.
Key ideas, with timestamps
- 1:01
The context window as RAM
He uses Karpathy's operating system analogy: the LLM is the CPU and the context window is limited working memory. Context engineering is the job of deciding what goes into that memory at each step. He lists the inputs as instructions, memories, few-shot examples, tool descriptions, knowledge and tool feedback.
- 2:02
Why agents hit the wall first
Agents run long and call tools, and every tool result stays in the transcript. At 2:33 he cites Drew Breunig's list of context failures and Cognition's claim that context engineering is the main job of people building agents.
- 4:07
Write: scratchpads versus memories
A scratchpad lives for one task; Anthropic's multi-agent researcher saves its plan to memory so it survives if the 200,000-token window overflows. Memories persist across sessions, and he points to ChatGPT, Cursor and Windsurf generating them automatically from user interactions.
- 6:39
Select: three memory types
He borrows human memory types. Procedural memory is instructions, such as rules files or CLAUDE.md, which Claude Code loads in full at start. Semantic memory is facts, which need embedding search or a graph to select from a large store. Episodic memory is past examples, used as few-shot prompts.
- 8:10
Too many tools, and RAG over tool descriptions
He cites a paper showing tool selection degrades after about 30 tools and fails near 100. The fix is to embed tool descriptions and retrieve only the relevant ones. At 8:41 he quotes Windsurf's CEO on code retrieval: chunk along code boundaries, and combine embeddings with grep, knowledge graphs and LLM re-ranking.
- 9:43
Compress: summarise and trim
Claude Code auto-compacts at 95% of the window, as of the July 2025 upload. Anthropic summarises only completed work, and Cognition summarises at the hand-off between agents. Trimming can be as simple as keeping recent messages or as involved as LLM-based pruning.
- 11:16
Isolate: sub-agents, sandboxes and state
Sub-agents each get their own window, which lets the whole system process more tokens. Hugging Face's open deep research runs tool calls as code in a sandbox that keeps state across turns, so heavy objects like images never reach the model. A typed state object with separate fields does the same inside one agent.
- 13:51
LangGraph mapping and the eval prerequisite
Before any of this, he says, trace token usage (LangSmith) and set up evals so you know a compression step did not hurt the agent. The rest maps the four moves to LangGraph: checkpointed state, long-term memory store, the big-tool package, summarisation helpers, supervisor and swarm multi-agent, and E2B sandboxes.
What it gets right
The taxonomy is the right shape. Write, select, compress and isolate covers nearly every trick in production agents, and naming them makes design reviews faster. When an agent goes wrong on a long task, you can ask which of the four is missing.
Selection is where the hard work is. The Windsurf example at 8:41 is the most practical part. Embeddings alone are not enough for code, and good systems combine several retrievers with a ranking step. The same holds for any large personal or team knowledge store.
Measure first. The advice at 13:51 to trace tokens and run evals before changing context is the line most teams skip. Compression that looks fine in a demo can drop the one detail a later step needed.
What it leaves out
Each strategy has a cost the talk does not price.
- Compression is lossy and breaks caches. A summary drops details for good, and the agent cannot know what it lost. Rewriting or trimming earlier messages also changes the prompt prefix, which defeats prompt caching. On long agent runs, cached prefixes are often a large share of cost savings, so trimming can make a run more expensive, not less.
- Multi-agent is not free. Anthropic's own write-up on its research system reported that multi-agent runs use many times more tokens than a plain chat. Cognition's post, which Martin cites only for its summarisation point, argues against splitting work across agents because sub-agents make conflicting decisions without shared context. The talk presents isolation as a clear win when the people it quotes disagree.
- Written memories go stale and can be wrong. Auto-generated memories in ChatGPT or Cursor can store a mistaken fact, and every later session inherits it. Context poisoning, which he lists at 2:33, applies to memory stores as much as to transcripts. Memory needs review and a way to delete entries.
No decision order. A practitioner wants to know where to start. A reasonable order: select first (fewer tools, better retrieval), because it is cheap and reversible; then isolate heavy tool output; compress last, because it is lossy.
About half is product mapping. From 13:51 onwards the talk shows where each idea lives in LangGraph. That is useful if you use LangGraph, but the ideas do not depend on it.
How this applies to memory across AI tools
Martin's write and select steps are what a personal knowledge store does for a person instead of an agent. You write material outside any single chat, and the assistant selects the relevant pieces when a task needs them. The same rules apply: keep the store outside the window, retrieve narrowly, and do not dump whole documents into context.
Most people's context is spread across tools, so the select step has to work in each of them. Exposing a store as an MCP server is one way to make that possible. BrainTube takes that approach, saving YouTube videos with their transcripts, articles, PDFs and podcasts into one compiled knowledge base that Claude, ChatGPT, Cursor and other MCP clients can search over MCP.
For background on the trade-off between window size and stored context, see persistent memory vs the context window and RAG explained.
Who should watch it
Watch it if you build agents and want a shared vocabulary for context problems, or a reading list: the talk links the Anthropic, Cognition, Breunig, Windsurf and Hugging Face posts that shaped the field in mid-2025. The framework itself is 3:35 to 13:51; watch that and skip the rest unless you use LangGraph, where 13:51 to 20:06 shows the matching features. Skip it entirely if you want benchmarks or a cost comparison: there are no measurements here, only patterns.
Frequently asked
- What is context engineering?
- Context engineering is the work of choosing what goes into a model's context window at each step of a task: instructions, memories, tool descriptions, retrieved knowledge and tool results. The term took off in mid-2025, with Karpathy describing it as filling the window with just the right information for the next step.
- What are the four context engineering strategies?
- In LangChain's framing they are write, select, compress and isolate. Write saves information outside the window, select pulls only relevant pieces back in, compress summarises or trims, and isolate splits work into separate contexts such as sub-agents or sandboxes.
- How many tools can an AI agent handle?
- The paper cited in the talk found tool selection getting worse beyond about 30 tools and failing near 100. A common fix is to embed tool descriptions and retrieve only the few relevant to the current task, instead of sending every tool definition each turn.
Keep videos like this one searchable
BrainTube saves YouTube videos with their transcripts into one compiled knowledge base that Claude, ChatGPT, Cursor and other MCP clients can search.
Go deeper
More video notes
- Context Engineering for AI Agents: 4 Failure Modes to Avoid · IBM Technology
- What Is Context Engineering? Why It Matters for AI Agents · IBM Technology
- Is RAG Still Needed? Choosing the Best Approach for LLMs · IBM Technology
- A Helping Hand for LLMs (Retrieval Augmented Generation) - Computerphile · Computerphile
Are you the creator and want this page changed or removed? Contact us and we will act within 48 hours.