All video notes
Video notes · RAG and context

What Is Context Engineering? Why It Matters for AI Agents

Video by IBM Technology on YouTube · 10 min · uploaded August 2026

Notes and analysis by BrainTube, written from the video. Not affiliated with IBM Technology. Watch on YouTube

In short

The video defines context engineering as shaping everything a model sees at inference time, not just the prompt, and argues that more context often makes agents worse. Its checklist of relevance, structure, timing and compression, plus the ongoing work of keeping context current, is a sound framework. It stays abstract: it names no techniques, shows no failure and leaves out how you would measure any of it.

The argument

Prompt engineering asks how to phrase an instruction. Context engineering asks what the whole information environment should be: system prompt, user query, retrieved documents, conversation history and the outputs of tools, APIs and other agents. That shift matters for agents because they work over many steps, call tools and update memory, so the context changes constantly. Bigger context windows do not remove the problem, because performance can fall as context grows. The job is to give the model the right information in the right form at the right moment, and to keep that information current as the task runs.

Key ideas, with timestamps

  1. 0:33

    A definition and a human analogy

    Context engineering is defined as deliberately structuring and optimising what is passed to a model or agent. The presenter compares it to human working memory, which the video says holds three to five items for about 20 seconds, and to how people chunk work into subtasks to cope.

  2. 1:42

    Prompt engineering is one part

    Phrasing the instruction is one piece of a larger job. Agents reason over several steps, use tools, retrieve data and update memory, so what the model sees at each step matters more than one well-worded prompt.

  3. 2:33

    Big windows do not fix it

    Even with windows of two million tokens or more, more context does not mean better results. The video cites context rot, where irrelevant, badly structured or missing data leads to weaker reasoning and hallucinations. The analogy is being handed one page of onboarding notes versus 4,000 pages of Slack messages, emails and notes.

  4. 3:59

    What counts as context

    Context is everything the model sees at inference: the system prompt, the user's query, documents retrieved from a vector database or document store, the history of the conversation, and outputs from tools, APIs or other agents.

  5. 4:48

    Four traits of good context

    Relevance means every item helps the task. Structure means labels and formatting that separate kinds of information. Timing means adding context only when the agent needs it. Compression means summarising or filtering instead of dumping raw data. Together the video calls these steps context processing.

  6. 5:50

    Context management over time

    The ongoing part has five tasks: decide what to keep and drop, keep continuity across turns, prioritise by relevance, recency or importance, handle the lifecycle by updating and invalidating stale data, and keep the whole thing consistent so retrieved pieces fit together.

  7. 7:34

    The clinic scheduling example

    A bare prompt to schedule an appointment leaves the model guessing about clinic rules, the doctor's availability and the patient's history. With context engineering, the system supplies scheduling policies and durations, real-time availability, a morning-only preference, relevant history and open slots from a scheduling API. The model can then propose a specific Wednesday 10 a.m. follow-up that fits.

What it gets right

The framing is correct and widely shared by people who build agents. Most agent failures come from what is in the window, not from wording: a missing fact, a stale fact, or the right fact buried under noise. Saying plainly that a large window is not a strategy is useful, because many teams still respond to bad answers by adding more context.

The split between processing (making each piece usable) and management (keeping the set right over time) is a good way to organise the work. The lifecycle point, that outdated data must be invalidated and not just outnumbered, is the one most teams skip.

The clinic example works because it shows that most of the context comes from systems, not the user: a policy store, a calendar, a patient record and an API.

What it leaves out

  • No techniques. The video names the goals but not the tools. In practice they are retrieval with ranking for relevance, clear sections or tags for structure, tool calls the agent makes when it needs something for timing, and summaries or compaction of old turns for compression. Sub-agents that work in a separate window and return only a result are another common method.
  • No measurement. There is no way to know whether context is good except by testing. Keep a set of tasks with known answers and check what changes when you add, cut or reorder context.
  • Compression loses things. Summaries drop details that turn out to matter later. Keeping the original source and pointing to it is often safer than replacing it.
  • Privacy in the example. Putting a patient's medical history into a scheduling agent's context should follow the minimum needed for the task. The video mentions relevance but not access control, which in healthcare is a legal question as well as a quality one.
  • The memory figure. The working memory numbers mix two separate research findings, one about how many items people hold and one about how long they last. The analogy holds, but the figure is a loose summary.

How this applies to memory across AI tools

Persistent memory is a context engineering problem. A memory system succeeds or fails on the same four traits: does it return the relevant item, in a clear form, at the moment it is needed, without flooding the window. Storing everything is easy. Deciding what reaches the model is the hard part, which is the point of persistent memory versus the context window.

The timing trait argues for memory the agent can query rather than memory pasted in up front. BrainTube saves YouTube videos with their transcripts, articles, PDFs and podcasts into one compiled knowledge base and exposes it as an MCP server, so Claude, ChatGPT, Cursor and other MCP clients can search it when a task needs it. For how retrieval fits into this, see RAG explained.

Who should watch it

Watch it if you are new to agents or need a short, shareable explanation of why context matters more than prompt wording, for example for a non-technical team. The useful core is 4:48 to 7:34, the four traits and the management tasks. Skip it if you already build agents: there is no code, no demo of a failure and no named technique, so you will learn more from a hands-on guide to retrieval or compaction.

Frequently asked

What is the difference between prompt engineering and context engineering?
Prompt engineering focuses on how you word an instruction. Context engineering covers everything the model sees when it answers: the system prompt, retrieved documents, conversation history and tool outputs. Prompt engineering is one part of it, and for multi-step agents the rest usually matters more.
Do large context windows make context engineering unnecessary?
No. Models can perform worse as context grows, especially when much of it is irrelevant or badly organised, an effect often called context rot. A large window gives you room, but you still need to choose what goes in and in what form.
What makes context good for an AI agent?
The video lists four traits: relevance to the task, clear structure, timing so information arrives when needed, and compression so raw data is filtered or summarised. Over time the context also has to be kept current, with stale information removed.

Keep videos like this one searchable

BrainTube saves YouTube videos with their transcripts into one compiled knowledge base that Claude, ChatGPT, Cursor and other MCP clients can search.

Start free

Go deeper

More video notes

Are you the creator and want this page changed or removed? Contact us and we will act within 48 hours.