Is RAG still needed with million-token context windows?

For small, bounded sets of documents, often not: pasting them into a long context is simpler and avoids retrieval misses. For large or frequently changing collections, retrieval is still needed because the data will not fit and reprocessing it on every query is expensive. Many systems combine both.

What is silent failure in RAG?

Silent failure is when the answer exists in your data but the retrieval step does not return it, so the model never sees it. The model then answers from the wrong chunks with no sign anything was missed. Hybrid keyword and vector search, reranking and agent-driven search reduce it.

When is long context better than RAG?

Long context works best when the material is bounded and the question needs reasoning across all of it, such as reviewing one contract, summarising a book or finding what is missing between two documents. Chunk-based retrieval only shows the model fragments, which can hide gaps and contradictions.

All video notes

Video notes · RAG and context

Is RAG Still Needed? Choosing the Best Approach for LLMs

Video by IBM Technology on YouTube · 11 min · uploaded March 2026

Notes and analysis by BrainTube, written from the video. Not affiliated with IBM Technology. Watch on YouTube

In short

IBM's video weighs RAG against putting whole documents into a long context window, with three arguments for each. Its conclusion is sound: long context for a bounded set of documents that needs global reasoning, retrieval for large and changing collections. It frames the choice as either-or, though, and most real systems now combine the two or let an agent search with tools.

The argument

Models only know their training data, so anything recent or private has to be injected into context. There are two ways to do it. RAG chunks documents, embeds them in a vector database and retrieves the closest chunks per question. Long context skips all that and pastes the documents into a window that can now hold a million tokens or more. Long context wins on simplicity, avoids retrieval misses and can reason across whole documents. RAG wins on cost for repeated queries, on focus when the context would be huge, and on scale, because enterprise data is far larger than any window. The answer depends on whether your data set is bounded.

Key ideas, with timestamps

0:52
The RAG pipeline in one pass
Documents such as PDFs, code files or books are chunked, turned into vectors by an embedding model and stored in a vector database. At question time a semantic search returns the closest chunks, which are added to the context next to the prompt. The whole approach depends on retrieval finding the right chunks.
2:50
Why long context is now an option
Early models had windows of around 4K tokens, so RAG was the only way. Some current models take a million tokens or more, roughly 700,000 words by his estimate, enough for all of The Lord of the Rings plus The Hobbit. That raises the question of whether RAG is now needless complexity.
4:11
The no-stack stack
A production RAG system needs a chunking strategy (fixed size, sliding window or recursive), an embedding model, a vector database, a reranker and a process to keep vectors in sync with the source. Long context removes all of it: get the data and send it.
5:06
Silent failure
Semantic search is probabilistic, so it can miss the relevant document. The answer existed in the data but the model never saw it, and nothing signals the miss. With long context there is no retrieval step to fail.
6:08
The whole book problem
Asked which security requirements were left out of a release, RAG retrieves chunks about security from both the requirements doc and the release notes, but it cannot retrieve the gap between them. Spotting an omission needs both documents in full.
7:43
Rereading costs
A 500-page manual is about 250K tokens, and long context reprocesses it on every query. RAG pays the processing cost once, at indexing. Prompt caching helps for static data, but for content that changes often you pay the full cost on each request.
8:36
Needle in a haystack
Being in the context window does not mean being used. As context grows toward 500K tokens, attention gets diluted, and a single paragraph in the middle of a 2,000-page document may be missed or confused with nearby text. Retrieving the top five chunks hands the model only the needles.
9:42
Bounded versus infinite data
Enterprise data lakes run to terabytes or petabytes, so a retrieval layer is unavoidable at that scale. His rule at 10:19: bounded data that needs global reasoning, such as one legal contract or one book, suits long context; enterprise knowledge needs a retrieval store.

What it gets right

The bounded versus unbounded rule is the right first question, and it is more useful than most "RAG is dead" takes. If the material fits comfortably and you will ask few questions of it, paste it in. If it is large, growing or queried constantly, you need retrieval.

Naming silent failure is valuable. Retrieval misses are the most common reason RAG answers are wrong, and they are hard to see because the model answers confidently from whatever it was given. The gap example at 6:08 is also a good test case: questions about what is missing, or what changed between two documents, defeat chunk retrieval.

The cost argument is correct too. Reprocessing a 250K-token manual per question adds up, and caching only helps when the content is stable.

What it leaves out

Hybrids. The two approaches combine well. A common pattern retrieves whole relevant documents rather than small chunks, then lets a long context window reason over them. That fixes the gap example: retrieve the requirements doc and the release notes in full, then compare.
Agentic search. A third option is to give the model search tools (keyword search, file reads, filters) and let it look, read and look again. It can notice a miss and try another query, which addresses silent failure without loading everything.
Better retrieval. Silent failure is partly a choice of method. Adding keyword search alongside vectors, using metadata filters and reranking all reduce misses. The video lists a reranker as overhead but not as a fix.
Access control and freshness. In companies, a big reason for retrieval is permissions: each user should only see documents they are allowed to. Retrieval can filter per user; a pasted context cannot. Keeping data current is also easier when you reindex one changed file than when you rebuild a giant prompt.
"The only viable warehouse". The closing line overstates the vector database. Large collections need a retrieval layer, but that layer can be keyword search, SQL or a mix, not only vectors.

How this applies to memory across AI tools

Personal knowledge sits between the video's two cases. One person's saved videos, articles and papers grow steadily and soon exceed any window, but most questions only need a handful of sources, read in full. That points to retrieval that finds the right whole sources first, then lets the model read them closely. See RAG explained and persistent memory vs the context window for the trade-offs.

BrainTube follows that pattern: it saves YouTube videos with their transcripts, articles, PDFs and podcasts into one compiled knowledge base and exposes it as an MCP server, so Claude, ChatGPT, Cursor and other MCP clients can search it and pull in only what a question needs.

Who should watch it

Watch it if you are deciding how to give a model access to documents and want a balanced 11-minute overview with a clear rule of thumb. It suits engineers and technical managers. The arguments start at 4:11; the silent failure (5:06) and gap example (6:08) are the most useful parts, and the decision rule is at 10:19. Skip it if you already build retrieval systems, since it does not cover hybrid search, reranking in depth, evaluation or agent-driven search.

Frequently asked

Is RAG still needed with million-token context windows?: For small, bounded sets of documents, often not: pasting them into a long context is simpler and avoids retrieval misses. For large or frequently changing collections, retrieval is still needed because the data will not fit and reprocessing it on every query is expensive. Many systems combine both.
What is silent failure in RAG?: Silent failure is when the answer exists in your data but the retrieval step does not return it, so the model never sees it. The model then answers from the wrong chunks with no sign anything was missed. Hybrid keyword and vector search, reranking and agent-driven search reduce it.
When is long context better than RAG?: Long context works best when the material is bounded and the question needs reasoning across all of it, such as reviewing one contract, summarising a book or finding what is missing between two documents. Chunk-based retrieval only shows the model fragments, which can hide gaps and contradictions.

Keep videos like this one searchable

BrainTube saves YouTube videos with their transcripts into one compiled knowledge base that Claude, ChatGPT, Cursor and other MCP clients can search.

Start free

Is RAG Still Needed? Choosing the Best Approach for LLMs

In short

The argument

Key ideas, with timestamps

The RAG pipeline in one pass

Why long context is now an option

The no-stack stack

Silent failure

The whole book problem

Rereading costs

Needle in a haystack

Bounded versus infinite data