Where Naive RAG Breaks: Common Failure Modes

Three ways the baseline fails

Naive RAG's fixed decisions turn into visible failures once the data or the question stops being simple. Recognizing which failure you are looking at is what points you to the right fix in a later lesson, so this lesson is worth slowing down for.

Failure 1: chunk boundaries cut through meaning

Fixed-size chunking (say, every 500 tokens) has no idea where a sentence, a table row, or an argument actually ends. A chunk boundary that lands in the middle of a critical sentence can leave both halves individually meaningless to an embedding model, so neither half ranks highly for a question that needed the whole sentence.

Failure 2: the top-k list is relevant-sounding but wrong

Cosine similarity measures topical closeness, not correctness or specificity. A chunk can be about the right subject while still being the wrong chunk, an old pricing page instead of the current one, a competitor's document instead of yours, and naive RAG has no mechanism to notice this. It hands over whatever ranked highest, confidently.

Failure 3: single-hop retrieval cannot answer multi-hop questions

"Which of our vendors raised prices after switching CEOs?" cannot be answered by any single chunk, it requires connecting a fact in one document (who the CEO changes were) to a fact in another (which vendors raised prices) and reasoning across both. A single top-k retrieval pass over one query embedding was never built to do that; it looks for one chunk that already contains the whole answer, and none does.

These three failures map directly onto three different fixes: better chunk and query handling (Advanced RAG, Lessons 6 to 7), composable retrieval strategies (Modular RAG, Lesson 8), and, for the multi-hop case specifically, explicit relationship modeling (Graph RAG, Lesson 12) or an agent that can retrieve more than once (Agentic RAG, Lesson 17).