Failure Modes by Hand

Where we left off

Lessons 10-15 each fixed one specific problem: a word split by a chunk boundary, unrelated topics blended into one chunk, an irrelevant top-k result, a citation-free answer. This lesson deliberately triggers two failures none of those fixes fully solve, so you've actually seen them happen before this course moves on to Advanced techniques.

Failure 1: a question needing two documents at once

"Is the weather station's wind speed reading representative of conditions in the garden?" only has a real answer if you combine garden.md (raised beds sit in a wind funnel, drying faster than the station's average suggests) with weather-station.md (what the station actually measures). With k=1, retrieval only returns garden.md, so the model can only ever partially reconstruct the picture.

This is Naive RAG's most fundamental retrieval limitation: it treats every question as "find the single best matching passage," with no concept of "this question needs several passages combined." Increasing k (as this lesson also shows) helps when the pieces exist as separate, individually-retrievable chunks, but Naive RAG has no way to know a question is multi-part in the first place, or to go looking for a second piece once the first one is found. Later courses in this series (Graph RAG, Agentic RAG) tackle this directly, one with explicit relationships between pieces of information, the other by letting the system decide to retrieve again.

Failure 2: a fact split across a chunk boundary

Lesson 10 already showed fixed-size chunking with no overlap splitting the word "readings" across two chunks. This lesson reuses that exact setup and asks a question about the split fact directly, using only the chunk that contains "re-oiling" (what a similarity search would likely rank highest). The model gets the first half of the answer clearly, and visibly struggles with the second half, the sentence is cut off mid-word in the actual retrieved text.