Where Dense Wins

Where we left off

Lesson 14 built a diagnostic for spotting a sparse-favoring query in advance. This lesson builds the mirror: a diagnostic for spotting a dense-favoring one, by directly measuring how much vocabulary a query actually shares with its own correct answer.

The code, piece by piece

STOPWORDS = {"a", "an", "and", ..., "why", ...}

def tokenize(text: str) -> set[str]:
    tokens = re.findall(r"[a-z0-9]+", text.lower())
    return {token for token in tokens if token not in STOPWORDS}

Function words ("why," "does," "my") show up in nearly every document regardless of topic. Counting them as "shared vocabulary" would be Lesson 3's raw-term-frequency mistake all over again, common words looking like a signal when they're really just noise every document has anyway.

def overlap_ratio(query, document_text) -> float:
    shared = tokenize(query) & tokenize(document_text)
    return len(shared) / len(tokenize(query))

What fraction of the query's content tokens appear literally anywhere in the document that actually answers it? High overlap: sparse retrieval has real tokens to latch onto. Low overlap: it doesn't, and dense retrieval has to carry the question on meaning alone.