Keyword Search by Hand

Where we left off

Lesson 1 showed dense retrieval's blind spot: exact tokens like a firmware build number carry almost no "meaning" for an embedding model to place precisely. This lesson builds the other half of Hybrid RAG, sparse retrieval, starting with the simplest possible version: counting words.

Nothing here calls Gemini. Sparse retrieval doesn't need an embedding at all, it works directly on the text.

The code, piece by piece

def tokenize(text: str) -> list[str]:
    return re.findall(r"[a-z0-9]+", text.lower())

Lowercase everything, then split on anything that isn't a letter or digit. "20240115" and a hyphenated part number both survive as clean tokens, punctuation is what splits them apart, not the digits themselves, this is exactly why sparse retrieval doesn't blur a rare ID the way an embedding model does.

def term_frequency_score(query_tokens, doc_tokens) -> int:
    return sum(doc_tokens.count(token) for token in query_tokens)

The entire algorithm: for every word in the query, count how many times it shows up in this document, add it up. No weighting, no notion that some words matter more than others, just raw counting. That simplicity is both this lesson's point and its limitation, see below.