BM25 by Hand

Where we left off

TF-IDF (Lesson 4) has a quiet flaw: a term's contribution to the score grows linearly with how many times it appears. A document that mentions a word twice scores exactly twice as high on that term as one that mentions it once, four times scores four times as high, and so on, forever. In practice, that's wrong: a document mentioning "firmware" five times isn't five times more about firmware than one mentioning it once, at some point it's just a longer, more repetitive document.

BM25 (Best Match 25) fixes this with two changes: term-frequency saturation (each additional occurrence of a term matters less than the last) and length normalization (a longer document doesn't win just by containing more words overall). It's the sparse-retrieval algorithm actually used in production search engines, and the direct ancestor of rank_bm25, the library this course graduates to in the Advanced tier.

The code, piece by piece

numerator = tf * (K1 + 1)
denominator = tf + K1 * (1 - B + B * doc_len / avg_doc_len)
score += term_idf * (numerator / denominator)

K1 (1.5 here, the standard default) controls how fast saturation kicks in. B (0.75) controls how much document length matters, B=0 would turn length normalization off entirely, B=1 would make it fully proportional to length. doc_len / avg_doc_len is where a document longer than average gets its score pulled down slightly, one shorter than average gets a small boost, correcting for length instead of rewarding it.