Prompting for Modality-Cited Answers

Where we left off

naive_rag Lesson 15 taught grounded answers that cite a source file: "(according to garden.md)". That's not quite enough information once a citation might point at an image instead of a document, "according to bike-repair.md" is misleading if the actual fact came from derailleur-hanger-diagram.png, a different file entirely, attached because it's the image that document references. This lesson extends the citation instruction to name which modality a fact came from, not just which file.

The code, piece by piece

def build_context_label(record: dict) -> str:
    if record["modality"] == "image":
        return f"an image ({record['source']})"
    return f"a text note ({record['source']})"

One small function that turns a record into a citation-ready label, "an image (derailleur-hanger-diagram.png)" or "a text note (bike-repair.md)". This label, not the raw filename, goes into the prompt's context section, so the model has the modality distinction available to cite directly.

prompt = f"""Answer the question using only the context below.

Rules:
- Every claim must cite its source, including whether it came from a text note or an image, like this: (according to the image derailleur-hanger-diagram.png).
- If the context doesn't contain the answer, say so, don't guess.

Context:
{context}

Question: {query}"""

naive_rag Lesson 15's exact rule structure, one clause added: "including whether it came from a text note or an image." Nothing else about the grounding rules changes, honest citation and admitting ignorance are exactly as important here as they were for text-only RAG, this just extends what "citation" means once a source can be a picture.