RAG is a retrieval problem, not a prompting problem

A retrieval-augmented system gives a confidently wrong answer. The instinct is to rewrite the prompt. Sometimes that helps at the margin. Usually it’s treating a symptom, because the model can only reason over what retrieval handed it — and retrieval handed it the wrong thing.

Retrieval quality is the ceiling

Generation can’t rise above its inputs. If the right passage never makes it into context, no prompt will conjure it. That makes retrieval the ceiling on answer quality, and it’s where the engineering effort belongs.

Two failure modes dominate:

• Pure vector search captures meaning but misses exact matches — names, codes, the literal string a user typed.
• Pure keyword search nails exact matches but misses paraphrase and intent.

Real corpora need both. Hybrid retrieval blends keyword precision and semantic recall, then re-ranks so the strongest evidence lands in context.

You can’t fix what you can’t see

The deeper problem is that most RAG systems are opaque. When an answer is wrong, no one can say what was retrieved, how it scored, or why a better passage lost. Without that visibility, tuning is guesswork.

We treat retrieval as an observable surface. Every query exposes its candidates and scores, so you can see exactly where it went wrong and tune with evidence — chunking, indexing, hybrid weighting, re-ranking — instead of intuition.

Grounding and honesty

Finally, a grounded system cites its sources and is allowed to say “I don’t know.” That single behavior — refusing to answer when the evidence isn’t there — does more for trust than any amount of prompt engineering.

So before you rewrite the prompt again, ask the upstream questions. What did we retrieve? How did we chunk and index it? Can we even see the scores? Fix retrieval, and the answers follow.