Vibe Coder · Unit 17: RAG and Your Data · Lesson 6 of 6
Checking RAG answers
A RAG helper can fail in two places: finding the right passage, and answering from it. Test both with a fixed set of questions, fix the most common causes, and publish the results.
Read the short notes and pictures.Do everything inside a blue frame.
Two places to fail
A RAG system is two machines in a row, and a wrong answer came from one of them. Looking only at final answers and tweaking the prompt is how teams waste weeks. Split the question in two and the cause is usually obvious.

Step one: retrieval
For each test question, look at the chunks that were retrieved, before the model sees them. Ask one thing: is the information needed to answer in there? Score it yes or no.
Log the retrieved chunks
In n8n, open the run and read the output of the vector store node. With hosted file search, ask for the search results to be included in the response.
Mark each question
Answer present in the chunks, or not.
If not: is it in your documents at all?
If no, it is a document problem. If yes, it is a chunking or search problem.
Step two: generation
Only for the questions where retrieval succeeded, read the answer against the chunks.
Supported
Every claim in the answer appears in a retrieved chunk.
Cited
The answer names where it came from.
Honest
When the chunks do not contain the answer, it says so.
Contained
It adds nothing from the model’s general knowledge.
The usual causes and fixes
Retrieval failures
The answer was never written down. A chunk split a table or a list in half. The user’s words differ from the document’s. Too few chunks were retrieved. Old and new versions are both indexed.
Fixes
Write it. This is the most common fix. Split at headings; keep tables whole. Add keyword search beside vectors (hybrid). Retrieve more, for example ten, not three. Remove superseded documents.
Generation failures
It answered from general knowledge. It merged two unrelated chunks. It was vague to be safe. It ignored a number in the chunk.
Fixes
Instruct: only from the passages. Ask it to quote the sentence it relied on. Ask for the answer first, then the source. Use a stronger model for this step.

Keeping score
Ten questions is enough to start; grow the set whenever a real user asks something that fails. Keep the scores in a table with one row per version of your system. This is the evidence that a change helped, and it is what makes your project credible to anyone reading it.
Your scores are: retrieval 9 out of 10, answers 5 out of 10. Where do you work, and what do you try first?
Score, fix and publish your helper
Score your helper on your test set: retrieval and answers separately. Make at least two improvements, re-scoring after each. Then publish the work to your GitHub repository.
Done when
Link the rag/ folder from your main README.
Keep these
- A wrong answer is a retrieval failure or a generation failure.
- Check the retrieved chunks before you touch the prompt.
- Re-run the same questions after every change and keep the scores.