AI Folks
  • Courses
  • Web Of AI
  • Pricing
  • Enterprise
Log in
PreviousVectors in SupabaseNextTransactional email with Resend

Vibe Coder · Unit 17: RAG and Your Data · Lesson 6 of 6

Checking RAG answers

A RAG helper can fail in two places: finding the right passage, and answering from it. Test both with a fixed set of questions, fix the most common causes, and publish the results.

Read the short notes and pictures.Do everything inside a blue frame.

Two places to fail

A RAG system is two machines in a row, and a wrong answer came from one of them. Looking only at final answers and tweaking the prompt is how teams waste weeks. Split the question in two and the cause is usually obvious.

A chalkboard, hand-drawn. Two cards: retrieval, where the right passage was not fetched; generation, where the passage was there and the model ignored it.
Find out which half failed before you change anything.

Step one: retrieval

For each test question, look at the chunks that were retrieved, before the model sees them. Ask one thing: is the information needed to answer in there? Score it yes or no.

  1. 1

    Log the retrieved chunks

    In n8n, open the run and read the output of the vector store node. With hosted file search, ask for the search results to be included in the response.

  2. 2

    Mark each question

    Answer present in the chunks, or not.

  3. 3

    If not: is it in your documents at all?

    If no, it is a document problem. If yes, it is a chunking or search problem.

Step two: generation

Only for the questions where retrieval succeeded, read the answer against the chunks.

Supported

Every claim in the answer appears in a retrieved chunk.

Cited

The answer names where it came from.

Honest

When the chunks do not contain the answer, it says so.

Contained

It adds nothing from the model’s general knowledge.

The usual causes and fixes

Retrieval failures

The answer was never written down.
A chunk split a table or a list in half.
The user’s words differ from the document’s.
Too few chunks were retrieved.
Old and new versions are both indexed.

Fixes

Write it. This is the most common fix.
Split at headings; keep tables whole.
Add keyword search beside vectors (hybrid).
Retrieve more, for example ten, not three.
Remove superseded documents.
Change one thing, re-run the whole set, and keep the change only if the score goes up.

Generation failures

It answered from general knowledge.
It merged two unrelated chunks.
It was vague to be safe.
It ignored a number in the chunk.

Fixes

Instruct: only from the passages.
Ask it to quote the sentence it relied on.
Ask for the answer first, then the source.
Use a stronger model for this step.
Asking the model to quote its source sentence is a cheap and effective check.
A chalkboard, hand-drawn. A table of three versions with a retrieval score and an answer score for each.
Score every version on the same questions.

Keeping score

Ten questions is enough to start; grow the set whenever a real user asks something that fails. Keep the scores in a table with one row per version of your system. This is the evidence that a change helped, and it is what makes your project credible to anyone reading it.

A model can help you grade

For larger sets, ask a second model to judge whether each answer is supported by the chunks. Spot-check its judgements yourself: a grader is also a model, and it can be wrong.

Add hostile cases

Include a question whose answer is not in your documents, and a document containing text such as "ignore your instructions". The helper should say it does not know in the first case and treat the text as content in the second.

ActivityYour turnWrite it yourself first, then compare.

Your scores are: retrieval 9 out of 10, answers 5 out of 10. Where do you work, and what do you try first?

ActivityBuild stepA helper that answers from your documents

Score, fix and publish your helper

Score your helper on your test set: retrieval and answers separately. Make at least two improvements, re-scoring after each. Then publish the work to your GitHub repository.

Done when

Link the rag/ folder from your main README.

Keep these

  • A wrong answer is a retrieval failure or a generation failure.
  • Check the retrieved chunks before you touch the prompt.
  • Re-run the same questions after every change and keep the scores.
Previous lessonVectors in SupabaseProject · milestone 17A helper that answers from your documents