What is RAG?

It has never seen your documents.

The model only knows what it learned during training. Your handbook, codebase, and company files were probably not part of it.

But it may still answer questions about them. And those answers can be completely made up.

A better prompt cannot fix missing information. The model needs to see the document.

“Just paste the whole document in.”

For a short document, this works well. It is often the easiest option.

But large collections can have thousands of pages. Sending everything becomes slow and expensive.

Simple rule: short document, paste it. Large collection, search it.

Search first. Then answer from what you found.

First, find the useful parts of your documents.

Then give only those parts to the model. It only needs to read what matters.

That is RAG: find the right information, then answer from it.

There are two stages.

First, prepare your documents so they can be searched.

Then, when a question arrives, find the best parts and send them to the model.

Prepare first. Search when the question arrives.

Split documents into chunks.

A 200-page PDF is too large to search as one piece.

So you split it into smaller sections called chunks. Each chunk should make sense on its own.

Split around headings and paragraphs. Try not to cut ideas in half.

Turn every chunk into numbers.

Each chunk goes through an embedding model. It turns the meaning into numbers.

Text with similar meaning gets similar numbers. Store those numbers with the original text.

A vector database stores these numbers and helps find similar ones.

Find the closest matches.

The question is turned into numbers too.

The system then finds chunks with similar meaning. The words do not need to match exactly.

"Time off" can still find a section about "annual leave".

Closest does not always mean best.

The closest chunk may be about the right topic but still miss the answer.

So the system can check the results again and keep the most useful ones.

This second check is called reranking.

Give the model only what it needs.

Send the model the useful chunks, the question, and clear instructions.

Tell it to say when the answer is not in the text.

Without this rule, it may still invent an answer.

The whole flow, end to end.

Pick a question and follow the steps.

The system searches the documents, finds useful chunks, and uses them to answer.

If nothing useful is found, the system should say so.

Usually the problem is the search.

A bad answer often starts with bad search results.

Maybe the wrong chunks were found. Maybe a document was missing. Maybe the chunks were split badly.

Check what was found before changing the prompt.

Three useful upgrades.

Keyword search: helps find exact words, names, codes, and numbers.

Filters: search only the right files or the latest version.

Reranking: check the search results again and keep the best ones.

None of these require changing the main model.

Test search and answers separately.

Do not only check the final answer.

First ask: did search find the right text? Then ask: did the model answer correctly from it?

Start with a small set of questions where you already know the answers.

RAG is actually simple.

Split the documents. Turn the chunks into embeddings. Search for the best chunks. Give them to the model.

That is the core of RAG.

If the documents do not contain the answer, say so.

Two phases, not one

Indexing runs once when documents change. Retrieval runs on every single question. Most confusion is mixing them up.

Chunking is a real decision

Split on structure, not on character count. A rule cut in half retrieves as two halves of an answer.

Same embedding model, both sides

Questions and chunks must be embedded by the same model, or the distances mean nothing.

A threshold is the refusal

Without a minimum score, the top result always comes back — even when nothing relevant exists.

Cite so it can be checked

An answer with a passage reference is auditable. One without is just a fluent claim.

Measure the halves apart

Did retrieval find the right passage? Did generation use it faithfully? Two questions, two fixes.