AI Folks
  • Courses
  • Web Of AI
  • Pricing
  • Enterprise
Log in
PreviousFine-tuning with UnslothNextHow image models work

Vibe Coder · Unit 19: Fine-Tuning a Model · Lesson 4 of 4

Is it better?

A tuned model has to beat what you already had. Compare it with the base model and with your best prompt on the hold-out set, count the cost, and publish the experiment.

Read the short notes and pictures.Do everything inside a blue frame.

The honest test

It is easy to be pleased with a model you trained yourself. The work only paid off if it beats what you already had. Often it does not, and finding that out cheaply is a good result: you keep the prompt, and you know why.

Three contenders

A pale contour map, hand-drawn. A table of three contenders: the untouched base model, a strong model with your best prompt, and your tuned small model.
Compare all three on the same held-out inputs.

A: the base model

The small open model you started from, with a plain prompt. Shows what training added.

B: your best prompt

A strong hosted model with the careful prompt and examples from the ladder. This is what you would ship without fine-tuning.

C: the tuned model

Your small model with its adapter and the short system message it was trained with.

Judging

  1. 1

    Run every hold-out input through all three

    Save the outputs in a table.

  2. 2

    Shuffle and hide the labels

    So you do not know which model wrote which.

  3. 3

    Score against a rubric

    A few yes-or-no questions per output.

  4. 4

    Reveal and total

    Then read the failures of each contender, not only the totals.

A weak rubric

Is it good? (1 to 10)

A usable rubric

Correct facts from the input?  yes / no
Sounds like our voice?         yes / no
Right format and length?       yes / no
Invented anything?             yes / no
Yes-or-no questions give scores two people can agree on.

With a larger set, let a model help

For more than a few dozen outputs, a strong model can apply the rubric for you. Check a sample of its judgements by hand first.

Cost

A pale contour map, hand-drawn. A table comparing a strong model with a prompt against a tuned small model on price per call, speed and hosting.
Quality is one column. Cost and upkeep are the others.

Per call

A small tuned model with a short prompt uses far fewer tokens than a large model with a long one.

Speed

Small models answer faster, if the hardware is adequate.

Hosting

A hosted model is someone else’s problem. A tuned open model has to run somewhere: your own machine, a rented server or a provider that hosts open models.

Upkeep

When your style or policy changes, a prompt is edited in a minute. A tuned model is retrained.

Deciding

Ship the tuned model when it clearly wins on the hold-out set, or matches the prompt version at meaningfully lower cost at your volume, and when someone will own retraining it. In every other case keep the prompt. For most early products the answer is "not yet", and the dataset you built is still valuable: it is now your test set.

Check for memorised data

Before shipping any tuned model, try to make it repeat its training examples. If it will output a real customer’s details, your dataset was not cleaned well enough. Fix the data and retrain.

ActivityYour turnWrite it yourself first, then compare.

Results on 20 hold-out inputs: base model 8 pass, best prompt 17 pass, tuned model 16 pass. The tuned model costs a tenth as much per call. You handle 200 requests a day. What do you ship?

ActivityBuild stepA model tuned for one job

Compare, decide and publish

Run your hold-out inputs through the base model, your best prompt and your tuned model. Score them blind against a rubric, add the costs, and write your decision. Publish the experiment to GitHub.

Done when

Model files are large. Leave them out of the repository and say in the README where they are kept.

Keep these

  • Compare the base model, your best prompt and the tuned model.
  • Judge blind, with a yes-or-no rubric, on held-out inputs.
  • Ship the tuned model only if it wins on quality or cost.
Previous lessonFine-tuning with UnslothProject · milestone 19A model tuned for one job