Vibe Coder · Unit 2: How Claude and ChatGPT Work · Lesson 1 of 7

What a language model does

Claude and ChatGPT do one thing underneath: predict what comes next, one token at a time. That one idea explains what they do well and badly.

Read the short notes and pictures.Do everything inside a blue frame.

One job: predict what comes next

Claude and ChatGPT are built on large language models. A language model does one thing: it reads the text so far and predicts what comes next. Writing an email, planning a product and writing code are all the same job to the model, done again and again.

ThecapitalofFranceis?ParisParis92%Lyon3%a2%located1%
The model reads the text so far and scores every word that could come next.

Text becomes tokens

The model does not read letters or whole words. Your text is first split into small pieces called tokens. A short, common word is usually one token. A long or rare word is split into several. Each token is turned into numbers, because numbers are what the model works with.

textunbelievably goodtokensun403believ6667ably1346good922IDsvectorsthe IDs here are examples; every tokeniser has its own numbering
Your text is split into tokens, and each token becomes numbers the model can work with.

A probability for every next token

For the text "The cat sat on the", the model gives every token it knows a probability of coming next. "mat" gets a high one. "moon" gets a low one. It picks one, adds it to the text, and predicts again. A full answer is this loop run hundreds of times.

Wooden token blocks on an oak table: The, cat, sat, on, the and a question mark. Behind them, columns show mat 62 percent, sofa 17, floor 11 and moon 1.
Every possible next token gets a probability. The model picks one and goes again.
  1. Read

    The model reads all the text so far: your message, and anything it has already written.

  2. Predict

    It gives every possible next token a probability.

  3. Pick

    It picks one. Usually a likely one, but not always the most likely, which is why the same question can get different answers.

  4. Repeat

    It adds the token to the text and starts again, until the answer is finished.

How it learned to do this

Nobody wrote rules for grammar, facts or code into the model. It was trained: shown a huge amount of text with the next token hidden, asked to guess it, and adjusted a little after each guess. After enough rounds, its guesses became good enough to write essays and working programs.

A row of wooden token blocks on an oak table: The, cat, sat, on, the, mat, and, then a question mark.
One token at a time: read, predict, add, repeat.

What this means for you

It is fluent

It has seen a great deal of good writing and code, so what it writes reads well and usually runs.

It writes what is likely

Likely is often right. It is not the same as true, so facts and code need checking.

It follows what you give it

Its prediction depends on the text in front of it. A clearer request gives a better answer.

It can vary

Ask twice and the wording can change, because it picks from several likely tokens.

ActivityBuild · step 1 of 6A fact-checked product brief

Draft the brief, twice

Give Claude your one-line idea from "My build" and ask for a one-page product brief: the problem, the people, what the first version does. Then ask again in a new chat. Compare the two, and paste the better parts of both into "My build" under the heading "Brief".

Done when

Two drafts show you which ideas the model is sure of and which it made up on the day.

Keep these

  • A language model predicts the next token, again and again.
  • Text is split into tokens, and tokens become numbers.
  • It learned by guessing hidden text and being adjusted, not from written rules.
  • It writes what is likely, which is often but not always what is true.