Vibe Coder · Unit 2: How Claude and ChatGPT Work · Lesson 7 of 7

Tokens and cost

When your code calls a model you pay by the token. Input, output, the context window, and three levers that cut the bill.

Read the short notes and pictures.Do everything inside a blue frame.

In your own product, you pay by the token

The Claude and ChatGPT apps charge a flat monthly price. When your own code calls a model through the API, the bill works differently: you pay for every token you send and every token the model writes. Understanding tokens is how you predict the cost and the speed of an AI feature before you build it.

What a token is

A token is a piece of text, often a short word or part of a longer one. As a rule of thumb for English, 1,000 tokens is about 750 words. Code, numbers and other languages usually take more tokens per word. Every API response tells you exactly how many tokens the request used.

textunbelievably goodtokensun403believ6667ably1346good922IDsvectorsthe IDs here are examples; every tokeniser has its own numbering
Text is split into tokens before the model sees it. The count of tokens is what you are billed for.

Input and output are priced separately

A 3D scene of coloured blocks and cards. : Input tokens, Output tokens.
Input is everything you send. Output is what the model writes, and it costs more per token.

Input tokens

Everything you send: instructions, the conversation so far, documents and tool results.

Output tokens

Everything the model writes, including its thinking. Priced several times higher than input.

What you send

system:  instructions      400 tokens
history: earlier turns    2,100 tokens
user:    new question        60 tokens

input total              2,560 tokens

What you get back

assistant: the answer       380 tokens

output total                380 tokens

cost = input x input price
     + output x output price
Most of the input here is history. That is typical: in a long conversation, earlier turns are the largest part of every request.

Every turn sends the whole conversation again

The API keeps no memory between requests. To continue a conversation, your code sends all of it again, plus the new message. Turn 20 of a chat sends turns 1 to 19 as input. Cost per turn grows as the chat grows, and so does the time to respond. Everything must also fit in the model’s context window.

the context window: what the model seesolder tokens: not seenoldestnewestprompt + conversation + the reply being written must all fit
Each request carries the whole conversation. A long history is paid for again on every turn.

Three levers that cut the bill

A 3D scene of coloured blocks and cards. Three levers: shorter context, smaller model, caching.
Shorter context, a smaller model, and caching.
  1. Shorter context

    Send only what the task needs. Summarise old turns, retrieve the relevant passages instead of whole documents, and cap the length of the answer.

  2. A smaller model

    Route simple jobs such as classifying, extracting and short replies to the small model, and keep the large one for hard requests. The price per token differs by several times between sizes.

  3. Caching

    Prompt caching stores the part of the prompt that repeats, such as long instructions or a reference document, so later requests read it at a fraction of the input price.

ActivityBuild · step 7 of 7A fact-checked product brief

Engineers: cost the AI feature

Pick the one AI feature your product will have. Estimate input and output tokens per request, requests per user per day and users in the first month, then work out a monthly cost on a small and a large model using the prices on the provider’s pricing page. Add it to the brief under "Cost".

Done when

You will check this estimate against real usage in the APIs unit.

Keep these

  • Through the API you pay per token: input and output, with output priced higher.
  • About 1,000 tokens is 750 English words.
  • Each turn sends the whole conversation again, so long chats cost more per turn.
  • Cut cost with shorter context, a smaller model for simple jobs, and caching.