Vibe Coder · Unit 10: APIs · Lesson 10 of 13

Caching and batching

Engineers: cut cost and latency by caching the part of a prompt that repeats, and batching work that does not need an instant answer.

Read the short notes and pictures.Do everything inside a blue frame.

Paying less for the same work

Most prompts repeat a long prefix: a system prompt, tool definitions, a reference document. Prompt caching lets the provider reuse that prefix at a fraction of the input price. Batch processing handles non-urgent jobs asynchronously for less.

A 3D scene of coloured blocks and cards. : First call, Later calls.
The first call writes the cache, later calls read it.

Using the cache well

Stable prefix

Any change before the cache point invalidates it.

Long enough

There is a minimum size to cache.

Used soon

Caches expire after a short time.

Measured

Usage reports cached tokens separately.

A 3D scene of coloured blocks and cards. : Live, Batch.
Batches trade speed for a lower price.

Good batch jobs

Backfills

Classify last year’s feedback.

Nightly reports

Summaries nobody is waiting for.

Evaluations

Running a test set against a prompt.

ActivityBuild stepOne AI feature

Cache your prompt

Restructure your feature’s prompt so the stable part comes first, enable caching, and confirm from the usage data that later calls read from the cache. Record cost before and after.

Done when

Use a batch job when you add a backfill.

Keep these

  • Cache the repeating prefix.
  • Keep stable content first.
  • Batch what does not need an instant answer.