Vibe Coder · Unit 10: APIs · Lesson 10 of 13
Caching and batching
Engineers: cut cost and latency by caching the part of a prompt that repeats, and batching work that does not need an instant answer.
Read the short notes and pictures.Do everything inside a blue frame.
Paying less for the same work
Most prompts repeat a long prefix: a system prompt, tool definitions, a reference document. Prompt caching lets the provider reuse that prefix at a fraction of the input price. Batch processing handles non-urgent jobs asynchronously for less.

Using the cache well
Stable prefix
Any change before the cache point invalidates it.
Long enough
There is a minimum size to cache.
Used soon
Caches expire after a short time.
Measured
Usage reports cached tokens separately.

Good batch jobs
Backfills
Classify last year’s feedback.
Nightly reports
Summaries nobody is waiting for.
Evaluations
Running a test set against a prompt.
Cache your prompt
Restructure your feature’s prompt so the stable part comes first, enable caching, and confirm from the usage data that later calls read from the cache. Record cost before and after.
Done when
Use a batch job when you add a backfill.
Keep these
- Cache the repeating prefix.
- Keep stable content first.
- Batch what does not need an instant answer.