AI Folks
  • Courses
  • Web Of AI
  • Pricing
  • Enterprise
Log in
PreviousErrors and retriesNextModels on your own computer

Vibe Coder · Unit 11: APIs · Lesson 13 of 14

Fast models on Groq

Not every AI feature needs the biggest model. Groq serves open models very fast through an API that looks like OpenAI’s, so trying it is a two-line change.

Read the short notes and pictures.Do everything inside a blue frame.

Speed is a feature

You added an AI feature with Claude and with OpenAI. Both are strong, and for many jobs stronger than you need. A feature that tags a support ticket, suggests the next word or classifies an expense should feel instant. For those, a smaller model served quickly is the better product decision.

Open models and who hosts them

Open model

A model whose weights are published. Anyone can run it, on their own hardware or a host’s.

Inference

Running a model to get an answer, as opposed to training it.

Inference provider

A company that runs open models for you and charges per token.

Groq

A provider whose documentation describes it simply as fast LLM inference, OpenAI-compatible.

A two-line change

Dark stripes, hand-drawn. A code window: an OpenAI client created with a base URL at api.groq.com and a key read from GROQ_API_KEY.
The same client, a different address and key.

"OpenAI-compatible" means the API accepts requests in the same shape as OpenAI’s. If your model call lives in one place, as the APIs unit recommended, switching provider means changing the address, the key and the model name.

Before

client = OpenAI()

# model: an OpenAI model

After

client = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)

# model: one from Groq's model list
The address is the one in Groq’s docs. Pick a current model id from their models page; the list changes often.

Low-code too

n8n has a Groq Chat Model node, so an AI Agent or a basic model step can use Groq by choosing that node and adding a credential. No code needed.

Choosing per job

Dark stripes, hand-drawn. A table of jobs and the kind of model each needs.
Small and fast for simple jobs; large and careful for hard ones.
  1. 1

    List your AI features

    For each: how hard is the thinking, and how long will a user wait?

  2. 2

    Try the small model first

    On your real test inputs.

  3. 3

    Keep the strong model where it earns its cost

    Long documents, subtle judgement, anything a mistake makes expensive.

  4. 4

    Consider routing

    Fast model first; hand over to the strong one when the fast one is unsure or the input is long.

Also on Groq

Speech to text and text to speech, and built-in tools such as code execution and browser search on some models.

Limits

Every provider has rate limits. Read the limits page before you launch.

Data

Check each provider’s terms on what they do with your prompts.

Test before you switch

A smaller model will be worse at some things. Run the test set you built for your AI feature against any model before it replaces the one you have. Speed is no use if the answers are wrong.

ActivityYour turnWrite it yourself first, then compare.

Your invoicing tool has two AI features: (a) suggest a category for each expense as it is typed; (b) write a polite but firm final-notice letter. Which model for each, and why?

ActivityBuild stepOne AI feature

Try your AI feature on a fast model

Get a Groq API key. Point your AI feature at Groq by changing the address, key and model, and run your test inputs through both providers. Record speed and quality side by side.

Done when

Keep the provider and model in configuration, not scattered through the code.

Keep these

  • Open models can be hosted by many providers.
  • Groq is fast and OpenAI-compatible: change the address, key and model.
  • Use small fast models for simple jobs, and test before switching.
Previous lessonErrors and retriesNext lessonModels on your own computer