Training vs Inference (and Why It Matters for Cost)

Two phases, two economies

Every model lives two lives. Training is the one-time construction: feeding enormous datasets through expensive hardware for weeks to set the model’s billions of parameters. It costs a fortune and happens rarely — the frontier labs do it, most people never do.

Inference is everything after: running the finished model to answer a prompt, generate an image, or transcribe audio. Each call is comparatively cheap, but you pay it every single time. For anyone building on AI, inference is the meter that actually runs.

Fine-tuning: the middle path

Between “use it as-is” and “train from scratch” sits fine-tuning — taking a pre-trained model and training it a bit more on your own narrower data to nudge its style or specialise it (say, on your support tickets). It’s far cheaper than training a base model, but it’s still a training process with data, cost, and maintenance. Crucially, it’s often unnecessary: good prompting plus retrieval (RAG) solves many problems people reach for fine-tuning to fix.

The habit that saves money

Because inference scales with usage, a successful feature’s bill grows month over month. That’s why the whole Best Practices track exists — routing easy work to cheaper models, caching, and trimming tokens all attack inference cost directly.

One more thing to internalise: standard inference does not retrain the model. Your conversation doesn’t teach it anything permanent, and (unless a provider says otherwise) shouldn’t leak into the next person’s answers — though you should still check each provider’s data-use policy and disable training-on-your-data for anything sensitive.

That completes the Foundations module. From here you can jump into Prompt Engineering to get dramatically better answers, or head to Using AI Tools to choose and verify a chatbot.

For each task, tell me whether it happens during TRAINING or INFERENCE:
1) Answering my question right now
2) Learning grammar from billions of web pages
3) Fine-tuning on a company's support tickets
4) Generating an image from my prompt
Give a one-line reason for each.

1) Inference — it's using the already-trained model to respond.
2) Training — the model is learning general patterns from data.
3) Training (fine-tuning) — adjusting a base model on new, specific data.
4) Inference — running the trained model to produce an output.