When would you choose batch inference over real-time (online) inference?

Choose batch inference when predictions are not needed instantly and can be precomputed on a schedule. You run the model over a large dataset periodically (e.g., nightly), store results in a database or cache, and look them up on demand. It fits use cases like product recommendations, churn scoring, or lead ranking where inputs change slowly. Batch is simpler to operate, cost-efficient (you can use spot instances and maximize throughput), and easy to backfill. Choose real-time inference when predictions depend on fresh, per-request inputs and users expect low latency, such as fraud detection at checkout, search ranking, or chatbots. Real-time serving requires an always-on service with strict latency SLAs, autoscaling, and careful handling of feature freshness. Some systems combine both: precompute heavy features in batch, then serve lightweight real-time scoring.