MLOps & Deployment: Interview Questions
Production AI questions that separate builders from tinkerers — monitoring, data and model drift, evaluation, cost control, and safely shipping and updating models and LLM apps.
easyWhat is MLOps?
MLOps is the set of practices for reliably deploying, monitoring, and maintaining machine-learning systems in production — the ML equivalent of DevOps. It covers data and feature pipelines, model versioning, automated training/evaluation, deployment, monitoring for drift and performance, and rollback. The key difference from traditional software: ML systems depend on data that changes, so they degrade silently over time and need continuous monitoring and retraining, not just code deployment.
mediumWhat is data drift and how do you detect it?
Data drift is when production input data diverges from the training distribution over time (e.g. user behaviour shifts, a new product launches), degrading model performance. Concept drift is when the input-output relationship itself changes. Detect it by monitoring feature distributions and comparing them to training (statistical tests like PSI or KS), tracking prediction distributions, and watching live accuracy where labels are available. Alerts on drift trigger investigation and retraining.
Learn this: Training vs Inference (and Why It Matters for Cost) →
mediumWhat should you monitor for a model in production?
Operational metrics (latency, throughput, error rate, cost), model-quality metrics (accuracy/precision/recall where labels arrive, or proxy metrics), input data health (drift, missing values, schema changes), and business KPIs the model is meant to move. For LLM apps, also track token usage per feature, hallucination/quality via sampling and evals, and safety flags. The principle: catch problems from data and quality, not just uptime — a model can be 'up' while quietly making bad predictions.
hardHow do you evaluate an LLM application before and after deployment?
Build a golden set of representative inputs with expected outputs or grading criteria, and run it in CI on every prompt or model change — block regressions. Use automated graders (exact match, rubric-based, or LLM-as-judge with care) plus human review for nuanced cases. In production, sample real traffic, log inputs/outputs, collect user feedback, and feed failures back into the eval set. Evals turn model upgrades and prompt tweaks from leaps of faith into measured engineering.
mediumHow do you control the cost of an LLM feature in production?
Route requests by difficulty — small model for mechanical work, flagship only for hard cases. Cache stable prompt prefixes and cache/store repeated results. Cap output tokens and request only needed fields. Use batch APIs for non-interactive jobs (often ~50% cheaper). Retrieve minimal context instead of stuffing documents. And measure tokens per feature in CI so regressions surface immediately rather than in the monthly bill. These commonly cut spend by more than half.
mediumHow do you safely deploy a new model version?
Use progressive rollout: shadow mode (run the new model alongside the old without serving its output) to compare, then a canary or A/B test on a small traffic slice with guardrail metrics, then full rollout — with instant rollback ready. Version the model, data, and config together for reproducibility, and keep an audit trail. For LLMs, pin model snapshots and run your eval suite before promoting. Never swap a production model based on offline metrics alone.
hardWhat is training-serving skew?
Training-serving skew is when features are computed differently during training than at serving time — different code paths, data sources, or timing — so the model sees inconsistent inputs and underperforms in production despite good offline metrics. Prevent it by sharing the same feature-transformation code between training and serving (or a feature store), logging serving features to compare against training, and testing the full pipeline end to end. It's a leading cause of 'great in testing, bad in production'.
mediumA newly deployed model is underperforming. What do you do?
Roll back to the previous known-good version immediately if user impact is real — recovery first, diagnosis second. Then investigate with your monitoring: compare input distributions (drift?), check for a broken feature pipeline (training-serving skew), review the deploy diff, and inspect failing cases. Reproduce offline, fix, re-run evals, and redeploy progressively (shadow/canary). Capture the root cause so the same class of failure is caught earlier next time. Good rollback and observability are what make this a controlled event rather than a crisis.
mediumWhat is a feature store, and what problem does it solve in ML systems?
A feature store is a centralized system for defining, computing, storing, and serving ML features. It typically has two parts: an offline store (a data warehouse or lake) that holds historical feature values for training, and an online store (a low-latency key-value store like Redis or DynamoDB) that serves fresh features for real-time inference. Its main value is consistency: the same feature definition and transformation logic produce values for both training and serving, which reduces training-serving skew. It also enables feature reuse across teams and models, point-in-time correct joins to avoid label leakage, versioning, and monitoring. Popular implementations include Feast, Tecton, and cloud-native offerings like Vertex AI and SageMaker Feature Store.
easyWhat is a model registry, and why is model versioning important?
A model registry is a central catalog that stores trained model artifacts along with their metadata: version, training data reference, hyperparameters, evaluation metrics, lineage, and stage (e.g., staging, production, archived). Tools like MLflow Model Registry, SageMaker, and Vertex AI provide this. Versioning matters because production models change frequently; you need to know exactly which model version served a given prediction for debugging, auditing, and compliance. It enables safe promotion workflows (approval gates before a model goes to production), fast rollback to a known-good version if a new one degrades, reproducibility by linking a model to its exact code and data, and A/B comparison across versions. Without a registry, teams lose track of what is deployed and cannot reliably reproduce or revert models.
mediumHow does CI/CD for machine learning differ from CI/CD for traditional software?
Traditional CI/CD tests and deploys code. ML CI/CD must also handle data and models, which adds pipeline stages. Beyond unit tests, CI validates data schemas, checks data quality, and tests feature-engineering code. It often includes CT (continuous training): an automated pipeline that retrains, evaluates against a baseline, and only promotes a model that beats the current one on holdout metrics. Deployment ships a model artifact plus its serving code, not just code. Extra concerns include validating that the model meets latency and accuracy thresholds, guarding against training-serving skew, and monitoring after deploy for drift. Because behavior depends on data, ML systems need triggers to retrain and redeploy when data changes, not only when code changes. Tools include Kubeflow Pipelines, TFX, GitHub Actions, and MLflow.
mediumWhat is the difference between canary and shadow deployment for ML models?
Both reduce the risk of releasing a new model, but differently. In a canary deployment, the new model actually serves a small slice of live traffic (e.g., 5%) and its predictions are returned to those users; you monitor business and quality metrics, then gradually ramp to 100% if healthy or roll back if not. In a shadow (or dark) deployment, the new model receives a copy of real production traffic and produces predictions, but those outputs are never returned to users; they are only logged and compared against the current model. Shadow testing validates latency, stability, and prediction behavior under real load with zero user risk, but it cannot measure downstream business impact since no one acts on its outputs. Canary measures real impact but exposes some users to a potentially worse model. Teams often shadow first, then canary.
easyWhen would you choose batch inference over real-time (online) inference?
Choose batch inference when predictions are not needed instantly and can be precomputed on a schedule. You run the model over a large dataset periodically (e.g., nightly), store results in a database or cache, and look them up on demand. It fits use cases like product recommendations, churn scoring, or lead ranking where inputs change slowly. Batch is simpler to operate, cost-efficient (you can use spot instances and maximize throughput), and easy to backfill. Choose real-time inference when predictions depend on fresh, per-request inputs and users expect low latency, such as fraud detection at checkout, search ranking, or chatbots. Real-time serving requires an always-on service with strict latency SLAs, autoscaling, and careful handling of feature freshness. Some systems combine both: precompute heavy features in batch, then serve lightweight real-time scoring.
mediumWhat is the difference between concept drift and data drift?
Data drift (covariate shift) means the distribution of input features changes over time while the relationship between inputs and target stays the same, P(X) changes but P(Y|X) does not. Example: a new user segment shifts the age distribution of your inputs. Concept drift means the relationship between inputs and target itself changes, P(Y|X) changes, so the same input now maps to a different correct output. Example: during an economic shock, income features that once predicted low default risk no longer do. The distinction matters for diagnosis and remedy: data drift is detectable by comparing input distributions (KS test, PSI) without labels and may or may not hurt accuracy; concept drift usually requires labels or delayed feedback to detect and almost always degrades performance, forcing a retrain on recent data. Monitoring should track both input distributions and, when labels arrive, prediction quality.
mediumWhat triggers should decide when to retrain a production model?
Retraining can be triggered by schedule, performance, or data signals. Scheduled retraining (e.g., weekly) is simple and predictable but may retrain unnecessarily or too late. Performance-based triggers fire when a monitored metric (accuracy, AUC, business KPI) drops below a threshold once ground-truth labels arrive, retraining only when quality actually degrades. Drift-based triggers fire when input distributions shift significantly (PSI or KS above a bound), useful when labels are delayed since drift is a leading indicator. Volume-based triggers retrain after enough new labeled data accumulates. In practice teams combine these: monitor drift and performance continuously, and gate automatic retraining behind an evaluation step that promotes the new model only if it beats the incumbent on a holdout set. Also consider retrain cost, label latency, and stability, since retraining too aggressively can add noise and instability.
mediumWhat must you track to make an ML experiment reproducible?
Reproducibility requires capturing everything that influences the result. Track: the code version (git commit), the exact dataset version or snapshot (via hashing or tools like DVC or LakeFS), all hyperparameters and config, the training environment (library versions, container image, hardware/GPU type), and random seeds. Also log the resulting metrics, the model artifact, and its lineage linking model to code and data. Nondeterminism is a common pitfall: GPU operations, parallel data loading, and some library kernels can produce different results even with fixed seeds, so you may need deterministic flags and to accept small tolerances. Tools like MLflow, Weights & Biases, and Neptune automate much of this tracking. Good reproducibility lets you rerun an experiment, compare runs fairly, audit a production model, and debug regressions by isolating which change caused a metric shift.
hardWhat is dynamic batching in model serving, and what tradeoff does it manage?
Dynamic batching groups multiple incoming inference requests that arrive close in time into a single batch that the model processes in one forward pass. GPUs are far more efficient on batched matrix operations, so batching dramatically increases throughput and hardware utilization compared to processing requests one at a time. The tradeoff is latency versus throughput: to form a batch, the server waits a short window (e.g., a few milliseconds) for more requests to arrive, adding queuing delay to each request. A larger max batch size and longer wait window raise throughput but also tail latency. Serving frameworks like NVIDIA Triton, TorchServe, TensorFlow Serving, and vLLM (with continuous batching for LLMs) expose knobs for max batch size and max wait time so you can tune to your SLA. For LLMs, continuous or in-flight batching further improves utilization by adding and removing sequences mid-generation.
mediumHow does a blue-green deployment work, and why use it for model serving?
Blue-green deployment runs two identical production environments: blue (the current live version) and green (the new version). You deploy the new model to green while blue keeps serving all traffic, then run smoke tests and validation against green. When green is verified healthy, you switch the router or load balancer to send all traffic to green instantly. Blue stays running and idle, so if the new model misbehaves you flip traffic back to blue for a near-instant rollback with no redeploy. The benefits for model serving are zero-downtime cutover, fast and safe rollback, and the ability to fully validate the new environment before it takes traffic. The costs are running two full environments (double the resources during the switch) and handling stateful concerns like warm caches or in-flight requests. It differs from canary, which shifts traffic gradually rather than all at once.
mediumHow do you ensure model interpretability in production?
Ensuring model interpretability in production involves several strategies. First, you can choose inherently interpretable models, like decision trees or linear models, when feasible. For black-box models, such as neural networks, you can use techniques like SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) to provide insights into model predictions. Additionally, you can implement logging mechanisms that track feature importance and changes over time. Providing documentation and visuals to stakeholders will also help contextualize model decisions, enhancing trust and transparency.
mediumWhat role do audit logs play in MLOps?
Audit logs are critical in MLOps for ensuring traceability and compliance. They record every change made to the model, data, features, and environments during the ML lifecycle. This allows teams to track who made changes, what changes were made, and why. Audit logs help in diagnosing issues, understanding model performance trends, and providing accountability, especially in regulated industries. They are also useful for facilitating reproducibility of experiments and models, aiding in both debugging and regulatory audits.
No questions match your filter.