LLMs & Transformers: Interview Questions
The questions that dominate 2026 AI interviews — how transformers and attention work, what tokens and context windows are, why models hallucinate, and how temperature and fine-tuning shape behaviour.
mediumWhat is a transformer, and why did it change NLP?
A transformer is a neural-network architecture (introduced in 'Attention Is All You Need', 2017) built around self-attention rather than recurrence. It processes all tokens in parallel and lets each token attend to every other, capturing long-range relationships efficiently. This parallelism made training on massive data practical and scaled far better than RNNs/LSTMs — which is why virtually every modern LLM is a transformer.
hardExplain self-attention in simple terms.
Self-attention lets each token decide how much to 'pay attention' to every other token when building its representation. Each token produces a query, key, and value; the query is compared against all keys to get attention weights, which then combine the values. In 'the animal didn't cross the street because it was tired', attention helps 'it' link to 'animal'. Multi-head attention runs several such mechanisms in parallel to capture different kinds of relationships.
easyWhat is a token in an LLM?
A token is the unit an LLM reads and writes — a chunk of text roughly 4 characters or ¾ of a word in English. Text is split by a tokeniser; common words are one token, rarer words split into several, and punctuation and spaces count too. Models have a maximum token limit (the context window), and APIs bill per token, so tokens matter for both capability and cost.
easyWhat is a context window and why does it matter?
The context window is the maximum number of tokens (input + output) a model can consider in a single request. Anything beyond it is truncated. It matters because it bounds how much document, conversation, or code the model can reason over at once — and because models tend to use the start and end of a long context better than the middle ('lost in the middle'). Bigger isn't automatically better; relevant, well-placed context beats sheer volume.
mediumWhy do LLMs hallucinate, and how do you reduce it?
LLMs are trained to produce plausible next tokens, not verified truth, so when they lack knowledge they still generate a confident-sounding answer. Reduce it with retrieval-augmented generation (ground answers in real sources), instructing the model to say 'I don't know' when unsure, lowering temperature, asking for citations you can check, and adding verification steps. You can lower the rate substantially but not to zero — human verification of important facts remains necessary.
easyWhat does the temperature parameter do?
Temperature controls randomness in token selection. Low temperature (near 0) makes the model pick the highest-probability tokens — more deterministic and focused, good for factual or structured tasks. Higher temperature flattens the distribution so less-likely tokens get chosen more often — more varied and creative, good for brainstorming, but also more prone to drifting or hallucinating. Top-p (nucleus) sampling is a related control over the candidate pool.
hardWhen would you fine-tune an LLM versus use RAG?
Use RAG when answers must be grounded in factual, changing, or private data — you update documents, not weights, and get citations. Use fine-tuning to teach a consistent style, format, or narrow behaviour the base model can't reliably follow through prompting. RAG is usually cheaper to keep current; fine-tuning needs curated data and re-runs when things change. They combine well: RAG for knowledge, light fine-tuning for tone. Reach for prompting first — it solves more than people expect.
mediumWhat's the difference between pre-training and fine-tuning?
Pre-training builds the base model from scratch on enormous general text to learn language and world patterns — hugely expensive, done by frontier labs. Fine-tuning takes that pre-trained model and trains it further on smaller, task- or domain-specific data to specialise it (including instruction-tuning and RLHF that make base models good at following instructions). Pre-training gives broad capability; fine-tuning shapes it toward specific behaviour.
Learn this: Training vs Inference (and Why It Matters for Cost) →
mediumWhat is the difference between encoder-only, decoder-only, and encoder-decoder transformer architectures?
Encoder-only models (e.g. BERT) process the full input bidirectionally with unmasked self-attention, producing rich contextual embeddings. They excel at understanding tasks like classification, NER, and retrieval, but can't generate text autoregressively. Decoder-only models (e.g. GPT, Llama) use causal (masked) self-attention so each token attends only to prior tokens, making them natural next-token generators used for most modern LLMs. Encoder-decoder models (e.g. T5, original Transformer) pair a bidirectional encoder with an autoregressive decoder that cross-attends to the encoder output; they suit sequence-to-sequence tasks like translation and summarization where input and output are distinct sequences. Decoder-only dominates today because it scales simply and handles both understanding and generation via prompting.
mediumWhy do transformers need positional encodings, and how do sinusoidal and RoPE approaches differ?
Self-attention is permutation-invariant: it treats input as a set, so without position information 'dog bites man' and 'man bites dog' look identical. Positional encodings inject order. The original transformer added fixed sinusoidal vectors (sines/cosines of different frequencies) to token embeddings, letting the model infer relative distances and extrapolate somewhat to unseen lengths. Learned absolute embeddings (BERT/GPT-2) instead train a position vector per index, but don't extrapolate past training length. RoPE (Rotary Position Embedding, used in Llama, etc.) rotates the query and key vectors by an angle proportional to position, so the dot product naturally encodes relative position. RoPE is now standard because it preserves relative-distance information, works well with long contexts, and supports interpolation/scaling tricks to extend context length.
mediumWhat is multi-head attention and why is it better than a single attention head?
Multi-head attention runs several attention operations in parallel, each with its own learned query, key, and value projections. The model embedding is split into h lower-dimensional subspaces; each head computes scaled dot-product attention independently, and the outputs are concatenated and projected back. The benefit is representational diversity: different heads can specialize, one tracking syntactic dependencies, another coreference, another local n-gram patterns, so the model attends to information from multiple representation subspaces at different positions simultaneously. A single head must average all these relationships into one attention distribution, which is a bottleneck. Importantly, multiple heads cost roughly the same as one because the per-head dimension is d/h, keeping total compute constant. Variants like grouped-query and multi-query attention share keys/values across heads to shrink the KV cache.
hardWhat is the KV cache in transformer inference and what problem does it solve?
During autoregressive generation, a decoder produces one token at a time, and each new token attends to all previous tokens. Naively, you'd recompute the key and value vectors for the entire sequence at every step, making generation O(n^2) redundant work. The KV cache stores the key and value tensors already computed for past tokens, so at each step you only compute Q, K, V for the single new token and append its K,V to the cache. This turns per-step attention cost from quadratic to linear in sequence length and is the main reason generation is fast. The tradeoff is memory: the cache grows linearly with context length and batch size (2 * layers * heads * head_dim * tokens), often becoming the dominant memory consumer for long contexts. Techniques like multi-query/grouped-query attention, paged attention (vLLM), and quantized caches reduce its footprint.
easyWhat is the difference between a token and an embedding in an LLM?
A token is a discrete unit of text produced by the tokenizer, typically a subword piece (via BPE, WordPiece, or SentencePiece) represented as an integer ID from a fixed vocabulary. Tokens are symbolic: ID 1234 carries no inherent meaning. An embedding is the dense, continuous vector (e.g. 4096 floats) that the model looks up for each token ID from its embedding matrix. Embeddings live in a learned geometric space where semantically related concepts sit near each other, so the model can compute with meaning rather than opaque IDs. In short: tokenization maps text to token IDs, and the embedding layer maps each ID to a vector. Every transformer layer then transforms these vectors into increasingly contextual representations. 'Embeddings' can also refer to the final pooled sentence/document vectors used for search and RAG.
mediumHow do top-k and top-p (nucleus) sampling control LLM text generation?
Both truncate the probability distribution over the next token before sampling, to avoid picking absurd low-probability tokens while keeping variety. Top-k keeps only the k highest-probability tokens, renormalizes, and samples from them; k is fixed, so in a very confident context you may still include unlikely options, and in a flat distribution you may exclude good ones. Top-p (nucleus) instead keeps the smallest set of tokens whose cumulative probability exceeds p (e.g. 0.9), so the candidate pool grows when the model is uncertain and shrinks when it's confident, making it more adaptive. They're often combined with temperature, which reshapes the distribution first. Lower p/k yields safer, more deterministic text; higher values yield more diverse, creative, or risky output. Setting k=1 is equivalent to greedy decoding.
hardWhat is RLHF and why is it used to align language models?
RLHF (Reinforcement Learning from Human Feedback) aligns a pretrained/instruction-tuned model with human preferences. It has three stages: (1) supervised fine-tuning on demonstration data to get a reasonable base; (2) train a reward model by having humans rank multiple model responses to the same prompt, teaching a model to predict which output people prefer; (3) optimize the LLM against that reward model using RL, typically PPO, with a KL-divergence penalty that keeps the policy from drifting too far from the SFT model and degrading language quality. RLHF is what makes models helpful, harmless, and honest, reducing toxic or off-task outputs, because raw next-token prediction optimizes for likelihood, not human usefulness. Simpler preference-optimization methods like DPO now often replace the explicit reward-model + PPO loop, directly optimizing on preference pairs while achieving similar alignment.
mediumWhat is instruction tuning and how does it differ from standard pretraining?
Pretraining teaches a model general language and world knowledge by next-token prediction over massive unlabeled text; the result is a strong text completer but not a good assistant, it may continue a question rather than answer it. Instruction tuning is supervised fine-tuning on a curated dataset of (instruction, desired response) pairs spanning many task types, formatted as natural-language commands. This teaches the model to follow instructions and generalize to unseen tasks in a zero/few-shot manner, aligning its behavior with how users actually prompt. Datasets like FLAN, Alpaca, and Dolly popularized it. It typically precedes preference alignment (RLHF/DPO). Compared to pretraining, it uses far less data, is much cheaper, and changes behavior/format more than underlying knowledge. It's the step that turns a raw base model into a usable chat/assistant model.
hardWhat is a Mixture-of-Experts (MoE) model and what advantage does it provide?
In a Mixture-of-Experts transformer, the dense feed-forward layer is replaced by many parallel expert sub-networks plus a lightweight gating/router network. For each token, the router selects a small subset (e.g. top-2 of 8 or more) of experts to process it, so only a fraction of parameters activate per token. This decouples total parameter count from per-token compute: a model can have hundreds of billions of parameters but only run a few billion FLOPs per token, giving greater capacity at roughly constant inference cost (sparse activation). Mixtral, DeepSeek, and others use this. Challenges include load balancing (auxiliary losses to keep experts evenly used), higher memory since all experts must be resident, and communication overhead in distributed training. MoE is a key technique for scaling model quality without proportionally scaling compute.
mediumWhat is model quantization and what are the tradeoffs of running an LLM in INT8 or INT4?
Quantization stores model weights (and sometimes activations) at lower numeric precision, converting FP16/BF16 to INT8, INT4, or FP8. Since LLM inference is largely memory-bandwidth bound, this roughly halves or quarters memory and speeds up loading and generation, letting large models fit on smaller GPUs. Post-training quantization (GPTQ, AWQ, bitsandbytes) needs no retraining; quantization-aware training bakes robustness in during training. The tradeoff is accuracy: aggressive INT4 can degrade quality, especially on reasoning, and outlier activations are hard to represent, so methods keep sensitive channels in higher precision or use per-group scales. INT8 is usually near-lossless; 4-bit is a popular quality/size sweet spot for local deployment. KV caches can also be quantized to save memory during long-context generation.
hardHow does LoRA make fine-tuning large models cheaper, and what is parameter-efficient fine-tuning?
Full fine-tuning updates all of a model's billions of weights, needing large memory for gradients and optimizer state and producing a full model copy per task. Parameter-efficient fine-tuning (PEFT) freezes the base weights and trains only a small set of new parameters. LoRA (Low-Rank Adaptation) is the most common: for a weight matrix W, it learns two small low-rank matrices A and B such that the update is delta_W = B*A, adding only a few million trainable parameters. Because rank r is tiny (e.g. 8-64), you cut trainable parameters and optimizer memory by orders of magnitude, train faster, and store lightweight per-task adapters that can be swapped or merged into W at inference with no added latency. QLoRA combines LoRA with a 4-bit quantized frozen base, enabling fine-tuning of huge models on a single GPU. Quality is usually close to full fine-tuning for most tasks.
hardWhy does self-attention scale quadratically with sequence length, and how do long-context models cope?
Standard self-attention compares every token with every other token: for a sequence of n tokens it computes an n-by-n attention matrix, so both compute and (naive) memory scale as O(n^2). Doubling context roughly quadruples attention cost, which is why long contexts are expensive. Coping strategies fall into a few families. Exact but memory-efficient kernels like FlashAttention avoid materializing the full matrix, reducing memory to O(n) and speeding things via tiling, without changing the math. Approximate/sparse attention (Longformer, BigBird, sliding-window as in Mistral) restricts each token to a local or structured subset of others, giving near-linear cost. Linear-attention and state-space models (Mamba) replace softmax attention with recurrences that scale linearly. Retrieval and context compression sidestep the issue by feeding only relevant chunks. RoPE scaling/interpolation extends usable length. The KV cache also grows linearly with n, compounding long-context memory pressure.
mediumWhat is multi-task learning in the context of LLMs, and why is it important?
Multi-task learning in the context of LLMs refers to the training approach where a model is trained on multiple tasks simultaneously rather than focusing on a single specific task. This is achieved by sharing parameters across tasks, allowing the model to learn generalized representations that can be beneficial across various applications. The importance of multi-task learning lies in its ability to improve the overall performance and robustness of the model since the model can leverage knowledge from related tasks, potentially reducing overfitting on any individual task and enhancing its ability to generalize to new, unseen data.
mediumWhat are the main techniques to improve training efficiency for LLMs?
To improve training efficiency for LLMs, several techniques are commonly employed: (1) **Gradient Accumulation**: This allows for larger effective batch sizes without requiring more GPU memory, making training more efficient; (2) **Mixed Precision Training**: Using lower precision (e.g., FP16) reduces computational resource requirements, speeding up training and allowing for larger batch sizes; (3) **Distributed Training**: Distributing the training process across multiple GPUs or nodes can significantly reduce the time needed to train large models; (4) **Curriculum Learning**: Training the model on easier tasks before advancing to more difficult tasks can help improve convergence; and (5) **Network Pruning**: Reducing the number of parameters in the model can lead to faster training times while maintaining similar performance.
mediumHow are LLMs trained and what are the stages involved?
LLMs are typically trained through a multi-stage process involving pre-training and fine-tuning. In the pre-training phase, the model learns language representations from a large corpus using unsupervised methods, usually through masked language modeling or next-token prediction. This phase captures grammar, semantics, and world knowledge. The fine-tuning phase involves supervised training on a narrower dataset tailored to specific tasks, utilizing labeled examples to improve the model’s performance on those tasks. Additional techniques like reinforcement learning from human feedback (RLHF) can further align the model’s outputs with human preferences.
mediumWhat are the trade-offs of using LLMs in practical applications?
Using LLMs in practical applications comes with several trade-offs including computational cost, interpretability, and responsiveness. LLMs require significant computational resources for both training and inference, which can be costly and limit deployment in resource-constrained environments. Additionally, their outputs can be complex and difficult to interpret, hindering user trust. While LLMs often produce high-quality responses, they can also generate irrelevant or incorrect information, necessitating effective methods for user oversight or filtering. Balancing performance, cost, and reliability is key to successful LLM deployment.
No questions match your filter.