LLMs & Transformers: Interview Questions
The questions that dominate 2026 AI interviews — how transformers and attention work, what tokens and context windows are, why models hallucinate, and how temperature and fine-tuning shape behaviour.
mediumWhat is a transformer, and why did it change NLP?
A transformer is a neural-network architecture (introduced in 'Attention Is All You Need', 2017) built around self-attention rather than recurrence. It processes all tokens in parallel and lets each token attend to every other, capturing long-range relationships efficiently. This parallelism made training on massive data practical and scaled far better than RNNs/LSTMs — which is why virtually every modern LLM is a transformer.
hardExplain self-attention in simple terms.
Self-attention lets each token decide how much to 'pay attention' to every other token when building its representation. Each token produces a query, key, and value; the query is compared against all keys to get attention weights, which then combine the values. In 'the animal didn't cross the street because it was tired', attention helps 'it' link to 'animal'. Multi-head attention runs several such mechanisms in parallel to capture different kinds of relationships.
easyWhat is a token in an LLM?
A token is the unit an LLM reads and writes — a chunk of text roughly 4 characters or ¾ of a word in English. Text is split by a tokeniser; common words are one token, rarer words split into several, and punctuation and spaces count too. Models have a maximum token limit (the context window), and APIs bill per token, so tokens matter for both capability and cost.
easyWhat is a context window and why does it matter?
The context window is the maximum number of tokens (input + output) a model can consider in a single request. Anything beyond it is truncated. It matters because it bounds how much document, conversation, or code the model can reason over at once — and because models tend to use the start and end of a long context better than the middle ('lost in the middle'). Bigger isn't automatically better; relevant, well-placed context beats sheer volume.
mediumWhy do LLMs hallucinate, and how do you reduce it?
LLMs are trained to produce plausible next tokens, not verified truth, so when they lack knowledge they still generate a confident-sounding answer. Reduce it with retrieval-augmented generation (ground answers in real sources), instructing the model to say 'I don't know' when unsure, lowering temperature, asking for citations you can check, and adding verification steps. You can lower the rate substantially but not to zero — human verification of important facts remains necessary.
easyWhat does the temperature parameter do?
Temperature controls randomness in token selection. Low temperature (near 0) makes the model pick the highest-probability tokens — more deterministic and focused, good for factual or structured tasks. Higher temperature flattens the distribution so less-likely tokens get chosen more often — more varied and creative, good for brainstorming, but also more prone to drifting or hallucinating. Top-p (nucleus) sampling is a related control over the candidate pool.
hardWhen would you fine-tune an LLM versus use RAG?
Use RAG when answers must be grounded in factual, changing, or private data — you update documents, not weights, and get citations. Use fine-tuning to teach a consistent style, format, or narrow behaviour the base model can't reliably follow through prompting. RAG is usually cheaper to keep current; fine-tuning needs curated data and re-runs when things change. They combine well: RAG for knowledge, light fine-tuning for tone. Reach for prompting first — it solves more than people expect.
mediumWhat's the difference between pre-training and fine-tuning?
Pre-training builds the base model from scratch on enormous general text to learn language and world patterns — hugely expensive, done by frontier labs. Fine-tuning takes that pre-trained model and trains it further on smaller, task- or domain-specific data to specialise it (including instruction-tuning and RLHF that make base models good at following instructions). Pre-training gives broad capability; fine-tuning shapes it toward specific behaviour.
Learn this: Training vs Inference (and Why It Matters for Cost) →
mediumWhat is the difference between encoder-only, decoder-only, and encoder-decoder transformer architectures?
Encoder-only models (e.g. BERT) process the full input bidirectionally with unmasked self-attention, producing rich contextual embeddings. They excel at understanding tasks like classification, NER, and retrieval, but can't generate text autoregressively. Decoder-only models (e.g. GPT, Llama) use causal (masked) self-attention so each token attends only to prior tokens, making them natural next-token generators used for most modern LLMs. Encoder-decoder models (e.g. T5, original Transformer) pair a bidirectional encoder with an autoregressive decoder that cross-attends to the encoder output; they suit sequence-to-sequence tasks like translation and summarization where input and output are distinct sequences. Decoder-only dominates today because it scales simply and handles both understanding and generation via prompting.
mediumWhy do transformers need positional encodings, and how do sinusoidal and RoPE approaches differ?
Self-attention is permutation-invariant: it treats input as a set, so without position information 'dog bites man' and 'man bites dog' look identical. Positional encodings inject order. The original transformer added fixed sinusoidal vectors (sines/cosines of different frequencies) to token embeddings, letting the model infer relative distances and extrapolate somewhat to unseen lengths. Learned absolute embeddings (BERT/GPT-2) instead train a position vector per index, but don't extrapolate past training length. RoPE (Rotary Position Embedding, used in Llama, etc.) rotates the query and key vectors by an angle proportional to position, so the dot product naturally encodes relative position. RoPE is now standard because it preserves relative-distance information, works well with long contexts, and supports interpolation/scaling tricks to extend context length.
mediumWhat is multi-head attention and why is it better than a single attention head?
Multi-head attention runs several attention operations in parallel, each with its own learned query, key, and value projections. The model embedding is split into h lower-dimensional subspaces; each head computes scaled dot-product attention independently, and the outputs are concatenated and projected back. The benefit is representational diversity: different heads can specialize, one tracking syntactic dependencies, another coreference, another local n-gram patterns, so the model attends to information from multiple representation subspaces at different positions simultaneously. A single head must average all these relationships into one attention distribution, which is a bottleneck. Importantly, multiple heads cost roughly the same as one because the per-head dimension is d/h, keeping total compute constant. Variants like grouped-query and multi-query attention share keys/values across heads to shrink the KV cache.
hardWhat is the KV cache in transformer inference and what problem does it solve?
During autoregressive generation, a decoder produces one token at a time, and each new token attends to all previous tokens. Naively, you'd recompute the key and value vectors for the entire sequence at every step, making generation O(n^2) redundant work. The KV cache stores the key and value tensors already computed for past tokens, so at each step you only compute Q, K, V for the single new token and append its K,V to the cache. This turns per-step attention cost from quadratic to linear in sequence length and is the main reason generation is fast. The tradeoff is memory: the cache grows linearly with context length and batch size (2 * layers * heads * head_dim * tokens), often becoming the dominant memory consumer for long contexts. Techniques like multi-query/grouped-query attention, paged attention (vLLM), and quantized caches reduce its footprint.
easyWhat is the difference between a token and an embedding in an LLM?
A token is a discrete unit of text produced by the tokenizer, typically a subword piece (via BPE, WordPiece, or SentencePiece) represented as an integer ID from a fixed vocabulary. Tokens are symbolic: ID 1234 carries no inherent meaning. An embedding is the dense, continuous vector (e.g. 4096 floats) that the model looks up for each token ID from its embedding matrix. Embeddings live in a learned geometric space where semantically related concepts sit near each other, so the model can compute with meaning rather than opaque IDs. In short: tokenization maps text to token IDs, and the embedding layer maps each ID to a vector. Every transformer layer then transforms these vectors into increasingly contextual representations. 'Embeddings' can also refer to the final pooled sentence/document vectors used for search and RAG.
mediumHow do top-k and top-p (nucleus) sampling control LLM text generation?
Both truncate the probability distribution over the next token before sampling, to avoid picking absurd low-probability tokens while keeping variety. Top-k keeps only the k highest-probability tokens, renormalizes, and samples from them; k is fixed, so in a very confident context you may still include unlikely options, and in a flat distribution you may exclude good ones. Top-p (nucleus) instead keeps the smallest set of tokens whose cumulative probability exceeds p (e.g. 0.9), so the candidate pool grows when the model is uncertain and shrinks when it's confident, making it more adaptive. They're often combined with temperature, which reshapes the distribution first. Lower p/k yields safer, more deterministic text; higher values yield more diverse, creative, or risky output. Setting k=1 is equivalent to greedy decoding.
hardWhat is RLHF and why is it used to align language models?
RLHF (Reinforcement Learning from Human Feedback) aligns a pretrained/instruction-tuned model with human preferences. It has three stages: (1) supervised fine-tuning on demonstration data to get a reasonable base; (2) train a reward model by having humans rank multiple model responses to the same prompt, teaching a model to predict which output people prefer; (3) optimize the LLM against that reward model using RL, typically PPO, with a KL-divergence penalty that keeps the policy from drifting too far from the SFT model and degrading language quality. RLHF is what makes models helpful, harmless, and honest, reducing toxic or off-task outputs, because raw next-token prediction optimizes for likelihood, not human usefulness. Simpler preference-optimization methods like DPO now often replace the explicit reward-model + PPO loop, directly optimizing on preference pairs while achieving similar alignment.
mediumWhat is instruction tuning and how does it differ from standard pretraining?
Pretraining teaches a model general language and world knowledge by next-token prediction over massive unlabeled text; the result is a strong text completer but not a good assistant, it may continue a question rather than answer it. Instruction tuning is supervised fine-tuning on a curated dataset of (instruction, desired response) pairs spanning many task types, formatted as natural-language commands. This teaches the model to follow instructions and generalize to unseen tasks in a zero/few-shot manner, aligning its behavior with how users actually prompt. Datasets like FLAN, Alpaca, and Dolly popularized it. It typically precedes preference alignment (RLHF/DPO). Compared to pretraining, it uses far less data, is much cheaper, and changes behavior/format more than underlying knowledge. It's the step that turns a raw base model into a usable chat/assistant model.
hardWhat is a Mixture-of-Experts (MoE) model and what advantage does it provide?
In a Mixture-of-Experts transformer, the dense feed-forward layer is replaced by many parallel expert sub-networks plus a lightweight gating/router network. For each token, the router selects a small subset (e.g. top-2 of 8 or more) of experts to process it, so only a fraction of parameters activate per token. This decouples total parameter count from per-token compute: a model can have hundreds of billions of parameters but only run a few billion FLOPs per token, giving greater capacity at roughly constant inference cost (sparse activation). Mixtral, DeepSeek, and others use this. Challenges include load balancing (auxiliary losses to keep experts evenly used), higher memory since all experts must be resident, and communication overhead in distributed training. MoE is a key technique for scaling model quality without proportionally scaling compute.
mediumWhat is model quantization and what are the tradeoffs of running an LLM in INT8 or INT4?
Quantization stores model weights (and sometimes activations) at lower numeric precision, converting FP16/BF16 to INT8, INT4, or FP8. Since LLM inference is largely memory-bandwidth bound, this roughly halves or quarters memory and speeds up loading and generation, letting large models fit on smaller GPUs. Post-training quantization (GPTQ, AWQ, bitsandbytes) needs no retraining; quantization-aware training bakes robustness in during training. The tradeoff is accuracy: aggressive INT4 can degrade quality, especially on reasoning, and outlier activations are hard to represent, so methods keep sensitive channels in higher precision or use per-group scales. INT8 is usually near-lossless; 4-bit is a popular quality/size sweet spot for local deployment. KV caches can also be quantized to save memory during long-context generation.
hardHow does LoRA make fine-tuning large models cheaper, and what is parameter-efficient fine-tuning?
Full fine-tuning updates all of a model's billions of weights, needing large memory for gradients and optimizer state and producing a full model copy per task. Parameter-efficient fine-tuning (PEFT) freezes the base weights and trains only a small set of new parameters. LoRA (Low-Rank Adaptation) is the most common: for a weight matrix W, it learns two small low-rank matrices A and B such that the update is delta_W = B*A, adding only a few million trainable parameters. Because rank r is tiny (e.g. 8-64), you cut trainable parameters and optimizer memory by orders of magnitude, train faster, and store lightweight per-task adapters that can be swapped or merged into W at inference with no added latency. QLoRA combines LoRA with a 4-bit quantized frozen base, enabling fine-tuning of huge models on a single GPU. Quality is usually close to full fine-tuning for most tasks.
hardWhy does self-attention scale quadratically with sequence length, and how do long-context models cope?
Standard self-attention compares every token with every other token: for a sequence of n tokens it computes an n-by-n attention matrix, so both compute and (naive) memory scale as O(n^2). Doubling context roughly quadruples attention cost, which is why long contexts are expensive. Coping strategies fall into a few families. Exact but memory-efficient kernels like FlashAttention avoid materializing the full matrix, reducing memory to O(n) and speeding things via tiling, without changing the math. Approximate/sparse attention (Longformer, BigBird, sliding-window as in Mistral) restricts each token to a local or structured subset of others, giving near-linear cost. Linear-attention and state-space models (Mamba) replace softmax attention with recurrences that scale linearly. Retrieval and context compression sidestep the issue by feeding only relevant chunks. RoPE scaling/interpolation extends usable length. The KV cache also grows linearly with n, compounding long-context memory pressure.
mediumWhat is multi-task learning in the context of LLMs, and why is it important?
Multi-task learning in the context of LLMs refers to the training approach where a model is trained on multiple tasks simultaneously rather than focusing on a single specific task. This is achieved by sharing parameters across tasks, allowing the model to learn generalized representations that can be beneficial across various applications. The importance of multi-task learning lies in its ability to improve the overall performance and robustness of the model since the model can leverage knowledge from related tasks, potentially reducing overfitting on any individual task and enhancing its ability to generalize to new, unseen data.
mediumWhat are the main techniques to improve training efficiency for LLMs?
To improve training efficiency for LLMs, several techniques are commonly employed: (1) **Gradient Accumulation**: This allows for larger effective batch sizes without requiring more GPU memory, making training more efficient; (2) **Mixed Precision Training**: Using lower precision (e.g., FP16) reduces computational resource requirements, speeding up training and allowing for larger batch sizes; (3) **Distributed Training**: Distributing the training process across multiple GPUs or nodes can significantly reduce the time needed to train large models; (4) **Curriculum Learning**: Training the model on easier tasks before advancing to more difficult tasks can help improve convergence; and (5) **Network Pruning**: Reducing the number of parameters in the model can lead to faster training times while maintaining similar performance.
mediumHow are LLMs trained and what are the stages involved?
LLMs are typically trained through a multi-stage process involving pre-training and fine-tuning. In the pre-training phase, the model learns language representations from a large corpus using unsupervised methods, usually through masked language modeling or next-token prediction. This phase captures grammar, semantics, and world knowledge. The fine-tuning phase involves supervised training on a narrower dataset tailored to specific tasks, utilizing labeled examples to improve the model’s performance on those tasks. Additional techniques like reinforcement learning from human feedback (RLHF) can further align the model’s outputs with human preferences.
mediumWhat are the trade-offs of using LLMs in practical applications?
Using LLMs in practical applications comes with several trade-offs including computational cost, interpretability, and responsiveness. LLMs require significant computational resources for both training and inference, which can be costly and limit deployment in resource-constrained environments. Additionally, their outputs can be complex and difficult to interpret, hindering user trust. While LLMs often produce high-quality responses, they can also generate irrelevant or incorrect information, necessitating effective methods for user oversight or filtering. Balancing performance, cost, and reliability is key to successful LLM deployment.
mediumWhat is fine-tuning in the context of LLMs, and when is it typically applied?
Fine-tuning in the context of LLMs refers to the process of taking a pre-trained language model and continuing its training on a smaller, task-specific dataset. This is typically applied when you want to adapt a general-purpose model to perform better on a specific task, such as sentiment analysis, question answering, or domain-specific language generation. Fine-tuning allows the model to learn more nuanced patterns relevant to the specific application while leveraging the general knowledge acquired during pre-training, thereby enhancing performance on the intended task.
mediumWhat is the purpose of layer normalization in transformer architectures?
Layer normalization is a technique used in transformer architectures to stabilize and improve the training process by normalizing the outputs of each layer across the features, instead of across the batch. This helps to mitigate issues related to internal covariate shift, where the distribution of layer inputs changes during training. By ensuring that the mean and variance of inputs to the layer are consistent, layer normalization enhances convergence speed, reduces sensitivity to hyperparameters, and often leads to better overall performance. It is especially beneficial in deep networks, where managing gradients is crucial.
mediumWhat are attention masks in transformer models and how do they function?
Attention masks are binary or float vectors that indicate which tokens in the input sequence should be attended to and which should not. In most transformer implementations, they are used to prevent the model from attending to padding tokens or future tokens in the case of decoder models. For example, in sequence-to-sequence tasks, attention masks ensure that the model processes only the relevant parts of the input, thereby enhancing performance and efficiency.
mediumWhat is layer-wise learning rate decay, and why is it beneficial for training transformers?
Layer-wise learning rate decay is a training strategy that applies different learning rates to different layers of a neural network, typically with lower rates for lower layers and higher rates for upper layers. This approach is beneficial for transformers because it allows the model to learn more complex features in deeper layers while stabilizing the training of earlier layers. It helps prevent overfitting in lower layers that are already well-trained and encourages faster convergence overall.
mediumHow can bias in LLMs be identified and mitigated?
Bias in LLMs can often be identified through testing with diverse datasets that capture a wide range of perspectives and contexts. One common method is to evaluate model outputs for potentially biased associations or language. To mitigate bias, techniques such as data augmentation, adversarial training, and employing fairness-aware algorithms can be used. Additionally, fine-tuning on carefully curated datasets aimed at reducing bias can help improve model behavior. It's important to continuously monitor and assess bias in model outputs even after deployment.
mediumWhat is zero-shot learning in LLMs and how does it work?
Zero-shot learning in LLMs refers to the model's ability to understand and generate responses for tasks it was not explicitly trained on. It relies on the model's generalization ability and the semantic similarity of the information it has seen during training. For instance, when a query is presented that involves a new task or domain, the LLM can leverage its understanding of language and context to provide a relevant answer based on analogous tasks it has learned from. This capability is particularly useful in applications where specific training data for every potential task is impractical.
mediumWhat are the main parameters of a transformer model, and how do they affect model performance?
The main parameters of a transformer model include the number of layers (depth), the hidden size (width), the number of attention heads, and the feed-forward network size. The number of layers determines how deep the model is and can capture more complex patterns but may lead to overfitting if too many layers are used without sufficient data. The hidden size affects the capacity of the model to learn and represent information; larger sizes can enhance performance but require more memory and computation. The number of attention heads allows the model to focus on different parts of the input simultaneously, improving its ability to capture relationships within the data. Finally, the size of the feed-forward network between the attention layers can influence the model's ability to combine features effectively.
mediumHow does the choice of training data influence the performance of LLMs?
The choice of training data significantly impacts the performance of LLMs because it determines the knowledge, biases, and linguistic styles that the model learns. Inclusive and diverse datasets lead to more generalized models that can better understand and generate content across various topics and demographics. Conversely, if the training data is limited or biased, the resulting model may produce skewed outputs or fail to generalize well to unseen instances. Data quality, volume, and relevance directly correlate with the model's effectiveness, making careful dataset curation essential for optimal performance.
hardWhat are contrastive learning techniques and how do they benefit LLMs?
Contrastive learning techniques aim to learn representations by contrasting positive pairs (similar items) with negative pairs (dissimilar items). In the context of LLMs, these techniques can enhance model understanding of nuanced language variations and contextual meanings. By training models on the task of differentiating between similar and dissimilar contexts, LLMs develop richer embeddings that capture semantic relationships more effectively. This method can improve various downstream tasks, such as sentiment analysis and information retrieval, leading to better performance and more coherent generation.
mediumWhat is the rationale behind the pretraining phase of LLMs?
The pretraining phase of Large Language Models (LLMs) aims to teach the model to understand language structure and semantics before it is fine-tuned on specific tasks. During this phase, models are usually trained on large corpora of text data with unsupervised learning methods, allowing them to learn linguistic patterns, word associations, and contextual information. This foundational knowledge is critical as it enables the model to perform well on a variety of downstream tasks with minimal additional training.
mediumWhat are common evaluation metrics used to assess the performance of LLMs?
Common evaluation metrics for assessing the performance of LLMs include BLEU (Bilingual Evaluation Understudy) for translation tasks, ROUGE (Recall-Oriented Understudy for Gisting Evaluation) for summarization, and accuracy or F1 scores for classification tasks. For generative tasks, metrics like perplexity are also used to quantify how well a probability model predicts a sample. Additionally, human evaluations are often utilized to assess fluency, relevance, and coherence, providing a qualitative measure of model performance.
No questions match your filter.