How does LoRA make fine-tuning large models cheaper, and what is parameter-efficient fine-tuning?

Full fine-tuning updates all of a model's billions of weights, needing large memory for gradients and optimizer state and producing a full model copy per task. Parameter-efficient fine-tuning (PEFT) freezes the base weights and trains only a small set of new parameters. LoRA (Low-Rank Adaptation) is the most common: for a weight matrix W, it learns two small low-rank matrices A and B such that the update is delta_W = B*A, adding only a few million trainable parameters. Because rank r is tiny (e.g. 8-64), you cut trainable parameters and optimizer memory by orders of magnitude, train faster, and store lightweight per-task adapters that can be swapped or merged into W at inference with no added latency. QLoRA combines LoRA with a 4-bit quantized frozen base, enabling fine-tuning of huge models on a single GPU. Quality is usually close to full fine-tuning for most tasks.