What is gradient descent and how does it work?

Gradient descent is an iterative optimization algorithm that minimizes a loss function by adjusting model parameters in the direction opposite to the gradient of the loss with respect to those parameters. At each step it computes the gradient, then updates weights: w := w - learning_rate * gradient. The learning rate controls step size. Variants differ by how much data is used per update: batch gradient descent uses the full dataset, stochastic (SGD) uses one example, and mini-batch uses a small subset, trading off gradient noise against computational efficiency. It converges to a minimum of convex functions; for non-convex problems like neural networks it finds a local minimum or saddle-adjacent region, which in practice is often good enough.