What is a loss function, and how do you choose one?

A loss function measures how far a model's predictions are from the true values; training minimizes it. The choice depends on the task. For regression, mean squared error (MSE) penalizes large errors heavily and assumes Gaussian noise, while mean absolute error (MAE) is more robust to outliers. For binary classification, use binary cross-entropy (log loss), which penalizes confident wrong predictions; for multi-class, use categorical cross-entropy. Hinge loss is used for SVMs. The loss should match the output distribution and the cost of different error types in your problem. Note the distinction from an evaluation metric: the loss must be differentiable for gradient-based training, whereas a metric like accuracy is what you actually care about but may not be optimizable directly.