What is a Mixture-of-Experts (MoE) model and what advantage does it provide?

In a Mixture-of-Experts transformer, the dense feed-forward layer is replaced by many parallel expert sub-networks plus a lightweight gating/router network. For each token, the router selects a small subset (e.g. top-2 of 8 or more) of experts to process it, so only a fraction of parameters activate per token. This decouples total parameter count from per-token compute: a model can have hundreds of billions of parameters but only run a few billion FLOPs per token, giving greater capacity at roughly constant inference cost (sparse activation). Mixtral, DeepSeek, and others use this. Challenges include load balancing (auxiliary losses to keep experts evenly used), higher memory since all experts must be resident, and communication overhead in distributed training. MoE is a key technique for scaling model quality without proportionally scaling compute.