Mixture-of-Experts

An architecture where a model contains multiple specialized 'expert' sub-networks, but only activates a subset of them for any given input. This allows models to have massive total parameter counts while keeping inference costs manageable — only the relevant experts fire for each token. MoE is the architecture behind Llama 4's ability to pack 2 trillion parameters while only using 17-288 billion per forward pass, enabling frontier-class performance at a fraction of the compute cost of dense models.