Mixture of Experts (MoE)

Mixture of Experts (MoE) is a neural-network architecture in which the feedforward layer of a transformer block is replaced by many parallel "expert" sub-networks, with a learned router selecting a small number of experts to process each token. Only the selected experts' parameters are used for a given token, so total parameters can grow while compute per token stays roughly constant. It is the dominant architecture for compute-efficient frontier models and the reason model descriptions now distinguish "total parameters" from "active parameters."

Dense versus sparse models

In a dense model, every parameter participates in every forward pass; GPT-3, Llama, and older Claude checkpoints are dense. In a sparse (MoE) model, most parameters sit idle for any given token and only the active experts contribute. Training and inference FLOPs scale with active parameters, while model capacity scales with total parameters.

This divergence is the origin of the "total params / active params" notation now used to describe MoE models:

Mechanism

A router, usually a small linear layer, scores each token against each expert. Top-k routing then selects the top k experts (often k=2 or k=8) per token, the token is dispatched to those experts, and their outputs are weighted-combined. Load-balancing losses discourage the router from collapsing onto a few popular experts, which would leave the rest of the model unused.

Common variants include top-2 gating (Switch), expert-choice routing (in which experts pick tokens rather than the reverse), shared-plus-routed expert hybrids (DeepSeek-V3), and fine-grained experts. DeepSeek-MoE's contribution was the use of many small experts rather than a few large ones.

Origin and history

Modern MoE in deep learning was introduced by Shazeer et al. in "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (2017). GShard (Google, 2020) scaled MoE transformers past 600B parameters using expert parallelism. The Switch Transformer (Fedus et al., 2021) simplified routing to top-1 and demonstrated training-efficiency gains, and GLaM (Google, 2021), a 1.2T MoE, matched dense GPT-3 at roughly a third of the energy. Mixtral 8x7B (Mistral, December 2023) was the first widely deployed open-weights MoE and brought the pattern into mainstream use. DeepSeek-MoE and DeepSeek-V3 popularized fine-grained experts, shared experts, and auxiliary-loss-free load balancing (see DeepSeek-V3), and the Qwen3 and Kimi K2 technical reports (2025) extended the MoE recipe to multi-trillion-parameter scale.

Adoption in frontier models

Compute-constrained labs, the fast-follow cohort described in Fast-Follow Problem, gravitated to MoE for several reasons. It provides more capacity per FLOP: a trillion-parameter MoE can be trained with compute budgets closer to those of a 50B dense model. Inference is cheaper because serving cost tracks active rather than total parameters; Kimi K2's 32B active parameters make it feasible to serve a 1T model on modest hardware. MoE also offers distillation synergy, since large MoE teachers are cheap to run and provide rich signal for distilling into dense students (see Distillation). Closed Western labs use MoE as well — GPT-4 is widely reported to be an MoE model — though they have historically disclosed less about their architectural choices.

Every credible open frontier model released after mid-2024 has been MoE. Hardware vendors including Nvidia (GB200), AMD (MI300), and Cerebras now market MoE throughput as a first-class benchmark. Active research areas include memory-efficient expert offloading, expert merging, and cross-modal expert specialization.

Relation to policy and safety

Regulations indexed to training FLOPs, such as the EU AI Act (Regulation 2024/1689) GPAI thresholds and EO 14110's now-rescinded 10^26 threshold, interact awkwardly with MoE: total-parameter counts over-state capability while training FLOPs understate the effective capacity gained. MoE partially offsets export controls by letting compute-poor labs reach higher capability per FLOP. It also complicates evaluation, since benchmark-claim comparability suffers when a "235B model" can correspond to very different compute regimes.

Relationships

• depends-on: Scaling Laws — MoE is a scaling-efficiency lever
• related: Fast-Follow Problem — MoE is central to how DeepSeek/Qwen/Kimi narrowed the gap
• related: Distillation — MoE teachers distill well into dense students
• related: Compute Governance — complicates FLOP-based thresholds
• instance-of: DeepSeek-V3, Qwen3, Kimi K2 — concrete frontier MoE deployments