AIght_
ToolsCompareLearnUniverseWorkflowsSignalHuman
Take the quiz
← All concepts

Concept

Mixture of Experts

How modern models pretend to be huge while doing the work of something smaller.

Moon·Updated May 17, 2026
◈ Learn it interactively8 steps · ~6 min →

Where this idea lives

PREREQUISITESTOOLS THAT SHOW ITMixture of ExpertsTransformersTransformers — The architecture that changed what AI could do with language — and then everything elseScaling LawsScaling Laws — Why bigger keeps working — and the question of where it stops.How AI Models Are TrainedHow AI Models Are Trained — From random noise to a model that can reason — the actual pipelineDeepSeekDeepSeekClaudeClaudeGeminiGeminiCommon misconception: A 670B MoE model uses 670B parameters per token.Common misconception: More experts always means better quality.Common misconception: MoE replaces dense models.
prereqsrelatedtoolsmisconceptions
shows up in:Software EngineeringPhysics & Engineering
You might think:A 670B MoE model uses 670B parameters per token.More experts always means better quality.MoE replaces dense models.

Common misconception

“A 670B MoE model uses 670B parameters per token.”

A typical MoE routes each token through only 2 of, say, 64 experts. So a 670B-parameter MoE might activate ~40B per token. The headline number is the capacity; the per-token compute is much smaller. This is the trick — huge model, cheap-ish inference.

A mixture-of-experts (MoE) model splits each feed-forward block into N "experts." A small router network picks which experts to send each token through. Only the chosen experts compute; the rest sit idle.

You get the benefit of a vast parameter pool — different experts can specialise on different patterns — without paying full compute per token.

◉ INTERACTIVE

Step 1 of 4

8 experts in the layer

Every feed-forward block contains N specialist networks. At rest, none are active.

E1
E2
E3
E4
E5
E6
E7
E8

All 8 experts exist; only 2 fire. Sparse compute, dense capability.

Why it works

  • Capacity without cost. Total parameters grow without proportional compute increase. DeepSeek-V3 has 670B parameters but activates ~37B per token.
  • Implicit specialisation. Experts naturally end up handling different domains: code, math, multilingual, etc. — without anyone designing this.

Where it bites

  • Memory. All experts must live in GPU memory; activating a few doesn't save VRAM, only FLOPs.
  • Routing instability. Bad routing wastes capacity; training MoEs is finicky.
  • Inference batching. Different tokens route to different experts, hurting batch efficiency.

What to read next

Scaling laws explain why bigger keeps working. Quantization and distillation are the alternative paths to capability-per-dollar.

◈ Check yourself

You just read it — now see if it stuck. No score, no penalty for a wrong pick.

Question 1 of 3

What does a mixture-of-experts (MoE) model do?

Question 2 of 3

A 700B-parameter MoE model can run cheaper than a 70B dense model. How?

Question 3 of 3

What does the "router" in a mixture-of-experts model do?

← Prerequisite

Transformers

Next →

Fine-Tuning

by Moon · poet · engineer · third-year CS, Chandigarh University · Updated May 17, 2026

← Back to all conceptsBrowse tools →
intermediate
Read time5 min read
UpdatedMay 2026
Sources4

Read next

  1. Transformers →
  2. Scaling Laws →
  3. How AI Models Are Trained →