Goal: understand the architecture that most large models now use.
Why this step matters¶
A Mixture of Experts (MoE) model replaces one big feed-forward layer with many smaller expert layers, plus a small router that picks only two or three of them for each token.
Think of a company with fifty specialists. For any one job you only call in two of them. You get the knowledge of fifty people at the cost of consulting two.
What you do¶
Understand total versus active parameters. A model may have 30 billion parameters in total but use only 3 billion for any given token. Both numbers matter, for different reasons: total decides your memory, active decides your speed.
Understand routing. A small network looks at each token and picks the top few experts for it.
Understand load balancing. Left alone, the router sends everything to a few favourite experts and the rest never learn anything. An auxiliary loss — or newer loss-free balancing methods — prevents this.
Understand shared experts. Some designs keep one expert that every token always uses, for the general knowledge, while the routed experts specialise.
Understand the honest cost. MoE saves compute but not memory. Every expert has to sit in memory even though most are idle at any moment. Serving is also more complex, because experts may live on different devices.
Know when not to use it. For a single-GPU model, MoE is usually the wrong choice. Say so plainly in your write-up.
Learn about upcycling. You can turn an already-trained dense model into an MoE by copying its feed-forward layer into several experts and then training the router. Cheaper than starting over.
You are ready to move on when¶
You can explain in two sentences why MoE helps and what it costs.