The AI school where most of the staff sit out

Some of the largest AI language models contain more than a trillion learned numbers, yet for each word they process they use only about three percent of them. The design behind that is called mixture of experts.

It is now common enough that you have probably heard the phrase and nodded. This piece is for the nod. It walks through what the idea means, why it saves work, what it costs, and why the big number on a model's spec sheet tells you much less than you would think.

What does a model's size actually measure?

Start with the word "parameter." A parameter is one of the adjustable numbers a model sets during training. Taken together they hold everything the model learned. When a model is described as having 600 billion parameters, or a trillion, that is a count of those numbers. It describes the model's capacity, how much it could in principle hold. ^[3]

The obvious assumption is that every one of those numbers goes to work every time you type a question. In older, so-called dense models that is what happens. Each piece of text, which the field calls a token and which is usually a word or a chunk of one, passes through all of them.

Mixture of experts breaks that assumption. The model keeps a big pool of parameters but sends each token through only a small part of it. ^[3]

What is a mixture of experts?

Picture a school with a great many helpers. A question arrives and is cut into little pieces. For each piece, a teacher points to a few helpers, who work on it. The teacher then combines their answers. The school employs every helper, but it does not make all of them work on every piece.

In a model, the teacher is called the router and the helpers are the experts. The router is a small component that learns, during training, how to choose. ^[1]

Here is what happens in a single layer, according to the authors of one research paper on the subject. The router scores all the available experts for a token. It selects a small number of the top scorers. Then it combines their outputs, giving each a weight.

The experts themselves are feed-forward sub-networks, the parts of the model that transform the token's internal representation. ^[4]

Two details matter. First, the answer is a weighted mixture, not a verdict from one winner. Second, this happens again and again. A model has many layers, and routing is repeated across its mixture-of-experts layers, so a token's path can differ from layer to layer and from token to token. ^[4]

How big is the saving?

The numbers are striking.

The team behind Kimi K2 reports about 1.04 trillion total parameters and 32 billion activated ones. Each of its mixture-of-experts layers holds 384 experts. Eight are active for any token, plus one shared expert that is always used. ^[1]

DeepSeek-V3 is reported at 671 billion total parameters and 37 billion active per token. One paper describes it as using 256 routed experts, with eight selected per token, plus a shared expert. ^[3]

Do the division. Kimi K2 activates roughly one parameter in 32. DeepSeek-V3 activates roughly one in 18. Inside a layer of Kimi K2, the router picks eight experts out of 384, so it calls on one expert in 48.

That is why total size is a poor stand-in for the work done on each request. Two models with the same headline size can behave very differently if one runs the whole pool each time and the other runs a thirtieth of it. These ratios belong to these models, though. They are not a universal rule for the technique. ^[1][3]

"Active parameters" has a precise meaning. It is the count of parameters used for one token's forward pass, the single trip of that token through the model. It is not the whole pool. It is also not a tally of everything used across all the tokens in a long conversation, which would be far larger. ^[3][4]

Why does that save compute?

Because most experts sit out. The saving comes from not running every expert for every token. ^[3]

It is not a saving on everything, though. The router has to do its scoring. The chosen experts still do their work.

And a model has other parts besides the experts. Attention, the step that lets each token look at the other words in the text, still contributes to every forward pass. ^[3][4]

So the honest description is this. Mixture of experts trims the expensive part of the model, the bulk of its parameters, down to a slice per token. It does not shrink the model's whole bill to a slice.

So where is the catch?

You might reasonably suspect there is one. A model with a trillion parameters that runs like a 32-billion one sounds like a free lunch. It is not.

The first cost is memory. Sparse activation does not make the unused experts vanish. The complete set of expert parameters still has to be available, both for training and for answering questions, even though only a subset is used for any one token. ^[3] The school must keep all its helpers, with their desks and rooms, even on days when most of them stand idle. A large pool means a substantial memory footprint across the hardware that serves the model. ^[3]

The second cost is movement. A big model's experts are spread over many devices. When the router picks experts for a token, the token's data has to travel to whichever devices hold them. In training, the system must also update all those experts and keep the traffic organised. ^[2]

NVIDIA's technical report on training these models with its Megatron Core software describes the problem as three linked challenges: memory, communication between devices and compute efficiency. They are coupled, the report says, because easing one constraint can shift pressure onto another. ^[2] That is a vendor's own engineering document, so it is better evidence about the difficulty of the work than about how one provider compares with another.

The third cost is balance. Routing and balancing expert workloads carry real costs of their own. If the router keeps favouring a few experts, those become bottlenecks while the rest idle. ^[1][2]

Put together, these costs mean the active-parameter count is not a full measure of speed, energy use or what a request costs to serve. Hardware, memory movement, batching, how evenly the router spreads the load and the rest of the architecture all matter. ^[1][2][3]

Do the experts really specialise?

The word "expert" invites a tidy picture. One expert for grammar, one for chemistry, one for French. It is a good story, and the research does not support it as settled fact.

The router decides by mathematical scores, not by reading job titles. The experts are learned sub-networks inside a layered system, and nobody assigns them roles. ^[4] Studies of routing report that selections can reflect features such as syntax rather than broad subject areas. ^[4]

There is evidence that the choice of experts matters, though. In experiments, researchers found that changing which experts are active could alter a model's behaviour. ^[4] That shows the experts are doing distinct work. It does not show that each has a simple, stable specialty you could describe in a sentence. ^[4]

So "specialist" is best read as what the design intends, not as a promise that each expert has a clean human-readable label. The school picture needs the same care. The teacher is a set of scores learned in training, not a person making a judgement, and the choice is made again at every layer, for every token. ^[4]

What to watch

The framework is clear. The details in deployed models are not. The question to put to any new model is how much of its theoretical cut in active computation survives contact with real hardware, once memory, communication between devices and uneven use of experts take their share. ^[2]

Until that is answered, treat a model's total parameter count as the size of its staff and its active count as the number of helpers called on for each word. Neither tells you, on its own, how fast or how cheaply the model runs.