How a mixture-of-experts model activates only part of its parameters

The Switch Transformers paper says models typically reuse the same parameters for all inputs, and that Mixture of Experts instead selects different parameters for each incoming example. Quote: "models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example."

In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example.

The Switch Transformers paper says the result is a sparsely-activated model with outrageous numbers of parameters but a constant computational cost. Quote: "The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost."

The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost.

The Mixtral paper says each layer of Mixtral 8x7B is composed of 8 feedforward blocks (experts) and that for every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Quote: "each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs."

each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs.

The Mixtral paper says its gating is the softmax over the Top-K logits of a linear layer, and that if the gating vector is sparse the outputs of experts whose gates are zero can be avoided. Quote: "If the gating vector is sparse, we can avoid computing the outputs of experts whose gates are zero."

If the gating vector is sparse, we can avoid computing the outputs of experts whose gates are zero. There are multiple alternative ways of implementing G(x) [6, 15, 35], but a simple and performant one is implemented by taking the softmax over the Top-K logits of a linear layer [28].

The Mixtral paper distinguishes the total parameter count, which grows with the number of experts n, from the active parameter count used to process an individual token, which grows with K up to n. Quote: "the model’s total parameter count (commonly referenced as the sparse parameter count), which grows with n n , and the number of parameters used for processing an individual token (called the active parameter count), which grows with K K up to n n ."

This motivates a distinction between the model’s total parameter count (commonly referenced as the sparse parameter count), which grows with n, and the number of parameters used for processing an individual token (called the active parameter count), which grows with K up to n.

The Mixtral paper says the memory costs for serving Mixtral are proportional to its sparse parameter count, 47B. Quote: "The memory costs for serving Mixtral are proportional to its sparse parameter count, 47B"

The memory costs for serving Mixtral are proportional to its sparse parameter count, 47B, which is still smaller than Llama 2 70B.

The Mixtral paper reports that it did not observe obvious patterns in the assignment of experts based on topic. Quote: "Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic."

Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic.

Mistral says Mixtral has 46.7B total parameters but only uses 12.9B parameters per token, and so processes input and generates output at the same speed and for the same cost as a 12.9B model. Quote: "Mixtral has 46.7B total parameters but only uses 12.9B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12.9B model."

Mistral describes Mistral Large 4 as a 1 trillion-parameter natively multimodal model with 49 billion active parameters. Quote: "ML4 is a 1 trillion-parameter natively multimodal model with 49 billion active parameters."

ML4 is a 1 trillion-parameter natively multimodal model with 49 billion active parameters.

Mistral lists Mistral Large 4 as an open-weight hybrid instruct-and-reasoning MoE. Quote: "Open-weight hybrid instruct-and-reasoning MoE with multimodal input"

Open-weight hybrid instruct-and-reasoning MoE with multimodal input

Mistral says that, as it works toward releasing the weights, it will share further details on the model architecture. Quote: "As we work toward releasing the weights, we will share further details on the model architecture"

As we work toward releasing the weights, we will share further details on the model architecture, additional benchmarks, and our post-training methodology.