Mixtral of Experts

Problem

Framing

Dense decoder LMs trade quality for inference cost: matching 70B-class performance usually requires activating 70B-class parameters on every token. Mixtral closes this gap with top-2 sparse MoE feedforward blocks, giving each token access to 47B parameters while activating only 13B, and still reaching 70.6% MMLU plus 58.4% GSM8K.

Currently Used Methods

Foundational

• @vaswaniAttentionAllNeed2017 — decoder transformer backbone for autoregressive language modeling.
  • Limitation in context: dense FFNs activate all parameters for every token.
• @fedusSwitchTransformer2021 — sparse expert routing to raise parameter count cheaply.
  • Limitation in context: top-1 routing drops expert diversity per token.
• @brownGPT3_2020 — large dense scaling improves few-shot language performance.
  • Limitation in context: serving cost scales with total parameter count.
• @hoffmannChinchilla2022 — compute-optimal dense scaling laws for transformers.
  • Limitation in context: scaling remains dense, so inference stays expensive.
• @chameleonTeam2024 — competitive open multimodal MoE-style scaling in a large foundation model.
  • Limitation in context: not a compact open text-only 8x7B serving point.

Proposed Method

Architecture

Mixtral keeps the Mistral decoder stack and replaces every FFN with an 8-expert MoE layer. The model uses 32 layers, $d=4096$, hidden size $14336$, 32 attention heads, 8 KV heads, and context length $32768$. Each token is routed to 2 experts, so compute stays near a 13B dense model while total parameters rise to 47B.

Loss / Objective

The MoE block replaces the dense FFN with sparse gated expert aggregation:

Sampling Rule / Algorithm

Autoregressive decoding uses the standard next-token factorization over the sparse decoder:

Training Procedure

• Context length: $32768$
• Layers: $32$
• Model dimension: $4096$
• Hidden dimension per expert: $14336$
• Attention heads: $32$
• KV heads: $8$
• Experts per layer: $8$
• Active experts per token: $2$

Evaluation

Datasets

• Reasoning: MMLU, HellaSwag, ARC Challenge, Winogrande, PIQA, SIQA, OpenBookQA, CommonsenseQA
• World knowledge: NaturalQuestions, TriviaQA
• Reading comprehension: BoolQ, QuAC
• Math: GSM8K, MATH
• Code: HumanEval, MBPP
• Multilingual: French, German, Spanish, Italian variants of ARC-c, HellaSwag, MMLU
• Long context: passkey retrieval, proof-pile perplexity
• Instruction following: MT-Bench, LMSys human eval
• Bias: BBQ, BOLD

Metrics

• Accuracy for classification and QA benchmarks
• pass@1 for HumanEval and MBPP
• maj@8 for GSM8K
• maj@4 for MATH
• Retrieval accuracy for passkey task
• Perplexity on proof-pile
• MT-Bench score for instruct models

Headline results

• MMLU: 70.6%
• ARC Challenge: 85.8%
• MBPP (pass@1): 60.7%
• GSM8K (maj@8): 58.4%
• MT-Bench, Instruct: 8.30
• Passkey retrieval: 100% accuracy across tested context lengths and positions

Table 1: Comparison of Mixtral with Llama 2 70B and GPT-3.5

Ablations

• Context position in passkey retrieval: accuracy stays at 100%.
• Context length in passkey retrieval: accuracy stays at 100%.
• Longer proof-pile context: perplexity decreases monotonically.
• Multilingual data upsampling: gains appear across French, German, Spanish, and Italian benchmarks.

Method Strengths and Weaknesses

Strengths

• 13B active parameters still beat Llama 2 70B on MMLU, ARC-c, MBPP, and GSM8K.
• Top-2 routing preserves expert diversity without dense 47B compute.
• 32k context is usable, with 100% passkey retrieval.
• Apache 2.0 release makes the model practically reusable.

Weaknesses

• WinoGrande trails Llama 2 70B: 81.2% versus 83.2%.
• HellaSwag also trails Llama 2 70B: 86.7% versus 87.1%.
• MT-Bench slightly trails GPT-3.5 for instruct tuning: 8.30 versus 8.32.
• Training recipe details are sparse, which limits reproducibility.

Suggestions from the authors

• Study expert specialization across domains and layers.
• Optimize expert-parallel serving under router locality and load imbalance.
• Leverage routing locality for caching during training and inference.
• Correct residual bias and sentiment issues with stronger preference tuning.

Links

Prior Papers

• @fedusSwitchTransformer2021 — establishes sparse expert routing, the direct architectural precedent for Mixtral's MoE FFN replacement.
• @vaswaniAttentionAllNeed2017 — provides the transformer decoder backbone that Mixtral keeps outside the FFN blocks.
• @brownGPT3_2020 — serves as a dense large-model baseline for capability versus inference-cost tradeoffs.
• @hoffmannChinchilla2022 — frames dense scaling efficiency, which Mixtral challenges with sparse activation.
• @chameleonTeam2024 — another vault MoE-related foundation model relevant to sparse scaling comparisons.

Further Papers

• @shaoGRPO2024 — later open LLM work on post-training and alignment that can build on Mixtral-class backbones.
• @luUnifiedIO2_2024 — contemporary multimodal scaling work that intersects with efficient large-model design.
• @chameleonTeam2024 — overlapping 2024 foundation-model scaling paper with adjacent MoE and deployment concerns.