Chameleon: Mixed-Modal Early-Fusion Foundation Models

Problem

Framing

Late-fusion multimodal models still split text and images across separate encoders or decoders, which weakens arbitrary interleaved generation and cross-modal reasoning. Chameleon closes this by tokenizing images into the same autoregressive stream as text and training one transformer end-to-end on mixed-modal data.

Currently Used Methods

Foundational

• @alayracFlamingo2022 — frozen language model with visual resampling for few-shot vision-language tasks.
  • Limitation in context: late fusion blocks full mixed-modal token generation.
• @rameshDALLE2021 — autoregressive image generation from discrete visual tokens.
  • Limitation in context: not a unified image-text reasoning model.
• @luUnifiedIO2_2024 — unified seq2seq modeling across many perception and language tasks.
  • Limitation in context: still relies on modality-specific I/O design.
• @wuNExTGPT2023 — multimodal LLM with external encoders and decoders.
  • Limitation in context: separate modules weaken native early fusion.
• @brownGPT3_2020 — large autoregressive transformer for pure text next-token prediction.
  • Limitation in context: no native image token stream.

Proposed Method

Architecture

Chameleon maps text and images into one discrete token stream, then runs a LLaMA-2-style autoregressive transformer over the joint sequence. Images are quantized into codebook tokens; the tokenizer uses a 65,536-word BPE vocabulary that includes 8,192 image tokens. Stability comes from QK-normalization and reordered residual norms.

Loss / Objective

Pre-training uses standard autoregressive next-token prediction over mixed text-image sequences.

SFT masks prompt tokens and optimizes only answer tokens.

Algorithm

The main architectural update reorders normalization around the residual branches:

Training Procedure

• Vocabulary size: 65,536, including 8,192 image codebook tokens.
• Optimizer: AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-5}$.
• Warmup: 4,000 steps.
• LR schedule: exponential decay to $0$.
• Weight decay: $0.1$.
• Gradient clipping: $1.0$.
• Dropout: $0.1$ for 7B; none for 34B pre-training.
• Batch size: 7B uses $2^{23}$ tokens; 34B uses $3 \times 2^{22}$ tokens.
• Total training: 2.1 epochs, 9.2T tokens seen.

Evaluation

Datasets

• Text: PIQA, SIQA, HellaSwag, WinoGrande, ARC-E, ARC-C, OBQA, MMLU, GSM8K, MATH.
• Image-to-text: COCO, Flickr30k, VQAv2.
• Human evaluation: mixed-modality and text-only prompt sets against Gemini and GPT-4V variants.

Metrics

• Text benchmarks: accuracy or exact match.
• Captioning: CIDEr.
• VQA: accuracy.
• Human evaluation: task-fulfillment and pairwise win rate.

Headline results

• COCO captioning: Chameleon-MultiTask 139.1 CIDEr.
• Flickr30k captioning: Chameleon-MultiTask 76.2.
• VQAv2: Chameleon-MultiTask 69.6.
• MMLU: Chameleon-34B 70.6.
• PIQA: Chameleon-34B 83.6.

Ablations

• QK-normalization: removing it makes 7B diverge after about 20% of an epoch.
• Dropout: needed for 7B stability, but does not fix 34B divergence.
• Norm reordering: stabilizes 34B where LLaMA-2 ordering diverges.
• Image generation objective: disabling it avoids one instability mode in 34B.

Results figure

Table 4: Training-loss ablation for norm reordering.

Method Strengths and Weaknesses

Strengths

• One token stream supports arbitrary text-image interleaving.
• Strong image-to-text results: 139.1 COCO CIDEr after multitask SFT.
• Text ability stays competitive: 70.6 MMLU at 34B.
• Stability recipe is concrete: QK-norm plus norm reordering.

Weaknesses

• Training stability is fragile without bespoke normalization changes.
• Image tokenization adds a learned bottleneck before generation.
• VQAv2 score 69.6 trails the strongest specialist systems.
• 34B training cost is high: 9.2T tokens seen.

Suggestions from the authors

• Improve identification and reasoning failure modes in human evaluation.
• Push alignment methods tailored to mixed-modal generation.
• Extend efficient inference for interleaved autoregressive decoding.
• Study scaling laws for early-fusion mixed-modal training.

Links

Prior Papers

• @luUnifiedIO2_2024 — another unified multimodal model, useful for contrasting early fusion against broader task-unification designs.
• @wuNExTGPT2023 — representative modular multimodal LLM that Chameleon departs from with end-to-end early fusion.
• @alayracFlamingo2022 — strong late-fusion baseline for image-conditioned text generation.
• @rameshDALLE2021 — antecedent for discrete visual-token autoregressive generation.
• @brownGPT3_2020 — text-only autoregressive transformer baseline that Chameleon extends to mixed-modal tokens.

Further Papers

• @luUnifiedIO2_2024 — closely related 2024 multimodal foundation model for comparison on unified modeling choices.
• @jiangMixtral2024 — useful text-only contemporary baseline because Chameleon claims competitive unimodal language performance.