Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

Problem

Framing

Prior multimodal autoregressive models cover at most two modalities or specialize to narrow task families. Unified-IO 2 closes this gap with one encoder–decoder model that ingests and generates text, images, audio, and actions through a shared discrete-token interface.

Currently Used Methods

Foundational

• @wuNExTGPT2023 — multimodal instruction-tuned generator with text, image, and audio outputs.
  • Limitation in context: lacks a unified four-modality pretraining and action interface.
• @alayracFlamingo2022 — few-shot vision-language model with strong interleaved image-text understanding.
  • Limitation in context: understanding-heavy design, not a unified multimodal generator.
• @liBLIP2_2023 — bridges frozen vision encoders to language models efficiently.
  • Limitation in context: does not natively generate images, audio, or actions.
• @rameshDALLE2021 — autoregressive text-to-image generation with discrete visual tokens.
  • Limitation in context: image generation only, without cross-modal generality.
• @raffelT5_2020 — text-to-text encoder–decoder scaling recipe for multitask transfer.
  • Limitation in context: text-only abstraction misses multimodal tokenization and decoding.

Proposed Method

Architecture

Text passes through BPE embeddings. Images use a ViT encoder, audio uses an AST encoder, and image/audio history use Perceiver resamplers before concatenation into one encoder sequence. A shared encoder–decoder transformer emits discrete tokens, then modality-specific decoders map them to text, image, audio, or action outputs.

Loss / Objective

Training uses standard autoregressive next-token prediction over the target token sequence.

Algorithm

Inference autoregressively decodes modality-specific discrete tokens conditioned on the packed multimodal input.

Training Procedure

• Batch size: 512
• Model sizes: 1.1B, 3.2B, 6.8B parameters
• Optimizer: Adam-style with $\beta_1 = 0.9$
• Second moment: $\beta_2 = 1.0 - k^{-0.8}$
• Packing: dynamic packing for multimodal sequences
• Image input pretraining size: $384 \times 384$
• Audio sample rate: $16000$ Hz

Evaluation

Datasets

• Vision-language: VQAv2, OKVQA, ScienceQA, Tally-QA, RefCOCO, RefCOCO+, RefCOCO-g, COCO Caption, POPE, SEED, MMB
• Video/audio: Kinetics-400, MSR-VTT, MSVD-QA, SEED-T, VGG-Sound, AudioCaps
• Action: robotic manipulation benchmarks including VIMA-style tasks

Metrics

• Accuracy / Exact Match for QA and classification
• CIDEr for captioning
• Task success or normalized scores for action tasks
• FID and TIFA for generative image evaluation

Headline results

• VQAv2: UIO-2XXL 79.4, UIO-2XL 78.1, UIO-2L 75.3
• OKVQA: UIO-2XXL 55.5, UIO-2XL 53.7, UIO-2L 50.2
• ScienceQA: UIO-2XL 88.8, UIO-2XXL 88.7
• RefCOCO-g: UIO-2XXL 86.6, UIO-2XL 84.0, UIO-2L 79.0
• MMB: UIO-2XXL 71.5, UIO-2XL 68.1, UIO-2L 62.1

Table 4: Vision-language generalist comparison across question answering, referring expression, captioning, hallucination, and multimodal benchmarks.

Ablations

• Modality mixture: balanced mixtures reduce loss and stabilize gradient norms.
• Scale: larger models improve next-token accuracy across modalities.
• Packing: dynamic packing supports efficient large-batch multimodal pretraining.
• Image modeling paradigm: the paper compares autoregressive, masked, and diffusion-style choices.

Method Strengths and Weaknesses

Strengths

• One token interface spans text, image, audio, and action generation.
• Scale helps consistently: 6.8B beats 1.1B on most reported multimodal benchmarks.
• Strong breadth: one model handles QA, captioning, grounding, audio, and robotics.
• Shared architecture reuses standard encoder–decoder machinery across modalities.

Weaknesses

• Still trails single-task specialists by large margins on several benchmarks.
• Autoregressive decoding over visual and audio tokens is computationally expensive.
• Performance is uneven across tasks; COCO captioning drops from XL to XXL.
• System complexity is pushed into tokenizers, resamplers, and modality-specific decoders.

Suggestions from the authors

• Improve modality balancing during large-scale multimodal pretraining.
• Expand instruction tuning coverage for harder cross-modal reasoning tasks.
• Strengthen action modeling and embodied control beyond current benchmark settings.
• Refine image and audio generation tokenizers for higher-fidelity outputs.

Links

Prior Papers

• @wuNExTGPT2023 — Another unified multimodal generative model spanning text, image, and audio, serving as a direct antecedent.
• @alayracFlamingo2022 — A strong multimodal generalist baseline for interleaved visual-language understanding.
• @liBLIP2_2023 — Relevant as an efficient vision-language connector design that contrasts with full end-to-end unification.
• @rameshDALLE2021 — Important precursor for autoregressive generation over discrete image tokens.
• @raffelT5_2020 — Encoder–decoder transfer and scaling recipe underlying the model family design.

Further Papers

• @chameleonTeam2024 — Another 2024 large multimodal autoregressive model with closely overlapping unification goals.