NExT-GPT: Any-to-Any Multimodal LLM

Problem

Framing

Multimodal LLMs mostly stop at understanding or call external generators in a brittle cascade. NExT-GPT closes this gap by aligning one LLM to both multimodal encoders and diffusion decoders, enabling any-to-any input and output with only about 1% trainable parameters.

Currently Used Methods

Direct antecedents

• @brownGPT3_2020 — large language modeling establishes the central autoregressive reasoning backbone.
  • Limitation in context: text-only pretraining cannot ingest or emit image, audio, or video.
• @alayracFlamingo2022 — cross-attention adapters connect frozen visual encoders to frozen LLMs.
  • Limitation in context: focuses on perception; it does not natively generate multimodal outputs.
• @liBLIP2_2023 — Q-Former aligns frozen vision encoders with frozen LLMs efficiently.
  • Limitation in context: one-way image-to-text bridging, not unified any-to-any generation.
• @radfordCLIP2021 — contrastive image-text alignment supplies transferable visual semantics.
  • Limitation in context: retrieval-style embeddings are not an instruction-following multimodal interface.
• @rombachLatentDiffusion2022 — latent diffusion gives strong controllable image synthesis backbones.
  • Limitation in context: standalone generators lack unified cross-modal reasoning and dialogue control.

Proposed Method

Architecture

NExT-GPT connects modality encoders, lightweight projection layers, a Vicuna-7B LLM, modality-specific output projectors, and frozen diffusion decoders. Image, audio, and video inputs are encoded by ImageBind; outputs are generated by Stable Diffusion, AudioLDM, and Zeroscope. Only the grouping/projection modules and LoRA adapters are updated.

Loss / Objective

Training uses cross-entropy captioning and instruction-following losses across three stages.

Sampling Rule / Algorithm

Inference maps LLM output representations into modality-specific signal tokens, then invokes the matched diffusion decoder.

Training Procedure

• Three stages: encoding-side alignment, decoding-side alignment, end-to-end instruction tuning.
• Optimizer: Adam.
• Trainable modules: input projections, output projections, LoRA adapters.
• Frozen modules: ImageBind encoder, diffusion U-Nets, most LLM weights.
• LLM: Vicuna-7B with 33M LoRA parameters.
• Input projection: grouping block, 28M parameters.
• Output projections: 31M for image, 31M for audio, 32M for video.

Evaluation

Datasets

• COCO, NoCaps, Flickr30K.
• VQAv2, VizWiz, OKVQA.
• MSRVTT, MSVD-QA, MSRVTT-QA, NExT-QA.
• AudioCaps.
• MMBench, SEED-Bench.

Metrics

• CIDEr, BLEU-4, METEOR.
• Accuracy.
• FID, CLIPSim.
• FD, IS.
• Human evaluation scale.

Headline results

• NoCaps zero-shot captioning: CIDEr 123.7.
• Flickr30K zero-shot captioning: CIDEr 84.5.
• COCO zero-shot captioning: CIDEr 124.9.
• VQAv2 zero-shot QA: 66.7.
• VizWiz / OKVQA zero-shot QA: 48.4 / 52.1.
• MMBench / SEED-Bench: 58.0 / 57.5.

Table 4: Zero-shot multimodal understanding results across image captioning, question answering, and evaluation-only benchmarks.

Ablations

• Training only caption-style alignment underuses instruction-following ability.
• Decoding-side alignment is required for multimodal generation.
• End-to-end instruction tuning improves complex multi-turn multimodal behavior.
• Grouping-based projection keeps adaptation lightweight at about 1% updated parameters.

Method Strengths and Weaknesses

Strengths

• One LLM handles multimodal understanding and generation in a unified interface.
• Lightweight tuning updates about 1% of parameters.
• Covers image, audio, and video on both input and output sides.
• Strong zero-shot captioning and QA scores across several benchmarks.

Weaknesses

• Generation still depends on separate frozen diffusion backbones.
• Modality coverage excludes tighter token-level joint generation inside the LLM.
• Training requires a staged pipeline, not a single simple objective.
• Video and audio generation quality trails specialized standalone generators.

Suggestions from the authors

• Build richer multimodal instruction-tuning datasets with more modality combinations.
• Reduce error accumulation between LLM planning and downstream generators.
• Improve end-to-end updating across understanding and generation modules.
• Extend the system to more modalities and more complex interactions.

Links

Prior Papers

• @brownGPT3_2020 — establishes the LLM backbone paradigm that NExT-GPT turns into a multimodal hub.
• @alayracFlamingo2022 — an early frozen-LLM multimodal adapter design that NExT-GPT extends toward generation.
• @liBLIP2_2023 — lightweight encoder-to-LLM alignment is a direct precursor to NExT-GPT's projection strategy.
• @radfordCLIP2021 — supplies the aligned visual-semantic representation tradition behind multimodal bridging.
• @rombachLatentDiffusion2022 — provides the diffusion-decoder style that NExT-GPT plugs into its output side.

Further Papers

• @chameleonTeam2024 — pursues tighter multimodal generation with a unified model instead of external modality decoders.
• @luUnifiedIO2_2024 — extends the any-to-any multimodal ambition with a broader unified I/O formulation.