GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Problem

Framing

Accurate post-training quantization did not scale to 100B+ transformers; greedy second-order methods were too slow, and cheap rounding collapsed at 3 bits. GPTQ closes this gap with one-shot, row-wise Hessian-aware quantization that keeps OPT-175B near full precision: WikiText2 perplexity 8.37 at 4-bit versus 8.34 FP16.

Currently Used Methods

Direct antecedents

• "Post-Training Quantization for Vision Transformer" — adaptive rounding for accurate PTQ on vision backbones.
  • Limitation in context: optimization-heavy PTQ is too slow for 100B-scale transformers.
• "BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction" — block reconstruction with second-order corrections.
  • Limitation in context: designed for vision models, not full GPT-family quantization.
• "Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning" — second-order greedy weight updates via inverse Hessian.
  • Limitation in context: cubic input-dimension cost makes billion-parameter deployment impractical.
• "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" — 8-bit inference for large transformers.
  • Limitation in context: lower compression and no 3-bit or 4-bit regime.

Proposed Method

Architecture

GPTQ is not a new transformer. It quantizes each linear layer row-wise, using a shared input Hessian across rows and processing columns in fixed-order blocks. The key systems change is a Cholesky-form inverse Hessian update plus batched column updates.

Loss / Objective

Each layer solves a reconstruction problem on calibration activations:

Algorithm

For the current unquantized set $F$, GPTQ quantizes one weight and applies the optimal second-order compensation to the remaining weights:

The inverse Hessian is updated after each quantized coordinate by:

Training Procedure

• Post-training only; no weight retraining.
• Calibration data: a few thousand samples.
• Context length for perplexity evaluation: 2048 tokens.
• Target precisions: 3-bit and 4-bit.
• Grouped variants evaluated: $g1024$, $g128$.
• Generation latency benchmark: batch size 1, output length 128.

Evaluation

Datasets

• WikiText2
• Penn Treebank
• C4
• LAMBADA
• ARC-easy
• ARC-challenge
• PIQA
• StoryCloze

Metrics

• Perplexity
• Accuracy
• Average per-token latency
• Quantization runtime
• GPU count reduction

Headline results

• OPT-175B, WikiText2, 4-bit: PPL 8.37; FP16 8.34; RTN 10.54.
• BLOOM-176B, WikiText2, 4-bit: PPL 8.21; FP16 8.11; RTN 8.37.
• OPT-175B, LAMBADA, 4-bit: 76.80%; FP16 75.59%; RTN 71.34%.
• OPT-175B, 3-bit on A100-80GB: 71 ms/token, 3.24$\times$ speedup, 5$\to$1 GPUs.
• A6000-48GB, OPT-175B, 3-bit: 130 ms/token, 4.53$\times$ speedup, 8$\to$2 GPUs.

Table 2: GPTQ runtime for full quantization of the 4 largest OPT and BLOOM models.

Ablations

• Bit-width: 4-bit stays near FP16; 3-bit remains usable for GPTQ but collapses for RTN.
• Model size: larger models are usually easier to quantize.
• Group size: $g1024$ improves 3-bit perplexity by about 0.2 on average.
• Finer grouping: $g128$ adds another roughly 0.1 perplexity gain.

Method Strengths and Weaknesses

Strengths

• Preserves OPT-175B WikiText2 perplexity at 4-bit: 8.37 versus 8.34 FP16.
• 3-bit remains functional where RTN collapses on perplexity and zero-shot tasks.
• Full 175B-model quantization finishes in 4.2 hours for OPT.
• Enables real latency gains, not just memory savings.

Weaknesses

• Depends on second-order Hessian approximations from calibration activations.
• C4 calibration samples come from the C4 training set.
• Kernel speedups target low-batch generative inference, mainly matrix-vector workloads.
• 3-bit still degrades smaller and medium models substantially.

Suggestions from the authors

• Extend kernels beyond low-batch generative inference.
• Combine GPTQ with improved distribution strategies and faster attention kernels.
• Explore alternative quantization grids and grouping schemes.
• Study even lower-bit settings with extra stabilization tricks.

Links

Prior Papers

• @dettmersQLoRA2023 — follows the same compression agenda for LLM deployment, but adds low-rank finetuning after quantization.

Further Papers

• @dettmersQLoRA2023 — builds directly on 4-bit LLM quantization as a practical foundation for parameter-efficient finetuning.
• @kwonvLLM2023 — extends the low-bit LLM inference line with system-level optimizations for serving.