Toolformer: Language Models Can Teach Themselves to Use Tools

Problem

Framing

Large LMs still hallucinate facts, miss recent information, and fail at arithmetic or time-sensitive queries. The paper closes this gap by teaching the LM to insert API calls into its own text stream, then keeping only calls that lower next-token loss. A 6.7B GPT-J variant then beats larger models on several zero-shot tasks, including 40.4 on ASDiv.

Currently Used Methods

Direct antecedents

• @chenCodex2021 — code-specialized LM with in-context execution and synthesis abilities.
  • Limitation in context: no self-supervised policy for deciding when external tools help.
• @lewisRAG2020 — retrieval-augmented generation injects documents into generation.
  • Limitation in context: retrieval is fixed, not model-chosen API use during free-form decoding.
• @yaoReAct2022 — interleaves reasoning traces with actions over external environments.
  • Limitation in context: relies on prompting trajectories, not self-labeled training data.
• "GPT-3: Language Models are Few-Shot Learners" — large in-context learner with strong zero-shot transfer.
  • Limitation in context: scale alone does not fix arithmetic, recency, or temporal grounding.
• "LaMDA: Language Models for Dialog Applications" — tool-augmented dialogue via human-designed demonstrations.
  • Limitation in context: needs substantial annotation and task-specific tool supervision.

Proposed Method

Architecture

Toolformer keeps the base LM architecture fixed and augments training text with inline tool calls of the form $e(c)=\texttt{<API>}\ a_c(i_c)\ \texttt{</API>}$ or $e(c,r)=\texttt{<API>}\ a_c(i_c)\ \to r\ \texttt{</API>}$. The tool set spans question answering, Wikipedia search, calculator, machine translation, and calendar APIs. Inference runs normal decoding until the model emits $\to$, then executes the chosen API and resumes generation.

Loss / Objective

Candidate calls survive only if their returned result improves prediction of future tokens.

Sampling Rule / Algorithm

The LM first proposes where a tool call should start, then samples call strings at those positions.

For each $i\in I$, it samples up to $m$ calls and keeps at most top-$k$ positions.

Training Procedure

• Base LM: GPT-J, 6.7B parameters.
• Data: subset of CCNet, then augmented to $C^{\ast}$.
• Tools: QA, WikiSearch, Calculator, Calendar, MT.
• Weighting: $\tilde{w}_t=\max(0,1-0.2t)$, normalized to $w_t$.
• Finetuning examples: up to 25k per API.
• Max sequence length: 1,024.
• Effective batch size: 128.
• Learning rate: $1\times 10^{-5}$.
• Warmup: first 10% of training.
• Decoding for tool use: top-$k=10$ trigger for $\texttt{<API>}$.
• Inference restriction: at most one API call per input.

Evaluation

Datasets

• LAMA: SQuAD, Google-RE, T-REx.
• Math: ASDiv, SVAMP, MAWPS.
• QA: WebQS, Natural Questions, TriviaQA.
• Multilingual QA: MLQA.
• Temporal: T EMP LAMA, DATESET.
• LM retention: WikiText, held-out CCNet.

Metrics

• LAMA: correct answer appears in first 5 generated words.
• Math: first predicted number matches target.
• QA: correct answer appears in first 20 generated words.
• MLQA: answer appears in first 10 generated words.
• Language modeling: perplexity.

Headline results

• LAMA SQuAD: 33.8 for Toolformer vs 26.8 for GPT-3 and 21.6 for OPT-66B.
• LAMA T-REx: 53.5 vs 39.8 for GPT-3.
• ASDiv: 40.4 vs 14.0 for GPT-3.
• SVAMP: 29.4 vs 10.0 for GPT-3.
• DATESET: 27.3 vs 0.8 for GPT-3.

Results table

Table 2: Number of examples with API calls in $C^{\ast}$ for different filtering thresholds $\tau_f$

Ablations

• API-disabled decoding still improves math, showing tool-augmented finetuning teaches latent skills.
• Calendar helps DATESET, but barely triggers on T EMP LAMA; one-call inference is too restrictive.
• Tool use emerges near 775M parameters; smaller GPT-2 variants gain little.
• Finetuning on API-augmented text preserves perplexity: 10.5 on CCNet with or without disabled tool calls.

Method Strengths and Weaknesses

Strengths

• Self-supervised filtering turns tool use into a measurable likelihood gain.
• Beats GPT-3 on LAMA T-REx: 53.5 vs 39.8.
• Beats GPT-3 on arithmetic tasks by large margins.
• Preserves language-model perplexity after API-call finetuning.

Weaknesses

• One API call per example blocks multi-step tool compositions.
• Search tool is shallow; no iterative reformulation or browsing.
• Benefits emerge late; sub-775M models barely exploit tools.
• CCNet finetuning hurts some multilingual settings, limiting MLQA gains.

Suggestions from the authors

• Add interactive search with query reformulation and multi-result browsing.
• Learn multi-step tool chains instead of independent single calls.
• Study stronger tool use in smaller models.
• Improve multilingual finetuning to avoid distribution-shift regressions.

Links

Prior Papers

• @chenCodex2021 — relevant as a strong code-capable LM baseline for external execution and prompting.
• @lewisRAG2020 — relevant as the canonical retrieval-augmented LM that injects external knowledge.
• @yaoReAct2022 — relevant as an action-and-reasoning framework for tool-interacting language agents.

Further Papers

• @kwonvLLM2023 — extends the systems question of how LLMs use external memory and serving-time resources.
• @luUnifiedIO2_2024 — pushes toward a broader multi-tool, multimodal model interface beyond text-only APIs.