ReAct: Synergizing Reasoning and Acting in Language Models

Problem

Framing

Chain-of-thought improves reasoning, but it cannot query the world, and action-only prompting lacks explicit planning. ReAct closes this gap by interleaving free-form thoughts with text actions and observations, improving grounded multi-hop QA, fact verification, and interactive decision making.

Currently Used Methods

Foundational

• @weiCoT2022 — chain-of-thought prompting elicits explicit intermediate reasoning.
  • Limitation in context: no external actions, so factual errors can persist uncorrected.
• @kojimaZeroShotCoT2022 — zero-shot reasoning via simple prompting cues.
  • Limitation in context: reasoning remains detached from environment feedback.
• @schickToolformer2023 — language models learn tool use through API-call supervision.
  • Limitation in context: tool use is not framed as interleaved reasoning trajectories.
• @brownGPT3_2020 — large language models support few-shot in-context task adaptation.
  • Limitation in context: prompting alone does not coordinate planning with action execution.
• @yaoToT2023 — tree search expands multiple reasoning paths before committing.
  • Limitation in context: search over thoughts still omits external environment interaction.

Proposed Method

Architecture

ReAct uses a single LM as a policy over an alternating trajectory of thought, action, and observation tokens. Thoughts decompose goals and plan retrieval; actions query an external environment; observations feed back into the next reasoning step.

Algorithm

At step $t$, the model conditions on the full interaction history and emits either a reasoning trace or an action:

The environment then returns a new observation after each action:

Training Procedure

• Prompting: PaLM-540B for main results.
• CoT-SC baseline: 21 samples, decoding temperature $0.7$.
• ReAct $\rightarrow$ CoT-SC fallback: 7 steps on HotpotQA, 5 on FEVER.
• Finetuning data: 3,000 correct ReAct-generated trajectories.
• Finetuning models: PaLM-8B and PaLM-62B.
• Batch size: 64.
• Finetuning steps: 4,000 for ReAct/Act; 2,000 or 1,000 for Standard/CoT.

Evaluation

Datasets

• HotpotQA: multi-hop question answering over Wikipedia.
• FEVER: fact verification with Wikipedia evidence.
• ALFWorld: text-based household decision making.
• WebShop: online shopping in a noisy web environment.

Metrics

• HotpotQA: exact match.
• FEVER: accuracy.
• ALFWorld: task success rate.
• WebShop: score and success rate.

Headline results

• HotpotQA (PaLM-540B, prompting): ReAct $27.4$ EM; CoT $29.4$; Act $25.7$.
• FEVER (PaLM-540B, prompting): ReAct $60.9$ Acc; CoT $56.3$; Act $58.9$.
• HotpotQA (best hybrid): ReAct $\rightarrow$ CoT-SC $35.1$ EM.
• FEVER (best hybrid): CoT-SC $\rightarrow$ ReAct $64.6$ Acc.
• ALFWorld: best ReAct $71\%$ success; best Act $45\%$; BUTLER $37\%$.
• WebShop: ReAct $66.6$ score, $40.0\%$ SR; Act $62.3$, $30.1\%$ SR.

Ablations

• Reasoning + acting vs acting only: ReAct beats Act on all four tasks.
• ReAct vs CoT: ReAct is more grounded, but slightly worse on HotpotQA EM.
• Hybrid fallback with CoT-SC: both switch rules beat plain CoT-SC with fewer samples.
• Finetuning vs prompting: ReAct is weakest in small-model prompting, strongest after 3k-example finetuning.

Method Strengths and Weaknesses

Strengths

• Grounds reasoning in retrieved observations, reducing HotpotQA hallucination failures versus CoT.
• Improves FEVER accuracy over CoT: $60.9$ vs $56.3$.
• Large gains in decision making: ALFWorld $71\%$ vs Act $45\%$.
• Finetunes well: 3k trajectories make ReAct strongest on PaLM-8B and 62B.

Weaknesses

• Slightly trails CoT on HotpotQA prompting: $27.4$ vs $29.4$ EM.
• Interleaving constraints raise reasoning-error rate on HotpotQA failure analysis.
• Search failures derail performance; non-informative retrieval forms $23\%$ of ReAct errors.
• Long-horizon prompting needs many demonstrations and hits context-length limits.

Suggestions from the authors

• Collect more high-quality human-written reasoning and action annotations.
• Improve long-context learning for tasks needing many demonstrations.
• Scale ReAct with multi-task training across domains.
• Combine ReAct with reinforcement learning and human feedback.

Links

Prior Papers

• @weiCoT2022 — ReAct extends chain-of-thought by inserting environment actions and observations into the reasoning loop.
• @kojimaZeroShotCoT2022 — zero-shot CoT is a direct reasoning-only baseline that clarifies what acting adds.
• @brownGPT3_2020 — the paper relies on large few-shot language models as the substrate for prompted trajectories.

Further Papers

• @yaoToT2023 — extends LM reasoning beyond single trajectories by searching over multiple thought branches.
• @schickToolformer2023 — studies explicit tool-use training, a close descendant of LM acting with external interfaces.