Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.

2021 · arXiv

Evaluating Large Language Models Trained on Code

Problem

Framing

Code LMs were scored with token overlap, not executable correctness, so benchmark gains did not imply working programs. The paper closes this with HumanEval and a pass@ $k$ estimator, then shows code-specialized GPT reaches 28.8% pass@1 and 77.5% pass@100 with reranking on HumanEval.

Currently Used Methods

Foundational

@brownGPT3_2020 — large autoregressive language modeling at scale.
- Limitation in context: near-zero HumanEval pass@1 at 12B after text-only pretraining.
@kaplanScalingLaws2020 — power-law scaling laws for language-model loss.
- Limitation in context: loss scaling alone does not measure executable program correctness.
"Measuring Coding Challenge Competence With APPS" — benchmarked program synthesis with hidden tests.
- Limitation in context: contest-style tasks mix specification parsing with coding ability.
"GPT-Neo" — open GPT-style models used for code generation comparison.
- Limitation in context: lower HumanEval pass rates than similarly sized Codex models.
"GPT-J-6B" — larger open autoregressive baseline for code completion.
- Limitation in context: still trails code-finetuned Codex on functional correctness.

Proposed Method

Architecture

Codex keeps the GPT-3 decoder-only Transformer and changes the data mix: large-scale Python pretraining plus code-focused fine-tuning. The main variants span roughly 85M to 12B non-embedding parameters, with an extra Codex-S stage tuned on correctly implemented standalone functions.

Results plot on page 2: HumanEval pass rate versus non-embedding parameters for GPT-3, Codex, and Codex-S, including reranking curves.

Loss / Objective

Training remains next-token maximum likelihood on code sequences.

\mathcal{L}(\theta) = - \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})

Held-out code loss after fine-tuning follows an empirical scaling law:

\mathcal{L}_{\mathrm{test}}(N) \approx \left(\frac{N}{5.92 \times 10^{7}}\right)^{-0.13}

Sampling Rule

The evaluation target is functional success under $k$ samples, estimated from $n$ draws with $c$ correct programs.

\mathrm{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}

Training Procedure

Base model family: 85M to 12B non-embedding parameters.
Domain adaptation: large-scale Python GitHub corpus.
Codex-S: further supervised fine-tuning on standalone Python functions.
Sampling temperatures are tuned per pass@ $k$ evaluation.
APPS results use temperature $0.6$ to cover multiple $k$ values.

Evaluation

Datasets

HumanEval: 164 hand-written Python programming problems with unit tests.
APPS: coding-challenge benchmark with hidden tests.
Docstring generation set: human-graded generation tasks.

Metrics

pass@1, pass@10, pass@100.
BLEU for reference-similarity analysis.
Held-out code cross-entropy loss.

Headline results

HumanEval, Codex-12B: pass@1 28.8%.
HumanEval, Codex-S: pass@1 37.7%.
HumanEval, Codex-S mean-logp reranking: 44.5% solved.
HumanEval, Codex-S oracle over 100 samples: 77.5% solved.
APPS: Codex improves over GPT-Neo and GPT-J baselines across pass@ $k$ .

Ablations

Model size: pass rate rises smoothly with non-embedding parameters.
Additional standalone-function tuning: Codex-S beats plain Codex at fixed size.
Sample count: more draws sharply raise solvable-problem coverage.
Reranking heuristic: mean log-probability beats random and back-translation; BLEU separates correct from wrong solutions poorly.

Result plot: pass rate versus number of generated samples under different reranking heuristics, with oracle, mean-logp, back-translation, and random selection.

Method Strengths and Weaknesses

Strengths

Replaces proxy string metrics with executable pass@ $k$ on hidden tests.
HumanEval is hand-written to reduce simple memorization of benchmark items.
Code fine-tuning yields a large jump over GPT-3 at equal scale.
Multi-sample decoding exposes much higher latent competence than pass@1 alone.

Weaknesses

Training objective stays token-level likelihood, not direct program correctness.
Reported gains depend strongly on sampling budget and reranking.
BLEU fails to track correctness, so common text metrics remain misleading.
Sandbox execution and hidden tests cover only observable behavior, not full software quality.

Suggestions from the authors

Improve automatic evaluation beyond hidden tests and overlap-based metrics.
Study alignment for code models under more capable real-world deployment.
Reduce harmful memorization, insecure suggestions, and adversarially triggered code.
Develop better ways to select correct samples without executing every candidate.

Evaluating Large Language Models Trained on Code

Evaluating Large Language Models Trained on Code

Problem

Framing

Currently Used Methods

Foundational

Proposed Method

Architecture

Loss / Objective

Sampling Rule

Training Procedure

Evaluation

Datasets

Metrics

Headline results

Ablations

Method Strengths and Weaknesses

Strengths

Weaknesses

Suggestions from the authors

Links

Prior Papers

Further Papers