What Outcome RL Buys: Finishing, Reliability, Rarely Reach

Yang, Shan

TL;DR

Outcome RL raises pass@1, and every pass@1 is scored under a token budget. We read 20 released RL models (1.5B to 32B, seven bases) against the thinking models they started from, at fixed budgets, and split each change in pass@1 into three parts that add up exactly: finishing, reliability and reach. Where the base often runs out of budget, most of the change is finishing (82% at 4k and 8k tokens). Where the base has enough tokens, the gain is reliability. Reach adds at most 2.1 percentage points on any of our four physics and math sets. The release rebuilds every table in the paper from row-level data with the paper's own scripts, without a GPU.

Findings

82%

of the change is finishing at 4k and 8k tokens

Short budgets: the gain is finishing

Where the base often runs out of budget, most of the change in pass@1 is the RL model answering before the budget ends. Given more tokens, the base closes most of that gap by itself.

62%

reliability share one budget above the cap

Enough tokens: the gain is reliability

Where the base has enough tokens, the RL model is graded right more often, even on problems both models finish.

≤ 2.1 pp

reach on any of four physics and math sets

RL rarely solves what its base cannot

By our definition, reach is small everywhere: on physics, MATH-500, a harder unfiltered physics set and competition math from 2025 and 2026.

3.5%

of 144 RL-only physics wins hold up

Most new solves do not survive a second look

Of 144 physics problems only the RL model solved, 3.5% hold up when resampled or read. In a random sample, nine of ten new RL steps a human confirmed also appear in the base's samples.

8B

small-scale controlled RL run

The same split in our own training

In a small-scale RL run on an 8B model, the gain on new problems is finishing, with reliability flat. Supplying each problem's physical setup was not shown to improve on plain RL.

One Change, Three Parts

Each change in pass@1 at a fixed token budget splits exactly into three parts.

Finishing

Answering before the budget runs out.

Reliability

Being right once finished.

Reach

Solving problems the base answered wrongly in all eight tries, minus the problems the base solves that the RL model answered wrongly in all eight.

The Split at Each Recipe's Own Cap

Every recipe also trains under a budget, so we read each one at its own training cap and at the next budget above it. Shares are each part's size over the three parts' total, pooled over the 20 recipes on physics; brackets are 95% intervals from resampling the seven bases.

Budget	Finishing	Reliability	Reach, net
At the training cap	45% [18, 59]	49% [38, 70]	5%
Next budget above the cap	28% [11, 37]	62% [56, 73]	10%

Per-recipe bars for all 20 recipes on physics, at the training cap and one budget above it, each split into finishing, reliability and reach, with side panels counting the problems only the base or only the RL model solves. — Each recipe at its own training cap and the next budget above it, on physics. Rows are sorted by how often the base finishes at the cap, least first. The side panels count problems that only the base (grey) or only the RL model (black) solves.

Released Artifacts

Everything needed to recompute the paper's measurements from row-level data, without a GPU.

Artifact	Contents	Where
Metrics	The exact three-part split, the mirror control, cap selection, per-cell problem-clustered intervals, crossed and base-resampled bootstraps, and the frozen answer matcher with the judge protocol that audits its misses.	`pairaudit/`
Test sets	300 physics problems with texts; ids, gold answers and text hashes for 671 harder unfiltered physics problems and 276 competition-math problems (2025 contests, plus AIME and HMMT February 2026), whose texts a builder script rebuilds from their sources and checks against each hash.	`sets/`
Rows	One line per sample (problem, sample index, finished, score, length) for every model and budget in the study, plus the 24-sample rounds, judged scores and judge verdicts.	`data/`
Paper tables	Every table as CSV, the scripts that wrote them, and tests that rerun those scripts on the rows and match the paper's numbers.	`paper/`

Quick start

git clone https://github.com/shanyang-me/outcome-rl-buys
cd outcome-rl-buys && pip install -e ".[dev]"
pytest -m "not slow"              # the split's identities and the first release's numbers (~20 s)
pytest -m slow                    # the paper's own scripts rebuild its tables from the rows (~4 min)
python sets/build_sets.py         # rebuild the harder and competition-math texts from their sources

BibTeX

@misc{yang2026outcomerl,
  title         = {What Outcome {RL} Buys: Finishing, Reliability, Rarely Reach},
  author        = {Yang, Shan},
  year          = {2026},
  eprint        = {2610.34894},
  archivePrefix = {arXiv}
}