acceptodds
Under review as a conference paper at ICLR 2027

When Does Longer Reasoning Help? Predicting Mathematical Reasoning Through Discovery and Execution

Abstract

Test-time compute can improve mathematical reasoning, but can short-budget runs predict how mathematical reasoning scales with additional compute? We introduce a Discovery–Execution (DE) framework that predicts the aggregate held-out scaling curves through a convolution of strategy discovery and conditional execution. From independent short-budget attempts and oracle-sketch-conditioned runs, the framework estimates cumulative success along held-out reasoning trajectories under alternate compute allocations. We evaluate four models on 35 fresh Olympiad problems and non-geometry problems from IMO-ProofBench Advanced. Under the DE framework, near-saturated execution predicts geometric scaling, as observed for the GPT models. For Claude Opus 4.8, incorporating measured execution substantially improves held-out forecasts over geometric extrapolation across one- and two-arm allocations. As a secondary application, regularized DE (R-DE) decisions to continue or restart yield lower average regret than the best model-specific retrospective policy. Together, these results show that measuring conditional execution provides information about longer reasoning that short-budget success rates do not always capture.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mathematical reasoning, test time scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xLSbOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Discovery-Execution framework to model the relationship between increasing computational resources and final reasoning performance. The framework divides mathematical problem-solving into two stages: discovering a feasible problem-solving strategy and executing that strategy. It then models the success rate under different computational budgets through the convolution of the strategy discovery probability and the conditional execution success rate. The experiments cover four different large language models and validate the framework on multiple mathematical datasets.

Critical strengths
  1. The experimental design is relatively comprehensive, including multiple baselines and ablation studies.
  2. The research problem addressed by the paper is meaningful: whether model performance under longer reasoning budgets and different allocations of computational resources can be predicted using a limited set of measurements.
  3. The experimental design is well conceived. The Discovery–Execution decomposition is intuitive and provides good interpretability.
Critical weaknesses
  1. Can the Oracle Sketch represent the distribution of strategies naturally discovered by the model? Since the Oracle Sketch provides a manually designed strategy, is it possible that using a different strategy that better aligns with the language distribution of the LLM would lead to significant variations in performance?

  2. Regarding the reliability of Discovery Probability estimation, the paper indirectly estimates Discovery Probability by dividing the normal problem-solving success rate by the Oracle Execution success rate. However, this estimation heavily relies on the assumption that Oracle Execution can accurately represent the model's execution ability after naturally discovering a strategy. Therefore, the estimation may be biased.

  3. Using Cumulative Success as the main evaluation metric may mask performance degradation that occurs during long reasoning processes.

Questions for authors
  1. Is it possible to extract feasible strategies naturally proposed by the model from unaided reasoning trajectories and conduct a set of controlled experiments on these strategies using human evaluation or stronger models?
  2. Can the authors directly analyze the model's reasoning trajectories to verify whether the indirectly estimated Discovery Probability is accurate and evaluate the magnitude of the estimation bias?
  3. Could the authors further report Final Proof Accuracy under different computational budgets and compare it with Cumulative Success?
1 comment

Thank you for the thoughtful and detailed review!

We view DE as a predictive approximation for the aggregate performance across inference budgets, rather than a description of LLMs' internal reasoning trajectories. Its discovery–execution components are modeling abstractions whose usefulness we evaluate through held-out forecasts. Oracle representativeness remains important because mismatch can affect both predictions and the interpretation of inferred discovery.

1. Execution with model-generated strategies

The model-generated sketch control shows no consistent original-reference advantage. Appendix E.2/Table 8 compares sketches from verified model-generated proofs with original references on 38 matched problems (table and verification protocol in general response Section 5).

This tests strategy choice; supplying a sketch still differs from continuing the trajectory that discovered it, and the executing solver need not be the proof generator. Appendix E.1's initial-versus-delayed sketch control shows little change for Opus (20/48 versus 20/48), GPT-5.4 (30/45 versus 29/45), and GPT-5.5 (11/12 versus 11/12). Only Muse shows a substantial drop (34/71 to 17/71), identifying a limitation of history independence for Muse in this experiment.

2. Discovery estimates and bias sensitivity

We directly inspected held-out unaided trajectories to check inferred discovery. At block 1, we count an attempt as positive if all three reference steps are recognized or its proof is correct. Across all 57 problems, mean plain-DE discovery differs from the observed fraction by +2.9 pp for GPT-5.5, +3.3 for GPT-5.4, −0.4 for Muse, and +9.2 for Opus (general response Section 5). Discovery is fitted from separate short-attempt and oracle measurements. The acquisition check uses six seeds for Opus/GPT-5.5 and three for the other models.

These observations provide a lower-bound diagnostic, assuming the positive annotations indicate acquisition: missing write-ups and unfinished alternative strategies can go undetected. They do not reveal the true acquisition rate, so the differences above are not estimates of discovery bias.

We also tested how an error in oracle execution would affect discovery estimates and forecasts. We multiplied the measured execution probabilities at every checkpoint by 0.8, 0.9, 1.0, 1.1, or 1.2, capped them at 100%, and refit plain DE. Short attempts, measurement weights, the estimator's boundary constraint, and held-out outcomes were unchanged. For six-seed Opus:

Assumed change to oracle executionMean inferred discoveryForecast RMSE (pp)
20% lower66.45%9.14
10% lower64.42%5.79
Unchanged62.11%2.18
10% higher61.46%2.04
20% higher60.86%1.91

Changes are relative: a measured probability of 60% becomes 54% under the 10% reduction. RMSE measures aggregate Parallel-1 forecasts over blocks 1–8.

Lower assumed execution leads DE to infer more discovery, and the downward adjustments substantially worsen forecasting error. Upward adjustments modestly improve it. This quantifies sensitivity to a specified mismatch; it does not show that such a mismatch exists or measure its actual magnitude.

3. Final-proof versus cumulative accuracy

Final-proof and cumulative accuracy differ by at most 1.17 percentage points across the proof checkpoints. For the AIME extension, final-answer accuracy at block 5 is 528/960 (55.00%), compared with cumulative success of 556/960 (57.92%): a larger 2.92-point gap. We will report both metrics: DE forecasts production of a correct proof within budget, but does not select a correct earlier proof for delivery.

Opus's forecasting advantage persists under final-proof scoring. Evaluating the six-seed forecasts over blocks 1–8 gives RMSE of 4.44 pp (SG), 4.48 (R-SG), 1.85 (DE), and 1.72 (R-DE). Both DE variants still outperform both geometric baselines without refitting.

Each entry is final-proof accuracy / cumulative success, in percent, on all 57 proof problems. Opus uses six held-out seeds (342 runs); Muse, GPT-5.4, and GPT-5.5 use the original three seeds (171 runs each).

Budget (blocks)MuseGPT-5.4GPT-5.5Opus
132.7 / 32.749.1 / 49.180.7 / 80.752.3 / 52.3
236.8 / 36.859.6 / 60.286.0 / 86.067.0 / 67.3
337.4 / 37.462.6 / 63.788.9 / 89.571.6 / 71.6
437.4 / 37.464.3 / 64.990.1 / 90.673.7 / 74.0
537.4 / 37.466.7 / 67.891.2 / 91.873.7 / 74.3
637.4 / 37.467.3 / 68.491.8 / 92.474.3 / 75.1
737.4 / 37.469.6 / 70.891.8 / 92.475.1 / 76.0
839.8 / 39.870.2 / 71.391.8 / 92.475.1 / 76.3

We hope these controls and the final-proof comparison address your concerns and support reconsideration of your assessment.

Reviewer_99UFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether short-budget measurements can predict the benefits of longer test-time mathematical reasoning. It proposes a Discovery–Execution (DE) framework that models proof success as a convolution of strategy discovery and conditional execution probabilities. Discovery is estimated from independent short-budget attempts, while execution is measured using oracle-provided strategy sketches. A regularized variant (R-DE) uses hierarchical Bayesian estimation to improve prediction stability.

Experiments on 57 Olympiad-level problems across four language models show that execution-aware forecasting substantially improves prediction accuracy for Claude Opus 4.8, while providing limited gains for other models. The authors also investigate the application of the proposed framework to adaptive inference decisions.

Critical strengths
  1. Meaningful problem formulation. The paper addresses an important question in test-time scaling: when is longer reasoning preferable to repeated independent attempts? The Discovery–Execution (DE) decomposition provides an intuitive probabilistic framework, and its connection to geometric scaling under saturated execution is theoretically coherent.

  2. Improved prediction through Bayesian regularization. The proposed R-DE framework incorporates hierarchical Bayesian estimation to reduce uncertainty from limited observations, improving the stability of long-budget success predictions.

  3. Adaptive compute allocation. The authors extend the predictive framework to guide decisions between continuing an existing reasoning trajectory and starting a new attempt, demonstrating its potential utility for inference-time resource allocation.

Critical weaknesses
  1. Restrictive modeling assumptions. The framework assumes geometric strategy discovery and separates discovery from execution. However, mathematical reasoning often involves multiple strategies and iterative strategy refinement, which may not fit these assumptions.

  2. Reliance on oracle strategies. Execution probabilities are estimated using oracle-provided strategies, which may not accurately reflect execution after naturally discovering a strategy.

  3. Limited generalizability. The evaluation is limited to 57 non-geometry problems and four proprietary models. Experiments on broader mathematical reasoning benchmarks and open-source models are needed to establish generalizability, especially since R-DE underperforms simpler baselines on GPT-5.5.

  4. Limited evaluation of prediction reliability. The evaluation mainly focuses on aggregate prediction errors, with limited analysis of prediction accuracy and uncertainty at the problem level. This raises concerns about the reliability of the framework for problem-specific compute allocation.

Questions for authors
  1. Modeling assumptions. How well does the proposed framework capture mathematical reasoning involving multiple solution strategies and iterative interactions between strategy discovery and execution? Could the authors further justify these assumptions?

  2. Oracle-based execution. Can the authors provide evidence that execution probabilities estimated using oracle-provided strategies are representative of those following naturally discovered strategies?

  3. Generalizability. Could the authors extend the evaluation to broader mathematical reasoning benchmarks and open-source models? What explains the weaker performance of R-DE on GPT-5.5?

1 comment

Thank you for the thoughtful and constructive review! We provide additional evidence and clarifications.

Our AIME results and new seeds strengthen the aggregate forecasting evidence. We also test sketch sensitivity and clarify how DE accommodates multiple strategies and refinement. GPTs and Opus favor values close to the geometric case, while Muse shows a limitation. (Appendix E.3)

1. Multiple strategies and iterative refinement

Multiple strategies preserve DE under our assumptions. For a fixed problem, let αj\alpha_j be the constant probability of first discovering strategy jj in a block, conditional on no previous discovery. Assume mutually exclusive first discoveries, 0<α=∑jαj≤10<\alpha=\sum_j\alpha_j\leq1, and execution curves εj(r)\varepsilon_j(r) stable across acquisition times and prior histories. Then P(success by H)=∑t=1H(1−α)t−1α⋅ε‾(H−t+1)P(\text{success by }H)=\sum_{t=1}^H(1-\alpha)^{t-1}\alpha\cdot\overline{\varepsilon}(H-t+1), where $\overline{\varepsilon}(r)=\sum_j\frac{\alpha_j}{\alpha}\varepsilon_j(r). $ Execution becomes a discovery-weighted mixture. Balanced oracle sampling approximates it if the sketches represent relevant strategies and their discovery weights are approximately equal or execution curves are similar.

On 39 AIME problems with two distinct sketches, either sketch yields lower aggregate RMSE than SG and R-SG. Errors in pp:

Oracle measurementSGR-SGDER-DE
Eight sketch A6.756.744.724.90
Eight sketch B6.756.745.455.47
Four A + four B (200 samples)6.756.745.445.29
Eight A + eight B6.756.745.105.46

The balanced row averages 200 random four-A/four-B selections, refitting each time. Either tested sketch gives lower aggregate error; this supports robustness to sketch choice.

2. Execution can include iterative refinement

Define ε(r)\varepsilon(r) as completion within rr blocks after first discovery, including checking, failed derivations, revisions, and further discovery. The convolution still applies if this conditional curve is stable across acquisition times and preceding histories (Assumption 1). Empirically, Appendix E.1 tests Assumption 1: initial versus delayed sketches give 20/48 versus 20/48 successes for Opus, 30/45 versus 29/45 for GPT-5.4, and 11/12 versus 11/12 for GPT-5.5; Muse falls from 34/71 to 17/71, showing a limitation. AIME additionally tests forecasting in a solve–critique–revise setting. We view our method as a predictive approximation that isolates a dominant discovery bottleneck via oracle sketches, rather than a description of internal reasoning dynamics of LLMs.

3. Oracle representativeness

We cover oracle representativeness in section 5 of our general response comment. Model-generated sketches show no consistent original-reference advantage, and auditing held-out trajectories provide a discovery diagnostic. These checks support our oracle execution approximation.

4. Broader evaluation and GPT-5.5

The evaluation now includes 120 AIME problems with an open model and three additional Opus/GPT-5.5 seeds (general response 1–2). Opus's advantage persists; pooled GPT-5.5 R-DE and R-SG errors are essentially tied (1.19 versus 1.21 pp).

GPT-5.5's weaker benefit is consistent with near-saturated execution: 162/171 original oracle trajectories succeed in block 1, versus 168/171 by block 8. Near execution saturation leaves little additional execution signal to exploit, and DE reduces to geometric scaling (Proposition 1).

Submitted Appendix E.7/Table 13 tests transfer over 50 splits, fitting priors on 38 problems and evaluating 19. With test-problem short attempts but no oracle sketches, R-DE improves on R-SG for Opus (5.04 versus 6.24 solved-count RMSE) and Muse; benefits are not uniform across models.

5. Problem-level reliability and practical scope

We discuss problem-level results and practical scope in general response Section 4. Per-problem uncertainty remains noisy. Our practical use is benchmark-wide forecasting: the reusable three-seed 4→8 measurement uses 50% less allocated output-token budget than the separate allocation sweep.

Problem-level solved-count RMSE (Appendix D.3/Table 5 for frontier models; plus eight-oracle analysis for AIME):

Model / benchmarkTrajectories per problemSGR-SGDER-DE
Muse61.271.231.191.09
GPT-5.461.221.211.201.16
GPT-5.560.710.690.740.71
Opus60.930.920.920.86
Qwen3-4B / AIME81.461.411.491.46

Frontier targets are six individual Parallel-2 arms over blocks 1–4; AIME uses eight held-out trajectories over blocks 1–5. On Table 5's targets, R-DE improves on both geometric baselines for three frontier models.


We hope the broader evaluation and assumption checks address your concerns and support reconsideration of your assessment.

Reviewer_MGX2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose Discovery–Execution (DE), a framework that predicts how an LLM's success on olympiad proofs grows with more test-time compute. A model can fail for two reasons. It may never find the right proof strategy, or it may find the strategy but not finish the proof. So two models with the same short-budget success rate can gain differently from longer reasoning.

DE sums over the block where the strategy is first found: the chance of first finding it in block k times the chance of finishing it in the remaining K − k + 1 blocks. Discovery is assumed geometric. One-block success comes from 24 fresh attempts per problem. Execution comes from three 8-block runs that start with a verified sketch of at most 25 words. If execution always finishes within the first block, DE reduces to a simple geometric curve (SG). R-DE adds Beta and Dirichlet priors shared across problems.

The authors build AOBench, 35 non-geometry problems from 2026 olympiads, and add 22 problems from IMO-ProofBench Advanced. They test four models with 8 blocks of 200k output tokens. GPT-5.6 Sol grades the proofs, and IMO medalists agree with it on 48 of 50 audited proofs. DE then predicts held-out runs that split the budget into 1 or 2 attempts.

Averaged over models, R-DE lowers RMSE (in solved trials) from SG's 5.94 to 4.36 at N = 1 and from 5.84 to 3.76 at N = 2 (Table 1). At N = 4, run for two models, all predictors underestimate Muse-Spark (Table 4). For continue-or-restart choices, R-DE's regret is 1.50 versus 1.79 for the best fixed action (Table 3).

Critical strengths

The idea is simple, and the authors test it carefully. For Opus, DE helps exactly where it should. On the 28 problems where at least one of three sketch-conditioned runs needs more than one block, DE lowers the error from 5.50 to 1.86. On the other 29 problems, SG is slightly better (Table 2). A sketch taken from another problem in the same domain does not help any model (Table 10). This tells me the execution curve carries real information, at least for Opus.

The paper is honest about its failures. It reports that R-SG beats R-DE for GPT-5.5 (Table 6) and that all methods underestimate Muse at N = 4 (Table 4). AOBench uses new 2026 problems with checked reference proofs, and an IMO-medalist audit agrees with the grader on 48 of 50 proofs (Appendix F). This is useful for the community.

Critical weaknesses

The evidence that measuring execution improves prediction comes from one of four models. In the bootstrap test (Table 6), Opus is the only model where R-DE is clearly better than both SG and R-SG. For Muse and GPT-5.4, both intervals include zero. For GPT-5.5, R-SG is clearly better than R-DE. Even for Opus, the gain comes from 28 problems with 3 runs each (Table 2). The allocation results are also limited. N = 4 is not run for Opus, the one model where measuring execution clearly helps. For Muse at N = 4, R-DE predicts 78.5 solved trials, but 93 are observed (Appendix D.1). I would like to see N = 4 for Opus and at least one more model where execution is not saturated.

The framework does not predict beyond the budget it measures. The sketch-conditioned runs are 8 blocks long, the same as the longest target run, so DE never predicts an execution budget it has not measured. The measurement is also not cheap. It takes about 48 blocks per problem, while one target allocation costs 24. It also needs a verified sketch for each problem. Without test-problem sketches, the Opus error rises from 3.40 to 5.04 (Table 13). I do not expect DE to be cheaper than running one allocation. My point is that the abstract asks whether short-budget runs can predict scaling, but the execution measurement is as long as the longest target. A simple fix is to fit execution on blocks 1 to 4 and predict blocks 5 to 8.

Questions for authors

Questions that could change my score:

  1. Can you fit the execution curve on the first 4 blocks only and predict Opus's Parallel-1 curve at blocks 5 to 8? This is the result that matters most for my score.
  2. Can you run N = 4 for Opus? For each model, does DE pick the right allocation among N = 1, 2 and 4? Section 3.4 defines the optimal allocation, but no experiment tests it.
  3. Many unaided Muse runs stop early without a correct proof (38 of 171, Appendix A.2). How many tokens do the runs actually use under each allocation? If long runs stop early, N = 1 uses less compute than N = 4.

Need Clarification:

  1. For Opus, R-DE lowers the aggregate error from 7.16 to 2.87 (Table 1), but the problem-level error only from 0.93 to 0.86 (Table 5). Table 5 also uses different targets (single Parallel-2 arms, blocks 1 to 4). Does the aggregate gain mostly come from correcting SG's average bias, while problem-level errors cancel in the sum?
  2. The grader is a GPT model, and two of the solvers are GPT models. The audit sample is stratified by score, not by solver (Appendix F). Does agreement between the human audit and the grader differ by solver?
  3. Appendix D.5 says the supplement reports uncertainty for the regret results. Can you add those intervals to Table 3? The gain over the best fixed action is small (1.50 vs. 1.79), and R-DE is worse than it for Muse (1.57 vs. 0.57).
1 comment

We thank you for the thoughtful review and for suggesting the extrapolation test!

Using only four oracle blocks, DE and R-DE outperform both geometric baselines on Opus's blocks 5–8. The AIME extension and additional seeds are reported in the general response (Sections 1–2).

1. Prediction beyond the measured horizon

We performed the requested Opus 4->8 and a Qwen/AIME 3->5 check. Both outperform SG and R-SG. Fits use the original short-attempt banks and oracle checkpoints through cutoff cc, censoring unfinished runs. Flat extrapolation sets ε(c+j)=ε(c)\varepsilon(c+j)=\varepsilon(c); linear extrapolation uses ε(c+j)=min⁡(1,ε(c)+j[ε(c)−ε(c−1)])\varepsilon(c+j)=\min(1,\varepsilon(c)+j[\varepsilon(c)-\varepsilon(c-1)]).

Opus oracle success improves from 146/171 to 153/171 in block 4 to block 8.

Aggregate Parallel-1 RMSE on later blocks only, pp:

Model / oracle and target seedsMeasured oracle blocksEvaluated target blocksSGR-SGDE, flatDE, linear
Opus / original 1–31–45–83.253.152.412.41
Opus / pooled 1–61–45–84.134.032.372.23
Qwen3-4B / original 0–71–34–55.535.811.772.35

R-DE flat/linear RMSE is 0.78/0.41 pp for original Opus, 0.67/1.56 pp for pooled Opus, and 2.16/3.13 pp for Qwen. Its shared priors are refitted using only the measured prefix.

Original Opus flat/linear forecasts (top row) are identical: 54 problems have zero tail increment or are already at probability 1; the other three have fitted discovery probability zero.

2. Opus N=4 and allocation selection

We have now run Opus at N=4, as was requested: three banks per problem, each of 4 independent 2-block runs. Opus N=4 success is at 76.02%, versus 74.51% predicted by DE and 79.28% by R-DE. Observed best allocation for Opus is N=2, and both DE and R-DE select it. Both also select the observed-best allocation for Muse; only R-DE does so for GPT-5.5, whose N=2 advantage is 1 success out of 171. R-DE therefore selects the observed-best allocation for all three models, and DE for two.

Success rates and predicted choices at total 8-block ceiling:

Solver / evaluation sourceOracle seeds for fittingN=1 successN=2 successN=4 successDE choiceR-DE choice
Muse / dedicated banks339.77%45.03%54.39%N=4N=4
GPT-5.5 / dedicated banks692.40%92.98%92.40%N=4N=2
Opus / dedicated banks675.44%77.78%76.02%N=2N=2

Choices use all available oracle seeds: six for Opus/GPT-5.5 and three for Muse, with the original 24 short attempts fixed. Fixed-parameter SG predicts a tie, approximately observed for GPT-5.5. Opus N=4 now includes all 171 allocation trials, three per problem. We will add these results to the paper.

3. Actual output-token usage

We agree that realized output-token usage differs across allocations despite equal ceilings. The ceiling is 1.6M output tokens per trial. Muse uses approximately 42% more tokens at N=4 than N=1; its 38 unsuccessful early-stopped N=1 runs average 495k tokens. We will describe these as equal-ceiling comparisons, not matched-spend evidence. The AIME protocol provides complementary forecasting evidence without early stopping.

Mean recorded output tokens per allocation:

SolverN=1N=2N=4
Muse843k940k1,193k
GPT-5.5282k380k643k
Opus742k976k1,326k

4. Aggregate versus individual accuracy

We decompose Opus’s problem-level error (Table 5) into bias and residual variance (square roots reported). General response section 4 provides detailed results. Both components improve numerically, but indeed, bias correction is more noteworthy. The problem-specific improvement is noisier because each observation comes from only six trajectories.

Accurate aggregate forecasts support benchmark-wide compute planning. The three-seed 4→8 measurement uses 50% less allocated output-token budget than separately evaluating all allocations.

5. Grading agreement by solver

Human–grader agreement is 96.2% for GPT proofs (25/26) and 95.8% for non-GPT proofs (23/24). Individually, agreement is 14/14 for GPT-5.5, 11/12 for GPT-5.4, 12/12 for Opus, and 11/12 for Muse. Each group has one grader-fail/human-pass disagreement; neither has a grader-pass/human-fail disagreement.

6. Regret uncertainty

The estimated regret reductions have 95% intervals that include zero. Paired whole-problem bootstrapping, with fitted policies fixed, gives an Opus reduction of 0.86 [−0.14, 2.71] using the dedicated restart bank and a four-model average reduction of 0.29 [−0.54, 0.96] using the saved banks. We will describe the decision application as exploratory.

We hope the requested horizon test and additional evidence address your concerns and support reconsideration of your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New experiments and common reviewer concerns

We thank the reviewers for their thoughtful feedback. The main concerns were about generalizability, prediction beyond the measured execution horizon, the modeling assumptions and oracle representativeness, and the practical value of aggregate forecasts.

We address these concerns with AIME experiments on an open model, three additional seeds for Opus and GPT-5.5, beyond-horizon forecasting tests, and dedicated Opus N=4 runs. We clarify the modeling assumptions in our individual replies and examine oracle representativeness through alternative sketches and unaided trajectories. Our primary application is benchmark-wide forecasting: reusing measurements across three allocations requires 50% less allocated solver budget in the comparison below, while individual-problem gains remain model-dependent.

1. Broader evaluation: Qwen on AIME (2023-2026)

On 120 AIME problems, Qwen3-4B-Instruct-2507 uses 32 short attempts and 8 oracle runs per problem to forecast 8 unaided runs (960 targets): 5 solve–critique–revise blocks, up to 8,192 output tokens each, without early stopping, tools, or internet. Its execution curve was unsaturated (block 1: 53.54%; block 5: 65.00%). Aggregate RMSE over blocks 1–5 is:

PredictorRMSE (p.p.)
SG3.70
R-SG3.88
DE1.53
R-DE1.89

DE and R-DE reduce error by 59% and 51% relative to SG and R-SG.

2. Three additional Opus/GPT-5.5 seeds

We added three seeds each for 8-block oracle and held-out Parallel-1 runs on all 57 problems, keeping the 24 short attempts fixed. Rows use the indicated seeds for separate oracle and unaided runs. Aggregate RMSE over blocks 1–8 (pp):

ModelSeedsSGR-SGDER-DE
OpusOriginal 1–34.194.212.151.68
OpusAdditional 4–65.645.651.901.46
OpusAll 1–64.884.902.181.13
GPT-5.5Original 1–31.401.091.772.14
GPT-5.5Additional 4–61.891.632.291.43
GPT-5.5All 1–61.511.211.951.19

Opus's advantage persists. GPT-5.5's pooled R-DE and R-SG errors are essentially tied, consistent with near-saturated execution and Proposition 1.

3. Extrapolating the execution curve

Using only the first 4 oracle blocks for Opus and 3 for Qwen, flat and linear execution extrapolations both improve later Parallel-1 forecasts over SG and R-SG.

Aggregate RMSE on later blocks only, percentage points:

Model / oracle and target seedsMeasured oracle blocksEvaluated target blocksSGR-SGDE, flatDE, linear
Opus / original 1–31–45–83.253.152.412.41
Opus / pooled 1–61–45–84.134.032.372.23
Qwen3-4B / original 0–71–34–55.535.811.772.35

4. Practical application: benchmark-wide compute planning

Individual-problem gains are noisy. We decompose Opus’s Table 5 errors into aggregate bias and problem-specific variation to assess the benefit for benchmark-wide forecasting.

SGR-DER-DE − SG
Problem-level RMSE0.93 ± 0.100.86 ± 0.08−0.07 ± 0.08
Aggregate0.29 ± 0.090.12 ± 0.06−0.17 ± 0.09
Problem-specific0.88 ± 0.080.85 ± 0.07−0.03 ± 0.06

SEs use 10,000 bootstrap resamples, with predictions held fixed.

For three-seed Opus 4→8, measurement allocates 24 + 3 × 4 = 36 blocks per problem, versus 3 allocations × 3 trials × 8 blocks = 72 to evaluate N=1, 2, 4 separately: 50% less allocated output-token budget, excluding sketch preparation/grading.

5. Oracle representativeness

Sensitivity to sketch choice. Appendix E.2/Table 8 compares original sketches with sketches from verified proofs generated by the evaluated models on 38 matched problems. Both use the same format and at most 25 words; GPT-5.6 Sol and IMO medalists checked the proofs.

SolverOriginalModel-generated alternativeChange (pp)
Muse58/11466/114+7.02
GPT-5.5109/114111/114+1.75
Opus88/11482/114−5.26

There is no consistent original-sketch advantage.

Inferred versus observed acquisition. At block 1 on all 57 problems, we count all three reference steps recognized or a correct proof, including correct alternative routes:

ModelObservedDE discoveryΔ (DE − observed, pp)
Opus52.9%62.1%+9.2
GPT-5.579.2%82.1%+2.9
GPT-5.449.1%52.4%+3.3
Muse34.5%34.1%−0.4

There are 342 attempts each for Opus/GPT-5.5 and 171 for the others. Treating positives as valid acquisition evidence gives a lower-bound diagnostic: missing write-ups and unfinished alternative strategies may go undetected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.