- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a Discovery-Execution framework to model the relationship between increasing computational resources and final reasoning performance. The framework divides mathematical problem-solving into two stages: discovering a feasible problem-solving strategy and executing that strategy. It then models the success rate under different computational budgets through the convolution of the strategy discovery probability and the conditional execution success rate. The experiments cover four different large language models and validate the framework on multiple mathematical datasets.
Critical strengths
- The experimental design is relatively comprehensive, including multiple baselines and ablation studies.
- The research problem addressed by the paper is meaningful: whether model performance under longer reasoning budgets and different allocations of computational resources can be predicted using a limited set of measurements.
- The experimental design is well conceived. The Discovery–Execution decomposition is intuitive and provides good interpretability.
Critical weaknesses
-
Can the Oracle Sketch represent the distribution of strategies naturally discovered by the model? Since the Oracle Sketch provides a manually designed strategy, is it possible that using a different strategy that better aligns with the language distribution of the LLM would lead to significant variations in performance?
-
Regarding the reliability of Discovery Probability estimation, the paper indirectly estimates Discovery Probability by dividing the normal problem-solving success rate by the Oracle Execution success rate. However, this estimation heavily relies on the assumption that Oracle Execution can accurately represent the model's execution ability after naturally discovering a strategy. Therefore, the estimation may be biased.
-
Using Cumulative Success as the main evaluation metric may mask performance degradation that occurs during long reasoning processes.
Questions for authors
- Is it possible to extract feasible strategies naturally proposed by the model from unaided reasoning trajectories and conduct a set of controlled experiments on these strategies using human evaluation or stronger models?
- Can the authors directly analyze the model's reasoning trajectories to verify whether the indirectly estimated Discovery Probability is accurate and evaluate the magnitude of the estimation bias?
- Could the authors further report Final Proof Accuracy under different computational budgets and compare it with Cumulative Success?
Thank you for the thoughtful and detailed review!
We view DE as a predictive approximation for the aggregate performance across inference budgets, rather than a description of LLMs' internal reasoning trajectories. Its discovery–execution components are modeling abstractions whose usefulness we evaluate through held-out forecasts. Oracle representativeness remains important because mismatch can affect both predictions and the interpretation of inferred discovery.
1. Execution with model-generated strategies
The model-generated sketch control shows no consistent original-reference advantage. Appendix E.2/Table 8 compares sketches from verified model-generated proofs with original references on 38 matched problems (table and verification protocol in general response Section 5).
This tests strategy choice; supplying a sketch still differs from continuing the trajectory that discovered it, and the executing solver need not be the proof generator. Appendix E.1's initial-versus-delayed sketch control shows little change for Opus (20/48 versus 20/48), GPT-5.4 (30/45 versus 29/45), and GPT-5.5 (11/12 versus 11/12). Only Muse shows a substantial drop (34/71 to 17/71), identifying a limitation of history independence for Muse in this experiment.
2. Discovery estimates and bias sensitivity
We directly inspected held-out unaided trajectories to check inferred discovery. At block 1, we count an attempt as positive if all three reference steps are recognized or its proof is correct. Across all 57 problems, mean plain-DE discovery differs from the observed fraction by +2.9 pp for GPT-5.5, +3.3 for GPT-5.4, −0.4 for Muse, and +9.2 for Opus (general response Section 5). Discovery is fitted from separate short-attempt and oracle measurements. The acquisition check uses six seeds for Opus/GPT-5.5 and three for the other models.
These observations provide a lower-bound diagnostic, assuming the positive annotations indicate acquisition: missing write-ups and unfinished alternative strategies can go undetected. They do not reveal the true acquisition rate, so the differences above are not estimates of discovery bias.
We also tested how an error in oracle execution would affect discovery estimates and forecasts. We multiplied the measured execution probabilities at every checkpoint by 0.8, 0.9, 1.0, 1.1, or 1.2, capped them at 100%, and refit plain DE. Short attempts, measurement weights, the estimator's boundary constraint, and held-out outcomes were unchanged. For six-seed Opus:
Changes are relative: a measured probability of 60% becomes 54% under the 10% reduction. RMSE measures aggregate Parallel-1 forecasts over blocks 1–8.
Lower assumed execution leads DE to infer more discovery, and the downward adjustments substantially worsen forecasting error. Upward adjustments modestly improve it. This quantifies sensitivity to a specified mismatch; it does not show that such a mismatch exists or measure its actual magnitude.
3. Final-proof versus cumulative accuracy
Final-proof and cumulative accuracy differ by at most 1.17 percentage points across the proof checkpoints. For the AIME extension, final-answer accuracy at block 5 is 528/960 (55.00%), compared with cumulative success of 556/960 (57.92%): a larger 2.92-point gap. We will report both metrics: DE forecasts production of a correct proof within budget, but does not select a correct earlier proof for delivery.
Opus's forecasting advantage persists under final-proof scoring. Evaluating the six-seed forecasts over blocks 1–8 gives RMSE of 4.44 pp (SG), 4.48 (R-SG), 1.85 (DE), and 1.72 (R-DE). Both DE variants still outperform both geometric baselines without refitting.
Each entry is final-proof accuracy / cumulative success, in percent, on all 57 proof problems. Opus uses six held-out seeds (342 runs); Muse, GPT-5.4, and GPT-5.5 use the original three seeds (171 runs each).
We hope these controls and the final-proof comparison address your concerns and support reconsideration of your assessment.