- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents an empirical study on test-time compute in LLMs. Specifically, this paper focuses on the correlation between retrospective coverage and deployable policies. The major finding is that retrospective coverage can hardly be a strong indicator for determining action policies, although there are some correlations.
Critical strengths
- This paper targets an interesting topic.
- This paper does provide some interesting finding that may serve as a starting point for further investigation.
Critical weaknesses
- This paper does not provide any conclusive results. Although I think the major finding of this paper interesting, it is unclear whether this finding can be used to improve test-time compute in LLMs.
- The benefit of test-time compute for LLMs in general is typically limited as existing empirical results have demonstrated. Most exploration of test-time compute focuses on domains like maths and coding. However, this paper does not consider any domain-specific characteristics in test-time compute, although a mathematical benchmark is used in the study. In my opinion, this decision may make the findings of this paper less interesting and less useful.
- The scope of the reported study is limited. It is unclear whether the findings of this paper are generalizable for cutting-the-edge LLMs.
- Somewhere in this paper, the presentation seems to be difficult to follow. In particular, I do not quite understand what Section 4 aim to present.
Questions for authors
Q1: What are the takeaway messages from this paper? How can this paper be used to improve the practice? Q2: How is Section 4 related to the empirical study reported in this paper? Q3: If domain specialty is further considered, how would this impact the findings of this study?
5 comments
Q3. Domain characteristics and generalization
The distinction between candidate coverage and choosing an action applies beyond a particular domain. The quantitative gap does not have to be domain-invariant. Appendix F.3 reports metadata-defined groups, averaging models within each question before computing problem-level intervals.
On MATH, the original sixteen-outcome, eight-selection gap is 2.91 points for level-one problems (11 questions, interval [0.55, 6.42]) and 5.50 for level-five problems (32 questions, [3.91, 7.21]). This is descriptive evidence of different magnitudes, not a causal difficulty effect or a statistical test of their difference. The example is not a subgroup interaction test, and the small denominators remain explicit.
On the extended GPQA bank, the eight-selection gaps are 10.29 points for biology (12 questions, [7.12, 13.81]), 13.77 for chemistry (63, [11.48, 16.06]) and 13.25 for physics (57, [11.32, 15.21]). Biology has a particularly small denominator. These results support examining domain composition rather than presenting one pooled number as universal; they do not establish a significant domain interaction.
We also retain the near-ceiling qualification for the stronger MATH models. Neither these strata nor the additional Qwen3-8B routing test establish quantitative generalization to frontier LLMs. Section 9 distinguishes the general measurement definitions from the empirical findings for these three open models, these action definitions and these evaluation populations.
Thanks for the response. I think the response helps clarify some issues. However, I still think this paper may not be mature enough for publication in the current form, although I also think some findings are interesting.
Thank you for asking us to make the practical purpose and empirical scope clearer. The paper is an evaluation audit, not a new algorithm for improving reasoning accuracy. Its practical contribution is to identify which claim an oracle measurement can support and which additional evaluation is necessary before choosing a deployed policy.
Q1. Takeaways and practical use
There are three different decisions.
For a concrete example from the submitted model-action experiment on GPQA, the fixed model scores 46.31%, product-coupled retrospective coverage is 68.53%, and eight-outcome split-selected value is 60.06%. These correspond to roughly 22.22 and 13.75 points of two different diagnostic headrooms. Neither number is a measured deployable routing gain. The practitioner should therefore not select a routing system from either headroom alone; its held-out policy improvement and cost are separate measurements.
We have expanded the empirical check: GPQA fitting now contains 316 questions/model instead of 66, and evaluation has 64 outcomes per stochastic action on the same 132 problems. Expanded structured prompt routing gains 0.21 points, with nominal interval [-1.27, 1.70], relative to its fit-selected fixed action. Homogeneous MATH fitting, however, gives a small positive structured gain of 1.08 points [0.25, 1.97], which does not survive exploratory within-benchmark multiplicity adjustment. Thus the conclusion is not that routing never helps; it is that coverage and policy gains must be measured separately.
This reporting workflow makes the evidence for deployment decisions explicit. We do not claim to have demonstrated a new accuracy or efficiency improvement for test-time compute.
Q2. How Section 4 connects to the experiments
Section 4 supplies the interpretation of the measured quantities. The Introduction and the opening of Section 4 clarify its role as statistical context for the empirical targets, not a standalone theoretical contribution:
The last point does not rule out useful estimation. We now provide panel-size curves and model-based component estimates. On GPQA with 64 outcomes/action, the coverage-minus-split-selected gap remains 11.60 points [10.22, 13.01] at selection size thirty-two. This is a more resolved finite-panel measurement, not a claim that the population component has been identified without assumptions.
For transparency, the revision also corrects the fitting count (748 total states, including 500 legacy MATH states, rather than 248 total), the model-router replication description (eight on MATH and sixteen on GPQA), and legacy MATH history-feature and stop tie inconsistencies. Figure 3 separates tie breadth from changes in maximizing sets; we withdraw the unsupported link between independent-choice disagreement and utility-router failure. Appendices C and F clarify scoring after truncated GPQA reasoning. Marking all truncated fresh continuations incorrect in a separate sensitivity, with prefix/stop utilities unchanged and no attempts omitted, gives an extended coverage-minus-V32 gap of 11.69 points [10.31, 13.13]. The primary labels and original coverage results are retained.