- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines two distinctions in evaluating LLM reasoning trajectories: predicting correctness across different problems versus distinguishing successful and unsuccessful attempts on the same problem, and measuring successful continuation from a prefix versus demonstrating an advantage over restarting with a comparable nominal budget.
The primary study evaluates early hidden-state probes on repeated R1-Distill-Qwen-7B generations. On the same 22-problem evaluation set, a probe achieves pooled AUROC of 0.761 but within-problem AUROC of 0.515 at four tokens. Additional probes reveal some within-problem signals, qualifying the main negative result. A complementary study evaluates continuation and restart behavior across 89 MATH problems and two models. The paper contributes an explicit AUROC decomposition, controlled empirical comparisons, and extensive sensitivity analyses, while emphasizing that its findings do not establish a general absence of correctness information or prefix value.
Critical strengths
-
The primary comparison directly addresses a consequential evaluation mismatch. Evaluating pooled and within-problem AUROC using the same fitted scores and evaluation rows is a strong design choice. The contrast between 0.761 pooled and 0.515 within-problem AUROC on panel W cannot be attributed simply to comparing different evaluation populations. The offline selection analysis further shows why the distinction matters: at four tokens, selecting among two, four, or eight attempts produces estimated gains below one percentage point, with intervals including zero despite substantial oracle headroom as shown in Sections 2.3–2.4, Table 2. This is useful evidence for researchers evaluating probes intended for inference-time selection.
-
The paper actively investigates evidence that qualifies its initial interpretation. The problem-centered and per-problem probes reveal heterogeneous positive signals, and the paper explicitly distinguishes these results from the pooled-trained probe's weak within-problem performance. It also examines shared token prefixes, cohort attrition, probe specifications, numerical backends and additional continuation branches. These analysis strengthen the credibility of the narrower conclusion and avoid turning a negative probe result into an unsupported claim that the representations contain no relevant information.
Critical weaknesses
-
The strongest result comes from one generating model, quantized replayed representations and 22 problems selected using their observed success rates. The broader panel uses the same model and overlapping trajectories. Moreover, the strongest pooled-versus-within contrast occurs at four tokens. Table 9 shows that pooled performance on W is itself much closer to chance at subsequent anchors. The result therefore establishes a striking early-window example, rather than a persistent separation throughout reasoning.
-
The alternative probes also change several factors simultaneously, including training population, centering, and access to other attempts from the evaluation problem. They demonstrate that some within-problem information is available, but do not isolate why the original probe fails to capture it. Since the paper acknowledges that difficulty prediction and within-problem evaluation already exist in prior work, its contribution depends heavily on the empirical insight added by this case study. A more controlled comparison of pooled versus within-problem training objectives, or replication on another model, would materially strengthen that contribution.
-
Continuation rates use four to eight samples, and restart rates use four samples per budget. Nine of the ten highlighted extra-batch comparisons rely on interpolated restart estimates rather than measurements at the actual comparison budget. Figure 3 shows uncertainty at the measured endpoints, but does not provide uncertainty intervals for the continuation-minus-restart differences that incorporate both sampling uncertainty and the interpolation assumption.
-
The extra-batch analysis usefully avoids selecting on those additional continuation outcomes, but still concerns initially selected terminal candidates. Meanwhile, the headline “one of 178” prefix-limited count depends on a large advantage threshold and taxonomy precedence, while other categories explicitly contain positive advantages. The authors acknowledge these limitations, so I do not regard the presentation as a claim that prefix benefits are universally rare. However, the study leaves the magnitude and robustness of prefix advantages insufficiently established. Direct matched-budget restart measurements on the highlighted candidates would be substantially more informative than additional taxonomy refinements.
Questions for authors
-
The within-problem bootstrap weights a problem sampled w_c times by w_c^2. For inference over sampled problems, why is this preferable to treating each sampled problem copy as a separate cluster and weighting its within-pair contribution by w_c? What population estimate does the quadratic weighting target, and do the conclusions change under linear cluster multiplicities?
-
Can you directly measure restart success at the exact nominal budgets of the ten in-grid extra-batch comparisons, with enough samples to estimate continuation-minus-restart uncertainty? Would the positive aggregate pattern remain?
-
With the same representations, training problems, and problem-disjoint evaluation, does a training objective explicitly targeting within-problem discrimination outperform pooled logistic classification? This would help distinguish an objective mismatch from weak transferable information in the tested representations.
2 comments
Thank you for the detailed response and additional experiments. The bootstrap correction addresses my concern about cluster multiplicities and the matched-objective comparison helps clarify that changing the training loss alone does not establish an improvement in this setting. The broader BF16 evaluation also strengthens the evidence that the early pooled-versus-within gap extends beyond the original selected panel. However, the fact that nine prefixes already contain the correct answer limits what we can conclude about the source of that advantage. Grading only newly generated text rules out direct credit for the old answer, but does not separate answer reuse from further reasoning, as you acknowledge.
My remaining concern is about the scope of the empirical contribution. The strongest pooled-versus-within separation remains concentrated in the early window of the original model, while the second-generator results are mixed. The published-probe evaluation is useful, but also shows meaningful within-problem discrimination. Together, these results support the value of the proposed evaluation controls, while leaving the broader conclusions about reasoning trajectories less firmly established. Overall, the revision addresses several of my original concerns and makes the evidence more convincing. After considering the additions, I still find the empirical contribution somewhat limited in scope, so I will maintain my rating.
Thank you. We completed all three requested analyses and added broader evaluation, a published-probe test and a second generator.
You are right: for inference over questions, each sampled copy should contribute its within-question pairs once. A question drawn w times now contributes its ranking sum and pair count w times. Their ratio retains the pair-weighted target, not an equal-question average. The old implementation merged copies under the original identity, yielding quadratic weights instead of resampling separate question clusters. Linear multiplicities are now primary. Pooled AUROC still compares the full duplicated row population.
With saved baseline reanalysis scores and 2,000 draws, four-token within AUROC remains 0.5155. Its interval changes from [0.4733, 0.5985] to [0.4830, 0.5627]; pooled-minus-within is 0.2455 [0.1129, 0.3438]. All ten W-anchor within intervals include 0.5. Table 4 retains both conventions; intervals condition on fitted scores (Section 2.2; Appendix A.1).
All ten fixed in-grid candidates received 64 fresh outputs per arm. Restart reasoning budgets are exactly 2,048, 3,072 or 5,120 tokens, matching the archived prefix plus 1,024 continuation tokens; both arms retain a 512-token answer reserve. The new comparison requires no interpolation.
Continuation succeeds on 552/640 outputs versus 434/640 restarts. The equal-candidate mean advantage is +18.44 percentage points, with conservative Hoeffding interval [10.85, 26.03] accounting for sampling in both arms. Eight estimates are positive, one zero and one negative. Figure 1 and Table 11 give all candidate estimates and pointwise Newcombe/Wilson intervals. The positive aggregate pattern remains (Section 3.2; Appendix A.5).
This BF16 Transformers follow-up uses archived 4-bit prefixes and estimates advantage on the ten selected candidates. The original 1/178 count describes a threshold-and-precedence classification, not all positive advantages (Appendix B.3).
Nine prefixes already state the correct answer; the tenth reaches a symbolic sign repair without stating the evaluated answer. Answer reuse is a possible contributor. Grading only new text gives 549/640 versus 434/640, retaining +17.97 points [10.38, 25.56]. Direct credit for old text cannot explain the whole advantage. The model still sees the prefix; separating reuse from repair or further reasoning needs content interventions. The inspection is post-hoc, unblinded and AI-assisted (Section 3.3; Appendix A.5).
We held the 22-question panel, question-disjoint folds, training-only preprocessing and regularization fixed across global classification, problem-balanced classification and within-problem pairwise ranking. At four tokens, pairwise-minus-global within AUROC is −0.0064 [−0.0276, 0.0089] on original states and +0.0143 [−0.0014, 0.0355] on BF16 states.
Neither interval establishes an improvement from changing the loss alone in this matched linear-probe setup; a modest BF16 benefit remains possible. This isolates the loss change; it does not establish absence of transferable within-question information. These fits remain separate from the original union-trained probe (Section 2.3, Table 3; Appendix A.2, Tables 5–6).
BF16 evaluation covers all 128 archive questions and 32,768 attempts, retaining training rows and question-disjoint evaluation. At four tokens, pooled/within is 0.714/0.504: gap 0.210 [0.083, 0.322], with positive gap intervals in all three splits. The early gap extends beyond the selected panel; later contrasts are weaker (Appendix A.2, Table 7).
Zhang et al.'s released Qwen-7B/MATH probe gives archive-weighted pooled/within 0.800/0.643 on 2,185 stratified attempts from 49 mixed-outcome questions. The gap's sampling interval is [0.139, 0.173]; its separate conditional question interval is [0.057, 0.246]. Within-question signal remains. This tests answer-endpoint states on a separate archive with NF4 replay, not an exact replication (Section 2.4; Appendix A.3, Tables 8–9).
We completed 512 Ministral solutions: 32 questions × 16 seeds. At 32 tokens, pooled/within is 0.637/0.522, 0.650/0.545 and 0.711/0.541 for the three objectives. Only eight questions supply within pairs; all gap intervals include zero, and the primary four-token global result is 0.346/0.393. Later problem-balanced and pairwise selection gains are +1.48/+1.53 points, with pointwise intervals excluding zero. We report this variation rather than a persistent cross-model gap (Section 2.5; Appendix A.6, Tables 12–14).
A one-page change map locates these results; the main section order is preserved.