acceptodds
Under review as a conference paper at ICLR 2027

Problem or Path? Controls for Evaluating LLM Reasoning Trajectories

Abstract

High pooled correctness prediction and rising continuation success do not, by themselves, identify within-attempt information or a prefix specific advantage. We examine two controls: within-problem evaluation and budget-indexed restart comparisons. On public R1-Distill Qwen-7B trajectories, a separately versioned early-window probe reanalysis yields pooled AUROC 0.761 but pair-weighted within-problem AUROC 0.515 on the same 22-problem set at four tokens. Within-problem intervals include chance at all ten anchors, while post-hoc within-targeted probes reveal heterogeneous positives, including a failure-weighted mean of 0.556 at 32 tokens. A complementary truncation study covers 178 problem–model cells from 89 MATH problems and two small models. One cell receives a frozen prefix-limited label, although additional cells exhibit prefix advantages. An extra-batch check on all initial terminal candidates finds higher continuation success in all ten in-grid Ministral-3 comparisons against nominal-budget restart references, nine of which are interpolated. Selection, small branch counts, and unresolved numerical differences from archived probe reports limit interpretation. The results motivate problem conditioned discrimination and explicit restart-budget controls, not a general absence of outcome information or prefix value.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM reasoning, correctness probing, within-problem evaluation, test-time compute, restart baselines

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uW1zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines two distinctions in evaluating LLM reasoning trajectories: predicting correctness across different problems versus distinguishing successful and unsuccessful attempts on the same problem, and measuring successful continuation from a prefix versus demonstrating an advantage over restarting with a comparable nominal budget.

The primary study evaluates early hidden-state probes on repeated R1-Distill-Qwen-7B generations. On the same 22-problem evaluation set, a probe achieves pooled AUROC of 0.761 but within-problem AUROC of 0.515 at four tokens. Additional probes reveal some within-problem signals, qualifying the main negative result. A complementary study evaluates continuation and restart behavior across 89 MATH problems and two models. The paper contributes an explicit AUROC decomposition, controlled empirical comparisons, and extensive sensitivity analyses, while emphasizing that its findings do not establish a general absence of correctness information or prefix value.

Critical strengths
  • The primary comparison directly addresses a consequential evaluation mismatch. Evaluating pooled and within-problem AUROC using the same fitted scores and evaluation rows is a strong design choice. The contrast between 0.761 pooled and 0.515 within-problem AUROC on panel W cannot be attributed simply to comparing different evaluation populations. The offline selection analysis further shows why the distinction matters: at four tokens, selecting among two, four, or eight attempts produces estimated gains below one percentage point, with intervals including zero despite substantial oracle headroom as shown in Sections 2.3–2.4, Table 2. This is useful evidence for researchers evaluating probes intended for inference-time selection.

  • The paper actively investigates evidence that qualifies its initial interpretation. The problem-centered and per-problem probes reveal heterogeneous positive signals, and the paper explicitly distinguishes these results from the pooled-trained probe's weak within-problem performance. It also examines shared token prefixes, cohort attrition, probe specifications, numerical backends and additional continuation branches. These analysis strengthen the credibility of the narrower conclusion and avoid turning a negative probe result into an unsupported claim that the representations contain no relevant information.

Critical weaknesses
  • The strongest result comes from one generating model, quantized replayed representations and 22 problems selected using their observed success rates. The broader panel uses the same model and overlapping trajectories. Moreover, the strongest pooled-versus-within contrast occurs at four tokens. Table 9 shows that pooled performance on W is itself much closer to chance at subsequent anchors. The result therefore establishes a striking early-window example, rather than a persistent separation throughout reasoning.

  • The alternative probes also change several factors simultaneously, including training population, centering, and access to other attempts from the evaluation problem. They demonstrate that some within-problem information is available, but do not isolate why the original probe fails to capture it. Since the paper acknowledges that difficulty prediction and within-problem evaluation already exist in prior work, its contribution depends heavily on the empirical insight added by this case study. A more controlled comparison of pooled versus within-problem training objectives, or replication on another model, would materially strengthen that contribution.

  • Continuation rates use four to eight samples, and restart rates use four samples per budget. Nine of the ten highlighted extra-batch comparisons rely on interpolated restart estimates rather than measurements at the actual comparison budget. Figure 3 shows uncertainty at the measured endpoints, but does not provide uncertainty intervals for the continuation-minus-restart differences that incorporate both sampling uncertainty and the interpolation assumption.

  • The extra-batch analysis usefully avoids selecting on those additional continuation outcomes, but still concerns initially selected terminal candidates. Meanwhile, the headline “one of 178” prefix-limited count depends on a large advantage threshold and taxonomy precedence, while other categories explicitly contain positive advantages. The authors acknowledge these limitations, so I do not regard the presentation as a claim that prefix benefits are universally rare. However, the study leaves the magnitude and robustness of prefix advantages insufficiently established. Direct matched-budget restart measurements on the highlighted candidates would be substantially more informative than additional taxonomy refinements.

Questions for authors
  • The within-problem bootstrap weights a problem sampled w_c times by w_c^2. For inference over sampled problems, why is this preferable to treating each sampled problem copy as a separate cluster and weighting its within-pair contribution by w_c? What population estimate does the quadratic weighting target, and do the conclusions change under linear cluster multiplicities?

  • Can you directly measure restart success at the exact nominal budgets of the ten in-grid extra-batch comparisons, with enough samples to estimate continuation-minus-restart uncertainty? Would the positive aggregate pattern remain?

  • With the same representations, training problems, and problem-disjoint evaluation, does a training objective explicitly targeting within-problem discrimination outperform pooled logistic classification? This would help distinguish an objective mismatch from weak transferable information in the tested representations.

2 comments

Thank you. We completed all three requested analyses and added broader evaluation, a published-probe test and a second generator.

  1. Question-bootstrap multiplicities.

You are right: for inference over questions, each sampled copy should contribute its within-question pairs once. A question drawn w times now contributes its ranking sum and pair count w times. Their ratio retains the pair-weighted target, not an equal-question average. The old implementation merged copies under the original identity, yielding quadratic weights instead of resampling separate question clusters. Linear multiplicities are now primary. Pooled AUROC still compares the full duplicated row population.

With saved baseline reanalysis scores and 2,000 draws, four-token within AUROC remains 0.5155. Its interval changes from [0.4733, 0.5985] to [0.4830, 0.5627]; pooled-minus-within is 0.2455 [0.1129, 0.3438]. All ten W-anchor within intervals include 0.5. Table 4 retains both conventions; intervals condition on fitted scores (Section 2.2; Appendix A.1).

  1. Direct restart measurements and advantage uncertainty.

All ten fixed in-grid candidates received 64 fresh outputs per arm. Restart reasoning budgets are exactly 2,048, 3,072 or 5,120 tokens, matching the archived prefix plus 1,024 continuation tokens; both arms retain a 512-token answer reserve. The new comparison requires no interpolation.

Continuation succeeds on 552/640 outputs versus 434/640 restarts. The equal-candidate mean advantage is +18.44 percentage points, with conservative Hoeffding interval [10.85, 26.03] accounting for sampling in both arms. Eight estimates are positive, one zero and one negative. Figure 1 and Table 11 give all candidate estimates and pointwise Newcombe/Wilson intervals. The positive aggregate pattern remains (Section 3.2; Appendix A.5).

This BF16 Transformers follow-up uses archived 4-bit prefixes and estimates advantage on the ten selected candidates. The original 1/178 count describes a threshold-and-precedence classification, not all positive advantages (Appendix B.3).

Nine prefixes already state the correct answer; the tenth reaches a symbolic sign repair without stating the evaluated answer. Answer reuse is a possible contributor. Grading only new text gives 549/640 versus 434/640, retaining +17.97 points [10.38, 25.56]. Direct credit for old text cannot explain the whole advantage. The model still sees the prefix; separating reuse from repair or further reasoning needs content interventions. The inspection is post-hoc, unblinded and AI-assisted (Section 3.3; Appendix A.5).

  1. Matched objectives on the same representations and training rows.

We held the 22-question panel, question-disjoint folds, training-only preprocessing and regularization fixed across global classification, problem-balanced classification and within-problem pairwise ranking. At four tokens, pairwise-minus-global within AUROC is −0.0064 [−0.0276, 0.0089] on original states and +0.0143 [−0.0014, 0.0355] on BF16 states.

Neither interval establishes an improvement from changing the loss alone in this matched linear-probe setup; a modest BF16 benefit remains possible. This isolates the loss change; it does not establish absence of transferable within-question information. These fits remain separate from the original union-trained probe (Section 2.3, Table 3; Appendix A.2, Tables 5–6).

  1. Broader coverage, a published probe, and a second generator.

BF16 evaluation covers all 128 archive questions and 32,768 attempts, retaining training rows and question-disjoint evaluation. At four tokens, pooled/within is 0.714/0.504: gap 0.210 [0.083, 0.322], with positive gap intervals in all three splits. The early gap extends beyond the selected panel; later contrasts are weaker (Appendix A.2, Table 7).

Zhang et al.'s released Qwen-7B/MATH probe gives archive-weighted pooled/within 0.800/0.643 on 2,185 stratified attempts from 49 mixed-outcome questions. The gap's sampling interval is [0.139, 0.173]; its separate conditional question interval is [0.057, 0.246]. Within-question signal remains. This tests answer-endpoint states on a separate archive with NF4 replay, not an exact replication (Section 2.4; Appendix A.3, Tables 8–9).

We completed 512 Ministral solutions: 32 questions × 16 seeds. At 32 tokens, pooled/within is 0.637/0.522, 0.650/0.545 and 0.711/0.541 for the three objectives. Only eight questions supply within pairs; all gap intervals include zero, and the primary four-token global result is 0.346/0.393. Later problem-balanced and pairwise selection gains are +1.48/+1.53 points, with pointwise intervals excluding zero. We report this variation rather than a persistent cross-model gap (Section 2.5; Appendix A.6, Tables 12–14).

A one-page change map locates these results; the main section order is preserved.

Thank you for the detailed response and additional experiments. The bootstrap correction addresses my concern about cluster multiplicities and the matched-objective comparison helps clarify that changing the training loss alone does not establish an improvement in this setting. The broader BF16 evaluation also strengthens the evidence that the early pooled-versus-within gap extends beyond the original selected panel. However, the fact that nine prefixes already contain the correct answer limits what we can conclude about the source of that advantage. Grading only newly generated text rules out direct credit for the old answer, but does not separate answer reuse from further reasoning, as you acknowledge.

My remaining concern is about the scope of the empirical contribution. The strongest pooled-versus-within separation remains concentrated in the early window of the original model, while the second-generator results are mixed. The published-probe evaluation is useful, but also shows meaningful within-problem discrimination. Together, these results support the value of the proposed evaluation controls, while leaving the broader conclusions about reasoning trajectories less firmly established. Overall, the revision addresses several of my original concerns and makes the evidence more convincing. After considering the additions, I still find the empirical contribution somewhat limited in scope, so I will maintain my rating.

Reviewer_1JQuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines the problems in the interpretation of two measurements of language model reasoning: probe and truncated prefix, in which the pooled prediction and conditional continuation success can make the measurements useless. The authors study two empirical controls: a primary case that compares pooled and within-problem correctness rankings, and a complementary case that compares continuation from a model’s own prefix against a measured restart-budget curve. Evaluation studies cover 178 problem-model cells from 89 MATH problems with Gemma-4 E4B and Ministral-3 3B models.

Critical strengths

The paper examines an important issue in LLM reasoning and challenges common metrics with two empirical controls. The problem is useful and important in LLM reasoning research. The main novelty focuses on the early-window, same-score, same-set comparison with explicit residual positives and evaluation weights.

Critical weaknesses
  • The paper's novelty is limited. It focuses mainly on evaluations, but the evaluation methods show limited novelty, and the generalizability of the results is limited. Writing and formatting need improvement.
  • The two distinct experimental components (i.e., prediction and continuation) are isolated, and they lack connections.
  • The generalizability of the study is limited. The probe used to evaluate correctness depends only on a single model and 22 problems.
  • In the continuation analysis, only one of the 178 cells receives the prefix-limited label, but the authors claim that this is not the prevalence of all prefix benefits. Therefore, the empirical values of prefix advantage should be reported to clarify the claim.
Questions for authors
  • Could the authors further justify and clarify the details of how the 22 mid-band problems are selected? How representative are they?
  • Can the two components be connected to make a unified analysis?
  • Can the AUROC decomposition be generalized beyond within-problem and between-problem comparisons?
2 comments

Thank you. We expanded evaluation, tested a published probe and a second generator, and directly measured prefix advantages.

  1. How were the 22 questions selected, and how representative are they?

The panel contains questions with 14–242 successes among 256 archived attempts, selected before state extraction without inspecting probe scores. It captures 93.76% of archive failures and 84.19% of within-question correct–incorrect pairs. The panel targets within-question ranking, which requires both outcomes, rather than all mathematics questions (Section 2.2).

We expanded BF16 evaluation to all 128 archive questions and 32,768 attempts, preserving the original training rows and excluding held-out questions from fitting. At four tokens, pooled/within AUROC is 0.714/0.504: gap 0.210 [0.083, 0.322], with positive gap intervals in all three fixed splits. The early gap extends to the full archive; later contrasts are weaker (Section 2.3; Appendix A.2, Table 7).

We also completed 512 Ministral solutions: 32 questions × 16 independent seeds. At 32 tokens, pooled/within is 0.637/0.522 for global classification, 0.650/0.545 for problem-balanced classification and 0.711/0.541 for within-problem pairwise ranking. Only eight questions supply within pairs; all gap intervals include zero. The primary four-token global result is 0.346/0.393, showing variation across anchors (Section 2.5; Appendix A.6, Tables 12–14).

  1. What is new beyond proposing evaluation controls?

We quantify the practical consequence: despite pooled AUROC 0.761, within AUROC is 0.515, and best-of-four selection gains only 0.50 percentage points [−0.99, 2.60] over random selection, against oracle headroom of 21.92 points. The three training objectives share rows, folds, preprocessing and regularization. Changing the loss alone did not establish an early ranking improvement; a modest BF16 benefit remains possible (Sections 2.2–2.3, Tables 1–3).

We applied Zhang et al.'s released Qwen-7B/MATH probe (COLM 2025), without retraining, to 2,185 stratified attempts from all 49 eligible mixed-outcome questions. Archive-weighted pooled/within AUROC is 0.800/0.643: gap 0.157, sampling interval [0.139, 0.173], and separate conditional question interval [0.057, 0.246]. Within-question signal remains, but pooled performance overstates it. This uses answer-endpoint states on a separate archive with NF4 replay, not an exact replication (Section 2.4; Appendix A.3, Tables 8–9).

  1. What does “one of 178” mean, and how large are prefix advantages?

The count follows the original rule: success ≥0.75 and advantage ≥0.5 at successive anchors, with instant and budget-limited labels taking precedence. Three non-instant cells cross the primary advantage threshold; four additional instant cells show advantages. Therefore 1/178 is not the prevalence of all beneficial prefixes (Section 3.2; Appendix B.3, Table 15).

All ten fixed in-grid candidates now have 64 fresh outputs per arm at exact nominal comparison budgets. Continuation succeeds on 552/640 outputs versus 434/640 restarts: equal-candidate mean advantage +18.44 points [10.85, 26.03]. Eight estimates are positive, one zero and one negative. Figure 1 and Table 11 report every candidate's rates and intervals. The estimand is the selected-panel advantage, not population prevalence (Section 3.2; Appendix A.5).

Nine prefixes already state the correct answer; the tenth reaches a symbolic sign repair. Answer reuse is therefore a possible contributor. Grading only newly generated text retains +17.97 points [10.38, 25.56], ruling out direct credit for archived text as the sole explanation. The model still sees that text; our post-hoc, unblinded AI-assisted inspection does not isolate reuse, repair or further reasoning (Section 3.3; Appendix A.5).

  1. Can the two components be connected?

Both test whether a success measure supports the intended decision: within-question ranking for choosing a solution, and comparison with restart for deciding whether to retain a prefix at a given budget. We explain this framework before the mathematics, foreground results, and provide a one-page concern-to-result map. The empirical targets remain distinct (Sections 1, 2.1 and 5).

  1. Can the AUROC decomposition extend beyond problems?

Yes. Any specified disjoint partition of correct–incorrect pairs gives a pair-count-weighted sum of ranking averages. Task-family or model-group partitions define other targets; problem identity addresses our selection question. This identity does not itself identify a causal difficulty effect (Section 2.1; Appendix C.3). We also corrected within-bootstrap question multiplicities to be linear and report both old and corrected intervals transparently (Appendix A.1, Table 4).

Thank you for the rebuttal and additional experiments. The new results have addressed some of my concerns, particularly regarding the experimental scale and scope of evaluation, and have strengthened the empirical findings.

However, I still have concerns about the limited novelty and the weak connection between the two components. While the additional experiments improve the paper, they do not fully address these issues, which I believe would require a more substantial revision.

Therefore, I decide to maintain my original rating.

Reviewer_MBZDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines two methodological controls for evaluating LLM reasoning trajectories. First, the authors reanalyze hidden-state probes for predicting final-answer correctness, comparing pooled AUROC across problems with within-problem discrimination among repeated attempts. Second, they investigate the effects of reasoning prefixes by comparing continuation success against restarting with comparable nominal token budgets.

Critical strengths
  • S1. Interesting and meaningful research problem. The paper investigates an important question about how to evaluate LLM reasoning trajectories more comprehensively. Better evaluation methods could benefit applications such as reasoning data selection and the development of techniques for improving LLM reasoning capabilities.

  • S2. Detailed methodology and reproducibility. The authors provide extensive descriptions of their experimental setups, methodological derivations, and implementation details. The additional analyses and documentation demonstrate a strong effort toward transparency and reproducibility.

Critical weaknesses
  • W1. Poor presentation substantially hinders understanding. I generally prioritize scientific merit over presentation and do not consider writing quality a central criterion in evaluating research. However, the writing here is sufficiently confusing that it substantially hinders understanding of the research questions and contributions. The introduction introduces two measurements without adequate conceptual background. Section 2.1 moves directly into mathematical formalism without sufficiently establishing the underlying intuition. Furthermore, the main scientific findings are frequently obscured by extensive methodological and implementation details, many of which could be moved to the appendix. These issues go beyond writing style and make it unnecessarily difficult to evaluate the scientific contribution.

  • W2. Misalignment between the problem formulation and scientific objective. There are two distinct prediction objectives: problem-level correctness prediction, P(Y=1∣Q)P(Y=1\mid Q), and trajectory-conditioned correctness prediction, P(Y=1∣Q,T)P(Y=1\mid Q,T). A probe trained to predict final-answer correctness across problems can legitimately exploit problem difficulty, which is naturally informative about success. Such behavior is not inherently problematic. In fact, the probe is performing its assigned task. If the intended objective is to isolate trajectory-specific information, the training setup should explicitly encourage this distinction, rather than relying primarily on a pooled correctness objective. More importantly, the authors do not sufficiently establish that failure to isolate trajectory-specific information is a prevalent, unresolved limitation of existing probing research, especially since prior approaches already use trajectory representations and some conduct within-problem evaluations. Consequently, the observed performance gap does not clearly establish a novel scientific finding beyond the distinction between these evaluation objectives.

  • W3. Insufficient consideration of prefix quality limits scientific insight and potential applicability. The usefulness of a reasoning prefix plausibly depends heavily on its content and quality. A prefix containing misleading reasoning may harm subsequent performance, whereas one representing meaningful progress could substantially reduce the remaining reasoning effort. However, the authors truncate individual model-generated trajectories at different positions without systematically examining the correctness, quality, or reasoning progress represented by those prefixes. Consequently, although the experiments measure continuation success relative to restart references, they provide limited insight into what makes a prefix beneficial or harmful. Without such characterization, it is also unclear how these findings could generalize across reasoning trajectories or inform meaningful evaluations of the reasoning generated so far.

Questions for authors

Please see the Weaknesses section.

2 comments

Thank you. The revised paper connects the evaluation target to its practical use: choosing among solutions to the same question. We test whether pooled correctness prediction supports that decision and whether changing the training objective closes the gap.

  1. W2: Contribution and relevance to prior work.

Across-question prediction is valid, but its success does not establish within-question ranking. On identical scores and attempts, our early probe gives pooled/within AUROC 0.761/0.515. Choosing the highest-scored of four attempts gains 0.50 percentage points over random selection [−0.99, 2.60], despite oracle headroom of 21.92 points. This connects the evaluation gap to limited observed selection benefit (Section 2.2, Tables 1–2).

We also evaluated Zhang et al.'s released Qwen-7B/MATH probe (COLM 2025), without retraining, on 2,185 stratified attempts from all 49 eligible mixed-outcome questions. Restoring archive row and pair weights gives pooled/within AUROC 0.800/0.643: gap 0.157, fixed-archive sampling interval [0.139, 0.173], and separate question-bootstrap interval [0.057, 0.246]. The probe retains within-question signal, but pooled AUROC overstates that discrimination. This tests a published probe on a separate repeated-attempt archive with NF4 replay, rather than exactly replicating its original experiment (Section 2.4; Appendix A.3, Tables 8–9).

We acknowledge prior within-problem full-trace work, including Yuan et al. (2026). Our contribution is empirical evidence connecting the pooled–within discrepancy to solution selection, matched objective tests, and evaluation of a published probe (Sections 1 and 4).

  1. W2: Does training for the intended objective help?

We compared global classification, problem-balanced classification and within-problem pairwise ranking, holding rows, question-disjoint folds, preprocessing and regularization fixed within each representation condition. At four tokens, pairwise-minus-global within AUROC is −0.0064 [−0.0276, 0.0089] on original states and +0.0143 [−0.0014, 0.0355] on BF16 states. Neither interval establishes an improvement from changing the loss alone; a modest BF16 benefit remains possible. These matched linear-probe fits use the 22-question panel and are separate from the original union-trained probe (Section 2.3, Table 3; Appendix A.2, Tables 5–6).

  1. Coverage beyond the original case.

Expanding BF16 evaluation to all 128 archive questions and 32,768 attempts gives pooled/within AUROC 0.714/0.504 at four tokens: gap 0.210 [0.083, 0.322] (Table 7).

We also completed 512 Ministral attempts: 32 fixed questions × 16 independent seeds. At 32 tokens, pooled/within is 0.637/0.522, 0.650/0.545 and 0.711/0.541 for the three respective objectives. This gives directional evidence at the later anchor, with positive selection gains for problem-balanced and pairwise training. Only eight questions supply within pairs, all gap intervals include zero, and the primary four-token global result is 0.346/0.393. We report this variation explicitly (Section 2.5; Appendix A.6, Tables 12–14).

  1. W3: Prefix content and its relation to benefit.

We inspected all ten fixed prefixes for answer availability, derivations, errors and repair. Nine already state the correct answer; the tenth reaches a symbolic sign repair without stating the evaluated answer. Continuation can therefore reuse information, including answers, already present at truncation: a concrete possible explanation for the observed benefit.

We separately checked whether the grader merely credited the old prefix. With 64 fresh outputs per arm at exact nominal restart budgets, the mean advantage is +18.44 points [10.85, 26.03]. Grading only newly generated text retains +17.97 points [10.38, 25.56]. The benefit survives this grading control; answer reuse by the model remains possible. The post-hoc, unblinded AI-assisted audit characterizes prefix content, but separating reuse, repair and further reasoning or establishing a general quality–benefit relationship requires content interventions (Sections 3.2–3.3; Appendix A.5, Table 11).

  1. W1: Presentation and connection between the studies.

We now explain both decisions before the formalism, foreground results, and shorten historical audits while retaining their records separately. The main section order is preserved; a one-page change map identifies each concern, result and location. Both studies test the comparison required for the intended decision: within-question selection for probes, and budget-indexed restart comparisons for prefixes (Sections 1 and 5).

I thank the authors for the detailed response. In details,

W1: Thanks for the updated file. However, I still think the paper's presentation can be improved. A good introduction that contextualizes the research background and the rationale behind the Section 2 measures can be built to better formalize the problem and situate the work among its peers. The updated version partially addresses my concern.

W2: I think my questions are more related to the probe training setup. In the paper, the problem difficulty is considered a confound. As I understand it, the authors intend to use the probe to capture and model P(Y=1∣Q,T)P(Y=1| Q, T), but without sufficient information about the trajectory, the aggregated measures encourage P(Y=1∣Q)P(Y=1| Q). In this setup, I think the problem's difficulty becomes a legitimate and unavoidable consequence of this optimization goal; therefore, it is a phenomenon invited by the design rather than a byproduct. On that basis, I think the author's responses do not address my concern.

W3: In this area of weakness, I argue that the characterization of the reasoning prefix needs to be evaluated more carefully. A twisted, long reasoning path does not necessarily indicate a more advanced reasoning process. I think my concern is not addressed by the response.

Based on the current version and the author responses, I choose to maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.