acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Scaling via Error Localization

Abstract

Scaling inference-time computation has emerged as a reliable method to improve the performance of large language models on complex reasoning and programming tasks. However, standard approaches such as independent sampling and sequential multi-turn refinement operate without token-level credit assignment, resulting in computational inefficiency, since valid reasoning prefixes are frequently discarded. In this work, we introduce Test-Time Scaling via Error Localization (TTEL), an inference-time algorithm that utilizes fixed or environment feedback to perform token-level error localization. By comparing conditional probabilities under informed feedback against a non-diagnostic baseline, TTEL isolates the step at which an error occurred. The algorithm then truncates the trajectory and branches a new generation, maximally reusing the valid prefix. Extensive evaluations demonstrate that TTEL establishes strictly dominating Pareto frontiers across sequential reasoning domains, measured by pass-at-k vs. generated-token cost. With Qwen3-8B on LiveCodeBench, TTEL attains a pass@64 of 71.0% while generating approximately half as many tokens as independent sampling (360.4k vs. 735.0k). Generalizing to math benchmarks AIME-2025 and HMMT-2025, TTEL cleanly outperforms competing test-time baselines across both Qwen3-8B and Qwen3-4B-Thinking-2507.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-time scaling, Error Localization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.17 over 6 reviews, above 49% of the others.
Reviewer_HxepOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Test-Time Scaling via Error Localization (TTEL), an inference-time search algorithm that aims to improve reasoning efficiency by identifying potential error locations and reusing prefixes of previously generated trajectories. TTEL compares token probabilities under feedback-conditioned and non-diagnostic contexts to identify candidate error locations, then branches from the selected position to regenerate the remaining trajectory. Experiments on LiveCodeBench V6, AIME-2025, and HMMT-2025 demonstrate promising improvements in pass@k and token efficiency over independent sampling and multi-turn refinement baselines.

Critical strengths

S1. Fine-grained error localization from trajectory-level feedback.

The paper presents a promising approach to deriving token-level error localization signals from trajectory-level feedback. By comparing token probabilities under different feedback conditions, TTEL identifies potential error locations within failed reasoning trajectories and uses them to guide prefix-based branching. This provides a principled way to leverage coarse-grained feedback for fine-grained test-time search without additional model training.

S2. Effectiveness under limited diagnostic feedback.

An interesting finding is that TTEL achieves improvements even with generic feedback rather than detailed environment-generated error messages. The results on AIME-2025 and HMMT-2025 suggest that feedback-conditioned probability shifts can provide useful signals for guiding search, even when the feedback does not explicitly identify the underlying reasoning errors. This broadens the potential applicability of the approach beyond environments with rich execution feedback.

Critical weaknesses

W1. Insufficient validation of the reliability and interpretability of error localization.

The central premise of TTEL is that feedback-induced token probability shifts can reveal meaningful error locations. However, the paper does not provide a quantitative evaluation of localization accuracy against actual reasoning errors, making it difficult to assess how reliably the selected branching points identify erroneous reasoning steps. While Figure 8 presents a qualitative example and Appendix I analyzes branching behavior, neither directly validates the accuracy of error localization.

Furthermore, the interpretation of these probability shifts remains insufficiently justified. A token's probability may decrease under feedback because the model prefers a different expression or reasoning path, rather than because the original token represents an actual mistake. Although the proposed baseline filtering mitigates certain context-induced effects, it does not establish that the remaining spikes correspond to genuine reasoning errors. These concerns weaken the evidence supporting the proposed localization mechanism.

W2. Unclear necessity of the proposed error localization mechanism and missing comparisons with alternative approaches.

The paper demonstrates the effectiveness of TTEL as a complete framework, but does not sufficiently establish whether its specific error localization mechanism is necessary for the observed improvements. In particular, simpler branching strategies, such as random or fixed-position branching, could help isolate the contribution of feedback-guided localization from the benefits of prefix reuse.

Moreover, process reward models (PRMs) can also provide intermediate correctness signals and potentially guide branching from erroneous reasoning steps. Although PRMs are discussed in Appendix A, the paper does not include an empirical comparison with PRM-guided localization and branching. Such a baseline would be particularly valuable for assessing the accuracy, effectiveness, and practical trade-offs of TTEL's training-free localization strategy. Convincing evidence from these comparisons would substantially strengthen the paper, and I would consider increasing my score accordingly.

W3. Limited analysis of iterative refinement effectiveness compared with conventional multi-turn refinement.

Figure 4 compares TTEL and multi-turn refinement across different rollout budgets, while Appendix I analyzes TTEL's branching behavior across successive turns. However, these results do not directly reveal how their correction capabilities differ over consecutive refinement iterations.

In particular, it remains unclear whether TTEL is more effective at correcting an initially unsuccessful attempt after one or multiple refinement steps, and how this advantage evolves with additional iterations. A direct comparison of per-iteration correction success rates, starting from the same failed trajectories, would provide stronger evidence for the effectiveness of the proposed refinement mechanism.

W4. Insufficient analysis of the restart strategy and its contribution to search effectiveness.

TTEL triggers a restart when no valid localization spike is detected or the selected branching point exceeds the depth limit. While Appendix I provides some analysis of successive refinement turns after a restart, the paper does not clearly characterize how frequently restarts occur or how they contribute to the overall search performance.

In particular, when valid spikes cannot be identified, TTEL falls back to generating fresh trajectories from the original prompt, potentially behaving similarly to independent sampling. It remains unclear how much of the reported improvement comes from successful prefix-based refinement versus restarting and exploring new trajectories. Further analysis of restart frequency, refinement depth, and the respective contributions of these two operations would help establish the effectiveness of the proposed search strategy.

Questions for authors

Q1. Reliability and interpretability of error localization.

Could the authors provide a quantitative evaluation of error localization accuracy against annotated or synthetically introduced reasoning errors? For example, measuring localization accuracy or the distance between selected branching points and actual error positions would help establish the reliability of the proposed mechanism.

Additionally, could the authors clarify how genuine reasoning errors are distinguished from feedback-induced probability shifts caused by alternative valid reasoning paths or changes in expression?

Q2. Necessity of error localization and comparisons with alternative approaches.

Could the authors compare TTEL with simpler branching strategies, such as random or fixed-position branching, under matched inference budgets? This would help isolate the contribution of the proposed localization mechanism from the benefits of prefix reuse.

More importantly, a comparison with PRM-guided error localization and branching would be particularly valuable. It would help establish whether TTEL provides more accurate localization, better downstream performance, or a more favorable trade-off between effectiveness and computational cost.

Q3. Refinement effectiveness and restart behavior.

Could the authors provide a direct comparison of TTEL and conventional multi-turn refinement across consecutive refinement iterations, ideally starting from the same unsuccessful trajectories? Reporting correction success rates after one, two, or more refinement steps would help clarify the benefits of the proposed approach beyond the existing pass@k results.

Additionally, could the authors report the frequency of restarts, the distribution of consecutive refinement turns before each restart, and the relative contributions of prefix-based refinement and fresh exploration to overall search success? These analyses would help explain how the two components interact and whether the restart strategy effectively allocates the inference budget.

2 comments

Thanks for your helpful review, and for acknowledging the strengths of our work. We'll address your concerns point-by-point. We hope these responses are satisfactory and would be grateful if you would consider increasing your score.

W1 and Q1: Insufficient validation of the reliability and interpretability of error localization

Effective recovery does not require pinpointing the exact point where reasoning fails; branching at or before the earliest error is sufficient to prevent flawed reasoning from propagating.

To evaluate this directly, we take one independently sampled failed trajectory per question from Qwen3-8B across LCBv6, AIME-25, and HMMT-25, and use gemini-4-argon as an annotator to locate the earliest reasoning error (135 successfully annotated trajectories). TTEL's branching prefixes precede the error in 71.1% of trajectories, compared to 47.4% for uniform random (+23.7 percentage points):

MethodAll (n = 135)Code (n = 97)Math (n = 38)
Uniform Random47.4%49.0%43.6%
TTEL71.1%73.2%65.8%

This advantage holds consistently across both code (+24.2 pp) and math (+22.2 pp), confirming that TTEL reliably identifies clean prefixes before flawed reasoning sets in.

More details can be found here: [Error localization accuracy and interpretation (Appendix J.3, Figure 12)]


W2 and Q2: Unclear necessity of the proposed error localization mechanism and missing comparisons with alternative approaches

To isolate the effect of error localization, TTEL (random prefix depth) selects a branching token uniformly at random from the latest rollout (excluding any prior prefix), while keeping the search tree, feedback prompt, and restart dynamics identical to TTEL. We evaluate this ablation with Qwen3-8B on LiveCodeBench v6 (LCBv6) and HMMT-25 up to k = 64 rollouts (1 seed) against standard TTEL (3 seeds on LCBv6, 4 on HMMT-25):

Method (Qwen3-8B, k = 64)LCBv6 Pass@64LCBv6 TokensHMMT-25 Pass@64HMMT-25 Tokens
TTEL (random prefix depth)68.7%423k53.3%637k
TTEL71.0 ± 1.6%360k60.8 ± 3.6%528k

Across both benchmarks, TTEL achieves higher Pass@64 (+2.3 pp on LCBv6, +7.5 pp on HMMT-25) while using 15%–17% fewer tokens, confirming that targeted error localization improves both accuracy and token efficiency.

More details can be found here: [Random branching (Appendix J.3, Figure 13)]

W3 and Q3: Limited analysis of iterative refinement effectiveness compared with conventional multi-turn refinement.

Truncating at the selected spike position keeps valid upstream steps and drops the flawed suffix instead of conditioning on a wrong final answer. On the first branch from a failed attempt, TTEL recovers 40.6%, 18.5%, and 9.9% of trajectories on Qwen3-8B, and 58.5%, 33.2%, and 7.8% on Qwen3-4B-Thinking (AIME-25, HMMT-25, LCBv6). By the second consecutive branch, recovery falls to 1.9%–6.4% (8B) and 1.6%–17.3% (4B; full table in Q3 Part 2), where spike filtering restarts cleanly before errors compound.

Multi-Turn (Answer Only) discards prior reasoning each round, whereas our newly evaluated Multi-Turn (Full Reasoning) baseline on Qwen3-8B, run with the same feedback and thought traces as TTEL, appends failed traces until the context window fills and progress stalls after early turns. By k = 16, TTEL already matches or beats both k = 64 Multi-Turn variants across all three benchmarks with fewer tokens (under half of Answer Only), and keeps gaining through k = 64:

Method (Qwen3-8B)LCBv6 Pass@k (Tokens)AIME-25 Pass@k (Tokens)HMMT-25 Pass@k (Tokens)
TTEL (k = 16)63.6 ± 0.4% (96k)81.7 ± 1.7% (132k)54.2 ± 3.6% (147k)
TTEL (k = 64)71.0 ± 1.6% (360k)83.3 ± 0.0% (440k)60.8 ± 3.6% (528k)
Multi-Turn (Answer Only, k = 64)56.7 ± 0.7% (323k)76.7 ± 3.3% (342k)54.2 ± 2.8% (321k)
Multi-Turn (Full Reasoning, k = 64)55.7 ± 0.6% (191k)73.3 ± 4.1% (194k)50.0 ± 4.7% (195k)

More details can be found here: [Iterative refinement effectiveness vs. Multi-Turn across successive steps (Appendices J.4 & J.8, Figure 14, Tables 11, 13, and 14)]


W4 and Q3: Insufficient analysis of the restart strategy and its contribution to search effectiveness.

Across k = 64 turns, TTEL restarts on 27%–41% of post-Turn-0 turns; baseline filtering (no surviving spike) triggers most restarts, and only a small share hit the D_c = 16,384 prefix cap. Recovery concentrates on the first branch and drops steeply on the second, so chains stay at 1–2 branches per root:

ModelDatasetRestarts / 64Branches / RootD_c CutoffRecovery (1st → 2nd)First Solves on BranchBranch-Only Solves (N=1 / N≥2)
Qwen3-8BLCBv625.51.330.7%9.9% → 1.9%72.8% (83/114)14.0 / 1.0 (of 131)
Qwen3-8BAIME-2525.61.036.7%40.6% → 6.4%88.4% (38/43)2.5 / 0.5 (of 30)
Qwen3-8BHMMT-2524.11.277.6%18.5% → 2.2%89.7% (35/39)3.8 / 0.2 (of 30)
Qwen3-4B-ThinkingLCBv618.22.154.1%7.8% → 1.6%62.9% (66/105)11.7 / 0.7 (of 131)
Qwen3-4B-ThinkingAIME-2520.21.6313.9%58.5% → 17.3%98.4% (61/62)9.0 / 0.0 (of 30)
Qwen3-4B-ThinkingHMMT-2516.91.8714.8%33.2% → 6.7%95.2% (59/62)9.2 / 0.5 (of 30)

Post-Turn-0 gains come primarily from prefix branching: 80.5% (342/425) of post-Turn-0 first solves occur on branched turns, which on math reach a 5x–29x higher per-turn solve rate than restarts at roughly half the generated tokens per turn. Unbranched restarts alone cannot exceed Independent Pass@64 (61.7% vs. TTEL's 85.0% on 4B AIME-25; 47.5% vs. 69.2% on 4B HMMT-25; 64.6% vs. 71.0% on 8B LCBv6), and Branch-Only Solves counts problems per seed where Turn 0 and all restarts fail.

Restarts still help when an error precedes truncating at the selected spike position or the initial algorithm fails (as on LCBv6, where switching algorithms needs a clean slate) and seed new roots for later branches; never restarting (Multi-Turn Full Reasoning) trails TTEL by 10–15 points. Capping depth earlier at N = 1 or N = 2 branches on Qwen3-8B gives similar Pass@64 across all three benchmarks but costs 9%–34% more generated tokens, since baseline filtering already restarts once a prefix stops yielding spikes.

More details can be found here: [Restart frequency, refinement depth, and branching vs. restarting (Appendix J.8, Tables 13 and 14)]

Reviewer_hWSGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTEL, a training-free framework that improves inference-time reasoning by identifying potential errors in generated trajectories and selectively regenerating the subsequent steps. Specifically, TTEL compares token-level probabilities under error feedback and non-diagnostic feedback to identify positions with significant probability shifts. It then preserves the reasoning prefix before the identified error position and regenerates the remaining trajectory, reducing redundant computation. Experiments on mathematical reasoning and code generation benchmarks demonstrate improvements in both reasoning accuracy and token efficiency over existing test-time scaling baselines.

Critical strengths
  1. In general, the presentation of the paper is clear. The method is well illustrated and the algorithm is concisely specified.
  2. The proposed method introduces an interesting approach to test-time scaling by leveraging feedback-induced changes in token-level probabilities to identify potential reasoning errors and guide subsequent generation.
  3. The method consistently improves pass@k and token efficiency over the evaluated baselines across mathematical reasoning and code generation benchmarks.
Critical weaknesses
  1. A major concern is that all empirical results are reported in terms of pass@k, which only measures whether at least one correct solution exists among the generated candidates, rather than whether the method can identify and select the correct answer. This may overestimate the practical effectiveness of TTEL, especially in mathematical reasoning tasks where an external verifier is not readily available.
  2. While TTEL improves pass@k, the paper does not directly validate its central claim that feedback-induced probability shifts can reliably localize reasoning errors. The existing evidence is primarily based on end-to-end performance (Figs. 4–6) and qualitative examples (Fig. 8). A quantitative evaluation comparing the identified error positions against ground-truth error locations would provide stronger support for the proposed mechanism. In addition, ablations that compare the proposed error localization strategy with random branching under a matched branching-depth distribution would help clarify whether the improvements stem from accurate error localization or simply from prefix reuse.
  3. The experiments are conducted only on two relatively small models from the Qwen family, leaving the generalizability of the method to larger models and other model families unclear. Moreover, the mathematical benchmarks, AIME-2025 and HMMT-2025, each contain only 30 problems, which limits the robustness of the empirical conclusions.
  4. The baselines considered in this paper are relatively weak. The authors are encouraged to compare against other training-free, search-based approaches, such as Tree-of-Thoughts and MCTS-based reasoning methods such as RAP. In addition, comparisons with explicit self-reflection methods, such as Self-Refine, would help provide a more comprehensive evaluation.
Questions for authors
  1. Table 1 reports a pass@16 of 63.6% for TTEL on LiveCodeBench with Qwen3-8B, while Table 7 reports 65.9% under the same setting (τ=0.06\tau=0.06, k=16k=16). Could the authors clarify the reason for this discrepancy?
  2. The average rollout lengths for independent sampling with Qwen3-4B-Thinking-2507 are close to the maximum generation length of 16,384 tokens (approximately 14.5k on AIME and 15.4k on HMMT). This raises concerns about potential truncation effects. Since TTEL continues from a retained prefix and can exceed this limit, truncation may favor TTEL. Could the authors report the proportion of rollouts that reach the length limit and investigate whether increasing the maximum generation length affects the relative performance of TTEL and the baselines?
4 comments

Thanks for the thorough review and helpful suggestions. We have added Pass@1 selection methods, error-localization and random-branching ablations, a non-Qwen model family (OLMo-3-7B-Think at 28k generation token budget), and stronger baselines in Appendix J of the revised PDF. We hope this addresses your concerns and would greatly appreciate it if you could reconsider your score.

W1: All empirical results report Pass@k rather than single-answer selection accuracy, which may overestimate practical effectiveness when no external verifier is available.

  • We focus on scaling pass@k so that at least one correct answer is produced, after which selection methods can be applied to pick a single final answer. This contrasts with methods like Recursive Self-Aggregation that mainly bridge the gap from pass@1 to pass@k without solving new problems that independent sampling misses.
  • We evaluate Pass@1 using two selection methods over the k=64 candidate rollouts (Appendix J.5, Figure 15, Table 12). Majority voting picks the most frequent final answer, which is applicable for math datasets. The LLM-Judge is the same model asked to pick the best of four candidates, repeated over three rounds in a tournament style (64, 16, 4, 1 selected answers per round).
  • TTEL has the highest accuracy with majority voting across all settings. For LLM-Judge selection, it is the best in 4 of 6 cases and within 2 points in the other two.
ModelDatasetMaj. Indep.Maj. Multi-TurnMaj. TTELJudge Indep.Judge Multi-TurnJudge TTEL
Qwen3-8BLCBv6N/AN/AN/A51.949.453.4
Qwen3-8BAIME-2570.874.278.370.072.575.8
Qwen3-8BHMMT-2545.846.749.243.345.844.2
Qwen3-4B-ThinkingLCBv6N/AN/AN/A53.951.953.4
Qwen3-4B-ThinkingAIME-2561.776.783.360.077.580.0
Qwen3-4B-ThinkingHMMT-2540.844.248.340.843.350.0

Reviewer Query

W2: Validate whether selected branch points actually correspond to ground-truth errors, and compare against random branching to separate localization from prefix reuse.

  • Error Localization vs. Branching Before Errors (Appendix J.3, Figure 12): We would like to state that precise error localization is not expected, but branching at or before an error is sufficient to avoid propagating flawed reasoning.

We take a failed independently sampled trajectory per question for Qwen3-8B across LCBv6, AIME-25, and HMMT-25. We use a frontier LLM annotator that locates the earliest reasoning error, which it does successfully in 135 trajectories. Branching prefixes for TTEL lie before the error in 71.1% of the traces versus 47.4% for uniform random:

MethodLCBv6 (n = 97)AIME-25 (n = 19)HMMT-25 (n = 19)
Uniform Random49.0%39.6%47.6%
TTEL73.2%68.4%63.2%
  • Random-branch control (Appendix J.3, Figure 13): TTEL (random prefix depth) selects a branching token uniformly at random from the latest rollout (excluding any prefix). The search tree, feedback prompt, and restart dynamics are identical to TTEL. We evaluated this baseline with Qwen3-8B over LiveCodeBench V6 and HMMT-25 up to k = 64 rollouts (1 seed) and compare it with standard TTEL (3 seeds on LCBv6, 4 on HMMT-25):
Benchmark (Qwen3-8B, k = 64)TTEL Pass@64TTEL TokensTTEL (random prefix depth) Pass@64TTEL (random prefix depth) Tokens
LCBv671.0 ± 1.6%360k68.7%423k
HMMT-2560.8 ± 3.6%528k53.3%637k

On both benchmarks, TTEL reaches higher Pass@64 while using 15% to 17% fewer tokens.

Reviewer Query

W3: Experiments cover only two small Qwen models (leaving other model families and larger scales untested), and AIME-25 / HMMT-25 each contain 30 problems.

  • Model families (Appendix J.6, Figure 16): We evaluated OLMo-3-7B-Think at a 28k-token generation cap up to k=64 rollouts (4 seeds). Applying a strict cutoff of 28k tokens for the prefix and rollout, TTEL still outperforms Multi-Turn refinement and matches Independent Sampling at roughly half the generated tokens:
BenchmarkTTEL Pass@64 (Tokens)TTEL Strict <=28k (Tokens)Multi-Turn Pass@64 (Tokens)Independent Pass@64 (Tokens)
AIME-2587.5 ± 4.3% (588k)86.7 ± 3.3% (588k)84.2 ± 1.4% (544k)86.7 ± 0.0% (1,202k)
HMMT-2555.8 ± 2.8% (640k)55.8 ± 2.8% (640k)50.8 ± 1.4% (563k)56.7 ± 2.4% (1,337k)
  • Larger models: We are running a larger model Qwen3-14B and will report back with the numbers soon.
  • Regarding math sets: AIME-25 and HMMT-25 have 30 problems each. We have chosen 2025 splits to avoid pretraining contamination and have run 4 seeds per method on these math benchmarks.

Reviewer Query

W4: Compare against stronger training-free search and self-reflection baselines (such as Self-Refine, Tree-of-Thoughts, and RAP).

We evaluated Self-Refine (32 solver attempts with critique tokens included) and Power Sampling (16 MCMC chains, α = 4.0) on Qwen3-8B. TTEL outperforms both in Pass@k, even at k = 16 with far fewer generated tokens (Appendix J.4, Figure 14, Table 11):

Method (Qwen3-8B)LCBv6 Pass@k (Tokens)AIME-25 Pass@k (Tokens)HMMT-25 Pass@k (Tokens)
TTEL (k = 16)63.6 ± 0.4% (96k)81.7 ± 1.7% (132k)54.2 ± 3.6% (147k)
TTEL (k = 64)71.0 ± 1.6% (360k)83.3 ± 0.0% (440k)60.8 ± 3.6% (528k)
Self-Refine (k = 32)60.3% (351k)80.0% (397k)53.3% (374k)
Power Sampling (k = 16)N/A63.3% (1,833k)46.7% (2,009k)

Reviewer Query

Q1: Table 1 reports a Pass@16 of 63.6% for TTEL on LiveCodeBench with Qwen3-8B, while Table 7 reports 65.9%. Why do they differ?

LocationPass@16SeedsFeedback ModePurpose
Figure 50.6304GenFBShorter ablations at k=16
Figure 50.6584EnvFBShorter ablations at k=16
Table 163.6 ± 0.4%3GenFBScaling run up to k=64 (3 seeds used given low variance and 4x higher cost)
Table 7 (τ=0.06)65.9 ± 1.3%3EnvFBSpike threshold τ sweep (matches Figure 5's EnvFB bar)

Figure 5 and Table 1 share identical GenFB results on seeds 0 to 2, differing only by Figure 5's fourth seed. We have added TTEL-EnvFB to the Table 7 and Table 8 captions (Appendix H) and aligned Table 7 to population SD. Thank you for bringing this up.

Reviewer Query

Q2: Qwen3-4B-Thinking trajectories on math approach 16k tokens; report truncation rates and test a larger generation cap.

  • TTEL with a strict 16k token cap (Appendix J.1, Figure 11, Table 9): Although TTEL allows branching such that the total depth of the prefix and continuation can reach ~32k tokens, it rarely exceeds ~16k tokens in practice. Retrospectively scoring every trajectory over 16,384 total tokens and its successor branches as failures (while still charging their full token cost) reduces mean TTEL Pass@64 across all six settings only from 73.1% to 71.5%:
ModelDatasetTTEL Pass@64Strict <=16k Pass@64Best Baseline Pass@64
Qwen3-8BLCBv671.0 ± 1.6%70.7 ± 1.3%64.6% (Indep.)
Qwen3-8BAIME-2583.3 ± 0.0%82.5 ± 1.4%76.7% (Multi-Turn)
Qwen3-8BHMMT-2560.8 ± 3.6%59.2 ± 4.9%55.8% (Indep.)
Qwen3-4B-ThinkingLCBv669.5 ± 0.6%69.2 ± 0.4%67.4% (Indep.)
Qwen3-4B-ThinkingAIME-2585.0 ± 1.7%83.3 ± 2.4%80.0% (Multi-Turn)
Qwen3-4B-ThinkingHMMT-2569.2 ± 3.6%64.2 ± 6.4%47.5% (Indep.)
  • Truncation rates (Appendix J.2, Table 10): Multi-Turn refinement and TTEL hit the 16k generation cap far less often than Independent Sampling, particularly on math (% of attempts hitting 16k):
Method8B LCBv68B AIME-258B HMMT-254B LCBv64B AIME-254B HMMT-25
Independent Sampling42.9%50.4%62.3%39.0%64.2%80.0%
Multi-Turn5.2%11.8%9.2%1.3%1.7%1.9%
TTEL14.6%20.8%27.4%10.1%27.4%31.7%
  • Evaluation at a 28k generation cap on OLMo-3-7B-Think (Appendix J.6, Figure 16): We evaluated OLMo-3-7B-Think at a 28k-token generation cap up to k=64 rollouts (4 seeds). Applying a strict cutoff of 28k tokens for the prefix and rollout, TTEL still outperforms Multi-Turn refinement and matches Independent Sampling at roughly half the generated tokens:
BenchmarkTTEL Pass@64 (Tokens)TTEL Strict <=28k (Tokens)Multi-Turn Pass@64 (Tokens)Independent Pass@64 (Tokens)
AIME-2587.5 ± 4.3% (588k)86.7 ± 3.3% (588k)84.2 ± 1.4% (544k)86.7 ± 0.0% (1,202k)
HMMT-2555.8 ± 2.8% (640k)55.8 ± 2.8% (640k)50.8 ± 1.4% (563k)56.7 ± 2.4% (1,337k)

All new tables and figures are included in Appendix J of the revised PDF. We are happy to clarify any remaining questions.

Reviewer_ikabOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTEL, an inference-time scaling method that uses the token-level distribution to locate the issue after a failed attempt. TL;DR: after a failed attempt, the same model re-scores its full trajectory (the reasoning trace plus the final answer) twice: (1) with the original context plus a "not successful / has issue" feedback, and (2) with a "just repeat it" instruction that gives no hint. The authors then check each generated token: if its probability drops under (1) but not under (2), and the argmax token under (1) differs from the original token, the token is regarded as a potential wrong point. TTEL cuts off the trajectory at the strongest such point and regenerates from there. If TTEL does not find any potential wrong point, it restarts and regenerates from scratch. The authors use Qwen3-8B and Qwen3-4B-Thinking-2507 to test this method, and they claim a pass@64 of 0.71 on LiveCodeBench with about half the tokens of independent sampling. Their ablation also shows that the baseline filter reduces the Turn-1 spikes from 486 to 19.3.

Critical strengths
  1. The idea is novel, and I would give a strong recommendation on it. The paper uses the token-level distribution of the same model as an error signal at inference time, without any training or extra model, which saved a lot of cost and that's pretty cool. Comparing the feedback context with a non-diagnostic baseline is a sensible design. Without this baseline, the Turn-1 spikes rise from 19.3 to 486.0 and pass@16 drops from 0.630 to 0.592 (Figure 6). So the contrast itself carries useful information.
  2. The token efficiency is clear on LiveCodeBench. At k=64, TTEL reaches a pass@64 of 0.71 with 360.4k generated tokens, while independent sampling reaches 0.646 with 735.0k. From an industrial view, saving half of the generated tokens is a real gain.
Critical weaknesses
  1. Formulation: I am not convinced that the signal locates the error. (a) The branch point is t* = argmax g_t, which finds the largest probability drop, not the first error. If the goal is to trace where the reasoning first goes wrong, the earliest qualifying token would match the goal better. The paper does not explain this choice or compare the two. (b) In the GenFB setting, the baseline prompt still contains the "unsuccessful" feedback; only the instruction changes to "repeat the previous attempt word by word" (Appendix B). So g_t measures "redo" vs. "repeat", not informative vs. non-diagnostic feedback as the paper states.

  2. Experiment: I cannot tell where the gain comes from, and the method has no fallback when it is stuck. (a) There is no control that branches at a random position, or at a fixed fraction of the trace, with the same tree and restart logic. Without it, I cannot tell whether the gain comes from redoing the generation at the right point, or purely from reusing the prefix and its prefill. (b) Once any spike is found, TTEL keeps the prefix before t*. If the real error is earlier, every later branch inherits it. Restart only happens when no spike is found, because the depth cap D_c = 16,384 equals the generation cap, so the condition t* <= D_c in Algorithm 1 never triggers. Figure 10 shows the correction rate decreases above 8k checkpoint depth, which is consistent with this concern.

To convince me, I would like to see: (i) the random-branch control at matched tokens; (ii) statistics on how many branches happen on the same root before a restart, and whether a forced restart after N branches helps; (iii) a multi-turn baseline with the full trace. If TTEL still wins under these controls, I would raise my rating.

Questions for authors

Clarifications that could change my assessment:

  1. Random-branch control (Weakness 2a). If you branch at a random position, or at a fixed fraction of the trace, with the same tree, feedback and restart logic, does TTEL still win at matched generated tokens? This is the main question for me, because it separates the effect of locating the error from the effect of reusing the prefix and its prefill.

  2. Choice of t* (Weakness 1a). Why do you use t* = argmax g_t instead of the earliest qualifying token in E_filtered? If the goal is to find where the reasoning first goes wrong, the earliest token seems a better match. Could you compare the two?

  3. Baseline prompt (Weakness 1b). In GenFB, the baseline prompt still contains "Your previous attempt was unsuccessful"; only the instruction changes. What happens if the feedback is replaced by "Your previous attempt was successful"? If the spikes stay, the signal mainly comes from the redo instruction, not from the error.

  4. Getting stuck on a wrong prefix (Weakness 2b). How many branches happen on the same root before a restart, and what fraction of questions never restart? Since D_c = 16,384 equals the generation cap, is the depth condition in Algorithm 1 ever triggered? Does a forced restart after N branches help, especially for checkpoints deeper than 8k tokens, where Figure 10 shows the correction rate decreases?

3 comments

Thank you for the thorough review. We ran each requested experiment and added the new tables and plots to Appendix J of the revised PDF. We hope these clarifications and additional results address your concerns. If so, we would be grateful if you would consider raising your score

Reviewer Query

(i) / Q1 / W2a: Add a random-branch control keeping the rest of TTEL fixed.

  • Random-branch control (Appendix J.3, Figure 13): TTEL (random prefix depth) selects a branching token uniformly at random from the latest rollout (excluding any prefix). The search tree, feedback prompt, and restart dynamics are identical to TTEL. We evaluated this baseline with Qwen3-8B over LiveCodeBench V6 and HMMT-25 up to k = 64 rollouts (1 seed) and compare it with standard TTEL (3 seeds on LCBv6, 4 on HMMT-25):
Benchmark (Qwen3-8B, k = 64)TTEL Pass@64TTEL TokensTTEL (random prefix depth) Pass@64TTEL (random prefix depth) Tokens
LCBv671.0 ± 1.6%360k68.7%423k
HMMT-2560.8 ± 3.6%528k53.3%637k

On both benchmarks, TTEL reaches higher Pass@64 while using 15% to 17% fewer tokens.

Reviewer Query

(iii): Evaluate a Multi-Turn baseline with access to the full prior reasoning trace.

We evaluated Multi-Turn refinement on Qwen3-8B using the same feedback as TTEL while retaining the previous thought trace in context, allowing reuse of information from up to 16k tokens of the prior attempt. TTEL outperforms this baseline by 10 to 15 percentage points across all three benchmarks (Appendix J.4, Figure 14, Table 11):

Method (Qwen3-8B, k = 64)LCBv6 Pass@64 (Tokens)AIME-25 Pass@64 (Tokens)HMMT-25 Pass@64 (Tokens)
TTEL71.0 ± 1.6% (360k)83.3 ± 0.0% (440k)60.8 ± 3.6% (528k)
Multi-Turn (Answer Only)56.7 ± 0.7% (323k)76.7 ± 3.3% (342k)54.2 ± 2.8% (321k)
Multi-Turn (Full Reasoning)55.7 ± 0.6% (191k)73.3 ± 4.1% (194k)50.0 ± 4.7% (195k)

Reviewer Query

(ii) / Q4 / W2b: Clarify how many branches occur per root, how D_c = 16,384 triggers restarts, whether deeper branches get stuck, and what happens with forced restarts after N branches.

  • After branching from a prefix, TTEL computes spikes only over the newly generated tokens, not over the retained prefix, so the prefix strictly grows and the same branch point is never re-selected. The student's next rollout uses the same feedback as the previous rollout's teacher, thus avoiding repetition of the same mistakes.
  • TTEL restarts from the question when either no spike survives filtering or the cumulative prefix length exceeds D_c = 16,384 (Algorithm 1, lines 15 to 20). The depth limit triggers up to 15% of all restarts, and a root averages 1.0 to 2.2 consecutive branches before restarting (Appendix J.8, Tables 13 and 14).
  • We report the recovery rate from consecutive failed branches alongside the number of questions solved exclusively via branching at minimum successful branch depth N = 1 versus N >= 2:
ModelDatasetRecovery (1st branch)Recovery (2nd branch)Branch-only solves (min N = 1)Branch-only solves (min N >= 2)
Qwen3-8BLCBv69.9%1.9%14.0 / 1311.0 / 131
Qwen3-8BAIME-2540.6%6.4%2.5 / 300.5 / 30
Qwen3-8BHMMT-2518.5%2.2%3.8 / 300.2 / 30
Qwen3-4B-ThinkingLCBv67.8%1.6%11.7 / 1310.7 / 131
Qwen3-4B-ThinkingAIME-2558.5%17.3%9.0 / 300.0 / 30
Qwen3-4B-ThinkingHMMT-2533.2%6.7%9.2 / 300.5 / 30

Consistent with this drop-off, adding a forced restart after N = 1 or N = 2 consecutive branches on Qwen3-8B matches or slightly improves Pass@64 across all three benchmarks (1 to 2 seeds, k = 64). The trade-off is 9% to 34% more generated tokens, since forced restarts discard reusable prefixes earlier.

Reviewer Query

Q2 / W1a: Why select t* = argmax g_t rather than the earliest qualifying spike in E_filtered?

We agree that branching at or before the first reasoning error is the right goal. Selecting the earliest spike in E_filtered is sensitive to noise and leads to a shorter prefix being reused. Comparing prefix reuse across the 145 of 162 failed Qwen3-8B trajectories (LCBv6, AIME-25, HMMT-25) with at least one spike:

Selection RuleMean Depth (tokens / % trace)Cuts Within First 1,000 Tokens
Earliest spike in E_filtered1,796 (14.4%)53.1%
t* = argmax g_t (TTEL)3,505 (26.7%)36.6%

Using the spike with the maximum magnitude retains roughly twice the prefix tokens as branching using the earliest spike. The maximum spike lies before the annotated root error on 71.1% of traces versus 47.4% for uniform random (Appendix J.3, Figure 12). On AIME-25, branching at the earliest spike gives similar Pass@64 while costing more generated tokens (553k vs. 440k) due to regenerating longer suffixes.

Reviewer Query

Q3 / W1b: In GenFB, the baseline replaces the revision instruction with a repetition instruction. What happens when holding the re-attempt instruction fixed and contrasting against a "Your previous attempt was successful" baseline?

  • Baseline prompt control: We thank the reviewer for noting this detail in the GenFB baseline prompt. Contrasting the "unsuccessful" teacher feedback against a "Your previous attempt was successful" baseline also isolates a sparse set of spikes. Across 162 failed Qwen3-8B trajectories the number of spikes decreases from 13.4 to 10.6 spikes per trace, while shifting the selected branch point deeper into the trace (mean spike depth 5,427 vs. 3,505 tokens).

We have addressed each requested check above and updated Appendix J accordingly. We would be glad to follow up on any remaining points.

Reviewer_kgVMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Test-Time Scaling via Error Localization (TTEL), a training-free test-time search algorithm that aims to localize trajectory errors using token-probability contrasts from the same language model. It compares probabilities recorded during the original rollout with those obtained by re-scoring the trajectory under a feedback-augmented context and a baseline reprompting context. TTEL then selects a truncation point, retains the preceding prefix, and regenerates the suffix. Without parameter updates, the method aims to improve the generated-token efficiency of test-time search in mathematical reasoning and code generation. Across two Qwen models and three benchmarks, the authors report improvements over Independent Sampling, Multi-Turn Refinement, and Recursive Self-Aggregation.

Critical strengths

S1. Clear methodology addressing a well-motivated problem. Using feedback-conditioned re-scoring to select branching positions and combining it with prefix reuse is a clear design with practical potential. Using the same model for generation and evaluation avoids the need to train an additional process reward model.

S2. Promising results with reduced generated-token cost. In Table 1, Qwen3-8B on LiveCodeBench improves from 64.6% pass@64 with Independent Sampling to 71.0% with TTEL, while the average generated-token count decreases from 735.0k to 360.4k. Notably, the main LiveCodeBench results use generic failure feedback rather than detailed execution feedback. Improvements on mathematical benchmarks further suggest that the strategy does not require rich execution feedback to be useful.

Critical weaknesses

W1. The independent contribution and semantic accuracy of the localization signal remain insufficiently validated.

TTEL simultaneously changes prefix reuse, historical context, and branching-position selection relative to the main baselines. Appendix B shows that TTEL retains the full reasoning trace of the latest attempt, whereas Multi-Turn retains only the final answers from previous rounds. The current comparisons therefore do not isolate the contribution of spike-based selection from the benefits of the other components. A comparison against random or simple rule-based branching, with matched historical context, comparable branching depths, and continuation budgets, would help establish the independent value of the localization signal.

Figure 10 in Appendix I shows that parent trajectories with correct final answers also produce branching signals. Since this analysis categorizes trajectories by terminal correctness, it cannot determine whether these peaks correspond to genuine reasoning errors, redundant expressions, or other modifiable positions. Figure 6 demonstrates that baseline filtering reduces detected spikes and improves downstream pass@k, but it does not directly establish whether the retained positions correspond more accurately to semantic errors. Comparing continuation success rates from retained and filtered-out positions would assess their intervention utility, while evaluating semantic localization accuracy would require independent error annotations.

Furthermore, the baseline in Section 4.1 and Appendix B retains the original feedback while replacing the revision instruction with a repetition instruction. The resulting probability differences may therefore also reflect instruction-induced changes in expression preferences. The theorem in Appendix E does not establish a formal connection between the spike score and the selection of favorable prefixes. These observations leave the interpretation of the signal as precise token-level error localization insufficiently supported.

W2. The length constraint on complete trajectories is insufficiently controlled.

Section 6 and Appendix B cap newly generated tokens per rollout at 16,384 while allowing TTEL to retain a prefix of approximately the same maximum length. Under the stated limits, a branched TTEL trajectory could approach 32k tokens, whereas an Independent Sampling trajectory remains bounded by 16k tokens.

The paper does not report the actual lengths of complete trajectories, the proportion of generations terminated by the token cap, or whether those generations had already produced a final answer. Consequently, the current reporting does not rule out greater available reasoning length within individual trajectories as a partial explanation for the gains. Please clarify whether an additional complete-trajectory length limit is applied, report the relevant length and termination statistics, and assess whether the advantage persists when this factor is controlled at matched cumulative cost.

Questions for authors

Please address the questions raised in W1–W2.

3 comments

Thank you for the detailed review and for pointing us towards consideration of matched reasoning length, effectiveness of error localization against a control and fair comparison with baselines. All new analyses are present in Appendix J of the revised PDF and summarized in the comments below. We hope this addresses your concerns and would greatly appreciate it if you could reconsider your score.

Reviewer Query

W2: Please clarify whether an additional complete-trajectory length limit is applied, report the relevant length and termination statistics, and assess whether the advantage persists when this factor is controlled at matched cumulative cost.

TTEL with a strict 16k token cap (Appendix J.1, Figure 11, Table 9): No separate complete-trajectory cap is applied in the main runs: TTEL allows a prefix up to D_c = 16,384 and a continuation up to 16,384 (~32k max), but stitched trajectories exceed 16k on only 1.3% to 16.8% of turns, and median complete length (5.6k to 14.2k tokens) is shorter than Independent Sampling (14.0k to 16.4k) in every setting. Retrospectively scoring every trajectory over 16,384 total tokens and its successor branches as failures (while still charging their full cumulative token cost) reduces mean TTEL Pass@64 across all six settings only from 73.1% to 71.5%, preserving TTEL's advantage across the Pareto curve:

ModelDatasetTTEL Pass@64Strict <=16k Pass@64Best Baseline Pass@64
Qwen3-8BLCBv671.0 ± 1.6%70.7 ± 1.3%64.6% (Indep.)
Qwen3-8BAIME-2583.3 ± 0.0%82.5 ± 1.4%76.7% (Multi-Turn)
Qwen3-8BHMMT-2560.8 ± 3.6%59.2 ± 4.9%55.8% (Indep.)
Qwen3-4B-ThinkingLCBv669.5 ± 0.6%69.2 ± 0.4%67.4% (Indep.)
Qwen3-4B-ThinkingAIME-2585.0 ± 1.7%83.3 ± 2.4%80.0% (Multi-Turn)
Qwen3-4B-ThinkingHMMT-2569.2 ± 3.6%64.2 ± 6.4%47.5% (Indep.)

Truncation rates (Appendix J.2, Table 10): Multi-Turn refinement and TTEL hit the 16k generation cap far less often than Independent Sampling, particularly on math (% of attempts hitting 16k):

Method8B LCBv68B AIME-258B HMMT-254B LCBv64B AIME-254B HMMT-25
Independent Sampling42.9%50.4%62.3%39.0%64.2%80.0%
Multi-Turn5.2%11.8%9.2%1.3%1.7%1.9%
TTEL14.6%20.8%27.4%10.1%27.4%31.7%

Evaluation at a 28k generation cap on OLMo-3-7B-Think (Appendix J.6, Figure 16): We evaluated OLMo-3-7B-Think at a 28k-token generation cap up to k=64 rollouts (4 seeds). Applying a strict cutoff of 28k tokens for the prefix and rollout, TTEL still outperforms Multi-Turn refinement and matches Independent Sampling at roughly half the generated tokens:

BenchmarkTTEL Pass@64 (Tokens)TTEL Strict <=28k (Tokens)Multi-Turn Pass@64 (Tokens)Independent Pass@64 (Tokens)
AIME-2587.5 ± 4.3% (588k)86.7 ± 3.3% (588k)84.2 ± 1.4% (544k)86.7 ± 0.0% (1,202k)
HMMT-2555.8 ± 2.8% (640k)55.8 ± 2.8% (640k)50.8 ± 1.4% (563k)56.7 ± 2.4% (1,337k)

Reviewer Query

W1 (Controls): TTEL simultaneously changes prefix reuse, historical context (full last trace vs. answers only in Multi-Turn), and branch selection. Compare against matched historical context and random branching.

Matched historical context: We evaluated Multi-Turn refinement on Qwen3-8B using the same feedback as TTEL while retaining the previous thought trace in context, allowing reuse of information from up to 16k tokens of the prior attempt. TTEL outperforms this baseline by 10 to 15 percentage points across all three benchmarks (Appendix J.4, Figure 14, Table 11):

Method (Qwen3-8B, k = 64)LCBv6 Pass@64 (Tokens)AIME-25 Pass@64 (Tokens)HMMT-25 Pass@64 (Tokens)
TTEL71.0 ± 1.6% (360k)83.3 ± 0.0% (440k)60.8 ± 3.6% (528k)
Multi-Turn (Answer Only)56.7 ± 0.7% (323k)76.7 ± 3.3% (342k)54.2 ± 2.8% (321k)
Multi-Turn (Full Reasoning)55.7 ± 0.6% (191k)73.3 ± 4.1% (194k)50.0 ± 4.7% (195k)

Random-branch control (Appendix J.3, Figure 13): TTEL (random prefix depth) selects a branching token uniformly at random from the latest rollout (excluding any prefix). The search tree, feedback prompt, and restart dynamics are identical to TTEL. We evaluated this baseline with Qwen3-8B over LiveCodeBench V6 and HMMT-25 up to k = 64 rollouts (1 seed) and compare it with standard TTEL (3 seeds on LCBv6, 4 on HMMT-25):

Benchmark (Qwen3-8B, k = 64)TTEL Pass@64TTEL TokensTTEL (random prefix depth) Pass@64TTEL (random prefix depth) Tokens
LCBv671.0 ± 1.6%360k68.7%423k
HMMT-2560.8 ± 3.6%528k53.3%637k

On both benchmarks, TTEL reaches higher Pass@64 while using 15% to 17% fewer tokens.

Reviewer Query

W1 (Localization & Prompt Contrast): Evaluate semantic localization with independent error annotations; explain spikes on already-correct parents (Fig. 10); check whether replacing the revision instruction with a repetition instruction in the baseline affects spike selection.

Error Localization vs. Branching Before Errors (Appendix J.3, Figure 12): We would like to state that precise error localization is not expected, but branching at or before an error is sufficient to avoid propagating flawed reasoning.

We take a failed independently sampled trajectory per question for Qwen3-8B across LCBv6, AIME-25, and HMMT-25. We use a frontier LLM annotator that locates the earliest reasoning error, which it does successfully in 135 trajectories. Branching prefixes for TTEL lie before the error in 71.1% of the traces versus 47.4% for uniform random:

MethodLCBv6 (n = 97)AIME-25 (n = 19)HMMT-25 (n = 19)
Uniform Random49.0%39.6%47.6%
TTEL73.2%68.4%63.2%

Spikes on already-correct parents (Appendix J.7):

  • In TTEL-GenFB, the message "The previous attempt was unsuccessful" is sent unconditionally after every turn regardless of whether the attempt was correct. This does not use any oracle verifier. Appendix I describes spikes from already-correct parents and the retention rate of correct solutions when branching from these tokens.
  • This unconditional message does not hurt scaling (Appendix J.7). On Maths datasets, branching from a correct parent trace retains correctness 95% to 99% of the time (raising mean@64 across all difficulty tiers). On LCBv6, mean@64 improves on medium and hard problems.

Baseline prompt control: We thank the reviewer for noting this subtlety in the GenFB baseline prompt. Contrasting the "unsuccessful" teacher feedback against a "Your previous attempt was successful" baseline still isolates a sparse set of spikes. Across 162 failed Qwen3-8B trajectories the number of spikes decreases from 13.4 to 10.6 spikes per trace, while shifting the selected branch point deeper into the trace (mean spike depth 5,427 vs. 3,505 tokens).

We hope these controlled baselines and error-localization results address your concerns. We are happy to discuss any remaining questions.

Reviewer_N7pkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TTEL (Test-Time Scaling via Error Localization), a simple inference-time method that regenerates part of an attempt rather than the whole attempt. The central idea adapts recent self-distillation methods in reinforcement learning, which compare a model's token probabilities with and without post-hoc feedback. The major difference is that TTEL uses this signal during inference, without updating the model and without observing the correct answer. After each attempt, the model re-scores the same tokens under two contexts: a teacher context with feedback and a baseline context with uninformative instructions. In the main variant, the feedback is a fixed "unsuccessful" message; in the code variant, an alternative uses execution feedback. The token with the largest feedback-specific probability drop is treated as a potential error; the attempt is truncated there, and a new continuation is generated from the retained prefix.

On LiveCodeBench with Qwen3-8B, TTEL surpasses independent sampling with fewer tokens.

Critical strengths
  1. The presentation is clear and easy to follow. I like that the authors pose three research questions and answer each with dedicated experiments.
  2. The method is easy to implement, needs no training, and reduces generated tokens by reusing prefixes.
  3. The method is reasonable at a high level. Using probability changes to identify critical tokens aligns well with self-distillation in RL.
  4. The experimental results are promising, with consistent pass@k gains over independent sampling, multi-turn refinement, and RSA across two models and three benchmarks.
  5. The ablations support the design: removing the baseline filter greatly increases detected spikes and lowers pass@k.
Critical weaknesses

Overall, I like the presentation and the method, but I think the experimental section could be more comprehensive, and I have a few concerns about the detailed implementation of the TTEL method:

  1. The reliability of the fixed feedback is questionable. Unlike self-distillation in RL, no label is available at test time, so the main variant uses the same "unsuccessful" message for every attempt, even the correct ones. This may matter more on tasks with high success rates than on difficult tasks where most attempts are wrong, but pass@k cannot reveal any harm, since the original correct attempt remains in the pool. Please state clearly when you send this message. Additional evaluation on tasks with different initial difficulty and report the mean@k can further clarify the question.

  2. Simple baselines are missing. TTEL is a test-time scaling method, yet it is not compared with Power Sampling [1], which also resamples parts of a generation, or with methods that branch at high-uncertainty tokens, such as high-entropy positions.

  3. The inference cost is not fairly reflected in the main claims. TTEL requires teacher and baseline forward passes over the full context for each attempt, but the main results and the "half the tokens" claim count only generated tokens. Under total tokens (appendix, Fig. 7), TTEL on LiveCodeBench with Qwen3-8B uses roughly 1.7× more tokens than independent sampling at k = 64, and with Qwen3-4B-Thinking it no longer dominates at most budgets. Please report measured wall-clock and GPU time, and qualify the "strictly dominating" claim.

  4. Localization is claimed but not verified. The paper claims precise token-level credit assignment but does not evaluate the selected tokens against annotated errors. Please report how often the selected token matches a real error.

  5. Pass@k does not show practical performance. Pass@k assumes a correct attempt can be identified afterward. With no label at test time, a selection step is required, but single-answer accuracy is not reported in the main results.

[1] Reasoning with Sampling: Your Base Model is Smarter Than You Think, Aayush Karan and Yilun Du, arXiv 2025.

Questions for authors

None

3 comments

Thank you for the thorough review and helpful suggestions. We have added verifier-free Pass@1 selection, additional baselines, prefill/decode wall-clock analysis, and error-localization results in Appendix J of the revised PDF, summarized across three short comments below. Hoping this address your concerns and would be grateful if you would consider raising the score.

Reviewer Query

W1: TTEL sends a fixed "unsuccessful" message even after correct attempts. State clearly when it is sent, evaluate by initial problem difficulty, and report mean@k.

  • In TTEL-GenFB, the message "The previous attempt was unsuccessful" is sent unconditionally after every turn regardless of whether the attempt was correct. This does not use any oracle verifier. Appendix I describes spikes from already-correct parents and the retention rate of correct solutions when branching from these tokens.

  • Sending this message unconditionally does not hurt scaling (Appendix J.7): on math, branching from an already-correct parent stays correct 95% to 99% of the time (raising mean@64 across all difficulty tiers). On LCBv6, mean@64 improves on medium and hard problems while dipping on easy ones (93.3% to 75.1% on Qwen3-8B), where Pass@64 remains 100%.

  • Mean@64 by initial difficulty tier: Stratifying questions by Independent Sampling solve rate (Easy >= 50%, Medium 0-50%, Hard 0%), TTEL raises mean@64 in every math tier and on Medium/Hard code tasks, while dipping only on Easy code (where Pass@64 stays 100%):

ModelDatasetEasy (Indep. / TTEL)Medium (Indep. / TTEL)Hard (Indep. / TTEL)
Qwen3-8BAIME-2586.8 / 88.014.7 / 46.50.0 / 2.7
Qwen3-8BHMMT-2586.3 / 88.911.9 / 22.70.0 / 0.1
Qwen3-8BLCBv693.3 / 75.112.1 / 20.20.0 / 1.4
Qwen3-4B-ThinkingAIME-2590.1 / 95.19.2 / 61.50.0 / 19.1
Qwen3-4B-ThinkingHMMT-2581.8 / 91.29.9 / 41.00.0 / 6.2
Qwen3-4B-ThinkingLCBv688.7 / 69.811.1 / 16.00.0 / 1.4

Reviewer Query

W5: Pass@k assumes an oracle selector; report single-answer Pass@1 selection accuracy.

  • We focus on scaling pass@k so that at least one correct answer is produced, after which selection methods can be applied to pick a single final answer. This contrasts with methods like Recursive Self-Aggregation that mainly bridge the gap from pass@1 to pass@k without solving new problems that independent sampling misses.
  • We evaluate Pass@1 using two selection methods over the k=64 candidate rollouts (Appendix J.5, Figure 15, Table 12). Majority voting picks the most frequent final answer, which is applicable for math datasets. The LLM-Judge is the same model asked to pick the best of four candidates, repeated over three rounds in a tournament style (64, 16, 4, 1 selected answers per round).
  • TTEL has the highest accuracy with majority voting across all settings. For LLM-Judge selection, it is the best in 4 of 6 cases and within 2 points in the other two.
ModelDatasetMaj. Indep.Maj. Multi-TurnMaj. TTELJudge Indep.Judge Multi-TurnJudge TTEL
Qwen3-8BLCBv6N/AN/AN/A51.949.453.4
Qwen3-8BAIME-2570.874.278.370.072.575.8
Qwen3-8BHMMT-2545.846.749.243.345.844.2
Qwen3-4B-ThinkingLCBv6N/AN/AN/A53.951.953.4
Qwen3-4B-ThinkingAIME-2561.776.783.360.077.580.0
Qwen3-4B-ThinkingHMMT-2540.844.248.340.843.350.0

Reviewer Query

W2: Compare against additional training-free inference-time baselines such as Power Sampling (Karan & Du, 2025) and Self-Refine.

We evaluated Self-Refine (32 solver attempts with critique tokens included) and Power Sampling (16 MCMC chains, α = 4.0, 1 seed) on Qwen3-8B. TTEL outperforms both in Pass@k, even at k = 16, with far fewer generated tokens (Appendix J.4, Figure 14, Table 11):

Method (Qwen3-8B)LCBv6 Pass@k (Tokens)AIME-25 Pass@k (Tokens)HMMT-25 Pass@k (Tokens)
TTEL (k = 16)63.6 ± 0.4% (96k)81.7 ± 1.7% (132k)54.2 ± 3.6% (147k)
TTEL (k = 64)71.0 ± 1.6% (360k)83.3 ± 0.0% (440k)60.8 ± 3.6% (528k)
Self-Refine (k = 32)60.3% (351k)80.0% (397k)53.3% (374k)
Power Sampling (k = 16)N/A63.3% (1,833k)46.7% (2,009k)

Reviewer Query

W4: Directly evaluate whether selected branching positions precede annotated reasoning errors.

  • Error Localization vs. Branching Before Errors (Appendix J.3, Figure 12): We would like to state that precise error localization is not expected, but branching at or before an error is sufficient to avoid propagating flawed reasoning.

We take a failed independently sampled trajectory per question for Qwen3-8B across LCBv6, AIME-25, and HMMT-25. We use a frontier LLM annotator that locates the earliest reasoning error, which it does successfully in 135 trajectories. Branching prefixes for TTEL lie before the error in 71.1% of the traces versus 47.4% for uniform random:

MethodLCBv6 (n = 97)AIME-25 (n = 19)HMMT-25 (n = 19)
Uniform Random49.0%39.6%47.6%
TTEL73.2%68.4%63.2%

Reviewer Query

W3: The inference cost is not fully reflected when counting only generated tokens, since TTEL requires teacher and baseline forward passes over the full context for each attempt.

  • Prefill vs. decode cost: Teacher and baseline re-scoring are parallel prefill passes over the trace, whereas generating new tokens is sequential decode. Although TTEL processes more total (input + output) tokens (already reported in Figure 7), prefill accounts for under 20% of TTEL's runtime on an A100 GPU.
  • Sequential wall-clock time per question (Qwen3-8B, k = 64, A100 GPU): Because TTEL generates roughly half as many output tokens, it takes less sequential GPU time per question than Independent Sampling (Appendix J.9, Table 15):
Benchmark (Qwen3-8B, k = 64)TTEL (min / Q)Indep. Sampling (min / Q)Multi-Turn (min / Q)TTEL Prefill Share
LCBv698.7161.680.419.9%
AIME-25117.4184.484.517.9%
HMMT-25140.7201.379.917.7%

If all 64 independent attempts for one question fit in GPU memory at once, Independent Sampling has lower single-question latency since TTEL turns are sequential, whereas across a batch of questions TTEL parallelizes across questions.

Thanks again for the thoughtful suggestions. Please let us know if you have any follow-up questions.

Reviewer_8K3zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces TTEL, an inference-time search method that reuses the valid prefix of a failed reasoning trajectory instead of resampling from scratch. After a failure, the model re-scores its own trajectory under (1) a feedback context and (2) a "non-diagnostic" baseline context. Tokens whose probability drops under feedback but not under the baseline are treated as error locations, and the trajectory is truncated and regenerated from the strongest one (or restarted if none survive). The signal is borrowed from self-distillation (SDPO) but used purely at inference time. On LiveCodeBench v6, AIME-25, and HMMT-25 with Qwen3-8B and Qwen3-4B-Thinking-2507, TTEL reports better pass@k-vs-token frontiers than independent sampling, multi-turn refinement, and RSA (e.g., 71.0% pass@64 at ~360k tokens vs. 64.6% at ~735k on LCB).

Critical strengths
  • [Section 4, Figure 2] Neat, simple idea.
    • Repurposing the π(·|x)/π(·|x,f) contrast as a training-free branching signal is creative.
    • The non-diagnostic baseline is a sensible control, and Fig. 6 shows it matters (~19 → ~486 spikes per turn without it).
  • [Appendix F] Thorough raw reporting.
    • Per-seed tables for every method / budget / model / benchmark, plus total-token accounting (Fig. 7). This is what let me check the concerns below.
Critical weaknesses
  • [Section 6, Tables 1–6] The gains may be an artifact of the 16,384-token cap.
    • All methods are capped at 16,384 new tokens per rollout, but TTEL continues from a prefix of up to Dc = 16,384, so a branched trajectory can reach ~32k tokens while only being charged for the suffix.
    • The baselines look heavily truncated: independent sampling averages ~14.5k tokens/rollout on AIME-25 and ~15.4k on HMMT-25 with Qwen3-4B-Thinking (i.e., at the cap).
    • Independent-sampling pass@64 on AIME-25 for Qwen3-4B-Thinking is only 61.7%, which seems well below this model's commonly reported single-sample accuracy.
    • If baselines are cut off mid-reasoning, "branch and keep going" helps regardless of where you branch.
  • [Section 7.3, Appendix E] The paper doesn't isolate the localization itself.
    • The missing control: the same tree search with a random (or fixed-fraction) branch point. Prefix reuse alone mechanically lowers token cost.
    • The "no filter" ablation changes the spike distribution, which is not the same as comparing to an uninformed choice.
    • Theorem 1 holds for any prefix; it's only an advantage if we already assume s_a > s_¬a, which is the claim that needs evidence.

[Overall] I like the idea, but I'm not yet convinced the gains come from error localization rather than prefix reuse under a tight cap. A random-branch baseline and a larger generation cap would go a long way, and I'd revise upward if the frontiers hold.

Questions for authors

Clarifications:

  1. [Section 6] What fraction of rollouts hit the 16,384 cap, per method? Can branched TTEL trajectories exceed 16,384 total tokens? How do results change with a matched effective max length (e.g., 32k+)?
  2. [Section 6, GenFB] Is "Your previous attempt was unsuccessful" triggered by ground truth / hidden tests? If yes, that's an oracle signal and should be stated. Appendix I mentions spikes with a "correct parent". Is the message sent regardless of correctness?
  3. [Section 7.3] Can you add a random-branch-point baseline with everything else fixed?
  4. [Section 7.2] TTEL-GenFB pass@16 on LCB appears as 0.630 (Fig. 5), 63.6 (Table 1), and 65.9 (Table 7). Why do they differ? Fig. 5 also says 4 seeds vs. 3 elsewhere.
  5. [Section 6] Multi-turn sees only previous answers, while TTEL's teacher sees the full last trace. Since Section 7.2 shows the trace matters, would multi-turn with the full trace close the gap?

Suggestions:

  1. A selection metric (majority vote / verifier-selected) alongside pass@k, since prefix sharing makes candidates more correlated.
  2. Wall-clock time, since re-scoring adds prefill cost.

Minor:

  • [Abstract] "Strictly dominating" is too strong when curves overlap within 1 SD at low k (e.g., 4B-Thinking LCB k=4: 51.7 ± 2.5 vs. 52.4 ± 1.9).
  • [Section 4.1] Please state τ_B (0.06) in the main text.
  • [Appendix B] Top-100 log-probs, with the rest ≈0: how often is the original token outside the top-100? That would produce a max spike regardless of feedback.
  • [Figure 9] The caption looks garbled.
  • [Section 8] Stray colon in "Conclusion:".

Suggested Citations:

  • [Section 2] More discussion of step-level search that branches from intermediate states using PRMs/verifiers (e.g., step-level beam search in Snell et al., 2024; REBASE in Wu et al., 2024; both already cited) would clarify what's new.
5 comments

Thanks for your insightful and helpful review. We hope these responses adequately address your points and would be grateful if you would consider increasing your score.

W1 and Q1: What fraction of rollouts hit the 16,384 cap, per method? Can branched TTEL trajectories exceed 16,384 total tokens? How do results change with a matched effective max length (e.g., 32k+)?

Truncation rates at the 16k cap (Appendix J.2, Table 10). Multi-Turn refinement and TTEL hit the 16,384-token generation cap far less often than Independent Sampling, particularly on math (% of attempts hitting 16k):

Method8B LCBv68B AIME-258B HMMT-254B LCBv64B AIME-254B HMMT-25
Independent Sampling42.9%50.4%62.3%39.0%64.2%80.0%
Multi-Turn5.2%11.8%9.2%1.3%1.7%1.9%
TTEL14.6%20.8%27.4%10.1%27.4%31.7%

Matched strict ≤16k total-token cap (Appendix J.1, Figure 11, Table 9). Yes—because TTEL continues generation from a branched prefix, total trajectory length (prefix + continuation) can reach up to ~32k tokens. However, TTEL's gains do not rely on exceeding 16,384 total tokens: retrospectively scoring every trajectory over 16,384 total tokens (and all of its successor branches) as a failure—while still charging their full token cost—reduces mean Pass@64 by only 1.6 pp (73.1% to 71.5%), and TTEL still outperforms the strongest baseline in every setting:

ModelDatasetTTEL Pass@64Strict ≤16k Pass@64Best Baseline Pass@64
Qwen3-8BLCBv671.0 ± 1.6%70.7 ± 1.3%64.6% (Indep.)
Qwen3-8BAIME-2583.3 ± 0.0%82.5 ± 1.4%76.7% (Multi-Turn)
Qwen3-8BHMMT-2560.8 ± 3.6%59.2 ± 4.9%55.8% (Indep.)
Qwen3-4B-ThinkingLCBv669.5 ± 0.6%69.2 ± 0.4%67.4% (Indep.)
Qwen3-4B-ThinkingAIME-2585.0 ± 1.7%83.3 ± 2.4%80.0% (Multi-Turn)
Qwen3-4B-ThinkingHMMT-2569.2 ± 3.6%64.2 ± 6.4%47.5% (Indep.)
Mean—73.1%71.5%65.3%

Matched 28k generation cap (Appendix J.6, Figure 16). To test a longer horizon, we evaluated OLMo-3-7B-Think with a 28k-token generation cap across all methods up to k = 64 rollouts (4 seeds). Even under a matched strict ≤28k cutoff on prefix + continuation, TTEL outperforms Multi-Turn refinement (+2.5 to +5.0 pp) and matches Independent Sampling while using roughly half the generated tokens (51%–52% fewer):

BenchmarkTTEL Pass@64 (Tokens)Strict ≤28k Pass@64 (Tokens)Multi-Turn Pass@64 (Tokens)Independent Pass@64 (Tokens)
AIME-2587.5 ± 4.3% (588k)86.7 ± 3.3% (588k)84.2 ± 1.4% (544k)86.7 ± 0.0% (1,202k)
HMMT-2555.8 ± 2.8% (640k)55.8 ± 2.8% (640k)50.8 ± 1.4% (563k)56.7 ± 2.4% (1,337k)

W2 & Q3: Can you add a random-branch-point baseline with everything else fixed, and validate whether TTEL actually branches at or before reasoning errors?

To isolate the effect of branch-point selection with everything else fixed, we added TTEL (random prefix depth) in Appendix J.3 (Figure 13). This baseline selects a branching token uniformly at random from the latest rollout (excluding any prior prefix), while keeping the search tree, feedback prompt, and restart dynamics identical to TTEL. Evaluated with Qwen3-8B up to k = 64 rollouts on LiveCodeBench V6 (LCBv6) and HMMT-25, standard TTEL (3 seeds on LCBv6, 4 on HMMT-25) outperforms the random-depth baseline (1 seed) on Pass@64 while using 15–17% fewer tokens:

Benchmark (Qwen3-8B, k = 64)TTEL Pass@64TTEL TokensTTEL (Random Depth) Pass@64TTEL (Random Depth) Tokens
LCBv671.0 ± 1.6%360k68.7%423k
HMMT-2560.8 ± 3.6%528k53.3%637k

Regarding error localization (Appendix J.3, Figure 12), TTEL does not require exact token-level localization; branching anywhere at or before the earliest error is sufficient to avoid propagating flawed reasoning. To test this directly, we took one independently sampled failed Qwen3-8B trajectory per question across LCBv6, AIME-25, and HMMT-25, and used a frontier LLM annotator to locate the earliest reasoning error, which succeeded on 135 trajectories. TTEL's branching prefix precedes the error in 71.1% of traces overall, compared to 47.4% for uniform random selection:

MethodLCBv6 (n = 97)AIME-25 (n = 19)HMMT-25 (n = 19)Overall (n = 135)
Uniform Random49.0%39.6%47.6%47.4%
TTEL73.2%68.4%63.2%71.1%

Q2: Is "Your previous attempt was unsuccessful" triggered by ground truth / hidden tests? Appendix I mentions spikes with a "correct parent". Is the message sent regardless of correctness?

Yes—in TTEL-GenFB, the message "The previous attempt was unsuccessful" is sent unconditionally after every turn regardless of whether the attempt was correct, without using any ground-truth or hidden-test verifier. Appendix I examines spikes from already-correct parents and measures how often correct solutions are retained when branching from those tokens.

Sending this feedback unconditionally does not hurt scaling (Appendix J.7). On math benchmarks, branching from an already-correct parent stays correct 95%–99% of the time, raising mean@64 across all difficulty tiers. On LCBv6, mean@64 improves on medium and hard problems; while it dips on easy problems (from 93.3% to 75.1% on Qwen3-8B), Pass@64 on those problems remains 100%.

Suggestion 1: Reporting a single-answer selection metric (such as majority vote or verifier-free selection) alongside Pass@k would clarify practical gains.

Our primary focus is scaling Pass@k so that candidate generation produces at least one correct answer, after which downstream selection methods can pick a single final prediction. This contrasts with methods such as Recursive Self-Aggregation, which mainly close the gap from Pass@1 to Pass@k without solving new problems missed by independent sampling.

To show how Pass@k gains translate into single-answer accuracy, we evaluate verifier-free Pass@1 selection over the k = 64 candidate rollouts using two methods (Appendix J.5, Figure 15, Table 12). On math datasets, majority voting selects the most frequent final answer. Across all datasets, we also evaluate an LLM-Judge, prompting the same model to choose the best of four candidates in a three-round tournament (64 → 16 → 4 → 1).

TTEL achieves the highest accuracy under majority voting across every setting. Under LLM-Judge selection, TTEL performs best in 4 of 6 settings and comes within 2 points in the remaining two:

ModelDatasetMaj. Indep.Maj. Multi-TurnMaj. TTELJudge Indep.Judge Multi-TurnJudge TTEL
Qwen3-8BLCBv6N/AN/AN/A51.949.453.4
Qwen3-8BAIME-2570.874.278.370.072.575.8
Qwen3-8BHMMT-2545.846.749.243.345.844.2
Qwen3-4B-ThinkingLCBv6N/AN/AN/A53.951.953.4
Qwen3-4B-ThinkingAIME-2561.776.783.360.077.580.0
Qwen3-4B-ThinkingHMMT-2540.844.248.340.843.350.0

Comments on the paper

1 comment

We thank the reviewers for their insightful feedback and for highlighting our work’s core strengths:

  • Training-free, single-model token-level error localization: The same model conditioned conditioned on feedback can provide a branching signal through simple token probabilities, without additional training or an external PRM (8K3z, N7pk, kgVM, ikab, hWSG, Hxep).
  • Effective non-diagnostic baseline filter: Removing generic context shifts cuts Turn-1 spikes from ~486 to ~19 and raises Pass@16 (8K3z, N7pk, ikab).
  • Consistent Pass@k gains with ~50% fewer tokens: Strong gains on both coding and maths: for example Qwen3-8B on LiveCodeBench V6 raising Pass@64 from 64.6% to 71.0% compared to independent sampling, with tokens reducing from 735k to 360k. (N7pk, kgVM, ikab, hWSG).
  • Works with generic feedback: Gains hold without execution traces or verifiers, enabling math reasoning (kgVM, Hxep).
  • Clear presentation & algorithm specification: Well-illustrated mechanism, concise algorithm, and RQ-driven evaluation (N7pk, kgVM, hWSG).
  • Transparent raw reporting: Per-seed tables across all models, budgets, and benchmarks plus total-token accounting (8K3z).

The review process has driven us to conduct 13 new experiments and analyses (6 new figures and 7 new tables in Appendix J, plus 3 targeted controls in individual replies), fundamentally strengthening the paper’s empirical claims. We have updated the manuscript to reflect these findings:

1. Ruling Out 16k Length-Cap Artifacts

A primary question was whether TTEL’s gains stemmed from implicit context extension beyond the 16k per-attempt cap rather than error localization. Auditing per-attempt truncation rates alongside a strict total-length cap analysis shows otherwise: scoring every TTEL trajectory over 16,384 total tokens (prefix + continuation) and all downstream branches as failures—while still charging full token cost—reduces mean Pass@64 across all six settings only from 73.1% to 71.5%, keeping TTEL ahead of the best baseline in every setting.

(Added to Appendix J.1, Fig. 11, Table 9; Appendix J.2, Table 10)

2. Validating Error-Onset Localization & Branching Mechanism (Reviewers kgVM, ikab, Hxep)

To verify that TTEL pinpoints branching points before errors, we evaluated localization on 135 Gemini Argon annotated failures. TTEL branches before the earliest error on 71.1% of traces vs. 47.4% uniform random. Against TTEL (random prefix depth) (identical tree, prompt, and restarts), TTEL improves Pass@64 on LCBv6 (71.0% vs. 68.7%) and HMMT-25 (60.8% vs. 53.3%) with 15%–17% fewer tokens. Three targeted controls in our individual replies further evaluate argmax vs. earliest spike (ikab), "unsuccessful" vs. "successful" baseline prompts (kgVM, ikab), and PRM-guided branching (Hxep).

(Added to Appendix J.3, Figs. 12–13; targeted controls in individual replies)

3. Stronger Baselines & Verifier-Free Pass@1 Selection

On Qwen3-8B, TTEL beats Multi-Turn with full prior reasoning traces by 10–15 points, and even at k=16 surpasses 32-round Self-Refine and Power Sampling. Under verifier-free Pass@1 selection, TTEL achieves the highest accuracy on 4/4 math Majority Vote settings and 4/6 same-model LLM-Judge tournament settings (within 2 points on the other two).

(Added to Appendix J.4, Fig. 14, Table 11; Appendix J.5, Fig. 15, Table 12)

4. Generalizability Across Model Families, 28k Cap & Feedback Regimes

To verify generalizability beyond Qwen and a 16k cap, we evaluated OLMo-3-7B-Think under a 28k generation cap across 4 seeds (including strict 28k accounting): TTEL outperforms Multi-Turn and matches Independent Sampling at roughly half the generated tokens. We also added unconditional-feedback and difficulty-tier analyses across problem regimes.

(Added to Appendix J.6, Fig. 16; Appendix J.7)

5. Practical Runtime, Restart Dynamics & Logprob Sensitivity

We also profiled wall-clock cost, top-100 logprob sensitivity, and restart dynamics. Prefill re-scoring is <20% of A100 runtime (98.7–140.7 min/Q vs. 161.6–201.3 min/Q sequential for Independent Sampling), and 80.5% (342/425) of first solves after a failed Turn 0 come from branched turns rather than fresh restarts.

(Added to Appendix J.8–J.10; Tables 13–15)

Thanks for providing references for related works. We will cite these papers in the revised version of our paper.

We believe the revised paper now offers a comprehensive, rigorously validated solution for training-free, token-level error localization. Through additional experiments and analyses, we have addressed all concerns raised by the reviewers, with detailed point-by-point replies posted under each review. We really thank our reviewers for their time and service, and welcome any further suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.