- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Test-Time Scaling via Error Localization (TTEL), an inference-time search algorithm that aims to improve reasoning efficiency by identifying potential error locations and reusing prefixes of previously generated trajectories. TTEL compares token probabilities under feedback-conditioned and non-diagnostic contexts to identify candidate error locations, then branches from the selected position to regenerate the remaining trajectory. Experiments on LiveCodeBench V6, AIME-2025, and HMMT-2025 demonstrate promising improvements in pass@k and token efficiency over independent sampling and multi-turn refinement baselines.
Critical strengths
S1. Fine-grained error localization from trajectory-level feedback.
The paper presents a promising approach to deriving token-level error localization signals from trajectory-level feedback. By comparing token probabilities under different feedback conditions, TTEL identifies potential error locations within failed reasoning trajectories and uses them to guide prefix-based branching. This provides a principled way to leverage coarse-grained feedback for fine-grained test-time search without additional model training.
S2. Effectiveness under limited diagnostic feedback.
An interesting finding is that TTEL achieves improvements even with generic feedback rather than detailed environment-generated error messages. The results on AIME-2025 and HMMT-2025 suggest that feedback-conditioned probability shifts can provide useful signals for guiding search, even when the feedback does not explicitly identify the underlying reasoning errors. This broadens the potential applicability of the approach beyond environments with rich execution feedback.
Critical weaknesses
W1. Insufficient validation of the reliability and interpretability of error localization.
The central premise of TTEL is that feedback-induced token probability shifts can reveal meaningful error locations. However, the paper does not provide a quantitative evaluation of localization accuracy against actual reasoning errors, making it difficult to assess how reliably the selected branching points identify erroneous reasoning steps. While Figure 8 presents a qualitative example and Appendix I analyzes branching behavior, neither directly validates the accuracy of error localization.
Furthermore, the interpretation of these probability shifts remains insufficiently justified. A token's probability may decrease under feedback because the model prefers a different expression or reasoning path, rather than because the original token represents an actual mistake. Although the proposed baseline filtering mitigates certain context-induced effects, it does not establish that the remaining spikes correspond to genuine reasoning errors. These concerns weaken the evidence supporting the proposed localization mechanism.
W2. Unclear necessity of the proposed error localization mechanism and missing comparisons with alternative approaches.
The paper demonstrates the effectiveness of TTEL as a complete framework, but does not sufficiently establish whether its specific error localization mechanism is necessary for the observed improvements. In particular, simpler branching strategies, such as random or fixed-position branching, could help isolate the contribution of feedback-guided localization from the benefits of prefix reuse.
Moreover, process reward models (PRMs) can also provide intermediate correctness signals and potentially guide branching from erroneous reasoning steps. Although PRMs are discussed in Appendix A, the paper does not include an empirical comparison with PRM-guided localization and branching. Such a baseline would be particularly valuable for assessing the accuracy, effectiveness, and practical trade-offs of TTEL's training-free localization strategy. Convincing evidence from these comparisons would substantially strengthen the paper, and I would consider increasing my score accordingly.
W3. Limited analysis of iterative refinement effectiveness compared with conventional multi-turn refinement.
Figure 4 compares TTEL and multi-turn refinement across different rollout budgets, while Appendix I analyzes TTEL's branching behavior across successive turns. However, these results do not directly reveal how their correction capabilities differ over consecutive refinement iterations.
In particular, it remains unclear whether TTEL is more effective at correcting an initially unsuccessful attempt after one or multiple refinement steps, and how this advantage evolves with additional iterations. A direct comparison of per-iteration correction success rates, starting from the same failed trajectories, would provide stronger evidence for the effectiveness of the proposed refinement mechanism.
W4. Insufficient analysis of the restart strategy and its contribution to search effectiveness.
TTEL triggers a restart when no valid localization spike is detected or the selected branching point exceeds the depth limit. While Appendix I provides some analysis of successive refinement turns after a restart, the paper does not clearly characterize how frequently restarts occur or how they contribute to the overall search performance.
In particular, when valid spikes cannot be identified, TTEL falls back to generating fresh trajectories from the original prompt, potentially behaving similarly to independent sampling. It remains unclear how much of the reported improvement comes from successful prefix-based refinement versus restarting and exploring new trajectories. Further analysis of restart frequency, refinement depth, and the respective contributions of these two operations would help establish the effectiveness of the proposed search strategy.
Questions for authors
Q1. Reliability and interpretability of error localization.
Could the authors provide a quantitative evaluation of error localization accuracy against annotated or synthetically introduced reasoning errors? For example, measuring localization accuracy or the distance between selected branching points and actual error positions would help establish the reliability of the proposed mechanism.
Additionally, could the authors clarify how genuine reasoning errors are distinguished from feedback-induced probability shifts caused by alternative valid reasoning paths or changes in expression?
Q2. Necessity of error localization and comparisons with alternative approaches.
Could the authors compare TTEL with simpler branching strategies, such as random or fixed-position branching, under matched inference budgets? This would help isolate the contribution of the proposed localization mechanism from the benefits of prefix reuse.
More importantly, a comparison with PRM-guided error localization and branching would be particularly valuable. It would help establish whether TTEL provides more accurate localization, better downstream performance, or a more favorable trade-off between effectiveness and computational cost.
Q3. Refinement effectiveness and restart behavior.
Could the authors provide a direct comparison of TTEL and conventional multi-turn refinement across consecutive refinement iterations, ideally starting from the same unsuccessful trajectories? Reporting correction success rates after one, two, or more refinement steps would help clarify the benefits of the proposed approach beyond the existing pass@k results.
Additionally, could the authors report the frequency of restarts, the distribution of consecutive refinement turns before each restart, and the relative contributions of prefix-based refinement and fresh exploration to overall search success? These analyses would help explain how the two components interact and whether the restart strategy effectively allocates the inference budget.
2 comments
W3 and Q3: Limited analysis of iterative refinement effectiveness compared with conventional multi-turn refinement.
Truncating at the selected spike position keeps valid upstream steps and drops the flawed suffix instead of conditioning on a wrong final answer. On the first branch from a failed attempt, TTEL recovers 40.6%, 18.5%, and 9.9% of trajectories on Qwen3-8B, and 58.5%, 33.2%, and 7.8% on Qwen3-4B-Thinking (AIME-25, HMMT-25, LCBv6). By the second consecutive branch, recovery falls to 1.9%–6.4% (8B) and 1.6%–17.3% (4B; full table in Q3 Part 2), where spike filtering restarts cleanly before errors compound.
Multi-Turn (Answer Only) discards prior reasoning each round, whereas our newly evaluated Multi-Turn (Full Reasoning) baseline on Qwen3-8B, run with the same feedback and thought traces as TTEL, appends failed traces until the context window fills and progress stalls after early turns. By k = 16, TTEL already matches or beats both k = 64 Multi-Turn variants across all three benchmarks with fewer tokens (under half of Answer Only), and keeps gaining through k = 64:
| Method (Qwen3-8B) | LCBv6 Pass@k (Tokens) | AIME-25 Pass@k (Tokens) | HMMT-25 Pass@k (Tokens) |
|---|---|---|---|
| TTEL (k = 16) | 63.6 ± 0.4% (96k) | 81.7 ± 1.7% (132k) | 54.2 ± 3.6% (147k) |
| TTEL (k = 64) | 71.0 ± 1.6% (360k) | 83.3 ± 0.0% (440k) | 60.8 ± 3.6% (528k) |
| Multi-Turn (Answer Only, k = 64) | 56.7 ± 0.7% (323k) | 76.7 ± 3.3% (342k) | 54.2 ± 2.8% (321k) |
| Multi-Turn (Full Reasoning, k = 64) | 55.7 ± 0.6% (191k) | 73.3 ± 4.1% (194k) | 50.0 ± 4.7% (195k) |
More details can be found here: [Iterative refinement effectiveness vs. Multi-Turn across successive steps (Appendices J.4 & J.8, Figure 14, Tables 11, 13, and 14)]
W4 and Q3: Insufficient analysis of the restart strategy and its contribution to search effectiveness.
Across k = 64 turns, TTEL restarts on 27%–41% of post-Turn-0 turns; baseline filtering (no surviving spike) triggers most restarts, and only a small share hit the D_c = 16,384 prefix cap. Recovery concentrates on the first branch and drops steeply on the second, so chains stay at 1–2 branches per root:
| Model | Dataset | Restarts / 64 | Branches / Root | D_c Cutoff | Recovery (1st → 2nd) | First Solves on Branch | Branch-Only Solves (N=1 / N≥2) |
|---|---|---|---|---|---|---|---|
| Qwen3-8B | LCBv6 | 25.5 | 1.33 | 0.7% | 9.9% → 1.9% | 72.8% (83/114) | 14.0 / 1.0 (of 131) |
| Qwen3-8B | AIME-25 | 25.6 | 1.03 | 6.7% | 40.6% → 6.4% | 88.4% (38/43) | 2.5 / 0.5 (of 30) |
| Qwen3-8B | HMMT-25 | 24.1 | 1.27 | 7.6% | 18.5% → 2.2% | 89.7% (35/39) | 3.8 / 0.2 (of 30) |
| Qwen3-4B-Thinking | LCBv6 | 18.2 | 2.15 | 4.1% | 7.8% → 1.6% | 62.9% (66/105) | 11.7 / 0.7 (of 131) |
| Qwen3-4B-Thinking | AIME-25 | 20.2 | 1.63 | 13.9% | 58.5% → 17.3% | 98.4% (61/62) | 9.0 / 0.0 (of 30) |
| Qwen3-4B-Thinking | HMMT-25 | 16.9 | 1.87 | 14.8% | 33.2% → 6.7% | 95.2% (59/62) | 9.2 / 0.5 (of 30) |
Post-Turn-0 gains come primarily from prefix branching: 80.5% (342/425) of post-Turn-0 first solves occur on branched turns, which on math reach a 5x–29x higher per-turn solve rate than restarts at roughly half the generated tokens per turn. Unbranched restarts alone cannot exceed Independent Pass@64 (61.7% vs. TTEL's 85.0% on 4B AIME-25; 47.5% vs. 69.2% on 4B HMMT-25; 64.6% vs. 71.0% on 8B LCBv6), and Branch-Only Solves counts problems per seed where Turn 0 and all restarts fail.
Restarts still help when an error precedes truncating at the selected spike position or the initial algorithm fails (as on LCBv6, where switching algorithms needs a clean slate) and seed new roots for later branches; never restarting (Multi-Turn Full Reasoning) trails TTEL by 10–15 points. Capping depth earlier at N = 1 or N = 2 branches on Qwen3-8B gives similar Pass@64 across all three benchmarks but costs 9%–34% more generated tokens, since baseline filtering already restarts once a prefix stops yielding spikes.
More details can be found here: [Restart frequency, refinement depth, and branching vs. restarting (Appendix J.8, Tables 13 and 14)]
Thanks for your helpful review, and for acknowledging the strengths of our work. We'll address your concerns point-by-point. We hope these responses are satisfactory and would be grateful if you would consider increasing your score.
Effective recovery does not require pinpointing the exact point where reasoning fails; branching at or before the earliest error is sufficient to prevent flawed reasoning from propagating.
To evaluate this directly, we take one independently sampled failed trajectory per question from Qwen3-8B across LCBv6, AIME-25, and HMMT-25, and use gemini-4-argon as an annotator to locate the earliest reasoning error (135 successfully annotated trajectories). TTEL's branching prefixes precede the error in 71.1% of trajectories, compared to 47.4% for uniform random (+23.7 percentage points):
This advantage holds consistently across both code (+24.2 pp) and math (+22.2 pp), confirming that TTEL reliably identifies clean prefixes before flawed reasoning sets in.
More details can be found here: [Error localization accuracy and interpretation (Appendix J.3, Figure 12)]
To isolate the effect of error localization, TTEL (random prefix depth) selects a branching token uniformly at random from the latest rollout (excluding any prior prefix), while keeping the search tree, feedback prompt, and restart dynamics identical to TTEL. We evaluate this ablation with Qwen3-8B on LiveCodeBench v6 (LCBv6) and HMMT-25 up to k = 64 rollouts (1 seed) against standard TTEL (3 seeds on LCBv6, 4 on HMMT-25):
Across both benchmarks, TTEL achieves higher Pass@64 (+2.3 pp on LCBv6, +7.5 pp on HMMT-25) while using 15%–17% fewer tokens, confirming that targeted error localization improves both accuracy and token efficiency.
More details can be found here: [Random branching (Appendix J.3, Figure 13)]