- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TrajUQ, a method for predicting the probability of failure of an LLM reasoning trajectory from a small number of short continuations. The method combines the mean of branch-level failure predictions with information about their distribution, using a kernel regression model. The authors evaluate the approach on three QA datasets and 13 model configurations. They report consistent improvements in Brier score over a learned mean baseline, as well as some improvements in offline answer selection.
Critical strengths
The problem of predicting reasoning failure before generation is completed is interesting and practically relevant. Such predictions could potentially be used to stop unsuccessful reasoning trajectories early and reduce inference costs.
The paper presents a systematic evaluation across multiple models, with several baselines and ablations. The improvements over the learned mean baseline are relatively small but consistent. I also appreciate the attempt to evaluate the trade-off between prediction quality and observation budget.
Critical weaknesses
My main concern is the limited experimental setting. The evaluation uses only 180 questions from each of three QA datasets. In particular, there are no mathematical reasoning benchmarks, such as GSM8K, MATH, or AIME, or other tasks involving long reasoning trajectories. These seem to be the most natural applications of the proposed method. Without such experiments, it is difficult to assess its practical value.
Another limitation is that the method requires supervised training for each model–task pair. The paper does not investigate transfer to unseen tasks, which limits its applicability to general-purpose LLM inference.
I also find the baseline comparisons not fully convincing. Several existing uncertainty estimation methods are adapted to this specific setting and have different supervision requirements. Although the authors include a baseline based on second-order statistics, I would like to see comparisons with simpler supervised predictors using the same branch scores and training data, especially in the two-branch setting.
Finally, the reported reduction in observation tokens does not directly demonstrate a reduction in end-to-end inference cost. The answer selection experiments are offline and do not establish whether the method can improve practical reasoning efficiency.
Questions for authors
-
Can the authors evaluate TrajUQ on mathematical reasoning benchmarks such as GSM8K, MATH, or AIME, preferably using models with long reasoning trajectories? This is my main concern. I would particularly like to see whether the predicted failure risk can be used for early stopping or restarting reasoning, and whether this improves accuracy or reduces inference costs. A convincing positive result on mathematical reasoning tasks would be the main factor that could change my assessment of the paper.
-
How well does the method generalize across tasks? For example, can a probe and readout trained on two datasets be applied to a third without additional training?
-
Could the authors compare TrajUQ with simpler supervised models, such as an MLP or gradient boosting on the mean, variance, and covariance of branch scores, using the same training data and hyperparameter tuning budget?
-
Could the authors clarify the selection procedure for the 180 questions per dataset and explain why this sample size was chosen? It would also be useful to discuss how the results might depend on the selection of questions.
5 comments
Rebuttal (2/3)
Table 2. Online stopping and restarting. StrategyQA: Qwen2.5-72B-Instruct, 180 questions averaged over multiple random seeds. MATH-500: Qwen3.8-27B, 500 questions. Tokens denote total generated tokens per question.
| Policy | StrategyQA / Accuracy (%) | StrategyQA / Tokens | MATH-500 / Accuracy (%) | MATH-500 / Tokens |
|---|---|---|---|---|
| Single generation | 70.0 | 132.7 | 84.8 | 1697 |
| Random restart | 68.8 | 130.5 | 83.2 | 1674 |
| Restart using mean | 71.5 | 127.4 | 85.4 | 1651 |
| Restart using TrajUQ | 72.3 | 123.6 | 86.2 | 1605 |
W2
Another limitation is that the method requires supervised training for each model–task pair. The paper does not investigate transfer to unseen tasks, which limits its applicability to general-purpose LLM inference.
Q2
How well does the method generalize across tasks? For example, can a probe and readout trained on two datasets be applied to a third without additional training?
Response. We retain the readout advantage on all three held-out QA tasks without target-task fitting.
-
Source-only fitting. With Qwen2.5-7B-Instruct, we trained on two QA tasks and evaluated the third, rotating the target. We fitted preprocessing, the probe, and the readouts and selected hyperparameters using source tasks, then froze the full pipeline. We used , , and for both readouts. We used target labels only for evaluation.
-
Transfer results. We reduce Brier by 0.0013–0.0092 across all three targets compared with the learned mean readout (Table 3).
Table 3. Measured transfer results: training on two tasks and freezing the full pipeline for the third.
| Target | Learned mean Brier | TrajUQ Brier | TrajUQ learned mean (95% CI) |
|---|---|---|---|
| MMLU-Pro | 0.2367 | 0.2354 | $--$0.0031, $-$0.0008] |
| StrategyQA | 0.2377 | 0.2309 | $--$0.0101, $-$0.0038] |
| TriviaQA | 0.2498 | 0.2406 | $--$0.0137, $-$0.0052] |
W3
I also find the baseline comparisons not fully convincing. Several existing uncertainty estimation methods are adapted to this specific setting and have different supervision requirements. Although the authors include a baseline based on second-order statistics, I would like to see comparisons with simpler supervised predictors using the same branch scores and training data, especially in the two-branch setting.
Q3
Could the authors compare TrajUQ with simpler supervised models, such as an MLP or gradient boosting on the mean, variance, and covariance of branch scores, using the same training data and hyperparameter tuning budget?
Response. We obtain lower Brier by jointly reading the mean and branch-score distribution, with the clearest gains at . At , MLP and GBDT improve on the second-order readout, while we further reduce Brier by and , respectively (Table 4).
-
Input representation. We give the first three readouts the same mean, variances, and covariance (O,v_{16},v_{64},v_{16,64}) (Appendix B.7, p. 17, lines 873–878). With TrajUQ, we retain and the empirical distribution of complete, centered score vectors across depths (p. 4, lines 195–202, Eq. 2). At , we thereby preserve distributional structure beyond second moments. At , second moments determine , and the losses are close (p. 23, lines 1239–1240).
-
Matched comparison. We use Qwen2.5-72B-Instruct / StrategyQA with 180 questions, , and . We fit and test each readout separately at and , sharing branch scores, independent continuation labels, and question-grouped folds (p. 5, lines 237–241; p. 7, lines 365–367). We search nine parameter settings per method and select parameters and a blend with the endpoint mean using inner validation folds.
Table 4. Measured supervised readouts on Qwen2.5-72B-Instruct / StrategyQA at matched branch budgets (Brier ).
| Predictor | Input features | ||
|---|---|---|---|
| Kernel ridge regression | Mean, variance, covariance | 0.1880 | 0.1873 |
| MLP | Mean, variance, covariance | 0.1878 | 0.1871 |
| GBDT | Mean, variance, covariance | 0.1883 | 0.1865 |
| Kernel ridge regression (TrajUQ) | Mean + full empirical distribution | 0.1876 | 0.1841 |
Rebuttal (3/3)
Q4
Could the authors clarify the selection procedure for the 180 questions per dataset and explain why this sample size was chosen? It would also be useful to discuss how the results might depend on the selection of questions.
Response to Q4. We randomly sampled 180 questions from each benchmark (Appendix B.1, p. 14, lines 723–728). We chose this size to keep continuation collection manageable across 13 model configurations.
We collected 1,053,000 continuations (). Across the three benchmarks and 13 model configurations, we used five prefix positions per question and sampled 30 continuations per prefix (p. 5, lines 252–261; p. 14, lines 729–735). We compare readouts on the same questions and use question-grouped splits and bootstrap intervals to assess sampling variability (p. 3, lines 145–148).
We also evaluated all 500 MATH-500 questions, retaining the readout advantage on both models (Table 1).
If we have addressed your concerns, we would be grateful if you could raise your score in light of the additional evidence.
Rebuttal (1/3)
Thank you for emphasizing mathematical reasoning, inference cost, and transfer across tasks. Your questions have helped us improve the paper's clarity and strengthen its empirical evaluation. Below, we address each of your concerns. We present results on mathematical reasoning, online restarting, transfer across tasks, and MLP/GBDT comparisons, and explain how we sampled the benchmark questions.
Critical Weaknesses and Questions For Authors
W1
Q1
W4
Response. On MATH-500, we improve accuracy by 1.4 percentage points and reduce total generated tokens by 5.4% with TrajUQ-guided restarting.
Mathematical reasoning. We evaluated all 500 MATH-500 questions, fitting on training folds and testing held-out questions using our original question-grouped protocol (p. 5, lines 229–235). We used one prefix per question with H=64 and S=16,64. We kept the K=10 probe, readouts, and shrinkage fixed when evaluating K=2. With ten branches, TrajUQ lowers Brier by 0.0059 and 0.0064 on the two models. With two branches, we retain reductions of 0.0022 and 0.0013 against the ten-branch learned mean (Table 1).
Online stopping and restarting. On MATH-500 with Qwen3.8-27B, we increase accuracy from 84.8% to 86.2% and reduce average total generated tokens from 1,697 to 1,605 (Table 2). Compared with mean-guided restarting, we gain 0.8 percentage points in accuracy and use 2.8% fewer tokens. After a 512-token prefix, we observe the live and one auxiliary branch for up to 64 tokens each (K=2, H=64, S=16,64). We allow at most one restart and use a common 65,536-token total cap for all four policies.
On StrategyQA with Qwen2.5-72B-Instruct, we increase accuracy from 70.0% to 72.3% and reduce total generated tokens by 6.9%. We also gain 0.8 percentage points in accuracy and use 3.0% fewer tokens than mean-guided restarting. Here, we observe both branches for up to 16 tokens after a 16-token prefix (K=2, H=16, S=8,16), with at most one restart and a 192-token total cap.
For each task, we share the scorer and observation budget between the two readouts. We fit readouts and select thresholds on training questions. We count every generated token, including auxiliary branches, discarded attempts, and restarts.
Original QA coverage. We evaluated MMLU-Pro, StrategyQA, and TriviaQA (p. 5, lines 252–253), prompting models to provide reasoning, including multi-hop reasoning for StrategyQA (p. 14, lines 705–714). We explain question selection and sample size in Q4.
Table 1. Measured MATH-500 results with task-specific fitting and held-out evaluation.