- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether recent changes in a language model’s intermediate predictions can predict disagreement with a later output. The authors introduce Trajectory-to-Go (T2G), which summarizes two successive updates in output distributions and hidden states using magnitude and direction features. The main predictor, Risk-T2G, combines two regression heads to forecast terminal KL divergence. Experiments on recurrent-depth and conventional transformers report improvements in error ranking over confidence-based controls, together with corpus-transfer and state-augmentation results. The paper also evaluates KL-constrained computation allocation and introduces Task-T2G, a separate answer-change classifier for early exiting on MMLU. The authors conclude that recent trajectory information can help distinguish locally stable predictions from predictions that are close to a later output.
Critical strengths
-
The paper provides a concrete formulation for predicting disagreement with a later model output. The reported results offer some evidence that recent trajectory features improve terminal-error ranking over matched confidence controls in the evaluated settings.
-
The evaluation includes several model families and corpus-transfer checks, providing useful information about where the features help. The paper also acknowledges several limitations, including model-dependent feature contributions and the lack of a demonstrated latency advantage over learned controls.
Critical weaknesses
-
The distinction from prior methods is not sufficiently discussed.
-
CATs (Schuster et al., 2021) already learns intermediate-to-final prediction agreement from prediction history, but it is not discussed.
-
The paper discusses CALM (Schuster et al., 2022) in Section 2 and provides a CALM-inspired state baseline for the main KL prediction task in Appendix C.3. However, it does not sufficiently explain the relationship between Task-T2G and CALM’s learned consistency classifier.
-
-
The experiments do not fully establish the method’s contribution.
-
The specific contribution of the proposed geometric representation is not adequately isolated. The forecasting evaluation lacks a generic history predictor with the same prediction target, observation window, and comparable capacity. Improvements over confidence predictors demonstrate that trajectory information is useful, but do not establish the contribution of this particular geometric encoding relative to other ways of representing the same history.
-
The contribution of the complete Task-T2G feature combination is unclear. On Ouro-2.6B in Table 15, Task-T2G has a higher AURC (10.90) than inverse margin (10.13), linear confidence (10.28), the variant without the gate (10.48), and the variant without hidden features (10.62). The authors acknowledge model-dependent feature contributions and distinguish exit coverage from AURC. Nevertheless, the complete combination provides no ranking advantage over these simpler alternatives in the reported point estimates.
-
The practical contribution of Task-T2G is also limited by the reported comparisons. The authors explicitly acknowledge that they have not demonstrated a latency advantage over the learned control. On Ouro-1.4B in Table 14, the simpler inverse-margin policy also has a lower answer-change rate (1.46% versus 1.86%), higher accuracy (66.21% versus 66.11%), and greater time savings (14.23% versus 13.02%) than Task-T2G. Higher exit coverage therefore does not establish a better accuracy–consistency–latency trade-off. The paper does not report paired confidence intervals for these baseline differences. Moreover, omitting fixed pass 3 leaves the contribution of adaptive stopping relative to reduced fixed computation unresolved.
-
Questions for authors
-
Supplementary materials. The package described in the reproducibility statement on page 10 is absent from the submitted materials. Please provide the stated code, prompt hashes, saved fits and thresholds, evaluation scripts, results, and table checks in an anonymous artifact accessible to reviewers.
-
Relationship to CATs. Please compare T2G with CATs (Schuster et al., 2021) in model architecture, prediction target, historical inputs, and predictor design.
-
Relationship to CALM. Please compare Task-T2G with CALM’s learned consistency classifier (Schuster et al., 2022) in architecture, input features, and prediction target.
-
History baseline. Please add a CATs-inspired history predictor using ordinary prediction statistics and projected hidden states from the same three intermediate readings, without T2G’s engineered trajectory features.
-
Fixed-depth comparison. Please add fixed pass 3 to the MMLU evaluation on both Ouro models.
7 comments
Response to Reviewer PS7B
W2. Is the full Task-T2G combination needed?
Task-T2G is one calibrated score; the paper does not claim that each of its inputs is necessary, and Appendix F now gives the paired intervals. Control minus Task-T2G (AURC ×1,000; positive favors Task-T2G):
| Control | Ouro-1.4B | Ouro-2.6B |
|---|---|---|
| Inverse margin | 2.00 [.55, 3.43] | −.77 [−1.89, .36] |
| Linear confidence | 1.60 [.37, 2.83] | −.62 [−1.65, .46] |
| Confidence + gate | .41 [−.21, 1.08] | −.19 [−1.20, .87] |
| Without hidden motion | .37 [−.14, .91] | −.28 [−.98, .40] |
| Without gate | .54 [−.12, 1.20] | −.42 [−.85, −.04] |
On Ouro-1.4B, Task-T2G ranks better than inverse margin and linear confidence (intervals above zero) and is comparable to the three closer variants. On Ouro-2.6B (AURC 10.90) the controls have slightly lower point AURC; four of the five intervals include zero, and the variant without the gate ranks better by .42 [.04, .85].
W2. Inverse margin, paired intervals and latency
Task-T2G minus the comparator at the calibrated operating points, in percentage points (Appendix F, Table 20, new):
| Comparator | Model | Exit rate | Answer change | Time saved |
|---|---|---|---|---|
| Inverse margin | Ouro-1.4B | 4.98 [3.52, 6.54] | .39 [−.10, .98] | −1.21 [−2.32, .01] |
| Inverse margin | Ouro-2.6B | 3.32 [1.76, 4.88] | .49 [−.10, 1.07] | .20 [−.67, 1.22] |
| Confidence + gate | Ouro-1.4B | −.29 [−1.46, .78] | −.10 [−.59, .39] | −1.30 [−2.14, −.49] |
| Confidence + gate | Ouro-2.6B | −.59 [−1.95, .78] | −.29 [−.78, .20] | .35 [−.40, 1.17] |
Task-T2G exits on more questions than inverse margin at both scales (intervals above zero), with comparable answer change. Its scoring cost offsets the extra exits, so the time saved is comparable to inverse margin at both scales (both intervals include zero). On Ouro-1.4B the point estimates of answer change and time favor inverse margin, accuracy differs by one question in 1,024, and confidence + gate is faster. The MMLU study supports a different claim from latency: the forecast can be fitted to a task decision and calibrated (about 98% of pass-4 answers preserved at 75–81% exits). The gains we claim for trajectory inputs are in forecasting later change (Tables 1–4) and in iterations saved under a KL limit against learned confidence (Table 5); on MMLU the revision states that confidence and the gate remain competitive.
Exit and change use 1,024 questions; timing uses a 128-question subset at batch 1 with scoring included.
Q5. Fixed pass 3
Now in Table 6, on the same 1,024 questions (%):
| Model | Policy | Answer change | Exit | Accuracy |
|---|---|---|---|---|
| Ouro-1.4B | Fixed pass 3 | 11.72 | 100 | 66.21 |
| Task-T2G | 1.86 | 74.61 | 66.11 | |
| Fixed pass 4 | 0 | 0 | 66.50 | |
| Ouro-2.6B | Fixed pass 3 | 7.52 | 100 | 74.41 |
| Task-T2G | 1.66 | 80.76 | 74.80 | |
| Fixed pass 4 | 0 | 0 | 75.00 |
Fixed pass 3 changes 7.52–11.72% of pass-4 answers, four to six times the 2% target that Task-T2G meets. It exits on every question, so it saves more time: 23.45% and 24.07%, against 12.58% and 18.75% for Task-T2G in the same timing run (Appendix F; the paired run behind Table 6 did not include fixed pass 3). Its accuracy is similar because the changed answers gain and lose correctness about equally (a net loss of 3 and 6 questions). The adaptive policies are for the case where agreement with the full model's answer is the requirement.
Thank you for the clarifications, supplementary material, and additional experiments. The history baseline and fixed-pass-3 comparison address the missing experiments in my review. The following concerns remain.
Q1.
The supplementary package contains the stated categories of materials. However, the paired-timing script requires per-question latency arrays and a provenance file that are absent from the package. The table checks verify agreement with saved summaries, but these missing inputs prevent independent recomputation of the reported runtime intervals. Please provide the missing inputs or complete instructions for generating them and rerunning the analysis.
Q2/Q3.
The response clarifies the methodological differences. Based on this comparison, I find Task-T2G particularly close to CATs and CALM in its prediction objective and stopping procedure (Schuster et al., 2021; Schuster et al., 2022). Its trajectory inputs and recurrent setting distinguish the implementation, but a direct experimental comparison is needed to establish their contribution to answer-consistency prediction. The CATs-inspired history baseline and CALM-inspired state probe are evaluated on terminal-KL prediction and do not address this comparison. Please compare Task-T2G with a CATs-inspired history consistency classifier and a CALM-style current-state classifier, using the same pass-3/pass-4 answer-disagreement target and data splits.
Q4.
The neural comparisons support the contribution of the trajectory representation across the three evaluated models, with relative EAURC reductions of 5.8–12.6%. These results evaluate the input representations under the same neural architecture and should be distinguished from the improvements obtained by the original regression-based Risk-T2G predictor. For that predictor, the relative reductions are approximately 1.7% on Huginn and 7.7% on Ouro-1.4B, while the Ouro-2.6B difference has an interval containing zero. The additional experiments address the missing baseline, but the gains vary substantially across models, leaving it unclear whether regression-based Risk-T2G offers a consistent advantage across models.
W2.
Task-T2G has not established a clear overall advantage over simpler adaptive policies. Its local improvements in risk ranking and exit coverage provide limited evidence of a better practical stopping policy once answer consistency and execution cost are considered. This limits the contribution of the task-specific stopping extension and reduces the breadth of the benefits demonstrated by the paper.
Response to Reviewer PS7B
Thank you for the detailed review; the controls you asked for are now in the paper. Tables and sections refer to the revised PDF. Table 14 of the submission is now Table 6 and Table 15 is Table 19. Brackets are paired 95% intervals from 2,000 resamples of whole prompts or questions.
Q1. Supplementary package
The anonymous
T2G_supplementary.zipaccompanies the revision. It contains the code and evaluation scripts, prompt hashes, saved fits and thresholds, and the results of the looped-model forecasting, control and stopping experiments. Its table check (python check_tables.py, standard library only) confirms that 318 printed entries of Tables 1, 2, 5, 6, 12, 16, 19 and 20 match the saved results, and lists the result file and field behind each entry. Per-example arrays are recomputed from the public models with the included scripts. The transfer and layer-wise tables (Tables 3 and 4) are not part of the check.W1, Q2, Q3. CATs and CALM
Risk-T2G differs from CATs in the target (a continuous distance between distributions, not agreement of a task prediction), in the setting (successive passes of a recurrent model, which share its output head) and in how history enters (sizes and directions of two updates, not the states and predictions themselves). Task-T2G has the same kind of target as CATs and CALM and, like them, leaves the model unchanged; it differs in its trajectory inputs and in the recurrent setting. Section 2 now discusses both.
Q4 and W2. A CATs-inspired history predictor
Table 2 (new). The generic predictor sees the same three readings without engineered trajectory features: entropy, maximum probability and margin at each reading, both successive KL values, and the three hidden states after a random projection and PCA fitted on training prompts. It has the same target, losses, penalties and two-head rule as Risk-T2G, and 119 terms per head. EAURC ×1,000, lower is better:
This predictor uses the hidden coordinates linearly, whereas CATs combines them in an MLP. We therefore also fitted the same two 32-unit networks to each input set, with weight decay and training length chosen by four-fold validation within the fitting prompts:
The T2G inputs give lower EAURC in five of the six comparisons with intervals above zero (1.7% and 7.7% under regression; 5.8%, 7.2% and 12.6% under the networks, which have 1,026 parameters against 3,522). Under regression on Ouro-2.6B the two are comparable (the interval includes zero). These controls test the representation at the same target, window and comparable capacity; they do not reproduce the complete CATs system with its trained heads.