- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces Latent Trajectory Assay (LTA), a method for measuring how a model’s hidden states change when one word in a factual question is replaced with a synonym. It compares the hidden states for the original and reworded prompts, by comparing the hidden states when providing the model’s original answer after the initial or the reworded prompt. Experiments on six open-weight models use 500 TriviaQA questions. Experiments find that the changes in hidden states can help predict if a model will change its answer with an AUROC of 0.71 to 0.79.
Critical strengths
The paper provides an initial analysis into how a single-word, meaning-preserving change to a question affects a model’s hidden states during generation, across six open-weight models from 4B to 70B parameters.
Critical weaknesses
- The trajectory from the beginning of the token to the end of the sequence does not yield much difference. From just the first teacher forced position, the AUROC for answer changes is already 0.69 to 0.77. This merely rises 0.71 to 0.79, indicating that there is not much benefit from analyzing the whole sequence of hidden states from the fully generated answer.
- The paper does not filter for questions that the model would naturally answer correctly. Many of these flips may reflect model guessing instead of sensitivity to the wording, as we see that 35 percent of prompts give more than one answer for LLaMA-3.1-8B. Flips could occur because models switch from one wrong answer to the next. If the model's internal knowledge is not well calibrated for the question, these flips are meaningless. Additionally, they only use the answer the model originally produces, not the gold, correct answer.
- The experiments are way too narrow. Only a small portion of TriviaQA is used in its entirety, and there is only a single perturbation type. Questions requiring more reasoning or multi-step processes would be needed to further analyze how "trajectories" are affected with perturbations.
- Model question answering is not really observed here. The main way the paper measures hidden states is with the correct answer provided in the input after the initial or perturbed prompt. These hidden states provide information on how a model processes text, not how it would autoregressively generate its own answer.
Questions for authors
How would this LTA analysis work on mathematical reasoning questions, or commonsense reasoning questions?
How would uncertainty or confidence calibration be factored into this analysis, given that model random guessing could provide more information on why these flips occur for difficult questions?
2 comments
Thank you for your response. I think this submission is potentially promising, and I would recommend the authors to add more experiments and address the scope. I will be maintaining the score.
Thank you for the careful reading. Two of the four weaknesses rest on a description of the method that differs from what the paper does, so we address those first.
The forced continuation is the model's own answer, and question answering is observed directly (weakness 4, and the last sentence of weakness 2). No gold answer is ever placed in the input. Each pair is run under two conditions (Section 4.4). In the free-run condition the model generates greedily from the reworded question, and a flip means that this answer differs from its own answer to the original. In the teacher-forced condition, the tokens pushed through the model are the model's own clean generation, right or wrong. The forced pass is a control that holds the token path fixed so that state differences are attributable to the input change alone, and its relevance is empirical: divergence measured there predicts flips observed in free generation. Gold answers enter only when we grade correctness (Section 4.1), and the paper reports correctness prediction separately from flip prediction (Section 6).
Flips on questions the model does not know (weakness 2). The composition you suspect is real. On four of six models most flips are changes between two wrong answers (29 to 70% of flips across the grid). Appendix L reports this, together with the result that the score separates correct-to-wrong flips from non-flips at AUROC 0.76 to 0.87 and leads the confidence baseline there on all six models. We agree this belongs in the main text.
We ran the filter you describe. Restricted to questions the model answers correctly, the flip rate falls to 5 to 15% and the score's flip AUROC is 0.76 to 0.81, against 0.65 to 0.71 for the paper's best confidence baseline (a lead of 0.06 to 0.15, with a bootstrap interval that excludes zero on three models). Restricted further to questions answered correctly and stable under resampling (all five temperature-0.7 samples agree with the greedy answer), the flip rate is 4 to 11% and the AUROC is 0.82 to 0.91 against 0.67 to 0.74, a lead of 0.10 to 0.18 whose interval excludes zero on all six models. (The Llama-3.3-70B cell holds only 21 flips.) On resample-unstable questions the flip rate is 35 to 52% and the score is weaker, 0.63 to 0.73, matching your intuition about where most flips come from; the measurement is sharpest where the model is not guessing. Across pairs the score correlates with resampling disagreement at Spearman 0.03 to 0.27, lower than the confidence baseline does on every model, so it is not a restatement of sampling uncertainty. These strata were defined after the reviews and we report them as post hoc.
Calibration and guessing (question 2). The resampling control of Section 4.4 is the paper's handle on this, and Section 6.1 reports that for screening prompts in advance, black-box resampling disagreement beats every hidden-state statistic. Your question also prompted a sharper confidence baseline than the generation-averaged ones in the submission, the clean model's log-probability of its own first answer token. On the full set it reads 0.70 to 0.76 and the score's lead over it is 0.02 to 0.08, with an interval that excludes zero on one model only, so over all questions a well-chosen confidence read is close to the paired measurement. On the questions answered correctly and stably it falls to 0.59 to 0.66 while the score holds at 0.82 to 0.91, a lead of 0.17 to 0.30 that excludes zero on all six models (with the same caution for the 70B cell). The two are complementary: resampling and confidence find the questions on which the model is unsure, and the paired measurement finds the flips that remain where it appears sure.
The trajectory adds little over its first position (weakness 1). We agree, and the paper states it (Section 6 and Appendix L): much of the predictive signal is present at the prompt boundary, and what the trajectory supplies is the depth-resolved relaxation and its timing, not a better flip score. The claim is that a single benign substitution produces a displacement that is present before any answer token, grows with depth, and contracts along the generation in 92 to 97% of pairs, and that the relation of this displacement to answer flips validates it as a meaningful measurement.
Scope (weakness 3, question 1). We agree. One dataset, one answer format, and one perturbation type is the main limitation of the study, and Appendix M states it. We chose the answer-first short-answer format because it gives a clean flip label and a short shared forced path. Mathematical and multi-step questions need a reasoning-first format, in which the forced path is long and the answer arrives late, and we do not claim that the present results transfer to it.