acceptodds
Under review as a conference paper at ICLR 2027

Measuring the stability of autoregressive generation: Paired latent trajectories under benign input perturbation

Abstract

Replacing a single word in a factual question can change an instruction-tuned model's answer even when the question's meaning is preserved. Across six open-weight models, 8 to 34 % of our audited substitutions change the answer. We introduce the Latent Trajectory Assay (LTA) to measure how such changes propagate through generation. LTA pairs hidden-state trajectories for clean and perturbed prompts in two conditions. Letting the perturbed prompt generate its own continuation shows whether the answer changes. Under teacher forcing, the clean continuation is fixed through both prompts, which isolates the internal response to the input change. In all six models the internal divergence, and how well it predicts a changed answer, grow with depth. Most perturbed trajectories drift back toward their clean counterparts, but so do meaningless edits such as a change of letter case, and what a real substitution adds is a larger displacement that outlasts that baseline. Scored only on positions after the answer, so that it cannot read the flip itself, divergence predicts answer flips at AUROC 0.68 to 0.76 and beats an output-probability score from the same pass on every model by point estimate. A human audit of 360 substitutions rejects 14 percent as meaning-changing. The signal survives their removal (AUROC 0.68 to 0.78), and it flags the rejected substitutions on its own (AUROC 0.76). For screening fragile prompts before a rewording is known, black-box resampling beats every hidden-state statistic, and an erosion score from the teacher-forced pass improves that screen on five of six models without hidden-state access. LTA's contribution is therefore not a better detector but a controlled, depth-resolved measurement of the internal response to a specific input change. We release the instrument, capture archive, and tooling.

Primary area:
interpretability and explainable AI
Keywords:
hidden-state dynamics, input perturbation robustness, teacher forcing, representation stability, hallucination detection, uncertainty estimation, large language models, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yHNmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Latent Trajectory Assay (LTA), a method for measuring how a model’s hidden states change when one word in a factual question is replaced with a synonym. It compares the hidden states for the original and reworded prompts, by comparing the hidden states when providing the model’s original answer after the initial or the reworded prompt. Experiments on six open-weight models use 500 TriviaQA questions. Experiments find that the changes in hidden states can help predict if a model will change its answer with an AUROC of 0.71 to 0.79.

Critical strengths

The paper provides an initial analysis into how a single-word, meaning-preserving change to a question affects a model’s hidden states during generation, across six open-weight models from 4B to 70B parameters.

Critical weaknesses
  1. The trajectory from the beginning of the token to the end of the sequence does not yield much difference. From just the first teacher forced position, the AUROC for answer changes is already 0.69 to 0.77. This merely rises 0.71 to 0.79, indicating that there is not much benefit from analyzing the whole sequence of hidden states from the fully generated answer.
  2. The paper does not filter for questions that the model would naturally answer correctly. Many of these flips may reflect model guessing instead of sensitivity to the wording, as we see that 35 percent of prompts give more than one answer for LLaMA-3.1-8B. Flips could occur because models switch from one wrong answer to the next. If the model's internal knowledge is not well calibrated for the question, these flips are meaningless. Additionally, they only use the answer the model originally produces, not the gold, correct answer.
  3. The experiments are way too narrow. Only a small portion of TriviaQA is used in its entirety, and there is only a single perturbation type. Questions requiring more reasoning or multi-step processes would be needed to further analyze how "trajectories" are affected with perturbations.
  4. Model question answering is not really observed here. The main way the paper measures hidden states is with the correct answer provided in the input after the initial or perturbed prompt. These hidden states provide information on how a model processes text, not how it would autoregressively generate its own answer.
Questions for authors

How would this LTA analysis work on mathematical reasoning questions, or commonsense reasoning questions?

How would uncertainty or confidence calibration be factored into this analysis, given that model random guessing could provide more information on why these flips occur for difficult questions?

2 comments

Thank you for the careful reading. Two of the four weaknesses rest on a description of the method that differs from what the paper does, so we address those first.

The forced continuation is the model's own answer, and question answering is observed directly (weakness 4, and the last sentence of weakness 2). No gold answer is ever placed in the input. Each pair is run under two conditions (Section 4.4). In the free-run condition the model generates greedily from the reworded question, and a flip means that this answer differs from its own answer to the original. In the teacher-forced condition, the tokens pushed through the model are the model's own clean generation, right or wrong. The forced pass is a control that holds the token path fixed so that state differences are attributable to the input change alone, and its relevance is empirical: divergence measured there predicts flips observed in free generation. Gold answers enter only when we grade correctness (Section 4.1), and the paper reports correctness prediction separately from flip prediction (Section 6).

Flips on questions the model does not know (weakness 2). The composition you suspect is real. On four of six models most flips are changes between two wrong answers (29 to 70% of flips across the grid). Appendix L reports this, together with the result that the score separates correct-to-wrong flips from non-flips at AUROC 0.76 to 0.87 and leads the confidence baseline there on all six models. We agree this belongs in the main text.

We ran the filter you describe. Restricted to questions the model answers correctly, the flip rate falls to 5 to 15% and the score's flip AUROC is 0.76 to 0.81, against 0.65 to 0.71 for the paper's best confidence baseline (a lead of 0.06 to 0.15, with a bootstrap interval that excludes zero on three models). Restricted further to questions answered correctly and stable under resampling (all five temperature-0.7 samples agree with the greedy answer), the flip rate is 4 to 11% and the AUROC is 0.82 to 0.91 against 0.67 to 0.74, a lead of 0.10 to 0.18 whose interval excludes zero on all six models. (The Llama-3.3-70B cell holds only 21 flips.) On resample-unstable questions the flip rate is 35 to 52% and the score is weaker, 0.63 to 0.73, matching your intuition about where most flips come from; the measurement is sharpest where the model is not guessing. Across pairs the score correlates with resampling disagreement at Spearman 0.03 to 0.27, lower than the confidence baseline does on every model, so it is not a restatement of sampling uncertainty. These strata were defined after the reviews and we report them as post hoc.

Calibration and guessing (question 2). The resampling control of Section 4.4 is the paper's handle on this, and Section 6.1 reports that for screening prompts in advance, black-box resampling disagreement beats every hidden-state statistic. Your question also prompted a sharper confidence baseline than the generation-averaged ones in the submission, the clean model's log-probability of its own first answer token. On the full set it reads 0.70 to 0.76 and the score's lead over it is 0.02 to 0.08, with an interval that excludes zero on one model only, so over all questions a well-chosen confidence read is close to the paired measurement. On the questions answered correctly and stably it falls to 0.59 to 0.66 while the score holds at 0.82 to 0.91, a lead of 0.17 to 0.30 that excludes zero on all six models (with the same caution for the 70B cell). The two are complementary: resampling and confidence find the questions on which the model is unsure, and the paired measurement finds the flips that remain where it appears sure.

The trajectory adds little over its first position (weakness 1). We agree, and the paper states it (Section 6 and Appendix L): much of the predictive signal is present at the prompt boundary, and what the trajectory supplies is the depth-resolved relaxation and its timing, not a better flip score. The claim is that a single benign substitution produces a displacement that is present before any answer token, grows with depth, and contracts along the generation in 92 to 97% of pairs, and that the relation of this displacement to answer flips validates it as a meaningful measurement.

Scope (weakness 3, question 1). We agree. One dataset, one answer format, and one perturbation type is the main limitation of the study, and Appendix M states it. We chose the answer-first short-answer format because it gives a clean flip label and a short shared forced path. Mathematical and multi-step questions need a reasoning-first format, in which the forced path is long and the answer arrives late, and we do not claim that the present results transfer to it.

Thank you for your response. I think this submission is potentially promising, and I would recommend the authors to add more experiments and address the scope. I will be maintaining the score.

Reviewer_UnSbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper is an exploratory paper on LM's reliability. The authors propose the Latent Trajectory Assay (LTA) method to measure how such changes propagate through generation and use it to study 3 questions: how generation trajectories respond to a single measured, intent-preserving substitution, whether the size of the response predicts which answers will change, how it compares against confidence read.

Critical strengths

This is a good exploratory work with a novel interesting evaluation or measurement framework with a well-motivated experimental design. Another strong point is the extensive series of experiments over a set of popular model families. Third, the authors put careful attention to pertrubation validity.

Critical weaknesses

The main concern is that at this point the trajectory study is limited to short-answer factual questions (answer followed by a short justification), which imposes quite a rigid structure. As a result the patters found in the papers could be specific to this particular question type. This as a result limits the work applicability.

Questions for authors
  1. Have you considered whether your findings hold true for another question type?
  2. Could LTA be extended to other data formats?
1 comment

Thank you for the review and for the two questions, which point at the extension we consider most important.

Other question types. We have not tested them, and we do not claim that the patterns transfer. The short-answer factual format was chosen because it gives an unambiguous flip label and a short shared forced path. Two findings are likely to be specific to the format, and the paper flags both. The secondary bump in the divergence curve is the handoff from the answer to the justification (Appendix L), and the advantage of the state over the output distribution at the first position is observed under a format that fixes the first output token (Section 6).

Extending the method to other formats. The instrument needs only a clean generation, a perturbed prompt, and a forced pass, so it applies to any format. What changes is the validation target. In multi-hop and reasoning-first formats the forced path is longer and the answer arrives late, so the questions become where along the reasoning the displacement grows or decays and whether divergence before the answer predicts a changed answer. In formats without a short answer, the flip label has to be replaced by a graded semantic comparison.

One result obtained during this discussion bears on your concern about applicability. On questions the model answers correctly and consistently under resampling, the score's flip AUROC rises to 0.82 to 0.91 across the six models, against 0.67 to 0.74 for the best confidence baseline (details in our response to Reviewer yHNm). The measurement is most informative on the questions that confidence and resampling would pass as safe.

Reviewer_JFMhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method called Latent Trajectory Assay (LTA) aimed to measure how the answer of an LLM a question changes if one token in the input prompt is replaced with a synonym (e.g. largest - biggest). It uses and compares three ways to extract model internal state information, and original runs with teacher-forced ones, and explores whether the information can be used to predict answer flip. Figure 1 (right) shows that the three states provide quiet comparative results to predict answer flip, with the early measure performing slightly worse, as makes sense (less context).

The paper evaluates a range of LLMs from small to 70B Llama size. It uses questions sampled from one benchmark, TriviaQA, and samples 500 questions, out of which 494 end in the final items. The synonym replacement is done with a "counter-fitted word-vector space". Of the 494 samples, 360 are audited by one human, and a second examines a subset of 120 items.

A key point that is left open is the key motivation: perturbing input prompts is know to perturb outputs. The paper suggests to look at model internals, but does not motivate why or in which cases going to (more costly) model internal analysis is enough, instead of just examining the output ("thinking" trace) of the model. So, adding a key motivation of what additional benefit the model internal brings, and a comparison to surface level information, would make it a much stronger paper.

Overall a paper that is quiet fine to follow but rather narrow in methodology.

The construction of the perturbed prompt is overly complicated, and its audit is surprisingly complex, too.

Critical strengths

The paper is quiet easy to follow, but the motivation for the method could be stronger.

Critical weaknesses
  • narrow in methodology: a sampled subset of items from a single dataset (TriviaQA) is used for evaluation. This means results are specific to this dataset and may not generalize. An evaluation on datasets beyond TriviaQA would strenghten the paper.

  • the human audit of the 360 items is overly complex. it is surprising that not all 494 items were checked, and especially that from the 360 items only 120 are checked by a second annotator. given the simplicity of the task (replacing one token with a synonym) the human audit is very narrow and should be done on the entire item set with both annotators.

  • Synonym replacement. A rather specific and single method is used for synonym replacement. A comparison to a model-free variant such as WordNet would strengthen the paper.

  • Relation to prior work. The components of the methods are provided by prior work, i.e., SPUQ perturbing prompts, and S2R2 doing teacher-forced decoding. The proposed method is thus a combination of existing works - the motivation and novelty of the solution should be better explained.

  • Overall, a narrow analysis paper that seems not yet ready for publication at a leading ML conference in my view.

Questions for authors

Please elaborate on the choise of items for annotation, and why this procedure.

2 comments

Thank you for the review. We respond to each point in turn.

Why look inside the model.

The value of this approach has two major components: the theoretical and the practical.

On the theoretical side, we explore the relationship between internal model activations and inconsistent model outputs (i.e. answers that flip). We find that the difference in response to a benign rewording is present in the state before any answer token is produced (which is somewhat tautological, as you note), this response difference grows with depth through the model (an interesting result), and it decays as the token generation sequence continues (meaning the difference on the first teacher-forced token contains most of the signal). These results provide potentially actionable insight for modelers seeking to enforce increased stability of model responses.

On the practical side, this also yields an unsupervised signal at inference time of when the model is producing output that is likely to be inconsistent. In section 6, we demonstrate that this signal is better than model confidence at predicting answer flips and in section 8 we demonstrate that this signal is statistically indistinguishable from the best black-box score when used alone, and when the two are used in conjunction they predict flips even more strongly. For a practitioner, the signals this paper measures arrange themselves along a cost frontier. At zero additional generations, single-pass confidence is the only free signal, and it answers the correctness question while saying little about stability. At a five-sample budget, black-box resampling becomes the best standalone advance screen for fragile prompts, and the strongest screen at that budget stacks it with the log-odds erosion of a teacher-forced pass, still without touching a hidden state. Our instrument occupies a unique point on this cost frontier: One clean generation plus one teacher-forced forward pass per variant gives a per-inference, per-depth account of how the model responds to a specific rewording, with no labels and no second model. This is an offline stability audit at the effective cost of 2 inferences (rather than 5+).

It is important to note that reasoning traces, as the reviewer suggested, are not always available and reasoning has even been found to impair performance on some constrained tasks like natural language inference. In our setting the models were run without one (an answer followed by a short justification, with thinking mode disabled where a model has one), so there is no trace to inspect. Evaluating such reasoning traces for stability or validity generally requires inference passes from a powerful model (which, practically speaking, may be the same model that produced them and thus prone to the same stability issues). Our approach offers a numeric metric for predicting output stability for a relatively cheap, fixed computational cost.

Human audit design (weakness 2, and your question). The 360 items are a stratified sample, 90 in each cell of language-model audit category by flip status, drawn so that the rejection rate can be estimated in every cell and then weighted back to the 1,376 audited variants (Section 4.3 and Appendix D). The audit exists because the task is less simple than it appears: 44 percent of raw candidates from the synonym space are not meaning-preserving in context, and after the language-model audit a human annotator still rejects 14 percent. The second annotator rated a 120-item overlap to measure agreement. We agree that single annotation of the remaining 240 items is a limitation. The paper says so and reports the results under every acceptance rule, and removing the human-rejected variants lowers the pooled flip AUROC by 0.03.

Source of synonyms. A lexical-database source such as WordNet would be a useful second operator. The conclusions do not depend on the quality of the candidate source, since every candidate is audited in context wherever it came from, and one finding of the paper is that the standard source needs that audit.

Relation to prior work. SPUQ perturbs prompts and reads the dispersion of outputs, with no internal measurement. S$^2$R$^2$ compares decoder states between clean and perturbed inputs under a shared forced target, which is the structural core of our forced condition, and the related-work section says so. It does this at segment-mean granularity and uses the result as a fine-tuning loss. Neither validates the internal quantity against observed answer changes, audits the perturbation for meaning preservation, or resolves the response by position and depth. The contribution is the controlled measurement and its validation, not a new perturbation.

Single dataset. We agree, and say the same to the other reviewers. This is the main limitation of the study.

I thank the authors for the response, providing a motivation and acknowledging the paper's current limitations. I agree with my fellow reviewer that the paper contains a promising direction. Given the narrow evaluation, I however remain with my current evaluation, but recommend the authors to strengthen the experiments, motivation, and resubmit the paper in future.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.