acceptodds
Under review as a conference paper at ICLR 2027

Testing Final-State Sufficiency in Multimodal Graph Reasoning

Abstract

A model answering questions about a changing world must track what is true after the updates. When a question depends only on the final state, a model that relies on that state alone should give the same answer distribution for any history reaching it. We call this requirement final-state sufficiency. Accuracy can miss violations when history changes the likelihoods of valid answers without changing the preferred answer. We therefore compare the correct-answer margin, the correct answer's log-likelihood advantage over its strongest valid alternative. Final-state sufficiency requires this margin to be invariant across histories reaching the same state. Our matched graph histories reverse the same two updates while fixing the final graph and answer task; questions about unchanged properties adjust for general order sensitivity. Both Qwen3-VL and Gemma violate the required margin equality in the complete graph on four nodes. A separate held-out study extends the Gemma finding to four larger, non-complete graph families in conditions where the model can answer the graph questions and follow the updates. Correct endpoint answers can therefore conceal history dependence. Dynamic-reasoning evaluations should test invariance across state-equivalent histories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reasoning, graph reasoning, state tracking, final-state sufficiency, order sensitivity, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tkMVOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether multimodal models produce consistent answer distributions across different update histories that reach the same final graph. By reversing two edge-addition instructions, the authors find changes in the correct answer’s likelihood margin in two models, with further evidence on larger graphs for Gemma. The results motivate evaluating history invariance alongside accuracy.

Critical strengths
  1. The matched-history design carefully controls the initial graph, final graph, and question while varying only update order.Only Gemma passes calibration and supports the findings beyond the four-node setting; whether the results extend to Qwen remains unresolved.
  2. The paper clearly separates behavioral findings from mechanistic interpretations and transparently reports unsuccessful controls.
Critical weaknesses
  1. Only Gemma passes calibration and supports the findings beyond the four-node setting; whether the results extend to Qwen remains unresolved.
  2. Without reporting how often both histories yield correct answers, it is unclear how frequently the proposed test reveals history dependence that accuracy would miss.
Questions for authors

How were nonconstant update-span RDMs obtained when the questions appear after the updates? Please clarify the actual prompt order, activation extraction positions, and whether the differences exceed numerical noise.

2 comments

Thank you ever so much for your encouraging feedback and thoughtful review!

In response to your comments, we have revised the paper and uploaded a new PDF with every change marked in blue; the changes are summarized in a table at the end of this response. To make the paper easier to read, the revision also adds a worked example of the task (Sec 3.1), a table of the experiments (Table 22), and a notation table (Table 23).

Q1: Whether the findings beyond the four-node setting extend to Qwen (W1).

A1: Thank you for raising this point. We respectfully clarify that the Qwen checkpoints show the history dependence beyond four nodes as well (Table 26, Appendix I).

Where the Qwen checkpoints pass calibration, they show the effect as well. Section 4.3 gives a verdict only when a panel passes calibration for every input format and question type (Appendix F.2). With the same thresholds applied to each input format and question type separately, the control-adjusted order effect on shortest-path questions with image input is negative, with a 95% interval below zero, in all 5 cases where Qwen3-VL passes and in 6 of 8 for Qwen3.5. Across the three models, the interval is below zero in 18 of the 21 cases that pass (Sec 4.3, Table 26).

Appendix I adds a second model on graphs of up to 10 nodes. This study of 26 further graph settings compares a graph-changing update with a state-preserving instruction that reaches the same final graph. With text input, Qwen3.5-9B shows a stable effect in 7 of its 12 eligible settings for edge additions (Tables 16, 17, and 19).

Prompted by this comment, we have added the condition-level results of all 30 panels to the revised PDF (Table 26, new Appendix L), and Section 4.3 now reports them together with a short summary of Appendix I.

Q2: How often both histories yield correct answers, and how often the test reveals dependence that accuracy misses (W2).

A2: Thank you for this suggestion. We respectfully clarify that both histories yield the correct answer in 181 of the 192 matched pairs for Gemma and in 147 for Qwen3-VL, and that the test reveals history dependence in three of the four conditions where every pair is answered correctly (Sec 4.2).

For Gemma, accuracy is identical under the two orders, yet the test detects the dependence. Gemma answers 183 of the 192 prompts correctly under each order, so accuracy shows no history dependence, while the test on the correct-answer margin gives θ^=3.671\widehat\theta = 3.671 [2.581, 4.855] (Sec 4.2).

Where every pair is answered correctly, the test still finds the dependence. In four conditions both orders yield the correct answer in all 48 pairs: text input with both question types for Gemma, and degree questions with both input formats for Qwen3-VL. Accuracy records no difference there, yet the 95% interval of the order effect excludes zero in three of them. One is the largest condition-level effect of the core study, −3.623 [−4.131, −3.123] for Gemma on shortest-path questions with text input (Table 5).

Accuracy can register a difference in only a small fraction of pairs. Correctness can differ between the orders only where the top answer changes. In the two core models and five further Qwen and Gemma checkpoints, this happens in at most 21 of the 192 pairs for any checkpoint, and all seven show the history dependence (Table 24). Qwen3.8-27B, for example, answers 361 of the 384 prompts correctly and changes its top answer in only 1 pair, yet its order effect on shortest-path questions with text input is −0.547 [−0.766, −0.328].

Prompted by this comment, we have added to Section 4.2 of the revised PDF how often both orders yield the correct answer, and the conditions in which every pair is answered correctly.

Q3: Prompt order, extraction positions, and how the update-span RDMs are obtained (Question).

A3: Thank you for this careful question. We respectfully clarify that each prompt gives the initial graph, the two updates, and then the question, and activations are read at the tokens of the second update and at the last token before the answer, exactly as documented (Appendices B.1 and C.2).

At the second update, the RDMs are set by the prompt, not by the question. The question has not been read at those tokens, so the states there are shared across the question conditions (derivation in Appendix G).

The paper now reports the agreement just before the answer. There the question has been read, and cross-model agreement is close to zero: .060 for text and −.024 for images (Sec 4.5, Table 10). The behavioral results and the replacement results of Section 4.5 are computed from answer margins and do not depend on these matrices.

Following your question, Sections 1, 3.5, 4.5, and 5 of the revised PDF now report the agreement just before the answer, and Appendix G derives the value at the second update.

For convenience, we collect below the changes made in the revised PDF in response to your review; all are marked in blue.

LocationChange in the revised PDFAddresses
Section 4.3; new Appendix LThe condition-level results of all 30 panels (Table 26); the seven checkpoints (Table 24); a short summary of Appendix IQ1, Q2
Section 4.2How often both orders yield the correct answer; the conditions in which every pair is answered correctlyQ2
Sections 1, 3.5, 4.5, 5; Appendix GRelational agreement reported just before the answer; the value at the second update derived in Appendix GQ3
Section 3.1; new Appendix KA worked example of the task; a table of the experiments (Table 22); a notation table (Table 23)Presentation

Thank you again for your support of this work and for questions that improved the paper.

Reviewer_KPuvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors address the following evaluation problem. A language model is given an object, a sequence of edits to it, and a question about the result. The paper asks whether the model's answer depends only on the final state of the object, or also on the order of the edits. The test bed is a small graph: the initial graph comes as a text adjacency matrix or an image, the edits are textual edge insertions, and the question asks for a node degree or a shortest-path distance. The authors claim that accuracy alone cannot detect such order dependence: the correct answer can stay top-ranked while its log-likelihood lead over the wrong candidates shifts with the edit order. They also claim that state-tracking benchmarks test whether a model recovers the current state, not whether it relies on that state alone.

To measure such dependence, the authors compare pairs of inputs that apply the same two edits in opposite orders, so that both reach the same final graph. Instead of accuracy, the authors score each complete candidate answer from a fixed set by its log-likelihood. They then take the gap between the correct answer and the best wrong answer as a measure of how strongly the model favors the correct answer (the authors call this the correct-answer margin). They compute how much this gap changes between the two edit orders for a question affected by the edits. They then subtract the same change for a control question about an unaffected property. This adjustment removes additive prompt effects but not every source of order sensitivity. The result is further differenced across the two question types and the two presentation formats, giving one number per instance. A nonzero value refutes the invariance, whereas a zero value would not establish it. Three supporting analyses are added: an extension to larger non-complete graphs, a diagnostic that inserts a description of the final graph before the question, and an inspection of hidden activations in the two models.

Experiments on 24 held-out instances show that the aggregated statistic is positive with intervals excluding zero for Qwen3-VL-8B and Gemma-4-12B, which the authors read as a violation in both models. The extension rejects the invariance in all seven settings in which Gemma passes a competence check. No Qwen model passes the check, so the authors' own criterion for cross-model breadth is not met. Inserting a final-graph description reduces a related contrast in three of four cases. Swapping activations before the answer has a large effect on the aggregated statistic, but the authors' own selectivity controls fail. However, the paper does not show empirically what the margin reveals beyond accuracy, and one internal result appears inconsistent with the documented prompt design. All data are synthetic graphs constructed by the authors.

Critical strengths

S1. The authors propose a controlled test that changes only the order of two edits and keeps everything else fixed (initial graph, final graph, question, candidate answers, presentation). The design and the statistics were fixed before the held-out evaluation. This turns a conceptual distinction, answering correctly versus relying only on the final state, into a testable one.

S2. For Gemma, the extension beyond the four-node graph rejects the invariance in all seven eligible family and size combinations, which is the strongest evidence in the paper. The authors also report transparently which of their own criteria failed (the selectivity controls, the cross-model breadth criterion, and the 23 ineligible extension panels), so the scope of each claim is easy to assess.

Critical weaknesses

W1. The empirical evidence should be reinforced in order to claim history dependence as a general phenomenon.

W1.1. The authors should add graph settings and models beyond K4 and the two tested checkpoints, or state the finding as specific to them. The authors perform their experiment under a situation where the correct answer is one constant per question type, as all 24 instances are variants of the same four-node graph that differ only in node names, question wording, and image layout (Appendix B.1). The core result for both models (Section 4.2) therefore rests on one case, and it is questionable whether it generalizes to other tasks. Two checkpoints of 8B and 12B likewise cannot support a general claim. Each model's nonzero θ refutes invariance on its own, but the two violations arise from different conditions (Section 4.2, lines 337 to 342): Gemma's from distance questions on text input, Qwen3-VL's from a sign reversal between text and image input. The two results therefore do not add up to one shared pattern. Other open-source multimodal models and sizes (for example Qwen3-VL-4B, Qwen2.5-VL-32B, Qwen3.5 at 4B and 27B, another Gemma size) and other state-tracking tasks would be needed; each of these models fits the one 96 GB device per run reported in Appendix C (lines 1193 to 1196).

W1.2. The authors should explain why both Qwen checkpoints fail the competence screen of the extension (Section 4.3), and should report the size of the Gemma effects. Only Gemma passes the screen, so the only evidence beyond K4 comes from one model. Table 9 gives Qwen's minimum direct-task accuracy as 0.000 in 11 of 20 settings without saying where or why, so a genuine inability cannot be separated from an artifact of the scoring protocol (line 1181), which also produced the core Qwen3-VL margins. For Gemma, seven rejections of a zero effect do not give its size, and Figure 5 has intervals without numbers.

W2. The paper does not state what the test adds over prior order-sensitivity evaluations.

The authors should state what their test can diagnose that prior order-sensitivity tests cannot. Swapping the two update sentences is the same manipulation as premise reordering [4]. The authors distinguish their test from [4] only by the intermediate graph states (lines 120 to 122). The real differences are three: the final graph is verified to be the same, the margin is computed over complete answers, and a control question is subtracted. The authors never say what new conclusion these three differences allow. Without such a statement the finding reduces to a known one, that output likelihoods depend on statement order, under a new name. The authors should also show that the dependence is specific to the intermediate states, for example by recomputing θ under a meaning-preserving wording change that leaves those states unchanged ("between D and A" to "between A and D"), scored the same way. Both manipulations can move θ. The distinction from [4] is nominal if they move it by similar amounts. The reordered-description condition in Table 1 belongs to a different experiment (Sections 3.4 and 4.4) and has no reported results. Consistency testing [1, 2] and dynamic-graph benchmarks [3] belong in the discussion as context.

W3. Table 10 reports a result that the documented prompt design should not produce.

The authors should explain how Table 10 obtains .964 at the second-update tokens. As I understand the design, no correlation should be computable there. The question comes after both updates (Table 2; Appendix B.1, lines 1027 to 1038), so a causal decoder yields identical hidden states at those tokens across the four question conditions of a history, up to numerical noise. Each vector of Eq. (14) is H21 minus H12 for one condition, and both terms are shared across conditions, so the four vectors coincide, the six distances are zero, and the inclusion rule (line 1394) excludes every instance. One of three cases must hold: an undocumented prompt difference, a capture position other than the documented one, or numerical noise with a shared rank pattern. Until then, the Section 4.5 comparison cannot be relied on. The behavioral margins are unaffected.

Other comments

O1. Problem framing. The authors should state the model's task as input and output in the main text, with a worked example. Section 3.1 defines the invariance and the margin, but the candidate sets and the teacher-forced scoring first appear in Appendices B.4 and C.1. The general recommendation in the conclusion (line 485) is untested outside short synthetic graph tasks.

O2. Reporting. The authors should add intervals to the condition-level effects and give a reference point for the size of θ. Table 5 gives means only and Figure 2b has no intervals. A reference point could be the number of answer changes across the 192 matched prompt pairs. Neither addition would change the counterexample, which does not require an answer to change (Appendix E.1, lines 1549 to 1550), but both would let a reader judge the size of the effect.

O3. Presentation (the basis for the Presentation score). The authors should add an overview of the experiments and a notation guide. The abstract defines two terms (lines 015 to 018) and then uses ten more author-defined terms before any definition ("matched controls" 020, "independent calibration" 024, "prespecified interaction" 081, "relational agreement" 099, among others). Appendix D holds the normalization, eligibility, and selectivity rules that give Eqs. 1 to 6 their meaning, and the paper defines at least eight derived quantities. Minor: state the logarithm base and the Gemma configuration for the core study; Eq. (7) restates Eq. (4); Appendix I could be condensed.

O4. Final-state diagnostic. The authors should say plainly that the diagnostic does not test whether an explicit final state removes the order dependence of H1. H2 and H3 (Sections 3.4 and 4.4) use a one-update design from different initial graphs, with attenuation taken after averaging (Gemma: 3.459 to −0.719). The paper discloses these limits.

O5. Pre-answer intervention. The authors should present the replacement result (Section 4.5) as a check rather than a finding. At 80% depth the last prompt token feeds the logits almost directly, so a large effect is unsurprising, although not guaranteed, and the authors themselves report that the selectivity controls fail (Table 13, Appendix H.3). The paper already scopes the result to influence (Appendix D.5, line 1437).

O6. Reproduction materials. The authors should identify where the evidence files can be accessed. The reproducibility statement and Appendix J.3 refer to them, but I could not identify a supplementary archive or anonymized repository, so W3.1 and W1.3 could not be checked.

[1] Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. "Beyond accuracy: Behavioral testing of NLP models with CheckList." Proceedings of ACL 2020, 4902–4912.

[2] Elazar, Y., et al. "Measuring and improving consistency in pretrained language models." Transactions of the ACL 9 (2021), 1012–1031.

[3] Zhang, Z., et al. "LLM4DyG: Can large language models solve spatial-temporal problems on dynamic graphs?" Proceedings of KDD 2024.

[4] Chen, X., Chi, R. A., Wang, X., and Zhou, D. "Premise order matters in reasoning with large language models." Proceedings of ICML 2024, PMLR 235, 6596–6620.

Questions for authors

Please refer to the weaknesses.

6 comments

Thank you very much for such a careful and thorough review. It is clear that you read the paper closely, and we are grateful for the time this took.

Following your suggestions, we evaluated five further checkpoints and ran the wording control you proposed: the order dependence appears in all seven checkpoints, and the main result holds under the wording control. In response to your comments, we have revised the paper and uploaded a new PDF with every change marked in blue; the changes are summarized in a table at the end of this response.

Q1: Graph settings and models beyond K4K_4 and the two tested checkpoints, the constant correct answer, and whether the two violations form one pattern (W1.1).

A1: Thank you for raising this point. We respectfully clarify that the paper tests five graph families beyond K4K_4 (Sec 4.1, Sec 4.3), and that, following your suggestion, we have evaluated five further checkpoints, and all seven show the order dependence. The violations also share one pattern, in the two core models with text input and in the three models beyond K4K_4, and an order effect tied to the constant correct answer cancels against the matched control.

The order dependence is not specific to two checkpoints. We scored Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B, Gemma-4-E4B, and Gemma-4-26B-A4B on the same held-out instances with the same protocol. Every one of them shows it, as do the two original checkpoints (Holm-adjusted p≤.027p \le .027 for each, Table 24). As you note, a violation can arise in different conditions, so we test the four combinations of input format and question type jointly, with the test that the extension already uses (Appendix F.3), and this choice was fixed before the new checkpoints were scored. The dependence is present in an accurate model as well: Qwen3.8-27B answers 361 of the 384 prompts correctly and changes its top answer in only 1 of the 192 matched pairs, yet its control-adjusted order effect on shortest-path questions with text input is −0.547 [−0.766, −0.328].

The violations share one pattern. In the core study with text input, the two models show the same pattern: the order effect is positive for degree questions and negative for shortest-path questions, with every interval excluding zero (Table 5). Beyond K4K_4, the held-out extension uses five non-complete graph families, and Gemma shows the order dependence in all seven panels in which it is eligible (Sec 4.3). On these graphs the order effect on shortest-path questions with image input has the same direction in all three models: it is negative, with a 95% interval below zero, in 18 of the 21 cases in which a model meets the calibration thresholds (see A2).

An effect tied to the constant answer cancels. The update-relevant question and its matched control have the same correct answer and the same candidate answers (Appendices B.1 and B.4), so an order effect tied to a particular answer is removed when the control is subtracted.

Prompted by this comment, we have added the seven checkpoints to the revised PDF (Table 24, new Appendix L), and Section 4.2 now summarizes them in one sentence.

Q2: Why both Qwen checkpoints fail the competence screen, which leaves the evidence beyond K4K_4 resting on one model, and the size of the Gemma effects (W1.2).

A2: Thank you for this point. We respectfully clarify that both Qwen checkpoints fail the screen because their accuracy is below the threshold, not because of the scoring protocol, and that where they meet the calibration thresholds, they show the order effect as well. Gemma's effects reach several log units, and the numbers behind Figure 5 are now tabulated.

The scoring protocol is not the cause. The calibration questions were also answered by greedy generation, and accuracy from the generated answers equals accuracy from complete-answer scoring in 223 of the 240 strata (Table 25). In every panel, both Qwen checkpoints fall below the accuracy threshold on shortest-path questions with text input, under scoring and under generation. The screen requires a panel to pass for every input format and question type (Appendix F.2), so no Qwen panel receives a verdict.

Where the Qwen checkpoints pass, they show the order effect as well. When the thresholds of Appendix F.2 are applied to each input format and question type separately, the order effect on shortest-path questions with image input is negative, with a 95% interval below zero, in all 5 cases where Qwen3-VL passes (every estimate below −1.6), in 6 of 8 for Qwen3.5, and in 7 of 8 for Gemma (Table 26). The panel-level verdicts of Section 4.3 are unchanged.

Gemma's effects reach several log units. The table gives the control-adjusted order effect for shortest-path questions in the seven eligible panels, with 95% intervals.

PanelText inputImage input
Component bridging, 6−1.368 [−1.837, −0.899]−1.903 [−2.578, −1.256]
Component bridging, 8−1.033 [−1.504, −0.580]−2.692 [−3.438, −1.928]
Cycle closure, 6−0.295 [−1.598, 1.289]−3.963 [−5.266, −2.643]
Cycle closure, 8−1.828 [−2.651, −1.012]−5.465 [−6.293, −4.721]
Symmetry-breaking chord, 6−2.498 [−3.078, −1.899]−2.150 [−2.465, −1.834]
Symmetry-breaking chord, 8−2.252 [−2.942, −1.551]−3.516 [−4.047, −3.004]
Hub attachment, 60.268 [−0.543, 1.172]−1.250 [−1.641, −0.875]

With image input the effect lies between −1.250 and −5.465 and its interval is below zero in all seven panels. With text input the interval is below zero in five.

Prompted by this comment, we have added to Appendix L of the revised PDF the calibration accuracies under both protocols (Table 25) and the condition-level contrasts of all 30 panels with intervals (Table 26). Figure 5 now refers to Table 26 for its numbers.

Q3: What the test adds over prior order-sensitivity evaluations such as premise reordering, and the wording control you proposed (W2).

A3: Thank you for this point. We respectfully clarify that the three differences you list are what allow a conclusion that reordering premises does not support: a model can give the correct answer for the final graph while its support for that answer depends on how the graph was reached. Following your suggestion, we also ran the wording control: the interaction stays above zero for both models, and for Gemma the order swap and the rewording do not move it by similar amounts.

The test sees what accuracy misses and removes generic order effects. The final graph is the same under both orders, so any difference in answer likelihoods is a violation of Eq. 1. The margin is computed over complete answers, so the test registers a change when the top answer stays the same: for Gemma the top answer differs between the two orders in only 4 of the 192 matched pairs. Accuracy, the measure used by Chen et al. (2024), records such cases as no effect. The matched control is asked after the same two sentences and subtracted, so an effect of sentence order that is common to both questions cancels.

The result holds under the wording control you proposed. We exchanged the two node names in every update sentence ("between D and A" to "between A and D") and scored the held-out prompts in the same way. Averaged over the two wordings, the interaction stays above zero for both models: θ^\widehat\theta is 3.007 [1.867, 4.171] for Gemma and 3.802 [1.646, 6.083] for Qwen3-VL. For Gemma the two manipulations do not move it by similar amounts: the order swap moves it by 3.671 under the original wording and by 2.342 under the new one, and the rewording by −0.706 [−2.255, 0.873].

The control separates two kinds of dependence. For Qwen3-VL the rewording also moves the interaction (Table 27). The rewording leaves the final graph unchanged, as the order swap does, so this is a second departure from Eq. 1. In Gemma the dependence is specific to the order of the updates, and in Qwen3-VL it extends to how each update is written.

Prompted by this comment, we have added the wording control to the revised PDF (Table 27, Appendix L) and a paragraph to Section 2 that states the three differences and what each allows, with Chen et al. (2024), behavioral invariance testing [1], consistency measurement [2], and dynamic-graph benchmarks [3] as context.

Q4: The value at the second-update tokens in Table 10 (W3).

A4: Thank you for this careful derivation. We respectfully clarify that the prompt and the capture position are exactly as documented, and that the value at the second-update tokens is determined by the prompt design, the third of your three cases. The behavioral margins are unaffected, as you state, and so are the replacement results of Section 4.5, which are computed from margins.

At the second-update tokens, the question has not yet been read. As you derive, the states there are therefore shared across the question conditions, and the correlation takes a value set by the prompt (derivation in Appendix G).

The paper now reports the agreement just before the answer. There the question has been read, and cross-model agreement is close to zero: .060 for text and −.024 for images (Sec 4.5, Table 10).

Your derivation led us to revise Sections 1, 3.5, 4.5, and 5 and Appendix G of the revised PDF accordingly.

Reviewer_RHMtOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether multimodal models give different levels of support to the correct answer when the same final graph is reached through different update histories. The authors construct paired inputs that share the initial graph, final graph, and question but apply the same two edge additions in opposite orders. They measure history dependence on degree and shortest-path questions using the correct-answer margin, i.e., the log-likelihood gap between the correct answer and its strongest valid alternative, adjusted with matched control questions about unchanged properties. On 24 matched K4 instances, both Qwen3-VL-8B and Gemma-4-12B show significant update-order effects. In an extension to larger non-complete graphs, only Gemma passes the capability checks, and its seven eligible settings also show the effect.

Critical strengths

S1. The two histories contain the same updates, differing only in their order, while keeping the initial graph, final graph, and question consistent. By comparing the relative support for the correct answer, the paper can identify changes that are easily overlooked when considering only whether an answer is correct. This design provides a concrete method for examining whether the model is affected by irrelevant historical information.

S2. The authors report results that do not support the expectations, including that the activation intervention did not pass the selectivity criterion, that all extended settings of the two Qwen models failed the predefined capability checks, and that Gemma's correct textual descriptions did not significantly reduce the predefined overall contrast. These observations help readers assess the scope of applicability of the conclusions.

Critical weaknesses

W1. The core study is based on 24 label, question, and layout variants with a fixed graph topology. The updates consist of only two steps, and the queries cover only two types: degree and shortest path. Although the authors add experiments on larger, non-complete graphs, only seven settings for Gemma pass the capability checks. Therefore, the existing evidence for more general topologies remains concentrated on a single model. The current results are not yet sufficient to establish how prevalent this phenomenon is in longer update sequences and more complex state-tracking tasks.

W2. To my knowledge, some existing studies have discussed models' sensitivity to input order. This paper observes an effect of order in graph update tasks, but after reading it, I still do not fully understand the main differences between this work and existing studies, or why this finding is important. I hope the authors can use more intuitive examples to explain what problems this study can help us understand or solve.

Questions for authors

What new insights does this paper provide beyond existing studies on input-order sensitivity? What are the practical implications of this finding? Why is it necessary to use graph-based settings?

5 comments

Thank you for your time and feedback. We appreciate your comments very much.

We note that the review rates the soundness concerns as minor or fixable and recognizes both the matched-history design (S1) and the transparent reporting of results (S2). Each point below is answered with specific results, and we give the exact location of each so that it can be checked directly. We have also evaluated five further checkpoints, and together with the two original ones, all seven show the update-order effect (A1).

In response to your comments, we have revised the paper and uploaded a new PDF with every change marked in blue; the changes are summarized in a table at the end of this response.

Q1: Scope of the evidence: a fixed topology, two-step updates, two query types, and a single model on more general topologies (W1).

A1: Thank you for raising this point. We respectfully clarify that the evidence on more general topologies is not confined to a single model: the Qwen models show the effect on the larger graphs as well, in the conditions where they pass the capability checks (Table 26), and Appendix I reports a study of 26 graph settings that includes a second model on graphs of up to 10 nodes (pages 40–46). To address the breadth of models directly, we also scored five further Qwen and Gemma checkpoints on the held-out instances of the core study, and all seven checkpoints show the update-order effect (Table 24).

On larger graphs, the effect is not confined to one model. The held-out extension tests five non-complete graph families at 6 and 8 nodes, with 320 matched pairs (Appendix F.1). Gemma shows the effect in all seven settings that pass the capability checks, which cover four graph families and both sizes (largest adjusted pp of 8.9×10−48.9\times10^{-4}, Sec 4.3, Table 7). When the checks are applied to each input format and question type separately, the Qwen models show the effect as well. For shortest-path questions with image input, the effect is negative, with a 95% interval below zero, in 18 of the 21 cases that pass across the three models, including all five for Qwen3-VL (Sec 4.3, Table 26).

Appendix I adds 26 settings and a second model on graphs of up to 10 nodes. This earlier exploratory study covers seven graph families at 4, 6, 8, and 10 nodes (Table 15) and evaluates Gemma and Qwen3.5-9B. It compares a graph-changing update with a state-preserving instruction that reaches the same final graph, for edge additions and for edge removals. With text input, Gemma shows a stable effect in 13 of its 17 eligible settings for additions and in 13 of 17 for removals, and Qwen3.5-9B in 7 of its 12 for additions (Tables 16, 17, and 19).

Two updates are the shortest reversible history, and the two question families cover a local and a path-level property. Final-state sufficiency is defined for a history h=(u1,…,uL)h=(u_1,\ldots,u_L) of any length (Sec 3.1, Eq. 1). The core study uses two updates because this is the shortest history whose order can be reversed. The two question families are a local property, node degree, and a path-level property, shortest-path length, each paired with a control question on an unchanged property (Sec 3.2).

So that these results are visible in the main text rather than only in Appendix I, we have added to Section 4.3 of the revised PDF a short paragraph summarizing them, and a scope table (Table 22, new Appendix K) that lists each study's graph families, sizes, models, kind of update, and held-out or exploratory status. The seven checkpoints and the condition-level results are reported in the new Appendix L (Tables 24 and 26).

Q2: Difference from existing studies of input-order sensitivity, and the new insights (W2, Question 1).

A2: Thank you for this question. We respectfully clarify that, as the revised PDF now states explicitly (Sec 1, Sec 2, Appendix A.5), existing studies ask whether rearranging a prompt changes the output, whereas we ask whether a model's answer depends only on the final graph. We reverse two updates because this keeps the final graph identical while changing how it was reached. The new insight is that a model can answer correctly under both orders while its support for the correct answer depends on the order, and that this dependence can be changed just before the answer.

The two orders pass through different graphs and end at the same one. Section 2 states that prompt-order studies "reorder demonstrations, answer options, or context blocks" (Lu et al., 2022; Pezeshkpour & Hruschka, 2024; Ok & Lee, 2026), whereas "Our histories instead pass through different intermediate graph states before reaching the same endpoint". After the first update the two histories describe different graphs, and after the second they describe the same one. Appendix A.5 adds that reversing the updates "changes the trajectory through the world while fixing the endpoint, question, answer set, node labels, and answer order".

Equal answers are a requirement here, and general order effects are subtracted. The question depends only on the final graph, so equal answer likelihoods under the two histories are a requirement (Eq. 1), and any difference is a violation of it. Each question about a property the updates change is paired with a control question about a property they leave unchanged, asked after the same two updates, and the estimand subtracts the control (Sec 3.2). An order effect common to both questions cancels in this subtraction. What remains is above zero for both models: θ^=5.781\widehat\theta = 5.781 for Qwen3-VL and 3.6713.671 for Gemma (Table 4). For Gemma this holds although both orders yield the correct answer in 181 of the 192 matched pairs (Sec 4.2).

Stating the final graph explicitly reduces a related history effect, and the effect can be changed just before the answer. The reduction appears in three of four model–format comparisons (Sec 4.4). Inside the models, replacing the activations just before the answer with those from the opposite order reproduces .955–.960 of the effect for Qwen3-VL and .802–.804 for Gemma, and the same replacement at the second update leaves the effect unchanged (Sec 4.5).

So that this difference is visible on the first page, we have added to Section 1 of the revised PDF one sentence that states it next to the cited prompt-order studies, and to Section 2 a paragraph on what the test allows one to conclude that reordering premises (Chen et al., 2024) does not.

Q3: Why the finding is important, with an intuitive example and its practical implications (W2, Question 2).

A3: Thank you for the suggestion. We respectfully clarify that the finding matters for a simple reason: in practice the order in which updates arrive is often arbitrary, and the paper shows that this order alone changes how strongly a model favors the correct answer. The paper also shows that stating the current state explicitly reduces a related history effect (Sec 4.4).

An intuitive example. Suppose an assistant keeps a shared calendar and two colleagues each add a meeting. Which entry arrives first is a matter of chance, and the calendar is the same afterwards. Any question about the calendar should therefore receive the same answer, with the same confidence, in both cases. The paper tests this property on graphs, where two orders of updates provably give the same result (see A4), and finds that it does not hold for either model (Sec 4.2).

What it helps us understand: a correct answer does not show that a model relies only on the current state. The effect can be present even when the answer is correct, because it concerns how strongly the model prefers the correct answer over its closest competitor (Sec 1). For Gemma on shortest-path questions with text input, this preference differs between the two orders by 3.099 log units on average, a factor of about 22, compared with 0.524 for a control question (Table 5).

What it helps solve: evaluations can compare both orders, and systems can keep an explicit record of the current state. For evaluation, a benchmark with updates can add this check whenever two orders lead to the same outcome: present both orders and compare how strongly the model favors the correct answer (Sec 5). For system design, the paper shows that stating the current state explicitly helps: it reduces a related history effect in three of four comparisons, for example from 12.865 to 1.410 for Qwen3-VL (Sec 4.4, Table 6). This is consistent with keeping an explicit record of the current state in systems that receive many updates, as in Statler (Yoneda et al., 2024; Appendix A.2).

Following your suggestion, we have added this example to Section 1 of the revised PDF, and one sentence on these two uses to Section 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.