W1. The empirical evidence should be reinforced in order to claim history dependence as a general phenomenon.
W1.1. The authors should add graph settings and models beyond K4 and the two tested checkpoints, or state the finding as specific to them. The authors perform their experiment under a situation where the correct answer is one constant per question type, as all 24 instances are variants of the same four-node graph that differ only in node names, question wording, and image layout (Appendix B.1). The core result for both models (Section 4.2) therefore rests on one case, and it is questionable whether it generalizes to other tasks. Two checkpoints of 8B and 12B likewise cannot support a general claim. Each model's nonzero θ refutes invariance on its own, but the two violations arise from different conditions (Section 4.2, lines 337 to 342): Gemma's from distance questions on text input, Qwen3-VL's from a sign reversal between text and image input. The two results therefore do not add up to one shared pattern. Other open-source multimodal models and sizes (for example Qwen3-VL-4B, Qwen2.5-VL-32B, Qwen3.5 at 4B and 27B, another Gemma size) and other state-tracking tasks would be needed; each of these models fits the one 96 GB device per run reported in Appendix C (lines 1193 to 1196).
W1.2. The authors should explain why both Qwen checkpoints fail the competence screen of the extension (Section 4.3), and should report the size of the Gemma effects. Only Gemma passes the screen, so the only evidence beyond K4 comes from one model. Table 9 gives Qwen's minimum direct-task accuracy as 0.000 in 11 of 20 settings without saying where or why, so a genuine inability cannot be separated from an artifact of the scoring protocol (line 1181), which also produced the core Qwen3-VL margins. For Gemma, seven rejections of a zero effect do not give its size, and Figure 5 has intervals without numbers.
W2. The paper does not state what the test adds over prior order-sensitivity evaluations.
The authors should state what their test can diagnose that prior order-sensitivity tests cannot. Swapping the two update sentences is the same manipulation as premise reordering [4]. The authors distinguish their test from [4] only by the intermediate graph states (lines 120 to 122). The real differences are three: the final graph is verified to be the same, the margin is computed over complete answers, and a control question is subtracted. The authors never say what new conclusion these three differences allow. Without such a statement the finding reduces to a known one, that output likelihoods depend on statement order, under a new name. The authors should also show that the dependence is specific to the intermediate states, for example by recomputing θ under a meaning-preserving wording change that leaves those states unchanged ("between D and A" to "between A and D"), scored the same way. Both manipulations can move θ. The distinction from [4] is nominal if they move it by similar amounts. The reordered-description condition in Table 1 belongs to a different experiment (Sections 3.4 and 4.4) and has no reported results. Consistency testing [1, 2] and dynamic-graph benchmarks [3] belong in the discussion as context.
W3. Table 10 reports a result that the documented prompt design should not produce.
The authors should explain how Table 10 obtains .964 at the second-update tokens. As I understand the design, no correlation should be computable there. The question comes after both updates (Table 2; Appendix B.1, lines 1027 to 1038), so a causal decoder yields identical hidden states at those tokens across the four question conditions of a history, up to numerical noise. Each vector of Eq. (14) is H21 minus H12 for one condition, and both terms are shared across conditions, so the four vectors coincide, the six distances are zero, and the inclusion rule (line 1394) excludes every instance. One of three cases must hold: an undocumented prompt difference, a capture position other than the documented one, or numerical noise with a shared rank pattern. Until then, the Section 4.5 comparison cannot be relied on. The behavioral margins are unaffected.
Other comments
O1. Problem framing. The authors should state the model's task as input and output in the main text, with a worked example. Section 3.1 defines the invariance and the margin, but the candidate sets and the teacher-forced scoring first appear in Appendices B.4 and C.1. The general recommendation in the conclusion (line 485) is untested outside short synthetic graph tasks.
O2. Reporting. The authors should add intervals to the condition-level effects and give a reference point for the size of θ. Table 5 gives means only and Figure 2b has no intervals. A reference point could be the number of answer changes across the 192 matched prompt pairs. Neither addition would change the counterexample, which does not require an answer to change (Appendix E.1, lines 1549 to 1550), but both would let a reader judge the size of the effect.
O3. Presentation (the basis for the Presentation score). The authors should add an overview of the experiments and a notation guide. The abstract defines two terms (lines 015 to 018) and then uses ten more author-defined terms before any definition ("matched controls" 020, "independent calibration" 024, "prespecified interaction" 081, "relational agreement" 099, among others). Appendix D holds the normalization, eligibility, and selectivity rules that give Eqs. 1 to 6 their meaning, and the paper defines at least eight derived quantities. Minor: state the logarithm base and the Gemma configuration for the core study; Eq. (7) restates Eq. (4); Appendix I could be condensed.
O4. Final-state diagnostic. The authors should say plainly that the diagnostic does not test whether an explicit final state removes the order dependence of H1. H2 and H3 (Sections 3.4 and 4.4) use a one-update design from different initial graphs, with attenuation taken after averaging (Gemma: 3.459 to −0.719). The paper discloses these limits.
O5. Pre-answer intervention. The authors should present the replacement result (Section 4.5) as a check rather than a finding. At 80% depth the last prompt token feeds the logits almost directly, so a large effect is unsurprising, although not guaranteed, and the authors themselves report that the selectivity controls fail (Table 13, Appendix H.3). The paper already scopes the result to influence (Appendix D.5, line 1437).
O6. Reproduction materials. The authors should identify where the evidence files can be accessed. The reproducibility statement and Appendix J.3 refer to them, but I could not identify a supplementary archive or anonymized repository, so W3.1 and W1.3 could not be checked.
[1] Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. "Beyond accuracy: Behavioral testing of NLP models with CheckList." Proceedings of ACL 2020, 4902–4912.
[2] Elazar, Y., et al. "Measuring and improving consistency in pretrained language models." Transactions of the ACL 9 (2021), 1012–1031.
[3] Zhang, Z., et al. "LLM4DyG: Can large language models solve spatial-temporal problems on dynamic graphs?" Proceedings of KDD 2024.
[4] Chen, X., Chi, R. A., Wang, X., and Zhou, D. "Premise order matters in reasoning with large language models." Proceedings of ICML 2024, PMLR 235, 6596–6620.
Thank you ever so much for your encouraging feedback and thoughtful review!
In response to your comments, we have revised the paper and uploaded a new PDF with every change marked in blue; the changes are summarized in a table at the end of this response. To make the paper easier to read, the revision also adds a worked example of the task (Sec 3.1), a table of the experiments (Table 22), and a notation table (Table 23).
Q1: Whether the findings beyond the four-node setting extend to Qwen (W1).
A1: Thank you for raising this point. We respectfully clarify that the Qwen checkpoints show the history dependence beyond four nodes as well (Table 26, Appendix I).
Where the Qwen checkpoints pass calibration, they show the effect as well. Section 4.3 gives a verdict only when a panel passes calibration for every input format and question type (Appendix F.2). With the same thresholds applied to each input format and question type separately, the control-adjusted order effect on shortest-path questions with image input is negative, with a 95% interval below zero, in all 5 cases where Qwen3-VL passes and in 6 of 8 for Qwen3.5. Across the three models, the interval is below zero in 18 of the 21 cases that pass (Sec 4.3, Table 26).
Appendix I adds a second model on graphs of up to 10 nodes. This study of 26 further graph settings compares a graph-changing update with a state-preserving instruction that reaches the same final graph. With text input, Qwen3.5-9B shows a stable effect in 7 of its 12 eligible settings for edge additions (Tables 16, 17, and 19).
Prompted by this comment, we have added the condition-level results of all 30 panels to the revised PDF (Table 26, new Appendix L), and Section 4.3 now reports them together with a short summary of Appendix I.
Q2: How often both histories yield correct answers, and how often the test reveals dependence that accuracy misses (W2).
A2: Thank you for this suggestion. We respectfully clarify that both histories yield the correct answer in 181 of the 192 matched pairs for Gemma and in 147 for Qwen3-VL, and that the test reveals history dependence in three of the four conditions where every pair is answered correctly (Sec 4.2).
For Gemma, accuracy is identical under the two orders, yet the test detects the dependence. Gemma answers 183 of the 192 prompts correctly under each order, so accuracy shows no history dependence, while the test on the correct-answer margin gives θ=3.671 [2.581, 4.855] (Sec 4.2).
Where every pair is answered correctly, the test still finds the dependence. In four conditions both orders yield the correct answer in all 48 pairs: text input with both question types for Gemma, and degree questions with both input formats for Qwen3-VL. Accuracy records no difference there, yet the 95% interval of the order effect excludes zero in three of them. One is the largest condition-level effect of the core study, −3.623 [−4.131, −3.123] for Gemma on shortest-path questions with text input (Table 5).
Accuracy can register a difference in only a small fraction of pairs. Correctness can differ between the orders only where the top answer changes. In the two core models and five further Qwen and Gemma checkpoints, this happens in at most 21 of the 192 pairs for any checkpoint, and all seven show the history dependence (Table 24). Qwen3.8-27B, for example, answers 361 of the 384 prompts correctly and changes its top answer in only 1 pair, yet its order effect on shortest-path questions with text input is −0.547 [−0.766, −0.328].
Prompted by this comment, we have added to Section 4.2 of the revised PDF how often both orders yield the correct answer, and the conditions in which every pair is answered correctly.