- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies conflicts between the order in which events are presented and the chronological order in which they occur. Using synthetic four-event narratives evaluated on ten language models, the authors report that chronological order strongly disrupts text-order questions, while chronological-order questions are more robust. Linear probes recover both orderings from hidden states, and cache transplantation and access-mask interventions are used to argue that chronology-conditioned contextual states causally influence text-order answers. The paper then proposes ReFrame, a low-rank adaptation of attention query and value projections, which achieves near-perfect accuracy on held-out instances of the same controlled task and improves performance under several alternative question wordings. The empirical effects are interesting, but the paper’s mechanistic claims and the generality of the proposed adaptation are not sufficiently established.
Critical strengths
-
The paper identifies a clear and reproducible behavioral asymmetry: chronological order is substantially more likely to override text-order answers than vice versa. The factorial construction and evaluation across ten models make this a useful diagnostic phenomenon.
-
The authors provide more mechanistic analysis than a purely behavioral paper, including cross-model probing, carrier transplantation, K/V separation, and access controls. These experiments offer evidence that contextual states formed during passage processing influence later order judgments.
Critical weaknesses
-
The benchmark does not convincingly distinguish chronological-order interference from a much simpler instruction- following or positional heuristic failure. Text-order questions explicitly instruct the model to ignore dates, while chronological questions explicitly instruct it to ignore sentence order. The conflicting conditions therefore change not only the underlying order relation but also which salient cues must be suppressed. A model that defaults to dates, recency, or semantic temporal structure can produce the reported asymmetry without constructing two competing internal order representations. The paper needs stronger controls, such as non-temporal distractor attributes, randomized date formats, decoy numerical information, and conditions where text order is queried without explicit “ignore dates” instructions.
-
The causal interventions do not establish the claimed mechanism with sufficient specificity. Carrier transplantation replaces keys and values at every full-attention layer, producing hybrid activations that are not naturally generated by the recipient prefix. The carrier is also not an isolated memory slot: its contextual state depends on the entire preceding passage and can encode many features besides chronology. Similarly, context access masks affect broad token ranges and, for Qwen3.5, recurrent state updates. Thus, the results show that these hidden states can influence the answer, but not that chronological information is routed through a distinct carrier pathway or that values specifically carry the proposed order representation.
-
The probing evidence is weaker than the paper suggests. The probes operate on complete sentence-end hidden states with positional information intact. Near-perfect prediction of text rank may therefore be explained by absolute or relative position signals, without demonstrating a content-based representation of “mention order.” The paper acknowledges this limitation, but still uses the decoding result as central evidence for a gap between “retaining” and “using” text-order information. A position-ablated, content-matched, or amnesic probing analysis is needed.
-
The ReFrame evaluation is too close to the training problem to support the broad claims. ReFrame is trained on the same four-event synthetic task with shared templates, answer format, instruction structure, and controlled order manipulations. The “additional questions” change wording but preserve the same scenes, dates, options, and underlying task. Near-perfect conflict accuracy can therefore reflect task-specific supervised correction rather than a general mechanism for selecting reference frames in narrative understanding. The paper does not evaluate on genuinely different stories, longer narratives, natural flashbacks, unseen temporal expressions, or external temporal-order benchmarks.
-
The reported mechanistic explanation and the proposed method are not tightly linked. The interventions identify effects of contextual states, but ReFrame simply adapts Q and V projections across the model. The paper does not show that ReFrame changes the specific carrier pathway identified by transplantation, nor that the relevant causal effect is reduced through the proposed mechanism. The method may instead learn a direct benchmark-specific decision rule or strengthen instruction-following. Without pathway-level validation after adaptation, the claim that ReFrame is mechanism-inspired in a substantive sense remains largely rhetorical.
Questions for authors
-
How do the authors rule out the simplest explanation that the models rely on salient timestamp information or a chronological default, rather than representing two competing order coordinates? Please provide controls with randomized or non-date temporal expressions, irrelevant numerical distractors, and matched “ignore X” instructions.
-
Can the authors repeat the probing analysis after removing or randomizing positional information, or use an amnesic intervention that tests whether position information is causally necessary for text-order decoding? Without this, what evidence shows that the probe is decoding text-order content rather than position?
-
What is the quantitative difference between natural recipient states and the donor-transplanted hybrid states? Have the authors verified that the intervention remains on-manifold, for example by comparing hidden-state distributions, likelihoods, or downstream behavior under progressively smaller interventions?
-
Why should the “carrier” be interpreted as a specific chronological-information route rather than an arbitrary late-context bottleneck? Can the authors perform matched transfers from multiple equally sized regions, including earlier and non-timestamp regions, with controls matched for norm, attention mass, and state similarity?
-
Does ReFrame improve performance on genuinely unseen narrative structures, event descriptions, temporal expressions, and longer stories? Evaluation on only new question wordings over the same scenes is not sufficient to establish generalization.
-
Can the authors compare ReFrame against simpler baselines, such as supervised fine-tuning on the same examples, LoRA on MLP/output projections, prompt-level instruction improvements, or training only query/value layers without the consistency objective? This is necessary to determine whether the reported gains come from the proposed mechanistic design or simply from task-specific adaptation.
-
After ReFrame training, do the carrier-transplantation and K/V intervention effects change in the predicted direction? If not, what evidence supports the claim that ReFrame repairs the mechanism identified earlier?