- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how behaviors learned in context from earlier tasks in a conversation, referred to as the history, affect performance on a new task. Each task consists of a binary-choice reasoning problem, the user's answer to that problem, and an instruction specifying whether to follow the user's answer or to verify it independently. The authors examine whether each task instruction is followed under obedience and verification histories, and propose several hypotheses to explain the cases where it is not. Based on these hypotheses, C-DGE is proposed and empirically validated that it corrects decisions made incorrect by mismatched histories.
Critical strengths
None
Critical weaknesses
- The core phenomenon is not convincingly established, which undermines both the mechanistic analysis and C-DGE built upon it.
In task-aligned accuracy, mismatched histories change performance by only about 1 - 5% relative to matched histories as shown in Table 9. Matched accuracy under independent verification is 51.82%, which is almost random chance for a binary choice. The reported effects therefore sit on a task the model barely solves and a task leaving little room for change.
Moreover, the paper assumes that the "obedience" and "verification" histories create the corresponding interaction policies, but never verifies this. The observed margin differences are explained by these policies without direct evidence that the histories actually lead to obedient or verifying behavior.
The evaluation protocol also limits what can be concluded. Because decisions are restricted to a forced A/B choice, the most desirable behavior, complying with the delegated choice while flagging the factual error, cannot be observed. As a result, correcting the user's mistake is counted as a failure.
Since the causal localization and the repair method both presuppose a real and meaningful failure mode, this weakness propagates to the entire paper. It is reflected in C-DGE's marginal practical gains: it improves mismatch accuracy by only about one percentage point on the locked test, as the gap it targets is itself small.
- The claims far exceed the experimental scope, so the results cannot support the generality the paper claims.
The paper frames the problem as stale interaction policies in general, i.e., who should propose, verify, constrain, and decide, yet only a single axis, verification versus obedience, is tested. All histories are derived from one templated cost-comparison record repetition, with only two policies and two surface styles. Such histories are unlikely to occur in real conversations, and varying the source benchmarks does not diversify them. Furthermore, only two models from the same family are evaluated. Given one behavioral axis, one synthetic history template, and one model family, the findings may reflect artifacts of this specific construction rather than a general property of multi-turn LLM interaction.
Questions for authors
Please see the weaknesses.
4 comments
Otherwise, we would also like to clarify two aspects of your characterization. First, it is accurate that our central behavioral contrast is verification versus delegation; this is the specific transition the paper sets out to test. However, the complete evaluation is not limited to one repeated history template: it also varies history style and length, includes six naturalistic task scenarios, and evaluates sequential benchmark-task histories. Second, Qwen3-8B and Qwen3.5-9B belong to the same model family, but are distinct model architectures. And the MiniCPM analysis further extends the behavioral evidence to another model family.
We hope our current clarification addresses your concerns, and we thank you again for your suggestions on this work.
Response to the authors response
Thanks for the response. However, I will maintain my score for the following reasons:
- Even with the added scenarios, the study tests a single, unrealistic behavioral axis under a forced A/B protocol, whereas the cited works, e.g., CAA with seven behaviors and CAST with multiple refusal categories, cover far broader and realistic cases.
- The 44.4% flip rate is reported only in aggregate across benchmarks, so it is unclear whether the effect generalizes or is driven by specific benchmarks.
- Boundary-case analysis is beneficial to the research community only when the failure is substantial.
We understand your concern. You question whether the effect observed in the forced A/B contrast reflects a genuine failure rather than an artifact manufactured by the design itself. This is precisely why we chose this design, rather than something we avoided.
We should first be clear that a forced A/B design does not mean there is no substantive failure. The binary format is an identification device, not a claim about the richness of real interaction. The failures we report are real model errors that occur in real task transitions. The A/B format simply makes them measurable.
These flips are not imposed by the design. They are genuine model errors that would occur in real task transitions.
First, the design does not manufacture flips. It only makes them observable. In real use, models already switch between independently judging the evidence and following a user's delegation. Our design holds the question and answer options fixed and changes only the preceding policy. If a flip occurs, it can only come from the model's own decision process, not from differences in the question, options, or labels. The design itself does not produce errors. It merely exposes errors that would otherwise be masked by item differences.
Second, real-world cases confirm that these flips are genuine failures. We not only report flip rates under controlled conditions, but also test whether the same failure reappears after ten benchmark tasks executed in sequence, that is, after genuinely accumulated task history (C-DGE 17.61% NGR vs. RePS 9.60%). If the flips were merely a constructive artifact, they should not reappear after sequential, real task execution. The positive crossover interaction across all six benchmarks in both discovery and replication shows that this is reproducible, genuine model behavior, not a product of the evaluation format.
Third, the design is precisely what distinguishes genuine failures from design failures. We observe 0/266 flips among high-margin decisions and 91/205 flips among low-margin decisions. If the design were manufacturing failures, it would not land so precisely on the decision boundary. This selectivity itself shows that what we capture are genuine judgment errors that occur when evidence and delegation conflict, not artifacts imposed by the evaluation format. What we report are meaningful failure cases, not a failure of the design.
Finally, we agree that supplementing the controlled contrast with studies of failures on real open-ended tasks would help complete the picture of how this mechanism behaves in practice. We would note, however, that such supplementation would come at a cost to effect observation. Real tasks introduce uncontrolled variation in wording, difficulty, and context, which is exactly the confound our paired design removes. The two approaches are therefore complementary. The controlled design identifies the mechanism, and naturalistic tasks characterize its expression, but they serve different inferential goals. We sincerely appreciate the opportunity to clarify this motivation. If there are specific analyses or clarifications that would help address your remaining concerns or reconsider your assessment, we would be grateful for your comments.
We thank you for this thoughtful comment. We understand your concern that the conclusions should be supported by evidence appropriate to the scope of the evaluation. Our claim is specific: an interaction state established during one task can persist across a task boundary and influence the next task when its requirements differ.
Our study follows a common approach in mechanistic and intervention-based research: isolate a defined behavior, test it under controlled conditions, and evaluate a targeted intervention. For example, CAA ([ACL 2024] evaluates selected behaviors using multiple-choice questions and open-ended generation, while CAST ([ICLR 2025]) investigates conditional refusal across seven models. LoReFT ([NeurIPS 2024]) and RePS ([NeurIPS 2025]) likewise study representation interventions on defined tasks or behavioral targets.
Nevertheless, we reconsidered the paper from your perspective and added analyses to address the concerns about behavioral evidence, history diversity, and model coverage:
We directly measured the history-induced preference for the user-selected answer, as detailed in the response to Reviewer 6uhm.
We added a cross-family behavioral analysis on MiniCPM5-2B. As shown in Appendix D.2, the mismatch-gap reduction was 17.03%, which provides evidence that the measured task-dependent history effect is not confined to the Qwen family.
We evaluated C-DGE after ten consecutive benchmark-task turns. After the model completed a sequence of ten benchmark questions, we evaluated the subsequent task transition on 192 held-out test items. C-DGE achieved 17.61% NGR, compared with 9.60% for RePS. This tests the effect of context accumulated through successive benchmark tasks, beyond a single repeated history instance.
We would also like to clarify the interpretation of the 51.82% figure and of the relatively small change in aggregate accuracy. The effect is intuitive at the item level: very easy items may remain correct under either history, while items already beyond the model’s capability may remain incorrect under either history. In both cases, a history-induced shift in the model’s interaction state may not change the final answer. The effect is therefore most visible near the decision boundary, where a change in the model’s preference can change the selected answer. This is why we evaluated the full benchmark distribution rather than constructing a test set consisting only of items expected to flip.
Our locked-test analysis supports this account. Among matched-correct decisions, the low-versus-high-margin flip-rate difference was 44.4 percentage points for independent verification and 4.8 points for delegated choice; none of the 582 high-margin cases flipped. The margin bands were determined using the discovery split and frozen before evaluation on the prospective split. Thus, aggregate accuracy depends in part on how a benchmark’s items are distributed across easy, difficult, and decision-boundary cases. Averaging over all items can dilute a real, task-dependent effect concentrated near that boundary; it does not by itself show that the effect is absent.
This also motivates the practical question behind our study. In everyday use, people may recognize experiences such as an assistant seeming not to follow a new request, or seeming to defer too readily to a prior preference. These experiences motivate our investigation; they are not themselves evidence for the mechanism. Our experiments test whether a specific task-transition effect can be observed in open-source benchmarks and whether it can be reduced through intervention. The effect is measurable in the full benchmark evaluation and becomes clearer in the boundary-risk analysis. We therefore view the contribution as identifying when this failure is likely to affect an answer and testing whether it is repairable, rather than claiming a large uniform accuracy gain on every benchmark item.
Accordingly, this paper identifies and tests a specific form of context mismatch: an interaction state established during one task can persist across a task boundary and influence behavior when the next task requires a different policy. By measuring this effect across benchmark tasks and history conditions, examining where it changes decisions, and evaluating a targeted intervention, we characterize when this failure emerges and show that it can be reduced. We again thank you for raising these concerns. We hope the added analyses and clarifications address them and make the scope and contribution of the work clearer.