- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies instruction revision — when one requirement changes (e.g., destination country), the unchanged ones (budget, ranking) must still hold — via causal interventions on attention queries. Queries from a source prompt that restates the final rule are transferred into a recipient prompt at recipient RoPE coordinates. On controlled selection tasks with exact ground truth (six-record tables; category + threshold + ranking), the authors find: (1) query transfer improves revision accuracy comparably to direct restatement on Llama-3.1-8B and Qwen2.5-7B, but paired per-table scoring shows it breaks some previously correct answers, a loss invisible in aggregate accuracy; (2) adding record K/V improves reconstruction of a monitored source margin in Qwen yet can worsen selection, because a two-candidate margin omits eligible competitors — full-candidate scoring separates copied source errors from correct-source reversals; (3) record-free source queries shift scores toward changed requirements but do not reliably yield complete execution, with spillover across requirement axes and retained answer-letter preferences. Evaluation uses exact McNemar tests, Holm correction within and across families, and table-level bootstrap intervals.
Critical strengths
- Methodological rigor in evaluating causal interventions. Paired preservation (Eq. 6), the complete-candidate margin (Eq. 4), the mutually exclusive error taxonomy, and global multiplicity correction form a template that addresses known weaknesses in how patching results are reported. The finding that aggregate gains conceal broken previously-correct answers is shown cleanly.
- A well-isolated negative result. In the Figure 1/Figure 4 example, record-K/V transfer cuts the D-F reconstruction error from 9.21 to 0.54 logits while flipping the answer from correct D to incorrect C through the omitted competitor C; the correct-source reversal analysis (Table 1) shows this is not a one-off.
Critical weaknesses
- The writing obscures the contribution enough to limit its impact. The paper's value is methodological — it asks the community to adopt evaluation practices — and that depends on legibility. As written, even a sympathetic reader must reconstruct the experimental matrix from scattered definitions.
- Narrow scope and no positive mechanistic account. Two 7-8B models, one synthetic six-record task, one intervention family (suffix/last-token queries). The results are predominantly cautionary: no preservation advantage over simply restating the rule in the prompt, Llama and Qwen disagree on whether record-state matching helps reconstruction, and record-free effects "do not consistently improve complete execution." A constructive takeaway about how revision is implemented — e.g., which query content carries the retained requirements — is missing, so the net contribution is a warning plus a protocol rather than an advance in understanding.
Questions for authors
- Query transfer and restatement tie on full-cohort revision accuracy (63/96, 69/96) and damage counts. Is there any setting — subset, metric, or budget — where the internal intervention is strictly preferable to restating the rule? If not, what is the practical motivation for query transfer beyond diagnosis?
- How do the repair/damage conclusions change on the complement of the capability subset (the tables where the unmodified model already fails)?
- Do reconstruction-vs-selection discrepancies persist with more records/categories, where the candidate-omission failure should be more prevalent?
- (Future work) A per-head analysis of which transferred query content encodes the retained threshold/ranking requirements would turn the diagnostic framing into a positive result.