- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper swaps K/V bridge states retained after source-history removal between example- and instruction-format histories, then measures effects at hidden-state, logit, and answer-margin levels. Across five arms (Qwen, Mistral, OLMo; synthetic tasks and IOI), large hidden-state effects do not translate into answer-level changes.
Critical strengths
- The paper is unusually candid about its limitations, failed launches, retrospective analyses, and provenance gaps. It also releases all episode records with CPU scripts that reproduce every reported statistic.
Critical weaknesses
-
No arm tests a setting where the answer depends on the removed history. In the synthetic arms, the query restates the rule and the models are not competent (primary Qwen: exactly 50%, always choosing the second clause). In IOI, the model answers correctly without any history. Since the answer never depends on the history, it is unsurprising that swapping bridge states leaves answers unchanged.
-
The main dissociation has an untested, simpler explanation, and the other findings are largely expected. The hidden state is read at layer 8, just two blocks after the patched layers 0–5, while logits are computed after all layers. The gap may therefore reflect readout depth. The authors acknowledge this, but the inexpensive depth-control experiment was not run. The overall work is trivial.
Questions for authors
-
Can you report results on a source-dependent task, where the answer information appears only in the removed history (high full-history accuracy, low query-only accuracy)? If retained bridge states produce donor-directed answer changes there, my assessment of RQ2 would change substantially. See the weakness above.
-
How do hidden-state projections behave when read at the final layer, or at several depths after the patched layers? If they shrink toward the logit-level values, the dissociation would be largely explained by readout depth.
2 comments
We have also updated our paper as per ICLR 2027 guidelines.
Thank you for identifying the two tests needed to assess the interpretation.
Depth control. The early IOI projection decreases from 0.968 at the original readout to 0.0078 at the final hidden state, close to the centred-logit projection of 0.0072. This supports your explanation of the large intermediate-versus-output contrast. It narrows the interpretation of that result. Middle and late patches produce larger final hidden and logit effects, but candidate choices remain unchanged.
Source-dependent evaluation. The new tasks require information from the passage. When only the generic bridge is retained before the question, source replacement gives near-chance accuracy. When replacement occurs after question processing, accuracy remains high on lookup and adapted SQuAD, and the selected partial swaps increase donor-answer preference by approximately 96–99 percentage points relative to same-answer controls.
This second condition has an important boundary: the retained states have already processed the question and options. It establishes answer-relevant retention under that condition, rather than general recall after removal. The unseen-question experiment finds smaller accuracy gains. For its donor-choice effect, the interval excludes zero in Mistral but not Qwen under the tested swap.
Contribution. The follow-ups provide a more specific empirical account. The original early IOI effect attenuates across depth. Behavioral transfer depends on what the retained segment has processed. The effective patch locations also differ between the two tested models. We believe these comparisons add useful evidence to the measurement contribution, with conclusions limited to the tested settings.