- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether models can retain different evidence-selection policies despite achieving identical accuracy on subsequent training data. Paired models are initially trained to follow either semantic evidence or a metadata cue, then receive identical examples on which both cues agree. The resulting policy difference persists through 64k shared-example presentations. A retrieval experiment further shows that changing rank metadata while holding retrieved passages fixed can substantially alter source selection and downstream answers. The paper also examines parameter interpolation, weight and activation interventions, and whether small sets of cue-conflict examples can rewrite the learned policy.
Critical strengths
-
The rank intervention is well controlled. The passages and their display order stay the same while only the rank metadata changes. This makes the effect of metadata easy to interpret. Using the same downstream reader also helps connect source-selection errors to answer quality.
-
The controlled findings are supported by several complementary checks. Replications across model sizes, adaptation methods, and task constructions strengthen the persistence result. The rewrite experiments include aligned-only and random-label controls, supporting the importance of supervision that distinguishes the competing policies.
Critical weaknesses
I am not yet convinced that “objective imprinting” adds enough beyond existing work. Earlier studies already show that models with similar accuracy can use different features and that fine-tuning can preserve reliance on spurious cues [1-2].The controlled training-history experiments are useful, but the two rules are explicitly taught at the start, and later training deliberately provides no examples that distinguish them. It is also unclear how closely this setup reflects ordinary RAG behavior. Table 1 does not include an unmodified selector or one trained on ordinary retrieval data, and the 82.8-point gap is measured only on recoverable ranking errors.
The reported detection rates need clarification. Sections 4.6–4.7 report that 1,000 IID examples detect the policy difference only 54.9% of the time. If the correct-rank cases have identical outcomes, as the reported results suggest, the stated wrong-rank accuracies and detection rule instead give approximately 71.5–73.6%. The same calculation gives almost 100% at 5,000 examples, compared with 90.8% in Table A18. There may be differences in the outcome pool that explain this, but they are not clear from the paper. Since these numbers support the claimed advantage of targeted audits, I would like to see the underlying counts and calculation reconciled.
[1] Underspecification Presents Challenges for Credibility in Modern Machine Learning. JMLR, 2022.
[2] Mechanistic Mode Connectivity. ICML, 2023.
Questions for authors
-
How does an unmodified Qwen2.5-3B selector perform in the true-rank and corrected-rank conditions of Table 1?
-
Can you provide the outcome counts and calculation behind the 54.9% detection rate in Section 4.6?