- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether synthetic histories of earlier policy violations make language models disregard an explicit current rule. It measures decisions and policy reports, then uses activation steering and post-training comparisons to investigate the effect.
Critical strengths
The research question is relevant to agents whose earlier decisions remain in the conversation: an explicit approval rule should continue to govern subsequent actions.
Critical weaknesses
-
The paper is exceptionally difficult to read and understand. In the abstract, the setup, control comparisons, and recovery claim are compressed into terminology that makes the basic experiment difficult to follow. I found the paper exceptionally frustrating to read. In Figure 2, the main results are attributed to paraphrased records, but in Table 4, the same treatment is described as literal. This leaves the experimental condition unclear, even after consulting the appendix, and directly affects the claim that the result goes beyond copying answer tokens.
-
Claims and experiment results do not align. In Appendix I.1, rewording the prompt reduces the Qwen3-14B-distilled student's action effect from prose records from +0.94 to 0.00, while the control remains at 0.00. The effect subsequently reported with bare answer tokens does not reproduce that original comparison. In Appendix B, recovery is declared when the confidence interval includes zero. This means the remaining effect cannot be reliably distinguished from zero; it does not show that compliance has been restored or establish the claimed recovery asymmetry.
-
The training and steering experiments do not establish the proposed explanation. In Table 19, the UltraChat control is described as "human-written," although the original UltraChat paper by Ding et al. describes generating the conversations with ChatGPT. This comparison cannot establish that human-written supervision prevents the failure. In Appendix I.5, the distilled student also shows generation problems, raising the unresolved question of whether the training causes broader instruction-following failures. In Section 3.5 and Table 7, the steering experiments manipulate activations near the output using directions obtained from prompts that favor different answers. The intervention could therefore change the CALL/HOLD preference without showing that the model uses the policy differently.
Questions for authors
-
Which records produced the main result? Figure 2 describes paraphrased records, while Table 4 labels the treatment literal. Which description is correct, and does Table 4 contain the raw results underlying Figure 2?
-
What supports the recovery claim? Which readout gives the five-record recovery threshold? Does the existing analysis show a return close to the compliant baseline on that same readout, beyond the confidence interval merely including zero?
-
What was the actual source of the training control? Does "human-written supervision" in Table 19 refer to a separately verified human-authored dataset or to the original UltraChat responses? Which existing evaluations establish that the distilled students retain ordinary instruction-following ability despite their reported generation problems?
-
What distinguishes policy steering from answer steering? Which existing control separates a change in how the model uses the policy from a change in its general preference for CALL versus HOLD? This distinction determines whether the intervention explains the failure or simply changes the selected answer.