- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Nowadays, the agents work through long sessions with instructions that change midway, asking them to change their reasoning, ask for permission, and so on. The paper discusses that in order to apply this change, the first natural solution is the system prompt, but since the game prefix chase can't be used, changing the system prompt forces the whole conversation to be recomputed. Instead, a API can accept a system message in the middle of the conversation while keeping the case by only appending the changes. However, blind appending results in a long list of interfering instructions that the agent needs to infer what they all mean together. The contributions of this paper are as follows: 1) Providing a method for appending such that revokes some interaction and determines which is in force without removing the cache. 2) A benchmark that grades what a model stopped doing. 3) A set of instructions to make the protocol reliable.
Critical strengths
The paper is very well written. The downsides of changing the system instruction and appending method is very well discussed in both the introduction and related work. The problem definition is well motivated. The experiments are fully design to provide insights about the claims of the paper and the results are fully explained
Critical weaknesses
I do appreciate the authors acknowledging the limitation on using only Qwen3.8-27B. I do believe that it would be interesting to see the performance on the smaller models as well as other family of models such as Gemma family. Although the paper talks in detail about the implemented method, there is little justification on some design choices such as why DPO was used.
Questions for authors
As mentioned in the weaknesses, what do you expect would be the performance on smaller/other families of LLMs? justify design choices of the method such as DPO and finally, I would like to see some prints on the snapshots during training and how some of the instructions are revoked.
2 comments
I thank the authors for their rebuttal which clarified my questions. I will discuss this further with other reviewers with higher confidence to decide on increasing the score.
Thank you for your thoughtful review and positive assessment of our problem formulation, protocol design, and experimental analysis. Revisions are highlighted in blue in the updated manuscript, and our code and benchmark are available at https://anonymous.4open.science/r/Statecraft-Review-7C4D/. Below we address each of your questions in order.
1. Smaller models and other model families, including Gemma (Section 6.5, Table 5; Appendix E.4)
We applied the same self-sampled LoRA DPO recipe, sampling preference pairs from each model's own responses under its native chat template, to three additional open-weight models across families and parameter scales:
Self-sampled DPO improves overall accuracy and roughly doubles single-revocation compliance across all three models, confirming that the approach transfers beyond the Qwen family. Their post-DPO ceilings are lower than on Qwen3.8-27B (80.6% -> 96.4%) because 4B–20B base models start much weaker on multi-turn tool use and multi-key compliance even before any instruction changes. Since DPO teaches which snapshot applies on a turn rather than adding general tool-use ability, weaker base capability both limits the task ceiling and produces fewer passing rollouts (R=1) per prefix on multi-version sequences, which is also why gains on single revocations are much larger than on the full benchmark (Section 6.5, Appendix E.4). While academic GPU limits prevented training models above 27B, stronger base models produce more passing rollouts and should benefit even more clearly (Section 7).
2. Why offline DPO over SFT and online RL (Section 5, Section 6.2, Appendix E.1)
When an instruction is revoked by leaving it out of the new snapshot, earlier assistant replies in the prompt still show the old behavior, so the model has to stop doing something it was doing in previous turns (Section 5). SFT trains only on passing responses, which shows a valid next reply but applies no direct penalty against continuing the revoked behavior. DPO instead pairs a passing response and a failing response generated from the same conversation prefix and workspace state, directly penalizing the old behavior when it should stop. Compared with online RL, offline DPO also needs no live rollouts during training, while matching online GRPO on our held-out evaluation matrix at much lower compute cost (Section 6.2, Appendix E.1).
To test SFT against DPO head-to-head at full data scale, we trained both methods under identical settings on the full preference dataset (Section 6.2, Appendix E.1). DPO outperforms SFT by nearly 11 points on Statecraft-200 (96.2% vs. 85.2%). Both methods learn single-step revocations, but on 4–6-version sequences SFT drops to 43.3% whereas DPO reaches 81.1%. Even starting from the trained SFT checkpoint, a short DPO continuation lifts accuracy to 92.7%, compared with 86.1% for continued SFT.
3. Training
<snapshot>examples and revoking a subset of instructions (Section 5, Appendix C.1)Section 5 (An example training pair) shows the snapshots from the training pair illustrated in Figure 3(a):
Version 4 omits
message_header, revoking the prefix rule while three[ACK]replies remain in the conversation history. From the same post-revocation state, the chosen response answers directly, whereas the rejected response still begins with[ACK].When multiple instructions are active and only some are revoked, the new snapshot keeps the remaining active keys and omits the revoked ones (Equation 1). For example, if version 2 sets both
reply_languageandmessage_headerand version 5 retains onlyreply_language, thenmessage_headeris revoked whilereply_languagestays in force. Additional multi-key and forged-snapshot examples appear in Appendix C.1 andtraining_example.jsonin the repository.