- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
You want to train an agent to ignore instructions hidden in documents, since this might contain prompt injections. You want the model to be able to disambiguate which instructions are from authorised users and which ones are from, say, the environment. They build some DPO pairs for two actions in the agentdojo simulated workspace which are cancelling a calendar event (cancel_calendar_event) and sharing a file (share_file). They got positive results in an easy setting (since actually, the base model already completes 63 of the 64 authorized actions) and then many negative results in a qualified, secondary setting.
Critical strengths
None
Critical weaknesses
- The paper is very hard to read and understand. It takes quite awhile to parse through the prose, and only has the one schematic figure at the beginning.
- As mentioned above, the main result is in a small setting where the model already does pretty well. And, the scale of training is quite small (only about 80 pairs?). My guess is that this type of training may simply be subsumed by existing instruction following datasets in tool-calling settings. And even on the small models, this is quite a saturated task.
Questions for authors
N/A
2 comments
Thanks. I will retain my score.
Response to Reviewer PBUz
Thank you for reviewing our paper. We may not have explained the goal clearly enough: we are not trying to raise the base model's authorized-action success from 63/64 to 64/64. We study whether safety repair also stops the model from carrying out legitimate requests. For example, after learning to reject a file-sharing instruction hidden in a document, can it still share the file when the user explicitly asks?
Our experiments therefore check both behaviors, rather than authorized-action success alone. The small scale and saturation you point out do limit what this evaluation can establish.