- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose the safety-asymmetric file injection (SAFI). SAFI fixes the payload body and only changes the way the task is performed on that file. The operation uses one of two frames. The 'transcription' frame requests the agent to copy the file contents, while the 'execution' frame requests the agent to execute the file contents. SAFI calculates two scores for each trial. The first score checks for the presence of refusal-related words, and the second score measures the command-motivated response (CMR) ratio. CMR counts the number of attempts for marker strings included in Bash input.
Critical strengths
- The measurement insights are accurate and carry significant meaning. This paper shows that the command strings included in a tool call do not guarantee actual execution.
- The limitations of the study are described very candidly. The authors specify each confounding variable, distinguish between pre-registered and exploratory tests, and report failure to replicate them without concealment.
Critical weaknesses
- No comparison isolates a single factor. The authors admit this problem many times. The workflow transformation moves placement, packaging, and frame together. The pattern coverage moves the target set with the pattern. The frame and single-word comparisons use different targets.
- The central numbers are unstable and come from a proxy metric. The authors did not attach the kernel-level collector during the attack trials. So the paper cannot confirm that any command ran. The CMR only counts a substring match in a Bash input.
Questions for authors
- Since the kernel collector was not connected during the attack attempt, its execution could not be verified. Why was the eBPF collector connected only for the detection experiment and not for the attack attempt?
- For the E5c condition, the success rate dropped from 19/30 to 5/30 when reproducing in the same environment (in-place replication). What is causing this decline?