- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether quantization increases unauthorized changes during file-editing tasks, even when the requested task is completed. It introduces Tripwire, a synthetic benchmark that checks task success and scope violations separately. Experiments show that some quantized configurations increase violations without reducing task success. The paper also compares how often two LLM reviewers flag these violations.
Critical strengths
The paper studies an interesting and practically relevant question. An agent can complete a task while also changing content outside the user’s request. Examining whether quantization increases this behavior draws attention to a risk that task-success metrics alone can miss.
Critical weaknesses
-
The messy presentation makes the main contribution hard to assess. The central question is straightforward, but the paper introduces several overlapping terms and repeatedly shifts between violations across all runs, violations within successful runs, and violations missed by a reviewer. Section 4 devotes substantial space to restating counts, ratios, and significance tests already available in the tables, and, by the way, all the subsections seem to be presented not in a proper \subsection{} way, which is just a minor evidence of poor presentation. The main findings and their implications are difficult to follow. A clearer account of what each metric measures and which results support each claim is needed. This affects my assessment because the current presentation makes it difficult to determine the exact scope and significance of the contribution.
-
The contribution seems to be limited. Prior work has already explored agents that complete benign tasks while taking unauthorized actions [1]. [2] also shows that quantization can increase errors without a significant change in task scores. This paper extends that analysis to scope violations, which is a relevant distinction, but the added insight remains limited. Tripwire uses synthetic, single-turn editing tasks with whole-file writes and at most two writes per run. It does not test whether the findings extend to multi-turn tool use or longer agent workflows. Therefore, the whole contribution seems incremental.
[1] Qu, Yubin, et al. "Overeager coding agents: Measuring out-of-scope actions on benign tasks." arXiv preprint arXiv:2605.18583 (2026).
[2] Jang, Jiwon, et al. "Flat score, amplified failures: How the error budget masks damage in quantized LLM agents." arXiv preprint arXiv:2607.27275 (2026).
Questions for authors
-
What is the main contribution beyond the closest prior work? Prior work has studied similar problems. Could you state clearly what Tripwire and the present findings add beyond these studies, and identify the results that support this added contribution? A clearer explanation would help me assess the novelty and significance of the work.
-
How broadly does the finding at 7B apply? The result that violations increase while task success stays unchanged is established for certain quantized versions of Qwen2.5-7B. Is this intended as a finding about that model, or about 7B agents more generally? Do you have evidence that the same pattern holds in other model families, larger models, or multi-turn agent workflows? The 14B NF4 result does not establish the same pattern because task success also declines. If broader evidence is unavailable, please explain how the claims should be limited to the tested models and single-turn editing setting.
-
Does the strong reviewer identify the actual scope violations? The paper counts a run as detected whenever the reviewer lists any problem, without checking whether that problem matches the registered violation. Could you examine the 54 successful runs with violations and report how often the 72B reviewer correctly identifies the unauthorized change? This would clarify whether it detects the claimed failure or merely flags the same runs for other reasons.