- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper explores whether coding agents follow the minimization principle, i.e., they only access the minimal amount of data and privileges required to solve the task. For this, the authors propose Workflow Analyzer, which identifies cases where this principle was not followed. They additionally propose Patch-Bench, which is a set of coding tasks with minimization requirements. Using this benchmark, they discovered that agents regularly request a significant amount of data unrequired for the current task.
Critical strengths
- The proposed problem of analysing whether agents follow the minimisation principle is interesting, and to my knowledge quite novel. The benchmark considers realistic instances of sensitive data and discovered that they are retrieved in many instances.
- The evaluation is quite complex, involving multiple models and harnesses. The experiments demonstrate that high utility values do not translate to high safety, and that models do not follow the minimisation principle.
Critical weaknesses
- My main issue is with regards to the presentation of the paper. The paper often uses a high density of non-standard technical terms which are never introduced. While the paper is generally understandable, certain sections (most critically the methodology section) are very difficult to parse. Further, without knowledge about SkillBench, it was not fully clear to me what the considered tasks look like, and more critically, what an actual instance of such minimization principle violations look like. An example for both would be very helpful.
- From the current state of the experiments, it is not clear how much of the difference between the tested models can be attributed to the model vs. the harness. For instance, certain harnesses might implement features or instructions which prevent the model from safety violations. Experiments holding the harness fixed, and only changing the model would be much more insightful.
Questions for authors
- Can you give a concrete example of what a violation of the minimisation principle looks like in practice?
- Can you run experiments where you fix the harness and change the model? How much impact does the harness have vs. the model?
- Can you elaborate on what "tasks without complete verifier scores in both setups are excluded" (line 424) means? Why is it not available, and how does this influence the results?
Please engage in the rebuttal stage
Dear Reviewers,
Thank you for your time and thoughtful reviews. As the discussion period draws to a close, I would appreciate your consideration of the authors’ responses and any new evidence. Please share any remaining concerns and update your reviews and scores where appropriate. Thank you for your constructive participation.
Your AC