- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates whether third-party comments can induce coding agents to use a shortcut that conflicts with the user's request to fix a software issue. It evaluates 13 models on 12 SWE-bench Verified issues under seven conditions, varying comment attribution, urgency, and the number of agreeing AI comments, with ten planned runs per model–issue–condition combination. The authors report increased shortcut-tool use following third-party endorsements, greater uptake for a human maintainer than an automated assistant, and a further increase when urgency is added. Additional AI agreement has no clear average effect. Hierarchical Bayesian analyses quantify these contrasts and substantial variation between models; the manuscript reports 10,260 retained trajectories.
Critical strengths
-
The study examines whether comments from parties other than the user can redirect an agent's tool use. Measuring an action during task execution makes the question relevant to deployed agents.
-
The hierarchical analysis includes model and item effects, reports posterior intervals, and provides model-specific contrasts.
Critical weaknesses
-
The many trajectories come from only 12 coding issues and a narrow set of comment templates and account identities. Repetition improves estimation within these conditions but does not establish generality across different messages, tasks, or agent environments.
-
The distinction from indirect prompt injection is insufficiently established. Section 3.4 treats lower uptake of attributed comments as ruling out instruction following, although it establishes only sensitivity to source framing. The introduction and discussion cite relevant work, but need a more precise account of the incremental contribution relative to indirect injection and agent-security evaluations
Questions for authors
-
Can human/AI identity be crossed with repository role, for example maintainer versus ordinary contributor? Since the current distinction is only between human and AI, without diving into the detailed segmentation of different authority level.
-
Does the trend consist when expanding to more than current 12 repo issues, and even across different task domains?
-
Can you provide more discussions and perhaps experiment supports for how the current findings distinguish from agent's instruction following behavior, which is already widely discussed by many works.