- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the unnecessary defensive behaviors in coding agents, which may lead to a waste of time and unwanted code. It proposes a new benchmark called ParanoiaEval to evaluate the risk-treatment capabilities. It contains 200 evidence-controlled repo-level task pairs with differences in treatment-definition evidence. It benchmarks 8 models, showing unnecessary treatment occurs in 11.2% to 58.7% of runs, even when the evidence is present.
Critical strengths
- This paper investigates an important problem in coding agents, which is usually ignored under functional correctness. Overly cautious behavior is a real complaint from developers, and the paper gives it a clear definition and a way to measure it.
- ParanoiaEval is built on top of the ATMA software risk management framework, covering 44 concrete situations. ParanoiaEval: 200 evidence-controlled repository-level task pairs from 50 repositories. Each pair only differs on one fact, making it suitable to test the necessity of a specific behavior under different evidence.
- The paper uses 8 models under Claude Code and CodeX, 3 runs per setting, and 9,600 runs in total. It reports bootstrap confidence intervals. The judge reaches 0.965 accuracy and κ of 0.93 on 200 held-out human labels. A second judge from another model family gives similar results, with no sign of family bias.
- Their findings show that unnecessary behaviors lower users' satisfaction even in high success-ratio tasks.
Critical weaknesses
- My main concern is about the human label "unnecessary". The necessity may reflect the preference of the annotators instead of the real scenarios. For example, one user may think it is better to have such a check than not, as in Figure 1. Ethics Statement mentions that all annotations are from the author team without reporting biographics. Two experts annotate the same instance, but the agreement among annotators is not reported. Such unnecessary labels may reflect the author team's preferences rather than the real scenario.
- The other important missing thing is the trade-off between the unnecessary behaviors and the final safety. Evidence in real projects can be stale or wrong. A cheap extra check can be good insurance. The paper treats any treatment beyond the evidence as a violation, but it never weighs the cost against the benefit. For example, if such behaviors are forbidden, will security or task success ratio drop?
- The cost of unnecessary work is not measured. The paper claims that such work wastes time and resources. Appendix O gives total tokens and runtime, but not split by runs with and without violations. So the actual cost is unknown. If the actual cost is small, such unnecessary behaviors may not be a big problem
Questions for authors
Can you provide the agreement between two experts for annotation?
2 comments
Thank the author for their detailed explanation and additional experiments, which have addressed my concerns. I will increase the score accordingly.
Rebuttal to Reviewer XuX2
Thank you for the constructive review and for recognizing the importance of the problem and the rigor of our evaluation. We address your three concerns below.
W1 & Q: Whose Notion of "Unnecessary"
Thank you for raising this important point. We agree that annotator background and agreement should be reported, and we have added both to
App G.2and the Ethics Statement. The two annotation experts are members of our team, distinct from the two experts who built and validated the tasks, and both have at least five years of software engineering experience and prior research experience in coding-agent evaluation. Before adjudication, their agreement is as follows:The judge agrees with this consensus at 0.965 accuracy (κ = 0.93), above pairwise agreement because adjudication resolves individual errors.
Our labels also do not encode a general preference for or against checks. A check is labeled unnecessary only when it treats a risk that a verified fact and the boundary stated in the answer card settle, and the paired E− task offers the same check without that fact. In
Figure 1, the same guard is unnecessary only where the table is validated on every pull request.Developers outside the author team also penalize these runs. The 20 developers rated runs blind to judge labels and model identities. They rated runs with excess treatment 1.27 points lower on a 5-point scale and would accept them as is in only 38% of cases versus 78% otherwise, with a Krippendorff's α of 0.76 between raters (
Obs 3). They also marked at least one unnecessary action in 81.5% of runs with excess treatment, compared with 14.5% of runs without it.W2: Stale Evidence and the Insurance Value of Checks
Thank you for this point. We agree that a cheap check can be good insurance, and our rules keep such checks in the baseline. Ordinary inspection, the targeted test, focused linting, a justified rerun after a change, and the final diff review stay within the baseline unless the answer card explicitly lists them (
App G.1). Two experts also verify that every E+ fact entails its treatment before a pair enters the benchmark (Sec 4.2), so extra treatment on E+ addresses a risk that the stated evidence settles.Our tasks contain no attacks or adversarial inputs, and each risk is an engineering failure named in the answer card. Extra treatment is also not always insurance, since an extra check can itself put the user's work at risk. In one Transfer task where CI validates release notes, Opus-5 ran the release-notes compiler to double-check a new file, and it deleted 28 pending fragments before the agent restored them. We have added this case to
App N.4.To test whether withholding treatment costs task success, we use two analyses. Extra treatment is not associated with higher success. 92.7% of E+ runs with excess treatment pass the oracle, compared with 94.5% of those without it. We then appended the instruction "Do not write unnecessary defensive code." to both variants of all 200 pairs and reran all 8 configurations once, with all other settings as in
Sec 5.1:Discouraging defensive work leaves TSR unchanged (93.0% → 93.1%) and lowers VR from 24.2% to 18.8%, but it also lowers the E− rate from 48.4% to 31.6%. Suppressing treatment regardless of the evidence thus removes protection that the evidence leaves open, which is why ParanoiaEval measures whether treatment follows the evidence. The per-configuration results are in
Tab. 4of the newSec 5.3.W3: Cost of Unnecessary Work
Thank you for this suggestion. We agree that the cost should be measured directly. We split runtime, input and output tokens, and command count by whether an E+ run contains excess treatment. To separate this cost from task difficulty and model speed, we regress each log cost on the excess label with task and configuration fixed effects and 95% CIs from bootstrapping tasks:
The cost grows with the degree of excess. Runs with clear excess treatment take 53% longer than runs without it, compared with 19% for minor excess. Machine cost is also only part of the burden. Developers accept only 38% of these runs as is (
Obs 3), so most need further changes before use. We have added this split asTab. 3binSec 5.2, with the method inApp O.2.We hope these analyses address your concerns. Thank you again for your detailed review.