acceptodds
Under review as a conference paper at ICLR 2027

ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding

Abstract

As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important. Existing work evaluates related agent behaviors from separate perspectives, but lacks a systematic framework for unifying these behaviors. To bridge this gap, we introduce ParanoiaEval, the first benchmark for unified evaluation of risk-treatment capabilities in coding agents. Grounded in the well-established Avoidance–Transfer–Mitigation–Acceptance framework in software engineering risk management, ParanoiaEval operationalizes its 4 fundamental treatments for coding-agent settings and contains 200 evidence-controlled repository-level task pairs, each differing only in treatment-defining evidence. We further introduce dedicated metrics for risk-treatment violations and evidence responsiveness, using a human-calibrated agentic judge for reliable evaluation. Large-scale experiments on 8 representative models and a post-hoc human study reveal that (I) unnecessary risk treatment occurs in 11.2%–58.7% of runs despite explicit evidence, with substantial variation across agent configurations; (II) stronger task capability does not ensure more appropriate risk treatment, while treatment violations substantially harm developers' experience, establishing risk treatment as an independent capability dimension; and (III) agents exhibit systematic patterns consistent with established risk-management findings, suggesting that knowledge from human practice can guide the diagnosis and improvement of this capability.

Primary area:
datasets and benchmarks
Keywords:
coding agents, risk treatment, evaluation, defensive programming

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XuX2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the unnecessary defensive behaviors in coding agents, which may lead to a waste of time and unwanted code. It proposes a new benchmark called ParanoiaEval to evaluate the risk-treatment capabilities. It contains 200 evidence-controlled repo-level task pairs with differences in treatment-definition evidence. It benchmarks 8 models, showing unnecessary treatment occurs in 11.2% to 58.7% of runs, even when the evidence is present.

Critical strengths
  • This paper investigates an important problem in coding agents, which is usually ignored under functional correctness. Overly cautious behavior is a real complaint from developers, and the paper gives it a clear definition and a way to measure it.
  • ParanoiaEval is built on top of the ATMA software risk management framework, covering 44 concrete situations. ParanoiaEval: 200 evidence-controlled repository-level task pairs from 50 repositories. Each pair only differs on one fact, making it suitable to test the necessity of a specific behavior under different evidence.
  • The paper uses 8 models under Claude Code and CodeX, 3 runs per setting, and 9,600 runs in total. It reports bootstrap confidence intervals. The judge reaches 0.965 accuracy and κ of 0.93 on 200 held-out human labels. A second judge from another model family gives similar results, with no sign of family bias.
  • Their findings show that unnecessary behaviors lower users' satisfaction even in high success-ratio tasks.
Critical weaknesses
  • My main concern is about the human label "unnecessary". The necessity may reflect the preference of the annotators instead of the real scenarios. For example, one user may think it is better to have such a check than not, as in Figure 1. Ethics Statement mentions that all annotations are from the author team without reporting biographics. Two experts annotate the same instance, but the agreement among annotators is not reported. Such unnecessary labels may reflect the author team's preferences rather than the real scenario.
  • The other important missing thing is the trade-off between the unnecessary behaviors and the final safety. Evidence in real projects can be stale or wrong. A cheap extra check can be good insurance. The paper treats any treatment beyond the evidence as a violation, but it never weighs the cost against the benefit. For example, if such behaviors are forbidden, will security or task success ratio drop?
  • The cost of unnecessary work is not measured. The paper claims that such work wastes time and resources. Appendix O gives total tokens and runtime, but not split by runs with and without violations. So the actual cost is unknown. If the actual cost is small, such unnecessary behaviors may not be a big problem
Questions for authors

Can you provide the agreement between two experts for annotation?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer XuX2

Thank you for the constructive review and for recognizing the importance of the problem and the rigor of our evaluation. We address your three concerns below.


W1 & Q: Whose Notion of "Unnecessary"

Thank you for raising this important point. We agree that annotator background and agreement should be reported, and we have added both to App G.2 and the Ethics Statement. The two annotation experts are members of our team, distinct from the two experts who built and validated the tasks, and both have at least five years of software engineering experience and prior research experience in coding-agent evaluation. Before adjudication, their agreement is as follows:

SplitRecordsAgreementCohen's κ
Memory6493.8%0.87
Held-out validation20094.5%0.89

The judge agrees with this consensus at 0.965 accuracy (κ = 0.93), above pairwise agreement because adjudication resolves individual errors.

Our labels also do not encode a general preference for or against checks. A check is labeled unnecessary only when it treats a risk that a verified fact and the boundary stated in the answer card settle, and the paired E− task offers the same check without that fact. In Figure 1, the same guard is unnecessary only where the table is validated on every pull request.

Developers outside the author team also penalize these runs. The 20 developers rated runs blind to judge labels and model identities. They rated runs with excess treatment 1.27 points lower on a 5-point scale and would accept them as is in only 38% of cases versus 78% otherwise, with a Krippendorff's α of 0.76 between raters (Obs 3). They also marked at least one unnecessary action in 81.5% of runs with excess treatment, compared with 14.5% of runs without it.


W2: Stale Evidence and the Insurance Value of Checks

Thank you for this point. We agree that a cheap check can be good insurance, and our rules keep such checks in the baseline. Ordinary inspection, the targeted test, focused linting, a justified rerun after a change, and the final diff review stay within the baseline unless the answer card explicitly lists them (App G.1). Two experts also verify that every E+ fact entails its treatment before a pair enters the benchmark (Sec 4.2), so extra treatment on E+ addresses a risk that the stated evidence settles.

Our tasks contain no attacks or adversarial inputs, and each risk is an engineering failure named in the answer card. Extra treatment is also not always insurance, since an extra check can itself put the user's work at risk. In one Transfer task where CI validates release notes, Opus-5 ran the release-notes compiler to double-check a new file, and it deleted 28 pending fragments before the agent restored them. We have added this case to App N.4.

To test whether withholding treatment costs task success, we use two analyses. Extra treatment is not associated with higher success. 92.7% of E+ runs with excess treatment pass the oracle, compared with 94.5% of those without it. We then appended the instruction "Do not write unnecessary defensive code." to both variants of all 200 pairs and reran all 8 configurations once, with all other settings as in Sec 5.1:

Harness meanTSR (%) ↑VR (%) ↓E− treatment (%)ER (%) ↑
Claude Code92.0 → 91.227.4 → 20.051.4 → 33.551.2 → 46.0
Codex94.0 → 95.021.0 → 17.645.4 → 29.653.5 → 40.6

Discouraging defensive work leaves TSR unchanged (93.0% → 93.1%) and lowers VR from 24.2% to 18.8%, but it also lowers the E− rate from 48.4% to 31.6%. Suppressing treatment regardless of the evidence thus removes protection that the evidence leaves open, which is why ParanoiaEval measures whether treatment follows the evidence. The per-configuration results are in Tab. 4 of the new Sec 5.3.


W3: Cost of Unnecessary Work

Thank you for this suggestion. We agree that the cost should be measured directly. We split runtime, input and output tokens, and command count by whether an E+ run contains excess treatment. To separate this cost from task difficulty and model speed, we regress each log cost on the excess label with task and configuration fixed effects and 95% CIs from bootstrapping tasks:

MeasureIncrease with excess treatment [95% CI]
Runtime+39% [29, 51]
Input tokens+35% [25, 45]
Output tokens+35% [26, 49]
Commands+30% [21, 40]

The cost grows with the degree of excess. Runs with clear excess treatment take 53% longer than runs without it, compared with 19% for minor excess. Machine cost is also only part of the burden. Developers accept only 38% of these runs as is (Obs 3), so most need further changes before use. We have added this split as Tab. 3b in Sec 5.2, with the method in App O.2.


We hope these analyses address your concerns. Thank you again for your detailed review.

Thank the author for their detailed explanation and additional experiments, which have addressed my concerns. I will increase the score accordingly.

Reviewer_WHWxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ParanoiaEval, a benchmark for unnecessary defensive work by coding agents, organized around Avoidance, Transfer, Mitigation, and Acceptance. It contains 200 paired tasks across 50 repositories. Each pair changes evidence about a risk-treatment boundary while retaining the task objective and completion oracle. The evaluation separates task success, unnecessary treatment under boundary-establishing evidence, and responsiveness to that evidence. Experiments cover eight native model–harness configurations and three repetitions, totaling 9,600 runs. Human-calibrated judging and a developer study provide complementary assessments of the observed behavior and its practical relevance.

Critical strengths
  1. The paired benchmark makes evidence responsiveness measurable. The 44 situations are grounded in 18,922 real coding-agent sessions and instantiated in executable repository tasks. Each pair changes a treatment-defining fact while preserving a common objective and oracle (Section 4.2). Reporting task success, violation rate, and evidence responsiveness separately helps distinguish an agent that generally does little defensive work from one that changes its behavior in response to evidence. This is a useful methodological contribution for studying execution quality beyond functional completion.

  2. The evaluation includes substantial checks on judge reliability and practical relevance. The judging rules and memory are frozen before validation on 200 human-consensus labels from disjoint tasks. The primary judge achieves 0.965 accuracy and 0.93 Cohen's kappa, and a second judge audits 3,200 trajectories (Appendix K). The study with 20 experienced developers assesses 800 completed runs, with two ratings per run and blinded configuration and label information. These checks strengthen confidence that the reported pattern is reproducible under the benchmark rules and relevant to developers, while leaving the validity of particular rules open to scrutiny.

Critical weaknesses
  1. Some unnecessary-treatment labels depend on workflow assumptions that the stated evidence does not fully settle. In the Transfer example, existing CI is taken to make a local full-suite or multi-version validation run unnecessary (Appendix E.2). Transferring validation responsibility does not by itself establish whether additional local checking is worthwhile. The relevant question is whether earlier feedback justifies its cost within the agreed scope, considering execution cost, feedback latency, the change's consequences, and the maintainer's workflow. The answer card should explain why the additional action is excessive under those assumptions. Boundaries established by an explicit verification budget or an accepted scope decision are more compelling than boundaries inferred solely from another mechanism's existence. Appendix G.1 appropriately permits ordinary inspection, targeted tests, focused linting, and justified reruns; my concern is therefore about particular boundary judgments, rather than a blanket penalty on verification. Agreement between judges and annotators applying the same answer card establishes consistency under that card, but does not independently establish that its boundary is the right one.

  2. The paired design and completion oracles do not establish that necessary protection is preserved. The paper correctly states that removing the evidence in E− makes a treatment no longer ruled out; it need not make that treatment mandatory. Likewise, the E− marker is not a claim that the action violates E−. However, evidence responsiveness measures changes in the designated treatment, without independently determining when omitting that treatment leaves a real failure unaddressed. High completion, low violation rate, and high responsiveness therefore do not establish comprehensive risk-treatment competence.

Questions for authors
  1. How do the answer cards determine whether additional local validation is worth its cost within the agreed task scope when an external mechanism such as CI already owns validation?

  2. Does the E− task in Appendix E.1 require preserving compatibility with the documented Python 3.8 execution path? If so, how do the shared reference patch and oracle cover that requirement? If not, please clarify how this execution path affects the treatment boundary without changing the task's completion requirements. More generally, which E− cases require a safeguard for correctness, rather than merely permit it, and how are omissions assessed?

  3. Can the developer-study comparison be reported for the available same-task and same-configuration subsets, with the treatment of rater and task dependence made explicit? The existing minor/clear breakdown is already informative. I also encourage narrowing the capability-independence and causal-impact wording to match the currently available evidence.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer WHWx

We sincerely thank you for your careful reading and for recognizing that the paired design makes evidence responsiveness measurable. Your questions on where each treatment boundary lies and what our metrics can establish are highly valuable for our benchmark, and we address them below.


W1 & Q1: Boundaries Set

Thank you for this precise point. We agree that the existence of another mechanism does not by itself settle whether extra local checking is worthwhile, and that the reason an extra action is excessive should be stated explicitly. Our definition of Transfer states that the agent should rely on the owner, and that repeating the owner's checks duplicates treatment (Sec 3.2). Ordinary inspection, the targeted test, focused linting, a justified rerun, and the final diff stay in the baseline (App G.1). In App E.2, CI runs lint, build, and the full matrix over 3 operating systems and 6 Python versions on every pull request. For a one-line fix to an exception clause, the targeted test file gives early feedback on the changed code, while a local multi-version run repeats the feedback that the pull request receives. We have added this rationale to the Transfer example in App E.2.

To test the boundaries beyond the card, we ran two checks. First, the 20 developers in our study rated runs without seeing judge labels. They rated Transfer runs with excess treatment 1.20 points lower (95% CI [0.98, 1.42]) and accepted them as is in 40% of cases, compared with 77% otherwise (App L.2). Second, we reclassified as acceptable every Transfer violation that consists only of extra verification:

Violations countedOverall VRTransfer VR
As judged24.2%20.9%
Without verification-only Transfer cases21.8%11.2%

The remaining Transfer violations contain other forms, such as guards, scope expansion, and confirmation requests, which do not depend on how one values early local feedback. The overall prevalence of unnecessary treatment therefore does not depend on these verification boundaries. We have added this sensitivity analysis to App M.3.


W2 & Q2: Necessary Protection under E−

Thank you for this careful reading of the paired design. As you note, removing the evidence makes the treatment no longer ruled out, without making it mandatory. In App E.1, the E− task does not require Python 3.8 compatibility. Both variants ask for int.bit_count(), share the two-line reference patch, and share an oracle that this patch passes. The note on the Python 3.8 copy changes only the boundary. Under E−, a version guard is a defensible choice, and its label records the treatment without counting as a violation (App N). Omitting the guard is not penalized either, since the user asks for this exact change.

More generally, no E− case requires a safeguard for correctness. Acceptance Check (I) requires the reference solution, which contains no treatment, to pass the oracle in both variants (Sec 4.2). Any protection that a task needs belongs to its completion requirements and is checked by the shared oracle, so TSR captures required function, while VR and ER capture treatment beyond it. We have added a sentence on this division of roles to Sec 4.3.


Q3: Developer-Study Dependence and Wording

Thank you for this suggestion. We refit the satisfaction comparison with a linear mixed model that has random intercepts for rater and task, and we report it on the subsets you suggest:

SubsetRecordsΔ satisfaction [95% CI]Accept w/Accept w/o
All8001.25 [1.11, 1.39]38%78%
Same task7441.23 [1.09, 1.37]38%77%
Same task and configuration1861.16 [0.93, 1.39]40%76%

The gap remains 1.16 points (95% CI [0.93, 1.39], p < .001) within the same task and configuration, so it does not come from differences between tasks or agents. We agree that our current wording goes beyond this evidence. We have replaced "establishing risk treatment as an independent capability dimension" with "indicating that risk treatment is a capability distinct from task success" (Abstract, Sec 1, 5.2, 6). The title of Obs 3 now reads "Treatment violations are associated with lower developer satisfaction".


Thank you again for these questions. They have helped us define the treatment boundaries more clearly and align our claims more closely with the evidence, making the benchmark more rigorous. We look forward to your further thoughts.

Reviewer_XKTfOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ParanoiaEval, benchmarking the unnecessary defensive behavior in coding agents. The author constructs 200 evidence controlled repo-level tasks. Results show that unnecessary risk treatment shows up in up to 58.7% of the runs, and stronger agents may not have more proper risk treatment ability. They also find that human can provide knowledge to achieve a better risk treatment capability.

Critical strengths
  1. The paper is well motivated and studies an important agent behavior.

  2. The proposed benchmark and evaluation results can bring insights to model developers. The benchmark is also checked by experts to ensure quality.

Critical weaknesses
  1. There lacks analysis on how codex and claude code differ in terms of different defensive coding patterns. For example, one may prefer writing defensive code for certain functions to the other.

  2. Is there any code failure caused by defensive coding? It would be interesting even if there are only a few cases, which can further strengthen the motivation of the paper.

  3. Can an anti-over-defensive instruction like “do not write unnecessary defensive code” mitigate such behavior? And will it harm task performance?

Questions for authors

See above.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer XKTf

We sincerely appreciate your positive assessment and for these concrete suggestions. Your suggestions on comparing the two harnesses and inspecting failed runs led to interesting new findings on how different agents channel excess caution and how defensive work can disrupt a task. Below, we address each point with new analyses and a new prompting experiment.


W1: Defensive Patterns of Claude Code and Codex

Thank you for this suggestion. Code-level guards appear in a similar share of the excess-treatment runs in both harnesses, yet the two harnesses channel excess caution in different ways, as Claude Code externalizes uncertainty to the user whereas Codex internalizes it through repeated verification. We grouped the 1,161 E+ runs that the judge marks as excess treatment, 658 in Claude Code and 503 in Codex, across the six main action forms, with all forms in App L.3. A run can contain several forms, so the shares do not sum to 100%.

Action formClaude CodeCodex
Guard20.1%19.1%
Verification loop49.4%60.4%
Scope expansion51.5%36.4%
Confirmation request35.7%0.0%
Hedging13.7%1.2%
New infrastructure7.4%7.4%

Guards concentrate under Avoidance in both harnesses, matching the Avoidance situations in App D.1. The main difference lies in how agents act at the stopping boundary. The newer Claude models, Opus-5 and Sonnet-5, ask the user to reconfirm settled decisions, most clearly under Acceptance, and Sonnet-5 alone asks in 33.8% of its excess-treatment runs. By contrast, Codex agents never ask for confirmation under any treatment and instead repeat verification in the environment. For example, when the user states that 37.0.0 is the final version and that they will handle the release, Opus-5 makes the two requested edits, searches for other version strings, and asks whether to run the release script, whereas 5.6-Terra makes the two edits and stops. We present the per-configuration heatmap as Fig. 4 in Sec 5.2 and this paired case in App N.3.


W2: Failures Caused by Defensive Work

Thank you for this question. Such failures are rare, since 92.7% of E+ runs with excess treatment still pass the oracle (Obs 2). Because they are too few for a systematic analysis, we inspected each failed run that contains excess treatment or a confirmation request and analyzed the cases qualitatively. We found two typical ways in which defensive work affects the task:

  • Redundant verification. Under E+, where CI runs lint, type checking, and the full test suite on every pull request, Opus-5 fixed an approval check whose tests pass in 0.75 seconds, then ran a broad test set and nearly the whole suite and stashed its changes to rerun an unrelated failure. The run took 712 seconds, more than 500 of them spent waiting for these checks.
  • Side effect of an extra check. Under E+, where CI validates release notes, Opus-5 ran the repository's release-notes compiler to double-check a new skeleton file, and the compiler deleted 28 pending fragments. The agent restored them with git, so the run did not fail for this reason, but the extra check put the user's files at risk.

The failed runs themselves fail for reasons unrelated to the excess treatment. Defensive work therefore rarely breaks code directly. Its costs show up mainly as time and lower developer acceptance, as developers accept only 38% of runs with excess treatment as is, compared with 78% otherwise (Obs 3). We have added both cases to App N.4, each as a structured case of the task, evidence, actions, and outcome.


W3: An Explicit Anti-Defensive Instruction

Thank you for this suggestion. We appended your sentence, "Do not write unnecessary defensive code.", to the instruction of both variants of all 200 pairs and reran all 8 configurations once, with all other settings as in Sec 5.1.

Harness meanTSR (%) ↑VR (%) ↓E− treatment (%)ER (%) ↑
Claude Code92.0 → 91.227.4 → 20.051.4 → 33.551.2 → 46.0
Codex94.0 → 95.021.0 → 17.645.4 → 29.653.5 → 40.6

The instruction does not harm task performance, as TSR moves from 93.0% to 93.1% on average, but it lowers VR only from 24.2% to 18.8% (95% CI [16.1, 21.9]), and VR under transfer and mitigation barely changes. It lowers treatment on E− even more, from 48.4% to 31.6%, where the treatment is no longer ruled out, so ER falls from 52.3% to 43.3%. The instruction thus suppresses treatment regardless of the evidence without teaching agents when it is warranted, so a simple instruction does not suffice, and appropriate risk treatment calls for systematic study and targeted methods. The per-configuration results are in Tab. 4 of the new Sec 5.3.


Thank you again for your support. We hope these additions address your questions.

Thanks for your reply which clears out my concerns. I decide to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.