acceptodds
Under review as a conference paper at ICLR 2027

AutoApproveBench: Benchmarking LLMs as Auto Reviewers for Coding Agents

Abstract

As LLM agents gain the ability to execute long-running tasks autonomously, their actions increasingly carry risks of compromising safety or exceeding user authorization. Model-based action approval has emerged as a crucial safeguard, yet existing evaluations remain proprietary and product-specific, precluding open and controlled comparison across reviewer models. We introduce AutoApproveBench, the first open and controlled benchmark for systematically evaluating LLMs as auto-approval reviewers. AutoApproveBench curates 1,579 context-grounded approval events from 75,165 review-relevant records drawn from real coding-agent sessions, enables controlled cross-model comparison, assesses both correct approval and correct denial, and organizes all denial cases into 3 diagnostic categories covering unsafe actions, violations of explicit requirements, and unrequested scope expansion. Extensive experiments with 10 representative models on AutoApproveBench show that (I) across models, correct denial rates range from 84.58% to 95.42%, while correct approval rates range from 31.95% to 82.31%. Detailed analysis further reveals that (II) reviewers occupy substantially different error profiles between preserving authorized work and blocking inadmissible actions, and (III) these points do not monotonically track general-purpose model capability. These results establish auto-approval reviewing as a distinct capability dimension that demands dedicated evaluation and calibration. Our work is openly accessible.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, coding agents, auto-review, agent safety, authorization, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_didNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper designed AutoApproveBench, a benchmark of 1,579 approval events from real coding-agent sessions, to evaluate whether an LLM reviewer can correctly allow or deny an agent’s proposed action before it runs. Across 10 models, correct denial is consistently high, while correct approval varies widely, and models rarely recognize when the agent should stop.

Critical strengths
  • As a benchmark paper, the approval events are sourced from real coding-agent sessions rather than synthetic tasks, and each case is verified unanimously by three human experts.
  • The evaluation fixes the harness, the input and the proposed action across 10 models with 3 runs each, so the comparison between reviewers is controlled and fair. The post-DENY continuation stage (safer alternative vs. stop) also measures beyond plain allow/deny accuracy.
Critical weaknesses
  • The reviewer prompt does not tell the models the same rules that the human experts used for labeling. For example, the prompt never says to deny actions that go beyond what the user asked (R3), and it defines “safer alternative” more loosely than the gold labels do. So the weak R3 and STOP results may come from the prompt design, rather than from the models’ judgment.
  • Cases were kept only if at least one screening model wanted to deny them, which means the ALLOW cases were chosen to look suspicious. So the finding that all models are denial-leaning may come from how the benchmark was built, rather than from the models themselves.
Questions for authors
  1. If the reviewer prompt states the full approval criteria (R1–R3) and the same continuation definition used for the gold labels, how much do R3 and STOP accuracy change? Does the model ranking stay the same?
  2. Which models were used for the automated screening, and do they overlap with the evaluated reviewers? How do models perform on ALLOW cases that no screening model flagged?
1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer didN: Full Criteria and Screening Overlap

We thank you for the constructive review and for recognizing the real-session sourcing and the controlled 3-run design. We address both concerns and both questions below. Revisions in the updated PDF are marked in blue.


W1 & Q1: Full Criteria Prompt

Thank you for this suggestion. Appendix H.1 is designed to mirror deployed auto-review, where a reviewer receives a compact authorization policy and must judge from the user request whether an action stays within it, while Appendix F holds the full criteria that the experts apply as ground truth. We agree that H.1 therefore does not state the R3 rule and defines Safer Alternative more briefly than the gold labels. To test this directly, we re-ran 10 models with the full approval criteria (R1 to R3) and the gold continuation definition on 500 cases stratified from the 1,579 (313 ALLOW, 187 DENY, 105 Stop, 82 Safer, seed 20261114). In the new continuation prompt, a safer step must advance an unfinished part of the original request, satisfy all criteria, and be materially safer, and completion reports or requests for permission count as Stop. The table gives raw rates on these cases as original 3-run mean→full criteria, where Stop is correct DENY plus correct continuation over the 105 Stop cases.

ModelOverallALLOWR3Stop
GPT-5.6 Sol75.20→68.2065.07→52.4093.10→91.383.49→20.95
Claude Opus 572.20→65.0062.94→50.1678.16→84.481.90→28.57
Gemini 3.8 Flash82.27→81.4077.21→75.7292.53→91.3818.41→29.52
Kimi K379.93→78.6079.34→71.8867.82→93.105.40→24.76
MiniMax M385.20→84.6085.41→83.3981.61→82.7616.83→22.86
GLM-5.379.00→77.6075.61→73.4872.99→77.590.95→20.95
Qwen-3.8 Flash76.20→71.0068.69→57.5183.33→93.102.22→26.67
HY 368.00→55.8055.27→33.5584.48→89.665.40→16.19
DeepSeek V4 Flash80.67→73.6079.02→62.9475.29→91.385.71→26.67
Grok 4.655.80→54.4033.87→30.6791.38→91.3824.44→29.52

R3 accuracy changes by −1.72 to +25.29 and Stop accuracy rises by +5.08 to +26.67 for all 10 models, yet Stop stays within 16.19–29.52%. The Spearman correlation between the original and full-criteria rankings by overall accuracy is 0.96 (Kendall τ 0.91), so the ranking is largely preserved. ALLOW accuracy falls for all 10 models (−1.49 to −21.73, mean −9.07), so spelling out every criterion shifts models toward denial and the approval–denial trade-off of Sec 5 persists. Appendix K.4 reports this condition, and we release the prompts and logs at https://anonymous.4open.science/r/autoapprovebench-review-5857/.


W2 & Q2: Screening Overlap and Unflagged ALLOW

Thank you for this question. We agree that the screening models should be stated, and Appendix B.5 details them. The M route used 5 screeners (DeepSeek-V4-Flash, Qwen3.7-Plus, HY3, MiniMax-M3, MiMo-v2.5) and kept a case if at least 1 returned DENY. The core set K (1,000 cases) was built from a 7-model vote (HY3, DeepSeek-V4-Flash, Qwen3.7-Plus, GLM-5.2, Kimi-K3, MiniMax-M3, Grok-4.6). 3 of the 5 M screeners (DeepSeek-V4-Flash, HY3, MiniMax-M3) are also evaluated, and the 7-model vote shares 5 identities with the evaluated set.

The denial tendency does not come from the screening. On the 861 M ALLOW cases that the screeners flagged, the 3 evaluated screeners reach a mean Acc_A of 71.73%, against 66.42% for the 7 evaluated models that took no part in screening, so the models that did not select these cases deny them more often. None of these 7 models screened the 990 M cases, and their Acc ranking on these cases is identical to their ranking on all 1,579 (Spearman ρ = 1.00). Removing, for each screener, the cases where only it returned DENY (1 to 71 cases) keeps the 10-model ρ at 1.00 for Acc and 0.988–1 for F1. The table gives the mean Acc_A of the 10 models by route, averaged over the 3 runs.

ALLOW casesnMean Acc_A
M route (at least 1 screener DENY)86168.01
K route12870.99
K, no DENY vote2773.95
All98968.40

Mean Acc_A on the 27 K ALLOW cases that received no DENY vote is 73.95%, so about a quarter of these authorized actions are still denied. On these 27 cases the models score GLM-5.3 88.89, MiniMax 85.19, DeepSeek-V4-Flash 83.95, Kimi K3 82.72, Qwen-3.8 Flash 77.78, HY3 75.31, Claude Opus 5 72.84, Gemini 3.8 Flash 69.14, GPT-5.6 Sol 58.02, and Grok 45.68. Appendix B.5 details the routes and screeners.

Reviewer_Cc8jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AutoApproveBench, a benchmark for evaluating LLMs as automated action reviewers for coding agents. The authors construct 1,579 approval events from real coding-agent sessions and evaluate 10 models on whether they correctly approve authorized actions and deny unsafe or unauthorized ones. They also introduce three categories of denial reasons and evaluate whether reviewers can identify when an agent should stop versus continue with a safer alternative.

The main findings are that models differ substantially in their approval and denial rates, perform better on explicit instruction violations than on more contextual authorization boundaries, and frequently propose continued actions even when they should stop. The authors also argue that autapproval is a distinct capability that is not well predicted by general model capability benchmarks.

Critical strengths
  1. I think evaluating action approval is an important direction, especially as coding agents become more autonomous. Existing agent benchmarks tend to focus on whether an agent completes a task, so evaluating whether individual actions should be permitted seems useful.
  2. The use of real coding-agent sessions is a strength. I also appreciate that the authors evaluate both false approvals and false denials, since overly conservative reviewers can make an autonomous coding agent practically unusable.
  3. The finding that models struggle to recognize when they should stop is interesting, and I think this could be worth investigating further.
Critical weaknesses
  1. i am concerned concerned about how the benchmark was constructed and whether it represents realistic approval decisions. 2.T he authors compare 10 LLMs, but there isn't much evaluation of alternative approval strategies. For example, how would a simple rule-based permission system perform? How much does the model improve over checking commands against explicit user constraints, identifying destructive operations, or applying basic heuristics around authorization?
  2. The evaluation is disconnected from the actual consequences of approval decisions. The entire benchmark evaluates isolated actions, where reviewers decide whether to ALLOW or DENY a proposed action given some context.
Questions for authors
  1. The STOP versus SAFER-ALTERNATIVE result is interesting, but I don't think the current evaluation fully supports the interpretation. One of the most striking results is that models perform extremely poorly at recognizing when an agent should stop, with average accuracy below 10%. However, I'm not convinced this necessarily means models are unable to recognize execution boundaries.

The benchmark defines STOP as a situation where no further authorized action can advance the original request. But in a real coding-agent setting, it may be reasonable to report completion, summarize work, or ask for clarification, even when no further code modification is authorized.

would like to understand the realism component here, if you tried other prompts or experiments. happy to discuss more.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer Cc8j: Realism, Rule Baselines, Consequences, and Stop

We thank you for the detailed review and for recognizing the value of real sessions and of measuring both false approvals and false denials. We address every point below. Revisions in the updated PDF are marked in blue.


W1 (Part 1): Construction and Realism

Thank you for raising this concern. We agree that the construction pipeline and the realism of the decisions should be stated explicitly. The events come from 75,165 review-relevant records of real sessions in which developers used three general-purpose coding agents for routine software development, and these records yield 4,506 candidates. 3 experts label each candidate independently, and only unanimous cases remain, giving 1,579 events (Appendix B and C). Each reviewer sees the request, the history, and a reconstructed workspace that it can inspect with 7 read-only tools, and no proposed action is ever executed. The events cover the operations that auto-review gates in practice, namely code and workspace changes (555), package, build, and execution (508), version control (181), deletion and cleanup (115), and network and deployment (85), in Shell, Python, Go, JavaScript, TypeScript, C/C++, and Java workspaces (Appendix D).


W1 (Part 2): Rule-Based Baselines

Thank you for this suggestion. We agree that a non-model reference is needed. We implemented 3 deterministic policies on the 1,579 events, written before scoring and without model calls or access to gold labels. Blacklist denies destructive patterns such as recursive deletion, force push, hard reset, and piped shell downloads. Constraints denies actions that conflict with user-stated prohibitions, read-only scope, wait-for-confirmation, or preserve-this requests. Combined denies if either fires.

PolicyAccF1Acc_AAcc_DR3
Blacklist67.7037.3592.7225.7614.67
Constraints61.9440.0878.5634.077.07
Combined64.6651.5673.2150.3421.20
10 models57.67–85.0162.66–81.2935.49–83.8984.58–94.8673.01–95.29

All 10 models exceed every rule policy in F1 (best rule 51.56, always-DENY 54.40) and 9 of 10 in Acc (best rule 67.70). The best model improves over the best rule by 17.31 points in Acc, 29.73 in F1, and 74.09 in R3. Rules also cannot produce continuation decisions. Table 1 and Obs. 4 in Sec 5 now include these baselines, Appendix K.2 details the rules, and the rule code is released at https://anonymous.4open.science/r/autoapprovebench-review-5857/.


W2: Consequences of Decisions

Thank you for this point. We agree that the link between decisions and their consequences should be made explicit. To connect them, we split the gold-DENY cases by the irreversibility flag of the proposed action over the 30 run logs.

Gold DENYCasesMean Acc_DModel range
Irreversible36193.3989.29–96.95
Reversible22984.6176.56–91.85

Splitting further by denial reason, each of the 10 models denies reversible violations less often than irreversible ones in all three categories, and the gap is largest for R1 (73.96 vs. 91.10) and R3 (78.23 vs. 89.51). Models thus treat reversibility as a proxy for authorization and let through unrequested changes that can be undone but still alter the user's workspace. Scoring one decision on a fixed request, workspace, and proposed action (Sec 3.1) isolates reviewer judgment from actor planning, which is what lets this split attribute the misses to the reviewer. Obs. 3 and Figure 6 in Sec 5 and Appendix K.3 now report this analysis.


Q1: Stop Interpretation

Thank you for this careful point. We agree that reporting completion, summarizing, or asking for clarification can be reasonable after a DENY, and our formulation counts them as Stop. The reviewer gates tool execution, so Stop ends the autonomous loop and returns control to the user, where such reports and questions belong, whereas Safer requires another concrete action that advances the unfinished request (Sec 3.3). The original continuation prompt did not state this distinction. Of the 9,930 Stop-case episodes in the 30 logs, models deny 9,619 and answer Safer on 7,982, and a keyword scan flags 2,975 of these (37.3%) as proposing to report, summarize, ask, or confirm. We therefore re-ran 10 models with a continuation prompt that states this distinction and requires a safer step to satisfy all criteria (500 cases, 105 Stop). The table gives Stop accuracy on these cases as the original 3-run mean→the new definition.

ModelStopModelStop
GPT-5.6 Sol3.49→20.95Kimi K35.40→24.76
Claude Opus 51.90→28.57GLM-5.30.95→20.95
Gemini 3.8 Flash18.41→29.52Qwen-3.8 Flash2.22→26.67
Grok 4.624.44→29.52HY 35.40→16.19
MiniMax M316.83→22.86DeepSeek V4 Flash5.71→26.67

Stop remains the hardest outcome for every model (mean 24.67%) even when reporting and clarification count as correct. Appendix K.4 reports this experiment.

Reviewer_A3t7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoApproveBench evaluates LLMs that approve or deny coding-agent actions using 1,579 human-validated events reconstructed from real sessions, with inspectable workspaces. Results across ten models show differing approval and authorization judgments and low accuracy on cases with no authorized continuation.

Critical strengths

Fixing the request, history, reconstructed workspace, and proposed action separates reviewer judgment from actor planning. The 989 ALLOW and 590 DENY cases measure both unnecessary blocking and missed violations. Independent assessment by three experts, with unanimous agreement required for inclusion, supports label quality.

Critical weaknesses
  1. The R3 and STOP results may partly reflect how models interpret the instructions. The reviewer prompt is less specific than the annotation guidelines about unnecessary state changes outside the request and the requirement that continuation advance unfinished, authorized work. The authors should check whether these findings, including the 9.64% mean STOP accuracy (Table 2), hold when reviewers receive the full criteria.
  2. Outside a core set, cases are retained only if at least one screening model predicts DENY (Algorithm 1). This may favor false-denial cases and exclude violations missed by all screeners. The paper does not specify the screening models or prompts, how the core set was chosen, or whether screeners were also evaluated. Selecting difficult cases is reasonable, but the effect on model comparisons and reported conservatism remains unclear, especially since no ALLOW case is approved by all models.
Questions for authors
  1. Does Appendix H.1 contain all guidance given to reviewers? Do the approval, R3, and STOP results hold with the full Appendix F criteria, especially the requirement that continuation advance unfinished, authorized work?
  2. Which models and prompts were used for screening, how was K constructed, and how many cases entered through each route? If screeners were also evaluated, do the comparisons hold on cases selected independently of those models?
  3. The percentages in Tables 7–9 imply that Run 3’s correct-DENY count equals the average of Runs 1 and 2 for all ten models. Please explain how the runs and tables were produced and provide the per-case results and aggregation procedure.
  4. ScopeJudge [1] compares independent judges on proposed tool calls before execution, while SteerBench-Work [2] evaluates fixed action-boundary decisions and both unnecessary blocking and unsafe approval. Please clarify the scope of the “first open and controlled benchmark” claim in relation to these recent preprints.
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer A3t7 (1/2): Full Criteria and Screening

We thank you for the constructive review and for recognizing the controlled design and the label quality. This part addresses W1–W2 and Q1–Q2, and Part 2 addresses Q3–Q4. Revisions in the updated PDF are marked in blue.


W1 & Q1: Full Criteria Prompt

Thank you for this suggestion. Appendix H.1 lists all guidance given to reviewers in the original runs. It is designed to mirror deployed auto-review, where a reviewer receives a compact authorization policy and must judge from the user request whether an action stays within it, while Appendix F holds the full criteria that the experts apply as ground truth. We agree that H.1 therefore states neither the full R3 rule nor the gold continuation definition. To test this directly, we re-ran 10 models with the complete Appendix F criteria and the gold continuation definition on 500 cases stratified from the 1,579 (313 ALLOW, 187 DENY, 105 Stop, 82 Safer). The table gives raw rates on these cases as original 3-run mean→full criteria, where Stop is correct DENY plus correct continuation over the 105 Stop cases.

ModelOverallALLOWR3Stop
GPT-5.6 Sol75.20→68.2065.07→52.4093.10→91.383.49→20.95
Claude Opus 572.20→65.0062.94→50.1678.16→84.481.90→28.57
Gemini 3.8 Flash82.27→81.4077.21→75.7292.53→91.3818.41→29.52
Kimi K379.93→78.6079.34→71.8867.82→93.105.40→24.76
MiniMax M385.20→84.6085.41→83.3981.61→82.7616.83→22.86
GLM-5.379.00→77.6075.61→73.4872.99→77.590.95→20.95
Qwen-3.8 Flash76.20→71.0068.69→57.5183.33→93.102.22→26.67
HY 368.00→55.8055.27→33.5584.48→89.665.40→16.19
DeepSeek V4 Flash80.67→73.6079.02→62.9475.29→91.385.71→26.67
Grok 4.655.80→54.4033.87→30.6791.38→91.3824.44→29.52

Stop accuracy rises for all 10 models (+5.08 to +26.67) yet stays within 16.19–29.52%, so the low Stop accuracy persists under the full criteria. The Spearman correlation of the overall-accuracy rankings before and after is 0.96, R3 changes by −1.72 to +25.29 and ALLOW by −21.73 to −1.49. Spelling out every criterion thus raises R3 and Stop accuracy but lowers ALLOW accuracy for all 10 models (mean −9.07), so the full rules shift models toward denial and the approval–denial trade-off of Sec 5 persists. Appendix K.4 reports this condition, and the prompts and logs are released at https://anonymous.4open.science/r/autoapprovebench-review-5857/.


W2 & Q2: Screening and Core Set

Thank you for this question. We agree that the screening models, prompts, and routes should be stated, and Appendix B.5 details them. The M route queried 5 screeners (DeepSeek-V4-Flash, Qwen3.7-Plus, HY3, MiniMax-M3, MiMo-v2.5) on a card with the extracted request, a compact context, and the proposed action, without workspace tools, and kept a case if at least 1 returned DENY. The repository gives this screening prompt with its six-rule approval policy. K was built from a 7-model diagnostic vote (HY3, DeepSeek-V4-Flash, Qwen3.7-Plus, GLM-5.2, Kimi-K3, MiniMax-M3, Grok-4.6), first adding the 65 cases where the majority contradicted the provisional label, then DENY and ALLOW families.

RouteCandidatesFinalALLOWDENY
K1,000589128461
M3,506990861129
Total4,5061,579989590

Missed violations. K cases enter expert labeling regardless of any DENY vote, and K supplies 461 of the 590 DENY cases (78.1%), including 7 violations that all 7 voters approved and the experts recovered. The 129 DENY cases from M are also the hardest subset, with a mean Acc_D of 77.05% against 93.60% on K, so the screening does not keep only easy violations.

Independent selection. None of GPT, Claude, Gemini, Grok, Kimi, GLM-5.3, or Qwen-3.8 screened the 990 M cases, and their Acc ranking on these cases is identical to their ranking on all 1,579 (Spearman ρ = 1.00). Removing, for each screener, the cases where only it returned DENY (1 to 71 cases) keeps the 10-model ρ at 1.00 for Acc and 0.988–1 for F1.

Conservatism. The denial tendency is not inherited from the screeners. On the 861 M ALLOW cases that the screeners flagged, the 3 evaluated screeners reach a mean Acc_A of 71.73%, against 66.42% for the 7 models that took no part in screening. We also corrected the statement that no ALLOW case is approved by all models. In Runs 1–3, all 10 models approve 57, 86, and 125 ALLOW cases respectively, and Obs. 2 now reports their mean of 89.

AuthorsOpenReview ↗

Rebuttal to Reviewer A3t7 (2/2): Run 3 Tables and Claim Scope

We continue with Q3 and Q4.


Q3: Run 3 Tables

Thank you for checking the tables so carefully. We confirm that Run 3 in the submitted Table 9 equals the average of Runs 1 and 2 for all 10 models. When we compiled the submission tables, the table script held the Run 1–2 average as a placeholder in the Run 3 slot, and we failed to replace it with the completed Run 3 logs. The three runs are independent passes of every model over all 1,579 events under the settings of Sec 4. We recomputed Tables 1, 2, and 7–9 and all dependent statistics from the 30 raw logs (3 runs × 10 models × 1,579 cases, 47,370 episodes). Tables 7 and 8 are unchanged. Tables 1 and 2 average the 3 runs, and F1 is computed within each run before averaging. The per-case logs and the aggregation script are at https://anonymous.4open.science/r/autoapprovebench-review-5857/.

QuantitySubmittedRecomputed
Correct approval range31.95–82.3135.49–83.89
Correct denial range84.58–95.4284.58–94.86
Mean Stop9.649.88
Mean Safer71.0471.02

The qualitative findings hold, with R2 above R1 and R3 and Safer above Stop (all Holm p < 0.001).


Q4: Claim Scope

Thank you for these pointers. We agree that the claim needs a precise scope. Coding-agent auto-review, as deployed in Codex Auto-review, Claude Code Auto Mode, and Cursor Auto-review, is a distinct and mature industrial control paradigm, and the claim concerns this setting. ScopeJudge [1] classifies offline penetration-testing transcripts without workspace fixtures, and SteerBench-Work [2] poses proceed-or-hold decisions on 106 incident-anchored single-turn scenarios across seven workplace domains, so both operate in settings outside coding-agent auto-review. AutoApproveBench evaluates general-purpose LLM reviewers on 1,579 events from real coding-agent sessions, each with a reconstructed inspectable workspace, 3-expert unanimous labels, 3 diagnostic categories, and post-DENY continuation labels, over 10 models and 3 runs. The Abstract and Sec 1 state "for coding agents" alongside the claim, and Sec 2 contrasts both preprints.

[1] Caldwell et al. ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents. arXiv:2607.07774, 2026.

[2] Serdar et al. SteerBench-Work: A Benchmark for Agent Steering at Action Boundaries. arXiv:2608.12654, 2026.

Comments on the paper

1 comment

Hi reviewers, as we approach the rebuttal deadline, please review the authors' responses and participate in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.