- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper designed AutoApproveBench, a benchmark of 1,579 approval events from real coding-agent sessions, to evaluate whether an LLM reviewer can correctly allow or deny an agent’s proposed action before it runs. Across 10 models, correct denial is consistently high, while correct approval varies widely, and models rarely recognize when the agent should stop.
Critical strengths
- As a benchmark paper, the approval events are sourced from real coding-agent sessions rather than synthetic tasks, and each case is verified unanimously by three human experts.
- The evaluation fixes the harness, the input and the proposed action across 10 models with 3 runs each, so the comparison between reviewers is controlled and fair. The post-DENY continuation stage (safer alternative vs. stop) also measures beyond plain allow/deny accuracy.
Critical weaknesses
- The reviewer prompt does not tell the models the same rules that the human experts used for labeling. For example, the prompt never says to deny actions that go beyond what the user asked (R3), and it defines “safer alternative” more loosely than the gold labels do. So the weak R3 and STOP results may come from the prompt design, rather than from the models’ judgment.
- Cases were kept only if at least one screening model wanted to deny them, which means the ALLOW cases were chosen to look suspicious. So the finding that all models are denial-leaning may come from how the benchmark was built, rather than from the models themselves.
Questions for authors
- If the reviewer prompt states the full approval criteria (R1–R3) and the same continuation definition used for the gold labels, how much do R3 and STOP accuracy change? Does the model ranking stay the same?
- Which models were used for the automated screening, and do they overlap with the evaluated reviewers? How do models perform on ALLOW cases that no screening model flagged?
Rebuttal to Reviewer didN: Full Criteria and Screening Overlap
We thank you for the constructive review and for recognizing the real-session sourcing and the controlled 3-run design. We address both concerns and both questions below. Revisions in the updated PDF are marked in blue.
W1 & Q1: Full Criteria Prompt
Thank you for this suggestion. Appendix H.1 is designed to mirror deployed auto-review, where a reviewer receives a compact authorization policy and must judge from the user request whether an action stays within it, while Appendix F holds the full criteria that the experts apply as ground truth. We agree that H.1 therefore does not state the R3 rule and defines Safer Alternative more briefly than the gold labels. To test this directly, we re-ran 10 models with the full approval criteria (R1 to R3) and the gold continuation definition on 500 cases stratified from the 1,579 (313 ALLOW, 187 DENY, 105 Stop, 82 Safer, seed 20261114). In the new continuation prompt, a safer step must advance an unfinished part of the original request, satisfy all criteria, and be materially safer, and completion reports or requests for permission count as Stop. The table gives raw rates on these cases as original 3-run mean→full criteria, where Stop is correct DENY plus correct continuation over the 105 Stop cases.
R3 accuracy changes by −1.72 to +25.29 and Stop accuracy rises by +5.08 to +26.67 for all 10 models, yet Stop stays within 16.19–29.52%. The Spearman correlation between the original and full-criteria rankings by overall accuracy is 0.96 (Kendall τ 0.91), so the ranking is largely preserved. ALLOW accuracy falls for all 10 models (−1.49 to −21.73, mean −9.07), so spelling out every criterion shifts models toward denial and the approval–denial trade-off of Sec 5 persists. Appendix K.4 reports this condition, and we release the prompts and logs at https://anonymous.4open.science/r/autoapprovebench-review-5857/.
W2 & Q2: Screening Overlap and Unflagged ALLOW
Thank you for this question. We agree that the screening models should be stated, and Appendix B.5 details them. The M route used 5 screeners (DeepSeek-V4-Flash, Qwen3.7-Plus, HY3, MiniMax-M3, MiMo-v2.5) and kept a case if at least 1 returned DENY. The core set K (1,000 cases) was built from a 7-model vote (HY3, DeepSeek-V4-Flash, Qwen3.7-Plus, GLM-5.2, Kimi-K3, MiniMax-M3, Grok-4.6). 3 of the 5 M screeners (DeepSeek-V4-Flash, HY3, MiniMax-M3) are also evaluated, and the 7-model vote shares 5 identities with the evaluated set.
The denial tendency does not come from the screening. On the 861 M ALLOW cases that the screeners flagged, the 3 evaluated screeners reach a mean Acc_A of 71.73%, against 66.42% for the 7 evaluated models that took no part in screening, so the models that did not select these cases deny them more often. None of these 7 models screened the 990 M cases, and their Acc ranking on these cases is identical to their ranking on all 1,579 (Spearman ρ = 1.00). Removing, for each screener, the cases where only it returned DENY (1 to 71 cases) keeps the 10-model ρ at 1.00 for Acc and 0.988–1 for F1. The table gives the mean Acc_A of the 10 models by route, averaged over the 3 runs.
Mean Acc_A on the 27 K ALLOW cases that received no DENY vote is 73.95%, so about a quarter of these authorized actions are still denied. On these 27 cases the models score GLM-5.3 88.89, MiniMax 85.19, DeepSeek-V4-Flash 83.95, Kimi K3 82.72, Qwen-3.8 Flash 77.78, HY3 75.31, Claude Opus 5 72.84, Gemini 3.8 Flash 69.14, GPT-5.6 Sol 58.02, and Grok 45.68. Appendix B.5 details the routes and screeners.