- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies inference time refinement for secure code generation as a game between a generator and a critic. The generator writes and revises code according to critic feedback for up to five rounds, and the final code is evaluated on held out benchmark tests that neither agent sees. The study compares four feedback types under a shared critic backbone: executable testcases, open ended natural language critique, critique grounded in observed test failures, and a hybrid that pairs tests with Semgrep findings. Experiments cover three generators of different capability on SeCodePLT and BaxBench and report capability, security, and correct and secure rates. Three findings emerge. Refinement improves the correct and secure rate over direct generation, but no feedback type uniformly dominates. Reward hacking is a primary bottleneck: generators often emit edits that satisfy the critic while failing held out tests. The authors formalize this behavior with a taxonomy of five categories, validate syntactic indicators on 150 manually checked trajectories, and show that critic satisfaction keeps rising with more rounds while true performance plateaus.
Critical strengths
1.To my knowledge this is the first systematic study of reward hacking in secure code generation, and it delivers an instrument rather than an anecdote. The taxonomy is illustrated with convincing worked examples, the syntactic indicators are validated on 150 manually annotated trajectories with false alarm rates below 6 percent, and a further decomposition attributes about 74 percent of cases to generator shortcuts rather than incomplete feedback. The community can reuse this measurement setup to detect a failure mode that aggregate pass rates cannot reveal.
2.The joint correctness and security objective exposes failure modes that a single metric cannot see, and the experimental design isolates the feedback axis cleanly. Blanket rejection looks like pure improvement under a safety only lens, and only the joint metric reveals it. Holding the critic backbone fixed across feedback types, evaluating only on held out tests, and spanning generators from 1.7B to frontier scale make the comparison credible. The finding that grounding natural language critique in observed test failures outperforms either raw signal is not obvious and directly useful in practice.
Critical weaknesses
-
The central claim that reward hacking is a primary bottleneck is not adequately quantified. The phenomenon of models exploiting weak proxies is well known, so the contribution rests on quantitative rigor, and this is where gaps appear. The hacking pool is defined after the fact as games that end critic satisfied but benchmark failing, its absolute size per feedback type and generator is never reported, and the proxy reward uses a different scale for each feedback type, so satisfaction levels are not comparable across settings. Without pool sizes the reader cannot verify that hacking, rather than ordinary failure, explains the remaining gap. The data appears recoverable from existing logs, so the issue is fixable.
-
The critic is thin and effectively singular, and no mitigation is tested even though the data already suggests one. The critic writes only two tests per round, so proxy strength is a confound: hacking prevalence may partly reflect this budget and the choice of a single critic family, and the stronger critic ablation covers only SeCodePLT with no significance testing. More importantly, the conclusion is prescriptive, yet no guard is tried. Fig 3a shows that true reward gains concentrate in the first round, so best of rounds selection or an early stop rule triggered by destructive indicators costs nothing new to compute and directly tests the bottleneck claim.
Questions for authors
1.What fraction of all games, per feedback type and generator, end in the reward hacking pool, and what share of all failures does the pool constitute? Does grounded critique induce fewer hacking cases in absolute terms?
2.Does hacking shrink when the critic writes more tests per round, for example five to ten? This separates generators hacking weak proxies from generators hacking any local proxy.
3.Are the gaps between feedback types in Table 2 statistically significant, for example 61.0 versus 60.2 versus 58.4 for Claude Opus 4.7 on BaxBench? Please state the sample size per benchmark.
4.From existing per round data, what correct and secure rate does first round stopping or best of rounds selection achieve per feedback type?
4 comments
Author Response (Part-2)
From existing per round data, what correct and secure rate does first round stopping or best of rounds selection achieve per feedback type? Does hacking shrink when the critic writes more tests per round, for example five to ten?
Using our existing per-round trajectories, we compute the C&S rate under two alternatives to running the full refinement game. First-round stopping keeps the code after the first revision. Best-of-rounds is an oracle that selects any round whose code passes the held-out tests. Since best-of-rounds uses the held-out ground-truth tests, it is an upper bound on what any round-selection strategy could achieve. The table reports First-round / Full refinement (Table 2) / Best-of-rounds.
| Benchmark | Generator | Testcase | NL | AGNL | Hybrid |
|---|---|---|---|---|---|
| SeCodePLT | GPT-4o | 57.4 / 60.4 / 64.2 | 53.1 / 52.8 / 59.0 | 53.8 / 58.9 / 62.6 | 54.0 / 56.3 / 60.0 |
| SeCodePLT | Claude-Opus-4.7 | 61.5 / 61.3 / 64.6 | 59.6 / 58.6 / 61.7 | 60.6 / 63.0 / 66.3 | 59.6 / 59.8 / 61.4 |
| SeCodePLT | Qwen3-1.7B | 34.2 / 39.1 / 43.7 | 36.4 / 39.6 / 48.8 | 35.7 / 39.4 / 46.4 | 30.4 / 36.8 / 42.0 |
| BaxBench | GPT-4o | 23.7 / 25.0 / 34.4 | 22.7 / 23.0 / 38.3 | 26.8 / 27.0 / 38.0 | 25.3 / 24.5 / 35.5 |
| BaxBench | Claude-Opus-4.7 | 57.7 / 58.4 / 65.3 | 55.9 / 56.9 / 69.9 | 59.7 / 61.0 / 70.7 | 60.7 / 60.2 / 68.4 |
| BaxBench | Qwen3-1.7B | 2.0 / 2.0 / 2.3 | 2.3 / 5.9 / 7.2 | 1.8 / 7.7 / 8.4 | 2.8 / 2.8 / 3.6 |
We note that full refinement beats first-round stopping in most settings and slightly underperforms or ties in others. The gains are largest for the weakest generator. For Qwen3-1.7B, full refinement adds 3.2-6.4 points on SeCodePLT, and on BaxBench AGNL improves from 1.8 to 7.7. As expected, best-of-rounds exceeds full refinement, by 1.6-9.2 points on SeCodePLT and by up to 15.3 points on BaxBench. This gap measures code that was correct and secure at some round but was later broken by further refinement. It therefore gives an upper bound on the cost of the over-refinement and reward hacking analyzed in Section 4. Overall, these results support our conclusion that more refinement does not monotonically improve code. A practical stopping or selection rule that does not rely on the held-out tests could recover part of this gap, for example by regression-checking each revision against tests from earlier rounds. We see this as a promising direction and have added this analysis to the revised paper.
To test whether more feedback per round reduces reward hacking, we ran an additional ablation on SeCodePLT with GPT-4o as the generator. Since o4-mini has been retired from the OpenAI API, we use gpt-5.4-mini as the critic and re-run the default feedback types with it for a matched comparison. More executable tests help testcase feedback. With five capability and five safety tests per round instead of one each, testcase feedback reaches 62.5% C&S, compared with 59.6% by default and 55.1% for the baseline. It fixes more tasks (88 vs. 77) and regresses fewer (28 vs. 41), suggesting that richer tests leave less room for shortcut fixes. In contrast, asking the natural-language critic for more findings per round (8-12) hurts. NL and AGNL fall to 42.9% and 45.9%, below the baseline, and regress 173 and 149 tasks, compared with 33 and 12 by default. Most of the damage is to functionality: capability drops from 69.7% to 55.4% (NL) and 60.1% (AGNL), while security barely changes. Since the critic must keep producing findings, it rarely declares itself satisfied, and many of its findings go beyond the task specification or contradict earlier rounds. In one template-rendering task, for example, the critic asks the generator to both reject and allow literal braces, and the generator ends up failing both capability and security. More executable tests thus reduce regressions under testcase feedback, while more unverifiable critique leads the generator to break working code. We have added this ablation to the revised paper.
Author Response (Part-3)
Proportion of different failure modes. Does grounded critique induce fewer hacking cases in absolute terms?
Each feedback type is evaluated on 819 games on SeCodePLT and 392 games on BaxBench per generator. We compare the final code of each game with the zero-shot baseline on the same task and call a task “fixed” if the baseline fails it but the refined code is correct and secure, and “regressed” if the baseline is correct and secure but the refined code is not. The core failure mode in this setting is performance regression: across all generators and feedback types, feedback regresses roughly one task for every two it fixes (1,343 fixed vs. 653 regressed). For example, with testcase feedback, GPT-4o on SeCodePLT fixes 56 tasks while regressing 14. With AGNL, Claude-Opus-4.7 fixes 54 tasks while regressing 21 on SeCodePLT, and 58 while regressing 37 on BaxBench. The net gains in Table 2 are therefore much smaller than they would be if refinement preserved the tasks the baseline already solved. For example, AGNL fixes 62 tasks for GPT-4o on BaxBench, which alone would be a gain of 15.8 points, but it also regresses 31, so the net gain is 7.9 points. In some settings feedback hurts outright: plain NL regresses more tasks than it fixes for both GPT-4o (33 vs. 12) and Claude-Opus-4.7 (12 vs. 10) on SeCodePLT.
Regressions arise both in games where the critic is eventually satisfied and in games that run to the round budget. In the latter, the generator often satisfies one round’s concerns only for the critic to raise new ones, so regressions can accumulate through edits that address earlier feedback without solving the task. For the manual reward-hacking analysis, we focus on the former: games that converge before the final round, yet whose final code regresses in correctness or security relative to the game’s initial code. Here the generator has fully satisfied the critic, so the gap between the critic’s signal and the true objective is directly observable. This yields 124 cases on SeCodePLT and 147 on BaxBench. In terms of proportions, regressions account for about 8% of all games whose final code fails; the remaining failures are tasks the baseline also fails and refinement does not fix. Reward-hacking cases account for about 3.4% of failing games. Per generator, the pool covers 2.4% of all games for GPT-4o and 2.3% for Claude-Opus-4.7, vs. 0.9% for Qwen3-1.7B, since the weaker generator produces far less correct and secure code that refinement could regress. Per feedback type, it covers 1.8-2.0% of games (73 cases for AGNL, 68 for testcase, 66 for NL and 64 for Semgrep-hybrid), and individual generator × feedback × benchmark settings range up to 7.4% (GPT-4o with AGNL on BaxBench). Grounded critique therefore does not reduce reward hacking in absolute terms. Its advantage comes from fixing more tasks: AGNL fixes the most tasks in 5 of the 6 generator × benchmark settings, while its regressions and reward-hacking cases remain comparable to the other feedback types.
take part in the discussion
Dear reviewer 75Bn,
please take part in the discussion. time is runnning out.
thank you your ac
Author Response (Part-1)
Thank you Reviewer 75Bn for your feedback. We appreciate that you recognize that our work is the first systematic study of reward hacking in secure code generation and that our experiment design isolates the feedback axis cleanly and evaluation includes generators from 1.7B to frontier scale, making the comparison credible. We address your concerns below.
Are the gaps between feedback types in Table 2 statistically significant? Please state the sample size per benchmark
We first note that SeCodePLT has 819 tasks and BaxBench has 392 tasks. Each cell in Table 2 (generator × feedback type) is evaluated on all tasks of its benchmark. Since all methods are run on the same tasks, we use paired tests to evaluate whether feedback types differ significantly. Specifically, for each generator × benchmark setting, we run pairwise McNemar tests [1,2] with Holm correction across the 6 pairs [3]. The McNemar test suits this setting because each task gives a paired binary outcome (Correct & Secure or not) under both feedback types. It tests significance using only the discordant tasks (solved by one method but not the other), so it accounts for the strong correlation between methods evaluated on the same tasks. Holm correction controls the family-wise error rate across the multiple pairwise comparisons.
We report the significant pairs for each model × dataset in the table above; AGNL denotes auto-grounded natural-language critique. First, as expected, pairs with larger performance differences are more likely to be significant. For GPT-4o on SeCodePLT, Testcase vs. NL (60.4 vs. 52.8, a 7.6-point gap) and AGNL vs. NL (6.1 points) are highly significant, whereas Testcase vs. AGNL (60.4 vs. 58.9) is not (p = 0.24). Similarly, for Claude-Opus-4.7 on SeCodePLT, AGNL vs. NL (4.4 points) is significant, while Testcase vs. AGNL (1.7 points) is not. For Qwen3-1.7B on SeCodePLT, every feedback type significantly improves over the baseline (Table 2), yet none differs significantly from another. Crucially, however, non-significance does not mean the methods behave the same or fix the same tasks. For Qwen3-1.7B on SeCodePLT, each feedback type fixes a similar number of the tasks that fail at baseline, but these are largely different tasks. Of the tasks fixed by at least one feedback type, only 24% are fixed by all four, and 28% are fixed by only one. Similarly, on BaxBench there is no substantial difference in performance across feedback types for GPT-4o and Claude-Opus-4.7, so we observe no significant differences. Here too, the feedback types fix different tasks. Only 9% (GPT-4o) and 20% (Claude-Opus-4.7) of the fixed tasks are fixed by all four feedback types, while 37-41% are fixed by only one. For Qwen3-1.7B on BaxBench, AGNL and NL significantly outperform testcase feedback: most of its baseline failures are applications that do not work at all, and prose critique helps a weak generator get a working program.
We also note that some CWEs are improved more by some feedback types than by others. For Qwen3-1.7B on SeCodePLT, testcase feedback fixes significantly more vulnerabilities than NL critique on CWEs that can be triggered through execution, including code/eval injection and resource exhaustion (e.g., CWE-94/95/770; p = 0.008). Conversely, NL critique fixes significantly more vulnerabilities than testcase feedback on CWEs that require reasoning about the code’s meaning, including broken cryptography, injection logic, information exposure and authorization (e.g., CWE-327/74/200/863; p = 0.001). These opposing effects cancel in aggregate, which helps explain the overall non-significance. On BaxBench, critique-based feedback likewise helps most on credential protection and resource exhaustion (e.g., for Claude-Opus-4.7 on CWE-522), while testcase feedback remains competitive on execution-visible injection such as XSS and SQL injection (CWE-79/89). We present these per-CWE results as exploratory. They suggest that the feedback types are complementary, and that choosing feedback by vulnerability category is a promising direction for future work. We have added this expanded discussion to the revised paper.
[1] McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2).
[2] Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7).
[3] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2).