acceptodds
Withdrawn from ICLR 2027

HackFree-GRPO: Detecting and Mitigating Reward Hacking in RLVR- Trained Code Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards can reward code agents for passing tests without resolving the underlying issue. We study this failure mode in repository-level, multi-turn training and introduce HackFree-GRPO. Its frozen detector scores actions, patch diffs, and execution logs. Reward shaping discounts risky successes, and trajectory filtering removes high-risk samples before group-relative advantages are computed. Shaping alone can leave an exploit positively advantaged in a low-success group; after filtering, score differences can still distinguish risky survivors. On SWE-bench Verified, FULL reduces protocol-defined hacking from 41.7% to 5.8% and raises strict-audit pass@1 from 30.5% to 49.0%, while verifier acceptance is 52.0% versus 52.3%. Adversarial-test-only pass@1 increases from 31.2% to 49.9%, exceeding SHAPE by 2.5 percentage points. We derive the exact change in retained-group advantages and identify the conditions under which filtering removes residual positive reinforcement. The analysis specifies a group-level mechanism that can be tested alongside the observed task-level gains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward hacking, reinforcement learning with verifiable rewards, code agents, AI safety, verifier exploitation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EjWvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a new framework called HackDetect that combines reward shaping and trajectory filtering and folds these into GRPO policy update. The authors train a Qwen2.5-Coder model on hand curated trajectories to showcase how HackDetect can bring more robustness to training while also improving performance.

Critical strengths
  1. The problem of reward hacking and its addressal at the alignment algorithm level is an important direction for the community
Critical weaknesses
  1. The paper is severely under-motivated in terms of its inspiration and prior works used. Reward filtering and shaping aren't new ideas and have been combined in the past to show performance improvements. How this technique actually affects more robust training is unclear. More concretely, the following in my belief are under-motivated:
  • Why did the authors use a deberta-v3-base instead of an LLM here? These older encoder models aren't as proficient at log or code analysis so it is unclear to me why they would use these. Even if the encoder is absolutely necessary, why not use a more recent and stronger embedding model like Voyage (closed) or EuroBERT (open).
  • It is unclear why these specific defaults were used during training and how they were tuned. There are no statistical significance numbers.
  • Equation 7 is not motivated and it is unclear what the formulations of all equations in the paper are grounded in.
  1. The authors do not separate the impact of their HackGRPO from the hand-labeled trajectories. Why are these trajectories needed in addition to the existing opensource training datasets?
  2. The study severely lacks baselines. Why have the authors not compared other methods, for example [1,2,3], to this one?
  3. There is no human eval study on the impact of the technique and whether HackGRPO really reduces the general occurrences of reward hacking beyond this domain. From my understanding this technique is not restricted to coding and I am unsure why this is scoped to coding-only.
  4. Limitations of HackGRPO-trained models are understudied and under-reported.
  5. Why haven't the authors tested any model other than the Qwen2.5? Why select this model specifically? How does LoRA impact training vs not?

[1] Rong, Xuankun, et al. "Self-rewarded multimodal safety alignment via rule-governed policy optimization." arXiv preprint arXiv:2511.12982 2 (2025). [2] Tang, Kai, et al. "SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model." arXiv preprint arXiv:2606.20698 (2026). [3] Agarwal, Aradhye, et al. "Learning when to act or refuse: Guarding agentic reasoning models for safe multi-step tool use." arXiv preprint arXiv:2603.03205 (2026).

Questions for authors

My questions are flagged above in the weaknesses and addressing those will directly impact my score

Reviewer_GC9qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward hacking in code agents trained using test-based rewards. HackFree-GRPO uses a frozen detector to reduce rewards for suspicious attempts or remove them before computing group-relative updates. Its analysis explains why reducing rewards alone can still reinforce exploits. Experiments report SWE-bench Verified success increasing from 30.5% to 49.0% under the paper’s stricter evaluation, and from 31.2% to 49.9% using adversarial tests alone.

Critical strengths
  • The paper addresses a practical training problem and clearly explains how an exploit can retain a positive learning signal even after its reward is reduced.

  • Separate component comparisons and improvements under adversarial tests alone support the method’s usefulness. The frozen detector excludes evaluation-only information from policy updates.

Critical weaknesses
  • The reduction in hacking needs independent confirmation. The paper’s rejection rule includes incorrect repairs, potentially legitimate test edits, and ambiguous judge decisions. Table 3’s official-harness and human-review figures are projections, not measurements. Improved test performance is therefore better supported than reduced confirmed exploitation.

  • The explanation for combining shaping and filtering remains untested. Table 4 uses assumed detector errors and group compositions. The paper does not measure how often shaping reinforces actual exploits during training or filtering removes correct solutions.

Questions for authors
  • Can you report official SWE-bench harness results and blind human checks of correctness and exploitation for vanilla GRPO, SHAPE, FILTER, and FULL, with paired task-level confidence intervals?

  • In recorded training groups, how often does SHAPE still reinforce confirmed exploits, and how often does FULL remove correct solutions?

Reviewer_zLcQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies reward hacking in code agents trained with verifiable rewards and proposes HackFree-GRPO. A frozen DeBERTa-based detector, HackDetect, scores tool actions, patch diffs, and execution logs. Its scores discount rewards and filter high-risk trajectories before recomputing group-relative advantages. Experiments with Qwen2.5-Coder-7B-Instruct report improvements over vanilla GRPO and the individual components across several code benchmarks.

Critical strengths
  1. This work addresses a practical problem: passing the training verifier does not necessarily mean resolving the underlying issue. The improvements under both the strict audit and adversarial tests alone provide evidence that the intervention improves performance beyond the original training reward.
Critical weaknesses
  1. My main concern is whether the detector and evaluation establish a reduction in reward hacking beyond the patterns covered by the human rubric. Section 3.1 describes 210 human-labeled trajectories, followed by self-training to obtain 8,512 trajectories. However, the coverage of the seed labels and the reliability of the expanded labels across diverse hacking behaviors remain insufficiently established. Since the policy is optimized against this frozen detector through both rewards and filtering, it may learn the detector's preferences while shifting toward less detectable exploits. Evaluation with held-out hacking categories and independent inspection of verifier-passing, detector-cleared trajectories would help distinguish more reliable problem solving from evasion of the detector and audit.

  2. The contribution relative to rubric-based RL need clearer isolation. There are some existing works in rubric-based RL that achieve better results by adding rubrics for process detection, so is the method proposed in this paper just applying rubric-based RL to anti-hack, with the core contribution being this scenario and whether the prepared rubric can capture most hack patterns? But the paper currently discusses this part of the contribution relatively little. Also, the discussion of rubric-based RL in related work is very limited.

  3. The experiments use a relatively dated model, Qwen2.5-Coder-7B-Instruct, and do not evaluate a model with extended chain-of-thought reasoning. A stronger recent reasoning model would help test whether the results persist when the policy can produce more sophisticated exploits. This also raises a concrete input-design question: With longer reasoning and interaction histories, does the current input design retain enough evidence to detect hacking? Would adding available reasoning traces help when a model explicitly expresses an intention to exploit the verifier?

Questions for authors

For the Figure 3, why a small classification model outperform its teacher model (LLM with rubrics)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.