- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce a new framework called HackDetect that combines reward shaping and trajectory filtering and folds these into GRPO policy update. The authors train a Qwen2.5-Coder model on hand curated trajectories to showcase how HackDetect can bring more robustness to training while also improving performance.
Critical strengths
- The problem of reward hacking and its addressal at the alignment algorithm level is an important direction for the community
Critical weaknesses
- The paper is severely under-motivated in terms of its inspiration and prior works used. Reward filtering and shaping aren't new ideas and have been combined in the past to show performance improvements. How this technique actually affects more robust training is unclear. More concretely, the following in my belief are under-motivated:
- Why did the authors use a deberta-v3-base instead of an LLM here? These older encoder models aren't as proficient at log or code analysis so it is unclear to me why they would use these. Even if the encoder is absolutely necessary, why not use a more recent and stronger embedding model like Voyage (closed) or EuroBERT (open).
- It is unclear why these specific defaults were used during training and how they were tuned. There are no statistical significance numbers.
- Equation 7 is not motivated and it is unclear what the formulations of all equations in the paper are grounded in.
- The authors do not separate the impact of their HackGRPO from the hand-labeled trajectories. Why are these trajectories needed in addition to the existing opensource training datasets?
- The study severely lacks baselines. Why have the authors not compared other methods, for example [1,2,3], to this one?
- There is no human eval study on the impact of the technique and whether HackGRPO really reduces the general occurrences of reward hacking beyond this domain. From my understanding this technique is not restricted to coding and I am unsure why this is scoped to coding-only.
- Limitations of HackGRPO-trained models are understudied and under-reported.
- Why haven't the authors tested any model other than the Qwen2.5? Why select this model specifically? How does LoRA impact training vs not?
[1] Rong, Xuankun, et al. "Self-rewarded multimodal safety alignment via rule-governed policy optimization." arXiv preprint arXiv:2511.12982 2 (2025). [2] Tang, Kai, et al. "SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model." arXiv preprint arXiv:2606.20698 (2026). [3] Agarwal, Aradhye, et al. "Learning when to act or refuse: Guarding agentic reasoning models for safe multi-step tool use." arXiv preprint arXiv:2603.03205 (2026).
Questions for authors
My questions are flagged above in the weaknesses and addressing those will directly impact my score