acceptodds
Under review as a conference paper at ICLR 2027

Unhack: Robust RLVR Under Unreliable Specifications

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) improves models’ ability to solve tasks in isolation. In deployment, however, models interact with users, tools, and other models, which introduce additional context that may be wrong or outdated. We find that RLVR-trained models often follow this context even when it conflicts with the task objective, including on problems they solve correctly without it. We study this behaviour in three settings: mathematical reasoning with answer hints, code generation with tests, and agentic tool use with action plans. We refer to it as specification anchoring, examine how RLVR training affects it, and introduce UNHACK, an RLVR training method designed to mitigate it. Untrained models already exhibit the behaviour; standard RLVR does not remove it in any domain we test and sometimes amplifies it. Across six model configurations, UNHACK substantially reduces compliance with corrupted specifications in all three settings while matching or improving standard task performance. The improvement generalises to corruption families outside the training support, while models continue to benefit from correct specifications. Common test-time mitigations, including prompt warnings and self-verification, do not reliably prevent this behaviour.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
rlvr, specification gaming, reinforcement learning from verifiable rewards, reward hacking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_drb4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors investigate how corrupted task specifications can mislead LLMs. The method specifically adds corrupted task specifications or misleading hints to the task prompt in order to mislead and fool the LLM. In their evaluations, they show that LLMs are not able to reliably deal with these scenarios and often falter. To improve the robustness of LLMs, they propose UNHACK, where they specifically add corrupted and misaligned task specifications into RLVR training runs, only rewarding the initial original objective. In doing so, they effectively reduce the compliance with the corrupted specifications, while preserving model capabilities.

Critical strengths

The overall motivation sounds reasonable and important to me, and it is concerning to me that LLMs can be fooled that easily.

Critical weaknesses

I am concerned about the framing of hacking and cheating in the paper. It is unclear to me whether the LLM here is actually trying to hack or cheat the benchmark. If the LLM follows the corrupted or misaligned task specifications, this seems to me more a confusion or a misinterpretation of the LLM.

The paper misses a comparison to reasoning models. I would expect current reasoning models, such as the qwen3.6/3.8 models, to already cope quite well with these scenarios. Is this only an artefact of LLM confusion when they do not have the possibility to think it through?

Questions for authors

What would be the expected optimal model response in these scenarios? Especially if the user input is not coherent, what are the LLMs supposed to do? E.g. if the provided test does not match the task itself. Should the model just ignore the test case or the task instructions? Should it ask for clarification or just proceed with solving? How are questions regarding clarifications treated in your evaluation? How often do LLMs ask for clarification or point out inconsistencies in the provided user input?

Could the authors reiterate the three subsets and clarify what, specifically, each subset tests for with respect to hacking/cheating or misleading models?

Table 1-3 seems rather incomprehensible, and I believe restructuring or visualisation would help the presentation.

1 comment

Thank you for suggesting a clearer presentation. Your feedback helped us reorganize the main Results section around Pareto plots of task performance and copying or following. The numerical comparisons are collected in our general comment, Additional baseline results (ID), and in the revised version of the paper.


1. Terminology: what we mean by "cheating"

We agree that "cheating" can sound like a claim about intent. This is why we preferred to talk about “specification anchoring”; when we use “cheating,” we follow the same vocabulary of ImpossibleBench [1] and mean the observable behavior, not intent. Regarding our method's name, UNHACK: this stems from the fact that training RLVR with hints or other privileged information can create reward-hacking behavior, and we provide a way around this.

2. Does reasoning resolve this behavior?

This behaviour also occurs in reasoning models: ImpossibleBench reports 33% cheating for o3 even under its strictest instruction to stop when tests conflict with the specification. We use that same instruction as our defense prompt.

3. What should the model do when specifications conflict?

Great question! Standard RLVR leaves this situation unspecified: models are not explicitly trained to handle conflicting guidance, and typically never encounter it during training. We think a model should stick to the task, flag the conflict, and ask for clarification when needed. UNHACK is a first step—it trains models to finish the task without following corrupted guidance. A natural next step would be to reward clarification across turns. Even our strict defense prompt, which tells models to identify inconsistencies, leaves substantial anchoring.

4. How are clarification responses treated?

We don't count a response without a final answer (no \boxed{} field) as copying, but it also gets no credit for solving the problem.

5. How often do models flag the conflict?

As a rough proxy for clarification or confusion, we count responses that mention both the correct value and wrong hint without choosing either. This happens in 1.25% of RLVR responses, 0.96% for UNHACK, and 5.78% for Base. Responses matching this proxy are therefore uncommon. Even across five attempts, a correct answer appears for only 9.7–23.8% of problems under wrong hints, despite much higher accuracy without hints. Together with the low rate of responses expressing uncertainty, this suggests that models tend to follow the wrong hint rather than remain undecided between the two answers.

6. What does each subset test?

Your suggestion prompted us to simplify the presentation by combining the three types of accuracy into one score: their geometric mean, GM3GM_3. Each matters for a different reason: accuracy without guidance shows what the model can solve on its own; accuracy with correct guidance shows whether it benefits from useful information; and accuracy with wrong guidance shows whether it resists misleading information. They need to be interpreted together, because high correct-guidance accuracy alone can simply reflect copying. For coding and ALFWorld, the last score counts only tasks completed without following the mistake.

We can then plot GM3GM_3 against copying or following in a Pareto plot, making the balance between task performance and resistance to corrupted guidance much easier to see. The main-paper plots also include the additional baselines. Thank you for this suggestion, it helped us make the results much easier to compare.

[1] Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases. arXiv preprint arXiv:2510.20270, 2025.

Reviewer_2kynOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies specification anchoring, where language models follow incorrect answer hints, test cases, or action plans despite conflicts with the underlying task objective. It proposes UNHACK, an RLVR training method that mixes correct and corrupted specifications, evaluates outputs against the original ground truth, and penalizes compliance with corrupted specifications. Experiments cover mathematical reasoning, code generation, and agentic tool use across six model configurations. The results show reduced compliance with misleading specifications, including unseen corruption types, while generally preserving performance without auxiliary specifications and retaining benefits from correct ones; prompt warnings and self-verification provide less consistent mitigation.

Critical strengths

None

Critical weaknesses
  1. After reading some literature, I have a strong feeling that this paper does not adequately position UNHACK against relevant training-based approaches. For example, SMART studies RL-based resistance to incorrect user information while evaluating receptiveness to valid feedback, and Seirênes trains reasoning models under misleading hints using ground-truth correctness rewards, including comparisons with static-hint training. Neither is discussed in the current manuscript. These precedents substantially narrow the novelty of training models to resist unreliable context while preserving the original task objective. UNHACK’s rule-based corruptions and cross-domain evaluation provide additional contributions, but the paper does not sufficiently establish what its training recipe adds beyond existing robust-training approaches.
  2. The experiments omit the most direct baseline: training with the same 50/50 mixture of correct and corrupted specifications, under matched training conditions, using only the ordinary ground-truth correctness reward without the additional −1 penalty for following corrupted specifications. The main comparison changes both the training-context distribution and the reward, leaving their contributions unresolved. Unhack-Direct does not address this issue: it retains the anti-cheating signal while removing the requirement for ground-truth correctness. Consequently, the results establish the effectiveness of the complete recipe but do not determine whether the specialized penalty provides meaningful gains beyond exposure to corrupted context. Given the closely related approaches above, this missing control is particularly consequential: the component that could distinguish UNHACK from straightforward robust data augmentation is precisely the component whose independent benefit remains untested.
Questions for authors
  1. Could you add a baseline with the same training data but without the extra −1 penalty, and compare with prior methods like SMART and Seirênes?
  2. What’s the main new insight here? I’m not seeing a clear one beyond what prior work has already shown.
1 comment

Thank you for pointing us to SMART [1] and Seirênes [2], these are very adjacent and we regret overlooking them. We now include controlled adaptations of both, alongside 2 SFT baselines and a RLVR->SFT control.


W1/Q1: Controlled comparisons.

Comparison to SMART and Seirênes. We thank the reviewer for pointing us to these closely related approaches. Our aim is to understand how different ways of augmenting the training context affect specification anchoring. We therefore retain each method's way of varying the guidance during training while keeping our task data, specification generator, RL algorithm, and training budget fixed.

  • SMART-style outcome RL. SMART primarily differs from standard RL through UA-MCTS trajectory generation and intermediate progress rewards. We remove these additional sources of supervision and retain outcome-only RL on prompts containing unreliable context. In our setting, this reduces exactly to the reviewer's requested control: the same 50/50 correct/corrupted specification mixture as UNHACK, scored only by ground-truth correctness, without our cheating penalty. We label this SMART (adapted).

  • Seirênes-style static context. Seirênes jointly trains a reasoner and a co-evolving adversary that generates misleading context to induce "robustness under contextual perturbations". Alongside resistance to misleading context, we study whether models can still use the useful parts of guidance that contains mistakes. Its paper additionally studies offline/static-hint training, as you mentioned. To compare at the same level, we keep our specification generator fixed and adopt the corresponding reasoner-side curriculum: training on a mixture of no-specification and corrupted-specification prompts, using only ground-truth correctness. Unlike UNHACK, this baseline never observes correct specifications during training. We label this Seirênes (adapted).

These controls separate three questions: (1) is exposure to unreliable context alone sufficient; (2) does including reliable specifications during training matter; and (3) does the explicit compliance penalty provide additional benefit? Full SMART and Seirênes introduce orthogonal mechanisms, respectively process-level search/reward and a co-evolving adversary, which could in principle be combined with UNHACK and which we leave to future work.

SFT Baselines: Anti-syc SFT [3] uses correctness-filtered targets; BCT [4] imitates unfiltered clean self-targets under corrupted context. We evaluate both from the base model and after RLVR. The general response, Additional baseline results (ID), collects the GM₃/following values for all nine methods and four models. The main Results section shows the corresponding Pareto plots.

W2/Q1: Why the −1. Our new results clarify its role. On MATH, following a wrong final-answer hint fails the task, so correctness reward already discourages copying. SMART/Seirênes adaptations are competitive with UNHACK here. The penalty can also distinguish failures: when no rollout is correct, a non-copying failure (0) ranks above copying (−1), whereas equal correctness rewards give no relative signal.

Coding and ALFWorld expose a different issue: following the mistake can coexist with passing the task checks. For Qwen-7B ALFWorld, SMART has 91.0% task success under wrong plans but only 72.1% clean success. This is why we measure both success and corrupted-guidance following. UNHACK has the highest plotted ALFWorld GM₃ for all four models, while coding includes successful alternatives. These findings support a domain-dependent benefit, rather than universal method superiority.

Q2: Main insight. Coding and agentic tasks show why task success alone is not enough: a model can follow a wrong test or plan step and still solve the task. The same guidance can also contain useful information, so ignoring it altogether is not the goal. We study how models can learn to use the good parts while resisting mistakes, and when correctness reward alone is enough to teach this.

See the general response for numbers and the main Results section for plots and metric definitions.

[1] Beigi et al., SMART, https://arxiv.org/abs/2509.16742.

[2] Zhang et al., Seirênes, https://arxiv.org/abs/2605.11636.

[3] Wei et al., Simple synthetic data reduces sycophancy in large language models: https://arxiv.org/abs/2308.03958

[4] Bias-augmented consistency training reduces biased reasoning in chain-of-thought: https://arxiv.org/abs/2403.05518

Reviewer_2BJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the issue that model may follow incorrect task related context in prompt, highly related to reward hacking and hallucinations which are all critical and severe for modern LLMs. The paper proposes UNHACK, a training framework for mitigating these cheating or context misleading behaviors by mixing training data with correct and incorrect specification generated by fixed rules.

The paper has thorough experiments spanning math, code and agentic tool use, evaluating on 6 models. The results clearly show that RLVR does not solve the cheating issues but makes it more severe that wrong hint copying rises from 14-46% in base model to 78-89% in RLVR model. While the proposed UNHACK could effectively mitigate the cheating behaviors and improve original task performance such as wrong hint copying in UNHACK model is reduced to 7-15% on math. The test-time mitigation part introduces the prompted based method for RLVR that still cannot mitigate it effectively.

Critical strengths

The problem studied in the paper is really important and appears in many scenarios.

The proposed method is effective for mitigating the cheating behaviors with good reproducibility (fixed rule specification)

The experiments are thorough, covering math, code and agentic tool use with 5 seeds on 6 models. The results are well-stated.

Critical weaknesses
  1. The method lacks some novelty. The proposed method is to train a model on a mixture of data with correct and incorrect context while rewarding the true task objective, while is an established line of work including synthetic data training, training with biased data, etc. UNHACK applying this approach in RL training to mitigate the cheating behaviors is not something brand new. The gain on performance or contribution mainly come from the model being trained and exposed on the data with specification according to the figure 6 that the models trained with different ratio could have decent performance.

  2. The experiments are not well designed. The related baselines are missing. The main baseline is RLVR which is fine, while with the setup about appending hints into prompt, rewarding with rule based verifier (binary verifier), this work is highly related reward hacking as stated in section 2. Therefore, the reward hacking mitigation works especially training related works should be included. In addition, the approaches to robustness with corrupted data should also be considered or just applying DPO on those data. With only plain or prompt based RLVR in comparison, the paper only demonstrates that training on data with corrupted specifications helps compared with no specification, but not proving the proposed method has advantages over prior works.

  3. The contribution 2 in introduction is confusing. It states “In standard RLVR, exposing this information only in correct form…..” which actually refers to the situation of p=0 as impossible ratio (in Appendix H, figure 6) but the main baseline in section 4 is trained on prompts with only task descriptions (in section 4.3).

  4. The comparison is a bit unfair in experiment section. As shown in table 1, 2, 3, the metrics of “cheating” and “follow” are exactly the penalized behaviors during training with reward -1 (section 2.1). By design, it’s expected that RL training will penalize the exact behaviors.

  5. The results are not convincing. No hint accuracy has modest improvement especially on Qwen model with 0.4, 1.0, 1.3 pp improvement. In addition, the table 1 shows for Qwen models, RLVR has almost no improvement on Math. And for ALFWorld, RLVR is unstable and probably not well-tined. As shown in table 3, the performance of Qwen 7B model after RLVR is worse than 3B due to one seed collapses (Appendix Q.4). Llama 3B model is also collapsed after RLVR.

Some related works might be relevant for comparison: Bias-augmented consistency training reduces biased reasoning in chain-of-thought: https://arxiv.org/abs/2403.05518 Simple synthetic data reduces sycophancy in large language models: https://arxiv.org/abs/2308.03958 Consistency training helps stop sycophancy and jailbreaks: https://arxiv.org/abs/2510.27062

Questions for authors

Add some related baselines. Double check the RLVR training.

1 comment

W1: Novelty. Thanks for the references. We agree that learning from synthetic data or unreliable context is an established idea; Wei et al [1] was already cited in our related work. What we study is whether models learn to use context selectively across math, code and multi-step agentic tasks. Coding and ALFWorld expose something that answer-level evaluations do not capture as directly: models can complete the task and still "cheat". Our contribution therefore goes beyond the training recipe.

W2: Baselines. Following your suggestions, we added anti-syc SFT from Wei [1] and BCT [2]. Anti-syc SFT learns from correctness-filtered targets, while BCT teaches the model to give the same answer despite misleading context. We also run both after RLVR to test whether a subsequent SFT stage can fix copying behaviours present in RL.

To distinguish the effect of data augmentation from the penalty, we also added correctness-only RL controls. In particular, one uses the same correct/corrupted mixture as UNHACK, with the same task data and training budget, but without the cheating penalty. The general comment collects the numerical comparisons; the revised main paper contains the plots and discusses what these alternatives teach us.

W3: Contribution #2. That sentence referred to the p=0 ablation, where every training prompt includes correct guidance. Our main RLVR baseline is trained without added guidance (as in standard RLVR training). We have removed the confusing wording from contribution #2.

W4: Penalizing the metric. We agree that lower cheating alone is not enough when that behaviour is explicitly penalized. This is why we report accuracy without guidance, with correct guidance, and with wrong guidance: we want models to solve the task, benefit from useful advice, and resist mistakes. In MATH, copying a wrong answer means failing, so correctness-only training can suffice. In coding and ALFWorld, the same tests or plan can contain both useful information and mistakes, and following the mistake may not prevent success. UNHACK learns to use the good parts of a specification while avoiding the wrong ones. The three accuracies help distinguish this from simply ignoring guidance or reducing copying at the cost of task capability.

W5: RLVR quality. Modest clean-accuracy gains do not, by themselves, indicate faulty RLVR training. For context, DeepSeekMath reports a 4.9-point MATH gain (46.8% → 51.7%) using approximately 144K training questions and 64 sampled responses per question. One pass through that data involves about 9.2M responses; our MATH runs sample 20,000 responses (2,500 updates with a GRPO group of 8), roughly 460 times fewer, yet already reveal severe specification anchoring. Characterizing and mitigating this early-emerging failure mode, rather than fully maximizing benchmark accuracy, is our objective.

Regarding the agentic results on AlfWorld: Seed-dependent collapse does not, by itself, establish that our RLVR baseline is improperly tuned. Such instability is documented in multi-turn agentic RL, including reward-variability collapse and gradient spikes in RAGEN [3], and seed-dependent performance collapse in T2PO [4], which evaluates ALFWorld. Crucially, excluding Qwen’s collapsed RLVR seed does not change the conclusion: ID/OOD following remains 55.2/21.6%, versus UNHACK’s 1.8/1.0%. The behavioral gap therefore cannot be attributed to the collapsed run.

[1] Wei: https://arxiv.org/abs/2308.03958 [2] BCT: https://arxiv.org/abs/2403.05518 [3] RAGEN: https://arxiv.org/abs/2504.20073 [4] T2PO: https://arxiv.org/abs/2605.02178

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank the reviewers for their suggestions and for their patience while we completed the additional experiments. We added four baselines—SMART/Seirênes [reviewer 2kyn] and BCT/anti-syc SFT [reviewer 2BJb]—plus RLVR→BCT and RLVR→anti-syc SFT. SMART uses UNHACK's guidance mixture with correctness reward only, providing the control requested by reviewer 2kyn. Implementation details are in the revised appendix.

Additional baseline results (ID).

Cells show GM₃ ↑ / copying or following ↓, in percent. GM₃ is the geometric mean of accuracy without, with correct, and with wrong guidance. For coding and ALFWorld, wrong-guidance accuracy is clean success: solving the task without following the mistake.

MATH: when copying == failing

Training on wrong hints with correctness reward can be enough here: SMART and Seirênes substantially reduce copying and are competitive with UNHACK. SFT reduces copying further but can weaken capability, which is why we read both scores together.

MethodQwen-3BQwen-7BLlama-3BLlama-8B
Base51.1/39.956.2/50.232.5/14.535.3/16.2
Anti-syc SFT62.1/0.771.0/0.429.7/1.348.1/0.8
BCT63.2/0.770.8/0.315.2/0.645.2/0.4
RLVR31.8/86.837.4/87.926.2/89.924.5/86.0
SMART (adapted)68.1/22.878.0/16.458.4/13.863.1/15.8
Seirênes (adapted)65.7/14.178.6/6.958.3/5.261.9/3.6
RLVR→BCT60.7/0.972.2/0.640.0/1.450.6/1.1
RLVR→anti-syc SFT63.4/0.873.4/0.742.3/2.053.8/1.1
UNHACK66.9/12.177.8/10.157.4/7.861.6/7.1

ALFWorld: when a model can follow a wrong step and still finish the task.

An agent can try to slice a cellphone and still finish the task, just as code can hardcode a false test yet pass the correct checks as long as their inputs don't conflict. This is why our evaluation goes beyond answer-level tests such as SycophancyEval and MATH: models can succeed and "cheat" at the same time. Here, augmentation still leaves substantial following; UNHACK has the highest GM₃ on all four models and much lower following than these RL controls.

MethodQwen-3BQwen-7BLlama-3BLlama-8B
Base12.0/43.319.6/32.88.8/83.227.0/45.6
Anti-syc SFT0.0/14.527.6/4.51.7/3.720.7/3.5
BCT0.0/0.32.3/0.51.4/3.013.3/2.9
RLVR65.9/44.352.1/41.70.2/60.162.6/39.9
SMART (adapted)67.9/38.080.3/21.715.7/78.362.7/41.4
Seirênes (adapted)74.5/40.584.5/20.79.7/79.165.7/34.6
RLVR→BCT72.5/8.327.9/2.30.0/3.857.8/5.7
RLVR→anti-syc SFT79.9/7.432.7/2.46.0/17.765.1/5.7
UNHACK86.6/3.588.1/1.823.0/19.880.6/1.8

A “wrong plan” contains just one or two corrupted steps, as in Figure 1. UNHACK benefits from the remaining good steps: clean success exceeds no-plan success on all four models. Coding with mixed tests examines the same ability to use good advice while ignoring mistakes.

Low following alone is not enough, as the SFT rows show. RLVR→SFT helps some models but does not reliably preserve capability, and adds target generation and SFT after RLVR. UNHACK learns the task and discourages copying together in one RL stage.

The main paper shows Pareto plots and discusses these results. The appendix gives full accuracies, copying/following rates, SDs, and implementation details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.