acceptodds
Withdrawn from ICLR 2027

Harness-Agnostic Detection and Immunization of Reward Hacking in Self-Evolving Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-evolving language models improve by proposing candidate updates and keeping whatever raises a visible score. When that score is an imperfect proxy for the capability one actually wants, sustained selection widens the gap between the two. This is reward hacking. We introduce HackProbe, a monitor that attaches to an arbitrary self-evolving loop through two black-box hooks, with no access to weights or activations. It keeps a secret, distribution-fixed comparison core, whose frozen distribution makes its capability proxy comparable across generations, alongside a rotated fresh layer that hardens the bank against co-adaptation. Four tests built on that proxy cover the level gap, a scale-aligned divergence with online change-point detection, capability stagnation, and a conditional confidently-wrong rate; a Šidák correction turns them into a calibrated family-wise -value. Diagnosis alone recovers nothing, so a risk-aware immunization layer reselects an honest candidate from the proposal pool using the core together with a purely structural gaming footprint, disclosing at most bits per generation to the host. We prove a detectability bound that converts a target error rate into an explicit probe-size budget, and we delimit what probe rotation does and does not buy. On a controlled prompt-level host with four injected hacking channels and ground-truth labels, HackProbe reaches AUROC against for the strongest baseline and cuts the false-positive rate from to . Its bandwidth-limited reselection is the only immunization level that returns more true capability under hacking, points on average, than it forfeits on clean runs, ; per-channel effects are mostly not individually significant.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reward hacking, self-evolving language models, black-box monitoring, model evaluation, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nmt4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is about detecting reward hacking in self-evolving language models, what the authors propose or solve specifically remains unclear, see Weaknesses blow.

Critical strengths

None

Critical weaknesses

AI-generated paper. The paper is very difficult to follow and clearly AI-generated. The authors claim that AI was only used for language polishing. The authors are not honest about the AI use in their statement, the entire paper is full of Claude-ish words and phrases and does not follow scientific standards, clearly AI was used for more than simple language polishing.

Missing clear problem statement, contributions and take-aways. Due to the heavy use of AI-generated text, it remains entirely unclear what this paper is contributing, proposing or solving exactly. Figure 1 does not help identifying a core contribution or which problem is solved exactly either.

Unclear notation. Most notation is used before it's properly introduced. This is very difficult to follow for the reader, the notation overview in the Appendix is not sufficient to carefully follow the authors' text or arguments.

Missing related work. This paper is missing a proper positioning in the broader literature. The main text is further not self-sufficient, basically all references to related work are missing. This also holds in particular for core concepts used by the authors (to make concrete examples: "Page-Hinkley alarm", "Goodhart gap", "Sidak fusion"). The related work section in the provided Appendix is not enough.

Missing description of the experimental setup. The description of the experimental setup is not sufficient and not justified. Why do the authors select the specific datasets and hyperparameters? Most hyperparameters are not even sufficiently described or motivated (to make examples: what is T, n, S, q, L, τ\tau, ...?) . It remains unclear why the experiments are sensible for supporting the authors' claims.

Modified conference format. The authors modified the conference style files to remove page numbers and the conference header, the objective of this modification remains unclear.

Overall, the paper's quality is not sufficient for this venue, recommending clear rejection.

Questions for authors

No further questions.

Reviewer_KQV1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes HackProbe, a black-box monitor for reward hacking in self-evolving LM loops. It attaches through two hooks: it observes the host's visible score MM, and it queries the current candidate on held-out probes. A secret comparison core with a frozen distribution gives a capability proxy C^\hat C. A rotated "fresh layer" is used only for hardening. Four statistics are fused with a Šidák correction: a level gap (J0), the scale-aligned divergence Dt=ΔMt−λΔC^tD_t = \Delta M_t - \lambda \Delta \hat C_t with Page–Hinkley change-point detection (J1), capability stagnation (J2), and a conditional confidently-wrong rate (J3). An immunization layer reselects from the proposal pool at four levels (F0–F3). F2 limits what the host learns to log⁡2Π\log_2 \Pi bits per generation. The theory gives a sub-Gaussian detectability bound, which turns into a probe budget n≳8(λ2vC+vM)log⁡(1/α)/δ2n \gtrsim 8(\lambda^2 v_C + v_M)\log(1/\alpha)/\delta^2, and a memorization bound for rotation.

Critical strengths
  1. The problem framing is useful. The paper monitors reward hacking from black-box scores and outputs only. It uses a frozen-distribution core so that ΔC^\Delta \hat C is comparable across generations, and it accounts explicitly for how much information the reselection channel leaks back to the host (log⁡2Π\log_2 \Pi bits per generation). The leave-one-hacking-type-out evaluation (Table 4) and the leave-one-statistic-out ablation (Table 6) are the right tests for a fused detector. The paper is also clear about what its guarantees do not cover (rotation only bounds memorization; error control is per generation, not per run), which is bit weird in itself.
Critical weaknesses

First of all the paper doesnt use the ICLR template, which could make this paper desk rejected.

Writing: The entire paper seems to be AI written, the authors does say that they use AI to polish. But polishing means, improving the current condition of the writing but honestly its a mess and hodgepodge. The data and prose are so dense that it makes it difficult to understand whats going on, and the entire paper seems to be written by some Auto Research Loop. The introduction doesnt even seems like an introduction, the authors added zero effort in building the base of what the paper is about.

Experiments;

  1. The evaluation setup cannot support the main claims. There is one host, at prompt level, with a frozen proposer. The hacking channels are injected by the authors and labelled channel-on/channel-off. The title claims harness-agnosticism, but Section 4.4 admits this one host is the whole of the evidence. More importantly, every candidate pool contains "one enforced honest candidate produced under a clean context" whose true score is higher (Sections 3.3 and 4). The reselection gain in Sections 4.2–4.3 is therefore built into the setup. A real self-evolving loop gives no such candidate, and the paper does not measure what happens without it.

  2. Low statistical power and weak baselines. The final measurement uses ∣Aeval∣=64|A_{eval}| = 64 items and 6 seeds per channel, so interval endpoints move in steps of 1/64. Only 2 of the 12 gain intervals in Table 2 exclude zero. The headline claim that "only F2 returns more than it costs" rests on 0.052 vs 0.047 (ratio 1.11), with no interval on the ratio. For detection, 3 of the 5 baselines flag every generation, so their precision, recall and F1 carry no information. The only non-trivial competitor (the absolute-C^\hat C trajectory) is an ablation of the method itself. There is no comparison with an existing overoptimization or contamination detector, or with a calibrated threshold on a simple held-out gap. At the chosen operating point the FPR is 0.434. The rotation results in Table 7 have no seeds or intervals. Proposition 1 is a standard sub-Gaussian tail bound with no lower bound, so the theory does not compensate for the thin evidence.

Questions for authors
  1. How large is the reselection gain when the pool contains no enforced honest candidate, i.e. when all Π\Pi candidates come from the host's own proposer?
  2. Can you report an interval for the F2 gain/cost ratio (bootstrap over seeds and audit items jointly) and per-seed results? With 6 seeds and 64 audit items, is the ratio distinguishable from 1?
  3. Have you run HackProbe on any second host, e.g. GEPA or a weight-level loop such as R-Zero? Without one, how is "harness-agnostic" justified?
  4. Why is there no comparison with existing detectors, e.g. a held-out gold-score gap in the style of Gao et al. (2023), or a fixed held-out set with a calibrated threshold? How was the "fixed held-out set" baseline (AUROC 0.587) thresholded?
  5. How many seeds are behind Table 7, and what are the intervals?
Reviewer_ziyFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The work introduces HackProbe, a framework designed for detecting and mitigating reward-hacking in self-evolving language models without accessing model weights or internal activations.

Critical strengths

The four detection statistics idea seems interesting, and the ablation study supports the authors' argument that different statistics contribute to detecting different forms of reward hacking, and the AUROC improvement of 0.1 is clear evidence that the proposed method of monitoring signals is useful.

Critical weaknesses

The biggest drawback of the method is the false-positive rate of more than 0.4, which makes implementing the harness highly impractical.

The experimental setup explicitly ensures that every candidate pool contains one clean, honest candidate, generated under a clean context and does not determine how performance changes when the clean candidate is absent.

The reported benefit of the F2 immunization strategy is relatively small compared to its cost on clean runs (0.052 vs. 0.047). It is unclear whether this difference is statistically significant or a random fluctuation, especially given the limited number of runs and the relatively small final evaluation set.

Overall, for the claimed improved quality over arbitrary self-evolving loops, the experiment settings remain extremely limited, with only a prompt-evolution system and a restricted evaluation set.

Questions for authors
  1. How does HackProbe perform on other types of self-evolving systems?

  2. How does the immunization performance change if the candidate pool does not contain a guaranteed honest candidate?

  3. Could you please report detection performance at fixed false-positive rates, i.e. 0.05 or 0.1?

  4. Is the benefit from F2 immunization strategy statistically significant?

  5. How does the performance of HackProbe change over substantially longer evolution trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.