acceptodds
Under review as a conference paper at ICLR 2027

Catching RLHF Failures before Training

Abstract

RLHF plays a key role in post-training, but the most reliable way to debug an RLHF setup, including its reward model (RM), is still running the training end to end: static RM benchmarks and Best-of-N (BoN) sampling correlate poorly with downstream outcomes. We introduce Lookahead, a sampling strategy that anticipates training failures before they occur, using only the base model and the RM. At selected decoding positions, each candidate token's logits are shifted by the reward that a rollout continuing from it obtains, with a coefficient controlling the optimisation pressure applied. At high , the RM's preferences are amplified until its flaws become visible. Across four (base model, RM) pairs spanning two model families and two scales, Lookahead outperforms BoN at predicting which IFEval prompts degrade after training, by a margin of 0.11–0.26. Increasing the BoN sample budget up to does not close this gap. At the instruction-family level, our method flags 20 of the 24 degradation cases, while BoN wrongly predicts improvement in 17. Our results show that much of what RLHF will break is already determined by the (base model, RM) pair, and can be read out before training begins.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward models, RLHF, reward model evaluation, failure prediction, post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yHQFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Lookahead, a method for predicting RLHF training failures before training occurs.

  • Setup: a policy we would like to train, an RM, and an eval set (e.g. IFEval). The goal is to check whether training the policy against the RM using RLHF on some training set would improve or degrade the evaluation metric on a specific test point.
  • Method: sample rollouts from the base policy on the eval problem. For every sampled token position i, run a router on (x,y<i)(x, y_{<i}). If the router "fires", sample continuations from a set of candidate tokens at position ii, taken to be the top-k tokens. Score the resulting rollouts with the RM, and use this score to adjust (shift by α(rk−r^)\alpha(r_k − \hat{r})) the token logit distribution. Then sample the true ith token from the adjusted distribution, and continue. To decide on a final prediction, the authors sample many Lookahead completions, score them using the evaluation metric, and then take the sign of the mean minus the base model’s mean (without Lookahead) as the predicted direction. If this is positive, the prediction is that RLHF against this RM should improve performance on this evaluation point. The most competitive router looks at the base model’s top-2 candidate next tokens, extends each greedily by some number of tokens, scores both with the RM, and fires if the second candidate wins by more than some predefined margin.
  • Results: the authors show that Lookahead enables more accurate prediction of downstream RLHF success than a BoN baseline. In this baseline, N rollouts are sampled from the policy, scored using the RM, and the highest-scoring rollout is selected and scored with the evaluation metric. This is repeated (100 independent BoN-16 draws from a pool of 2,000 base rollouts per prompt) and averaged, and the sign of the difference from the base model’s mean gives BoN’s predicted direction. Across four (base model, RM) setups, Lookahead predicts which IFEval prompts degrade after training better than BoN (F1 on degradations higher by 0.11–0.26), and matches or exceeds it on improvements in three of four setups.
  • The authors motivate their method as an instantiation of the closed-form solution to KL-regularized RLHF, although this connection is asserted by citations rather than derived or shown anywhere.
Critical strengths
  • The problem they set out to solve is an interesting and useful one: predicting the effect that RLHF against an RM would have given a policy on a particular evaluation problem. If their approach were to work, we could then iterate on RM design and improve its suitability for RLHF for particular tasks without having to incur substantial training costs.
  • The method appears to be effective, improving over the BoN baseline, although it is unclear to whether whether the level of success demonstrated is enough to obviate training runs during RM design (I also do not know what such a level might be, practically speaking).
Critical weaknesses

Cost

The cost of using this method is significant, primarily from generating top-k completion rollout, but also from router scoring. The authors acknowledge this and provide an estimate of 2T+fTK2T + fTK RM calls and fTKfTK rollouts, vs. NN for BoN. The main motivation for the method however is as a proxy for assessing how RLHF might impact performance on the evaluation problem. I would appreciate some discussion on this. I put some numbers through Claude (please point out any mistakes) and it returned this back-of-the-envelope estimation:


Back-of-envelope: Lookahead prediction cost vs. PPO training (Tülu 8B setup)

Conventions. FLOPs counted in units of P (parameter count; policy, RM and critic are all ~8B here). Forward pass ≈ 2P per token; forward+backward ≈ 6P per token (as in standard compute accounting). Attention FLOPs and wall-clock effects (e.g., memory-bound decoding) are ignored.

Inputs from the paper

  • Completion length T ≈ 230 tokens; router fire rate f = 4.2%; K = 5 candidates; probe length s = 4 (§2.2, §3)
  • 100 LA completions per prompt; 541 IFEval prompts (§3)
  • PPO: 952 steps × 64 prompts × 1 sample = 60,928 rollouts; max 1,024 generated tokens (Table 5)

Assumptions (mine)

  • LA rollouts start at the routed position, so average length ≈ T/2 ≈ 115 tokens
  • IFEval prompt length ≈ 85 tokens (so an average router call scores ≈ 200 tokens uncached)
  • PPO response length 200–600 tokens; one PPO epoch per batch; prompt tokens ignored for PPO (underestimates PPO somewhat)

Lookahead, per completion

ComponentGenerated tokensRM calls
Main trajectory230—
Router probes: 2 candidates × 4 tokens × 230 positions1,840460
LA rollouts: f·T·K ≈ 48 rollouts × ~115 tokens~5,55048
Total~7.6K~510

RM tokens scored per completion:

  • Uncached: 460 router calls × ~200 tokens + 48 rollouts × ~315 tokens ≈ 107K tokens
  • With prefix caching: ≈ 8–17K new tokens (only probe/rollout suffixes are new)

Lookahead, all of IFEval (541 prompts × 100 completions = 54,100 completions)

  • Generation: 54,100 × 7.6K ≈ 410M tokens → ≈ 0.8B·P
  • RM scoring, uncached: 54,100 × 107K ≈ 5.8B tokens → ≈ 11.6B·P
  • RM scoring, cached: ≈ 0.4–0.9B tokens → ≈ 0.9–1.9B·P
  • Total: ≈ 1.7–2.7B·P (cached) to ≈ 12.4B·P (uncached)

PPO training (one full run)

  • Generated tokens: 60,928 × 200–600 ≈ 12–37M tokens
  • Per generated token: actor generation (2P) + actor fwd/bwd (6P) + critic fwd/bwd (6P)
    • reference forward (2P) + RM forward (2P) ≈ 18P
  • Total: ≈ 0.2–0.7B·P (≈ 0.3–1.0B·P allowing for prompt tokens)

Ratio

LA / PPO
With RM prefix caching≈ 2–13×
Without caching≈ 12–60×

For reference: the BoN baseline as run in the paper

  • 2,000 base rollouts per prompt × 541 prompts ≈ 1.08M rollouts × ~230 tokens ≈ 250M tokens → ≈ 0.5B·P
  • RM scoring ≈ 1.08M × ~315 tokens ≈ 340M tokens → ≈ 0.7B·P
  • Total ≈ 1.2B·P, i.e., also on the order of a full PPO run

Caveats

  • Order-of-magnitude only; depends on response lengths, caching, and PPO epochs.
  • Per-sample cost could be reduced (fewer than 100 completions per prompt; cheaper router), but the paper reports no sample-efficiency analysis.
  • Structural point: LA cost scales with every benchmark and sample predicted, whereas training is paid once and any number of benchmarks can then be evaluated cheaply.
  • Not included: the generative-RM setting, where each "RM call" is a gpt-oss-120b generation (≈ 510 judge generations per LA completion), which would make the ratio far larger.

Lookahead, per completion

ComponentGenerated tokensRM calls
Main trajectory230—
Router probes: 2 candidates × 4 tokens × 230 positions1,840460
LA rollouts: f·T·K ≈ 48 rollouts × ~115 tokens~5,55048
Total~7.6K~510

RM tokens scored per completion:

  • Uncached: 460 router calls × ~200 tokens + 48 rollouts × ~315 tokens ≈ 107K tokens
  • With prefix caching: ≈ 8–17K new tokens (only probe/rollout suffixes are new)

Lookahead, all of IFEval (541 prompts × 100 completions = 54,100 completions)

  • Generation: 54,100 × 7.6K ≈ 410M tokens → ≈ 0.8B·P
  • RM scoring, uncached: 54,100 × 107K ≈ 5.8B

My takeaway is that training incurs a one off cost, while Lookahead scales with evaluation problems that we care about, but at the evaluation-dataset scale Lookahead might actually be significantly more expensive than running RL! This makes the practical application of this method potentially quite limited. Also the Gen-RM setting gets even more expensive...

Theoretical motivation of method and of BoN baseline

The authors allude to some theoretical justification for their approach via citations (e.g. "closed-form solution to the RLHF problem, Korbak et al. 2022"), but there is no attempt to show this formally or informally. Why should we expect Lookahead to provide good estimates of evaluation metric performance after RLHF against the RM? How should we more formally understand the router design? Likewise, it is unclear why BoN is a good baseline here either. How does the RM's choice of best rollout via BoN relate to the optimal solution learned via RLHF? Perhaps this is discussed in Beirami et al., as cited in the paper Appendix: regardless, it should be discussed explicitly as it is important to understanding why we expect BoN to be a reasonable baseline in the first place. BoN is often used to evaluate RMs (use RM to select the predicted best of N, then compare with ground truth best of N), but it is unclear why this predicts downstream RL performance. Another potential issue I see is that RL itself is quite noisy and very dependent on many things (choice of algo, hyperparameters, training data etc.), so it is unclear whether these predictions would hold more broadly in practice.

Missing baselines

There are many missing baselines that, if included, could improve the paper:

  • Does cheap RL (e.g. RL on fewer steps, or a smaller dataset) also predict evaluation metric performance on a held out test sample? Is that a reasonable way of getting cheaper proxies?
  • Varying the number of Lookahead rollouts (hardcoded to 100 in the paper, or 25 for Gen-RM). This might help with cost reduction.
  • Varying the probe length (hardcoded to 4).
  • Varying for τ\tau
Questions for authors

To improve my score, I would need:

  • Better discussion on cost, and a clear description of when and why this method could be helpful vs. just doing RLHF.
  • Some arguments justifying the design of this method beyond citing a few papers. The paper doesn't help me build intuition for why this token reweighting method and selection using the router is a good idea (I have an inkling based on the citations and the limited discussion about optimal policies and the tilted distribution, but the paper really should make this explicit).
  • The baselines listed above, at a bare minimum.

While I find this idea quite interesting, as of now, I don't think this approach would be useful practically, it leaves many design choices unjustified, and I am not sure why it should (in theory) work.

Reviewer_zBcuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an interesting and practically significant problem, with the potential to help training teams avoid the high costs of trial and error. The paper proposes a lookahead method that amplifies the logit difference by directly invoking the reward model, thereby enabling us to gain some insight into the performance trend that would result from RL training—even without actually performing it. The experimental results demonstrate the effectiveness of the method to a certain extent.

Critical strengths
  1. The problem is clearly defined, the reasoning is logical, and it addresses a meaningful problem.
  2. Prediction before RL has economic value.
Critical weaknesses
  1. The experimental design of the paper itself is not flawed, but the evaluation approach differs significantly from the method I had anticipated when I read the paper’s motivation. If I understand correctly, the authors primarily evaluated the following: (1) For prompts where significant changes were confirmed after the fact, can Lookahead predict the direction of those changes? However, the actual practical need is: (2) Without knowing the training results, can Lookahead identify, from among all prompts, those that will experience severe degradation in the future? Clearly, evaluation method (1) carries a potential risk of “hacking” and requires the establishment of prior knowledge; from the perspective of evaluation metrics, it excludes a large number of samples that might generate false positives in the real world.
  2. The computational cost of Lookahead is not negligible, and its actual value has yet to be fully demonstrated. According to Section 3, under the Tülu 8B setup, generating a single Lookahead response requires approximately 50 full rollouts and 510 RM calls. To obtain stable prompt-level predictions, the authors sample 100 Lookahead responses for each prompt. Although these computational costs may be significantly lower than those of a full RL training run, they remain quite expensive compared to simple evaluation methods such as BoN. The paper should fully report actual wall-clock latency and GPU hours, and analyze the performance-cost trade-offs under different budget constraints.
  3. The authors claim that RLHF results in highly sparse token-level KL divergence and, based on this, propose performing lookahead only at a few key token positions. However, the fact that post-RL policy changes are sparse does not imply that RMPROBE can accurately pinpoint these positions prior to training. Notably, Section 2.1 argues that partial rollout rewards are unreliable and therefore adopts full rollouts; However, Section 2.2 then uses partial continuations consisting of only few tokens to determine whether to initiate routing. Although the authors believe that noisy partial rewards are sufficient to perform binary routing, this assumption lacks thorough validation. In the generative RM experiments, RMPROBE’s F1 improvement is low, whereas the oracle router achieves significantly higher values, suggesting that routing quality may be a performance bottleneck.
Questions for authors

please refer to "weaknesses"

1 comment

Thank you very much for your feedback! Our answers: 1. This is a great observation. Filtering on significant changes was put in place to drop prompts where changes were within sampling noise, but we agree with you that it makes the task easier. Table 7 attempts to address this limitation, but we believe your proposal to be better suited.

We reran the eval on all 541 prompts, setting the positive class as "degrades", and the negative class "improves" or "does not change". Here are the results: F1−, LA vs BoN:

  • Tülu 8B: 0.63 vs 0.41, paired difference [+0.16, +0.30]
  • Tülu 13B: 0.66 vs 0.56, paired difference [+0.05, +0.13]
  • OLMo-2 7B: 0.46 vs 0.42, paired difference [−0.03, +0.13]
  • OLMo-2 13B: 0.47 vs 0.39, paired difference [−0.02, +0.17]

As you can see, results stay directionally the same, but are less strong.

On false positives (prompts predicted to degrade which stay unchanged): it varies across models. Tülu 8B only shows 5% false positives across unchanged prompts, while Tülu 13B shows 32%. Olmo models both sit at around 10%. An important note on false positives: These are not always completely wrong predictions, but often early signs of degradation that could appear under higher optimisation pressure. To back up this claim, we have retrained Tülu 8B in two different optimisation pressure regimes, governed by β\beta (β=0.5\beta = 0.5 and β=0.05\beta = 0.05). I find that 50% of false positive that LA flags in the low optimisation pressure regime become degraded in the high optimisation pressure regime. For prompts that LA does not flag, this number is only 20%.

I am committing to adding the above analysis in a new appendix entry under Table 7.

Please refer to the shared Cost answer!

I agree with your point that the highly sparse token-level KL divergence result does not straight-forwardly inform RMPROBE. At the same time I believe that Table 2 does a good job of showing a clear advantage of RMPROBE against randomly picking token positions at the same rate. This makes it hard to argue against the fact that RMPROBE is exploiting some useful signal, leading to better token position selection. Regarding partial vs full rollouts, as you correctly stated the intuition is that a binary signal (fire vs skip) needs to be less precise than updating the logits. I am committing to making this logic clearer in the paper's explanation of RMPROBE. I also do agree that in the generative RM experiments routing quality may be a performance bottleneck.

Reviewer_b3woOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Lookahead, a reward-guided decoding method for predicting RLHF-induced regressions before training. It amplifies reward-model preferences at selected token positions using full-rollout rewards. Across four setups, it improves degradation prediction over Best-of-N (BoN), with F1− gains.

Critical strengths

Practical motivation: Predicting training-induced failures is valuable for evaluating reward models beyond static response ranking. Consistent results: Degradation-prediction gains hold across four setups, and increasing BoN to N=2,000 does not close the gap. Useful supporting analyses: Routing ablations, PPO replications, and instruction-family results strengthen the empirical evidence.

Critical weaknesses

Limited mechanistic evidence: The explanation that Lookahead reaches rare, high-reward failure modes remains a hypothesis. Detailed analyses of cases where BoN predicts the wrong direction would help. A diversified or prefix-stratified BoN baseline could distinguish improved candidate coverage from token-level steering. Unclear practical efficiency: Each diagnostic response requires substantial rollout and RM computation. End-to-end cost comparisons and performance with fewer samples per prompt are missing. Restricted generalization: Main results focus on IFEval and PPO. Broader claims about predicting RLHF outcomes require additional tasks and training settings. Deployment gap: The main evaluation selects prompts using realized post-training changes. Although threshold-sensitivity results are provided, false-positive rates on unchanged prompts remain insufficiently characterized.

Questions for authors

What specific response patterns explain the large sign reversals between BoN predictions and actual PPO outcomes? Does Lookahead retain its advantage over diversified BoN under matched computational cost? How reliable and economical is the method with fewer samples and a separately calibrated warning threshold?

1 comment

Thank you very much for your feedback! Our answers:

  1. this is a great suggestion. We did a small case study on the "wrap in double quotes" prompts, where BoN shows improvement, while LA catches degradation (ground truth is degradation). When looking at rollouts, BoN selects the ones from the base model that comply with the instruction, while both PPO and LA show rollouts that open with a double quote, but never close it. We are committing to adding this case study in a designed appendix section
  2. As a potential version of prefix-stratified BoN, we ran the first token LA (Section 3, last paragraph) on IFeval for Tulu 8B. F1− is 0.46 for BoN-16, 0.62 for first-token, 0.70 for rmprobe LA. First-token LA is worse than BoN on improvements (F1+ 0.54 vs 0.60) So a stratified version does better than vanilla BoN, but is still unable to close the gap with LA.
  3. Please refer to the shared cost comment!

A couple of other observations from your weaknesses:

  1. Agreed with restricted generalisation. At the same time, I would like to highlight the fact that the generative RM results used GRPO! This is not stated explicitly as we are following the training recipe of the RaR paper, but I commit to adding it to help with clarity.
  2. On deployment gap: please see point 1. of answer to reviewer zBcu for some interesting results on it!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion on Cost

A common weakness raised by all reviewers is cost of the method. Here is our answer:

We generally agree with reviewer yHQF's back of the envelope calculations. At the same time:

  1. A core contribution would be showing that a lot of signal can be extracted without training. That is why BoN was selected as a baseline, because it is a proposed solution to the same problem. I agree with all reviewers that this is not cleanly stated in the intro/discussion, and we commit to rephrasing these paragraphs accordingly.
  2. in this paper we use N=100 answers per prompt to be sure that our results are robust to noise. In practice evals are often run with smaller N. An example of performance in that case: at 10 rollouts per prompt family-level agreement is 26.0/36 and at 25 it is 26.7/36, against 28/36 at 100, while BoN-16 stays near 16/36.
  3. About 90% of RM calls come from the router. This part is the "cheapest" to reduce. An example would be to fit a linear probe using a small subset, and then using the probe signal instead of running RMPROBE at every token. This is out of the scope of the paper, but I think it is an interesting point of view to keep in mind when thinking about cost and the paper's contributions.
  4. LA is independent of training dataset size. This becomes increasingly beneficial as we move from smaller models like the Olmo/Tulu family to larger models
  5. I would image LA to shine in a situation where the training dataset is large, and we are interested to test performance on a small subset of prompts

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.