- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose Lookahead, a method for predicting RLHF training failures before training occurs.
- Setup: a policy we would like to train, an RM, and an eval set (e.g. IFEval). The goal is to check whether training the policy against the RM using RLHF on some training set would improve or degrade the evaluation metric on a specific test point.
- Method: sample rollouts from the base policy on the eval problem. For every sampled token position i, run a router on . If the router "fires", sample continuations from a set of candidate tokens at position , taken to be the top-k tokens. Score the resulting rollouts with the RM, and use this score to adjust (shift by ) the token logit distribution. Then sample the true ith token from the adjusted distribution, and continue. To decide on a final prediction, the authors sample many Lookahead completions, score them using the evaluation metric, and then take the sign of the mean minus the base model’s mean (without Lookahead) as the predicted direction. If this is positive, the prediction is that RLHF against this RM should improve performance on this evaluation point. The most competitive router looks at the base model’s top-2 candidate next tokens, extends each greedily by some number of tokens, scores both with the RM, and fires if the second candidate wins by more than some predefined margin.
- Results: the authors show that Lookahead enables more accurate prediction of downstream RLHF success than a BoN baseline. In this baseline, N rollouts are sampled from the policy, scored using the RM, and the highest-scoring rollout is selected and scored with the evaluation metric. This is repeated (100 independent BoN-16 draws from a pool of 2,000 base rollouts per prompt) and averaged, and the sign of the difference from the base model’s mean gives BoN’s predicted direction. Across four (base model, RM) setups, Lookahead predicts which IFEval prompts degrade after training better than BoN (F1 on degradations higher by 0.11–0.26), and matches or exceeds it on improvements in three of four setups.
- The authors motivate their method as an instantiation of the closed-form solution to KL-regularized RLHF, although this connection is asserted by citations rather than derived or shown anywhere.
Critical strengths
- The problem they set out to solve is an interesting and useful one: predicting the effect that RLHF against an RM would have given a policy on a particular evaluation problem. If their approach were to work, we could then iterate on RM design and improve its suitability for RLHF for particular tasks without having to incur substantial training costs.
- The method appears to be effective, improving over the BoN baseline, although it is unclear to whether whether the level of success demonstrated is enough to obviate training runs during RM design (I also do not know what such a level might be, practically speaking).
Critical weaknesses
Cost
The cost of using this method is significant, primarily from generating top-k completion rollout, but also from router scoring. The authors acknowledge this and provide an estimate of RM calls and rollouts, vs. for BoN. The main motivation for the method however is as a proxy for assessing how RLHF might impact performance on the evaluation problem. I would appreciate some discussion on this. I put some numbers through Claude (please point out any mistakes) and it returned this back-of-the-envelope estimation:
Back-of-envelope: Lookahead prediction cost vs. PPO training (Tülu 8B setup)
Conventions. FLOPs counted in units of P (parameter count; policy, RM and critic are all ~8B here). Forward pass ≈ 2P per token; forward+backward ≈ 6P per token (as in standard compute accounting). Attention FLOPs and wall-clock effects (e.g., memory-bound decoding) are ignored.
Inputs from the paper
- Completion length T ≈ 230 tokens; router fire rate f = 4.2%; K = 5 candidates; probe length s = 4 (§2.2, §3)
- 100 LA completions per prompt; 541 IFEval prompts (§3)
- PPO: 952 steps × 64 prompts × 1 sample = 60,928 rollouts; max 1,024 generated tokens (Table 5)
Assumptions (mine)
- LA rollouts start at the routed position, so average length ≈ T/2 ≈ 115 tokens
- IFEval prompt length ≈ 85 tokens (so an average router call scores ≈ 200 tokens uncached)
- PPO response length 200–600 tokens; one PPO epoch per batch; prompt tokens ignored for PPO (underestimates PPO somewhat)
Lookahead, per completion
| Component | Generated tokens | RM calls |
|---|---|---|
| Main trajectory | 230 | — |
| Router probes: 2 candidates × 4 tokens × 230 positions | 1,840 | 460 |
| LA rollouts: f·T·K ≈ 48 rollouts × ~115 tokens | ~5,550 | 48 |
| Total | ~7.6K | ~510 |
RM tokens scored per completion:
- Uncached: 460 router calls × ~200 tokens + 48 rollouts × ~315 tokens ≈ 107K tokens
- With prefix caching: ≈ 8–17K new tokens (only probe/rollout suffixes are new)
Lookahead, all of IFEval (541 prompts × 100 completions = 54,100 completions)
- Generation: 54,100 × 7.6K ≈ 410M tokens → ≈ 0.8B·P
- RM scoring, uncached: 54,100 × 107K ≈ 5.8B tokens → ≈ 11.6B·P
- RM scoring, cached: ≈ 0.4–0.9B tokens → ≈ 0.9–1.9B·P
- Total: ≈ 1.7–2.7B·P (cached) to ≈ 12.4B·P (uncached)
PPO training (one full run)
- Generated tokens: 60,928 × 200–600 ≈ 12–37M tokens
- Per generated token: actor generation (2P) + actor fwd/bwd (6P) + critic fwd/bwd (6P)
- reference forward (2P) + RM forward (2P) ≈ 18P
- Total: ≈ 0.2–0.7B·P (≈ 0.3–1.0B·P allowing for prompt tokens)
Ratio
| LA / PPO | |
|---|---|
| With RM prefix caching | ≈ 2–13× |
| Without caching | ≈ 12–60× |
For reference: the BoN baseline as run in the paper
- 2,000 base rollouts per prompt × 541 prompts ≈ 1.08M rollouts × ~230 tokens ≈ 250M tokens → ≈ 0.5B·P
- RM scoring ≈ 1.08M × ~315 tokens ≈ 340M tokens → ≈ 0.7B·P
- Total ≈ 1.2B·P, i.e., also on the order of a full PPO run
Caveats
- Order-of-magnitude only; depends on response lengths, caching, and PPO epochs.
- Per-sample cost could be reduced (fewer than 100 completions per prompt; cheaper router), but the paper reports no sample-efficiency analysis.
- Structural point: LA cost scales with every benchmark and sample predicted, whereas training is paid once and any number of benchmarks can then be evaluated cheaply.
- Not included: the generative-RM setting, where each "RM call" is a gpt-oss-120b generation (≈ 510 judge generations per LA completion), which would make the ratio far larger.
Lookahead, per completion
| Component | Generated tokens | RM calls |
|---|---|---|
| Main trajectory | 230 | — |
| Router probes: 2 candidates × 4 tokens × 230 positions | 1,840 | 460 |
| LA rollouts: f·T·K ≈ 48 rollouts × ~115 tokens | ~5,550 | 48 |
| Total | ~7.6K | ~510 |
RM tokens scored per completion:
- Uncached: 460 router calls × ~200 tokens + 48 rollouts × ~315 tokens ≈ 107K tokens
- With prefix caching: ≈ 8–17K new tokens (only probe/rollout suffixes are new)
Lookahead, all of IFEval (541 prompts × 100 completions = 54,100 completions)
- Generation: 54,100 × 7.6K ≈ 410M tokens → ≈ 0.8B·P
- RM scoring, uncached: 54,100 × 107K ≈ 5.8B
My takeaway is that training incurs a one off cost, while Lookahead scales with evaluation problems that we care about, but at the evaluation-dataset scale Lookahead might actually be significantly more expensive than running RL! This makes the practical application of this method potentially quite limited. Also the Gen-RM setting gets even more expensive...
Theoretical motivation of method and of BoN baseline
The authors allude to some theoretical justification for their approach via citations (e.g. "closed-form solution to the RLHF problem, Korbak et al. 2022"), but there is no attempt to show this formally or informally. Why should we expect Lookahead to provide good estimates of evaluation metric performance after RLHF against the RM? How should we more formally understand the router design? Likewise, it is unclear why BoN is a good baseline here either. How does the RM's choice of best rollout via BoN relate to the optimal solution learned via RLHF? Perhaps this is discussed in Beirami et al., as cited in the paper Appendix: regardless, it should be discussed explicitly as it is important to understanding why we expect BoN to be a reasonable baseline in the first place. BoN is often used to evaluate RMs (use RM to select the predicted best of N, then compare with ground truth best of N), but it is unclear why this predicts downstream RL performance. Another potential issue I see is that RL itself is quite noisy and very dependent on many things (choice of algo, hyperparameters, training data etc.), so it is unclear whether these predictions would hold more broadly in practice.
Missing baselines
There are many missing baselines that, if included, could improve the paper:
- Does cheap RL (e.g. RL on fewer steps, or a smaller dataset) also predict evaluation metric performance on a held out test sample? Is that a reasonable way of getting cheaper proxies?
- Varying the number of Lookahead rollouts (hardcoded to 100 in the paper, or 25 for Gen-RM). This might help with cost reduction.
- Varying the probe length (hardcoded to 4).
- Varying for
Questions for authors
To improve my score, I would need:
- Better discussion on cost, and a clear description of when and why this method could be helpful vs. just doing RLHF.
- Some arguments justifying the design of this method beyond citing a few papers. The paper doesn't help me build intuition for why this token reweighting method and selection using the router is a good idea (I have an inkling based on the citations and the limited discussion about optimal policies and the tilted distribution, but the paper really should make this explicit).
- The baselines listed above, at a bare minimum.
While I find this idea quite interesting, as of now, I don't think this approach would be useful practically, it leaves many design choices unjustified, and I am not sure why it should (in theory) work.
Thank you very much for your feedback! Our answers: 1. This is a great observation. Filtering on significant changes was put in place to drop prompts where changes were within sampling noise, but we agree with you that it makes the task easier. Table 7 attempts to address this limitation, but we believe your proposal to be better suited.
We reran the eval on all 541 prompts, setting the positive class as "degrades", and the negative class "improves" or "does not change". Here are the results: F1−, LA vs BoN:
As you can see, results stay directionally the same, but are less strong.
On false positives (prompts predicted to degrade which stay unchanged): it varies across models. Tülu 8B only shows 5% false positives across unchanged prompts, while Tülu 13B shows 32%. Olmo models both sit at around 10%. An important note on false positives: These are not always completely wrong predictions, but often early signs of degradation that could appear under higher optimisation pressure. To back up this claim, we have retrained Tülu 8B in two different optimisation pressure regimes, governed by β (β=0.5 and β=0.05). I find that 50% of false positive that LA flags in the low optimisation pressure regime become degraded in the high optimisation pressure regime. For prompts that LA does not flag, this number is only 20%.
I am committing to adding the above analysis in a new appendix entry under Table 7.
Please refer to the shared Cost answer!
I agree with your point that the highly sparse token-level KL divergence result does not straight-forwardly inform RMPROBE. At the same time I believe that Table 2 does a good job of showing a clear advantage of RMPROBE against randomly picking token positions at the same rate. This makes it hard to argue against the fact that RMPROBE is exploiting some useful signal, leading to better token position selection. Regarding partial vs full rollouts, as you correctly stated the intuition is that a binary signal (fire vs skip) needs to be less precise than updating the logits. I am committing to making this logic clearer in the paper's explanation of RMPROBE. I also do agree that in the generative RM experiments routing quality may be a performance bottleneck.