- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces PFN-OPE, a prior-data fitted network that amortizes off-policy evaluation across many LLM evaluation tasks. Rather than fitting a new reward model for every behavior policy, target model, or reward definition, the method is pretrained on synthetic OPE tasks constructed from a pool of stored LLM responses scored by multiple reward functions. At inference time it receives a logged dataset together with one sampled target response per prompt and predicts the target policy value without additional fitting. The task prior generates both reward shifts and policy shifts. Experiments on HelpSteer2 and UltraFeedback report substantially lower estimation error than direct-method, in-context, and importance-sampling baselines, including cases in which the target LLM and reward model were absent during pretraining.
Critical strengths
Amortizing the OPE problem itself, rather than merely amortizing a reward regressor, is a novel and practically motivated idea. The method directly targets repeated deployment settings where both candidate models and evaluation criteria change, which is precisely where repeatedly fitting a separate direct-method model becomes cumbersome.
The empirical results are broad and unusually consistent. PFN-OPE is reported to outperform the strongest baseline by roughly two- to nine-fold across the main reward experiments, and it remains best in almost all behavior/target pairs when both the target model and reward model are unseen during pretraining. The policy-prior ablations further support the proposed mechanism rather than only reporting aggregate wins.
Critical weaknesses
Performance is inherently tied to the task prior, and the experiments do not yet establish robustness to substantial prior misspecification. The paper itself notes that a PFN is only as good as its prior. The proposed prior is deliberately engineered using combinations of existing reward functions and reweighted/routed policies, so strong performance on related deployment shifts does not imply reliability for qualitatively different target models, domains, or evaluation criteria.
The evaluation setting remains more controlled than the motivating deployment setting. Training labels are available because policies and rewards are defined over a finite response pool, and the same general construction is used to make evaluation ground truth available. This is a sensible experimental strategy, but it leaves open whether the gains persist on naturally collected production logs, fresh unconstrained target generations, and human-defined reward shifts that are not representable by the pretraining reward family.
Questions for authors
How does PFN-OPE behave when the test task lies well outside the support of the pretraining prior? Can the model expose a reliable indicator that its value estimate is out of distribution rather than simply returning an overconfident correction?
Have you evaluated a genuinely cross-domain setting, for example, pretraining the PFN on one corpus and evaluating on a different prompt distribution, while also holding out the target model and reward model?
Can the method be validated on real logged human feedback, with fresh target-model generations and human evaluation of a subset of those generations to provide an independent target-value estimate?
2 comments
Rebuttal by Authors (2/2)
Q1: Tasks outside the prior, and an out-of-distribution indicator
How PFN-OPE behaves when the test task lies "well outside the support of the pretraining prior", and whether it can "expose a reliable indicator that its value estimate is out of distribution".
Outside the prior, PFN-OPE's advantage shrinks and it can fail. Under GRM, a reward model outside the prior's reward family, its gain over the baselines falls to 2.0 to 4.1 times but remains (Figure 1). Where the prior has no tasks at all, a PFN has its highest error (Figure 4). Details are under W1.
PFN-OPE outputs 99 quantiles of the correction, so each estimate comes with an interval. We can use the interval width as a risk indicator.
We tested the intervals in three steps:
1. The raw intervals are overconfident. On tasks drawn from the prior at n = 512, the nominal 90% interval covers the truth 67.6% of the time.
2. Post-hoc calibration repairs this. We widen the intervals with a procedure fitted on 512 calibration tasks and evaluate on 512 separate tasks.
Coverage of PFN-OPE's intervals before and after calibration.
| Nominal coverage | Raw | Calibrated |
|---|---|---|
| 80% | 55.9% | 79.9% |
| 85% | 61.9% | 84.2% |
| 90% | 67.6% | 88.5% |
| 95% | 77.0% | 93.6% |
Each row gives how often the interval at that nominal level contains the true value, before and after calibration, on 512 held-out tasks drawn from the prior. The raw intervals cover the truth too rarely, 67.6% of the time at a nominal 90%.
3. Interval width ranks risk in one shifted setting. We rank the tasks of a population by calibrated interval width, with no access to target rewards, and discard the widest.
Change in the MSE of the estimates that remain after discarding those with the widest intervals.
| Discarded | 5% | 10% | 15% | 20% | 25% |
|---|---|---|---|---|---|
| In-prior tasks | −6.6% | −8.8% | −16.1% | −21.7% | −23.7% |
| Unseen targets under GRM | −5.0% | −7.8% | −8.9% | −8.7% | −6.5% |
On in-prior tasks the retained MSE decreases at every rate, by 6.6% to 23.7%. On unseen targets under GRM it also decreases at every rate, by 5.0% to 8.9%. Interval width is thus a potential risk indicator.
Q2: Cross-domain evaluation
Whether we evaluated "a genuinely cross-domain setting", pretraining on one corpus and evaluating on a different prompt distribution, "while also holding out the target model and reward model".
Yes, we ran it. With corpus, target LLMs, and reward model all held out, PFN-OPE's MSE is 91.9% below that of the best baseline ().
We pretrained a PFN on UltraFeedback alone. We then tested it on HelpSteer2. The target LLMs (Llama-3.2-3B and Gemma-3-1B) and the reward model (GRM) are also absent from its pre-training. Therefore, the corpus, target model, and reward are held out together. Rewards from HelpSteer2 enter only as context at test time.
Cross-corpus transfer, with the target LLMs and the reward model also held out. Value MSE over 18 estimates at n = 512. Lower in bold.
| Pretraining corpus | Test corpus | PFN-OPE MSE | Best baseline MSE | Reduction |
|---|---|---|---|---|
| UltraFeedback | HelpSteer2 | 0.0095 | 0.1175 (DM-TabPFN) | 91.9% |
We would also like to emphasize that the existing experiment already holds out everything but the corpus. No evaluation prompt appears in pretraining. Section 4.3 uses target LLMs absent from pretraining, one of them from an unseen model family, together with a reward model absent from pretraining, and PFN-OPE has the lowest MSE in 11 of 12 behavior and target pairs.
Q3: Validation on logged human feedback
Whether the method can be validated on "real logged human feedback, with fresh target-model generations and human evaluation of a subset".
We could not run this within the response period: it needs logged human feedback, fresh target generations, and new human ratings of those generations. The post-trained checkpoints and the held-out reward model address parts of the deployment question, but neither replaces validation against human feedback. We will state this as a limitation in the main text.
Rebuttal by Authors (1/2)
Thank you for the support and careful review. We have shortened your comments for reference and responded to each below. Tables are numbered across all our responses.
W1: Dependence on the task prior
We agree, the prior is PFN-OPE's inductive bias, and we do not claim reliability outside its support. The submission measures this dependence in three places, and two new experiments test targets further from the prior.
In the submission:
New: a held-out corpus. A PFN pretrained on UltraFeedback alone and tested on HelpSteer2, with the target LLMs and the reward model held out as well, has an MSE of 0.0095 against 0.1175 for the best baseline (details under Q2).
New: real DPO checkpoints, which the prior only approximates with reweighted policies. PFN-OPE has the lowest error at epochs 2 and 3, and a direct method is more accurate at epoch 1. We post-trained Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct with DPO for three epochs each and kept a checkpoint after every epoch. The preference pairs are built from cached responses on HelpSteer2 training prompts, scored by ArmoRM and GRM. Each checkpoint is a target policy and its base model is the behavior policy. We report the results with 100 logged draws for every checkpoint and estimator.
Table R1: DPO checkpoints as target policies. Mean correction: for each logged draw, the gap between the target value and the logged mean reward. Lower MSE in bold.
The degree of behavior-test policy shift increases for the later training epoch checkpoint. At epoch 1, where it is smallest, a direct method is more accurate on both. At epochs 2 and 3 PFN-OPE has the lowest MSE on both, by 8.9% and 2.0% for Qwen and by 43.6% and 52.4% for Llama. We will add these results in a new subsection, and state the dependence on the prior as a limitation in the main text.
W2: Controlled evaluation and deployment
The DPO experiment under W1 moves the evaluation a step toward deployment. Its targets are real post-trained checkpoints, and each generates fresh responses that were never part of the response pool. PFN-OPE has the lowest MSE there at the later epochs (Table R1). These are still new responses to existing evaluation prompts, scored by automated reward models. We have no production logs or human-defined rewards, and we will state this in the main text. We would also like to clarify the existing evaluation: the reweighted and routed policies appear only in pretraining, and every evaluation policy is a single LLM's own sampled responses on held-out prompts, eight per prompt at temperature 0.7 and top-p 0.9.