acceptodds
Under review as a conference paper at ICLR 2027

Amortized Off-Policy Evaluation for LLMs

Abstract

Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model. Teams therefore evaluate candidates offline, on data produced by already-deployed models. This is off-policy evaluation (OPE), and it faces two distribution shifts: as a model is updated in post-training, its responses diverge from the logged ones (policy shift), and the reward definition under which it is judged changes with business requirements (reward shift). Classical OPE methods are ill-suited to this continual-deployment setting because they are defined per task and require fitting from scratch on every new logged dataset or reward definition. To address this, we propose PFN-OPE, a prior-data fitted network that amortizes OPE across a distribution of contextual-bandit tasks. We pretrain it once on tasks constructed from a pool of LLM responses scored by several reward functions, in which both shifts occur. At test time it maps a logged dataset and one sampled target response per prompt to a value estimate in a single forward pass, with no per-task fitting. On HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies, PFN-OPE achieves 2.0 to 9.3 times lower error than the best baselines across all tested configurations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Prior-Fitted Network, Off-Policy Evaluation, Contextual Bandits, Amortized Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_K1C9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces PFN-OPE, a prior-data fitted network that amortizes off-policy evaluation across many LLM evaluation tasks. Rather than fitting a new reward model for every behavior policy, target model, or reward definition, the method is pretrained on synthetic OPE tasks constructed from a pool of stored LLM responses scored by multiple reward functions. At inference time it receives a logged dataset together with one sampled target response per prompt and predicts the target policy value without additional fitting. The task prior generates both reward shifts and policy shifts. Experiments on HelpSteer2 and UltraFeedback report substantially lower estimation error than direct-method, in-context, and importance-sampling baselines, including cases in which the target LLM and reward model were absent during pretraining.

Critical strengths

Amortizing the OPE problem itself, rather than merely amortizing a reward regressor, is a novel and practically motivated idea. The method directly targets repeated deployment settings where both candidate models and evaluation criteria change, which is precisely where repeatedly fitting a separate direct-method model becomes cumbersome.

The empirical results are broad and unusually consistent. PFN-OPE is reported to outperform the strongest baseline by roughly two- to nine-fold across the main reward experiments, and it remains best in almost all behavior/target pairs when both the target model and reward model are unseen during pretraining. The policy-prior ablations further support the proposed mechanism rather than only reporting aggregate wins.

Critical weaknesses

Performance is inherently tied to the task prior, and the experiments do not yet establish robustness to substantial prior misspecification. The paper itself notes that a PFN is only as good as its prior. The proposed prior is deliberately engineered using combinations of existing reward functions and reweighted/routed policies, so strong performance on related deployment shifts does not imply reliability for qualitatively different target models, domains, or evaluation criteria.

The evaluation setting remains more controlled than the motivating deployment setting. Training labels are available because policies and rewards are defined over a finite response pool, and the same general construction is used to make evaluation ground truth available. This is a sensible experimental strategy, but it leaves open whether the gains persist on naturally collected production logs, fresh unconstrained target generations, and human-defined reward shifts that are not representable by the pretraining reward family.

Questions for authors

How does PFN-OPE behave when the test task lies well outside the support of the pretraining prior? Can the model expose a reliable indicator that its value estimate is out of distribution rather than simply returning an overconfident correction?

Have you evaluated a genuinely cross-domain setting, for example, pretraining the PFN on one corpus and evaluating on a different prompt distribution, while also holding out the target model and reward model?

Can the method be validated on real logged human feedback, with fresh target-model generations and human evaluation of a subset of those generations to provide an independent target-value estimate?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for the support and careful review. We have shortened your comments for reference and responded to each below. Tables are numbered across all our responses.


W1: Dependence on the task prior

Performance is "inherently tied to the task prior", which is engineered from combinations of existing reward functions and reweighted or routed policies, so strong results on related shifts do "not imply reliability for qualitatively different target models, domains, or evaluation criteria".

We agree, the prior is PFN-OPE's inductive bias, and we do not claim reliability outside its support. The submission measures this dependence in three places, and two new experiments test targets further from the prior.

In the submission:

  • The gain shrinks with distance from the prior but does not vanish: 3.7 to 9.3 times for held-out rewards that the prior's combinations span, and 2.0 to 4.1 times under GRM, a reward model outside them (Figure 1).
  • Widening the prior helps: the full policy prior lowers MSE by 28% to 36% at every n (Figure 3).
  • Figure 4 shows the failure directly: the PFN pretrained without shifted policies has its highest error where its prior has no tasks.

New: a held-out corpus. A PFN pretrained on UltraFeedback alone and tested on HelpSteer2, with the target LLMs and the reward model held out as well, has an MSE of 0.0095 against 0.1175 for the best baseline (details under Q2).

New: real DPO checkpoints, which the prior only approximates with reweighted policies. PFN-OPE has the lowest error at epochs 2 and 3, and a direct method is more accurate at epoch 1. We post-trained Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct with DPO for three epochs each and kept a checkpoint after every epoch. The preference pairs are built from cached responses on HelpSteer2 training prompts, scored by ArmoRM and GRM. Each checkpoint is a target policy and its base model is the behavior policy. We report the results with 100 logged draws for every checkpoint and estimator.

Table R1:\color{blue}{\text{Table R1:}} DPO checkpoints as target policies. Mean correction: for each logged draw, the gap between the target value and the logged mean reward. Lower MSE in bold.

DPO runEpochMean correctionPFN-OPE MSEBest baseline MSE
Qwen10.02610.00850.0061 (DM-TabPFN)
Qwen20.03820.00570.0063 (DM-TabPFN)
Qwen30.07200.00680.0070 (DM-kernel)
Llama10.05300.01230.0046 (DM-kernel)
Llama20.10970.00480.0085 (DM-kernel)
Llama30.13060.00500.0106 (DM-kernel)

The degree of behavior-test policy shift increases for the later training epoch checkpoint. At epoch 1, where it is smallest, a direct method is more accurate on both. At epochs 2 and 3 PFN-OPE has the lowest MSE on both, by 8.9% and 2.0% for Qwen and by 43.6% and 52.4% for Llama. We will add these results in a new subsection, and state the dependence on the prior as a limitation in the main text.


W2: Controlled evaluation and deployment

Ground truth is available because policies and rewards are defined over a finite response pool. This "leaves open whether the gains persist on naturally collected production logs, fresh unconstrained target generations, and human-defined reward shifts".

The DPO experiment under W1 moves the evaluation a step toward deployment. Its targets are real post-trained checkpoints, and each generates fresh responses that were never part of the response pool. PFN-OPE has the lowest MSE there at the later epochs (Table R1\color{blue}{\text{Table R1}}). These are still new responses to existing evaluation prompts, scored by automated reward models. We have no production logs or human-defined rewards, and we will state this in the main text. We would also like to clarify the existing evaluation: the reweighted and routed policies appear only in pretraining, and every evaluation policy is a single LLM's own sampled responses on held-out prompts, eight per prompt at temperature 0.7 and top-p 0.9.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q1: Tasks outside the prior, and an out-of-distribution indicator

How PFN-OPE behaves when the test task lies "well outside the support of the pretraining prior", and whether it can "expose a reliable indicator that its value estimate is out of distribution".

Outside the prior, PFN-OPE's advantage shrinks and it can fail. Under GRM, a reward model outside the prior's reward family, its gain over the baselines falls to 2.0 to 4.1 times but remains (Figure 1). Where the prior has no tasks at all, a PFN has its highest error (Figure 4). Details are under W1.

PFN-OPE outputs 99 quantiles of the correction, so each estimate comes with an interval. We can use the interval width as a risk indicator.

We tested the intervals in three steps:

1. The raw intervals are overconfident. On tasks drawn from the prior at n = 512, the nominal 90% interval covers the truth 67.6% of the time.

2. Post-hoc calibration repairs this. We widen the intervals with a procedure fitted on 512 calibration tasks and evaluate on 512 separate tasks.

Table R11:\color{blue}{\text{Table R11:}} Coverage of PFN-OPE's intervals before and after calibration.

Nominal coverageRawCalibrated
80%55.9%79.9%
85%61.9%84.2%
90%67.6%88.5%
95%77.0%93.6%

Each row gives how often the interval at that nominal level contains the true value, before and after calibration, on 512 held-out tasks drawn from the prior. The raw intervals cover the truth too rarely, 67.6% of the time at a nominal 90%.

3. Interval width ranks risk in one shifted setting. We rank the tasks of a population by calibrated interval width, with no access to target rewards, and discard the widest.

Table R12:\color{blue}{\text{Table R12:}} Change in the MSE of the estimates that remain after discarding those with the widest intervals.

Discarded5%10%15%20%25%
In-prior tasks−6.6%−8.8%−16.1%−21.7%−23.7%
Unseen targets under GRM−5.0%−7.8%−8.9%−8.7%−6.5%

On in-prior tasks the retained MSE decreases at every rate, by 6.6% to 23.7%. On unseen targets under GRM it also decreases at every rate, by 5.0% to 8.9%. Interval width is thus a potential risk indicator.


Q2: Cross-domain evaluation

Whether we evaluated "a genuinely cross-domain setting", pretraining on one corpus and evaluating on a different prompt distribution, "while also holding out the target model and reward model".

Yes, we ran it. With corpus, target LLMs, and reward model all held out, PFN-OPE's MSE is 91.9% below that of the best baseline (Table R13\color{blue}{\text{Table R13}}).

We pretrained a PFN on UltraFeedback alone. We then tested it on HelpSteer2. The target LLMs (Llama-3.2-3B and Gemma-3-1B) and the reward model (GRM) are also absent from its pre-training. Therefore, the corpus, target model, and reward are held out together. Rewards from HelpSteer2 enter only as context at test time.

Table R13:\color{blue}{\text{Table R13:}} Cross-corpus transfer, with the target LLMs and the reward model also held out. Value MSE over 18 estimates at n = 512. Lower in bold.

Pretraining corpusTest corpusPFN-OPE MSEBest baseline MSEReduction
UltraFeedbackHelpSteer20.00950.1175 (DM-TabPFN)91.9%

We would also like to emphasize that the existing experiment already holds out everything but the corpus. No evaluation prompt appears in pretraining. Section 4.3 uses target LLMs absent from pretraining, one of them from an unseen model family, together with a reward model absent from pretraining, and PFN-OPE has the lowest MSE in 11 of 12 behavior and target pairs.


Q3: Validation on logged human feedback

Whether the method can be validated on "real logged human feedback, with fresh target-model generations and human evaluation of a subset".

We could not run this within the response period: it needs logged human feedback, fresh target generations, and new human ratings of those generations. The post-trained checkpoints and the held-out reward model address parts of the deployment question, but neither replaces validation against human feedback. We will state this as a limitation in the main text.

Reviewer_oPXQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Authors consider the problem of off-policy evaluation of LLMs, where we want to estimate the expected reward of an LLM given logged data from a different LLM (or a different checkpoint) and a reward function. Authors propose a modified prior-data fitted network (PFN), adapted to the off-policy evaluation task. More precisely, the model takes a set of contexts (prompts), actions (responses) for the two policies (LLMs: behavior and target), and rewards for the behavior policy; and predicts the correction value, i.e. the shift in the expected reward of the target policy compared to the average reward of the behavior policy. PFN-OPE is trained on a set of synthetic OPE datasets. Authors evaluate the method on a set of LLMs and two datasets, showing improved MSE compared to strong baselines like MIPS.

Critical strengths

Novelty: this reviewer is not aware of prior work applying in-context learning methods to the problem of OPE, though scalability becomes a concern (see weaknesses/questions).

Strong results and ablations: clear improvement in terms of MSE across datasets and policy pairs. Ablations demonstrating the impact of ground-truth correction, prior task coverage etc. on the method's performance.

Critical weaknesses

Little-to-no discussion of the limitations and computations cost of the method. In particular, I have concerns re. how the method would scale to large datasets (common in OPE problems -- logged datasets can contains millions and billions of rows in practice), given the squared complexity of the transformer wrt to the input size.

Related work discussion is sparse. In particular, it could be extended to in-context learning methods, which have a longer history (starting from https://arxiv.org/abs/1606.02185). There's other work like https://arxiv.org/abs/2605.10137, which also uses an in-context learner (PFN) for contextual bandits. The paper would be much stronger if positioned more clearly within existing literature.

Minor: no legend in Figure 2.

Questions for authors

Have authors conducted any experiments on large datasets? Do authors have any ideas for scaling the method to large datasets, e.g. by decomposing them hierarchically? (Running the method on batches from the dataset, and combining the results incrementally.)

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for the thoughtful review and the batching idea in Q1, which we have now implemented. We have shortened your comments for reference and respond to each below. Tables are numbered across all our responses.


W1 and Q1: Scaling to large logged datasets

Logged datasets can hold "millions and billions of rows in practice", and the transformer's cost is quadratic in the input size. Q1 asks whether we ran experiments on large datasets and suggests "running the method on batches from the dataset, and combining the results".

We implemented your suggestion. With chunked inference, cost grows linearly with the number of logged rows, and PFN-OPE's error at 4,096 rows matches its error at 512. Chunking buys scale but not accuracy: the error does not fall as rows are added.

Accuracy up to 4,096 rows. We split the logged rows into chunks of 512, run PFN-OPE on each chunk in one forward pass, and average the chunk estimates, so cost grows linearly with the number of rows and memory stays bounded. We tested all 4,096 logged responses available per behavior policy and corpus.

Table R9:\color{blue}{\text{Table R9:}} Value MSE with more logged rows: PFN-OPE by direct and by chunked inference, and every baseline fitted on 4,096 rows.

EstimatorLogged rowsSeen LLMs, ArmoRMNew targets, GRM
PFN-OPE, direct5120.01600.0453
PFN-OPE, direct1,0240.01590.0451
PFN-OPE, direct2,0480.01600.0452
PFN-OPE, 8 chunks of 5124,0960.01600.0455
DM-ridge4,0960.07270.0739
DM-kernel4,0960.07370.0768
DM-MLP4,0960.06870.0851
DM-TabPFN4,0960.07310.1437
DM-TabICLv24,0960.08040.1410
MIPS4,0960.13500.1540

Chunked inference at 4,096 rows matches direct inference at 512 rows to within about 1%. With 512 rows PFN-OPE has a lower MSE than every baseline fitted on 4,096. PFN-OPE's own error does not fall as rows are added, so chunking buys scale but not accuracy. The chunks share prompts, so their errors are correlated, and we do not claim that averaging reduces error at a 1B\frac{1}{B} rate (BB being the batch/chunk size).

Throughput at a million rows. To test scale itself we timed chunked inference on resampled rows, with 512-row chunks and 16 chunks per batch on one H100.

Table R10:\color{blue}{\text{Table R10:}} Throughput and memory of chunked inference on one H100.

Rows processedThroughputPeak GPU memory
100,35244,231 rows/s0.68 GiB
1,000,44842,546 rows/s0.68 GiB

A million rows take about 24 seconds, not counting checkpoint loading and embedding extraction. This shows that the computation scales. It is not an accuracy result: our evidence on accuracy ends at 4,096 rows.

Beyond chunking. Quadratic attention is a property of our current backbone, not of the PFN-OPE objective. One alternative is inducing-point attention, as in the Set Transformer (Lee et al., 2019): a fixed number s of learned tokens summarizes the whole logged dataset, and the target rows attend to that summary, at a cost of O((N+M)s+s2)\mathcal{O}((N + M)s + s^2) for NN logged rows and MM target rows. Another is a linear-time approximation of softmax attention, as in the Performer (Choromanski et al., 2021). Both keep the same amortized objective and avoid averaging separate chunk estimates. Both may also change accuracy and calibration, and both need retraining and evaluation.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W1: Computational cost

"little-to-no discussion" of the computational cost of the method.

We measured it according to your suggestion. Online, PFN-OPE's cost is about 12% of DM-MLP's, with a lower MSE than every baseline. The one-time cost is 2.52 A100-hours of pretraining plus at least 103 GPU-worker hours to build the response pool.

Online. The comparison is a matched workload at n = 512 (two rewards, two target policies, one H100). DM-ridge and DM-kernel cost about as much as DM-MLP, MIPS 2.5 times as much, DM-TabPFN 17 times and DM-TabICLv2 about 65 times. These are runtime ratios, and they exclude the costs all estimators share (Table R6\color{blue}{\text{Table R6}}). Within a chunk, attention is quadratic in the chunk size.

Table R6:\color{blue}{\text{Table R6:}} Online wall-clock cost relative to DM-MLP, and value MSE, at n = 512. Lowest in bold.

EstimatorSeen LLMs: costSeen LLMs: MSENew targets: costNew targets: MSE
PFN-OPE0.120.02580.120.0555
DM-ridge1.050.09081.020.0781
DM-kernel0.930.09290.890.0764
DM-MLP1.000.10431.000.0946
MIPS2.530.11892.490.1207
DM-TabPFN17.70.097417.10.0739
DM-TabICLv266.20.096063.80.1077

One-time. The full pretraining run takes 2.52 A100-hours, of which 0.91 reach the submitted checkpoint. Building the response pool took at least 103 GPU-worker hours, almost all of it response generation (Table R5\color{blue}{\text{Table R5}}). The pool figure is a lower bound, so we do not claim a saving over the whole lifecycle.

Table R5:\color{blue}{\text{Table R5:}} One-time cost of the response pool and of pretraining.

StageRecorded cost
Response generation for the pool100.31 GPU-worker hours
Scoring with the two reward models1.79 GPU-worker hours
Embedding1.14 GPU-worker hours
Pool subtotalat least 103.24 GPU-worker hours
Pretraining, full 4,000-update run2.52 A100-hours

W1: Limitations

"little-to-no discussion" of the limitations of the method.

We have included limitations in Appendix E. That being said, we will move a short version into the main text, covering context length, the finite evaluation bank, dependence on the prior, the embedding model, calibration under shift, and one-time cost.


W2: Related work

The related work is sparse. It should be positioned within in-context learning, which starts from arXiv:1606.02185, and should discuss arXiv:2605.10137, which uses a PFN for contextual bandits.

We agree and will extend the related work in both directions, adding both papers. The new text will read:

Dataset-level inference. The neural statistician (Edwards & Storkey, 2017) uses a hierarchical generative model with an exchangeable statistic network to infer a latent context from a dataset. PFN-OPE shares the aim of amortizing inference over sets, but its pretraining is supervised: the target is an OPE functional of the labelled logged rows and the unlabelled target rows.

PFNs for contextual bandits. PFN-TS (Tan et al., 2026) converts a PFN's predictive distribution into mean-reward samples for Thompson sampling, and evaluates policy value offline with SNIPS and a cross-fitted doubly robust check. It uses the PFN to select actions. PFN-OPE trains the PFN to output the policy value directly.


W3: Legend in Figure 2

Thank you very much for catching this! We will add it.

Reviewer_VHE4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles an important problem in llm evaluation which is off policy evaluation. It introduces PFN-OPE, a pretrained transformer that estimates a target LLM’s expected reward from scored behavior-model responses and unscored target responses. It learns across tasks constructed from a response pool with varying reward and policy shifts, enabling evaluation without per-task training.

Critical strengths

The paper addresses a relevant and important problem with an approach that are inspired by previous work but takes a substantial next step towards amortizing evaluations. One thing that I like about this work is that the method is fairly simple yet provide meaningful improvements. The paper is fairly well written and authors have covered major evaluation axis with experiments. Results across two datasets show strong improvements, including transfer to unseen rewards and target models, while ablations are also carried out on a few evaluations dimensions.

Critical weaknesses

I appreciate the experiments that the authors have already conducted. One that that left me wondering is though, the potential mismatch between the motivating setting and the policy shifts evaluated. The paper motivates PFN-OPE through continual post-training, yet evaluation only tests changes between different LLMs. These are useful experiments, but post-training may produce subtler shifts, with responses remaining similar in wording and style while changing in reward-relevant properties that frozen embeddings may not capture. Alternatively, smaller distribution shifts could make direct-method baselines more competitive, reducing PFN-OPE’s relative advantage. Although the tilted and routed policies are intended to approximate post-training shifts, this approximation is not directly validated on actual SFT, DPO, or RLHF checkpoints of the same model. Evaluating these checkpoints would clarify whether the reported gains extend to the motivating deployment scenario. It is possible that I may be overlooking something here given authors have thought more about this so appreciate authors take on this.

Questions for authors
  1. Does predicting the correction improve accuracy over predicting V(π)V(\pi) directly? Have the authors conducted an ablation comparing these objectives under the same architecture and training tasks? Centering the target around the logged mean seems reasonable, but its effect on accuracy and training stability remains unclear. What observation motivated this choice?
1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

We would like to thank the reviewer for the support and the careful questions. We ran both experiments you asked about and will add them to the revision.


W1: Evaluation on real post-trained checkpoints

The paper is motivated by continual post-training, yet "evaluation only tests changes between different LLMs". Post-training may produce subtler shifts: (a) responses stay similar in wording while reward-relevant properties change, which frozen embeddings may not capture; (b) smaller shifts "could make direct-method baselines more competitive". The tilted and routed policies are "not directly validated on actual SFT, DPO, or RLHF checkpoints".

To address the concern, we ran the experiment on real DPO checkpoints. Your point (b) holds for the smallest update: at epoch 1, a direct method baseline is more accurate. At epochs 2 and 3, as the degree of behavior-test policy shift grows, PFN-OPE has the lowest error on both models.

We post-trained two of the behavior LLMs, Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct, with DPO for three epochs each and kept a checkpoint after every epoch. The preference pairs are built from cached responses on HelpSteer2 training prompts, scored by ArmoRM and GRM. Each checkpoint is a target policy, and its base model is the behavior policy. We report results for 100 logged draws.

Table R1:\color{blue}{\text{Table R1:}} DPO checkpoints as target policies. Mean correction: for each logged draw, the gap between the target value and the logged mean reward. Lower MSE is in bold.

DPO runEpochMean correctionPFN-OPE MSEBest baseline MSE
Qwen10.02610.00850.0061 (DM-TabPFN)
Qwen20.03820.00570.0063 (DM-TabPFN)
Qwen30.07200.00680.0070 (DM-kernel)
Llama10.05300.01230.0046 (DM-kernel)
Llama20.10970.00480.0085 (DM-kernel)
Llama30.13060.00500.0106 (DM-kernel)

The results speak to both parts of the concern.

  • (a) These results show that PFN-OPE transfers to real DPO updates with frozen embeddings. They do not show that the embeddings capture every reward-relevant change, and we keep sensitivity to the embedding model as a limitation.
  • (b) Smaller shifts favor direct methods. The shift grows along both runs: the mean correction rises from 0.026 to 0.072 for Qwen and from 0.053 to 0.131 for Llama. At epoch 1, where the shift is smallest, a direct method is more accurate on both. At epochs 2 and 3 PFN-OPE has the lowest MSE on both, by 8.9% and 2.0% for Qwen and by 43.6% and 52.4% for Llama. So in these runs your expectation holds for the smallest update, and PFN-OPE's advantage appears as the update grows.

Q1: Predicting the correction versus the value

Whether predicting the correction improves accuracy over predicting V(π) directly, whether an ablation "under the same architecture and training tasks" exists, and what observation motivated the choice.

We include this ablation with the architecture, training tasks and budget matched. Centering on the logged mean lowers MSE by 5.7% on seen LLMs and 6.9% on new targets, relative to the uncentered value at the same scale. The targets are the correction V − r̄, which is ours, the uncentered value at the same scale, V, and per-response rewards that are then averaged (row-only).

Table R8:\color{blue}{\text{Table R8:}} Value MSE by training target. Lowest in bold.

Training targetSeen LLMsNew targets
Row-only0.04700.1225
Correction, V − r̄ (ours)0.01980.0530
V directly0.02100.0569
  • Accuracy. Centering lowers MSE by 5.7% on seen LLMs and 6.9% on new targets relative to V. The larger effect is the set-level target itself: the correction lowers MSE by 57.9% and 56.7% relative to averaging per-response predictions.
  • Motivation. We center so that the observed logged mean serves as the baseline and the network learns only the adjustment for the dataset in hand (Sections 3.1 and 3.2). The ablation supports this design choice with a modest improvement.
Reviewer_BVm5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a way to estimate how well a new language model will perform using answers and scores collected from older models. The new model answers the same questions, but its answers do not need to be scored.

The authors train a system called PFN-OPE on many simulated evaluation tasks. It learns to compare old, scored answers with new, unscored answers and predict the new model’s average score. Once trained, it can evaluate different models and scoring rules without being retrained each time.

Critical strengths

The problem matters. Evaluating every new language model can require expensive scoring and repeated training. The paper aims to reduce this work as models and evaluation criteria change.

The approach is useful. Once trained, PFN-OPE estimates a new model’s average score from old scored answers and new unscored answers, without retraining for each task.

The training setup is well motivated. The authors reuse a pool of scored answers to create many evaluation tasks with known answers, making training data cheaper to produce.

The experiments are encouraging. The method performs well on two datasets, including tests with models and scoring functions not used during training. Additional experiments help explain which parts of the method produce the gains.

Critical weaknesses

A key baseline is missing. The paper should compare PFN-OPE with a reward predictor trained on the same scored answer pool. This would show how much of the gain comes from its design and how much comes from its training data.

The tests are limited. All scores come from automated reward models. Performance on human-rated answers and actual sequences of updated model checkpoints remains unclear.

Model selection is not evaluated. The paper is motivated by choosing which model to deploy, but it reports only the error of the estimated score, not whether PFN-OPE ranks candidate models correctly.

Reliability under larger changes is uncertain. The method may struggle when a new model produces answers or errors that its training tasks do not cover.

The efficiency benefit is not fully measured. The paper avoids retraining for each evaluation, but does not clearly measure the total cost of building the answer pool, pretraining, and running evaluations.

Questions for authors
  1. How does PFN-OPE compare with an in-context reward predictor pretrained on the same response pool with a comparable budget, then applied to target responses? Separately, how does it compare with a rubric-based LLM judge in a practical evaluation setting?

  2. How does PFN-OPE scale beyond n=512? If its error plateaus, can you distinguish limitations caused by the training prior, the embeddings, or the training context length?

  3. How sensitive are the results to the choice of embedding model?

  4. How does PFN-OPE perform when the target is a post-trained (e.g., DPO or RLHF) checkpoint of the behavior LLM?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Thank you for the careful review and for the specific experiments you proposed.


W1 and Q1: Baseline pretrained on the same response pool

The paper should compare PFN-OPE with a reward predictor "trained on the same scored answer pool", with a comparable budget, to show how much of the gain comes from its design and how much from its training data.

As requested, we trained reward predictors on our proposed prior and their value estimation improved: the MSE of the public TabPFN and TabICLv2 falls by 6.3% to 13.2%. This result shows that our proposed prior is effective. The prediction objective matters as well: for two PFNs with the same pretraining data, architecture and budget, changing the objective from per-response prediction to the aggregate correction lowers MSE by 57.9% and 56.7%.

Data. We trained the public TabPFN and TabICLv2 models on tasks from our prior. Both estimate the policy value by predicting the reward of each target response and averaging.

Design. We compare two PFNs pretrained on our prior with the same initialization, architecture, and prior tasks with the same budget: PFN-DM, trained to predict the reward of each target response and average, and PFN-OPE, trained to predict the aggregate correction.

Table R2:\color{blue}{\text{Table R2:}} MSE of value estimation at n=512n = 512 on the same evaluation tasks, for seen LLMs and for unseen targets (Llama-3.2-3B and Gemma-3-1B). Lowest in bold.

EstimatorPretrainingSeen LLMsNew targets
DM-TabPFNOriginal0.10770.0769
DM-TabPFNOriginal + our prior0.09840.0668
DM-TabICLv2Original0.10300.0798
DM-TabICLv2Original + our prior0.09360.0748
PFN-DM, mean of per-response predictionsOur prior, per-response target0.04700.1225
PFN-OPE, aggregate correctionOur prior, correction target0.01980.0530

Fine-tuning on our prior lowers TabPFN's MSE by 8.7% on seen LLMs and 13.2% on new targets, and TabICLv2's by 9.1% and 6.3%, so the prior helps.


Q1: Comparison with a rubric-based LLM judge

How PFN-OPE compares with "a rubric-based LLM judge in a practical evaluation setting".

PFN-OPE estimates the policy value more accurately than a calibrated judge: its MSE is 55.3% lower for helpfulness and 72.8% lower for GRM.

The judge is Prometheus-2 (7B), with an affine map from judge score to reward that is fitted on the logged rows only and applied to the target rows. The comparison uses 128 HelpSteer2 prompts, selected by hash from the 348 that have a human-rated reference answer for the judge, all six ordered pairs of the three seen LLMs, and ten logged draws per pair and reward, at n=128n = 128.

Table R3:\color{blue}{\text{Table R3:}} Value MSE of PFN-OPE and of the calibrated Prometheus-2 judge, at n=128n = 128. Lower in bold.

RewardPFN-OPE MSEJudge MSE
Helpfulness0.01160.0258
GRM0.02380.0874

PFN-OPE has the lowest MSE of value estimation among all the estimators on this subset.


W2 and Q4: Post-trained checkpoints as target policies

Performance on "actual sequences of updated model checkpoints remains unclear" (W2). Q4 asks how PFN-OPE performs when the target is a DPO or RLHF checkpoint of the behavior LLM.

We ran this experiment on real DPO checkpoints of two behavior LLMs. PFN-OPE has the lowest error at epochs 2 and 3 of both runs. At epoch 1, where the shift is smallest, a direct method is more accurate.

We post-trained Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct with DPO for three epochs each and kept a checkpoint after every epoch. The preference pairs are built from cached responses on HelpSteer2 training prompts, scored by ArmoRM and GRM. Each checkpoint is a target policy and its base model is the behavior policy. We report the results with the same 100 logged draws for every checkpoint and estimator.

Table R1:\color{blue}{\text{Table R1:}} DPO checkpoints as target policies. Mean correction: for each logged draw, the gap between the target value and the logged mean reward. Lower MSE in bold.

DPO runEpochMean correctionPFN-OPE MSEBest baseline MSE
Qwen10.02610.00850.0061 (DM-TabPFN)
Qwen20.03820.00570.0063 (DM-TabPFN)
Qwen30.07200.00680.0070 (DM-kernel)
Llama10.05300.01230.0046 (DM-kernel)
Llama20.10970.00480.0085 (DM-kernel)
Llama30.13060.00500.0106 (DM-kernel)

The degree of behavior-test policy shift increases for the later training epoch checkpoint: the mean correction rises from 0.026 to 0.072 for Qwen and from 0.053 to 0.131 for Llama. At epoch 1, a direct method is more accurate on both. At epochs 2 and 3, PFN-OPE has the lowest MSE on both, by 8.9% and 2.0% for Qwen and by 43.6% and 52.4% for Llama.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W3: Model selection

The paper is motivated by choosing which model to deploy, but it reports only the error of the estimated score, "not whether PFN-OPE ranks candidate models correctly".

We evaluated selection directly. In this experiment, PFN-OPE has higher top-1 accuracy by 6% and 9% on Qwen and by 3% on Llama under ArmoRM.

We test LLM policy selection among checkpoints obtained from post-training. The three DPO checkpoints of each model and, for Qwen, a separate SFT checkpoint were evaluated over 100 logged draws. For each draw, every estimator scores all checkpoints of a family and selects the highest.

Table R4:\color{blue}{\text{Table R4:}} Top-1 selection accuracy among post-trained checkpoints, over 100 logged draws. Higher in bold.

PanelCandidatesPFN-OPE top-1Best baseline top-1
Qwen, ArmoRM497%91% (DM-TabICLv2)
Qwen, GRM499%90% (DM-TabPFN)
Llama, ArmoRM377%74% (DM-MLP)
Llama, GRM385%93% (DM-kernel)

W4: Reliability under larger changes

The method "may struggle when a new model produces answers or errors that its training tasks do not cover".

In our submission, we include tests of large changes in both policy and reward. To supplement these, we additionally add two new experiments: a held-out corpus, and real DPO checkpoints as targets.

In the submission:

  • Section 4.3 holds out the target LLM and the reward model together, including Gemma-3-1B from a family absent from pretraining, and PFN-OPE has the lowest MSE in 11 of 12 pairs (Table 1).
  • Its advantage grows with the size of the shift: in the largest correction bin of Figure 2a the baselines reach an MSE of 0.21 to 0.32 while PFN-OPE stays between 0.04 and 0.07.
  • No evaluation prompt or response appears in pretraining.

New experiments:

  • A held-out corpus. A PFN pretrained on UltraFeedback alone and tested on HelpSteer2, with the target LLMs and the reward model also held out, has a lower MSE than the best baseline (Table R13\color{blue}{\text{Table R13}}).
  • Real DPO checkpoints, which no pretraining task contains (Table R1\color{blue}{\text{Table R1}}). PFN-OPE has the lowest MSE at the later epochs, where the shift is larger, and a direct method is more accurate at the first epoch.

Table R13:\color{blue}{\text{Table R13:}} Cross-corpus transfer, with the target LLMs and the reward model also held out. Value MSE over 18 estimates at n=512n = 512. Lower in bold.

Pretraining corpusTest corpusPFN-OPE MSEBest baseline MSEReduction
UltraFeedbackHelpSteer20.00950.1175 (DM-TabPFN)91.9%

The concern is right in principle: a PFN is only as good as its prior, and Figure 4 shows that PFN-OPE's error is highest where the prior has no tasks. We will write this in the main-text limitations.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

W5: Total cost

The paper "does not clearly measure the total cost of building the answer pool, pretraining, and running evaluations".

We now report all three. Building the pool took at least 103 GPU-worker hours and pretraining 2.52 A100-hours (Table R5\color{blue}{\text{Table R5}}). Online, PFN-OPE costs about 12% of DM-MLP and has the lowest MSE (Table R6\color{blue}{\text{Table R6}}).

One-time cost.

Table R5:\color{blue}{\text{Table R5:}} One-time cost of the response pool and of pretraining.

StageRecorded cost
Response generation for the pool100.31 GPU-worker hours
Scoring with the two reward models1.79 GPU-worker hours
Embedding1.14 GPU-worker hours
Pool subtotalat least 103.24 GPU-worker hours
Pretraining, full 4,000-update run2.52 A100-hours

Online cost. We measured it on a matched workload: one logged dataset of 512 rows, two rewards (the ArmoRM aggregate and GRM), and two target policies with 512 target rows each, giving 18 workloads and 72 estimates per panel on one H100. Cost is wall-clock time: the sum of the measured preprocessing, fitting and hyperparameter selection, and prediction times, all on that H100. Every method may reuse fitted state across rewards and targets. We report the mean of paired cost ratios with DM-MLP set to 1. Generating, scoring and embedding the responses is shared by all estimators and excluded.

Table R6:\color{blue}{\text{Table R6:}} Online wall-clock cost relative to DM-MLP, and value MSE, at n = 512. Lowest in bold.

EstimatorSeen LLMs: costSeen LLMs: MSENew targets: costNew targets: MSE
PFN-OPE0.120.02580.120.0555
DM-ridge1.050.09081.020.0781
DM-kernel0.930.09290.890.0764
DM-MLP1.000.10431.000.0946
MIPS2.530.11892.490.1207
DM-TabPFN17.70.097417.10.0739
DM-TabICLv266.20.096063.80.1077

PFN-OPE costs about 12% of DM-MLP and has the lowest MSE on both panels.

  • More fitting does not close the gap. In separate runs with 1, 2 and 4 times the fitting allowance, DM-MLP's new-target MSE is 0.0949, 0.0915 and 0.0911, and MIPS's stays at 0.1207.
  • Nor does oracle tuning. Choosing ridge and kernel hyperparameters with access to the true target value, which no deployment can do, brings their new-target MSE to 0.0682 and 0.0655, still above PFN-OPE's 0.0555.

Q2: Scaling beyond n = 512

How PFN-OPE scales beyond n = 512 and, if its error plateaus, whether the cause is the training prior, the embeddings or the training context length.

To see how PFN-OPE scales beyond n=512, we have results scaling PFN-OPE up to 2,048 logged rows (where PFN-OPE was pretrained with at most 512 rows). Additionally, we include baseline results fitted on 4,096 rows. Empirically, we see the MSE of PFN-OPE with n=512n = 512 stays well below the best baselines fitted on 4,096 rows.

Unfortunately, we were unable to fully identify the three causes you list, because that requires retraining PFN-OPE from scratch with different priors, embedding models, and context lengths, which did not fit within the response period. However, one observation points to the prior. Within each setting, the squared mean error across logged draws is much larger than the variance across draws, so the remaining error is mostly bias, and more logged rows cannot correct a prior that misses the task. We will state this as a limitation.

Table R7:\color{blue}{\text{Table R7:}} Value MSE of PFN-OPE by direct inference beyond its training context, and of the best baseline fitted on 4,096 rows.

Logged rowsSeen LLMs, ArmoRMNew targets, GRM
5120.01600.0453
1,0240.01590.0451
2,0480.01600.0452
Best baseline, fitted on 4,096 rows0.0687 (DM-MLP)0.0739 (DM-ridge)

Q3: Sensitivity to the embedding model

How sensitive the results are to the choice of embedding model.

We have not measured this yet. Checking the sensitivity takes three steps: re-embedding the response pool and the evaluation responses with a second embedding model, pretraining PFN-OPE again on the new embeddings, and rerunning the evaluation with every baseline on the same embeddings. This takes longer than the response period allowed, so we will add the experiment to the camera-ready version. In the submission, every estimator uses the same frozen Qwen3-Embedding-4B features, so the comparison between estimators is controlled.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.