acceptodds
Under review as a conference paper at ICLR 2027

Great Expectations under Language Models with Potentials

Abstract

Many applications of language models hinge not on individual samples but on the expectation of a test functional under the model. Estimating such expectations reliably can be computationally expensive. In this paper, we show how to make estimation more efficient by exploiting the next-token conditional probabilities which are available as a by-product of sampling. We do so through potentials: real-valued functions on prefixes that decompose the test functional additively. We construct an estimator whose variance depends on the chosen potential, and derive conditions under which a potential reduces this variance. We then develop practical potentials for several estimands and applications, and show that for every estimand we study our estimator achieves lower variance than existing baselines at comparable computational cost.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
estimation, language model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_doybOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a way to estimate expectations under LMs (such as KL divergence, n-gram counts, or event probabilities) with lower variance than standard Monte Carlo. The idea is to reuse the next-token probabilities computed during sampling: the quantity of interest is split into step-by-step pieces using a “potential” function, and each piece is replaced by its average over the possible next tokens. The resulting estimator is unbiased, and a control-variate version can fall back to the original estimator. Experiments on losses and divergences, n-gram statistics, rare-event estimation, and watermark detection show consistent variance reductions over the baselines.

Critical strengths

This paper proposes a simple and practical method that works across many tasks. It runs after sampling, reuses probabilities the model already computes, and often costs almost nothing extra. It consistently reduces variance across quite different tasks, and the paper honestly reports where it fails.

Critical weaknesses
  1. Limited novelty and missing related works. Though the work demonstrates effectiveness and efficientcy over MC, the potential estimator equals the MC estimate minus a sum of martingale differences. This seems very related to the previous works [1, 2], and I have the feeling that this is the application of these methods to the regime of LLMs.

  2. More indications of practical value are needed. Baselines are limited to MC, MBAR, and [3]. There are no comparison with twisted SMC [4], which can also estimate the KL divergence between inference and target distributions in both directions. More practical indications are expected, such as a model trained on this proposed estimator, which is included in [3]

[1] Approximating martingales for variance reduction in Markov process simulation. 2002

[2] Doubly robust off-policy value evaluation for reinforcement learning. ICML 2016

[3] Better estimation of the Kullback–Leibler divergence between language models. NeurIPS, 2025.

[4] Probabilistic inference in language models via twisted sequential Monte Carlo. ICML, 2024.

Questions for authors
  1. There are some erroneous references, e.g., line 321 and line 1405, which dramatically decrease the readability of the paper.

  2. The method works best when a good potential is easy to write down, like for KL or n-gram counts. For a new quantity with no obvious structure, how should someone choose a potential? Is there a simple way to tell in advance whether it will help enough to be worth the extra cost?

2 comments

We thank the reviewer for their constructive feedback and for recognizing the simplicity, practicality, and broad applicability of our method. We address the concerns below:

  1. Our work indeed builds on an established statistical literature. Our related work section (Appendix A) already discusses connections to conditional Monte Carlo, control variates, reinforcement learning, and twisted SMC. We thank the reviewer for suggesting references [1, 2], which we have added. Our contribution focuses on adapting these general statistical principles to autoregressive language models and developing a practical framework for a range of estimation problems. We see this as analogous to recent work adapting importance sampling to language models, which similarly builds on long-established statistical principles.
  2. Estimating quantities of interest with lower variance at comparable computational cost has direct practical value beyond training: it can reduce the resources, such as GPU hours, needed to achieve a given precision. In relation to quantities relating to training procedures, we would also like to note that Appendix C.3 shows gradient variance reductions relative to Monte Carlo in all 135 evaluations, covering different quantities of interest and model checkpoints along a DPO training trajectory. Regarding baselines, we would like to note that our reproduction of [3] includes the Horvitz–Thompson estimator and Schulman’s control-variate estimator with fixed and fitted coefficients (Appendix C.2.4). We agree that using SMC could be used as an additional baseline and thank the reviewer for the suggestion.

Answers to the questions:

  1. We thank the reviewer for spotting the two missing cross-references. We have now fixed them in the revised manuscript.
  2. This is a great question! When a function does not admit an obvious additive structure, we propose approximating the oracle potential directly using a learned regressor; we describe this approach in Section 4.3 and demonstrate it in our event-analysis experiments. As a preliminary indication of whether learning a potential may be useful, one could examine the correlation between the available prefix-level features and the final function values in the samples. For a more reliable assessment, we refer the reviewer to our nested cross-fitting procedure (Algorithm 1), which compares candidate potentials based on held-out variance.

Thanks for the response, which solves most of my concerns. I maintain my positive judgement. Good luck!

Reviewer_cSr5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to estimate expectations of functions of language-model outputs more efficiently. It represents a sequence-level function as increments of a potential defined on prefixes, then averages each increment over the next-token distribution. For direct sampling, the authors establish unbiasedness under an integrability condition, identify a conditional-value oracle with zero variance, and derive a control-variate formulation. They also develop analytical potentials and a nested cross-fitting procedure for fitting and selecting learned potentials. The evaluation covers losses and divergences, gradients, n-gram counts and frequencies, watermark detection, and readability-event probabilities estimated using importance sampling. The reported gains are particularly large for selected divergence settings and for n-gram potentials that look ahead to future occurrences. The learned-potential experiment reports a more modest improvement over an existing MBAR estimator. The overall aim is to extract more information from generated samples by reusing conditional token probabilities.

Critical strengths

S1: The paper addresses a useful question: how much more accurately can we estimate an expectation when the model has already computed a distribution over possible next tokens? The direct-sampling construction gives a coherent way to answer this question for different terminal functions. The oracle and control-variate analysis also explain why the quality of the potential matters and why unbiasedness alone does not guarantee lower variance. Also, The comparison between the observed-prefix and lookahead potentials is particularly informative. The simple prefix potential gives almost no improvement, whereas marginalizing future occurrences produces much larger gains, including after the paper’s model-evaluation cost adjustment. This is a useful demonstration that the choice of decomposition materially affects the estimator, and it provides a concrete example of how to design a better potential.

S2: The counterexample in Appendix C.2.4 correctly shows that the variance guarantee in Amini et al. does not hold in general. In the example, the sequence-level Monte Carlo estimate is constant, while the tokenwise estimator still varies. This isolates a real issue with conditioning different summands separately: their covariances can change. The paper also reproduces the earlier experiment, so this contribution combines a clear theoretical correction with an empirical comparison.

Critical weaknesses

W1. The importance-sampling definition and proof describe different estimators

Appendix B.2 defines the importance-weighted potential estimator by replacing the target distribution pp with the proposal distribution qq in the conditional computations and applying the potential estimator to an importance-weighted potential (lines 805–814). However, the variance proof of Proposition 4 (line 832) treats this estimator as the original target-model potential estimator multiplied by the complete-trajectory likelihood ratio p(Y)/q(Y)p(Y)/q(Y). These two constructions may have the same expectation but need not have the same variance.

For example, consider two possible strings, the empty string and "a", with p=(1/4,3/4)p=(1/4,3/4), q=(1/2,1/2)q=(1/2,1/2), and f(y)=1f(y)=1 for both strings. Using the target-model oracle potential, Definition 3 returns 11 for either sampled string, giving a variance of 00. In contrast, Proposition 4 predicts a variance of 1/41/4 for a single sample. This discrepancy calls into question the stated variance formula under importance sampling. The authors should clarify which estimator is intended, reconcile the definition with the proof, and explain which construction is used in the MBAR experiments.

W2. The stability of the learned-potential gains is not fully established

The learned potentials achieve a median 12% variance reduction over MBAR, but the confidence intervals in Table 19 are computed with the fitted potentials, selected candidates, and coefficients held fixed during bootstrapping. While resampling whole chains appropriately accounts for within-chain dependence, it does not capture the variability introduced by fitting and selecting potentials on different data. Since learning and selection are central to the proposed method, it would be useful to show whether the reported gains remain stable when these steps are repeated, or clarify why the current uncertainty estimates are sufficient to support the conclusions.

Questions for authors

Q1. Importance-sampling estimator and variance proof

Regard W1 above, could the authors clarify the relationship between the estimator defined in Appendix B.2 and the trajectory-weighted estimator used in the proof of Proposition 4? In particular, why does line 832 treat them as having the same variance when they may only agree in expectation? Please clarify which construction is used in the MBAR experiments and whether the variance formula needs to be revised. If this concern arises from an ambiguity in the notation or a misunderstanding of the intended construction, a clarification or correction in the rebuttal would be appreciated.

Q2. Stability of learned potentials

Table 19 keeps the fitted potentials and selection results fixed during bootstrapping. Does the reported variance reduction remain stable when potential fitting and nested selection are repeated with different chain splits? A small sensitivity analysis would help clarify whether the improvement is consistent across different fitted models.

3 comments

We thank the reviewer for recognizing the usefulness of our method and for their constructive feedback. We have addressed their concerns in the revised manuscript, as detailed below:

  1. Definition 3 in Appendix B.5 contained an error, which we have now corrected. The correct estimator is the one used in the proof of Proposition 5, obtained by multiplying the potential estimator under the target model by the complete trajectory likelihood ratio. Importantly, this is exactly the estimator we implemented and used throughout our importance-sampling experiments. The error was therefore confined to the written definition and does not affect our reported experimental results. We sincerely apologize for the confusion and thank the reviewer again for their careful reading.
  2. We thank the reviewer for suggesting this sensitivity analysis. We repeated the full nested fitting and selection procedure with five additional seeds (1–5), alongside the original seed (0). For each seed, we reassigned the Monte Carlo chains to folds, refitted the learned potentials, repeated model selection, and re-estimated the control-variate coefficients. Across the six runs, the median variance ratio ranged from 0.865 to 0.881 (SD across runs = 0.006), supporting the stability of the gains under these changes in fold assignment, fitting, and selection. We have included this analysis in Appendix C.5.4, with the results reported in Table 21.

If these revisions address the concerns, we would be grateful if the reviewer could reconsider their assessment and score. We would be happy to clarify any remaining points.

Dear Reviewer,

Thank you again for your review! As the discussion period is coming to an end in a few days, we wanted to check whether our response addresses your concerns. Thank you.

Thank you for the detailed response and revisions. The clarification regarding Definition 3 addresses my main theoretical concern, and the additional analysis provides convincing evidence that the learned-potential gains are stable across fitting and selection runs. These revisions substantially address both of my major concerns. I have therefore raised my rating from 2 to 3.

Reviewer_ib3GOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes approach to estimation of expectations of functions over predictive distributions of LLMs. The approach makes use of potentials - real-valued functions of generated prefixes, which coincide precisely with the value of the target function when generation is complete (EOS is generated). A new estimator based on these potentials is shown to be unbiased, and a calibration approach for equal or lower variance compared to a simple Monte-Carlo estimator is proposed. Experiments on several models ranging from GPT-2 to ~7b and various target functions (including entropy of the predictive distribution, frequency of substring occurence and several others) show that when properly calibrated, proposed estimator has almost universally lower variance than Monte-Carlo estimator across models and target functions.

Critical strengths
  1. The math checks out and the proposed estimator does seem to have the claimed properties.
  2. Experiments are sound and support the predicted variance reduction strongly.

Generally I found the paper to be very thorough, clear and insightful.

Critical weaknesses

The choice of the models is a bit non-standard. I get the need for large number of samples for variance estimation, hence GPT2. Fine-tuned pairs are also understandable for KL estimation. It would be great though to have an experiment on a mainline non-finetuned modern off-the-shelf LLM. Some experiments (watermark and events) justify choice of model through replication efforts, but adding a more mainstream model to some of them would definitely strengthen the evaluation. Note that this is a minor complaint.

Questions for authors
1 comment

We thank the reviewer for their positive assessment and find their suggestion helpful.

In response, we have extended our watermark-detection probability experiment to Mistral-7B-v0.3 using 128 prompts and 1,000 continuations per prompt under the same protocol as for OPT-1.3B. The results are consistent with those obtained on the smaller model: the median variance ratio against Monte Carlo is 0.68 (interquartile range 0.60–0.79) for Mistral-7B-v0.3 and 0.64 (0.55–0.85) for OPT-1.3B. We refer the reviewer to Table 7 in Appendix C.1 of the revised manuscript for the full comparison.

We also note that our experiments on losses and divergences already include relatively recent language models, such as Gemma-2-9B and Tülu-3-8B. More generally, as the reviewer notes, our model choices were guided mainly by two considerations: (1) the trade-off between model size and the sample sizes needed to reliably estimate variance reductions, and (2) the experimental setups used in prior work, which we adopted to enable direct comparisons with existing baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.