- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a way to estimate expectations under LMs (such as KL divergence, n-gram counts, or event probabilities) with lower variance than standard Monte Carlo. The idea is to reuse the next-token probabilities computed during sampling: the quantity of interest is split into step-by-step pieces using a “potential” function, and each piece is replaced by its average over the possible next tokens. The resulting estimator is unbiased, and a control-variate version can fall back to the original estimator. Experiments on losses and divergences, n-gram statistics, rare-event estimation, and watermark detection show consistent variance reductions over the baselines.
Critical strengths
This paper proposes a simple and practical method that works across many tasks. It runs after sampling, reuses probabilities the model already computes, and often costs almost nothing extra. It consistently reduces variance across quite different tasks, and the paper honestly reports where it fails.
Critical weaknesses
-
Limited novelty and missing related works. Though the work demonstrates effectiveness and efficientcy over MC, the potential estimator equals the MC estimate minus a sum of martingale differences. This seems very related to the previous works [1, 2], and I have the feeling that this is the application of these methods to the regime of LLMs.
-
More indications of practical value are needed. Baselines are limited to MC, MBAR, and [3]. There are no comparison with twisted SMC [4], which can also estimate the KL divergence between inference and target distributions in both directions. More practical indications are expected, such as a model trained on this proposed estimator, which is included in [3]
[1] Approximating martingales for variance reduction in Markov process simulation. 2002
[2] Doubly robust off-policy value evaluation for reinforcement learning. ICML 2016
[3] Better estimation of the Kullback–Leibler divergence between language models. NeurIPS, 2025.
[4] Probabilistic inference in language models via twisted sequential Monte Carlo. ICML, 2024.
Questions for authors
-
There are some erroneous references, e.g., line 321 and line 1405, which dramatically decrease the readability of the paper.
-
The method works best when a good potential is easy to write down, like for KL or n-gram counts. For a new quantity with no obvious structure, how should someone choose a potential? Is there a simple way to tell in advance whether it will help enough to be worth the extra cost?
2 comments
Thanks for the response, which solves most of my concerns. I maintain my positive judgement. Good luck!
We thank the reviewer for their constructive feedback and for recognizing the simplicity, practicality, and broad applicability of our method. We address the concerns below:
Answers to the questions: