- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an importance-sampling method for estimating extremely rare events in language-model outputs. Instead of relying on ordinary sampling or searching directly over generated trajectories, their proposed method perturbs the language model's weights to construct a proposal distribution that increases the frequency of target events. It uses differentiable, event-specific surrogates to guide proposal learning and adaptive KL regularization to control importance-weight dispersion. Samples from the learned proposal are then reweighted to estimate event probabilities under the original model. The authors evaluate the method on GPT-2 Small and Gemma-2 across three experimental settings: single-token presence, compositional token events requiring specified tokens in order, and threshold-based events defined by a profanity classifier. They compare IU against naive Monte Carlo, restricted activation- and logit-based variants, cross-entropy methods, and additional MCMC and sequential Monte Carlo baselines. Using independently estimated reference probabilities, they report substantial sampling-efficiency improvements, including over 800× compute-weighted gains on rare token-presence events.
Critical strengths
-
Meaningful algorithmic contribution. The paper proposes a practical approach to rare-event importance sampling by learning expressive, weight-space proposals for language models. Its use of differentiable, event-specific surrogates allows proposal optimization even when the target event has not been observed, which addresses an important limitation of methods that primarily reinforce sampled trajectories.
-
Careful consideration of importance-weight stability. The method explicitly addresses the trade-off between amplifying rare events and maintaining adequate proposal coverage through KL regularization and adaptive effective-sample-size feedback. The variance decomposition provides a clear theoretical motivation, and the authors transparently discuss the limitations of their regularization and surrogate assumptions.
-
Substantial controlled empirical validation. The paper evaluates several event types on GPT-2 Small and Gemma-2, with independently estimated reference probabilities. Comparisons with naive Monte Carlo, cross-entropy methods, restricted proposal variants, and other sampling approaches demonstrate substantial efficiency improvements, including gains that account for proposal-training costs. These results provide convincing evidence of the method's effectiveness in the controlled settings considered.
Critical weaknesses
W1. Comparison with variance-reduced estimators (Section 5 and Appendix B.3). The authors use unbiased first-hit and first-completion estimators to obtain accurate reference probabilities, including for events near . While these methods exploit event-specific structure, they are also relevant alternatives for the token-based benchmarks. The paper would benefit from a compute-matched comparison to better contextualize IU's efficiency gains over naive Monte Carlo and include these comparisons in the main results section.
W2. Clarifying the scope of the claims. The paper motivates the problem through autonomous-agent failures, whereas its formulation and experiments focus on language-model outputs under fixed prompts. Since agent behavior also involves tool interactions and environmental feedback, the two settings are not directly equivalent. Although the authors acknowledge this distinction, more clearly scoping the motivation, introduction and conclusions would help communicate where the results apply.
W3. Dependence on surrogate quality. IU's effectiveness depends on having informative surrogates that adequately cover the target event. The token-based benchmarks provide relatively direct surrogates, while Figure 14 in Appendix C.4 shows reduced accuracy when these signals only partially capture the event. The paper could benefit from further discussion of when suitable surrogates are available and how their quality affects the reliability of IU.
W4. Evaluation across model scales and sequence lengths. The experiments focus on GPT-2 Small and Gemma-2-2B-IT, with relatively short generations (up to 50 tokens). While the computational constraints are understandable, a discussion of how IU might scale to larger models and longer sequences would help clarify its practical applicability, particularly given the additional proposal-training costs and potential importance-weight instability.
Questions for authors
Q1. Surrogate selection and reliability. How are suitable surrogates identified for a given rare event? Are there practical ways to assess whether a surrogate sufficiently captures the target event, particularly when important event pathways may remain unobserved?
Q2. Variance-reduced baselines. Could the authors clarify how IU's compute efficiency compares with the first-hit and first-completion estimators used for reference estimation? These appear to be effective alternatives for the token-based benchmarks, and understanding this comparison would help contextualize the reported gains.