acceptodds
Under review as a conference paper at ICLR 2027

Rare Event Estimation via Iterative Unalignment

Abstract

As agents are deployed with increased autonomy, even extremely rare events along their stochastic output trajectories can occur and prove catastrophic. Safe deployment therefore does not depend on whether these events can occur, but on how often they might. We study how to estimate the probability of rare failures arising from stochastic variation in an agent's own outputs. Estimating this type of risk requires searching over the combinatorially vast space of trajectories. Naive Monte Carlo is computationally prohibitive in this regime, and constructing effective importance sampling (IS) proposals requires coordinated changes to a context-dependent chain of conditional distributions. We develop a new IS method that perturbs the original model's weights to construct the proposal. The proposal is itself a differentiably parameterized language model, enabling gradient-based search over weight space. We formulate an objective that combines a differentiable surrogate for event amplification and an adaptive regularization scheme that dynamically balances amplification against estimator stability. We evaluate our approach on 120M and 2.6B models across three event families spanning 300+ rare events, verifying probabilities down to . In our most verifiable settings, our IS estimator achieves over compute-weighted efficiency gains over naive Monte Carlo for events with probabilities lower than .

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Rare Event Estimation, Rare Event Simulation, Importance Sampling, Large Language Models, Autoregressive Models, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8dBuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an importance-sampling method for estimating extremely rare events in language-model outputs. Instead of relying on ordinary sampling or searching directly over generated trajectories, their proposed method perturbs the language model's weights to construct a proposal distribution that increases the frequency of target events. It uses differentiable, event-specific surrogates to guide proposal learning and adaptive KL regularization to control importance-weight dispersion. Samples from the learned proposal are then reweighted to estimate event probabilities under the original model. The authors evaluate the method on GPT-2 Small and Gemma-2 across three experimental settings: single-token presence, compositional token events requiring specified tokens in order, and threshold-based events defined by a profanity classifier. They compare IU against naive Monte Carlo, restricted activation- and logit-based variants, cross-entropy methods, and additional MCMC and sequential Monte Carlo baselines. Using independently estimated reference probabilities, they report substantial sampling-efficiency improvements, including over 800× compute-weighted gains on rare token-presence events.

Critical strengths
  1. Meaningful algorithmic contribution. The paper proposes a practical approach to rare-event importance sampling by learning expressive, weight-space proposals for language models. Its use of differentiable, event-specific surrogates allows proposal optimization even when the target event has not been observed, which addresses an important limitation of methods that primarily reinforce sampled trajectories.

  2. Careful consideration of importance-weight stability. The method explicitly addresses the trade-off between amplifying rare events and maintaining adequate proposal coverage through KL regularization and adaptive effective-sample-size feedback. The variance decomposition provides a clear theoretical motivation, and the authors transparently discuss the limitations of their regularization and surrogate assumptions.

  3. Substantial controlled empirical validation. The paper evaluates several event types on GPT-2 Small and Gemma-2, with independently estimated reference probabilities. Comparisons with naive Monte Carlo, cross-entropy methods, restricted proposal variants, and other sampling approaches demonstrate substantial efficiency improvements, including gains that account for proposal-training costs. These results provide convincing evidence of the method's effectiveness in the controlled settings considered.

Critical weaknesses

W1. Comparison with variance-reduced estimators (Section 5 and Appendix B.3). The authors use unbiased first-hit and first-completion estimators to obtain accurate reference probabilities, including for events near 10−910^{-9}. While these methods exploit event-specific structure, they are also relevant alternatives for the token-based benchmarks. The paper would benefit from a compute-matched comparison to better contextualize IU's efficiency gains over naive Monte Carlo and include these comparisons in the main results section.

W2. Clarifying the scope of the claims. The paper motivates the problem through autonomous-agent failures, whereas its formulation and experiments focus on language-model outputs under fixed prompts. Since agent behavior also involves tool interactions and environmental feedback, the two settings are not directly equivalent. Although the authors acknowledge this distinction, more clearly scoping the motivation, introduction and conclusions would help communicate where the results apply.

W3. Dependence on surrogate quality. IU's effectiveness depends on having informative surrogates that adequately cover the target event. The token-based benchmarks provide relatively direct surrogates, while Figure 14 in Appendix C.4 shows reduced accuracy when these signals only partially capture the event. The paper could benefit from further discussion of when suitable surrogates are available and how their quality affects the reliability of IU.

W4. Evaluation across model scales and sequence lengths. The experiments focus on GPT-2 Small and Gemma-2-2B-IT, with relatively short generations (up to 50 tokens). While the computational constraints are understandable, a discussion of how IU might scale to larger models and longer sequences would help clarify its practical applicability, particularly given the additional proposal-training costs and potential importance-weight instability.

Questions for authors

Q1. Surrogate selection and reliability. How are suitable surrogates identified for a given rare event? Are there practical ways to assess whether a surrogate sufficiently captures the target event, particularly when important event pathways may remain unobserved?

Q2. Variance-reduced baselines. Could the authors clarify how IU's compute efficiency compares with the first-hit and first-completion estimators used for reference estimation? These appear to be effective alternatives for the token-based benchmarks, and understanding this comparison would help contextualize the reported gains.

Reviewer_bUbeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses estimating the probability of rare but potentially catastrophic events in LLM outputs, a growing concern as LLM agents act autonomously with little or no oversight. The paper proposes Iterative Unalignment (IU), a method based on importance sampling, whose proposal distribution is built by perturbing the weights of the reference LLM. The proposal is therefore itself a differentiable LLM and can be optimized with gradient-based methods. Training iterates over two terms: a differentiable surrogate that amplifies trajectories leading to the rare event, addressing event scarcity, and a regularizer, that limits how far the proposal deviates from the reference model and keeps importance weights from becoming too dispersed. In the empirical part of the paper, the authors evaluate UI on over 300 events across three different types of events (presence of a single target token, of a potentially ordered set of target tokens and threshold-scored events). They report large efficiency gains over naive Monte Carlo in the number of samples and when accounting for compute.

Critical strengths

S1 The paper is generally well-written and in particular everything up to and including the preliminaries is easy to follow and clear.

S2 The paper addresses are relevant and timely question. Complementing work on whether specific inputs can make a model fail, this paper asks how often a model fairs for a random fixed input.

S3 Decomposing the variance of the importance sampling estimator into event scarcity and weight dispersion is a clear and elegant step, that makes the separation of surrogate to estimate event signal and regularizer to address importance-weight dispersion intuitive.

Critical weaknesses

W1 The evaluation setup is underspecified. The experimental section names the number of events, but gives no description of the data used for the experiments, in particular what data was used, which and how were target tokens chosen? How were the lowest probabilities verfied? How are reference probabilities are distributed across events?

W2 The conclusion sections is too short, and its claims are unclear. It doesn't restate the contributions or the key numbers, or position the work. It's unclear whether the paper shows a gap, closes one, or shows an inherent tension. Statements like "informative search signal" and "deeper perturbations" (L482–484) are vague, and the failure of trajectory-reinforcing methods is neither explained nor quantified. If estimates are often poor, the alignment claim may not follow. It seems like a mismatch with the strong claims of the appendix.

W3 The reporting requires more transparency. Geometric means over events (L431) can mask cases where IU does no better than naive sampling, which matters for a safety-motivated method where worst cases count. The paper doesn't report how many such failures there are or how it handles events that never reach the error threshold. Absolute log error (L404) also hides whether estimates are too high or too low. A smaller related point: the 0.5 std shading in Figs. 4–5 is an unusual choice and isn't justified. Clarify which claims the results support and under which conditions (event type, rarity, model) IU is reliable.

Questions for authors

Clarifications

  • The approach assumes that a good proposal, close to the base model conditioned on the event, can be reached by a (low-rank) perturbation of the base weights. Why should this hold, particularly since closeness in output distribution does not imply closeness in weight space?

  • "Figure 1: [...] We seek a proposal that amplifies all event trajectories by similar factors to stabilize importance weights." Turned around, does that also mean that a malicious developer could use this work to amplify catastrophic events? Put differently, could the method be misused to amplify harmful behavior, given that the surrogate and proposal are designed to do exactly that? Please add a broader-impact discussion.

  • L240: The motivating example comes too late; introduce it in the intro and use it throughout. It also covers only one path to the event (the file being deleted), while others exist (deleting the parent folder, renaming, overwriting). Event recall hints at this, but if the proposal misses such paths, the regularizer can’t recover them, since it only sees trajectories the proposal produces (alluded to in L296).

  • L288: The equivalence of the two divergences is stated for unrestricted conditional distributions. Does this hold in your setting, and why? With a restricted proposal family, the optimal allocations may differ: χ² is mass-covering, while reverse KL is mode-seeking. Since you are using LoRa, I assume we are in the restricted setting. In this case discuss in more detail.

  • Lack of clarity in Section 4

    • The section intro relies on the preliminaries. Make it readable for someone who skips them.
    • L178: What is meant by "tilt"? Exponential reweighting and renormalization, or something else?
    • L182: Define ω.
    • L187: For completeness, I suggest to point out that amplifying event trajectories by 1μ\frac{1}{\mu} gives all other trajectories zero probability.
    • L191:What does "almost no optimization signal" mean?
    • L194: Specify which "differentiable components" of the proposal model the surrogate uses.
    • L197: Provide more information on the regularizer: the regularizer's formal definition, what it penalizes, and how it behaves on non-event trajectories. When it's applied to all proposal trajectories, most of them will be no rare events - does distortion behave in the same way for those?
    • L199: Note here, not only in L235, that amplification requires the event to already be captured by the proposal.
    • L242: "no low-variance estimator", what about naive MC?
    • L252: Specify which gap is demonstrated.
    • How should the error-threshold values in Fig. 4 be interpreted?

Minor Comments

  • L136: Use \citet for Angell et al.
  • L256–266: Trim, this passage is largely redundant. Also rewrite note on batching, because it still yields an unbiased estimate of the expected value.
  • L418: Consider spelling out the acronyms.
  • Fig. 1: The numbers are hard to read, enlarge those and consider adding brief labels
  • Figs. 4–5: Justify the ±0.5 std shading and state it in the legend.

Suggestions

  • L240: Introduce the motivating example in the intro, use it throughout, and cover events that can occur through multiple paths.
  • Related work:
    • L237: Is the nesting of rare events something that has been covered by prior work?
    • consider to discuss Zhao et al. (2024), “Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo,” which seems closely related: for a fixed prompt, they sample from the model's autoregressive distribution and reweight the resulting sequences
  • L31: Sharpen the opening claim by explaining why autonomous agents make harms more likely or more severe.
  • Sec. 5: Rework the introductory sentences.
  • State the scope of the 800×800\times claim (event type, rarity band, model) wherever it appears. (same for other claims of this form)
Reviewer_ouQnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper proposes Iterative Unalignment, which learns importance-sampling proposals in LM parameter space using differentiable surrogates and ESS-adaptive regularization to estimate rare output-event probabilities under a fixed prompt.
Critical strengths
  • Quantifying rare-event probabilities is an important and relatively underexplored problem in generative modeling. Weight-space proposal learning offers an interesting direction for follow-up research.

  • Experiments on approximately 120M- and 2.6B-parameter models show promising results, with reference probabilities down to (10^{-9}) and over (800\times) compute-weighted gains over naive Monte Carlo in the strongest settings.

Critical weaknesses
  • The (10^{-9}) results concern single-token occurrence. Other events involve token combinations or a bag-of-words profanity classifier; realistic semantic or tool-use failures remain untested.
  • Evaluation uses one prompt per model and outputs of at most 50 tokens, with Gemma restricted to 10 tokens. Longer outputs and diverse prompts would better test proposal stability and practical scalability.
  • With 512,000 raw-indicator samples, Table 2’s GPT-2 profanity probability of (8.59\times10^{-5}) implies approximately 15.1% relative standard error, not 3.82%; the corresponding Gemma entry implies approximately 17.6%, not 5.69%. This conflicts with the claimed below-10% precision and requires clarification.
Questions for authors

Please see the weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.