acceptodds
Under review as a conference paper at ICLR 2027

Estimating Rare Events in Language Models via Activation-Steered Committor Proposals

Abstract

At deployment scale, stochastic language generation can expose rare unintended or harmful behaviors that standard evaluation may never observe. We present SIREN, a white-box framework for rare-event simulation and low-probability estimation in large language models (LLMs). SIREN uses activation-steered lookahead rollouts to estimate candidate-wise future event probabilities and converts these estimates into a support-preserving sequential importance-sampling proposal over the model's high-probability token nucleus. To amortize the cost of nested lookahead, we further learn a lightweight committor-tilt surrogate: an MLP maps each candidate token's hidden state to a scalar score and is trained by cross-entropy to match the rollout-derived tilted distribution over the nucleus. At inference, this replaces candidate-specific sub-rollouts with a single cached candidate evaluation plus the MLP, while exact likelihood-ratio correction preserves unbiased probability estimation regardless of surrogate error. Across rare reading-level and jailbreak events, including probabilities on the order of , SIREN reaches event-bearing trajectories with far fewer outer samples than direct Monte Carlo, while the learned surrogate provides a practical accuracy–computation tradeoff.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Rare Event Simulation, Low Probability Estimation, Activation Steering, Mechansitic Interpretability, Alignment, LLMs, Diffusion Models.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QuKtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents SIREN, a white-box framework for estimating rare-event probabilities in language models. SIREN leverages steered lookahead rollouts to approximate candidate-wise committor values and construct mass-preserving proposals over high-probability tokens. To reduce computational overhead, the framework trains a lightweight MLP to predict relative proposal tilts from model hidden states. The authors further establish theoretical guarantees for estimator unbiasedness and proposal stability. Experiments on rare events involving reading level, toxicity, and jailbreak behavior demonstrate improvements in importance-weight stability and inference efficiency.

Critical strengths
  • Important and practically relevant research problem. The paper addresses rare-event probability estimation in LLMs, a problem with significant implications for safety evaluation and deployment risk assessment.
  • Principled importance-sampling proposal construction. SIREN leverages activation steering to approximate future event probabilities, while its mass-preserving formulation provides a sound theoretical foundation.
  • Meaningful computational efficiency improvements. The amortized committor-tilt surrogate substantially reduces inference overhead.
Critical weaknesses
  • Unclear support-preservation guarantees. Equations (9)–(10) may assign zero proposal probability to tokens with positive nominal probability when their estimated potentials vanish. Since the fallback only addresses the all-zero case, the assumptions required for the unbiasedness guarantee remain insufficiently justified.
  • Limited cost-matched comparisons and ablations. The experiments do not fully account for offline supervision costs or provide comprehensive comparisons under matched computational budgets. Additional ablations isolating lookahead, active-set restriction, and prefix-only intrvention would better establish their individual contributions.
  • Minor presentation issue. At line 350, the heading "4.4 REDUCING PROPOSAL-CONSTRUCTION COST" appears too close to Table 1, resulting in insufficient visual separation.
Questions for authors
  1. How does SIREN preserve nominal support when individual committor estimates vanish? Could the authors clarify whether positive smoothing or other mechanisms ensure that the assumptions underlying Theorem 2 hold in practice?
  2. How reliable are the reported results for events with probabilities at or below 10−410^{-4}? Higher-fidelity reference estimates, confidence intervals, or controlled benchmarks with known event probabilities would strengthen the empirical evaluation.
  3. Which components are essential to SIREN's performance? Systematic ablations could clarify the individual contributions.
8 comments
AuthorsOpenReview ↗

Rebuttal Comment Part 1

W1 [Support Preservation] We thank the reviewer for raising this issue. Notably, this is an issue in the exposition, but was explicitly handled in the implementation via a defensive mixture on the proposal. See our response to muzD W1 for details.

W2 [Compute Matching] We appreciate the reviewer’s thoughtful consideration of compute equivalences between methods and the contribution of different components of our method. Here, we discuss compute comparisons, and we ablate over different components of our method in response to Q3. We also note that our LMTailRisk results use the authors’ official public implementation, so they reflect the method as originally designed.

[Offline] Since MLP and LMTailRisk have similar inference time cost, here we perform benchmarking of the offline stages of each method, and summarize the results in the table below. Importantly, LMTailRisk also requires a substantial offline phase, in which it calibrates its steering parameters with cross-entropy method (CEM) optimization against ground-truth probability estimates obtained from 10^4-sample Monte Carlo runs. This CEM calibration is necessary for LMTailRisk to be effective; we also provide example results without CEM, and find that the method is degenerate. We split the offline runtime comparison between LMTailRisk and the MLP variant of SIREN into two phases: data collection, and training (CEM optimization for LMTailRisk).

Table: Offline runtime comparison in GPU hours on an H200 GPU. 100 particles used for CEM. LMTailRisk uses 10% of the prompt dataset by default, SIREN is fixed at 20.

MethodModel# PromptsData CollectionTrainingTotalPer Prompt
LMTailRiskLlama1b230.940.641.580.069
MLPLlama1b200.650.00430.650.033
LMTailRiskLlama3b171.031.92.930.17
MLPLlama3b201.340.00331.340.067
LMTailRiskQwen7b173.942.36.240.37
MLPQwen7b202.550.00442.550.13

As seen in the table, the MLP variant requires >2x less offline compute than LMTailRisk in total, and 2.1-2.9x less per prompt. This follows from how each method obtains data for supervision: LMTailRisk CEM calibration requires full 10^4 particle MC simulations, followed by a grid search over steering parameters. In contrast, SIREN requires no ground truth estimates, uses few rollouts per prompt (10 in our experiments) to generate thousands of token-level supervisory signals, and training the 0.6M-parameter MLP takes ~15 seconds. In our comparisons, LMTailRisk therefore receives more offline compute than the MLP variant of SIREN, so the results in the original manuscript do not favor MLP through additional compute. Matching budgets exactly would require either increasing SIREN’s offline compute via a larger training dataset or reducing LMTailRisk’s CEM budget, and neither would be expected to benefit LMTailRisk; as shown below, removing CEM entirely substantially degrades its estimates.

Table: LMTailRisk with and without CEM, evaluated on Llama3b.

Rarity RegimeHit Rate ↑Mean Log Err. ↓w Log Var ↓
With CEM (100 particles)0.001–0.010.881.2937.07
With CEM (100 particles)0.0001–0.0010.891.8829.8
No CEM (100 particles)0.001–0.010.983.15195.73
No CEM (100 particles)0.0001–0.0010.982.57215.99
No CEM (300 particles)0.001–0.011.03.14229.36
No CEM (300 particles)0.0001–0.0011.02.51261.69
AuthorsOpenReview ↗

Rebuttal Comment Part 2

W2 [Compute Matching - Online] We evaluate inference-compute-matched variants of our method and LMTailRisk. We fix the SIREN Rollout variant at 100 particles and scale each other method’s particle count by its speedup over Rollout, using the runtimes in App. A. For example, on Llama-1B, LMTailRisk is ~3× faster than Rollout and receives 300 particles, and SIREN-MLP is ~4× faster and receives 400.

Table: Jailbreaking results under matched inference compute, compared against an MC reference with 10^4 samples. Reference probabilities fall between 10^-4 and 10^-3.

MethodModel# ParticlesHit Rate ↑MAE ↓Mean Log Error ↓w-Log Var. ↓
LMTailRiskLlama1b3000.756.75e-041.4810.29
MCLlama1b5000.298.30e-042.01-
MLPLlama1b4000.636.46e-041.341.95
RolloutLlama1b1000.714.83e-041.779.59
MCLlama3b5000.207.18e-042.21-
LMTailRiskLlama3b3000.906.52e-041.4531.08
MLPLlama3b3000.685.16e-041.596.17
RolloutLlama3b1000.585.90e-041.969.83
LMTailRiskQwen7b4000.568.65e-041.665.48
MCQwen7b6000.165.40e-042.24-
MLPQwen7b4000.525.11e-041.813.63
RolloutQwen7b1000.164.86e-042.400.16

Both MLP and LMTailRisk improve on the particle-matched setting in hit rate and log error, and both beat 100-particle Rollout in log error. LMTailRisk has the highest hit rate on all models and the lowest log error on Llama 3B and Qwen 7B; MLP has the lowest log error on Llama 1B. However, both SIREN variants beat LMTailRisk on MAE for every model. Extra particles also do not reduce LMTailRisk’s weight variance on either Llama model (9.88 → 10.29 and 29.8 → 31.08), and SIREN MLP’s variance is 1.5–5× lower than LMTailRisk’s on all three models. All importance-sampling methods beat MC on hit rate and both error metrics, with two exceptions on Qwen 7B: LMTailRisk has higher MAE than MC (8.65e-04 vs. 5.40e-04); and Rollout, which has the largest particle gap to MC (100 vs. 600), matches MC’s hit rate (0.16) and has higher log error. On Qwen 7B, MLP substantially beats both MC and Rollout in hit rate and log error. Overall, LMTailRisk’s higher hit rate yields lower log error (which heavily penalizes floored zero estimates), while SIREN avoids large absolute errors (MAE) and yields more stable weights; under matched compute, MLP is the more effective SIREN variant on hit rate and log error.

Below, we evaluated on the prompts from JYYF W1, with a 10^6 sample MC reference, in the same compute-matched setting.

Table: Jailbreaking results with inference compute-matched particles, compared against an MC reference with 10^6 samples.

MethodModel# ParticlesHit Rate ↑MAE ↓Mean Log Error ↓w-Log Var. ↓
MCLlama1b5000.204.83e-041.90-
LMTailRiskLlama1b3000.751.32e-041.159.18
MLPLlama1b4000.252.87e-041.850.27
RolloutLlama1b1000.551.73e-041.720.68
MCLlama3b5000.052.44e-042.04-
LMTailRiskLlama3b3000.951.46e-041.4726.63
MLPLlama3b3000.152.84e-041.840.00
RolloutLlama3b1000.206.12e-041.921.41
MCQwen7b6000.005.69e-051.28-
LMTailRiskQwen7b4000.605.84e-051.1146.07
MLPQwen7b4000.505.68e-051.287.20
RolloutQwen7b1000.205.50e-051.120.00

On these prompts, LMTailRisk gains more from extra particles than SIREN MLP. Relative to the particle-matched setting, its hit rate rises on every model (e.g., 0.45 → 0.75 on Llama 1B), and its MAE beats or slightly trails SIREN’s (e.g., 1.38e-03 → 5.84e-05 on Qwen 7B). At matched compute, LMTailRisk therefore has the highest hit rate on all models and the lowest MAE and log error on both Llama models, while all three methods are comparable on Qwen 7B (MAE 5.5–5.8e-05, log error 1.11–1.28). By contrast, in the particle-matched setting SIREN had lower MAE on all three models and Rollout outperformed LMTailRisk on every metric on Llama-1B. Extra particles do not consistently stabilize LMTailRisk’s weights; its log-weight variance increases on Llama-1B (6.65 → 9.18) and Qwen 7B (8.98 → 46.07), and stays at least 6× higher than either SIREN variant’s on every model. SIREN MLP’s variance also rises on Qwen 7B (to 7.20), alongside a hit-rate rise from 0.20 to 0.50. Overall, SIREN produces more stable weights than LMTailRisk, while LMTailRisk scales more effectively with compute in the rarest regime, at the cost of higher weight variance. The particle-matched comparisons in the main text isolate proposal quality; there, the best SIREN variant has lower MAE than LMTailRisk in every setting, with more stable weights. Improving the compute scaling of SIREN is a promising direction for future work.

AuthorsOpenReview ↗

Rebuttal Comment Part 3

W3 [Presentation] We thank the reviewer for identifying this issue and have revised the manuscript to correct all identified presentation errors.

Q1 [Support Preservation] We thank the reviewer for this important clarification. In our implementation, we implement a defensive mixture to ensure the proposal carries nonzero support wherever the nominal distribution has nonzero mass. See muzD W1 for details.

Q2 [Reference fidelity] We agree that a MC estimate with 10^4 samples is a noisy surrogate for ground truth. We conduct a supplementary experiment with a higher fidelity 10^6 MC sample reference with a subset of evaluation prompts. See JYYF W1 for details.

[Confidence intervals] We note that the manuscript already reports uncertainty on the estimators and reference. For the MC reference, we report its approximate error in Sec. 5.2, which follows directly from the binomial sampling error of the 10^4-sample estimate. For the estimators, we report the per-run importance weight variance (Tab. 1,2,3), which measures the variability of each method’s estimate. Finally, the higher-fidelity 10^6-sample reference (see JYYF W1) yields results consistent with the 10^4 sample performance, indicating that our conclusions are not driven by reference noise. We will make these quantities more prominent in the revision and explicitly connect them to the main results.

[Controlled benchmarks] For semantic events such as jailbreaks, defined by a judge over LLM outputs, the true event probability is not analytically computable, and to our knowledge no benchmark with known probabilities exists for this setting. High-sample MC is therefore the strongest reference available, and this evaluation setting is consistent with existing literature (Angell et al.; Dorman et al., ICML 2026). We view controlled benchmarks for semantic rare events as an important direction for future work.

Q3 [Component Contributions] Active set restriction and lookahead rollouts are intrinsic to the formulation and therefore cannot be isolated directly—lookahead rollouts are required to construct the proposal, and active-set restriction is required for tractability. The MLP variant, which replaces rollout-based estimation with a learned approximation, also serves as an ablation of how the proposal is constructed. Removing the tilt altogether reduces to direct MC.

Here we ablate the remaining components on Llama-1B, varying the active set size (J) , lookahead horizon (τ), mixture strength (ρ), number of nested samples (K), steering parameters, and the steering horizon (T), around a base configuration with J=8, K=7, τ=10, ρ=0.1, T=25, LQR costs Q=0.1/R=10/Qf=0.1

Table: Alternate SIREN Rollout configurations for Llama 1B evaluated over 19-20 prompts with a 10^6 MC sample reference, with 100 particles per trial.

SettingHit rateMAEMean log errLog var
Base0.601.68e-041.583.48
ρ = 0.010.551.73e-041.720.68
ρ = 0.050.451.75e-041.710.89
ρ = 0.20.451.77e-041.734.47
ρ = 0.30.401.81e-041.8512.68
LQR costs 1/10/0.10.352.20e-021.9322.67
K = 160.553.88e-041.644.16
τ = 200.451.77e-041.740.67
τ = 300.471.71e-041.782.23
T = 150.301.77e-041.830.40
T = 400.421.74e-041.781.29
J = 100.551.75e-041.585.57
J = 120.451.70e-041.740.47

We observe that most deviations trade hit rate for lower weight variance. Mixture strength ρ controls this trade-off most directly: variance increases monotonically with ρ (from 0.68 at ρ=0.01 to 12.68 at ρ=0.3), while hit rate and log error are best at ρ=0.1. Raising the LQR state cost Q degrades every metric. Increasing the number of nested samples K or the lookahead horizon τ does not improve hit rate or error despite raising computational cost. We also find that too low and too high of a steering horizon T can worsen accuracy, which we hypothesize is a result of a short horizon reaching fewer target events, while a long horizon accumulates weight underestimation. The active-set size J has non-monotonic effects, though increasing J from 8 to 10 raised variance without improving hit rate or error. Given the small prompt set, hit-rate differences of ~0.1 (two prompts) or less should be interpreted cautiously, and we will include a more extensive ablation in the appendix of the final revision.

Reviewer_JYYFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SIREN, a white-box framework for estimating rare-event probabilities in large language models using sequential importance sampling. SIREN leverages activation-steered lookahead rollouts to approximate future event probabilities and constructs a support-preserving proposal distribution over high-probability candidate tokens. To reduce computational overhead, the authors introduce a lightweight MLP that learns to approximate the rollout-based proposal.

Critical strengths

The paper addresses the challenging problem of estimating rare-event probabilities in LLMs, where standard Monte Carlo sampling can be computationally expensive.

The paper provides a theoretical framework for the proposed importance-sampling approach, including variance characterization of committor estimation, unbiasedness guarantees under support-preserving proposals, and stability analysis for the learned surrogate.

Critical weaknesses

The reference probabilities are estimated using only 10,000 Monte Carlo samples, which may be highly unreliable for extremely rare events. Moreover, prompts with zero observed events are excluded from evaluation.

The method claims to preserve the nominal distribution's support and guarantee unbiased probability estimation. However, the estimated committor values can be zero, potentially assigning zero proposal probability to tokens with positive nominal probability. The paper only discusses the fallback when all estimated potentials vanish.

The paper contains several noticeable formatting problems that affect readability. For example, Table 1 overlaps with the Section 4.4 heading on page 7, and several figures in the appendix have incomplete captions (e.g., Figures 3–8).

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Rebuttal Comment

W1 [MC Reference Fidelity] We thank the reviewer for this thoughtful comment. We agree that a MC estimate with 10^4 samples is a noisy surrogate for ground truth in the rarest evaluated regime. In fact, because the smallest nonzero estimate from 10^4 samples is exactly 10^-4, every prompt in the rarest regime has exactly one hit and a reference value of exactly 10^-4. Error metrics in this bin therefore measure each method’s distance from a constant rather than from the true probabilities, and cannot significantly distinguish methods from a trivial zero estimate, whose score is determined by the probability floor of 10^-6. We adopted this evaluation protocol from prior work (Angell et al., ICML 2026), but agree that it does not strongly support accuracy claims in this regime. We do argue, however, that the results as presented remain meaningful in that they demonstrate improved stability of the estimator over LMTailRisk while retaining a higher hit rate than plain MC.

To evaluate accuracy in the rarest regime, we conduct an additional experiment with a 10^6 sample MC reference. Running this many MC samples for all extreme-rare prompts across all three models would require an estimated 3,000 GPU hours simply due to base model inference time, so we restricted this experiment to a subset. We fix 20 prompts on Llama 1B and Llama 3B, and 10 prompts on Qwen 7B (as Qwen inference times are substantially slower). Half of this prompt set was estimated as p =10-4 and the other half yielded no positive samples with 10^4 MC simulations. We perform 10^6 fresh MC simulations on this set, to obtain a more fine-grained reference. All prompts in this regime yielded a nonzero estimate. In the Table below, we summarize the results of this experiment, where all methods use 100 particles to isolate the utility of the proposal.

Table: Jailbreaking results with 100 particles, compared against an MC reference with 10^6 samples.

MethodModelHit Rate ↑MAE ↓Mean Log Error ↓w-Log Var. ↓
MCLlama1b0.001.86e-042.19-
LMTailRiskLlama1b0.452.57e-041.756.65
MLPLlama1b0.154.02e-042.030.00
RolloutLlama1b0.551.73e-041.720.68
LMTailRiskLlama3b0.859.38e-041.7636.02
MCLlama3b0.056.39e-042.06-
MLPLlama3b0.151.57e-041.910.00
RolloutLlama3b0.206.12e-041.921.41
MCQwen7b0.005.69e-051.28-
LMTailRiskQwen7b0.401.38e-031.188.98
MLPQwen7b0.202.47e-041.110.00
RolloutQwen7b0.205.50e-051.120.00

Rollout achieves lower MAE than LMTailRisk on all three models, with weight variance roughly one to two orders of magnitude lower. LMTailRisk attains the highest hit rate on Llama 3B and Qwen 7B, but its weights are highly variable (w-log variance up to 36.0). We emphasize that a higher hit rate does not imply a better estimate, as when weights vary widely, only a few samples dominate the estimate. Tab. 1 of the manuscript shows an extreme case: direct IS with R=10 achieves the highest hit rate, but with degenerate weights. Both SIREN variants achieve higher hit rates than MC, and log error is comparable across importance-sampling methods at this sample size. Overall, committor-based steering trades some hit rate for substantially more stable weights and thus more reliable estimates at a given sample budget, which is the motivation for SIREN’s formulation.

We have updated the manuscript with the additional results and revised discussion of prior results; in particular, we soften claims of improved accuracy in the rarest region from Tab. 3.

W2 [Support preservation] We sincerely thank the reviewer for highlighting this issue. In our implementation, we use a defensive mixture to ensure the proposal carries nonzero support wherever the nominal distribution has nonzero mass. Please refer to our response to muzD W1 for details.

W3 [Presentation] We appreciate the reviewer’s feedback on the presentation of the original manuscript and have made revisions to the formatting, particularly with regards to the compressed spacing. We have also made updates to the writing to improve clarity, and highlight the revised sections in blue in the updated manuscript (other small corrections or notational inconsistencies have been addressed in other sections, but we don’t believe this is notable for the reviewers). The underlying technical content is unchanged, aside from the mixture proposal discussion as described in muzD W1.

AuthorsOpenReview ↗

Rebuttal Follow-up

Dear Reviewer JYYF,

We thank you again for your constructive feedback on our submission. As the discussion period nears its end, we would appreciate your feedback on our rebuttal, including the higher-fidelity reference experiments [W1], support condition clarification [W2], and presentation updates [W3]. We would be grateful if you would consider these in your assessment, and please let us know if any points require further clarification.

Best regards,

Authors

Reviewer_muzDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper does sequential importance sampling for rare events in white-box LLMs. Activation-steered lookahead rollouts estimate a committor for each token in the nucleus, and these estimates tilt only the nucleus mass. An MLP then distills this into an amortized proposal.

Critical strengths

Using steering as an oracle for the proposal, rather than as the proposal itself, is an interesting design. The theory is simple and correct: it covers unbiasedness and the optimal prefix-only proposal. The amortized version keeps 74 to 86% of base throughput, and its log-weight variance is lower than that of LMTailRisk.

Critical weaknesses
  1. The support condition can be violated. If the committor estimate is zero for a nucleus token whose true probability is positive (for example when ARI scores of 4 or less are set to zero), that token gets zero proposal mass. The fallback only applies when all potentials vanish, so the unbiasedness claim (Thm 2) does not hold for the method as implemented.
  2. The reference probabilities come from 1e4 Monte Carlo samples, and prompts are kept only if they have at least one hit, which biases the reference upward. Around p = 1e-4, every method's error is close to that of answering zero, so the 1e-4 claim is not supported.
  3. The results are mixed. LMTailRisk has a higher hit rate on Llama-3B and Qwen, and the method does worse than Monte Carlo on Llama-3B. The comparisons are not compute-matched: the nested variant is 3 to 7x slower, and the cost of training the MLP is not counted.
  4. Twisted SMC, SMC with resampling, and multilevel splitting are missing as baselines. Several of the "jailbreak" examples are benign. Figure captions read "Caption", and two appendices are placeholders.

Minor: paper format has been "butchered" with multiple negative vspaces and padding areas removed. The paper looks quite terrible to the eye in this stage.

Questions for authors
  • How do you ensure the proposal stays positive on the nucleus?
  • What are the results at matched compute, with a 1e6-sample Monte Carlo reference?
  • Are the MLP's training prompts disjoint from the evaluation prompts?
  • How does the method compare with twisted SMC?
4 comments
AuthorsOpenReview ↗

Rebuttal Comment Part 1

W1 [Support preservation] We thank the reviewers for identifying this issue. They are correct that the statement “In (10), p^t\hat p_t … preserves the full nominal support” (l270 original submission) does not hold in general. With a finite number of rollouts KK, or with thresholded scores (e.g., ARI ≤ 4, as noted by Reviewer muzD), the estimate q~t\widetilde q_t can be zero for a token that has positive probability of reaching EE. Such a token would then receive zero proposal mass and introduce bias to the estimator, and the all-zero fallback does not cover this case.

This was an error in the exposition, not in the method used for our experiments. In all reported experiments, the proposal is the standard defensive mixture (1−ρ)p^+ρp(1-\rho) \hat p + \rho p, with ρ=0.01\rho = 0.01. In this formulation, every token with positive nominal probability has positive proposal probability by construction, so the support condition for Thm. 2 holds for the method as implemented (see below for a correction on the code provided with the initial submission).

We have revised the manuscript accordingly. We corrected the aforementioned statement at line 270 and included a discussion of the mixture proposal, amended Section 4.1 with explicit discussion of support conditions which must be satisfied by the committor approximations.

Code. We note that the code included with the original submission was an outdated snapshot that did not match the version used for our experiments. It differed from the experimental code in three ways that affect correctness: (1) In both committor_steering.py and committor_steering_gpt.py, a KV-caching bug caused the base probabilities in the nested lookaheads to be computed from a cache built on steered hidden states, which biased the lookahead-derived weights. (2) committor_steering.py omitted the mixture proposal, although committor_steering_gpt.py included it. (3) committor_steering.py did not stop generation at the EOS token, so text potentially kept being generated unnaturally (e.g., after a hard refusal). None of these issues were present in the code used to produce the reported results. We have replaced the snapshot with our experimental version, which also includes minor consistency updates elsewhere in the codebase. The updated code matches the pipeline described in the revised paper.

W2 [MC reference] We thank the reviewer for this insightful comment, and agree that 10^4 MC samples is an imprecise surrogate for ground truth in the rarest regime, and that retaining only prompts with at least one hit biases the reference upward. Please refer to our response to JYYF W1 for a more detailed discussion and experimental results.

W3 [Mixed Results] We agree that the original manuscript did not clearly establish the methods' relative strengths; our expanded evaluations with a higher fidelity reference and matched inference-time compute (JYYF W1, QuKt W2) help clarify them. At matched particle budgets, the best SIREN variant between MLP and Rollout achieves lower MAE than LMTailRisk in every setting, typically with substantially lower weight variance. At matched compute in the 10^-4–10^-3 regime, both SIREN variants retain lower MAE on all models and MLP retains 1.5–5× lower weight variance, while LMTailRisk has a higher hit rate on all models and lower log error on two of three. In the rarest regime, LMTailRisk benefits more from additional particles, outperforming SIREN on hit rate and on error for both Llama models, while the methods are comparable on Qwen 7B. In summary, SIREN avoids large absolute errors and produces more stable importance weights. In the rarest regime, LMTailRisk benefits more from additional compute, mainly because it finds more hits as compute increases, though this comes with higher weight variance. Improving how SIREN scales with compute in this regime is a promising direction for future work.

We acknowledge that Rollout's MAE exceeds MC's on Llama 3B in some settings. In both, MC hits 4–6× fewer prompts than Rollout, and Rollout has lower log error. Because MAE is driven primarily by overestimates, it may occasionally favor MC which rarely hits the event. The same effect causes LMTailRisk MAE, using its official implementation, to exceed MC's in a few settings (e.g., Qwen 7B in Tab. 3).

[Compute Matching] We agree that compute matching is important. SIREN MLP's offline cost is lower than LMTailRisk's due to LMTailRisk relying on ground truth estimates and a grid search CEM optimization (QuKt W2). Thus, existing MLP comparisons are compute-matched, if not favoring the baseline (e.g., 2.93 GPU hours for LMTailRisk offline vs. 1.34 for MLP on Llama 3B). We report inference-compute-matched results in our response to QuKt W2.

AuthorsOpenReview ↗

Rebuttal Comment Part 2

W4 [TwistedSMC] We thank the reviewer for this comment. We agree that TwistedSMC (Zhao et al., ICML 2024) warrants further discussion. We train a learned twist function as in their official codebase with lr=0.003, 1000 train particles, and 8 epochs (as determined by a small tuning sweep), on the StrongReject dataset. Note that this setup exceeds the offline compute requirements of LMTailRisk or SIREN MLP: TwistedSMC used 1.84 GPU hours for training the twist on Llama 1B, while LMTailRisk used 1.58, and MLP used 0.65. We were unable to achieve meaningful validation loss improvements for Llama 3B and Qwen 7B, despite committing 4.04 and 7.20 GPU hours, respectively, to training each tilt. We therefore only report Llama 1B here, as the other models revert to direct MC.

Table: Jailbreaking results with TwistedSMC 100 particles, compared against the 0.0001–0.001 MC reference of Tab. 3 in the manuscript

MethodModelHit Rate ↑MAE ↓Mean Log Error ↓w-Log Var. ↓
TwistedSMCLlama1b0.041.14e-032.590.05
RolloutLlama1b0.714.83e-041.779.59

TwistedSMC has substantially lower hit rate and higher errors than Rollout. We hypothesize that this is because twist learning relies on positive samples for supervision, which are scarce in this setting. This is consistent with limitations noted in the original paper, and their binary rare-event experiment uses a single prompt and a small model. Our setting, with larger models and twists that must transfer across a diverse prompt set, likely exacerbates this limitation.

[SMC, Splitting] SIREN is a method for constructing a proposal distribution, and is complementary to SMC with resampling and multilevel splitting rather than an alternative. In particular, the committor function that SIREN estimates is the optimal twist function for twisted SMC targeting EE, and the optimal importance function for multilevel splitting [1]. SIREN could therefore be integrated into either framework. We focus on per-trajectory importance sampling because it directly isolates the quality of the proposal distribution, which is the core of our contribution. We have added a discussion of these connections to the revised conclusion and view combining SIREN with resampling and splitting as a promising direction for future work.

[Benign jailbreaks] The jailbreak event is defined by an LLM-as-a-judge, as is standard in existing literature (Angell et al.; Skifstad et al., ICML 2026), so some flagged outputs may be benign from a human perspective as a result of inconsistencies/subjectivity in the judge. Thus, in practice the event targeted by the estimators is defined by the judge, which may not always align with human intuition. We agree this is an important clarification, and we include a brief discussion along with the qualitative examples in the revision.

[Presentation] We apologize for the presentation errors and have corrected them along with the revisions discussed in JYYF W3.

[1] Cérou et al. “On the Asymptotic Normality …” 2019

W5 [Presentation] We thank the reviewer for their feedback on the presentation of the original manuscript, and have updated the manuscript accordingly. Please refer to JYYF W3 for details.

Q1 [Proposal support] We thank the reviewer for this question. In our implementation we leverage a simple defensive mixture to ensure the proposal carries nonzero support wherever the nominal distribution has nonzero mass. See W1 for details.

Q2 [Compute matching, reference fidelity] We conduct additional experiments with a 10^6 MC sample reference, and summarize the results in our response to JYYF W1. We extend this comparison to an inference-compute-matched setting in response to QuKt W2.

Q3 [MLP Training/Test Set] Yes. The MLP training prompts were explicitly selected such that they remain disjoint from the evaluation prompts.

Q4 [TwistedSMC] We evaluate TwistedSMC in the jailbreak setting in response to W4.

AuthorsOpenReview ↗

Rebuttal Follow-up

Dear Reviewer muzD,

We thank you again for your thoughtful review. As the discussion period nears its end, we would appreciate your feedback on our rebuttal, including the support condition clarification [W1], higher-fidelity reference experiments [W2], compute-matched experiments [W3], baseline discussion [W4], and presentation updates. We would be grateful if you would consider these in your assessment, and please let us know if any points require further clarification.

Best regards,

Authors

Reviewer_muzDOpenReview ↗

thanks for your rebuttal.

Thank you for the detailed response, the additional experiments and the revised manuscript.

I understand that the paper leans heavily in its presentation of results that describe various trade-offs, as is common in ML when competing (perhaps irreconciliable) metrics compete. Those are classic pareto front tradeoffs. I think the paper is doing a poor job in that area, since all results are crammed in tables without adequate visualizations. The authors do not propose a solid "metric" hierarchy (L. 488) that could very clearly spell out scenarios in which one of these metrics would take precedence over the others, so I think this could be improved. In the end the uneducated reader on this topic is just left with a bunch of unannotated numbers, and none of the methods dominates consistently the others.

In terms of results, at matched compute, LMTailRisk has higher hit rates and lower log error on both Llama models (App. A.1). Weight stability therefore does not yet translate into more accurate estimates, yet the abstract still highlights probabilities on the order of 10^-4. I find this still confusing.

More broadly, I feel the paper would benefit from stating its intended use: finding examples of the rare behavior, or estimating its probability with error bars. The reported metrics favor different methods depending on which goal is targeted and this is all a bit confusing.

I encourage the authors to improve this presentation aspect to convince a wider readership of the practical interest of these methods, and while not opposed to acceptance, can't really support it actively more than with the current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.