- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents SIREN, a white-box framework for estimating rare-event probabilities in language models. SIREN leverages steered lookahead rollouts to approximate candidate-wise committor values and construct mass-preserving proposals over high-probability tokens. To reduce computational overhead, the framework trains a lightweight MLP to predict relative proposal tilts from model hidden states. The authors further establish theoretical guarantees for estimator unbiasedness and proposal stability. Experiments on rare events involving reading level, toxicity, and jailbreak behavior demonstrate improvements in importance-weight stability and inference efficiency.
Critical strengths
- Important and practically relevant research problem. The paper addresses rare-event probability estimation in LLMs, a problem with significant implications for safety evaluation and deployment risk assessment.
- Principled importance-sampling proposal construction. SIREN leverages activation steering to approximate future event probabilities, while its mass-preserving formulation provides a sound theoretical foundation.
- Meaningful computational efficiency improvements. The amortized committor-tilt surrogate substantially reduces inference overhead.
Critical weaknesses
- Unclear support-preservation guarantees. Equations (9)–(10) may assign zero proposal probability to tokens with positive nominal probability when their estimated potentials vanish. Since the fallback only addresses the all-zero case, the assumptions required for the unbiasedness guarantee remain insufficiently justified.
- Limited cost-matched comparisons and ablations. The experiments do not fully account for offline supervision costs or provide comprehensive comparisons under matched computational budgets. Additional ablations isolating lookahead, active-set restriction, and prefix-only intrvention would better establish their individual contributions.
- Minor presentation issue. At line 350, the heading "4.4 REDUCING PROPOSAL-CONSTRUCTION COST" appears too close to Table 1, resulting in insufficient visual separation.
Questions for authors
- How does SIREN preserve nominal support when individual committor estimates vanish? Could the authors clarify whether positive smoothing or other mechanisms ensure that the assumptions underlying Theorem 2 hold in practice?
- How reliable are the reported results for events with probabilities at or below ? Higher-fidelity reference estimates, confidence intervals, or controlled benchmarks with known event probabilities would strengthen the empirical evaluation.
- Which components are essential to SIREN's performance? Systematic ablations could clarify the individual contributions.
8 comments
Rebuttal Comment Part 2
W2 [Compute Matching - Online] We evaluate inference-compute-matched variants of our method and LMTailRisk. We fix the SIREN Rollout variant at 100 particles and scale each other method’s particle count by its speedup over Rollout, using the runtimes in App. A. For example, on Llama-1B, LMTailRisk is ~3× faster than Rollout and receives 300 particles, and SIREN-MLP is ~4× faster and receives 400.
Table: Jailbreaking results under matched inference compute, compared against an MC reference with 10^4 samples. Reference probabilities fall between 10^-4 and 10^-3.
| Method | Model | # Particles | Hit Rate ↑ | MAE ↓ | Mean Log Error ↓ | w-Log Var. ↓ |
|---|---|---|---|---|---|---|
| LMTailRisk | Llama1b | 300 | 0.75 | 6.75e-04 | 1.48 | 10.29 |
| MC | Llama1b | 500 | 0.29 | 8.30e-04 | 2.01 | - |
| MLP | Llama1b | 400 | 0.63 | 6.46e-04 | 1.34 | 1.95 |
| Rollout | Llama1b | 100 | 0.71 | 4.83e-04 | 1.77 | 9.59 |
| MC | Llama3b | 500 | 0.20 | 7.18e-04 | 2.21 | - |
| LMTailRisk | Llama3b | 300 | 0.90 | 6.52e-04 | 1.45 | 31.08 |
| MLP | Llama3b | 300 | 0.68 | 5.16e-04 | 1.59 | 6.17 |
| Rollout | Llama3b | 100 | 0.58 | 5.90e-04 | 1.96 | 9.83 |
| LMTailRisk | Qwen7b | 400 | 0.56 | 8.65e-04 | 1.66 | 5.48 |
| MC | Qwen7b | 600 | 0.16 | 5.40e-04 | 2.24 | - |
| MLP | Qwen7b | 400 | 0.52 | 5.11e-04 | 1.81 | 3.63 |
| Rollout | Qwen7b | 100 | 0.16 | 4.86e-04 | 2.40 | 0.16 |
Both MLP and LMTailRisk improve on the particle-matched setting in hit rate and log error, and both beat 100-particle Rollout in log error. LMTailRisk has the highest hit rate on all models and the lowest log error on Llama 3B and Qwen 7B; MLP has the lowest log error on Llama 1B. However, both SIREN variants beat LMTailRisk on MAE for every model. Extra particles also do not reduce LMTailRisk’s weight variance on either Llama model (9.88 → 10.29 and 29.8 → 31.08), and SIREN MLP’s variance is 1.5–5× lower than LMTailRisk’s on all three models. All importance-sampling methods beat MC on hit rate and both error metrics, with two exceptions on Qwen 7B: LMTailRisk has higher MAE than MC (8.65e-04 vs. 5.40e-04); and Rollout, which has the largest particle gap to MC (100 vs. 600), matches MC’s hit rate (0.16) and has higher log error. On Qwen 7B, MLP substantially beats both MC and Rollout in hit rate and log error. Overall, LMTailRisk’s higher hit rate yields lower log error (which heavily penalizes floored zero estimates), while SIREN avoids large absolute errors (MAE) and yields more stable weights; under matched compute, MLP is the more effective SIREN variant on hit rate and log error.
Below, we evaluated on the prompts from JYYF W1, with a 10^6 sample MC reference, in the same compute-matched setting.
Table: Jailbreaking results with inference compute-matched particles, compared against an MC reference with 10^6 samples.
| Method | Model | # Particles | Hit Rate ↑ | MAE ↓ | Mean Log Error ↓ | w-Log Var. ↓ |
|---|---|---|---|---|---|---|
| MC | Llama1b | 500 | 0.20 | 4.83e-04 | 1.90 | - |
| LMTailRisk | Llama1b | 300 | 0.75 | 1.32e-04 | 1.15 | 9.18 |
| MLP | Llama1b | 400 | 0.25 | 2.87e-04 | 1.85 | 0.27 |
| Rollout | Llama1b | 100 | 0.55 | 1.73e-04 | 1.72 | 0.68 |
| MC | Llama3b | 500 | 0.05 | 2.44e-04 | 2.04 | - |
| LMTailRisk | Llama3b | 300 | 0.95 | 1.46e-04 | 1.47 | 26.63 |
| MLP | Llama3b | 300 | 0.15 | 2.84e-04 | 1.84 | 0.00 |
| Rollout | Llama3b | 100 | 0.20 | 6.12e-04 | 1.92 | 1.41 |
| MC | Qwen7b | 600 | 0.00 | 5.69e-05 | 1.28 | - |
| LMTailRisk | Qwen7b | 400 | 0.60 | 5.84e-05 | 1.11 | 46.07 |
| MLP | Qwen7b | 400 | 0.50 | 5.68e-05 | 1.28 | 7.20 |
| Rollout | Qwen7b | 100 | 0.20 | 5.50e-05 | 1.12 | 0.00 |
On these prompts, LMTailRisk gains more from extra particles than SIREN MLP. Relative to the particle-matched setting, its hit rate rises on every model (e.g., 0.45 → 0.75 on Llama 1B), and its MAE beats or slightly trails SIREN’s (e.g., 1.38e-03 → 5.84e-05 on Qwen 7B). At matched compute, LMTailRisk therefore has the highest hit rate on all models and the lowest MAE and log error on both Llama models, while all three methods are comparable on Qwen 7B (MAE 5.5–5.8e-05, log error 1.11–1.28). By contrast, in the particle-matched setting SIREN had lower MAE on all three models and Rollout outperformed LMTailRisk on every metric on Llama-1B. Extra particles do not consistently stabilize LMTailRisk’s weights; its log-weight variance increases on Llama-1B (6.65 → 9.18) and Qwen 7B (8.98 → 46.07), and stays at least 6× higher than either SIREN variant’s on every model. SIREN MLP’s variance also rises on Qwen 7B (to 7.20), alongside a hit-rate rise from 0.20 to 0.50. Overall, SIREN produces more stable weights than LMTailRisk, while LMTailRisk scales more effectively with compute in the rarest regime, at the cost of higher weight variance. The particle-matched comparisons in the main text isolate proposal quality; there, the best SIREN variant has lower MAE than LMTailRisk in every setting, with more stable weights. Improving the compute scaling of SIREN is a promising direction for future work.
Rebuttal Comment Part 3
W3 [Presentation] We thank the reviewer for identifying this issue and have revised the manuscript to correct all identified presentation errors.
Q1 [Support Preservation] We thank the reviewer for this important clarification. In our implementation, we implement a defensive mixture to ensure the proposal carries nonzero support wherever the nominal distribution has nonzero mass. See muzD W1 for details.
Q2 [Reference fidelity] We agree that a MC estimate with 10^4 samples is a noisy surrogate for ground truth. We conduct a supplementary experiment with a higher fidelity 10^6 MC sample reference with a subset of evaluation prompts. See JYYF W1 for details.
[Confidence intervals] We note that the manuscript already reports uncertainty on the estimators and reference. For the MC reference, we report its approximate error in Sec. 5.2, which follows directly from the binomial sampling error of the 10^4-sample estimate. For the estimators, we report the per-run importance weight variance (Tab. 1,2,3), which measures the variability of each method’s estimate. Finally, the higher-fidelity 10^6-sample reference (see JYYF W1) yields results consistent with the 10^4 sample performance, indicating that our conclusions are not driven by reference noise. We will make these quantities more prominent in the revision and explicitly connect them to the main results.
[Controlled benchmarks] For semantic events such as jailbreaks, defined by a judge over LLM outputs, the true event probability is not analytically computable, and to our knowledge no benchmark with known probabilities exists for this setting. High-sample MC is therefore the strongest reference available, and this evaluation setting is consistent with existing literature (Angell et al.; Dorman et al., ICML 2026). We view controlled benchmarks for semantic rare events as an important direction for future work.
Q3 [Component Contributions] Active set restriction and lookahead rollouts are intrinsic to the formulation and therefore cannot be isolated directly—lookahead rollouts are required to construct the proposal, and active-set restriction is required for tractability. The MLP variant, which replaces rollout-based estimation with a learned approximation, also serves as an ablation of how the proposal is constructed. Removing the tilt altogether reduces to direct MC.
Here we ablate the remaining components on Llama-1B, varying the active set size (J) , lookahead horizon (τ), mixture strength (ρ), number of nested samples (K), steering parameters, and the steering horizon (T), around a base configuration with J=8, K=7, τ=10, ρ=0.1, T=25, LQR costs Q=0.1/R=10/Qf=0.1
Table: Alternate SIREN Rollout configurations for Llama 1B evaluated over 19-20 prompts with a 10^6 MC sample reference, with 100 particles per trial.
| Setting | Hit rate | MAE | Mean log err | Log var |
|---|---|---|---|---|
| Base | 0.60 | 1.68e-04 | 1.58 | 3.48 |
| ρ = 0.01 | 0.55 | 1.73e-04 | 1.72 | 0.68 |
| ρ = 0.05 | 0.45 | 1.75e-04 | 1.71 | 0.89 |
| ρ = 0.2 | 0.45 | 1.77e-04 | 1.73 | 4.47 |
| ρ = 0.3 | 0.40 | 1.81e-04 | 1.85 | 12.68 |
| LQR costs 1/10/0.1 | 0.35 | 2.20e-02 | 1.93 | 22.67 |
| K = 16 | 0.55 | 3.88e-04 | 1.64 | 4.16 |
| τ = 20 | 0.45 | 1.77e-04 | 1.74 | 0.67 |
| τ = 30 | 0.47 | 1.71e-04 | 1.78 | 2.23 |
| T = 15 | 0.30 | 1.77e-04 | 1.83 | 0.40 |
| T = 40 | 0.42 | 1.74e-04 | 1.78 | 1.29 |
| J = 10 | 0.55 | 1.75e-04 | 1.58 | 5.57 |
| J = 12 | 0.45 | 1.70e-04 | 1.74 | 0.47 |
We observe that most deviations trade hit rate for lower weight variance. Mixture strength ρ controls this trade-off most directly: variance increases monotonically with ρ (from 0.68 at ρ=0.01 to 12.68 at ρ=0.3), while hit rate and log error are best at ρ=0.1. Raising the LQR state cost Q degrades every metric. Increasing the number of nested samples K or the lookahead horizon τ does not improve hit rate or error despite raising computational cost. We also find that too low and too high of a steering horizon T can worsen accuracy, which we hypothesize is a result of a short horizon reaching fewer target events, while a long horizon accumulates weight underestimation. The active-set size J has non-monotonic effects, though increasing J from 8 to 10 raised variance without improving hit rate or error. Given the small prompt set, hit-rate differences of ~0.1 (two prompts) or less should be interpreted cautiously, and we will include a more extensive ablation in the appendix of the final revision.
Rebuttal Comment Part 1
W1 [Support Preservation] We thank the reviewer for raising this issue. Notably, this is an issue in the exposition, but was explicitly handled in the implementation via a defensive mixture on the proposal. See our response to muzD W1 for details.
W2 [Compute Matching] We appreciate the reviewer’s thoughtful consideration of compute equivalences between methods and the contribution of different components of our method. Here, we discuss compute comparisons, and we ablate over different components of our method in response to Q3. We also note that our LMTailRisk results use the authors’ official public implementation, so they reflect the method as originally designed.
[Offline] Since MLP and LMTailRisk have similar inference time cost, here we perform benchmarking of the offline stages of each method, and summarize the results in the table below. Importantly, LMTailRisk also requires a substantial offline phase, in which it calibrates its steering parameters with cross-entropy method (CEM) optimization against ground-truth probability estimates obtained from 10^4-sample Monte Carlo runs. This CEM calibration is necessary for LMTailRisk to be effective; we also provide example results without CEM, and find that the method is degenerate. We split the offline runtime comparison between LMTailRisk and the MLP variant of SIREN into two phases: data collection, and training (CEM optimization for LMTailRisk).
Table: Offline runtime comparison in GPU hours on an H200 GPU. 100 particles used for CEM. LMTailRisk uses 10% of the prompt dataset by default, SIREN is fixed at 20.
As seen in the table, the MLP variant requires >2x less offline compute than LMTailRisk in total, and 2.1-2.9x less per prompt. This follows from how each method obtains data for supervision: LMTailRisk CEM calibration requires full 10^4 particle MC simulations, followed by a grid search over steering parameters. In contrast, SIREN requires no ground truth estimates, uses few rollouts per prompt (10 in our experiments) to generate thousands of token-level supervisory signals, and training the 0.6M-parameter MLP takes ~15 seconds. In our comparisons, LMTailRisk therefore receives more offline compute than the MLP variant of SIREN, so the results in the original manuscript do not favor MLP through additional compute. Matching budgets exactly would require either increasing SIREN’s offline compute via a larger training dataset or reducing LMTailRisk’s CEM budget, and neither would be expected to benefit LMTailRisk; as shown below, removing CEM entirely substantially degrades its estimates.
Table: LMTailRisk with and without CEM, evaluated on Llama3b.