- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RARETRAP, a black-box framework for estimating how likely an LLM is to exhibit rare but severe behaviors under a specified prompt distribution. It defines a prompt distribution by mapping random latent vectors to discrete prompts through a surrogate model’s token embeddings, then progressively concentrates sampling on more severe behaviors to estimate small probabilities efficiently while collecting the prompts that trigger them. The authors evaluate the framework on over-generation and degenerate repetition across 10 open-weight models and two frontier models. The experiments show it applies broadly across models, check its estimates against direct Monte Carlo in selected settings, and analyze its design through sensitivity studies.
Critical strengths
-
The paper addresses a timely and important problem. The problem formulation clearly distinguishes the reference prompt distribution, behavioral severity metrics, and estimator. The paper also situates itself properly within the existing literature, explaining how its objective differs from prior work and why quantifying the probability of severe behaviors is important.
-
The experiments cover a wide range of models (10 open-weight models and two frontier black-box LLMs), evaluating two behavioral outcomes. The authors also conduct sensitivity analyses on prompt length and latent dimensionality, and examine higher severity thresholds. These experiments demonstrate the applicability of the proposed framework across different models and settings.
Critical weaknesses
-
It is unclear how to interpret probabilities estimated under the proposed artificial reference distributions in practical settings. Section 3.2 describes how to generate prompts, and it does not require the resulting tokens to be meaningful sentences. In Appendix F, the author acknowledges the resulting prompts ‘consist of a short mixed-script string of Unicode characters carrying no clear request.’ My concern is that the evaluations largely measure model behavior on unusual, incoherent inputs rather than realistic user queries. It is unclear how the estimated probabilities translate into actionable insights for model developers or real-world safety evaluation, as suggested in the abstract (i.e., ‘‘prioritizing alignment effort to improve safety and mitigate risks’’). Also, Tables 1 and 2 show that the measured probabilities vary a lot when only the surrogate model is changed, with other settings held fixed. I did not find any robustness checks across different random projection realizations. Although the dependence on the reference distribution is expected and acknowledged by the authors, the paper provides limited guidance on how to select a meaningful reference distribution or interpret model comparisons across different distributions.
-
While the paper shows RARETRAP can discover severe behaviors and produce probability estimates, evidence for estimation accuracy is limited. The comparison with direct Monte Carlo is limited to four configurations (Sec 4.3), leaving the more extreme probability estimates unvalidated. The main tables report single runs without confidence intervals or variance estimates, and no experiment compares estimation error against direct Monte Carlo or alternative rare-event estimators at a matched evaluation budget.
Questions for authors
-
The theoretical formulation assumes deterministic model responses, while the frontier-model experiments use stochastic decoding. Could the authors clarify how response randomness is handled in the conditional sampling procedure and whether the resulting probability estimates remain valid under stochastic decoding?
-
Could the authors compare RARETRAP with the rare-event estimation methods discussed in the related work, such as Wu & Hilton (2025), Cao et al. (2026), and Liu et al. (2026), with matched target-model evaluation budgets? How do these methods compare in estimation accuracy and efficiency? If any work cannot be directly applied in this black-box setting, can the authors explain why?
-
The paper acknowledges that ties in discrete severity metrics can bias the conditional probability estimates. How prevalent are such ties in the main experiments, and to what extent might they affect the reported probabilities, particularly for deeper conditional levels?
-
The weakness I mentioned above.
4 comments
W1 introduces multiple points, which we group under four main categories.
- Practical interpretability & actionable insights
Our goal is to provide a common basis for comparing models evaluated under a given distribution intended for controlled stress testing and model comparison. We envision many mappings may potentially converge to a few canonical distributions. Moreover, the estimates support three practical uses. (1) Prioritizing interventions: helping developers identify if a model needs interventions such as length calibration, compared to how other models perform under a shared mapping. (2) Comparing models: our experiments show that model rankings remain stable across surrogates, even when absolute probability estimates change (Table A below). This suggests that relative model comparisons may be informative across different mappings, while using a fixed mapping provides a consistent basis for comparison. (3) Building regression tests / training data: the conditional populations provide tail-enriched prompt-response pairs that developers can use for regression testing or post-training.
- Input Prompt Coherence
While many of the prompts do not resemble natural language, the behavior it exposes is nonetheless a real reliability failure. A model given unreadable input should ask for clarification and stop rather than generating endlessly. Moreover, unnatural inputs occur in deployment, for example, through OCR and encoding errors, tool and retrieval output, mixed-language pastes, and intentional malicious inputs. However, we consider this a limitation of the mapping, not the estimator. An alternative fixed map from latent to prompt space can provide more natural user prompts. To implement this, two directions look promising: (1) using a learnt latent-to-prompt map, for example, a diffusion language model that decodes latent input into fluent text; (2) conditioning the reference distribution on a readability score to ensure it's below a certain threshold, while keeping the distribution explicit (e.g., estimate into ). This can provide an estimate of the probability of the behavior among readable prompts, and is a key direction of future work.
- Surrogate sensitivity & selection
To evaluate the sensitivity of our estimates to surrogate choice, we added SmolLM2-360M and Gemma-3-1B as surrogates for three target models (Table A). Absolute estimates vary by one to three orders of magnitude across surrogates, but the model ranking (DeepSeek > Mistral > OLMo) is preserved in every case. This suggests that although surrogate choice substantially affects absolute estimates, relative model comparisons can remain robust across different surrogate mappings. We therefore recommend using a fixed external surrogate for cross-model comparisons and evaluating multiple surrogates to assess the robustness of the rankings. Absolute estimates should not be compared directly across different reference distributions. These findings have been added to the revised manuscript.
Table A. Over-generation probability under four surrogates. Each entry is the mean over the number of runs in parentheses. "Not reached" means the event was not reached within five levels.
| Target | Self | Qwen3-0.6B | SmolLM2-360M (new) | Gemma-3-1B (new) |
|---|---|---|---|---|
| DeepSeek-8B | 0.36 | 0.17 | 2.4e-2 | 8.4e-2 |
| Mistral-7B | 2.5e-2 | 8.4e-2 | 2.1e-3 | 1.6e-2 |
| OLMo-3-7B | 4.2e-3 | 1.7e-2 | < 1e-5 (not reached) | 2.6e-3 |
- Robustness to the random projection We further evaluated multiple configurations under two different projection draws (Table B). Their estimates differ by a median factor of 1.17 , and all are within a factor of 2. This is no larger than the difference between repeated runs under a single projection (see W2), and well below the effect of changing the surrogate. This demonstrates the robustness of our framework to the random projection. This suggests that projection choice has a relatively modest effect on the estimates in the tested configurations, compared with the observed sensitivity to surrogate choice.
Table B. Effect of the projection draw under the Qwen3-0.6B surrogate, estimated under two seeds. Each value is the mean over the number of runs in parentheses (additional runs are in progress).
| Target | Behavior | seed A | seed B | Ratio |
|---|---|---|---|---|
| GPT-OSS-20B | length | 1.82e-2 | 1.88e-2 | 1.03 |
| Nemotron-9B | length | 1.35e-2 | 1.78e-2 | 1.32 |
| DeepSeek-8B | length | 0.165 | 0.225 | 1.36 |
| GPT-OSS-20B | repetition | 6.36e-2 | 4.89e-2 | 1.30 |
| Qwen3-14B | repetition | 6.18e-3 | 5.94e-3 | 1.04 |
| DeepSeek-8B | repetition | 0.250 | 0.240 | 1.04 |
W2
Estimation accuracy (The comparison with direct Monte Carlo) is limited
We are adding four additional configurations and will report them once available. These comparisons cover probabilities from about 6e-3 to 4e-2. We acknowledge that estimates for rarer events remain unvalidated, as validating them with direct MC would require substantially more computational resources. We will clarify this limitation in the revised manuscript.
The main tables report single runs
We reran every Table 1 entry twice more with new estimator seeds, keeping each entry's projection fixed, and we now report the mean with a 95% confidence interval across runs (Table D). Entries that are estimated directly from the initial sample (probability of 0.1 or more) reproduce within a factor of 1.2. Of the 14 entries that need conditional levels, ten agree within a factor of 2 across the three runs. Only the rarest ones differ by a factor of 3 to 4.
Table D. Table 1 with confidence intervals across runs: mean and 95% confidence interval over three independent runs (the reported run and two repeats).
| Target | Surrogate | Over-generation, mean [95% CI] | Repetition, mean [95% CI] |
|---|---|---|---|
| DeepSeek-R1-Distill-Llama-8B | Self | 0.362 [0.323, 0.402] | 0.467 [0.426, 0.507] |
| DeepSeek-R1-Distill-Llama-8B | Qwen3-0.6B | 0.165 [0.136, 0.197] | 0.250 [0.216, 0.287] |
| Phi-4-reasoning | Self | 1.000 [0.994, 1.000] | 1.000 [0.994, 1.000] |
| Phi-4-reasoning | Qwen3-0.6B | 1.000 [0.994, 1.000] | 1.000 [0.994, 1.000] |
| Qwen3.5-9B | Self | 0.995 [0.985, 0.999] | 0.993 [0.983, 0.998] |
| Qwen3.5-9B | Qwen3-0.6B | 0.405 [0.365, 0.446] | 0.425 [0.385, 0.466] |
| NVIDIA-Nemotron-Nano-9B-v2 | Self | 0.510 [0.469, 0.551] | 0.543 [0.503, 0.584] |
| NVIDIA-Nemotron-Nano-9B-v2 | Qwen3-0.6B | 1.4e-2 [5.7e-3, 3.0e-2] | 1.5e-2 [9.5e-3, 2.5e-2] |
| Mistral-7B-Instruct-v0.3 | Self | 2.5e-2 [1.4e-2, 4.2e-2] | 3.0e-2 [2.1e-2, 4.4e-2] |
| Mistral-7B-Instruct-v0.3 | Qwen3-0.6B | 8.4e-2 [4.9e-2, 1.4e-1] | 7.7e-2 [6.7e-2, 8.8e-2] |
| OLMo-3-7B-Instruct | Self | 4.2e-3 [6.4e-4, 2.0e-2] | 9.3e-3 [5.5e-3, 1.5e-2] |
| OLMo-3-7B-Instruct | Qwen3-0.6B | 1.5e-2 [1.2e-2, 1.9e-2] | 2.4e-2 [1.7e-2, 3.3e-2] |
| Qwen3-14B | Qwen3-0.6B | 3.4e-3 [7.8e-4, 1.2e-2] | 6.2e-3 [4.2e-3, 9.0e-3] |
(additional runs in progress)
We will also run repeat runs for Table 2.
Comparison at matched evaluation budget
The comparison with direct MC in Table 4 uses a matched evaluation budget. For probabilities around , where a single conditional level suffices, RARETRAP is comparable with direct MC. Its advantage emerges for rarer events, which direct MC cannot reliably resolve at these budgets.
**W2** Additional Comparisons with Monte Carlo (estimation accuracy)
We revisit W2 regarding the Estimation accuracy (The comparison with direct Monte Carlo is limited) and include additional more extreme runs.
The Tables below demonstrate how many RARETRAP runs fall within the MC 95% CI for all the LLM/surrogate/task combinations from Table 2 in the manuscript:
Over-generation
| Target / surrogate | Direct MC [95% CI] | RARETRAP, mean ± SD | Runs within 95% range of MC |
|---|---|---|---|
| Qwen3.8-27B / Qwen3-0.6B | .0681 [.0617, .0749] | .0721 ± .0079 | 4/5 |
| Qwen3.8-27B / Qwen2.5-0.5B | .0840 [.0770, .0915] | .0842 ± .0049 | 5/5 |
| Nemotron-3.5-30B / Qwen3-0.6B | .0040 [.0029, .0053] | .0049 ± .0022 | 3/5 |
| Nemotron-3.5-30B / Qwen2.5-0.5B | .0113 [.0092, .0138] | .0104 ± .0032 | 4/5 |
Degenerate repetition
| Target / surrogate | Direct MC [95% CI] | RARETRAP, mean ± SD | Runs within 95% range of MC |
|---|---|---|---|
| Qwen3.8-27B / Qwen3-0.6B | .0382 [.0334, .0436] | .0374 ± .0036 | 5/5 |
| Qwen3.8-27B / Qwen2.5-0.5B | .0626 [.0565, .0692] | .0613 ± .0047 | 5/5 |
| Nemotron-3.5-30B / Qwen3-0.6B | .0040 [.0029, .0053] | .0037 ± .0018 | 4/5 |
| Nemotron-3.5-30B / Qwen2.5-0.5B | .0096 [.0077, .0120] | .0109 ± .0033 | 4/5 |
The tables show that most runs fall within the 95% CI of MC, a total of 34 of 40 across the two tasks. The runs outside it lie on both sides of the MC estimate, so there is no sign of a systematic bias in either direction. Agreement holds for both behaviors and under both surrogates, including the rarest events in the table (Nemotron-3.5-30B, ), which requires up to three levels.
We thank Reviewer dHvw heir acknowledgement of the importance, clarity, and timeliness of our work, and raising insightful questions. We address the questions first, followed by the weaknesses.
Q1. Could the authors clarify how response randomness is handled in the conditional sampling procedure and whether the resulting probability estimates remain valid under stochastic decoding?
A1. Under stochastic decoding, the response depends on both the input prompt and the decoding randomness. In our conditional sampling procedure, a proposal is accepted only if its sampled response satisfies the current intermediate event. One consequence is that a prompt in the final population is not guaranteed to reproduce the severe behavior if queried again. However, we have not investigated how our latent mapping and conditional sampling procedure extend to stochastic decoding. We therefore consider this an important limitation and leave a formal treatment and empirical evaluation to future work.
Q2. Could the authors compare RARETRAP with the rare-event estimation methods discussed in the related work, such as Wu & Hilton (2025), Cao et al. (2026), and Liu et al. (2026), with matched target-model evaluation budgets? How do these methods compare in estimation accuracy and efficiency? If any work cannot be directly applied in this black-box setting, can the authors explain why?
A2. These methods estimate rare-event probabilities but differ in their target events and access requirements. Wu & Hilton (2025) and Cao et al. (2026) estimate the probability of a specific next-token prediction, while Liu et al. (2026) estimate the probability of unsafe generation under stochastic decoding for a fixed prompt. None estimates the probability of a response-level behavior across a distribution of prompts. Moreover, these methods require access to model internals or token-level probabilities and therefore are not black-box. This is why matched-budget comparison with these methods is consequently not possible. Direct Monte Carlo (MC) remains the only unbiased, comparable black-box estimator.
Q3. The paper acknowledges that ties in discrete severity metrics can bias the conditional probability estimates. How prevalent are such ties in the main experiments, and to what extent might they affect the reported probabilities, particularly for deeper conditional levels?
A3. Ties at the intermediate threshold allow more than the nominal 100 of 1,000 samples to pass, which can bias the estimated failure probability downward. This effect is limited in 20 of the 22 analyzed open-weight multi-level runs, showing 99–109 samples pass at each level. This means the resulting changes in probability estimates are at most 9%. The two exceptions are Qwen3-14B with its own surrogate, where 132 and 155 samples pass. The surrogate mapping produces many duplicate prompts with identical severity scores, resulting in repeated thresholds and a larger bias. This issue is also illustrated in Table 7(b) (Appendix E) for Qwen3-14B at D=100, where ties prevent consistent threshold progression.