- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents ALE, an approach to per-prompt rare-failure estimation that extrapolates a high-failure model past the worse-failing of two base models, then extrapolates a failure rate linearly back to the base models. Experiments show competitive performance by ALE on a CWEval-based rare-failure estimation task relative to prior state of the art.
Critical strengths
The fundamental idea of sampling failures by extrapolating the policy difference between a more-failing policy and a less-failing one, and then extrapolating the failure rate back under an assumption of linearity, is an interesting one. The technique used to estimate the slope of the line using a single batch estimate of the failure rate is also clever. I believe this could be developed into a strong paper with clearer writing and more extensive experiments.
Critical weaknesses
The paper is extremely difficult to read. Particularly in the 4.5 pages used to explain the basic idea of the method, terms are introduced without explanation (for example, it takes several minutes to determine what the “unlabeled scan” in Figure 2 is referring to), in great complexity (see Table 2), and often with no intuitive motivation (for example, the notion of rollout generation being cheaper than failure labeling is fine, but this is not a standard assumption in the field, and it is not intuitively justified in this paper as far as I can find). While reading, the information contained in the next sentence nearly never answered the question I had in my head at that point. To be blunt, the paper reads like it was written by an LLM for LLMs, not by a human for humans, and as a human it is extremely hard to extract information from. There may be other weaknesses relating to the content of the paper, but it is very hard to tell due to the presentation style.
The results are presented strangely and are not particularly convincing. The results shown are not inherently better than the one baseline that is compared to, and furthermore, they do not match sample/label counts between the two methods, so direct comparisons are impossible. It is also not obvious why the setting of this paper is a fair comparison with prior work, which does not assume access to a model that is known to fail more often (if such access were known, the sampling methods in prior work would likely be adjusted). Thus, this should be a highly favorable comparison for the present method, and it still fails to outperform prior art convincingly. Furthermore, Table 2 is just a data dump (why do we care that targets are labeled cwe_329_0 in the code? What does that mean? If the target labels don’t matter, then shouldn’t this be presented as a figure?), with no interpretive assistance provided. Even if all of these issues were fixed, the results would still be limited, e.g., to this single task and single model (Qwen3-8B). Also, the basic baseline of Monte Carlo does not seem to be presented, but should be, as it looks like some failures are common enough that it could be competitive.
Questions for authors
In what regard is the comparison with Angell et al. (2026) fair, given the additional assumption of access to a second model that is known to fail more often?
Thank you for the review and the kind words on the slope estimator.
On your question, the adjustment you anticipate is our CE-IS baseline. In a controlled comparison, it applies Angell et al.'s proposal selection to ALE's checkpoint path, so both methods have identical model access (Sec. 5). Pernice & Aranguri (LessWrong 2026) likewise assume a less-safe second model (Sec. 1). Neither of ALE's checkpoints needs frequent failures, since amplification supplies them (1.8 x 10^-4 and 2.8 x 10^-5 in Fig. 3). Table 1 reports labelled and total rollouts for every estimator. ALE uses fewer of both than CE-IS.
Every estimate is compared against a direct Monte Carlo count (Fig. 4 and column P_MC of Table 2). At ALE's label budget, direct Monte Carlo finds no failure in the running example (Fig. 3) or for 30 of 49 targets in the broader paired evaluation (Sec. 5, Ablations, and Fig. 13). Table 2 is ordered by coverage. In its three lowest-coverage rows, from two tasks, ALE-SE is closest to the direct count at every target, while IS misses by up to 77x. At high coverage the order reverses (Sec. 5). Of the 41 targets, 29 are short expressions outside CWEval and 12 come from five CWEval tasks, all on Qwen3-8B (Table 1, Sec. 6). The next version will define the unlabeled scan where Fig. 2 first uses it. Thank you for your time and suggestions.