- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
ALERT is a white-box monitor that predicts long-context retrieval failures before the first answer token is generated. A supervised linear ranker uses attention at the final prompt position to select one earlier message. A second probe then predicts answer failure from that message's attention strength, its distribution across layers, and its competition with the question and the rest of the history. Both probes are fitted separately for each LLM and benchmark, with the LLM kept frozen. A threshold calibrated for at least 95% failure recall determines whether to repeat an excerpt of at most 64 tokens immediately before the unchanged question.
The evaluation tests ten open-weight models on NoLiMa, RULER, BABILong, and Scatter-MCQ adapted into multi-turn histories of 8K, 16K, and 32K tokens. Detection works best on targeted retrieval, with pooled AUROC of 0.970 on NoLiMa and 0.955 on RULER for Gemma-4-E4B in Table 1. Results are weaker on BABILong and Scatter-MCQ. For example, Qwen3-4B reaches BABILong AUROC of 0.786, 0.805, and 0.831 at 8K, 16K, and 32K in Table 2. Selective re-injection brings sizeable accuracy gains in Table 3, raising aggregate BABILong accuracy by 14.8–18.5 points and Scatter-MCQ accuracy by 5.7–9.5 points for the three intervention models, although it is triggered on 71.3–98.2% of prompts on those tasks.
Critical strengths
-
Distinguishing source localization from failure prediction and intervention makes the results much easier to interpret. Finding the supporting message does not necessarily mean the model will answer correctly. In the motivating example in Figure 4, Gemma-4-E4B gives that message 15.7% of its layer-35 attention at 16K but still produces the wrong location. Reporting source recall@1 alongside AUROC and false-positive rate in Tables 1 and 2, then accuracy changes and intervention rates in Table 3, helps distinguish errors in finding the source, assessing answer risk, and acting on the warning.
-
The fitting protocol takes several plausible sources of leakage seriously. Training the failure detector on out-of-fold ranked sources, as described in Section 4, exposes it to realistic localization errors rather than only ideal source selections. The alert threshold is chosen on a disjoint calibration set and then frozen. Section 5 further describes benchmark-specific partitions that hold out facts, relation families, background contexts, or question identities. These are appropriate safeguards for a supervised monitor, and the paper explains them clearly.
-
Several detection results are strong across the range of ten models and four tasks, and the intervention offers a simple way to recover some failures. Among the Gemma models, pooled AUROC in Table 1 ranges from 0.939–0.970 on NoLiMa and 0.941–0.955 on RULER. Qwen3-4B also remains relatively stable on BABILong as context length increases in Table 2. The sizeable paired BABILong accuracy gains for Gemma-4-E4B, Qwen3-4B, and Llama-3.2-11B in Table 3 are encouraging, particularly because re-injection preserves the original history and repeats only one short excerpt.
Critical weaknesses
-
The results do not yet isolate how much predictive value comes from routing to the selected source rather than from simpler confidence or attention features. The direct test of the central claim about information beyond next-token confidence uses only one 576-story BABILong cohort at 16K. ALERT reaches AUROC 0.710 for Gemma-4-E4B and 0.759 for Qwen3-4B there, compared with 0.518 and 0.517 for raw next-token margin. That is encouraging, but it covers only one benchmark, one context length, and one confidence feature. Tables 1 and 2 do not compare confidence-only, confidence-plus-routing, or source-agnostic attention baselines on the same splits, so the strong detection scores alone cannot establish the specific contribution of source routing.
-
The warning policy is close to always re-injecting in the settings where the intervention helps most, leaving its selective advantage unproven. Intervention rates in Table 3 are 71.3–91.5% on BABILong and 97.5–98.2% on Scatter-MCQ. The Introduction gives 43.6% accuracy for selective re-injection with Gemma-4-E4B on the 576-story cohort, compared with 44.4% for always re-injecting, and the abstract states that neither BABILong policy exceeds always re-injection accuracy. Although Section 4 describes always-on, confidence-triggered, lexical, random, and gold-source controls, their results are not shown in the main paper.
-
The broad detection study should not be taken as evidence that one fitted monitor is broadly reusable, or that intervention works across the same range of models. Detection is evaluated for ten models, whereas Table 3 tests re-injection for only three. Moreover, each ranker and detector is fitted separately for every model and benchmark, and the paper does not test transfer across models or tasks. Some model and task combinations in Tables 1 and 2 also have undefined policies or very high false-positive rates at the required recall. These limitations matter when judging how widely the reported gains would apply.
Questions for authors
-
Could you compare next-token confidence features, source-agnostic attention statistics, ALERT routing features, and confidence combined with routing across all four benchmarks? Please use the same training, calibration, and test partitions for every variant.
-
Could you report the always-on, confidence-triggered, lexical, length-matched random, and gold-source re-injection controls described in Section 4, together with rescue and harm counts? I would like to see how much benefit comes from the warning policy rather than from repeating any plausible context.