- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies run-time failure prediction for LLM-based multi-agent systems: given a prefix of an agent's trajectory, predict if the run will fail. Dream2Discern scores each prefix with a frozen LLM with LoRA adapters on its query/value projections and an MLP head. Training has two alternating steps, Dream and Discern. Dream makes one shared, norm-bounded, rank-r perturbation of the adapted query and value weights from a normalized gradient-ascent step on the batch's loss, plus SVD truncation and a step-halving acceptance test; Proposition 1 gives the closed-form linearized maximizer. Discern updates the adapters and head from the perturbed forward pass with the original labels; Proposition 2 ties this loss to the clean one plus a gradient-norm term. At test time, the auditor leaves out the perturbation and uses a calibrated threshold at each step. On AFTRAJ-2K and StepShield, the authors compare it with open and closed LLM judges, token-level uncertainty methods and adapted failure auditors SAFE, Hide-and-Seek, TRACES and AgentForesight, with a replication on Llama, a perturbation-strategy ablation, a radius sweep and two mechanism analyses.
Critical strengths
- The auditor does well in absolute terms, beating the adapted failure auditors on both benchmarks and backbone families, by a large margin on StepShield. On StepShield, Dream2Discern with Qwen3-4B gets 90.28 wACC against 79.17 for AgentForesight, and the wACC ordering holds on both Llama backbones. Along with the prefix-level, step-calibrated protocol and the accuracy-timeliness evaluation, it makes a good baseline for future learned runtime monitors. It counts for little in my rating, since Table 3 suggests the margin is due to LoRA fine-tuning of the LLM encoder, not to the proposed method.
- Dream has some effect of its own. On StepShield with Qwen3-4B, Dream training beats both the unperturbed and the randomly perturbed auditor by more than 5 wACC, and it has the highest AUROC in all four ablation cells. A Dream perturbation selected on training prefixes also raises held-out loss with the prediction head fixed, so the hard variants it finds are not specific to the batch that chose them.
Critical weaknesses
- The paper's ablation itself contradicts its claim that learning from hard latent variants is what drives the result. In Table 3, in all four cells, the unperturbed LoRA auditor already beats each failure-auditor baseline, and on AFTRAJ-2K Qwen3-4B it scores marginally above Dream (90.96 vs 90.36 wACC), while on StepShield Qwen3-8B, random perturbation scores above Dream; just one of four cells has a clear gain for Dream.
- The Table 1 margin, which the paper treats as evidence for hard latent variants, appears to compare LoRA fine-tuning with the frozen-encoder TRACES, LSTM, and GRU scorers. This is consequential because it's evidence already in the paper, not a missing run: further baselines couldn't make the hard variants necessary for the headline results.
- As far as I can tell, the Dream step is a sharpness-aware update with a rank constraint, not an augmentation. One normalized ascent step on the weights, followed by a descent step using the gradient at the perturbed point, is the Sharpness-Aware Minimization (SAM) update proposed by Foret et al., here with SVD truncation, a relative radius and backtracking, and Proposition 2 expresses the Discern loss as the clean loss with an added gradient-norm term.
- Because the perturbation is shared across the batch, it selects a sharp weight direction, rather than hard prefixes. Yet no gradient-norm or SAM baseline on the same adapters is reported, so whatever small effect Dream has is inseparable from that of a known regularizer.
Questions for authors
- What does a LoRA classifier with identical adapters, head, warm start and budget, but no perturbation score across three seeds on both benchmarks? How large is the Dream effect over it with matched seeds?
- Over three seeds, how does Dream2Discern compare against SAM or a gradient-norm penalty on the same LoRA q/v adapters? What about random perturbation whose norm matches that of the accepted Dream perturbation?
- Were SAFE, Hide-and-Seek, and TRACES trained with the same decisive-step prefix selection? What are the Table 1 metrics on a group-disjoint AFTRAJ-2K split and on StepShield's realistic-prevalence test set?
- Is there a held-out criterion for choosing the radius gamma? How sensitive are the Table 1 numbers to the choice of gamma?
- Can the Section 3.1 comparison be repeated with injected or label-preserving augmentations and more than one seed? How does the Dream step differ from low-rank flatness perturbations for LoRA such as Bi-LoRA and "Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape", and why isn't PrefixGuard a baseline?
7 comments
[2/4] Rebuttal by Authors
W3 & W4 (Q2): Relation to SAM and comparison on the same adapters
R3: Thanks for the detailed technical analysis. While Dream and Discern follow the two-step ascent and descent procedure of SAM, we would like to reclaim that our work is novel in introducing a new perspective for runtime failure prediction, i.e., learning from hard latent variants of recorded prefixes. The key difference from SAM lies in what the ascent step maximizes over and what it is used for, as clarified below.
Conceptually, Dream uses weight perturbation to construct hard variants of the training prefixes, whereas SAM uses weight perturbation to find flat solutions that generalize better. Our motivation is that new labelled trajectories require costly rollouts and verification, and editing a recorded trajectory can change its outcome label (Section 3.1). To this end, Dream keeps the recorded prefixes and labels unchanged and constructs their hard variants in the representation space. This is why we call Dream an implicit trajectory augmentation. In contrast, SAM is motivated by the geometry of the loss landscape and perturbs all trainable parameters to measure the sharpness of the loss around the current solution. Dream therefore differs from SAM in both the motivation and the role of the perturbation.
Technically, Dream differs from SAM in three aspects.
① Representation-only perturbation. Dream perturbs only the effective q/v projection weights under a fixed classifier (Section 3.2), whereas SAM perturbs all trainable parameters, including the LoRA factors and the classifier. Based on this, the prediction loss under Dream can rise only through a change in the representation of the same prefix, while a median of 30% (Qwen3-4B) and 46% (Qwen3-8B) of the first-order loss increase under SAM comes from moving the classifier.
② Operator-space perturbation. Dream perturbs the effective projection weights directly, so the perturbed weights can deviate from the frozen weights by a matrix of rank up to 16. SAM perturbs the rank-8 LoRA factors, so this deviation stays at rank at most 8 under SAM.
③ A different first-order term. As you point out, Proposition 2 has the form of a gradient-norm penalty, and SVD truncation and backtracking do not change this form. The penalized quantity differs from that of SAM. Dream penalizes , a norm of the loss gradients with respect to the projection weights, and contains no classifier gradient. SAM penalizes the norm of the gradients with respect to the LoRA factors and the classifier, and the gradients of the LoRA factors contain only through the products and . SAM therefore penalizes the classifier gradient in addition and does not penalize the part of outside the subspaces of the LoRA factors, so the two terms are not proportional.
Regarding the shared perturbation (W4), we would like to clarify that we do not claim that Dream selects real hard prefixes. Section 3.2 defines hardness by the batch loss under one shared perturbation, following the model-wise perspective of AWP (Appendix B.1), and this perturbation changes the representation of each prefix through the token states of that prefix (Eq. 5). A sharp weight direction and the hard latent variants of a batch are thus two descriptions of the same perturbation, and the distinction from SAM lies in the three aspects above.
Empirically (Q2), following your suggestion, we apply SAM, which is the gradient-norm penalty of Zhao et al. (ICML 2022) with α=1, and AWP to the same LoRA adapters and classifier as Dream2Discern on AFTraj-2K, with the settings of their original papers (ρ=0.05 for SAM, =5×10⁻³ for AWP). Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).
| Method | AUROC | wACC | TWA |
|---|---|---|---|
| AWP | 94.56±0.41 / 94.54±0.27 | 90.06±0.52 / 90.46±0.92 | 71.48±0.37 / 70.74±0.63 |
| SAM | 95.03±0.52 / 94.82±0.36 | 90.26±1.14 / 90.56±0.17 | 71.76±0.40 / 71.34±1.24 |
| Dream2Discern | 95.45±0.16 / 95.82±0.43 | 90.96±0.60 / 91.47±0.17 | 72.61±0.38 / 73.45±0.71 |
Dream2Discern consistently achieves better performance than both SAM and AWP on both backbones. Regarding the random perturbation, its norm in Table 3 equals at every step, which is never smaller than the norm of the accepted Dream perturbation. We have provided detailed analysis in the response of W1 (Q1).
We have added the comparison with SAM and AWP to Appendix E.3, this discussion to Section 3.2 and Appendix B.1, and SAM and the gradient-norm penalty to the related work in Appendix A.
[3/4] Rebuttal by Authors
Q3: Baseline prefixes, group-disjoint split and realistic prevalence
R4: Thanks for these questions.
Prefix selection. SAFE and TRACES are trained on the same training prefixes as Dream2Discern (Appendix D.2), while H&S follows its original objective on complete training trajectories with trajectory-level labels (Appendix D.4). The decisive-step annotations only select training prefixes and are never part of any auditor's input.
Group-disjoint AFTraj-2K. Following your suggestion, we re-evaluate the auditors of Table 1 on the 306 test trajectories whose parent groups do not occur in training. Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).
| Auditor | AUROC | FPR95 | wACC | TWA |
|---|---|---|---|---|
| SAFE | 94.91±0.29 / 93.19±0.89 | 40.86±3.25 / 55.48±11.19 | 84.97±2.04 / 85.84±1.36 | 69.39±1.11 / 66.59±0.57 |
| H&S | 91.37±1.28 / 90.32±0.64 | 72.69±13.60 / 90.97±3.87 | 89.65±1.15 / 89.22±1.31 | 70.98±0.71 / 70.42±1.26 |
| TRACES | 92.45±1.17 / 93.24±1.04 | 60.00±16.05 / 64.52±11.63 | 89.65±0.68 / 88.24±1.42 | 71.34±0.89 / 70.81±0.29 |
| Dream2Discern | 95.48±0.21 / 96.60±0.34 | 38.49±7.70 / 27.96±3.31 | 91.61±0.19 / 92.05±0.38 | 73.09±0.29 / 73.78±0.34 |
Dream2Discern still outperforms SAFE, H&S and TRACES on all four metrics on both backbones, and each metric is at least as good as in Table 1.
Realistic prevalence on StepShield. We would like to clarify that the 8.1%-prevalence set of StepShield is not an independent test set for trained auditors, since its 639 rogue and 639 matched clean records are copies of the StepShield training trajectories, as documented in the StepShield release. Table 1 uses the official held-out test set of StepShield (216 trajectories), on which StepShield itself reports its main results.
We have added the group-disjoint results to Appendix E.5.
Q4: Selection and sensitivity of the radius
R5: Thanks for these questions on the radius.
First, we would like to clarify that no held-out split was used to choose , since both benchmarks provide a training and a test partition only. We have explained the criterion for choosing in Appendix D.3, where is selected for each backbone according to the effect of the Dream step on training batches, so the radii are not obtained by tuning on the test sets.
Regarding the sensitivity, Table 11 varies from 10⁻⁴ to 10⁻², and StepShield is more sensitive to than AFTraj-2K. However, the comparison with trained auditors does not depend on the selected radius, since every between 10⁻⁴ and 3×10⁻³ yields a higher wACC than SAFE, H&S and TRACES on the same backbone on both benchmarks. To remove the per-backbone choice, we further fix =10⁻³ for both backbones and train three seeds on AFTraj-2K. Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).
| Method | AUROC | FPR95 | wACC | TWA |
|---|---|---|---|---|
| w/o Perturbation | 95.58±0.56 / 94.03±0.37 | 36.69±5.33 / 54.24±6.93 | 90.66±0.52 / 89.96±1.06 | 70.69±2.07 / 69.50±1.49 |
| Dream (=10⁻³) | 96.78±0.71 / 96.36±0.88 | 26.63±6.51 / 30.77±8.35 | 91.57±0.52 / 91.16±0.46 | 73.35±0.63 / 73.21±0.77 |
With this single radius, Dream improves all four metrics over the unperturbed auditor on AFTraj-2K with both backbones, and its AUROC, wACC and TWA are also higher than those of SAM and AWP in our response to W3 & W4 (Q2), which demonstrates that this improvement does not rely on the per-backbone selection of .
We have added these single-radius results to Appendix E.4.
[1/4] Rebuttal by Authors
Thank you for your detailed and careful review. Here are our detailed replies.
R1: Thanks for raising this central point. We would like to first clarify that the comparison quoted from Table 3 rests on a single training seed. Following your suggestion in Q1, we train the unperturbed LoRA auditor with identical adapters, classifier, warm start and budget on the same three seeds as Dream2Discern, together with the random perturbation. Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).
With three seeds, Dream achieves the highest wACC and TWA in three of the four settings, improving over the unperturbed auditor by up to 3.86 wACC and 3.95 TWA. Random perturbations of the same operators at the same radius fall below the unperturbed auditor in wACC and TWA in all four settings, which shows that the improvement of Dream comes from its loss-guided search. The only setting without improvement is StepShield with Qwen3-8B. We have clairified in Section 4.1 that the StepShield test trajectories come from incident templates never used in training, so the radius selected on training batches (Appendix D.3) is not necessarily the best one on this test set. The clear improvements in both wACC and TWA on the AFTraj-2K with both backbones demonstrate the effectiveness of Dream.
Regarding the effect with matched seeds, both auditors start from the same warm start and use the same seeds, so we pair them by seed within each backbone and report the difference between Dream2Discern and the unperturbed auditor trained on the same backbone, including the two Llama backbones of Table 2 (mean±std of the three paired differences).
Dream improves both wACC and TWA over the unperturbed auditor in seven of the eight settings, and the improvement in TWA on AFTraj-2K holds for all four backbones.
We have updated Table 3 in Section 4.3 with the three-seed results and added the paired comparison to Appendix E.3.
R2: Thank you for this comment. We would like to clarify the roles of Table 1 and Table 3 in the following two aspects.
① Comparison in Table 1. Table 1 compares complete runtime auditors. SAFE, H&S and TRACES follow their original designs, in which a sequence scorer is trained on frozen features of the same backbones, while the LoRA prefix auditor is a component of Dream2Discern (Section 3.2). The margin in Table 1 therefore combines the gain of the LoRA prefix auditor and the gain of Dream.
② Contribution of Dream. The contribution of Dream is measured in Section 4.3, where all variants share the same LoRA adapters, classifier, warm start and training budget. With three seeds, Dream further improves wACC and TWA over the unperturbed auditor in seven of the eight settings (please refer to the response of W1 (Q1)). Our best wACC values on AFTraj-2K (91.47 with Qwen3-8B) and on StepShield (90.28 with Qwen3-4B) are 1.51 and 3.86 points above the unperturbed auditor of the same backbone. These gains over an otherwise identical auditor demonstrate that the hard latent variants constructed by Dream are necessary for the headline results.
We have revised the discussion of Table 1 in Section 4.2, where the margin over SAFE, H&S and TRACES is now attributed to the complete auditor.