acceptodds
Under review as a conference paper at ICLR 2027

Dream2Discern: Learning from Hard Latent Variants for Runtime Failure Prediction in Multi-Agent Systems

29 views

Abstract

Reliable deployment of LLM-based multi-agent systems calls for runtime auditors that predict terminal outcomes from observed execution prefixes. These auditors typically learn from a limited corpus of trajectories with verified outcomes. Expanding this corpus requires costly execution and verification, yet broader failure coverage alone provides no explicit criterion for identifying prefixes that the current auditor finds difficult to classify. To this end, we propose Dream2Discern, a runtime auditor trained through implicit augmentation guided by its own prediction loss. During Dream, we select perturbations of the auditor’s representation operators to increase prediction loss on recorded prefixes, yielding hard latent variants. During Discern, the auditor learns to predict execution outcomes from these variants using the original labels. The variants are regenerated as the auditor learns, turning a limited corpus into a renewable source of hard examples and requiring no additional trajectory collection or outcome verification. Extensive evaluations on AFTRAJ-2K and StepShield show that Dream2Discern outperforms competitive failure auditors and achieves over 90% trajectory-level weighted accuracy on both benchmarks while balancing prediction accuracy and timeliness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Runtime Failure Prediction, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7MqtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies run-time failure prediction for LLM-based multi-agent systems: given a prefix of an agent's trajectory, predict if the run will fail. Dream2Discern scores each prefix with a frozen LLM with LoRA adapters on its query/value projections and an MLP head. Training has two alternating steps, Dream and Discern. Dream makes one shared, norm-bounded, rank-r perturbation of the adapted query and value weights from a normalized gradient-ascent step on the batch's loss, plus SVD truncation and a step-halving acceptance test; Proposition 1 gives the closed-form linearized maximizer. Discern updates the adapters and head from the perturbed forward pass with the original labels; Proposition 2 ties this loss to the clean one plus a gradient-norm term. At test time, the auditor leaves out the perturbation and uses a calibrated threshold at each step. On AFTRAJ-2K and StepShield, the authors compare it with open and closed LLM judges, token-level uncertainty methods and adapted failure auditors SAFE, Hide-and-Seek, TRACES and AgentForesight, with a replication on Llama, a perturbation-strategy ablation, a radius sweep and two mechanism analyses.

Critical strengths
  • The auditor does well in absolute terms, beating the adapted failure auditors on both benchmarks and backbone families, by a large margin on StepShield. On StepShield, Dream2Discern with Qwen3-4B gets 90.28 wACC against 79.17 for AgentForesight, and the wACC ordering holds on both Llama backbones. Along with the prefix-level, step-calibrated protocol and the accuracy-timeliness evaluation, it makes a good baseline for future learned runtime monitors. It counts for little in my rating, since Table 3 suggests the margin is due to LoRA fine-tuning of the LLM encoder, not to the proposed method.
  • Dream has some effect of its own. On StepShield with Qwen3-4B, Dream training beats both the unperturbed and the randomly perturbed auditor by more than 5 wACC, and it has the highest AUROC in all four ablation cells. A Dream perturbation selected on training prefixes also raises held-out loss with the prediction head fixed, so the hard variants it finds are not specific to the batch that chose them.
Critical weaknesses
  • The paper's ablation itself contradicts its claim that learning from hard latent variants is what drives the result. In Table 3, in all four cells, the unperturbed LoRA auditor already beats each failure-auditor baseline, and on AFTRAJ-2K Qwen3-4B it scores marginally above Dream (90.96 vs 90.36 wACC), while on StepShield Qwen3-8B, random perturbation scores above Dream; just one of four cells has a clear gain for Dream.
  • The Table 1 margin, which the paper treats as evidence for hard latent variants, appears to compare LoRA fine-tuning with the frozen-encoder TRACES, LSTM, and GRU scorers. This is consequential because it's evidence already in the paper, not a missing run: further baselines couldn't make the hard variants necessary for the headline results.
  • As far as I can tell, the Dream step is a sharpness-aware update with a rank constraint, not an augmentation. One normalized ascent step on the weights, followed by a descent step using the gradient at the perturbed point, is the Sharpness-Aware Minimization (SAM) update proposed by Foret et al., here with SVD truncation, a relative radius and backtracking, and Proposition 2 expresses the Discern loss as the clean loss with an added gradient-norm term.
  • Because the perturbation is shared across the batch, it selects a sharp weight direction, rather than hard prefixes. Yet no gradient-norm or SAM baseline on the same adapters is reported, so whatever small effect Dream has is inseparable from that of a known regularizer.
Questions for authors
  1. What does a LoRA classifier with identical adapters, head, warm start and budget, but no perturbation score across three seeds on both benchmarks? How large is the Dream effect over it with matched seeds?
  2. Over three seeds, how does Dream2Discern compare against SAM or a gradient-norm penalty on the same LoRA q/v adapters? What about random perturbation whose norm matches that of the accepted Dream perturbation?
  3. Were SAFE, Hide-and-Seek, and TRACES trained with the same decisive-step prefix selection? What are the Table 1 metrics on a group-disjoint AFTRAJ-2K split and on StepShield's realistic-prevalence test set?
  4. Is there a held-out criterion for choosing the radius gamma? How sensitive are the Table 1 numbers to the choice of gamma?
  5. Can the Section 3.1 comparison be repeated with injected or label-preserving augmentations and more than one seed? How does the Dream step differ from low-rank flatness perturbations for LoRA such as Bi-LoRA and "Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape", and why isn't PrefixGuard a baseline?
7 comments
AuthorsOpenReview ↗

[1/4] Rebuttal by Authors

Thank you for your detailed and careful review. Here are our detailed replies.

W1 (Q1): Contribution of Dream over the unperturbed LoRA auditor

R1: Thanks for raising this central point. We would like to first clarify that the comparison quoted from Table 3 rests on a single training seed. Following your suggestion in Q1, we train the unperturbed LoRA auditor with identical adapters, classifier, warm start and budget on the same three seeds as Dream2Discern, together with the random perturbation. Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

Pert. strategyAFTraj-2K wACCAFTraj-2K TWAStepShield wACCStepShield TWA
w/o Perturbation90.66±0.52 / 89.96±1.0670.69±2.07 / 69.50±1.4986.42±1.87 / 88.58±1.4954.79±1.62 / 57.04±1.42
w/ Random Pert.89.36±0.63 / 89.46±0.9069.30±1.23 / 69.15±1.7585.96±2.19 / 88.43±1.6754.27±1.29 / 56.57±1.26
w/ Dream Pert.90.96±0.60 / 91.47±0.1772.61±0.38 / 73.45±0.7190.28±0.46 / 87.50±0.9357.67±0.35 / 55.69±1.10

With three seeds, Dream achieves the highest wACC and TWA in three of the four settings, improving over the unperturbed auditor by up to 3.86 wACC and 3.95 TWA. Random perturbations of the same operators at the same radius fall below the unperturbed auditor in wACC and TWA in all four settings, which shows that the improvement of Dream comes from its loss-guided search. The only setting without improvement is StepShield with Qwen3-8B. We have clairified in Section 4.1 that the StepShield test trajectories come from incident templates never used in training, so the radius selected on training batches (Appendix D.3) is not necessarily the best one on this test set. The clear improvements in both wACC and TWA on the AFTraj-2K with both backbones demonstrate the effectiveness of Dream.

Regarding the effect with matched seeds, both auditors start from the same warm start and use the same seeds, so we pair them by seed within each backbone and report the difference between Dream2Discern and the unperturbed auditor trained on the same backbone, including the two Llama backbones of Table 2 (mean±std of the three paired differences).

Dream − w/o PerturbationQwen3-4BQwen3-8BLlama-3.2-3BLlama-3.1-8B
AFTraj-2K wACC+0.30±1.09+1.51±1.20+2.01±0.87+1.20±0.30
AFTraj-2K TWA+1.92±2.42+3.95±1.12+4.99±1.13+2.48±0.64
StepShield wACC+3.86±1.63−1.08±1.63+15.90±5.52+2.16±4.30
StepShield TWA+2.88±1.35−1.35±0.63+8.94±3.19+0.62±6.35

Dream improves both wACC and TWA over the unperturbed auditor in seven of the eight settings, and the improvement in TWA on AFTraj-2K holds for all four backbones.

We have updated Table 3 in Section 4.3 with the three-seed results and added the paired comparison to Appendix E.3.

W2: Attribution of the margin in Table 1

R2: Thank you for this comment. We would like to clarify the roles of Table 1 and Table 3 in the following two aspects.

① Comparison in Table 1. Table 1 compares complete runtime auditors. SAFE, H&S and TRACES follow their original designs, in which a sequence scorer is trained on frozen features of the same backbones, while the LoRA prefix auditor is a component of Dream2Discern (Section 3.2). The margin in Table 1 therefore combines the gain of the LoRA prefix auditor and the gain of Dream.

② Contribution of Dream. The contribution of Dream is measured in Section 4.3, where all variants share the same LoRA adapters, classifier, warm start and training budget. With three seeds, Dream further improves wACC and TWA over the unperturbed auditor in seven of the eight settings (please refer to the response of W1 (Q1)). Our best wACC values on AFTraj-2K (91.47 with Qwen3-8B) and on StepShield (90.28 with Qwen3-4B) are 1.51 and 3.86 points above the unperturbed auditor of the same backbone. These gains over an otherwise identical auditor demonstrate that the hard latent variants constructed by Dream are necessary for the headline results.

We have revised the discussion of Table 1 in Section 4.2, where the margin over SAFE, H&S and TRACES is now attributed to the complete auditor.

AuthorsOpenReview ↗

[2/4] Rebuttal by Authors

W3 & W4 (Q2): Relation to SAM and comparison on the same adapters

R3: Thanks for the detailed technical analysis. While Dream and Discern follow the two-step ascent and descent procedure of SAM, we would like to reclaim that our work is novel in introducing a new perspective for runtime failure prediction, i.e., learning from hard latent variants of recorded prefixes. The key difference from SAM lies in what the ascent step maximizes over and what it is used for, as clarified below.

Conceptually, Dream uses weight perturbation to construct hard variants of the training prefixes, whereas SAM uses weight perturbation to find flat solutions that generalize better. Our motivation is that new labelled trajectories require costly rollouts and verification, and editing a recorded trajectory can change its outcome label (Section 3.1). To this end, Dream keeps the recorded prefixes and labels unchanged and constructs their hard variants in the representation space. This is why we call Dream an implicit trajectory augmentation. In contrast, SAM is motivated by the geometry of the loss landscape and perturbs all trainable parameters to measure the sharpness of the loss around the current solution. Dream therefore differs from SAM in both the motivation and the role of the perturbation.

Technically, Dream differs from SAM in three aspects.

① Representation-only perturbation. Dream perturbs only the effective q/v projection weights under a fixed classifier (Section 3.2), whereas SAM perturbs all trainable parameters, including the LoRA factors and the classifier. Based on this, the prediction loss under Dream can rise only through a change in the representation of the same prefix, while a median of 30% (Qwen3-4B) and 46% (Qwen3-8B) of the first-order loss increase under SAM comes from moving the classifier.

② Operator-space perturbation. Dream perturbs the effective projection weights directly, so the perturbed weights can deviate from the frozen weights by a matrix of rank up to 16. SAM perturbs the rank-8 LoRA factors, so this deviation stays at rank at most 8 under SAM.

③ A different first-order term. As you point out, Proposition 2 has the form of a gradient-norm penalty, and SVD truncation and backtracking do not change this form. The penalized quantity differs from that of SAM. Dream penalizes grg_r, a norm of the loss gradients GjG_j with respect to the projection weights, and grg_r contains no classifier gradient. SAM penalizes the norm of the gradients with respect to the LoRA factors and the classifier, and the gradients of the LoRA factors contain GjG_j only through the products GjAj⊤G_jA_j^\top and Bj⊤GjB_j^\top G_j. SAM therefore penalizes the classifier gradient in addition and does not penalize the part of GjG_j outside the subspaces of the LoRA factors, so the two terms are not proportional.

Regarding the shared perturbation (W4), we would like to clarify that we do not claim that Dream selects real hard prefixes. Section 3.2 defines hardness by the batch loss under one shared perturbation, following the model-wise perspective of AWP (Appendix B.1), and this perturbation changes the representation of each prefix through the token states of that prefix (Eq. 5). A sharp weight direction and the hard latent variants of a batch are thus two descriptions of the same perturbation, and the distinction from SAM lies in the three aspects above.

Empirically (Q2), following your suggestion, we apply SAM, which is the gradient-norm penalty of Zhao et al. (ICML 2022) with α=1, and AWP to the same LoRA adapters and classifier as Dream2Discern on AFTraj-2K, with the settings of their original papers (ρ=0.05 for SAM, γ\gamma=5×10⁻³ for AWP). Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

MethodAUROCwACCTWA
AWP94.56±0.41 / 94.54±0.2790.06±0.52 / 90.46±0.9271.48±0.37 / 70.74±0.63
SAM95.03±0.52 / 94.82±0.3690.26±1.14 / 90.56±0.1771.76±0.40 / 71.34±1.24
Dream2Discern95.45±0.16 / 95.82±0.4390.96±0.60 / 91.47±0.1772.61±0.38 / 73.45±0.71

Dream2Discern consistently achieves better performance than both SAM and AWP on both backbones. Regarding the random perturbation, its norm in Table 3 equals γ\gamma at every step, which is never smaller than the norm of the accepted Dream perturbation. We have provided detailed analysis in the response of W1 (Q1).

We have added the comparison with SAM and AWP to Appendix E.3, this discussion to Section 3.2 and Appendix B.1, and SAM and the gradient-norm penalty to the related work in Appendix A.

AuthorsOpenReview ↗

[3/4] Rebuttal by Authors

Q3: Baseline prefixes, group-disjoint split and realistic prevalence

R4: Thanks for these questions.

Prefix selection. SAFE and TRACES are trained on the same training prefixes as Dream2Discern (Appendix D.2), while H&S follows its original objective on complete training trajectories with trajectory-level labels (Appendix D.4). The decisive-step annotations only select training prefixes and are never part of any auditor's input.

Group-disjoint AFTraj-2K. Following your suggestion, we re-evaluate the auditors of Table 1 on the 306 test trajectories whose parent groups do not occur in training. Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

AuditorAUROCFPR95wACCTWA
SAFE94.91±0.29 / 93.19±0.8940.86±3.25 / 55.48±11.1984.97±2.04 / 85.84±1.3669.39±1.11 / 66.59±0.57
H&S91.37±1.28 / 90.32±0.6472.69±13.60 / 90.97±3.8789.65±1.15 / 89.22±1.3170.98±0.71 / 70.42±1.26
TRACES92.45±1.17 / 93.24±1.0460.00±16.05 / 64.52±11.6389.65±0.68 / 88.24±1.4271.34±0.89 / 70.81±0.29
Dream2Discern95.48±0.21 / 96.60±0.3438.49±7.70 / 27.96±3.3191.61±0.19 / 92.05±0.3873.09±0.29 / 73.78±0.34

Dream2Discern still outperforms SAFE, H&S and TRACES on all four metrics on both backbones, and each metric is at least as good as in Table 1.

Realistic prevalence on StepShield. We would like to clarify that the 8.1%-prevalence set of StepShield is not an independent test set for trained auditors, since its 639 rogue and 639 matched clean records are copies of the StepShield training trajectories, as documented in the StepShield release. Table 1 uses the official held-out test set of StepShield (216 trajectories), on which StepShield itself reports its main results.

We have added the group-disjoint results to Appendix E.5.

Q4: Selection and sensitivity of the radius γ\gamma

R5: Thanks for these questions on the radius.

First, we would like to clarify that no held-out split was used to choose γ\gamma, since both benchmarks provide a training and a test partition only. We have explained the criterion for choosing γ\gamma in Appendix D.3, where γ\gamma is selected for each backbone according to the effect of the Dream step on training batches, so the radii are not obtained by tuning γ\gamma on the test sets.

Regarding the sensitivity, Table 11 varies γ\gamma from 10⁻⁴ to 10⁻², and StepShield is more sensitive to γ\gamma than AFTraj-2K. However, the comparison with trained auditors does not depend on the selected radius, since every γ\gamma between 10⁻⁴ and 3×10⁻³ yields a higher wACC than SAFE, H&S and TRACES on the same backbone on both benchmarks. To remove the per-backbone choice, we further fix γ\gamma=10⁻³ for both backbones and train three seeds on AFTraj-2K. Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

MethodAUROCFPR95wACCTWA
w/o Perturbation95.58±0.56 / 94.03±0.3736.69±5.33 / 54.24±6.9390.66±0.52 / 89.96±1.0670.69±2.07 / 69.50±1.49
Dream (γ\gamma=10⁻³)96.78±0.71 / 96.36±0.8826.63±6.51 / 30.77±8.3591.57±0.52 / 91.16±0.4673.35±0.63 / 73.21±0.77

With this single radius, Dream improves all four metrics over the unperturbed auditor on AFTraj-2K with both backbones, and its AUROC, wACC and TWA are also higher than those of SAM and AWP in our response to W3 & W4 (Q2), which demonstrates that this improvement does not rely on the per-backbone selection of γ\gamma.

We have added these single-radius results to Appendix E.4.

Reviewer_8TZYOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies runtime prediction of terminal failure from observed prefixes of LLM-based multi-agent executions. Dream2Discern alternates a Dream step, which searches bounded low-rank perturbations to LoRA-adapted query and value projections to increase batch binary cross-entropy with the head fixed, and a Discern step, which updates the adapters and head on the perturbed representations using original trajectory labels. At inference, perturbations are discarded and predictions use the unmodified prefix representation with step-dependent thresholds calibrated on successful training trajectories. Experiments use AFTRAJ-2K and StepShield with Qwen3-4B/8B and Llama-3.2-3B/Llama-3.1-8B backbones. Comparisons include prompted and uncertainty baselines, AgentForesight, SAFE, Hide-and-Seek, and TRACES. Table 1 reports 90.96 to 91.47% weighted accuracy on AFTRAJ-2K and 87.50 to 90.28% on StepShield for the Qwen backbones, averaged over three seeds. It also reports AUROC, FPR95, time-weighted accuracy, and detection-timing tradeoffs. Ablations compare loss-guided, random, and absent perturbations.

Critical strengths
  1. The method addresses a practical limitation of trajectory-level supervision: adding examples need not target the current auditor's errors. Shared operator perturbations create harder representations without changing recorded inputs or labels (Sections 3.2 and 3.3).

  2. Proposition 1 derives the closed-form solution to the linearized bounded low-rank search using truncated scaled gradients. Proposition 2 states the Taylor-remainder condition needed to interpret the accepted perturbation's loss increase (Section 3 and Appendix C).

  3. The empirical study includes two benchmarks, four backbones, recent learned auditor baselines, and three-seed reporting for the main trained systems (Tables 1 and 2). Dream2Discern improves weighted accuracy over these baselines in the reported Qwen and Llama comparisons.

  4. Table 3 compares learned, random, and absent perturbations, and Section 4.4 tests whether sensitivity tracks held-out errors. The limitations section distinguishes offline prediction from downstream intervention.

Critical weaknesses
  1. AFTRAJ-2K is not group-disjoint: Appendix D.1 reports 19 source-task parent groups shared across train and test, including 26 test trajectories. This may inflate results through task-family overlap. The leave-group-out check in Appendix E.1.2 applies only to a diagnostic. How does Table 1 change under a parent-group-disjoint split?

  2. The closed-source judge comparison uses different operating-point protocols. Appendix D.4 takes judges' returned labels for weighted and time-weighted accuracy, but calibrates the learned auditor on training successes at nominal α = 0.05. Thus, the claim that the 4B model exceeds Claude-Sonnet-4.6 on StepShield wACC and TWA is hard to interpret. Comparisons with trained auditor baselines do use the same protocol.

  3. Evidence for general runtime auditing is narrow: headline tests contain 332 AFTRAJ-2K and 216 StepShield trajectories (Appendix D.1), and auditors are trained separately per benchmark. StepShield's template holdout is useful, but there is no cross-benchmark transfer or closed-loop intervention experiment (Appendix F).

  4. Table 3 uses one seed, and the perturbation radii vary by backbone, selected using training-batch signed-margin reduction (Appendix D.3 and Table 6). Given the sensitivity in Table 8, it is unclear whether gains are robust to this rule or driven by radius selection.

Questions for authors
  1. Can you report Table 1 on a source-task-grouped AFTRAJ-2K split, including results for the 26 test trajectories whose groups occur in training?

  2. Can you calibrate closed-source judges on the same training successes, or limit comparison to AUROC and FPR95?

  3. Can you provide multi-seed results for the perturbation ablation and a radius-selection analysis that does not choose γ by training signed-margin reduction?

  4. How about the trade-off between the effectiveness and efficiency of the method?

5 comments
AuthorsOpenReview ↗

[1/2] Rebuttal by Authors

Thank you for your thorough review. Here are our detailed replies.

W1 (Q1): Parent-group overlap in AFTraj-2K

R1: Thanks for this careful check. First we would like to clarify that Table 1 follows the official train/test assignment of AFTraj-2K, and Appendix D.1 reports the 19 shared parent groups. Following your suggestion, we re-evaluate the auditors of Table 1 on the 306 test trajectories whose parent groups do not occur in training (first table) and on the 26 overlapping trajectories (second table, without FPR95 since only 14 of them are successful). Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

AuditorAUROCFPR95wACCTWA
SAFE94.91±0.29 / 93.19±0.8940.86±3.25 / 55.48±11.1984.97±2.04 / 85.84±1.3669.39±1.11 / 66.59±0.57
H&S91.37±1.28 / 90.32±0.6472.69±13.60 / 90.97±3.8789.65±1.15 / 89.22±1.3170.98±0.71 / 70.42±1.26
TRACES92.45±1.17 / 93.24±1.0460.00±16.05 / 64.52±11.6389.65±0.68 / 88.24±1.4271.34±0.89 / 70.81±0.29
Dream2Discern95.48±0.21 / 96.60±0.3438.49±7.70 / 27.96±3.3191.61±0.19 / 92.05±0.3873.09±0.29 / 73.78±0.34
AuditorAUROCwACCTWA
SAFE93.85±3.00 / 87.50±1.1982.05±2.22 / 84.62±3.8562.65±2.51 / 67.11±2.06
H&S85.71±1.03 / 71.63±10.4078.21±4.44 / 66.67±5.8863.49±4.54 / 53.94±4.58
TRACES85.12±1.57 / 81.15±6.2573.08±6.66 / 73.08±3.8553.90±6.89 / 60.71±6.30
Dream2Discern98.41±1.24 / 88.49±14.0683.33±9.68 / 84.62±3.8567.92±7.83 / 70.82±6.86

On the group-disjoint trajectories, Dream2Discern still outperforms SAFE, H&S and TRACES on all four metrics on both backbones, and each metric is at least as good as in Table 1. On the 26 overlapping trajectories, Dream2Discern obtains a lower wACC and TWA than on the group-disjoint ones, which indicates that the group overlap does not inflate Table 1. We have added both tables to Appendix E.5.

W2 (Q2): Operating-point protocol of closed-source judges

R2: Thank you for this precise observation. As stated in Section 4.1 and Appendix D.4, the returned labels of the closed-source judges define their decisions for wACC and TWA, while AUROC and FPR95 are computed from continuous scores for every method and do not depend on the operating point. Calibrating these judges on the training successes requires their scores on the training trajectories. We have not collected these scores, since each judge was queried in one recorded run on the test trajectories only (Appendix D.4).

Following your second suggestion, we limit the comparison with closed-source judges to AUROC and FPR95. As shown in Table 1, Dream2Discern achieves a higher AUROC and a lower FPR95 than all four closed-source judges on AFTraj-2K, and only falls behind Claude-Sonnet-4.6 on StepShield.

We have removed the statement that our 4B auditor exceeds Claude-Sonnet-4.6 in StepShield wACC and TWA from Section 4.2.

W3: Scope of the evidence

R3: Thank you for these constructive comments. We would like to clarify the scope of our evidence in the following three aspects.

① Test sizes. We use the held-out test sets released with the two benchmarks, report the mean and standard deviation over three seeds for Dream2Discern, SAFE, H&S and TRACES (Tables 1 and 2), and obtain consistent results on the group-disjoint subset of AFTraj-2K (please refer to the response of W1 (Q1)). Moreover, the StepShield test trajectories come from incident templates never used in training (Section 4.1), so Table 1 already evaluates generalization beyond the training templates, where Dream2Discern keeps its advantage over these baselines.

② Per-benchmark training. First, we would like to explain that the per-benchmark training in our work follows SAFE, H&S and TRACES, which train and evaluate their failure predictors within each benchmark. AFTraj-2K labels task failures of multi-agent systems, whereas StepShield labels policy violations of a code agent (Appendix D.1), so a transfer between them changes the agent system, the task domain and the label definition, which is beyond the scope of this work. We have added this to the limitations in Appendix F.

③ Closed-loop intervention. Our work targets runtime failure prediction (Section 2), and its contribution lies in how the auditor is trained, i.e., learning from hard latent variants of recorded prefixes without collecting or verifying new trajectories. The reaction to an alarm is a separate component left to the deployment system, and its evaluation requires closed-loop executions (Appendix F). What an intervention needs from the auditor is an accurate and early alarm, which we evaluate with TWA and the accuracy-timeliness trade-off (Figure 4).

AuthorsOpenReview ↗

[2/2] Rebuttal by Authors

W4 (Q3): Multi-seed ablation and radius selection

R4: Thanks for this helpful comment. Following your suggestion, we report the perturbation ablation with three seeds and a radius analysis in which γ\gamma is not selected per backbone.

Multi-seed ablation. Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

Pert. strategyAFTraj-2K AUROCAFTraj-2K wACCAFTraj-2K TWAStepShield AUROCStepShield wACCStepShield TWA
w/o Perturbation94.58±0.56 / 94.03±0.3790.66±0.52 / 89.96±1.0670.69±2.07 / 69.50±1.4991.38±1.40 / 93.64±0.8586.42±1.87 / 88.58±1.4954.79±1.62 / 57.04±1.42
w/ Random Pert.94.00±0.55 / 94.62±0.4689.36±0.63 / 89.46±0.9069.30±1.23 / 69.15±1.7591.85±0.62 / 93.73±0.3985.96±2.19 / 88.43±1.6754.27±1.29 / 56.57±1.26
w/ Dream Pert.95.45±0.16 / 95.82±0.4390.96±0.60 / 91.47±0.1772.61±0.38 / 73.45±0.7193.62±0.68 / 93.95±0.6390.28±0.46 / 87.50±0.9357.67±0.35 / 55.69±1.10

Dream consistently achieves higher AUROC in all settings and higher wACC and TWA in three of the four settings, improving over the unperturbed auditor by up to 3.86 wACC and 3.95 TWA. Random perturbations of the same operators at the same radius fall below the unperturbed auditor in wACC and TWA in all four settings, which shows that the improvement of Dream comes from its loss-guided search. The only setting without improvement in wACC and TWA is StepShield with Qwen3-8B. The StepShield test trajectories come from incident templates never used in training (Section 4.1), so the radius selected on training batches is not necessarily the best one on this test set.

Radius selection. We would like to emphasize that γ\gamma is selected on training batches (Appendix D.3), so the improvements above are not obtained by tuning γ\gamma on the test sets. To remove the per-backbone selection, we further fix γ\gamma=10⁻³ for both backbones and train three seeds on AFTraj-2K. Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

MethodAUROCFPR95wACCTWA
w/o Perturbation95.58±0.56 / 94.03±0.3736.69±5.33 / 54.24±6.9390.66±0.52 / 89.96±1.0670.69±2.07 / 69.50±1.49
Dream (γ\gamma=10⁻³)96.78±0.71 / 96.36±0.8826.63±6.51 / 30.77±8.3591.57±0.52 / 91.16±0.4673.35±0.63 / 73.21±0.77

With this single radius, Dream improves all four metrics over the unperturbed auditor on AFTraj-2K with both backbones.

We have updated Table 3 in Section 4.3 with the three-seed results and added the single-radius results to Appendix E.4.

Q4: Trade-off between effectiveness and efficiency

R5: Thank you for the question. We would like to claim that Dream2Discern trades additional training computation for prediction quality and leaves the runtime cost unchanged. We report the training time per step and the peak GPU memory on AFTraj-2K with batches of 32 prefixes, measured on one GPU (RTX PRO 6000 for Qwen3-4B, B200 for Qwen3-8B). Cells report Qwen3-4B / Qwen3-8B.

TrainingTime per step (s)Peak GPU memory (GiB)
Standard LoRA fine-tuning8.1 / 6.845.1 / 61.3
Dream2Discern w/o rank cap (no SVD)12.1 / 9.358.0 / 81.9
Dream2Discern18.9 / 12.058.0 / 81.9

Compared with standard LoRA fine-tuning, although Dream2Discern requires more but acceptable time and memory per training step, it delivers a significant improvements in performance (please refer to the response for W4 (Q3)). At runtime the perturbation is discarded, so each prediction requires one forward pass of the auditor, which is the same as the standard LoRA classifier. We have added this comparison to Appendix D.3.

AuthorsOpenReview ↗

Looking forward to your responses or further suggestions/comments!

Dear Reviewer 8TZY,

We sincerely thank you again for your positive assessment and constructive comments, in particular for recognizing that Dream2Discern addresses a practical limitation of trajectory-level supervision, the closed-form analysis in Propositions 1 and 2, and the breadth of our empirical study. Following your suggestions, we have addressed each of your comments.

  • Group-disjoint evaluation (W1, Q1). We have re-evaluated the auditors of Table 1 on the group-disjoint and the overlapping trajectories of AFTraj-2K in Appendix E.5 (Table 13).
  • Closed-source judges (W2, Q2). We have limited our comparison with closed-source judges to AUROC and FPR95 in the response, and removed the statement on Claude-Sonnet-4.6 from Section 4.2.
  • Scope of the evidence (W3). We have clarified the test sizes and the per-benchmark training, and added cross-benchmark transfer to the limitations in Appendix F.
  • Multi-seed ablation and radius selection (W4, Q3). We have updated Table 3 in Section 4.3 with three seeds and added a single radius for both backbones to Appendix E.4 (Table 12).
  • Effectiveness and efficiency (Q4). We have added the training cost comparison to Appendix D.3 (Table 7).

We hope these responses have addressed the questions you raised. As the discussion period is drawing to a close, we would be grateful to know whether our responses have addressed your concerns. We remain committed to addressing any remaining questions during the discussion period.

Best regards,

Authors of Submission #14607

Reviewer_1qcxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dream2Discern, a framework for runtime failure prediction in multi-agent systems. The method alternates between two stages: Dream, which generates hard latent variants through loss-guided low-rank perturbations to the model's query/value projections, and Discern, which trains the auditor on these variants using the original outcome labels. Experiments on AFTraj-2K and StepShield demonstrate promising improvements over existing failure prediction baselines.

Critical strengths
  1. Important problem. The paper addresses runtime failure prediction in multi-agent systems, which is an important problem for improving the reliability of agentic systems.

  2. Promising experimental results. The proposed method achieves strong performance across different benchmarks and model backbones, outperforming several existing failure prediction baselines.

  3. Clear methodology and analysis. The proposed framework is well structured and easy to follow. The paper also provides theoretical analysis and ablation studies to support the proposed method.

Critical weaknesses
  1. Limited novelty. The proposed method is closely related to existing approaches such as AWP [1] and SAM [2]. Although the authors introduce low-rank perturbations to query/value projections, the paper lacks direct comparisons with these methods and sufficient justification for these design choices.

  2. Unclear contribution of the perturbation strategy. Table 3 shows that the improvements from Dream are relatively limited and inconsistent across wACC and TWA. This raises concerns about how much the proposed perturbation strategy actually contributes to the overall performance.

  3. Concerns about the training setup. The training procedure selects failed trajectory prefixes starting from the annotated decisive error step, meaning that errors have already occurred in these training samples. This may shift the focus toward detecting existing errors rather than predicting failures in advance. The paper lacks an evaluation of prediction performance before the decisive error step.

[1]: Wu, Dongxian, Shu-Tao Xia, and Yisen Wang. "Adversarial weight perturbation helps robust generalization." Advances in neural information processing systems 33 (2020): 2958-2969. [2]: Foret, Pierre, et al. "Sharpness-aware minimization for efficiently improving generalization." arXiv preprint arXiv:2010.01412 (2020).

Questions for authors
  1. Computational overhead: What is the additional training cost introduced by Dream, particularly from SVD and repeated loss evaluations? Could the authors provide a comparison of training time and GPU memory usage against standard LoRA fine-tuning?

  2. Threshold calibration: The evaluation datasets have approximately balanced success/failure trajectories. How robust is the proposed threshold calibration when the failure rate or task distribution changes in real-world deployment?

  3. Trajectory length: The evaluation uses the maximum failure score across all prefixes for trajectory-level AUROC. Could this introduce a bias toward longer trajectories, which have more opportunities to produce high failure scores? How does performance vary with trajectory length?

6 comments
AuthorsOpenReview ↗

[1/3] Rebuttal by Authors

Thank you for your careful review. Here are our detailed replies.

W1: Novelty relative to AWP and SAM

Thanks for the constructive comment. We would like to reclaim that our work is novel in introducing a new perspective for runtime failure prediction, i.e., learning from hard latent variants of recorded prefixes. This has not been explored in previous failure auditors, and has both conceptual and technical distinctions from AWP and SAM with explicit knowledge advancement.

Conceptually, Dream uses weight perturbation to construct hard variants of the training prefixes, whereas SAM and AWP use weight perturbation to find flat solutions that generalize better. This design is motivated by a data constraint of trajectory auditing (Section 3.1), where directly augmenting trajectories requires costly rollouts and provides no explicit criterion for identifying hard cases to the current auditor. Dream therefore uses the auditor's own prediction loss to construct hard variants in the representation space. This is why we call Dream an implicit augmentation. In contrast, SAM is motivated by the geometry of the loss landscape and perturbs all trainable parameters to measure the sharpness of the loss around the current solution. Dream differs from SAM in both the motivation and the role of the perturbation.

Technically, Dream differs from SAM and AWP in three aspects.

① Representation-only perturbation. Dream perturbs the effective q/v projection weights under a fixed classifier (Section 3.2), whereas SAM and AWP perturb all trainable parameters, including the LoRA factors and the classifier. Based on this, the prediction loss under Dream can rise only through a change in the representation of the same prefix, while a median of 30% (Qwen3-4B) and 46% (Qwen3-8B) of the first-order loss increase under SAM comes from moving the classifier.

② Operator-space perturbation. The weights perturbed by Dream can deviate from the frozen weights by a matrix of rank up to 16, whereas SAM and AWP perturb the rank-8 LoRA factors and keep this deviation within rank 8.

③ Low-rank perturbation. The rank cap of Dream restricts the change at each projection to at most rr output directions shared across prefixes (Eq. 5), which yields structured hard variants for the auditor.

Empirically, we apply AWP and SAM to the same LoRA adapters and classifier as Dream2Discern on AFTraj-2K, with the settings of their original papers (γ\gamma=5×10⁻³ for AWP, ρ=0.05 for SAM). Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

MethodAUROCwACCTWA
AWP94.56±0.41 / 94.54±0.2790.06±0.52 / 90.46±0.9271.48±0.37 / 70.74±0.63
SAM95.03±0.52 / 94.82±0.3690.26±1.14 / 90.56±0.1771.76±0.40 / 71.34±1.24
Dream2Discern95.45±0.16 / 95.82±0.4390.96±0.60 / 91.47±0.1772.61±0.38 / 73.45±0.71

Dream2Discern consistently achieves better performance than both AWP and SAM on both backbones. We have added this comparison to Appendix E.3 and this discussion to Section 3.2 and Appendix B.1.

W2: Contribution of Dream

Thanks for raising this concern. We would like to first clarify that the improvements from Dream in Table 3 are not inconsistent: Dream improves both wACC and TWA over the unperturbed auditor by 1.51 to 5.56 points in two settings, and the differences in the other two settings are below one point under only a single training seed. For a more comprehensive comparison, we have rerun this ablation with three seeds. Cells report Qwen3-4B / Qwen3-8B (mean±std).

Pert. strategyAFTraj-2K wACCAFTraj-2K TWAStepShield wACCStepShield TWA
w/o Perturbation90.66±0.52 / 89.96±1.0670.69±2.07 / 69.50±1.4986.42±1.87 / 88.58±1.4954.79±1.62 / 57.04±1.42
w/ Random Pert.89.36±0.63 / 89.46±0.9069.30±1.23 / 69.15±1.7585.96±2.19 / 88.43±1.6754.27±1.29 / 56.57±1.26
w/ Dream Pert.90.96±0.60 / 91.47±0.1772.61±0.38 / 73.45±0.7190.28±0.46 / 87.50±0.9357.67±0.35 / 55.69±1.10

With three seeds, Dream achieves the highest wACC and TWA in three of the four settings, improving over the unperturbed auditor by up to 3.86 wACC and 3.95 TWA. Random perturbations of the same operators at the same radius fall below the unperturbed auditor, showing that the improvement of Dream comes from its loss-guided search. The only exception is StepShield with Qwen3-8B. We attribute this to the perturbation radius (shared by both benchmarks), while StepShield favors a smaller radius (Table 11). Furthermore, we have also explained in Section 4.1 that StepShield's test trajectories come from incident templates never used in training, so the radius selected on training batches is not necessarily the best one on this test set.

We have updated Table 3 in Section 4.3 with these three-seed results.

AuthorsOpenReview ↗

[2/3] Rebuttal by Authors

W3: Training setup and evaluation before the decisive error step

Thank you for the question. First, we would like to clarify that the decisive-step annotations only determine which recorded prefixes are used for training and are never part of the auditor's input (Appendix D.2). In addition, our evaluation already scores every nonterminal prefix, including those preceding the decisive step.

Following your suggestion, we further report prediction performance before the decisive step. On the failed test trajectories of AFTraj-2K, we report the percentage whose first alarm is raised at or before the decisive step KK (Pre-KK Rate), and the AUROC between failed and successful trajectories when failed ones are scored only on their prefixes before KK (Pre-KK AUROC). Cells report Qwen3-4B / Qwen3-8B (mean±std of three seeds).

AuditorPre-KK RatePre-KK AUROC
SAFE56.03±2.90 / 54.40±3.4967.51±0.44 / 64.11±1.87
H&S67.48±2.13 / 68.71±1.6257.35±3.74 / 49.19±3.40
TRACES69.53±4.52 / 73.01±1.6261.59±5.05 / 67.79±4.65
Dream2Discern80.37±0.61 / 80.78±2.7773.84±0.56 / 68.88±0.91

Dream2Discern still raises more timely alarms and achieves a higher Pre-KK AUROC than SAFE, H&S and TRACES on both backbones. We have added this analysis to Appendix E.5.

Q1: Training overhead of Dream

Thanks for the practical question. We would like to clarify that Dream adds computation only during training. At runtime the perturbation is discarded and each prediction requires only one forward pass of the auditor, same as a standard LoRA classifier.

Following your suggestion, we report the training time per step and the peak GPU memory on AFTraj-2K with batches of 32 prefixes, measured on one GPU (RTX PRO 6000 for Qwen3-4B, B200 for Qwen3-8B). Cells report Qwen3-4B / Qwen3-8B.

TrainingTime per step (s)Peak GPU memory (GiB)
Standard LoRA fine-tuning8.1 / 6.845.1 / 61.3
Dream2Discern w/o rank cap (no SVD)12.1 / 9.358.0 / 81.9
Dream2Discern18.9 / 12.058.0 / 81.9

We have added this comparison to Appendix D.3.

Q2: Robustness of threshold calibration

Thank you for raising these deployment questions. We would like to first clarify that the threshold calibration of Dream2Discern does not rely on the class balance of the datasets. As described in Section 3.4 and Appendix B.2, the step-dependent thresholds are fitted from successful trajectories of the training partition only at a nominal false-alarm level, i.e., the fraction of successful trajectories that are allowed to raise an alarm. We answer the two cases below.

  • Failure rate. Since no failed trajectory enters the calibration, a change in the failure rate alters neither the thresholds nor the false-alarm and detection rates of the auditor, as each rate is computed within one class. AUROC, FPR95 and TWA do not depend on the class proportions either, and wACC only re-weights the two rates. In deployment, the user can control the operating point through α\alpha, e.g., a smaller α\alpha when failures are rare and false alarms are costly. In Figure 4, we have varied α\alpha from 0.01 to 1.00, and the gains of Dream2Discern on StepShield extend across a broad range of these operating points (see Section 4.3).
  • Task distribution. In Table 1, we have already evaluated the thresholds under a template-level shift, since the StepShield test trajectories come from incident templates never used in training. With the thresholds fitted on the training templates, Dream2Discern still achieves 90.28 / 87.50 wACC with Qwen3-4B / Qwen3-8B. Under a larger shift, α\alpha remains a nominal level without a guarantee. In this case, the thresholds can be refitted from successful trajectories of the new deployment, which requires neither failure labels nor retraining of the auditor.

We have added this discussion to Appendix B.2.

AuthorsOpenReview ↗

[3/3] Rebuttal by Authors

Q3: Length bias of the maximum prefix score

Thanks for pointing out this potential confound. We would like to first clarify that the maximum over prefixes does not inflate our results through trajectory length. For wACC and TWA, the thresholds are calibrated on the same per-trajectory maximum over successful trajectories (Appendix B.2), so the nominal false-alarm level already covers the additional prefixes of longer trajectories. For AUROC, failed trajectories are shorter than successful ones on AFTraj-2K (9.5 vs. 11.5 steps on average) and as long as successful ones on StepShield (14.2 vs. 14.1 steps), so trajectory length alone gives an AUROC of only 34.86 and 50.45.

We further report the AUROC of Dream2Discern with failed and successful trajectories compared only within the same length quintile (length-matched), and within three ranges of the length TT (mean±std of three seeds).

AFTraj-2KAll (Table 1)Length-matchedT≤6T\le67≤T≤107\le T\le10T≥11T\ge11
Qwen3-4B95.45±0.1697.26±0.4398.91±0.3696.87±0.4495.41±1.54
Qwen3-8B95.82±0.4396.48±0.3798.95±0.0696.62±0.5291.81±0.77
StepShieldAll (Table 1)Length-matchedT≤13T\le1314≤T≤1614\le T\le16T≥17T\ge17
Qwen3-4B93.62±0.6893.85±0.6893.71±0.6691.29±0.8296.72±0.90
Qwen3-8B93.95±0.6393.81±0.7595.70±0.2692.08±0.5994.21±2.18

After matching the trajectory length, the AUROC rises by 0.66 to 1.81 points on AFTraj-2K and changes by at most 0.23 points on StepShield. We have added this analysis to Appendix E.5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback. We are glad that the reviewers find runtime failure prediction an important problem for reliable agentic systems (1qcx), and that our method addresses a practical limitation of trajectory-level supervision without changing the recorded inputs or labels (8TZY). We also appreciate that the reviewers find the framework well structured and supported by theoretical analysis (1qcx, 8TZY), the empirical study broad across two benchmarks, four backbones and recent learned auditors (8TZY), and the results strong across benchmarks and backbones (1qcx, 8TZY, 7Mqt), making Dream2Discern a good baseline for future learned runtime monitors (7Mqt).

We have responded to every comment point by point in the individual responses, supported by new experiments and analyses. All revisions are marked in red in the revised PDF, and we summarize them below.

Contribution of Dream (1qcx, 8TZY, 7Mqt)

  • Report the perturbation ablation over three seeds (see Table 3 in Section 4.3)
  • Add the seed-paired comparison with the unperturbed auditor on all four backbones (see Table 8 in Appendix E.3.1)
  • Clarify the attribution of the margin in Table 1 (see Section 4.2)

Relation to AWP and SAM (1qcx, 7Mqt)

  • Discuss the conceptual and technical differences (see Section 3.2 and Appendix B.1)
  • Compare with AWP and SAM on the same LoRA adapters and prediction head over three seeds (see Table 9 in Appendix E.3.1)
  • Add SAM, the gradient-norm penalty, Flat-LoRA and Bi-LoRA to the related work (see Appendix A)

Radius selection (8TZY, 7Mqt)

  • Evaluate a single radius for both Qwen3 backbones over three seeds (see Table 12 in Appendix E.4)

Evaluation protocol (1qcx, 8TZY, 7Mqt)

  • Re-evaluate the auditors on the group-disjoint and the overlapping trajectories of AFTraj-2K (see Table 13 in Appendix E.5)
  • Evaluate the prediction before the decisive step and the length-matched AUROC (see Tables 14 and 15 in Appendix E.5)
  • Discuss threshold calibration under a different failure rate and distribution shift (see Appendix B.2)
  • Remove the comparison with the label-based wACC and TWA of Claude-Sonnet-4.6 (see Section 4.2)

Training cost, motivation and scope (1qcx, 8TZY, 7Mqt)

  • Compare the training time and GPU memory with standard LoRA fine-tuning (see Table 7 in Appendix D.3)
  • Repeat the comparison of Section 3.1 over three seeds (see Appendix E.1.2)
  • Add cross-benchmark transfer to the limitations (see Appendix F)

We hope that these revisions address the reviewers' concerns, and we would be happy to clarify any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.