acceptodds
Under review as a conference paper at ICLR 2027

Diagnostic Resolution: What Agent Diagnoses Preserve for Downstream Use

Abstract

Failure diagnoses are often evaluated by whether they identify the right failure, but later use depends on what information the diagnostic record preserves. We show that task outcomes, failure labels, recurring patterns, and unchanged citations can each collapse distinctions that matter for subsequent decisions: a correct label may lack the evidence for the next check, similar failures may require different procedures, and previously valid judgments may require reassessment as evidence changes. We introduce Diagnostic Resolution (DiagRes), a consumer-conditional diagnotic protocol that evaluates whether a frozen diagnostic record supports correct later judgments under declared source access and reading budgets. DiagRes tests judgment support, check applicability, and reassessment, requiring answers to be correct, sufficiently supported, and bound to the appropriate requirement, execution, and version. Across 22 public tasks, DiagRes-S selection improves follow-up correctness over failure-label selection. Controlled studies across DiagRes-S/A/T distinguish material information loss from harmless record changes; with equal fact budgets and equal individual-decision accuracy, complete versus split evidence yields 50% versus 0% joint success. DiagRes-guided revision further raises correct coverage from 59.6% to 78.9% across 114 questions from 19 sources. These results suggest that reusable diagnosis depends not simply on retaining more information, but on preserving the complete grounds needed for later decisions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Evaluation, Failure Diagnosis, Trajectory Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_m4kfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Research Question:


The paper argues that agent failure diagnoses are often judged by whether they identify the correct failure. However, their value for later use depends on whether the diagnostic record preserves the information needed for follow-up decisions. The case studies show three ways this information can be lost. Task outcomes may hide unsupported reports or verified local improvements. Shared failure patterns do not determine whether a check can be reused. Conclusions can also become stale even when their citations remain unchanged.

Their Method:


To measure this, the authors propose Diagnostic Resolution (DiagRes). A diagnostic record is frozen and given to an LLM reader, who answers paired probe questions under a fixed source-access policy and reading budget. The probes test whether a claim is supported (S), whether a check applies to another case (A), or whether a judgment needs reassessment after an update (T). An answer counts only if it is correct, sufficiently supported, and tied to the right claim, execution, and version. Both items in a pair must pass.

Result:


The authors validate DiagRes through several studies with LLM readers. Across 22 tasks, selecting records by DiagRes-S instead of failure-label agreement improves follow-up correctness by 6.0 and 5.7 percentage points for two readers. In controlled cases, records with the correct label but mismatched evidence fail DiagRes and do not support correct follow-up decisions. Restoring a key fact raises task-macro correctness from 94.9% to 98.9%. With equal fact budgets, complete evidence achieves 50% joint success while split evidence achieves 0%. Finally, DiagRes-guided revision raises correct coverage across 114 follow-up questions from 59.6% to 78.9%.

Critical strengths
  1. A useful reframing. The core argument is convincing. Records with the same failure label are not interchangeable because one may preserve the evidence needed for the next check while another may lose it. Existing attribution benchmarks would score both records the same, so this points to a real gap in how diagnoses are evaluated.

  2. Sensible protocol design. The protocol freezes records before revealing the questions, fixes source access and reading budgets, and derives answer keys from the full sources. “Unresolved” counts as correct only when the sources leave the question open. Evidence must also be tied to the correct execution and version.

Critical weaknesses
  1. Limited evidence for generalization. Several key demonstrations use small case sets. Table 2 contains two cases with three follow-up questions, and template aliasing is examined through only two templates and ten pair comparisons. The equal-fact-budget experiment illustrates why complete evidence matters for joint decisions, but its deliberately constructed fact allocation provides limited evidence about naturally occurring diagnostic records. Similarly, restoring a necessary fact demonstrates the value of that information without establishing the added value of DiagRes as an evaluator. Validation is also uneven across task families: Section 5 focuses mainly on DiagRes-S, while applicability and reassessment rely more heavily on case comparisons and temporal studies, including 30 real source-update tasks.

  2. The quantitative gains are not established against stronger alternatives. The 22-task selection study reports improvements of 6.0 and 5.7 percentage points without confidence intervals, despite the cluster-bootstrap procedure described in Section 4.3. Label-only selection is the only baseline with reported results in this study; random selection is described, but its results are omitted. The evaluation also relies heavily on Astra and Sol across record construction, reading, revision, and some reference assessments, and no human readers are included. Separate contexts and execution-based checks provide useful safeguards, but independent validation remains limited. Finally, the paper gives insufficient detail about how probes and follow-up questions are generated and how much their information requirements overlap. These gaps make it difficult to establish whether DiagRes offers predictive value beyond simpler evaluation methods.

Questions for authors

Questions


  1. How are probes, follow-up questions, and reference answers created? Who writes them, the authors or a model, and which model? Are probes and follow-up questions generated through the same process? Who judges evidence sufficiency and binding, and how often is this checked by humans? If the same model family writes the questions, builds the records, and scores the answers, the selection results could be partly circular.

  2. Uncertainty and baselines for record selection. What are the confidence intervals for the +6.0 and +5.7 point gains using the cluster bootstrap described in Section 4.3? What does random selection achieve? How often do all three records share the same failure label, making label-only selection equivalent to random selection?

  3. Natural validation of DiagRes-A and DiagRes-T. Beyond the Section 3 cases and constructed workflows, are these two families evaluated on natural records with pair scoring? Do their scores show the same relationship with downstream use as DiagRes-S?

  4. Revision study. What is the overall correct coverage of self-revision across the 114 follow-up questions? Do the probe-based feedback messages overlap with the follow-up questions?

Suggestions


  1. Include human readers on at least a subset of the evaluation to test whether DiagRes scores reflect usefulness for people, not only LLMs.
  2. Use a reader and assessor from a different model family than the record builder.
  3. Scale up the controlled studies, report confidence intervals throughout, and compare against simpler signals such as record length, direct LLM quality ratings, and QA-based metrics such as QuestEval.
  4. Clarify the distinction between probes and follow-up questions early in the paper. Also avoid reusing the same letters for different concepts.
  5. The paper contains several typos and grammatical errors, for example "diagnotic" in the abstract, "Trajectory-level diagnostic of agents" in Section 1, and "We use GPT-6 Astra builds evidence-linked graphs" in Section 3. A careful proofread would help.
Reviewer_Q4x9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Diagnostic Resolution (DiagRes), an evaluation protocol for assessing whether agent failure diagnoses preserve information useful for downstream decisions. Instead of evaluating only failure attribution or task outcomes, DiagRes measures three aspects of diagnostic records: evidence support (DiagRes-S), applicability of diagnostic checks (DiagRes-A), and reassessment after evidence changes (DiagRes-T).

Critical strengths

The paper raises a reasonable question about whether failure diagnoses preserve sufficient information for downstream use. The distinction between producing a correct failure label and retaining the evidence needed for subsequent decisions is potentially useful.

Critical weaknesses

the paper does not provide direct comparisons against these stronger evaluation approaches or simpler LLM-as-a-judge baselines. The primary comparison against failure-label agreement is insufficient to establish the advantages of the proposed protocol.

The paper introduces DiagRes as a general evaluation protocol for agent failure diagnoses, yet it does not provide a systematic benchmark of existing failure diagnosis methods or different LLMs. In particular, there is no comprehensive comparison of representative diagnosis approaches across the proposed DiagRes-S, DiagRes-A, and DiagRes-T dimensions. The experiments primarily focus on selecting among a small number of generated diagnostic records, controlled information modifications, and feedback-guided revision.

The paper reports an improvement in correct coverage from 59.6% to 78.9% after DiagRes-guided revision. However, the revision procedure provides additional task-specific feedback that is not available to the self-revision baseline. It is therefore unclear whether the improvements arise from the proposed diagnostic resolution metric or simply from access to additional relevant information.

Questions for authors

NA

Reviewer_Qd4qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper argues that failure diagnoses should be evaluated not only by whether they identify the correct failure label, but by whether the retained record preserves enough information for correct downstream use. The authors identify three forms of diagnostic information loss—outcome blind spots, template aliasing, and diagnostic staleness—and introduce DiagRes, a reader-conditional protocol that tests judgment support, check applicability, and reassessment under fixed source access and reading budgets. Experiments show that DiagRes-based record selection improves follow-up correctness by 6.0 and 5.7 points over label-only selection, and that equal fact budgets can yield 50% versus 0% joint success depending on whether evidence is complete or split.

Critical strengths
  1. The paper moves beyond the conventional evaluation of whether a diagnostic label is correct, and instead asks whether a diagnostic record preserves enough information to support correct downstream decisions.

  2. Outcome blind spots, template aliasing, and diagnostic staleness are not proposed abstractly but are induced from 193 public trajectories across 76 tasks, and illustrated with concrete cases

  3. The paper combines natural selection experiments, controlled interventions, fact restoration, equal-budget studies, and guided revision, cross-validating the central claim from multiple angles.

Critical weaknesses
  1. The authors position DiagRes as a protocol, and "protocol" implies generality, yet the evaluation is predominantly software-oriented. Neither the title nor the abstract signals this scope limitation; although the Limitations section acknowledges it, the scope should be made explicit earlier (in the title, abstract, or introduction).

  2. The protocol requires retaining substantially more contextual information, but increased context can divert a reader's attention and is not guaranteed to be purely beneficial. While the equal-budget study shows that information quantity alone is insufficient, the paper does not directly test whether added context has a negative effect on reader attention—a dimension that is especially relevant given that reading budget is a core part of the protocol.

  3. Extracting the relevant evidence relies on LLMs and tools. In more flexible or open-ended settings where such tools do not apply, LLM-only extraction is likely to introduce error. The paper reports no error rate or sensitivity analysis for the extraction pipeline, and does not discuss how the protocol degrades when tooling is unavailable. This matters because the validity of the evaluation inherits any extraction error.

  4. The paper introduces DiagRes-A and DiagRes-T alongside DiagRes-S, and evaluates them in controlled and temporal studies. However, there is no natural experiment that selects records by A or T and measures downstream correctness, and no analysis isolating how much each of the three criteria contributes to downstream performance. The purpose of A and T would be clearer if their downstream impact were quantified in the same way as S.

  5. The protocol positions the reader as part of the evaluation condition, yet the main selection experiments use only two recent models. Since different LLMs may be differentially sensitive to enlarged context, the results should be validated on a broader set of readers (more models, open- vs. closed-source, different scales, and ideally human readers).

  6. The paper's formalization, while appropriate for a protocol paper, is at times opaque. It is better to provide concrete running examples

Questions for authors

Please check weak points above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.