- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Research Question:
The paper argues that agent failure diagnoses are often judged by whether they identify the correct failure. However, their value for later use depends on whether the diagnostic record preserves the information needed for follow-up decisions. The case studies show three ways this information can be lost. Task outcomes may hide unsupported reports or verified local improvements. Shared failure patterns do not determine whether a check can be reused. Conclusions can also become stale even when their citations remain unchanged.
Their Method:
To measure this, the authors propose Diagnostic Resolution (DiagRes). A diagnostic record is frozen and given to an LLM reader, who answers paired probe questions under a fixed source-access policy and reading budget. The probes test whether a claim is supported (S), whether a check applies to another case (A), or whether a judgment needs reassessment after an update (T). An answer counts only if it is correct, sufficiently supported, and tied to the right claim, execution, and version. Both items in a pair must pass.
Result:
The authors validate DiagRes through several studies with LLM readers. Across 22 tasks, selecting records by DiagRes-S instead of failure-label agreement improves follow-up correctness by 6.0 and 5.7 percentage points for two readers. In controlled cases, records with the correct label but mismatched evidence fail DiagRes and do not support correct follow-up decisions. Restoring a key fact raises task-macro correctness from 94.9% to 98.9%. With equal fact budgets, complete evidence achieves 50% joint success while split evidence achieves 0%. Finally, DiagRes-guided revision raises correct coverage across 114 follow-up questions from 59.6% to 78.9%.
Critical strengths
-
A useful reframing. The core argument is convincing. Records with the same failure label are not interchangeable because one may preserve the evidence needed for the next check while another may lose it. Existing attribution benchmarks would score both records the same, so this points to a real gap in how diagnoses are evaluated.
-
Sensible protocol design. The protocol freezes records before revealing the questions, fixes source access and reading budgets, and derives answer keys from the full sources. “Unresolved” counts as correct only when the sources leave the question open. Evidence must also be tied to the correct execution and version.
Critical weaknesses
-
Limited evidence for generalization. Several key demonstrations use small case sets. Table 2 contains two cases with three follow-up questions, and template aliasing is examined through only two templates and ten pair comparisons. The equal-fact-budget experiment illustrates why complete evidence matters for joint decisions, but its deliberately constructed fact allocation provides limited evidence about naturally occurring diagnostic records. Similarly, restoring a necessary fact demonstrates the value of that information without establishing the added value of DiagRes as an evaluator. Validation is also uneven across task families: Section 5 focuses mainly on DiagRes-S, while applicability and reassessment rely more heavily on case comparisons and temporal studies, including 30 real source-update tasks.
-
The quantitative gains are not established against stronger alternatives. The 22-task selection study reports improvements of 6.0 and 5.7 percentage points without confidence intervals, despite the cluster-bootstrap procedure described in Section 4.3. Label-only selection is the only baseline with reported results in this study; random selection is described, but its results are omitted. The evaluation also relies heavily on Astra and Sol across record construction, reading, revision, and some reference assessments, and no human readers are included. Separate contexts and execution-based checks provide useful safeguards, but independent validation remains limited. Finally, the paper gives insufficient detail about how probes and follow-up questions are generated and how much their information requirements overlap. These gaps make it difficult to establish whether DiagRes offers predictive value beyond simpler evaluation methods.
Questions for authors
Questions
-
How are probes, follow-up questions, and reference answers created? Who writes them, the authors or a model, and which model? Are probes and follow-up questions generated through the same process? Who judges evidence sufficiency and binding, and how often is this checked by humans? If the same model family writes the questions, builds the records, and scores the answers, the selection results could be partly circular.
-
Uncertainty and baselines for record selection. What are the confidence intervals for the +6.0 and +5.7 point gains using the cluster bootstrap described in Section 4.3? What does random selection achieve? How often do all three records share the same failure label, making label-only selection equivalent to random selection?
-
Natural validation of DiagRes-A and DiagRes-T. Beyond the Section 3 cases and constructed workflows, are these two families evaluated on natural records with pair scoring? Do their scores show the same relationship with downstream use as DiagRes-S?
-
Revision study. What is the overall correct coverage of self-revision across the 114 follow-up questions? Do the probe-based feedback messages overlap with the follow-up questions?
Suggestions
- Include human readers on at least a subset of the evaluation to test whether DiagRes scores reflect usefulness for people, not only LLMs.
- Use a reader and assessor from a different model family than the record builder.
- Scale up the controlled studies, report confidence intervals throughout, and compare against simpler signals such as record length, direct LLM quality ratings, and QA-based metrics such as QuestEval.
- Clarify the distinction between probes and follow-up questions early in the paper. Also avoid reusing the same letters for different concepts.
- The paper contains several typos and grammatical errors, for example "diagnotic" in the abstract, "Trajectory-level diagnostic of agents" in Section 1, and "We use GPT-6 Astra builds evidence-linked graphs" in Section 3. A careful proofread would help.