acceptodds
Withdrawn from ICLR 2027

From First to Last Without Changing a Prediction: Observability Is Not Specification in Agent Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent diagnoses increasingly guide repairs and incident investigations, yet a complete trajectory can leave open what a correct diagnosis should identify. Should it name the final answer to correct or the upstream source to inspect? This choice can reverse a system comparison: on 38 shared cases, the leading method becomes last among four methods when the diagnostic question changes, without changing a single prediction. We make the rules that define acceptable diagnoses a controlled experimental variable: change the rules with the model fixed, then change the executor with the rules fixed. On 720 frozen trajectories, supplying explicit rules raises fully correct diagnoses from 233 to 716. The repairs concern the meaning of an identified event as well as its location and whether a location is justified at all. Targeted role swaps produce the predicted changes among cases originally answered correctly under the rules. On 217 external trajectories, explicit rules improve acceptable answers and reduce unsupported claims of uniqueness; a model and a program agree on all three main scoring outcomes for 213 cases. These tests make diagnostic-object identification operational: they check which rules govern which answers, and whether their execution is reproducible. Before comparing diagnostic systems, specify and test what counts as a correct diagnosis.

Primary area:
datasets and benchmarks
Keywords:
AI agent evaluation, agent diagnosis, benchmark evaluation, evaluation specification, process evaluation, semantic evaluation, evaluation robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZrgGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This is an ambitious and intellectually interesting paper. Its central claim is not about improving agent diagnosis per se, but about diagnostic specification: the observation that a failure trajectory does not uniquely determine what a diagnosis ought to identify. The paper argues that the diagnostic target itself is an experimental variable, and that changing the specification can reverse rankings while leaving trajectories and predictions unchanged. In my view, this is potentially a publishable ICLR contribution, but the paper currently sits in an unusual position: The empirical result is strong. The methodological point is important. The formalism is weaker than the experimental story. The paper sometimes overstates what its experiments establish. The underlying theoretical object remains somewhat under-specified.

Critical strengths

The empirical result is strong. The methodological point is important.

Critical weaknesses

The formalism is weaker than the experimental story. The paper sometimes overstates what its experiments establish. The underlying theoretical object remains somewhat under-specified.

The paper appears to assume precisely the object whose variability it wishes to study. Without a prior theory of what constitutes a diagnosis and what constitutes a valid contract, the ranking-reversal result risks collapsing into the observation that changing the definition of success changes who wins. The entire contribution then hinges on whether the proposed contract family has an independent theoretical justification, and that justification does not appear to be developed sufficiently. The paper studies performance under a family of diagnostic contracts, but neither diagnosis nor contract is formally characterized. As a result, the significance of the observed ranking reversals is unclear. The results establish that rankings depend on the evaluation semantics chosen by the authors, but they do not establish whether this dependence reflects a fundamental property of diagnosis or merely the freedom to redefine the target concept. The paper appears to have the following structure: Consider a diagnosis method M. Evaluate it under contract C. Obtain ranking R_C. Change contract. Obtain ranking R_(C^' ). Observe R_C≠R_(C^' ). The conclusion is then: rankings are specification-sensitive. But to make that conclusion meaningful, one needs at least: a formal definition of diagnosis, a formal definition of contract, a characterization of permissible contracts. Otherwise there is an immediate objection: If I am allowed to arbitrarily redefine the target concept, then ranking changes are inevitable. In other words, the result becomes almost tautological. Suppose I define diagnosis as: root cause identification, and then redefine it as: first observed error, and then redefine it as: most repairable error, and then redefine it as: shortest explanatory description. Of course rankings may change. The paper then appears to demonstrate: "Different tasks produce different rankings".

which is true but not very informative.

A useful theory would need to answer: What properties must a diagnosis satisfy? Or: What class of concepts count as diagnoses? The paper instead seems to choose several examples and call them contracts. The problem is that without constraints on admissible contracts there is no nontrivial theorem available. For example, any method Acan be made superior to method Bby constructing a contract that rewards exactly the outputs produced by A. Thus: ∀A,B," "∃C

such that A>_C B.

If arbitrary contracts are allowed, ranking reversal is not surprising. It is unavoidable. A reviewer could therefore ask: What prevents the result from being a consequence of unrestricted relabeling? That question is not obviously answered.

The paper also seems to oscillate between two claims: Claim A "Diagnosis has no unique definition." and Claim B "We have discovered an important new diagnostic evaluation framework." These are in tension. If Claim A is correct, then the burden is on the authors to explain why their chosen framework should be preferred. If diagnosis genuinely lacks a canonical meaning, then EPCE is merely one possible semantics among many. The paper then becomes an ontology proposal rather than a scientific discovery. To justify the significance of ranking reversals, one must argue that the contract family being studied captures something essential about diagnosis. I did not see a convincing argument for that.

Questions for authors
  1. define contract precisely
  2. define diagnosis precisely
  3. define an order relation precisely over contracts/diagnoses
  4. How is this work related to theoretical definitions of diagnosis (of which there are a large number)?
Reviewer_M7cYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The method presented came to improve diagnosis and provide a framework of experiments for that cause. The authors highlight that diagnoses depend on rules defined and once rules are changed the evaluation changes as well. The contribution is the framework of experiments to evaluate the "diagnostic specification" and make it testable.

Critical strengths

Strengths: S1. Motivation is presented clearly along with the first example which demonstrate that ranking can change (first to last). S2. The question's asked to support the claim of the authors are extensive and demonstrate what's claimed, and it is clear thorough experiments were done for each claim (I liked section 3 specifically, in addition to comparison against executor which is exact)

Critical weaknesses

Weaknesses: W1. The question's asked and the experiment's results are not surprising or not very novel to me. The question is pretty obvious W2. This paper is somewhat hard to read, new experiments and benchmarks are presented in sections and it is somewhat hard to follow. (38, 217, 720 trajectories in multiple sections, each for different purpose). W3. A potential user to this framework would have to test many sets of rules and analyze multiple metrics (there is not one which is representative) to evaluate.

Questions for authors

Questions: Q1. I will be glad to see stronger baselines and a way to compare those, to understand the positioning against method which try to do the same. If possible, I would like to authors to position their method against [1]. Q2. Since the authors speak of ranking (from first to last) - can they provide ranking metrics on those mentioned experiments to show how ranking was changed? For example rank correlation. I would like to know how badly ranking can change. Q3. I would love to see a clean organized table, with clear metrics that evaluates the method wholly

[1] Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems (Zhang et al., 2025)

Reviewer_F5tuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that a complete agent trajectory does not, by itself, determine what a correct failure diagnosis should name. In the opening example an agent reads adjusted rather than unadjusted stock prices and submits a wrong year, and one could reasonably say the diagnosis should point either to the submitted answer or to the upstream source. The authors call the rules that settle this a diagnostic specification or contract, and instantiate one, EPCE (Evidence-Path Contamination Evaluation), which maps an observable event graph and task to a set of admissible diagnoses, a nonlocalizing output, or abstention. The paper first shows that rescoring four frozen methods on 38 shared external cases under a "commitment" view versus a "process" view moves Principle-Aware SOL from first to last. It then runs two kinds of tests. Specification sensitivity holds the model (gpt-5.6-sol) fixed and changes the instruction. On 720 internal trajectories, giving the model the full contract raises fully correct outputs from 233 to 716. Three role-swap "semantic placebos" change exactly the predicted cases among those Exact SOL originally got right. On 217 external trajectories from AgentRx and TELBench, the contract raises semantic exactness from 0/97 to 54/97 and admissible hits from 1/194 to 85/194, and cuts false specificity from 113/113 to 3/113. Implementation stability replaces the model with a program, Strong, that executes the same frozen semantic kernel; the two agree on all three main scoring outcomes for 213 of 217 cases. The paper closes with recommendations: declare the diagnostic question before ranking systems, report point exactness, admissible hit, and false specificity separately, and test whether rules are actually followed.

Critical strengths
  • The core message is correct and worth saying to the agent-diagnosis community, which is currently producing failure-attribution benchmarks quickly and often treats "the root step" as if it were uniquely defined. Separating point exactness, admissible hit, and false specificity, each on its own eligible population, is a good evaluation design. False specificity in particular (claiming a unique answer when the evidence supports several, or none) is a useful concept that most existing attribution metrics cannot express.

  • The experimental hygiene is careful. Predictions are frozen before a single join to a prediction-blind reference. The information boundary between Generic and Exact SOL is matched explicitly. The role swaps state in advance which cases should change and then check that no others do. Agreement between executors is reported case by case rather than on totals, which is how the authors notice that four balanced disagreements leave the aggregates identical. The challenger-removal sensitivity analysis on the external reference is also a thoughtful check.

Critical weaknesses
  • The headline improvement is close to circular. The scorer is defined by the EPCE contract, and Exact SOL is the same model given that contract, so the comparison from 233/720 to 716/720 mostly measures whether the model was told the vocabulary and decision rules it is graded on. The paper's own breakdown supports this reading. In 219 of the 485 repairs, Generic SOL already names the right event family and is marked wrong only on EPCE's semantic qualifiers for source, field, and transformation. Another 138 repairs consist of correctly withholding a location under the contract's nonlocalization rules. On the external set, Generic SOL scores 0/97 on semantic exactness and fails false specificity on all 113 eligible cases. Scores at the floor like these point to a mismatch of ontology and output conventions, not to a model that cannot diagnose. The internal tasks were also built around EPCE's own mechanism families (alias joins, revision and supersession, temporal validity, ambiguity and abstention), and Appendix C.1 says the ontology was not fixed before the tasks existed. A fair test of whether explicit rules help diagnosis would need a scorer the predictor was not handed, or at least a Generic condition that receives the output vocabulary and qualifier definitions without the decision rules.

  • The ranking-reversal result that motivates the paper is too thin to carry that weight, and the underlying point is already established. Under the process view, the four methods score 2, 0, 3, and 3 out of 38. Ordering methods on differences of one to three cases, with no interval or paired test, is not a reversal anyone should act on, and the commitment column (21 to 26 out of 38) is barely separated either. The authors also note that all four methods were frozen under a contract that prioritizes terminal submissions, so their poor process-view scores are expected, not a finding. The broader observation, that a ranking depends on the estimand and on defensible analytical choices, is the benchmark-lottery, construct-validity, and multiverse-analysis argument the paper itself cites. For failure attribution specifically, MP-Bench already argues against a single deterministic target, and TelemetrySuffBench already tests when the evidence supports a unique origin, both cited in Section 6.2. The paper frames diagnostic-object identification as a new experimental variable, but the experiments mostly confirm that changing the target changes the score.

  • The validation tests show that the contract is followed. They do not show that it is the right target. The role swaps show that when the instruction exchanges two labels, a strong model exchanges them in its output. That is a clean instruction-following result, but it says nothing about whether EPCE's objects are what someone repairing an agent needs. Implementation stability is similarly limited. Strong executes the frozen code of the same semantic kernel that Exact SOL receives as text, so 213/217 agreement shows the model can execute a declarative procedure, not that the rules are reproducible from their description (the authors flag this as future work). Meanwhile the external reference was built in Codex tasks locked to gpt-5.6-sol, which is the same model as the predictors, and 196 of its 576 candidates were non-unanimous among the model reviewers. No human expert judges whether the admissible sets are sensible, there is no comparison with AgentRx's or TELBench's own gold annotations, and no experiment links an EPCE-correct diagnosis to a better repair, even though repair and incident investigation are the motivation in the introduction.

  • The evidence base is narrow, and the paper is hard to read. Every condition is a single frozen run of one model, and Principle-Aware is a single stochastic realization, so none of the casewise comparisons have run-to-run variance. The 720 internal trajectories come from 120 scenario templates crossed with three generators, and the full internal set is private, with only a 122-row public core released. The external set is 217 cases from two sources. On top of this, the writing leans on a large private vocabulary ("carrier-decomposition admissible hit", "semantic core-metric vector", custody digests, V5 derivatives), and the main text repeatedly qualifies what is not claimed. It took me several passes to work out what the main comparisons actually measure, which is a problem for a paper whose message is that evaluation targets should be stated clearly.

Questions for authors
  • How much of the 233 to 716 gain remains under a scoring that does not require EPCE's qualifier vocabulary, for example matching on event family plus a lenient check of source and field? Alternatively, can you run a Generic condition that receives the output vocabulary and qualifier definitions but not the decision rules?

  • For the reversal result, can you instruct each method for the process question, rerun them, and report paired intervals on the shared cases? With scores of 0 to 3 out of 38, it is unclear whether the reversal reflects anything beyond noise and the terminal-submission priority built into the frozen contract.

  • How does the EPCE external reference relate to the original gold annotations in AgentRx and TELBench? Where they disagree, which one would a practitioner prefer, and did any human expert look at a sample of the admissible sets?

  • Have you tried a different model family as predictor, reference builder, or both? With predictors and reference built from the same model, shared interpretive preferences could inflate Exact SOL's agreement with the reference, and that is hard to rule out from the current design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.