- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This is an ambitious and intellectually interesting paper. Its central claim is not about improving agent diagnosis per se, but about diagnostic specification: the observation that a failure trajectory does not uniquely determine what a diagnosis ought to identify. The paper argues that the diagnostic target itself is an experimental variable, and that changing the specification can reverse rankings while leaving trajectories and predictions unchanged. In my view, this is potentially a publishable ICLR contribution, but the paper currently sits in an unusual position: The empirical result is strong. The methodological point is important. The formalism is weaker than the experimental story. The paper sometimes overstates what its experiments establish. The underlying theoretical object remains somewhat under-specified.
Critical strengths
The empirical result is strong. The methodological point is important.
Critical weaknesses
The formalism is weaker than the experimental story. The paper sometimes overstates what its experiments establish. The underlying theoretical object remains somewhat under-specified.
The paper appears to assume precisely the object whose variability it wishes to study. Without a prior theory of what constitutes a diagnosis and what constitutes a valid contract, the ranking-reversal result risks collapsing into the observation that changing the definition of success changes who wins. The entire contribution then hinges on whether the proposed contract family has an independent theoretical justification, and that justification does not appear to be developed sufficiently. The paper studies performance under a family of diagnostic contracts, but neither diagnosis nor contract is formally characterized. As a result, the significance of the observed ranking reversals is unclear. The results establish that rankings depend on the evaluation semantics chosen by the authors, but they do not establish whether this dependence reflects a fundamental property of diagnosis or merely the freedom to redefine the target concept. The paper appears to have the following structure: Consider a diagnosis method M. Evaluate it under contract C. Obtain ranking R_C. Change contract. Obtain ranking R_(C^' ). Observe R_C≠R_(C^' ). The conclusion is then: rankings are specification-sensitive. But to make that conclusion meaningful, one needs at least: a formal definition of diagnosis, a formal definition of contract, a characterization of permissible contracts. Otherwise there is an immediate objection: If I am allowed to arbitrarily redefine the target concept, then ranking changes are inevitable. In other words, the result becomes almost tautological. Suppose I define diagnosis as: root cause identification, and then redefine it as: first observed error, and then redefine it as: most repairable error, and then redefine it as: shortest explanatory description. Of course rankings may change. The paper then appears to demonstrate: "Different tasks produce different rankings".
which is true but not very informative.
A useful theory would need to answer: What properties must a diagnosis satisfy? Or: What class of concepts count as diagnoses? The paper instead seems to choose several examples and call them contracts. The problem is that without constraints on admissible contracts there is no nontrivial theorem available. For example, any method Acan be made superior to method Bby constructing a contract that rewards exactly the outputs produced by A. Thus: ∀A,B," "∃C
such that A>_C B.
If arbitrary contracts are allowed, ranking reversal is not surprising. It is unavoidable. A reviewer could therefore ask: What prevents the result from being a consequence of unrestricted relabeling? That question is not obviously answered.
The paper also seems to oscillate between two claims: Claim A "Diagnosis has no unique definition." and Claim B "We have discovered an important new diagnostic evaluation framework." These are in tension. If Claim A is correct, then the burden is on the authors to explain why their chosen framework should be preferred. If diagnosis genuinely lacks a canonical meaning, then EPCE is merely one possible semantics among many. The paper then becomes an ontology proposal rather than a scientific discovery. To justify the significance of ranking reversals, one must argue that the contract family being studied captures something essential about diagnosis. I did not see a convincing argument for that.
Questions for authors
- define contract precisely
- define diagnosis precisely
- define an order relation precisely over contracts/diagnoses
- How is this work related to theoretical definitions of diagnosis (of which there are a large number)?