- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The central question in EW-GET is how finely an evaluator can distinguish failure mechanisms in a learned sensor-to-decision system when only part of its internal behavior is observable. Rather than assuming that all predefined failures can be identified separately, the authors propose determining the level of distinction supported by the available evidence.
Their framework compares the joint distributions of observable measurements, termed witnesses. Mechanisms with identical witness distributions belong to the same population-level equivalence class. For practical diagnosis, a distance threshold and the connected components of its associated graph define an operational partition. The paper proves that richer, nested observations can refine the population partition and gives conditions for recovering the operational partition from finite samples, together with an abstention rule. The proposed witnesses measure acquisition information magnitude, directional fidelity, and decision overclaim.
Controlled experiments recover the expected partitions under Black, Grey, and White access. In HM3D navigation, richer internal observations substantially improve prediction of injected failure labels, while diagnosis-guided repair improves overall Success and SPL under a fixed repair library. These embodied results concern predictive discrimination and repair utility; they do not establish certified recovery of the operational partition in HM3D.
Critical strengths
1. A useful distinction between failure prediction and what the available evidence can justify.
The paper raises a meaningful question: before assigning a failure label, which distinctions can actually be supported by the evaluator's observations? Separating population identifiability, a chosen operational resolution, and finite-sample certification makes the target of diagnosis more precise. I particularly appreciate the distinction between recovering a partition of failure mechanisms and identifying a single mechanism from an individual trajectory. The controlled results show that the former can succeed even when substantial per-trajectory ambiguity remains. This perspective is useful beyond the particular classification models evaluated here.
2. Careful controlled validation, with informative positive and negative results.
The experiments examine not only partition recovery but also the contribution of different witnesses, sensitivity to the distance threshold, calibration sample size, and abstention. Appendix E.3 is particularly helpful: its explicit acquisition model shows how different measurements become available under different access levels and support progressively finer partitions. Appendix F.5 also strengthens the practical evidence by comparing diagnosis-guided repair with a diagnosis-blind allocation that preserves the scene-wise counts of each repair operation. I appreciate that the paper reports unfavorable results rather than assuming that every identifiable failure can be repaired.
Critical weaknesses
1. The theoretical results do not yet establish a sufficiently substantial new insight.
The motivation is interesting, but determining which faults can be distinguished from available observations is an established problem in fault diagnosability. For example, Fu et al. (2019) study data-driven fault distinguishability in dynamic systems. Their setting differs from the learned sensor-to-decision systems considered here, so this prior work does not make EW-GET redundant. However, it does mean that the paper needs to explain more clearly what is fundamentally new about its treatment of the problem.
I find that distinction difficult to identify in the main theoretical results. Proposition 3 relies on a standard information-processing argument; Proposition 4 follows from marginalization of joint distributions; and Theorem 6 recovers the threshold-graph partition when the assumed uniform distance-estimation error is smaller than the separation margin. These statements are relevant and appear sound under their assumptions, but they do not reveal a particularly surprising property of learned systems. Nor do they explain why the proposed magnitude, fidelity, and overclaim witnesses are especially suitable for distinguishing failures: the main partition results would apply to many other choices of measurements.
The formalization is useful, but I remain unconvinced that it provides enough new theoretical understanding for a strong Learning Theory contribution.
2. The main embodied experiments do not establish the practical significance of the proposed diagnostic resolution.
The controlled experiments recover the intended partitions, but their failure mechanisms and witness differences are deliberately constructed. This validates the procedure under those conditions. The more difficult question is whether the proposed measurements support meaningful failure distinctions in learned systems where those distinctions have not been built into the experiment.
HM3D provides a more realistic test of the learned navigation model, but its main results concern prediction of injected failure labels. White access achieves high Macro F1, and a generic MLP using the same summaries performs similarly. These results show that the internal features are predictive, not that the operational partition defined by EW-GET has been recovered or certified. The repair experiments demonstrate useful allocation decisions, but do not establish the missing partition-recovery result.
Appendix F.6 makes this limitation concrete. The primary Habitat witness-validity assessment did not meet its predefined criteria: In the predefined Habitat task-grounding assessment, the mean within-scene correlation was 0.004, with a 95% confidence interval of [−0.341, 0.356]. Only nine scenes met the support requirements, below the required twelve. On the evaluated severity grid, none of the 2,772 tested four-family configurations satisfied all predefined task-performance matching criteria, so the corresponding mechanism-identification experiment was not performed.
These negative findings do not invalidate the controlled experiments or the reported repair gains. I appreciate that the authors disclose them. Nevertheless, they leave the most distinctive part of the framework, namely the connection from the proposed measurements to justified diagnostic resolution, insufficiently established in the main application. This matters more to my recommendation than adding further benchmarks or routine ablations.
References:
Fangzhou Fu, Dayi Wang, Linlin Li, Wenbo Li, and Zhigang Wu. Data-driven method for the quantitative fault diagnosability analysis of dynamic systems. IET Control Theory & Applications, 13(8):1197–1203, 2019.
Questions for authors
Q1 — What is the main new theoretical insight beyond existing fault-diagnosability results?
The paper provides a clear formalization, but its principal guarantees appear to follow from familiar properties of information processing, marginalization, and estimation error. Could the authors identify the most important theoretical conclusion that goes beyond these properties and explain how it differs from existing diagnosability analyses? In particular, does any central result depend on the proposed acquisition-grounded witnesses rather than holding for other suitable measurements?
A more convincing account of the theoretical contribution would materially affect my assessment.
Q2 — What evidence shows that the proposed diagnostic resolution adds value beyond failure-label prediction?
The controlled experiments recover the intended operational partitions, whereas the HM3D results primarily measure label prediction and repair outcomes. Given the unsuccessful Habitat witness-validity assessment in Appendix F.6, could the authors clarify which existing results establish that their proposed measurements support meaningful mechanism distinctions in the embodied setting?
In particular, how does the formal partition-certification procedure enter the reported HM3D diagnosis and repair decisions, and what conclusion does it support that ordinary failure-label prediction would not? This would help determine whether the central theory and its practical validation are sufficiently connected.