acceptodds
Under review as a conference paper at ICLR 2027

EW-GET: Access-Conditioned Identifiability of Failure Mechanisms in Sensor-to-Decision Models

Abstract

Failure diagnosis is fundamentally resolution-limited: which mechanisms can be distinguished depends on what an evaluator can observe. We therefore treat diagnostic resolution, rather than a prespecified failure label, as the primary statistical object. EW-GET formalizes this principle for learned sensor-to-decision systems. Exposed traces are mapped to acquisition-grounded witness distributions for information magnitude, directional fidelity, and decision overclaim. Equality of joint witness laws defines a population access-conditioned failure quotient; at finite resolution, direct operational indistinguishability induces a threshold graph whose connected components define the reportable partition. We prove that richer access refines the population quotient and give finite-sample recovery and calibrated-abstention guarantees for the declared operational partition. Controlled benchmarks recover the predicted Black/Grey/White refinement. In HM3D RGB-D navigation, richer interfaces sharply improve mechanism-label predictability, yet a generic MLP with comparable F1 does not by itself establish evidence-supported diagnostic resolution. Resolution-compatible intervention improves aggregate recovery under a frozen repair library, while adverse representation-family effects show that identifiability does not imply intervention sufficiency. EW-GET thus turns diagnosis into an access-conditioned measurement problem with an explicit population target, finite-sample reporting rule, and bounded downstream use.

Primary area:
learning theory
Keywords:
Access-Conditioned Identifiability, Diagnostic Resolution, Failure Quotients, Information Geometry, Sensor-to-Decision Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JPQgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The central question in EW-GET is how finely an evaluator can distinguish failure mechanisms in a learned sensor-to-decision system when only part of its internal behavior is observable. Rather than assuming that all predefined failures can be identified separately, the authors propose determining the level of distinction supported by the available evidence.

Their framework compares the joint distributions of observable measurements, termed witnesses. Mechanisms with identical witness distributions belong to the same population-level equivalence class. For practical diagnosis, a distance threshold and the connected components of its associated graph define an operational partition. The paper proves that richer, nested observations can refine the population partition and gives conditions for recovering the operational partition from finite samples, together with an abstention rule. The proposed witnesses measure acquisition information magnitude, directional fidelity, and decision overclaim.

Controlled experiments recover the expected partitions under Black, Grey, and White access. In HM3D navigation, richer internal observations substantially improve prediction of injected failure labels, while diagnosis-guided repair improves overall Success and SPL under a fixed repair library. These embodied results concern predictive discrimination and repair utility; they do not establish certified recovery of the operational partition in HM3D.

Critical strengths

1. A useful distinction between failure prediction and what the available evidence can justify.

The paper raises a meaningful question: before assigning a failure label, which distinctions can actually be supported by the evaluator's observations? Separating population identifiability, a chosen operational resolution, and finite-sample certification makes the target of diagnosis more precise. I particularly appreciate the distinction between recovering a partition of failure mechanisms and identifying a single mechanism from an individual trajectory. The controlled results show that the former can succeed even when substantial per-trajectory ambiguity remains. This perspective is useful beyond the particular classification models evaluated here.

2. Careful controlled validation, with informative positive and negative results.

The experiments examine not only partition recovery but also the contribution of different witnesses, sensitivity to the distance threshold, calibration sample size, and abstention. Appendix E.3 is particularly helpful: its explicit acquisition model shows how different measurements become available under different access levels and support progressively finer partitions. Appendix F.5 also strengthens the practical evidence by comparing diagnosis-guided repair with a diagnosis-blind allocation that preserves the scene-wise counts of each repair operation. I appreciate that the paper reports unfavorable results rather than assuming that every identifiable failure can be repaired.

Critical weaknesses

1. The theoretical results do not yet establish a sufficiently substantial new insight.

The motivation is interesting, but determining which faults can be distinguished from available observations is an established problem in fault diagnosability. For example, Fu et al. (2019) study data-driven fault distinguishability in dynamic systems. Their setting differs from the learned sensor-to-decision systems considered here, so this prior work does not make EW-GET redundant. However, it does mean that the paper needs to explain more clearly what is fundamentally new about its treatment of the problem.

I find that distinction difficult to identify in the main theoretical results. Proposition 3 relies on a standard information-processing argument; Proposition 4 follows from marginalization of joint distributions; and Theorem 6 recovers the threshold-graph partition when the assumed uniform distance-estimation error is smaller than the separation margin. These statements are relevant and appear sound under their assumptions, but they do not reveal a particularly surprising property of learned systems. Nor do they explain why the proposed magnitude, fidelity, and overclaim witnesses are especially suitable for distinguishing failures: the main partition results would apply to many other choices of measurements.

The formalization is useful, but I remain unconvinced that it provides enough new theoretical understanding for a strong Learning Theory contribution.

2. The main embodied experiments do not establish the practical significance of the proposed diagnostic resolution.

The controlled experiments recover the intended partitions, but their failure mechanisms and witness differences are deliberately constructed. This validates the procedure under those conditions. The more difficult question is whether the proposed measurements support meaningful failure distinctions in learned systems where those distinctions have not been built into the experiment.

HM3D provides a more realistic test of the learned navigation model, but its main results concern prediction of injected failure labels. White access achieves high Macro F1, and a generic MLP using the same summaries performs similarly. These results show that the internal features are predictive, not that the operational partition defined by EW-GET has been recovered or certified. The repair experiments demonstrate useful allocation decisions, but do not establish the missing partition-recovery result.

Appendix F.6 makes this limitation concrete. The primary Habitat witness-validity assessment did not meet its predefined criteria: In the predefined Habitat task-grounding assessment, the mean within-scene correlation was 0.004, with a 95% confidence interval of [−0.341, 0.356]. Only nine scenes met the support requirements, below the required twelve. On the evaluated severity grid, none of the 2,772 tested four-family configurations satisfied all predefined task-performance matching criteria, so the corresponding mechanism-identification experiment was not performed.

These negative findings do not invalidate the controlled experiments or the reported repair gains. I appreciate that the authors disclose them. Nevertheless, they leave the most distinctive part of the framework, namely the connection from the proposed measurements to justified diagnostic resolution, insufficiently established in the main application. This matters more to my recommendation than adding further benchmarks or routine ablations.

References:

Fangzhou Fu, Dayi Wang, Linlin Li, Wenbo Li, and Zhigang Wu. Data-driven method for the quantitative fault diagnosability analysis of dynamic systems. IET Control Theory & Applications, 13(8):1197–1203, 2019.

Questions for authors

Q1 — What is the main new theoretical insight beyond existing fault-diagnosability results?

The paper provides a clear formalization, but its principal guarantees appear to follow from familiar properties of information processing, marginalization, and estimation error. Could the authors identify the most important theoretical conclusion that goes beyond these properties and explain how it differs from existing diagnosability analyses? In particular, does any central result depend on the proposed acquisition-grounded witnesses rather than holding for other suitable measurements?

A more convincing account of the theoretical contribution would materially affect my assessment.

Q2 — What evidence shows that the proposed diagnostic resolution adds value beyond failure-label prediction?

The controlled experiments recover the intended operational partitions, whereas the HM3D results primarily measure label prediction and repair outcomes. Given the unsuccessful Habitat witness-validity assessment in Appendix F.6, could the authors clarify which existing results establish that their proposed measurements support meaningful mechanism distinctions in the embodied setting?

In particular, how does the formal partition-certification procedure enter the reported HM3D diagnosis and repair decisions, and what conclusion does it support that ordinary failure-label prediction would not? This would help determine whether the central theory and its practical validation are sufficiently connected.

Reviewer_X2jDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose EW-GET, a method to determine which failure mechanisms can be statistically distinguished given the available observations in autonomous sensor-to-decision systems (e.g. autonomous robotic navigation). After proposing a mathematical formulation of failure distinguishability and defining the scope of the problem (i.e. assumptions), authors validate partition recovery in controlled settings and evaluate fault-diagnosis accuracy on a real-world dataset (HM3D).

Critical strengths

-The task is of general interest, and the formalization of certain aspects is interesting -The proposed mathematical framework (section 2.1) is sound. The equivalence relations, quotient spaces, threshold graphs make sense. -The experiments demonstrate the intended behavior of EW-GET in the controlled experiments.

Critical weaknesses

-Poor writing quality: The writing quality is poor, the flow of ideas is unclear, and the paper is packed with jargon. The abstract is particularly difficult to read and understand. -Insufficient coverage of recent relevant literature: The references are significantly outdated. The most recent reference is from 2023 (DINOv2) and is only a supporting reference. Overall, there are no citations addressing the core problem referenced from 2021 onward. Examples of relevant literature are: “Fault Diagnosis of Discrete-Event Systems from Abstract Observations” by Lamberti et al. (Computing and Informatics, 2022), “MD Tree: A Model-Diagnostic Tree Grown on Loss Landscape” by Zhou et al. (ICML 2024) and “Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems” by Zhang et al. (ICML 2025). -Weak novelty under the stated assumptions: The proposed method requires that all estimated distances between distributions are sufficiently accurate (assumption 5) and all true distances are sufficiently far from the threshold (theorem 6). Given these assumptions, the authors prove that thresholding the estimated distances gives the correct threshold decisions. This is simply a standard perturbation-stability argument, while the main statistical challenge is deferred to Assumption 5. Those are straightforward consequences of standard probability principles. -Assumptions are too strong: Assumption 5 means that all pairwise distances between witness distributions are simultaneously accurate within a known error bound. The combination of Assumption 5 with the margin condition of Theorem 6 makes the core of the method difficult in practice. Those are very strong assumptions/requirements, particularly in complex autonomous systems, which substantially limits the practical significance. -Limited real-world applicability: Proposition 4 makes sense under the nested-access assumption. However, it doesn’t automatically guarantee monotonic improvements in the actual diagnostic procedure. In real-world applications, richer observational access may increase estimation complexity and calibration requirements, so the population-level refinement guaranteed by Proposition 4 does not necessarily translate into improved finite-sample diagnostic performance. -Calibration claims are misleading: The calibration set checks that two failure mechanisms statistically distinguishable at the population level. However, recovering the boundary between the two mechanisms does not imply that an individual failure can be diagnosed confidently. -Results on HM3D do not validate the method’s claims: The HM3D experiments validate failure-label predictability, not the paper’s central statistical partition-recovery framework. Those results validate that more internal information makes the injected failure labels easier to predict, but they do not demonstrate recovery of the proposed operational quotient in HM3D. -Limited evaluations: the experiments should be extended to cover more commonly used fault diagnostic benchmarks.

Questions for authors

-Theorem 6 guarantees exact recovery of the finite-resolution operational partition under Assumption 5 and a separation margin. How can these conditions be verified or justified in realistic autonomous systems? Can the authors provide empirical estimates of the estimation error, separation margins, and calibration sample sizes required for partition recovery? -Can the authors explain why partition recovery was not evaluated on the HM3D dataset? Can the authors provide evidence that the proposed statistical framework remains effective beyond the controlled experiments? -How does successful partition recovery translate into reliable per-trace diagnosis, given the substantial abstention rates observed in the controlled experiments? -How can the authors establish that their chosen witnesses provide meaningful diagnostic distinctions, given the inconclusive Habitat validation? (Appendix F.6)

Reviewer_YobpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EW-GET treats diagnosis as a question of what distinctions are justified by the signals an evaluator can see. For a fixed finite set of candidate failure mechanisms, access profile, and witness functions, the paper groups mechanisms whose witness distributions are identical. With a nonzero tolerance, it instead forms a threshold graph and reports connected components. The paper proves that more access can only refine the population partition, and gives a conditional finite-sample recovery result. It evaluates the idea on an injected-failure benchmark and RGB-D PointNav in HM3D, where more internal access improves mechanism-label prediction and guides a fixed set of repair actions.

Critical strengths
  1. The paper cleanly separates population equality, a chosen finite-resolution partition, and a finite-sample estimate. That separation, together with abstention, is more precise than training a fault classifier and treating its accuracy as diagnosis.

  2. The authors go beyond a classifier leaderboard. They include nested-access experiments, witness ablations, leakage controls, and a repair experiment. They also correctly state that being able to distinguish a failure does not establish causal root cause or guarantee a successful repair.

Critical weaknesses
  1. The proposed “finest identifiable partition” is tied to a closed list of injected mechanisms and to selected witnesses. In the controlled benchmark, the Black/Grey/White partition is built into the data-generating setup, so perfect recovery mainly shows that the procedure recovers a known construction. In HM3D, high label-prediction accuracy could still be driven by simulator signatures, trajectory patterns, or outcomes that correlate with the injected label rather than by evidence about the actual failure mechanism. The current experiments do not rule this out.

  2. The main theorem needs a uniform distance estimator, independent calibration groups, and a separation margin. These assumptions are not established for the dependent, high-dimensional HM3D trajectories; the paper itself says its HM3D intervals are not operational-partition recovery guarantees. That leaves a gap between the clean theoretical story and the real-system evidence. The repair results are also limited to a fixed repair library and same-seed reset retries.

Questions for authors
  1. Can you test whether the witness contract and declared resolution transfer to unseen failure processes, mixtures of mechanisms, or naturally occurring failures? Controls that preserve trajectory/outcome statistics while changing the causal mechanism would be particularly useful.

  2. What concrete estimator and dependence assumptions would let Assumption 5 hold in HM3D? If that cannot be instantiated, please frame the real-system result as injected-label prediction rather than as certified operational diagnosis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.