acceptodds
Under review as a conference paper at ICLR 2027

Position: Why AI For PHM May Never Have Its "ImageNet Moment" (and Why It Probably Shouldn't)

Abstract

The "ImageNet moment”—a historical convergence in which a unified benchmark triggers methodological unification and discipline-wide reorganization—has become a default aspiration for ML entering new domains. We argue that the data-driven machine learning paradigm within prognostics and health management (PHM) may never, and should not, experience such a moment. Structural constraints including data inaccessibility, irreducible modality heterogeneity, and the absence of a dominant predictive architecture make the preconditions for such convergence structurally absent; an ImageNet-style leaderboard is therefore neither achievable nor desirable for physical-world industrial tasks. Instead, PHM requires small-scale, diagnostic, scenario-specific evaluation protocols that respect domain specificity over generic ranking. This position has implications beyond PHM: as ML expands into AI for Science and Engineering, the assumption that universal benchmarks are the natural endpoint of methodological maturity must be re-examined.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
benchmarking and evaluation, evaluation methodology, prognostics and health management (PHM), AI for science, AI for engineering, industrial machine learning, predictive maintenance, time-series classification, remaining useful life, position paper

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iDkTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This is a position paper making the statement that the field of prognostics and health management (PHM) will never have a single benchmark with a single leaderboard that will lead to a paradigm shift in the technology. The contributions mostly consist of arguments in favor of this position, with a single dataset from aviation maintenance in Appendix A used as direct evidence and support.

Critical strengths

None

Critical weaknesses
  • Overstated claims. The assertion that multimodal heterogeneity makes cross-modal pretraining and transfer effectively impossible (Section 2.2) is particularly questionable. Multimodal representation learning is an entire research field, and heterogeneous measurements do not intrinsically prevent useful transfer.
  • A false dichotomy, where the authors first define the "ImageNet" moment in much more narrow terms than the ImageNet challenge actually represented, and then argue that this wouldn't be enough for their field. ImageNet is a large dataset which started from image classification labels, but its significance comes from the realization that improvement on image classification is usually indicative of the improvement on other computer vision tasks. Yes, there are more tasks in PHM, but there are also more tasks in computer vision. It doesn't mean that a single task can't be important for uncovering consequential conceptual advancement in the field.
  • Another false dichotomy, contrasting universal, single number accuracy-only leaderboards with small, scenario-specific evaluations. Benchmarking does not need to be limited to optimizing a single accuracy metric. Benchmarks can evaluate robustness, calibration, computational efficiency, domain generalization, and any other criteria the domain experts consider important.
  • Questionable novelty. The proposed alternatives, such as scenario-specific evaluation, Pareto comparisons, failure-mode testing, and deployment-oriented metrics, seem closely related to established practices across different application domains. I don't find them to be sufficiently novel for an ICLR publication.
Questions for authors

I don't have questions, the authors' position is clear.

3 comments

We thank Reviewer iDkT for the review. For the convenience of the discussion period, we note at the outset which parts of the paper the review engages — the definitions of Sec. 1–2 and the genre question — and which it does not: the audit (Appendix A), the converging evidence reviewed in Sec. 2.3, the compensatory/non-compensatory analysis of Sec. 3.1, the constructive program of Sec. 4, and Sec. 5's argument for the AI4S audience. We address the four stated weaknesses in order.

1. Overstated claims (Sec. 2.2). As conceded in our response to Reviewer fZ1j (W2c), "effectively impossible" overstates our claim; we will rephrase to: no effective cross-modal pretraining–transfer protocol is known in PHM, with structural reasons for skepticism. The structural argument stands: physically distinct measurement channels share no representational space of the kind the ImageNet-pretraining narrative presupposes.

2. The definition of the ImageNet moment. The review holds that we "define the 'ImageNet' moment in much more narrow terms than the ImageNet challenge actually represented". The premise of the objection — that the term has a settled meaning from which our definition departs — does not hold. "ImageNet moment" is a rhetorical coinage: the earliest widely cited use (Ruder, 2018) already employs it as a cross-field metaphor for the arrival of a pretraining paradigm, and later usage has forked into operative definitions that do not coincide — deployment democratization (Veysov, 2020) being one prominent branch. A term of this kind has no canonical width to narrow; stipulating one's referent is the only available scholarly move, and Sec. 1 does so explicitly: "a single, well-defined central task; a homogeneous input modality; and abundant public data", with the historical account cited (Deng et al., 2009; Russakovsky et al., 2015). The reading the review defends — transfer across tasks — is not excluded by this stipulation; it is examined in Sec. 2.3. Lastly: the paper nowhere claims that other fields have not had, or cannot have, their ImageNet moments; Sec. 1 records as historical facts that 'vision, then language' inherited this template. The objection attributes to the paper a claim it does not make.

3. The second dichotomy (leaderboard criteria). The review argues that benchmarks "can evaluate robustness, calibration, computational efficiency, domain generalization". The paper does not claim otherwise; Sec. 3.1: such criteria appear "at best... as an afterthought column that nobody's ranking depends on". The claim of Sec. 3.1 concerns the decision rule, not the menu: leaderboard culture is compensatory (criteria collapse into a scalar or "hide it behind a weighted sum"), whereas industrial selection is non-compensatory, with hard thresholds that render a model inadmissible regardless of accuracy. This distinction — the analytical core of Sec. 3.1 and the design rationale of the two-stage protocol in Sec. 4 — is not addressed in the review.

4. Novelty. The review finds the alternatives "closely related to established practices across different application domains". Sec. 4 states its own frame: the four components are an "existence proof", "presented not as a deployment specification". The components are drawn from adjacent practice deliberately, because the position is that this practice — not a new invention — is the correct shape of progress in this field.

The review states that "the authors' position is clear" and poses no questions. We are glad the position is clear, and we hope the clarifications above support the discussion.

AuthorsOpenReview ↗

A partial bibliography on the term "ImageNet moment" (in case it is of use to the discussion.)

On the historical event:

  1. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. Proceedings of CVPR 2009.
  2. Russakovsky, O., Deng, J., Su, H., et al. (2015). ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252.

On the term's career as community rhetoric:

  1. Ruder, S. (2018). NLP's ImageNet moment has arrived. The Gradient, July 12, 2018.
  2. Veysov, A. (2020). Towards an ImageNet moment for speech-to-text. The Gradient, March 28, 2020.
  3. Ahmed, N. (2023). Scientific labor market and firm-level appropriation strategy in artificial intelligence research. Mack Institute Research Paper, The Wharton School, University of Pennsylvania. (The paper uses the post-2012 "ImageNet moment" as a treatment shock in a difference-in-differences design; its Appendix A3 collects usages of the term across voice recognition, protein folding, and ChatGPT.)
  4. Zuo, X.-N. (2026). The BrainNet imperative: why global brain science needs its own ImageNet moment. Science Bulletin, 71(7), 1587–1590.

I thank the authors for their response. I remain with my score, I believe this paper is better suited to a specialist journal of their community.

Reviewer_Pv32OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper is a position paper that focusing on how to correctly perform PHM (prognostics and health management, which uses signals from machine/equipment to predict future failure e.t.c.) research under the current AI hype. The author claims that PHM has data scarcity, no unified embodiment/signal representation, no dominant model family. Building a unifed benchmark like imageNet for PHM is difficult and 1) not representing real scenarios and 2) and not good for community because researchers will try to overfit it. The author suggest the community should focuse on small-scale, diagnostic, scenario-specific evaluation protocols that respect domain specificity over generic ranking

Critical strengths

The paper pointed out an important problem in PHM research, also seen in many AI4Science tasks. As the AI hype continues, learning a foundation/unified model and building a large-scale benchmark for a domain becomes main research focus. The author claims that this kind of direction is not always correct. The success in LLM/Vision is not going to happen in the PHM domain.
I agree with most of the authors argument, and I don't see clear logical flaws based on my understanding of time-series analysis and AI. This is the major strength.

Critical weaknesses
  1. The major weakness is that this paper is mostly the authors logical thinking and personal claim. Author cited a few external work as evidence but very weak. So I won't say the paper is rigorously supported by evidence. It is rather a paper with logical thinking based on the authors experiences. This is the major weakness and I don't see a clear guideline of how ICLR rates a position paper.
  2. Another weakness is that the authors claim may not be true and may not point the community to the right direction. In a data-driven/machine learning domain, although logically we can imagine that diverse and heterogeneous PHM data may not have human-understandable synergy, it does not prove that the synergy does not exist. Intellegence emerged from scaling, and a unified benchmark, even those entities theoretically share nothing in common, is still a valueable step towards scaling and worth exploring. Maybe the author is trying to say that this direction is over-rated and over-hyped and needs some more meaningful research, but I won't treat this as a rigorous paper that can be published on ICLR. The effort should be pushing for open-data, and experiments for scaling.
Questions for authors

None.

1 comment

We thank Reviewer Pv32 for this review, including the confidence declaration, which we read as a model of procedural honesty, and the explicit statement that the reviewer agrees with most of our argument. We address the two weaknesses in turn.

On how ICLR rates a position paper. ICLR has no dedicated position track, and the reviewer guidelines do not provide a genre-specific rubric — the gap the reviewer has identified is real, and it is the venue's, not the reviewer's. The community has nonetheless developed reference points: NeurIPS now runs a dedicated position paper track, and ICML's position track states as its criterion that a position be substantive and falsifiable. We ask to be evaluated on those terms: the importance of the thesis, the quality of argumentation, and the engagement with counterarguments — the last evidenced by our Alternative Views section, which answers five objections at thesis level. The position is falsifiable in a concrete sense: a single public PHM dataset exhibiting sustained maintenance and community convergence would overturn it.

On scaling and open data. We share the reviewer's enthusiasm for scaling experiments, and we note that this field has run them; the results are on record in Sec. 2.3: parameter-free methods statistically matching deep architectures on the standard time-series archives (Dempster et al., 2021), large-scale forecasting benchmarks where linear models outperform recently proposed deep models (Qiu et al., 2024), theoretical results on transformer failure modes in autoregressive time series, and PHM data-competition winners built on kernel regression, similarity matching, and feature-engineered ensembles (Zhao et al., 2024). The pattern these experiments reveal is why our position questions not scaling research but its evaluation culture. On open data: Sec. 2.1 argues that the scarcity is structural — owners of run-to-failure data have rational, permanent reasons not to release it — and Sec. 4 describes what can be built instead, with synthetic data of controlled failure coverage first among the options.

On the concern that the argument rests largely on logical structure: we agree that position papers must be anchored to evidence, and this one is — the NGAFID audit (Appendix A), the independent CWRU re-audit (Hendriks et al., 2022), the quantitative community statistics cited in Sec. 7, and the mechanisms analysis we develop in our responses. We are committed to strengthening this anchoring in the revision regardless of the decision.

We are grateful for the reviewer's engagement and for the agreement with the argument's core; we hope this response supplies the rubric dimensions the review was missing.

Reviewer_fZ1JOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This position paper argues that machine learning for prognostics and health management (PHM), which predicts failures in industrial equipment, will not have an "ImageNet moment", and should not want one. By an "ImageNet moment", the authors mean a point where a whole field comes together around one large public benchmark and one dominant model architecture. The paper makes two arguments. First, it argues that this cannot happen in PHM: companies keep their equipment data private, the public data that exists is too limited to tell methods apart, PHM data comes from many different types of sensors, and no single model architecture works best. Second, it argues that this should not happen: a leaderboard ranks models mainly by accuracy, but in industry a model is unusable if it fails basic requirements, such as running fast enough or not missing too many faults, and what matters most is reliable performance on a specific machine rather than good average performance.

Critical strengths
  • The paper is clearly written and accessible to ML researchers with no PHM background.
  • The distinction between leaderboard ranking and industrial selection is useful. Asking "which models meet the hard constraints, and which trade-off suits this asset?" instead of "which model scores highest?" is a practical way to frame evaluation.
  • The paper takes counterarguments and responds to five of them in Sec. 7.
Critical weaknesses

W1. The paper presents what the field is already doing as if it were a new position.

The paper’s own text shows that its main claims are already accepted and its proposals are already in use:

The paper says the field is already shifting from proposing new methods to improving evaluation (Sec. 1). The paper cites earlier work that already proposed evaluation in which a model is rejected if it fails practical requirements, however accurate it is (Salinas-Camus et al., 2025), and says that "the field already knows it" (Sec. 3.1). The paper says the time-series community has already reached the same conclusion that no single architecture dominates (Sec. 2.3). The paper says that industry already tests models on its own specific use cases, and that its proposal "merely makes this practice explicit" (Sec. 4).

A position paper is useful when it challenges a view that people actually hold. Here, the paper does not show that anyone is arguing for a single ImageNet-style benchmark for PHM, and its recommendations match what it says the field is already doing. Presenting these points as a new position does not add a new claim. The authors should either show who holds the opposing view and what harm it causes, or state clearly what the paper adds beyond describing an existing trend.

W2. The main claims are not supported by enough evidence.

  • Only one dataset is tested. The paper’s only experiment uses a single dataset (NGAFID). From this, the paper concludes that public PHM data in general cannot be used for meaningful benchmarking ("we do not believe that this dataset is an outlier", Sec. 2.1). However, no other public PHM dataset is examined, including the bearing datasets the paper itself mentions. Testing a few more datasets would be needed to support this general claim.
  • Close scores do not mean the data is useless. The paper argues that because the top methods score within 0.01 F1 of each other, the dataset has "no discriminative power" (Sec. 2.1). But small gaps between the best methods are common in many benchmarks, and they do not show that the data cannot rank methods. The paper does not explain why close scores here should mean the benchmark is uninformative.
  • The claim about transfer learning has no support. The paper states that pretraining on one sensor type and transferring to another is "effectively impossible" (Sec. 2.2), but gives no experiment or reference showing this. This claim could be tested directly, for example by pretraining on vibration data and fine-tuning on current or temperature data.

The authors should either add evidence for these claims or state them as hypotheses rather than conclusions.

W3. The proposals are described only in general terms.

Sec. 4 calls its proposals an "existence proof", but each one is stated as a general direction without the details needed to use it:

  • The paper does not say which hard constraints should be used in two-stage evaluation, how their thresholds should be set, or who should set them. It also does not say which metrics should appear in the Pareto comparison.
  • The paper does not define what counts as a "scenario" in small scenario-specific benchmarks, how many benchmarks are needed, or who would build and maintain them.

Without these details, it is hard to see how a researcher would act on the proposals. A worked example would help considerably, for example applying the two-stage evaluation to the NGAFID audit and showing how its conclusions differ from ranking by F1.

Questions for authors

Please see the Weaknesses section, where the questions are included.

1 comment

We thank Reviewer rZ1j for a careful, textually grounded review. W1–W3 in order.

W1. W1 conflates four statuses the paper keeps distinct: (1) a trend (Sec. 1), descriptive, and precisely the trend whose naive extrapolation we argue against; a trend is not a position; (2) a literature proposal (Salinas-Camus et al., 2025): "the field already knows it" (Sec. 3.1) means the insight exists in the literature ("the convergence is not ours alone"), not that it is practiced — a 2025 review still proposing this evaluation ("we propose a user-oriented, noncompensatory framework") is evidence it is not standard; (3) an empirical consensus (Sec. 2.3), an input rather than a target; (4) industrial practice (Sec. 4, "merely makes this practice explicit") — our target is the research community's culture, quantified in Sec. 7 (under 1% of surveyed PHM papers release code and data; von Hahn & Mechefske 2023). Awareness and adoption are different claims.

On who holds the opposing view: Sec. 1 already cites early benchmarking efforts (Hendriks et al., 2022; Zhao et al., 2024); the most recent instance is PDMBench (ICLR 2026 submission, oJhj8wOCNB) , a platform unifying 14 datasets, 22 models, and a public ranking board, described by its authors as "what ImageNet provided for computer vision and GLUE provided for NLP". The view exists concretely, on this very platform; that work's own results ("no universal best model") even corroborate Sec. 2.3. We will cite it in revision.

The implied criterion — superfluous given prior mentions — would rule out the genre itself. Our position is falsifiable: one public PHM dataset with a sustained maintenance-and-convergence trajectory would overturn it. None exists to date.

W2a. The NGAFID audit is a case study of a class: the generation of public datasets emerged in recent years ; newer releases such as SCANIA Component X and UTDTB v5; and competition datasets. The claim concerns the class's trajectory, not any member's quality: release, a burst of attention, then neglect, with none of its members exhibits, to date, community convergence of the kind a durable benchmark presupposes. Had one, it would be a counterexample. Hence no enumeration: listing members cannot bear on a claim about a lifecycle. For the older generation, Hendriks et al. (2022) independently re-audit CWRU, same failure class; newest and most-studied meet at the same conclusion.

Three mechanisms account for this regularity (analysis since submission): (i) publication-as-endpoint (real-world datasets released as finished artifacts, unmaintained); (ii) incumbency lock-in (old datasets are citation infrastructure; new ones must justify against the old, and never displace them); (iii) no bust-and-replace (an ImageNet successor inherits a shared problem space; a PHM dataset occupies a small, incommensurate slice — a consequence of Sec. 2.2 — so nothing inherits anything). An instrument's value presupposes a frozen protocol. These extend Sec. 2.1 ("you cannot standardize measurement on an instrument that does not exist") and converge on Sec. 4: evaluation returns to diagnostic value.

W2b. The claim does not rest on the F1 band alone; three observations converge: seven methods collapse into 0.63–0.76 F1 (top two within 0.01); only latency separates methods cleanly (three orders of magnitude); on the original multi-class task, classifiers collapse to majority classes >95% of the time. A benchmark whose rankings are dominated by non-methodological factors "measures noise, hardware, and luck" (Sec. 2.1), however common close scores are elsewhere.

W2c. Conceded: "effectively impossible" overstates our claim. Intended: no effective cross-modal transfer protocol is known, and Sec. 2.2 gives structural reasons (physically distinct channels share no representational space). The proposed test is welcome; our position predicts it will underperform within-modality baselines. We will rephrase.

W3. Sec. 4 sketches contours deliberately (an existence proof, not a deployment spec), but the requested worked example is computable from our audit. Edge budget 1 ms: six of seven methods inadmissible (best latency 2.8 ms); the two-stage choice is LogReg (0.629 F1, 0.1 ms) — last on the F1 leaderboard. At 5 ms: admissible {LogReg, InceptionTime, ConvMHSA}; LogReg→InceptionTime buys +0.119 F1 for 28× latency; InceptionTime→ConvMHSA buys +0.008 F1 for 1.1× — inside the ±0.009 standard error, i.e., noise. Full Pareto disclosure also surfaces CNN+BiMamba's ±0.112 instability. Under realistic budgets the two-stage protocol and the F1 leaderboard order methods oppositely; this inversion is the paper's claim. We will integrate this example.

The W2c rephrasing and the mechanisms paragraph are committed revisions, regardless of the decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.