- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This is a position paper making the statement that the field of prognostics and health management (PHM) will never have a single benchmark with a single leaderboard that will lead to a paradigm shift in the technology. The contributions mostly consist of arguments in favor of this position, with a single dataset from aviation maintenance in Appendix A used as direct evidence and support.
Critical strengths
None
Critical weaknesses
- Overstated claims. The assertion that multimodal heterogeneity makes cross-modal pretraining and transfer effectively impossible (Section 2.2) is particularly questionable. Multimodal representation learning is an entire research field, and heterogeneous measurements do not intrinsically prevent useful transfer.
- A false dichotomy, where the authors first define the "ImageNet" moment in much more narrow terms than the ImageNet challenge actually represented, and then argue that this wouldn't be enough for their field. ImageNet is a large dataset which started from image classification labels, but its significance comes from the realization that improvement on image classification is usually indicative of the improvement on other computer vision tasks. Yes, there are more tasks in PHM, but there are also more tasks in computer vision. It doesn't mean that a single task can't be important for uncovering consequential conceptual advancement in the field.
- Another false dichotomy, contrasting universal, single number accuracy-only leaderboards with small, scenario-specific evaluations. Benchmarking does not need to be limited to optimizing a single accuracy metric. Benchmarks can evaluate robustness, calibration, computational efficiency, domain generalization, and any other criteria the domain experts consider important.
- Questionable novelty. The proposed alternatives, such as scenario-specific evaluation, Pareto comparisons, failure-mode testing, and deployment-oriented metrics, seem closely related to established practices across different application domains. I don't find them to be sufficiently novel for an ICLR publication.
Questions for authors
I don't have questions, the authors' position is clear.
3 comments
A partial bibliography on the term "ImageNet moment" (in case it is of use to the discussion.)
On the historical event:
- Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. Proceedings of CVPR 2009.
- Russakovsky, O., Deng, J., Su, H., et al. (2015). ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252.
On the term's career as community rhetoric:
- Ruder, S. (2018). NLP's ImageNet moment has arrived. The Gradient, July 12, 2018.
- Veysov, A. (2020). Towards an ImageNet moment for speech-to-text. The Gradient, March 28, 2020.
- Ahmed, N. (2023). Scientific labor market and firm-level appropriation strategy in artificial intelligence research. Mack Institute Research Paper, The Wharton School, University of Pennsylvania. (The paper uses the post-2012 "ImageNet moment" as a treatment shock in a difference-in-differences design; its Appendix A3 collects usages of the term across voice recognition, protein folding, and ChatGPT.)
- Zuo, X.-N. (2026). The BrainNet imperative: why global brain science needs its own ImageNet moment. Science Bulletin, 71(7), 1587–1590.
I thank the authors for their response. I remain with my score, I believe this paper is better suited to a specialist journal of their community.
We thank Reviewer iDkT for the review. For the convenience of the discussion period, we note at the outset which parts of the paper the review engages — the definitions of Sec. 1–2 and the genre question — and which it does not: the audit (Appendix A), the converging evidence reviewed in Sec. 2.3, the compensatory/non-compensatory analysis of Sec. 3.1, the constructive program of Sec. 4, and Sec. 5's argument for the AI4S audience. We address the four stated weaknesses in order.
1. Overstated claims (Sec. 2.2). As conceded in our response to Reviewer fZ1j (W2c), "effectively impossible" overstates our claim; we will rephrase to: no effective cross-modal pretraining–transfer protocol is known in PHM, with structural reasons for skepticism. The structural argument stands: physically distinct measurement channels share no representational space of the kind the ImageNet-pretraining narrative presupposes.
2. The definition of the ImageNet moment. The review holds that we "define the 'ImageNet' moment in much more narrow terms than the ImageNet challenge actually represented". The premise of the objection — that the term has a settled meaning from which our definition departs — does not hold. "ImageNet moment" is a rhetorical coinage: the earliest widely cited use (Ruder, 2018) already employs it as a cross-field metaphor for the arrival of a pretraining paradigm, and later usage has forked into operative definitions that do not coincide — deployment democratization (Veysov, 2020) being one prominent branch. A term of this kind has no canonical width to narrow; stipulating one's referent is the only available scholarly move, and Sec. 1 does so explicitly: "a single, well-defined central task; a homogeneous input modality; and abundant public data", with the historical account cited (Deng et al., 2009; Russakovsky et al., 2015). The reading the review defends — transfer across tasks — is not excluded by this stipulation; it is examined in Sec. 2.3. Lastly: the paper nowhere claims that other fields have not had, or cannot have, their ImageNet moments; Sec. 1 records as historical facts that 'vision, then language' inherited this template. The objection attributes to the paper a claim it does not make.
3. The second dichotomy (leaderboard criteria). The review argues that benchmarks "can evaluate robustness, calibration, computational efficiency, domain generalization". The paper does not claim otherwise; Sec. 3.1: such criteria appear "at best... as an afterthought column that nobody's ranking depends on". The claim of Sec. 3.1 concerns the decision rule, not the menu: leaderboard culture is compensatory (criteria collapse into a scalar or "hide it behind a weighted sum"), whereas industrial selection is non-compensatory, with hard thresholds that render a model inadmissible regardless of accuracy. This distinction — the analytical core of Sec. 3.1 and the design rationale of the two-stage protocol in Sec. 4 — is not addressed in the review.
4. Novelty. The review finds the alternatives "closely related to established practices across different application domains". Sec. 4 states its own frame: the four components are an "existence proof", "presented not as a deployment specification". The components are drawn from adjacent practice deliberately, because the position is that this practice — not a new invention — is the correct shape of progress in this field.
The review states that "the authors' position is clear" and poses no questions. We are glad the position is clear, and we hope the clarifications above support the discussion.