- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates a constraint of Joint-Embedding Predictive Architectures (JEPAs) in medical imaging termed predictability bias—the tendency of point-prediction objectives to summarize the conditional target distribution and discard context-unpredictable variations. By pretraining a large-scale 3D CT model (TAP-JEPA) and auditing its predictive pathway, the authors demonstrate that training objectives should retain localized diagnostic features beyond what is predictable from surrounding anatomy.
Critical strengths
- Timely topic with large-scale empirical validation: As JEPA methods are increasingly adopted in medical image analysis, investigating their inductive biases, utility scope, and limitations on a competitive model pretrained on over 113k CT volumes is timely and practically relevant.
- Clarity and organization: The manuscript is well organized, clearly written, and easy to follow from theoretical formulation to empirical audits.
Critical weaknesses
- Limited conceptual novelty and over-attribution to JEPA: Predictability bias (Proposition 1 and Lemma 1) essentially renames a well-known property that point predictions regress to the conditional mean/median and discard context-orthogonal residuals—making the empirical finding that sporadic lesions are smoothed toward normal tissue largely unsurprising. Moreover, this mathematical property is not unique to JEPA; it applies equally to Masked Image Modeling (e.g., MAE, SimMIM) and other context-prediction methods, yet the paper frames it specifically around JEPA without auditing non-JEPA baselines for comparison.
- Insufficient final-layer token removal in the downstream intervention: In Section 5.6 (Table 3), measuring reliance on local pathology by deleting pathology-overlapping tokens only at the final layer is insufficient. Across a 12-layer ViT, self-attention continuously mixes information across tokens, allowing diagnostic signals from lesion tokens to diffuse into surrounding context tokens before the final layer. Removing or masking the target pathology tokens at the input level is needed to draw valid conclusions.
Questions for authors
Impact of partial labeling in ReXGroundingCT: Throughout Sections 5.3–5.6, non-annotated regions in the same organ/lobe are treated as healthy controls. Since ReXGroundingCT is partially labeled (and thoracic findings often involve multifocal lesions or diffuse parenchymal changes outside the bounding masks), how does this partial labeling affect the conclusions? Specifically, could unannotated lesions contaminate the "matched controls" in Sections 5.3–5.5, and could remaining unannotated pathology tokens explain why deleting only the annotated tokens in Section 5.6 barely reduces the scan-level decision margin?
3 comments
Thank you for the thorough rebuttal. The newly added experiments, analysis and clarification addresses my main soundness concerns. While I still view the theoretical framing as somewhat standard, the empirical audit is solid, so I maintain my positive score (3).
Thank you for the constructive engagement and for maintaining your positive assessment. We appreciate your recognition that the additional experiments and clarifications address your main soundness concerns. The novelty we wish to emphasize is not the standard point-prediction characterization itself, but identifying and systematically auditing a mismatch between contextual predictability and clinical relevance. Diagnostically important pathology need not be predictable from surrounding anatomy. Our audit demonstrates that strong scan-level performance can coexist with substantial attenuation of pathology-related information along the learned predictive pathway.
We thank the reviewer for recognizing the timeliness of the topic, the large-scale empirical validation, and the clarity and organization of the manuscript.
W1: Novelty and attribution to JEPA.
We agree that Proposition 1 and Lemma 1 are standard properties of point prediction, and the revised Section 3 presents them as such more clearly. Our contribution is to connect them to the JEPA training signal, derive testable consequences for anatomy and pathology, and demonstrate these consequences in a model trained on 113,611 CT volumes and evaluated across two independent datasets.
We also agree that the property is not specific to JEPA, as the Scope paragraph in Section 5.6 already noted. It applies to any point-prediction objective that predicts hidden content from visible context, including masked image modeling, as the revised introduction now also states more clearly. We study JEPA for two reasons. JEPAs predict learned teacher features, so features that the context cannot predict may also weaken in the targets themselves (Littwin et al., 2024; Sobal et al., 2022). And JEPAs are being adopted rapidly in medical imaging (Section 2). A matched audit of masked image modeling is a natural extension and would require separately matched model training and a pixel-space redesign of the recovery analysis, which we leave to future work.
W2: Input-level removal of pathology tokens.
Thank you for pointing out that pathology information may spread into surrounding tokens through self-attention before final-layer deletion. You are right that deleting the final-layer pathology-overlapping tokens alone cannot examine this possibility directly. We therefore repeated the intervention at the input level, where all patches that overlap same-category pathology annotations are removed before the first transformer block, the remaining tokens are re-encoded, and the unchanged MIL head is applied.
Values are the median logit drop as a percentage of the fixed decision margin, as in Table 3. The effects remain small for kidney, liver, scarring/fibrosis and nodules/masses, decrease for ground-glass opacity and pancreatic lesions, and remain largest for atelectasis/consolidation. Diffusion of annotated pathology information through self-attention therefore does not explain the small effects. The input-level estimates include the model’s complete response to re-encoding after token removal.
We restrict the conclusion to the annotated pathology-overlapping patches and a fixed readout; unannotated disease and correlated cues elsewhere in the scan can remain (see Q1). We add these additional analysis insights to Section 5.6 and Appendix F.2.
Q1: Partial labeling in ReXGroundingCT.
Thank you for raising this. This is a valid and important point. It may affect individual ReX comparisons, but ATLAS provides an independent consistency check.
Without an exhaustive additional expert review of these partially annotated cases, we cannot provide a comprehensive assessment of the extent to which unannotated findings affect individual ReX control comparisons. The official ReX partial annotation protocol specifies that up to three representative instances per finding were segmented, and diffuse-finding exclusion rules were applied (Baharoon et al., 2026). The 41 exhaustively annotated scans in our evaluation cohort yield only 4–25 patient groups per category, too few for precise, category-specific confirmatory estimates.
However, our results on the independent ATLAS cohort provide an important counterpoint to the concern raised. Its lesion masks were proposed by AI and then verified and corrected by radiologists, who added missed tumors and removed false positives (Bassi et al., 2025). ATLAS shows the same behavior as ReX: the pathology-versus-anatomy shortfall is 25.5 points in ATLAS and 25.6 in ReX; prediction attenuates pathology–control separation in both independent evaluation cohorts, and removing kidney or liver lesion tokens before encoding changes the decision margin by only 0.19% and 0.55%. Partial labeling therefore cannot explain these results on its own.
For Section 5.6, unannotated instances could carry part of the ReX decision evidence, which is why we state the downstream result for annotated pathology-overlapping tokens. We now formulate this limitation more clearly, together with the annotation procedures in Section 4.
We thank the reviewer again for the useful comments. They helped us make the intervention claims more precise and clarify our contributions and limitations.