acceptodds
Under review as a conference paper at ICLR 2027

The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging

Abstract

Joint-embedding predictive architectures (JEPAs) learn by predicting teacher representations of masked regions from visible context. Point prediction summarizes the conditional target distribution and cannot reproduce variation that the context does not resolve. We call this constraint predictability bias. In computed tomography (CT), recurrent anatomy is predictable from context, whereas the sporadic, localized pathology that determines clinical interpretation often is not. We introduce TAP-JEPA, a ViT-B/3D trained on 113,611 CT volumes that is competitive with state-of-the-art encoders (COLIPRI, Merlin, VoxelFM) in scan-level classification. We then audit the learned predictive pathway by replaying the exact masking, teacher, and predictor computations used during training. Prediction lowers the mean Skill (prevalence-normalized average precision) from 0.388 to 0.271 for pathology, but only from 0.921 to 0.873 for anatomy. Along the teacher direction, separating pathology from matched control regions in the same organ, prediction moves pathology 34.7% toward the controls and controls 13.6% toward pathology, with larger shifts farther from visible context. Replacing visible matched-control representations increases surrounding-target prediction loss more than fully replacing pathology representations across four pathology categories. Downstream, removing all final-layer pathology tokens reduces the decision margin by at most 0.45% in four of seven scan-level probes. Together, these results show that contextual predictability is not a sufficient proxy for clinical relevance in medical representation learning. Training objectives should therefore retain diagnostic features regardless of their predictability from surrounding anatomy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, joint-embedding predictive architectures, medical imaging, predictability bias

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CeFbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a constraint of Joint-Embedding Predictive Architectures (JEPAs) in medical imaging termed predictability bias—the tendency of point-prediction objectives to summarize the conditional target distribution and discard context-unpredictable variations. By pretraining a large-scale 3D CT model (TAP-JEPA) and auditing its predictive pathway, the authors demonstrate that training objectives should retain localized diagnostic features beyond what is predictable from surrounding anatomy.

Critical strengths
  1. Timely topic with large-scale empirical validation: As JEPA methods are increasingly adopted in medical image analysis, investigating their inductive biases, utility scope, and limitations on a competitive model pretrained on over 113k CT volumes is timely and practically relevant.
  2. Clarity and organization: The manuscript is well organized, clearly written, and easy to follow from theoretical formulation to empirical audits.
Critical weaknesses
  1. Limited conceptual novelty and over-attribution to JEPA: Predictability bias (Proposition 1 and Lemma 1) essentially renames a well-known property that point predictions regress to the conditional mean/median and discard context-orthogonal residuals—making the empirical finding that sporadic lesions are smoothed toward normal tissue largely unsurprising. Moreover, this mathematical property is not unique to JEPA; it applies equally to Masked Image Modeling (e.g., MAE, SimMIM) and other context-prediction methods, yet the paper frames it specifically around JEPA without auditing non-JEPA baselines for comparison.
  2. Insufficient final-layer token removal in the downstream intervention: In Section 5.6 (Table 3), measuring reliance on local pathology by deleting pathology-overlapping tokens only at the final layer is insufficient. Across a 12-layer ViT, self-attention continuously mixes information across tokens, allowing diagnostic signals from lesion tokens to diffuse into surrounding context tokens before the final layer. Removing or masking the target pathology tokens at the input level is needed to draw valid conclusions.
Questions for authors

Impact of partial labeling in ReXGroundingCT: Throughout Sections 5.3–5.6, non-annotated regions in the same organ/lobe are treated as healthy controls. Since ReXGroundingCT is partially labeled (and thoracic findings often involve multifocal lesions or diffuse parenchymal changes outside the bounding masks), how does this partial labeling affect the conclusions? Specifically, could unannotated lesions contaminate the "matched controls" in Sections 5.3–5.5, and could remaining unannotated pathology tokens explain why deleting only the annotated tokens in Section 5.6 barely reduces the scan-level decision margin?

3 comments

We thank the reviewer for recognizing the timeliness of the topic, the large-scale empirical validation, and the clarity and organization of the manuscript.

W1: Novelty and attribution to JEPA.

We agree that Proposition 1 and Lemma 1 are standard properties of point prediction, and the revised Section 3 presents them as such more clearly. Our contribution is to connect them to the JEPA training signal, derive testable consequences for anatomy and pathology, and demonstrate these consequences in a model trained on 113,611 CT volumes and evaluated across two independent datasets.

We also agree that the property is not specific to JEPA, as the Scope paragraph in Section 5.6 already noted. It applies to any point-prediction objective that predicts hidden content from visible context, including masked image modeling, as the revised introduction now also states more clearly. We study JEPA for two reasons. JEPAs predict learned teacher features, so features that the context cannot predict may also weaken in the targets themselves (Littwin et al., 2024; Sobal et al., 2022). And JEPAs are being adopted rapidly in medical imaging (Section 2). A matched audit of masked image modeling is a natural extension and would require separately matched model training and a pixel-space redesign of the recovery analysis, which we leave to future work.

W2: Input-level removal of pathology tokens.

Thank you for pointing out that pathology information may spread into surrounding tokens through self-attention before final-layer deletion. You are right that deleting the final-layer pathology-overlapping tokens alone cannot examine this possibility directly. We therefore repeated the intervention at the input level, where all patches that overlap same-category pathology annotations are removed before the first transformer block, the remaining tokens are re-encoded, and the unchanged MIL head is applied.

FindingFinal layerInput level [95% CI]
Kidney0.18%0.19% [0.15, 0.29]
Liver0.45%0.55% [0.40, 0.86]
Pancreas1.69%−0.28% [−0.80, 0.39]
Scarring/fibrosis−0.04%0.29% [0.06, 0.54]
Atelectasis/consol.10.47%8.02% [5.63, 12.90]
Ground-glass opacity2.39%0.81% [0.05, 1.81]
Nodules/masses0.08%0.09% [0.05, 0.13]

Values are the median logit drop as a percentage of the fixed decision margin, as in Table 3. The effects remain small for kidney, liver, scarring/fibrosis and nodules/masses, decrease for ground-glass opacity and pancreatic lesions, and remain largest for atelectasis/consolidation. Diffusion of annotated pathology information through self-attention therefore does not explain the small effects. The input-level estimates include the model’s complete response to re-encoding after token removal.

We restrict the conclusion to the annotated pathology-overlapping patches and a fixed readout; unannotated disease and correlated cues elsewhere in the scan can remain (see Q1). We add these additional analysis insights to Section 5.6 and Appendix F.2.

Q1: Partial labeling in ReXGroundingCT.

Thank you for raising this. This is a valid and important point. It may affect individual ReX comparisons, but ATLAS provides an independent consistency check.

Without an exhaustive additional expert review of these partially annotated cases, we cannot provide a comprehensive assessment of the extent to which unannotated findings affect individual ReX control comparisons. The official ReX partial annotation protocol specifies that up to three representative instances per finding were segmented, and diffuse-finding exclusion rules were applied (Baharoon et al., 2026). The 41 exhaustively annotated scans in our evaluation cohort yield only 4–25 patient groups per category, too few for precise, category-specific confirmatory estimates.

However, our results on the independent ATLAS cohort provide an important counterpoint to the concern raised. Its lesion masks were proposed by AI and then verified and corrected by radiologists, who added missed tumors and removed false positives (Bassi et al., 2025). ATLAS shows the same behavior as ReX: the pathology-versus-anatomy shortfall is 25.5 points in ATLAS and 25.6 in ReX; prediction attenuates pathology–control separation in both independent evaluation cohorts, and removing kidney or liver lesion tokens before encoding changes the decision margin by only 0.19% and 0.55%. Partial labeling therefore cannot explain these results on its own.

For Section 5.6, unannotated instances could carry part of the ReX decision evidence, which is why we state the downstream result for annotated pathology-overlapping tokens. We now formulate this limitation more clearly, together with the annotation procedures in Section 4.

We thank the reviewer again for the useful comments. They helped us make the intervention claims more precise and clarify our contributions and limitations.

Thank you for the thorough rebuttal. The newly added experiments, analysis and clarification addresses my main soundness concerns. While I still view the theoretical framing as somewhat standard, the empirical audit is solid, so I maintain my positive score (3).

Thank you for the constructive engagement and for maintaining your positive assessment. We appreciate your recognition that the additional experiments and clarifications address your main soundness concerns. The novelty we wish to emphasize is not the standard point-prediction characterization itself, but identifying and systematically auditing a mismatch between contextual predictability and clinical relevance. Diagnostically important pathology need not be predictable from surrounding anatomy. Our audit demonstrates that strong scan-level performance can coexist with substantial attenuation of pathology-related information along the learned predictive pathway.

Reviewer_89rtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the concept of predictability bias in JEPA-style architectures. The authors argue that models trained to predict hidden medical image representations from visible context will prefer learning predictable information (e.g., anatomy) and underrepresent unpredictable information (e.g., pathology). They train a JEPA-style model on approx. 113k 3D CT scans and throughout their experiments, they show that hidden-target prediction preserves anatomical discrimination better than pathology discrimination. In the end, the authors claim that strong SSL representations and downstream classification performance don't really guarantee perserving localised diagnostic information.

Critical strengths
  • The paper is interesting. I like the research question you are asking and I believe it's relevant to ask for the medical imaging field.
  • The experimental section is strong and I like that you are checking the predictability bias through multiple experiments such as masking analysis and feature-space interventions.
  • The model has been trained at scale and it has been evaluated on separate datasets to see the pathology-related effects.
Critical weaknesses
  • The paper is a bit hard to read. Also, I believe some improvements can be made for, e.g. Section 3, so that it's easier to follow. The conditional mean/median characterization and the gradient factorization are quite standard math facts and what I find novel there is mostly connecting them to the JEPA training and demonstrating the differences in predicting the anatomy vs. pathology. I would rather try to use a simplified presentation in the main paper, keep a short proof of e.g., conditional gradient identity and just add any fully measure-theoretic justifications in the Appendix section. This should make your paper easier to read and understand (for an eventual larger audience).
  • I see that you only trained a single JEPA model and then you made all experiments on it. Have you tried other JEPA configurations? What is the reasoning for choosing this exact experimental configuration?
  • To my understanding, in section 5.3, you compare teacher representations of hidden targets with the corresponding representations predicted from visible context. However, during training, the teacher will always "see" the complete image, whereas the predictor does not "see" directly the hidden region. I was wondering whether it wouldn't be the case that the pathology discrimination reduction that you talk about in that section happens because of the predictor's limited context rather than the information lost by the encoder? Could you comment more on this?
Questions for authors

See weaknesses section.

5 comments

We thank the reviewer for recognizing the relevance of the research question, the breadth of the experimental analysis, and the scale of the model and evaluation. The comments help us clarify the contribution, its empirical scope, and the distinction between the predictive training pathway and full-context inference.

W1: Readability and Section 3.

Thank you for the suggestion to make Section 3 more accessible. We revised Section 3 along the lines you suggested. The main text now gives the setup and the loss, states the conditional median/mean characterization as a standard result, and keeps the conditional gradient identity. The formal statements and measure-theoretic details are now in Appendix B. Section 3 now makes it clearer that we use these standard facts to trace what JEPA training on CT can preserve and to derive the anatomy and pathology signatures that Section 5 measures.

W2: Single JEPA configuration.

We used one production-scale configuration by design. Predictability bias is a property of the training objective, and the optimal prediction and gradient results in Section 3 hold for any encoder, predictor, and mask distribution. Configuration choices determine how informative the visible context is and how closely the trained model approaches the optimum, and thus how strongly the bias manifests. Our aim was to measure it through the training computation of one strong volumetric model, rather than to compare architectures or configurations. We therefore fixed the configuration before training, basing it on established volumetric JEPA and CT designs.

Masking, background down-weighting of the loss, and 3D RoPE follow the recent volumetric JEPAs NeuroVFM (Kondepudi et al., 2026) and Neuro-JEPA (Huang et al., 2026), with the token size scaled from NeuroVFM's ≈16 mm to accommodate the larger thorax–abdomen–pelvis field of view. The narrow predictor, the feature-wise L1 loss, and the optimization schedules are standard I-JEPA (Assran et al., 2023) and V-JEPA (Bardes et al., 2024) choices, and the three HU input windows follow previous multi-window CT input approaches as in NeuroVFM or Pillar-0 (Agrawal et al., 2025).

Considering configuration insights already visible within the single model setup investigated, the attenuation of pathology–control separation appears in all three mask families mixed during training, is most pronounced for the single-crop context with the most compact visible region (Table 26), and broadly grows with distance to visible context (Table 25), the quantity through which masking choices act. How further masking and objective design change the size of the effect is a natural next step, which we name as future work in Section 5.6 (Scope).

W3: Limited predictor context versus encoder information.

Thank you for raising this distinction. The predictor is not used during downstream inference, but its loss supplies the learning signal that shapes the student encoder during pretraining. Section 5.3 therefore examines the predictive training pathway, not the information available when the encoder receives a fully visible image.

The complete-view teacher measures the discriminative information available at the hidden target, while the predictor measures how much of that information the training objective can recover from visible context. Some reduction is expected when the target is hidden. The relevant finding is the anatomy–pathology difference under identical masks and evaluation folds. The normalized shortfall is 36.7% for pathology versus 11.2% for anatomy on ATLAS, and 31.8% versus 6.2% on ReX. The paired anatomy–pathology differences are 25.5 percentage points ([20.2, 32.5]) and 25.6 percentage points ([21.4, 29.9]), respectively.

This asymmetry matters for representation learning because gradients from the prediction loss update the student encoder. Information that cannot be resolved from context provides less consistent training signal through this pathway, even though downstream inference later uses a complete image. Whether the encoder's complete-view representation is used downstream is tested separately in Section 5.6. The revised conclusion makes this separation between prediction-level and downstream evidence more explicit.

We thank the reviewer again for the remarks. They helped us to improve the readability of the manuscript and clarify the reasoning behind the examined JEPA configuration and predictor evaluations.

Hi, thank you for your reply. Section 3 is way more readable and the clarifications for W3 are clear!

However, for W2, your response says that configuration choices determine how strong the bias is. Thus, I do believe that this claim would need some evidence that's more than just evaluating a single run. And even though Tables 25 and 26 are informative, they still test a single model configuration.

Since the main topic is predictability bias of JEPA training objectives in medical imaging, I think you can still show whether or how much/less it is pronounced under different training configurations. I don't think you'd need a production-scale config and/or train it for so long, but you could have tested on e.g., ViT-S and do some small-scale ablatioms like L1 vs. L2 (you discuss both in Seciton 3 but only L1 is tested?) and how much context to have visible (in your config you use 16% but perhaps using less or more would affect the predictability bias?).

I find them important, given the strong claims you make about predictability bias as a general characteristic of JEPA models. Therefore, I'm gonna keep my score as it is.

Thank you for the follow-up and for confirming that Section 3 and W3 are now clear.

We followed your suggestion with small-scale ViT-S ablations of the loss and the visible-context fraction, and evaluated each with our hidden-target recovery analysis. The predictability-bias signature it measures, a larger loss of pathology than of anatomy discrimination under hidden-target prediction, appears in every configuration.

We trained three ViT-S models from scratch with a matched, reduced-scale recipe: same initialization, data, masking families and optimizer, one seed each, and 12 epochs. They vary the two factors you raised: L1 with 16% or 32% visible context, and L2 (squared error) with 16% context. The analysis compares teacher and predicted Skill for anatomy and pathology on the same scans, masks, and folds as in the paper, with probes refitted for each model. All models are evaluated with the original 16% context masks; the 32% context model is also evaluated at its training context level. Results on the ReX cohort:

Configuration, context train / eval (%)Teacher Skill, A / PPredicted Skill, A / PShortfall (%), A / PP − A gap (pp) [95% CI]
ViT-B, L1, 16 / 16 (paper)0.956 / 0.3410.925 / 0.2436.2 / 31.825.6 [21.4, 29.9]
ViT-S, L1, 16 / 160.868 / 0.2200.845 / 0.1615.5 / 31.626.1 [20.5, 31.7]
ViT-S, L1, 32 / 160.879 / 0.2260.826 / 0.16412.7 / 32.219.5 [15.3, 24.3]
ViT-S, L1, 32 / 32‡0.879 / 0.2260.865 / 0.1633.4 / 33.229.7 [25.0, 34.9]
ViT-S, L2, 16 / 160.858 / 0.2130.842 / 0.1654.2 / 27.223.0 [17.6, 29.0]

A / P: anatomy/pathology. Normalized shortfall: (teacher − predicted Skill) / (teacher − geometry-only Skill). Pointwise 95% patient-bootstrap CIs. ‡ 32% context at evaluation, by adding nearby tokens to each original context while all scored targets stay hidden.

Pathology shortfall exceeds anatomy shortfall in every configuration, with positive intervals. It stays at 27–33%, whereas anatomy shortfall ranges from 3 to 13%. Across configurations and evaluation contexts, the gap ranges from 19.5 to 29.7 pp (paper model: 25.6). With L2, the gap is 3.1 pp smaller than with L1, which is not significant after Holm adjustment (p = 0.16). The 32% model has a 6.6 pp smaller gap than the 16% L1 model at 16% evaluation (Holm-adjusted p = 0.008) because its anatomy shortfall rises (12.7% vs 5.5%) while its pathology shortfall barely changes (32.2% vs 31.6%); at its training context, the gap is 29.7 pp.

On the ATLAS cohort, the asymmetry has the same direction in all runs. For the 32% model, the gap is 53.8 [43.3, 69.0] pp at 16% and 61.9 [52.2, 75.8] pp at 32% evaluation (paper model: 25.5). The short-run teachers separate ATLAS pathology only weakly from the geometry-only baseline (teacher pathology Skill 0.14–0.19 vs 0.43 for the paper model), so the normalized gaps are larger and less precise than for the paper model. For the two 16% runs, this teacher–geometry difference is below 0.1 (0.096 and 0.080), so their gaps (68.5 and 56.1 pp, same direction) are descriptive.

We will add these runs, with the full methodology and further details, to the appendix. Thank you again for this constructive suggestion.

Reviewer_Z5etOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether predicting masked CT features preserves information about local disease. The authors train TAP-JEPA on 113,611 CT volumes and report competitive scan-level classification results. They then replay the training masking and prediction pathway on held-out ATLAS and ReX scans. Compared with full-view teacher features, predicted features show a larger normalized drop in pathology probe performance than in anatomy probe performance (Section 5.3). The matched-region analysis also finds reduced pathology-control separation. The paper further tests how replacing visible context affects prediction and how deleting final-layer pathology tokens affects downstream decisions.

Critical strengths
  1. The main result is consistent across two datasets. The normalized pathology-minus-anatomy shortfall is 25.5 percentage points on ATLAS and 25.6 on ReX, and remains positive with a richer geometry baseline (Section 5.3; Tables 17-18). This is the main reason I lean toward acceptance.
  2. The matched-region analysis includes useful controls. The authors compare regions within the same organ or lung lobe, use random directions, and test alternative control regions (Appendix F.8). The results vary across disease categories, which helps show where the proposed explanation holds most strongly.
Critical weaknesses
  1. The deletion test is narrower than the conclusion. It removes final-layer pathology tokens from the full-view student readout without re-encoding the image. Disease information could remain in other tokens. Section 5.6 and Appendix F.2 acknowledge this, but Section 6 still describes removing the local finding itself. I would narrow that statement; the masked-prediction shortfall is a separate result.
  2. The context-replacement result depends on the setup. Table 2 uses shared targets, but their distances to the two regions need not match. The common-event, sampled-donor and equal-norm checks help (Tables 32-33). Table 37's token-disjoint and adjusted analyses use own-neighbourhood targets, so they do not directly settle the shared-target concern. In Table 35, the raw shared-target pathology-minus-control contrasts for liver, atelectasis/consolidation and ground-glass opacity are positive at 25% and 50% replacement but negative at full replacement, with pointwise intervals excluding zero. Section 5.5 and Appendix F.9 acknowledge this. It does not contradict Table 2, but Section 6 should keep the full-replacement/shared-target qualification.
Questions for authors
  1. Clarification: What can we conclude about pathology's contribution to prediction when the raw shared-target pathology-minus-control effects in Table 35 reverse sign with replacement strength? Section 5.5 already acknowledges this. I am comparing raw effects within Table 35, not partial-dose results with the adjusted estimates in Table 2.
  2. Additional check: Does the Table 2 result remain when the two intervened regions share no tokens and their distances to the shared targets are balanced? Please report retained pair and study/patient counts, remaining distance differences and confidence intervals. Table 37 uses own-neighbourhood targets rather than this shared-target setting.
3 comments

We thank the reviewer for the careful and technically precise review. Your questions helped us align the intervention claims with their scope and motivated two complementary analyses that we have added to the paper.

W1: Final-layer pathology-token removal.

You are right that deleting only the final-layer pathology tokens cannot rule out information having spread to other tokens through self-attention, as we note in Section 5.6 and Appendix F.2. We therefore also removed the same pathology-overlapping tokens before the first transformer block, re-encoded the remaining tokens, and applied the unchanged MIL head.

FindingFinal layerInput level [95% CI]
Kidney0.18%0.19% [0.15, 0.29]
Liver0.45%0.55% [0.40, 0.86]
Pancreas1.69%−0.28% [−0.80, 0.39]
Scarring/fibrosis−0.04%0.29% [0.06, 0.54]
Atelectasis/consol.10.47%8.02% [5.63, 12.90]
Ground-glass opacity2.39%0.81% [0.05, 1.81]
Nodules/masses0.08%0.09% [0.05, 0.13]

Values are the median logit drop in % of the fixed decision margin, as in Table 3. The effects remain small for kidney, liver, scarring/fibrosis and nodules/masses. We add this to Section 5.6 and Appendix F.2. In Section 6, we keep this downstream result separate from the masked-prediction results of Sections 5.3–5.4, as you suggest, and replace the statement with:

In several downstream tasks, removing the annotated pathology-overlapping tokens, at the final layer or before encoding, changes the decision margin by less than 0.6%.

W2: Dependence of the context-replacement result on the setup.

Q1 and Q2 below address replacement strength and target geometry. Section 5.5, the conclusion, and the abstract now qualify it as full replacement on shared targets and note the dependencies on target geometry; Section 5.5 also integrates the distance-caliper result.

Q1: What can we conclude from the change in direction across replacement strengths?

Thank you for this question. You are right that the raw shared-target pathology-minus-control contrasts in Table 35 change direction between partial and full replacement for liver, atelectasis/consolidation, and ground-glass opacity, so the relative ordering of the two replacement costs depends on replacement strength. Our analyses do not identify why the ordering changes, and we therefore draw no conclusion about pathology's contribution beyond the full-replacement setting. As already noted in Section 5.5 and Appendix F.9, Table 2 supports a specific full-replacement result, and Section 5.5, the conclusion, and the abstract now carry this qualification more explicitly. We agree that this dependence on replacement strength is an interesting question that warrants a dedicated analysis across strengths and layers, as acknowledged in the revised Scope paragraph (Section 5.6).

Q2: Token-disjoint regions and a shared-target distance caliper.

We repeated the full-replacement, reference-adjusted comparison of Table 2 after (i) excluding events in which the pathology and control regions share a token and (ii) keeping only shared targets whose distances to the two region centroids differ by at most one token width, ∣d+−d−∣≤24|d_+ - d_-| \le 24 mm.

FindingPairs (studies/patients)Signed d+−d−d_+ - d_-, mm∣d+−d−∣\lvert d_+ - d_- \rvert, mm% of control cost% of baseline loss
Kidney†23 (23)2.55 [1.05, 4.01]11.40 [10.80, 12.05]−0.9 [−28.5, 20.9]−0.022 [−0.568, 0.516]
Liver*279 (169)0.89 [0.56, 1.21]11.15 [11.05, 11.24]+13.5 [7.9, 19.0]+0.558 [0.308, 0.845]
Scarring/fibrosis*166 (104)4.17 [3.46, 4.88]11.46 [11.30, 11.61]+14.3 [8.3, 19.8]+1.093 [0.610, 1.564]
Atelectasis/consol.199 (119)1.61 [1.20, 2.02]11.37 [11.26, 11.47]+6.1 [0.3, 11.5]+0.430 [0.020, 0.827]
Ground-glass opacity291 (178)1.37 [1.08, 1.66]11.35 [11.26, 11.44]+2.4 [−2.1, 6.8]+0.182 [−0.153, 0.516]
Nodules/masses178 (135)1.22 [0.71, 1.74]11.26 [11.13, 11.39]+1.2 [−6.8, 8.2]+0.044 [−0.249, 0.324]

Pointwise 95% CIs from the same bootstrap as Table 2. Positive values indicate a higher control replacement cost, as in Table 2. * Supported after Holm (ATLAS) / Benjamini–Hochberg (ReX) correction. † Descriptive (< 100 pairs).

Table 2 supports the ordering after correction for liver lesions, scarring/fibrosis, atelectasis/consolidation and ground-glass opacity. The stricter caliper analysis additionally supports it after correction for liver lesions and scarring/fibrosis and pointwise for atelectasis/consolidation; ground-glass opacity and nodules/masses keep smaller, unresolved positive estimates. We add this analysis after Table 2 and to Appendix F.9, and qualify the corresponding intervention claims.

We thank the reviewer again for the questions. They helped us make the intervention claims more precise under complementary input-level and geometric analyses.

Thank you for the response and the additional experiments. The input-level deletion test addresses my concern about information spreading through self-attention before final-layer removal. The small median effects in several categories strengthen the downstream finding, with the conclusion appropriately limited to annotated pathology patches and a fixed readout. The new shared-target analysis with token-disjoint regions and a 24 mm distance caliper is also useful. Support remains after multiplicity correction for liver lesions and scarring/fibrosis. Atelectasis/consolidation has pointwise support, while ground-glass opacity and nodules/masses remain inconclusive. The remaining distance differences justify keeping the qualification about target geometry. I also appreciate the acknowledgment that the pathology-versus-control ordering reverses between partial and full replacement in some categories. The reason remains unresolved, but restricting the claim to full replacement on shared targets addresses my concern about its scope. The main reason I support acceptance remains the larger normalized drop in pathology discrimination than in anatomical discrimination during masked prediction, seen in both ATLAS and ReX. The added analyses strengthen the paper. The remaining sensitivity to intervention settings and reliance on a single model configuration limit how broadly the findings can be applied. I therefore maintain my rating of 3.

Thank you for the careful follow-up and for your constructive suggestions throughout the discussion. The input-level deletion and the token-disjoint caliper analysis clearly helped to improve the paper.

Regarding the reliance on a single model configuration: in response to Reviewer 89rt, we trained three additional reduced-scale ViT-S models that vary the loss (L1 vs L2) and the visible context (16% vs 32%), and repeated the hidden-target recovery analysis. The result you highlight as your main reason for supporting acceptance, the larger normalized drop in pathology than in anatomical discrimination, appears in all of them. Results on the ReX cohort:

Configuration, context train / eval (%)Teacher Skill, A / PPredicted Skill, A / PShortfall (%), A / PP − A gap (pp) [95% CI]
ViT-B, L1, 16 / 16 (paper)0.956 / 0.3410.925 / 0.2436.2 / 31.825.6 [21.4, 29.9]
ViT-S, L1, 16 / 160.868 / 0.2200.845 / 0.1615.5 / 31.626.1 [20.5, 31.7]
ViT-S, L1, 32 / 160.879 / 0.2260.826 / 0.16412.7 / 32.219.5 [15.3, 24.3]
ViT-S, L1, 32 / 32‡0.879 / 0.2260.865 / 0.1633.4 / 33.229.7 [25.0, 34.9]
ViT-S, L2, 16 / 160.858 / 0.2130.842 / 0.1654.2 / 27.223.0 [17.6, 29.0]

A / P: anatomy/pathology. Normalized shortfall: (teacher − predicted Skill) / (teacher − geometry-only Skill). Pointwise 95% patient-bootstrap CIs. ‡ 32% context at evaluation, by adding nearby tokens to each original context while all scored targets stay hidden.

The pathology–anatomy gap ranges from 19.5 to 29.7 pp (paper model: 25.6 pp) with positive intervals throughout, and ATLAS shows the same direction. Details are in our reply to 89rt, and we will include this information in the paper.

Comments on the paper

3 comments

We thank all reviewers for their constructive and detailed feedback, which helped to improve the manuscript. The reviewers found the research question relevant and timely (89rt, CeFb), the experiments strong and the matched-region controls useful (89rt, Z5et), and the main pathology-versus-anatomy result consistent across two datasets (Z5et). Below, we summarize the analyses added during the discussion and the changes in the revised PDF. Table, proposition, and lemma numbers in our replies follow the submitted version; in the revised PDF, the new Table 8 shifts later appendix tables by one, and Proposition 1 and Lemma 1 are now Proposition 2 and Lemma 1 in Appendix B.

New analyses

  • Input-level removal of pathology tokens (CeFb, Z5et). Pathology-overlapping patches are removed before the first transformer block, and the scan is re-encoded. The small downstream dependence for kidney, liver, scarring/fibrosis, and nodules/masses persists (at most 0.55% of the decision margin). Section 5.6, Appendix F.2.
  • Token-disjoint, distance-caliper context intervention (Z5et). This stricter test, with token-disjoint regions and shared targets within a token-scale 24 mm distance caliper, additionally supports the Table 2 ordering after correction for liver lesions and scarring/fibrosis and pointwise for atelectasis/consolidation, with smaller, unresolved estimates in the other categories. Section 5.5, Appendix F.9.
  • Reduced-scale configuration checks (89rt). Three ViT-S models trained with L1 at 16% or 32% visible context and with L2 at 16% show the same pathology–anatomy asymmetry in the hidden-target recovery analysis as the paper model (ReX gap 19.5–29.7 pp vs 25.6 pp, all intervals positive). Details are in our reply to reviewer 89rt; we will add these runs, along with further details, to the appendix.

Revisions

  • Section 3 revised for better accessibility; the optimal-prediction and gradient results are communicated more clearly as standard properties of point prediction, with formal statements in Appendix B (89rt, CeFb).
  • Abstract, Section 5.5, and the conclusion communicate more clearly what the interventions support; prediction-level and downstream results separated (Z5et, 89rt).
  • Partial annotation of ReX controls and the ATLAS annotation procedure clarified in Section 4 (CeFb).
  • Generality of predictability bias beyond JEPA clarified in the introduction (CeFb).
  • The rationale for the training configuration used is communicated more clearly, with a connection to design sources (Section 4, Appendix D) (89rt).
Area_Chair_sNhbOpenReview ↗

Reminder: Current Rebuttal Phase

Dear Reviewers,

I hope this message finds you well.

I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.

Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.

If you have any questions or need further assistance, please do not hesitate to reach out.

Best regards,

AC

We thank all reviewers again for a constructive discussion. Z5et stated that the added analyses strengthen the paper and address their concerns about the deletion test and the scope of the context-replacement claim, and kept their positive rating (3). 89rt acknowledged the revisions and additional experiments, raising their score to 3. CeFb stated that the additional experiments and clarifications address their main soundness concerns and kept their positive score (3). Overall, we added three analyses that support the robustness of our main findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.