acceptodds
Withdrawn from ICLR 2027

Predict-Then-Adapt: Cross-Plane Joint-Embedding for Test-Time Medical Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time medical image segmentation adapts source-trained models to unlabeled target volumes under distribution shifts. Existing methods mainly derive adaptation signals from model predictions or individual slices, leaving the intrinsic three-dimensional structure of test volumes underexploited. In particular, orthogonal views observe the same anatomy with exact voxel-wise correspondence, providing natural within-volume supervision. However, a representation that is predictable across views may not remain compatible with the source-trained segmentation task. We propose CP-JEPA, a Cross-Plane Joint-Embedding Predictive Adaptation framework following a predict-then-adapt principle. CP-JEPA models cross-plane prediction and task-space translation as two complementary stages. It first predicts masked target-view representations from geometrically corresponding orthogonal views, constructing a JEPA-style objective within each test volume. It then compares the cross-plane prediction with a capacity-matched axial control to isolate geometry-specific counterfactual evidence and translates this evidence into bounded, source-anchored segmentation corrections. CP-JEPA supports both single- and continual-volume adaptation by resetting volume-specific predictive states while selectively retaining reliable task-space evidence across cases. We evaluate CP-JEPA on multi-organ MRI segmentation under single- and continual-volume deployment, with and without labeled target-development data used only for configuration. Across DINOv3- and U-Net-based backbones, CP-JEPA achieves the strongest macro-average performance in three of four evaluation settings and ranks second in the remaining setting, outperforming the strongest TTA baselines by approximately Dice point on average.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Medical Image Segmentation, Joint-Embedding Predictive Architecture, Test-Time Adaptation, Domain Shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sUeiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a test-time medical image segmentation method, namely CP-JEPA. Instead of updating the segmentation network, the method trains predictors to estimate masked axial representations using coronal and sagittal views. A source-trained adapter then translates the predictive advantage of these cross-plane views into bounded segmentation corrections. The method also equips a memory grid to perform task-space corrections across volumes for continual adaptation. The method is evaluated on prostate, cardiac, and liver MRI under various adaptation protocols.

Critical strengths
  • Although there are many test-time adaptation objectives for segmentation tasks, leveraging the geometry of 3D medical scans, which is in this case, matching anatomical locations across orthogonal planes is a smart way to generate a test-time signal.

  • The authors conducted evaluations across multiple anatomies, MRI modalities, and adaptation setups. The results on the cardiac benchmark are also encouraging.

Critical weaknesses
  • The experiments don't distinguish the benefit of the test-time cross-plane prediction from the source-trained fully supervised adapter. During training, the control branches are penalized and forced to mimic the source prediction. They show no improvement by design indeed. It is also possible the gains come from just having a source-trained refinement module looking at orthogonal features, rather than test-time predictive learning itself. Also, the reported 79.4% cross-plane predictive advantage in Fig. 4 is measured on only 350 sampled voxels from a single prostate volume, far too thin to show it generalizes across subjects and shifts.

  • The paper reported a large number of TTA baselines, but Appendix A.5 acknowledges the baselines are not reproductions rather rough adaptations of original ones, and some like ASR coming from different fields like speech recognition. Also more relevant multi-view medical TTA paper (Joshi et al.) is completely skipped in the main accuracy tables, and just appeared only in a supplementary compute table. In addition, the method does not consistently dominate, for example standard TENT actually beats it in several setups. Finally, despite claiming to run three random seeds, there are no error bars or standard deviations for Dice improvements. There are also a few numerical contradictions between the text and tables, e.g., single-volume Dice of 0.7114, whereas Table 1 reports 0.7189.

Questions for authors
  • Could the authors provide a baseline comparison that uses a source-trained 3D refinement module on the same cross-plane views (without any test-time JEPA optimization) and ideally with a fair setup of aligned compute and parameters?
  • Why was the paper Joshi et al. excluded from the segmentation performance tables? Can you also provide standard deviations or paired confidence intervals for your main results, and clarify the results inconsistent CHAOS and TENT scores between the main text and appendix?
  • Since the continual setup relies on task-specific residuals across patients, how sensitive is it to patient ordering differences? Please report performance across different patient permutations and detail how often the gating mechanism described in teh paper rejects historical corrections.
2 comments

We thank the reviewer for recognizing the value of using orthogonal geometric correspondence to construct test-time supervision. Your comments helped us further distinguish the roles of test-time prediction, the source-supervised task adapter, and continual memory. We have added independently trained controls, cross-patient analyses, a MuVi comparison using the same source model, paired confidence intervals, and case-order experiments.

1. Do the gains primarily come from the source-supervised adapter rather than test-time prediction?

We added independently trained controls without requiring the control branches to remain close to the source prediction, allowing every branch to learn effective corrections through source-domain segmentation supervision.

All branches use the same frozen DINOv3 + 3D U-Net checkpoint, source training data, target cases, and 50 epochs of adapter training. Results across three independent adapter training seeds are:

MethodMean Dice across seven prostate target cohorts
Frozen source model0.6830
Axial-context prediction + adapter0.7113 ± 0.0059
Shuffled orthogonal correspondence + adapter0.7147 ± 0.0049
Same adapter with predictive evidence set to zero0.7160 ± 0.0106
Static three-view averaging + adapter0.7160 ± 0.0126
Static 3D refinement module without test-time predictive optimization0.7202 ± 0.0104
Exact cross-plane prediction + adapter0.7306 ± 0.0045

Exact cross-plane prediction outperforms static three-view averaging by 1.46 Dice percentage points, with positive paired differences for all three seeds. After averaging across seeds, the patient-level paired 95% confidence interval is [1.04, 1.91] percentage points. Thus, the additional source-supervised adapter and static multi-view inputs do not fully explain the gains.

Following your suggestion, we also added a static 3D refinement module with a closely matched total parameter count: approximately 607,100 parameters, compared with approximately 606,800 parameters for the CP-JEPA adapter and temporary predictors combined. CP-JEPA achieves a mean advantage of 1.04 percentage points. These results support an additional contribution from test-time cross-plane prediction and show higher mean performance than static 3D refinement in this comparison.

2. Does the cross-plane predictive advantage generalize across patients?

We expanded the original single-case example to 248 independent patients across seven target cohorts.

In all seven cohorts, exact cross-plane prediction achieves a positive mean validation cosine-similarity advantage over axial prediction. When keeping the same adapter fixed, exact predictive evidence also provides a positive mean segmentation gain over zero evidence in every cohort.

These results show that cross-plane predictability is not an isolated observation from a single case. They support the additional predictive information provided by orthogonal geometry. However, this information requires task-compatible translation and should not be directly equated with segmentation improvement or a safety guarantee. This distinction is one of the central motivations of our work.

3. Why was MuVi omitted from the accuracy comparison, and are the other baselines fair?

We acknowledge that methods such as ASR are not well suited to this setting. We will therefore remove comparisons with ASR, IMSE, and similarly unsuitable methods from the revision.

We considered MuVi and attempted to use it in our earlier experiments. Under its default settings, we observed overfitting, with source-to-target Dice gaps reaching 20 percentage points on some datasets, and consequently omitted it from the main table. To provide a comparison under matched conditions, we have now added experiments based on the core of its official training code, using the same DINOv3 hybrid backbone, source checkpoint, and seven target cohorts as CP-JEPA:

MethodMean single-volume Dice
Shared source model with static three-view readout0.6830
MuVi’s official adaptation strategy with an interface extension for the shared hybrid backbone0.6760
CP-JEPA0.7030

Under this setting, CP-JEPA outperforms MuVi by 2.70 Dice percentage points. MuVi exhibits performance degradation, which may reflect our limited exploration of its adaptation hyperparameters under the available time constraints. We will continue investigating these settings during the remaining rebuttal period.

We also retain the results showing TENT’s advantages in some settings. We appreciate the reviewer’s correction and will remove any wording suggesting that our method outperforms existing approaches under every setting.

4. How will standard deviations, confidence intervals, and numerical inconsistencies be addressed?

Space constraints prevented us from including standard deviations in Table 1. We now provide additional statistics from three independent training seeds and will incorporate these into the revised manuscript in the coming days.

DatasetSingle-volume improvement over the frozen model, with 95% CIContinual-volume improvement over the frozen model, with 95% CI
Prostate T2+3.34 [2.40, 4.58] percentage points+4.47 [3.40, 5.74] percentage points
Cardiac M&Ms+3.34 [2.86, 3.88] percentage points+6.01 [5.16, 6.93] percentage points
CHAOS+0.30 [0.14, 0.46] percentage points+3.91 [−0.45, 8.24] percentage points

The mean improvements on prostate and cardiac segmentation are supported by paired statistical analyses. The continual CHAOS result has greater uncertainty; we will explicitly retain this qualification.

We investigated the discrepancy between the CHAOS results of 0.7114 and 0.7189, as well as the TENT differences between the main text and appendix. During experimentation, we initially explored configurations using one seed and added two further seeds after finalizing the method to evaluate its stability. Our checks confirmed the reliability of the results reported in Table 1.

5. How sensitive is continual memory to case order, and how often does gating reject historical corrections?

We added experiments on four prostate cohorts with memory enabled, covering three adapter-seed combinations and four case orders.

The main adapter configuration improves over single-volume adaptation under all four orders, with gains ranging from 2.04 to 2.48 Dice percentage points. This indicates that its continual gains do not depend on a single favorable order. Across all 12 conditions, seven yield positive gains and five yield negative gains, with an average improvement of approximately 0.32 percentage points. Differences across adapters also indicate that the effectiveness of continual memory depends substantially on adapter quality.

In the evaluation across four target cohorts and three risk-model training seeds, there were 345 repeated case exposures eligible for historical correction. Of these, 261 benefited from historical corrections, while 42 exhibited substantial degradation. The strategy retained 98.3% of the original positive gain magnitude, and mean Dice increased from 0.6924 to 0.6978. These results provide encouraging evidence of an improved balance between benefits and risks.

We thank the reviewer for these suggestions. The additional experiments clarify that cross-plane prediction provides information beyond the controls, the source-supervised adapter translates that information into segmentation corrections, and continual memory can yield average gains across case orders while requiring careful attention to stability.

Reviewer_EyrfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the distribution shifts that medical image segmentation models encounter during deployment due to previously unseen scanners, acquisition protocols, and imaging contrasts. These shifts can degrade the performance of source-trained models on new data. The authors aim to exploit cross-plane geometric correspondence within a single 3D case to provide supervision for test-time adaptation, while ensuring that this supervision translates into improved segmentation. To this end, the paper proposes CP-JEPA, which performs test-time adaptation using cross-plane correspondence within an unlabeled 3D medical image. The method predicts axial features from coronal and sagittal views, then uses a source-trained task adapter to translate the resulting cross-plane predictive evidence into bounded segmentation corrections anchored to the source model’s prediction. During continual adaptation, the predictors are reset for each case, and only reliable historical task-space information is retained.

Critical strengths
  1. Cross-plane geometric supervision: The method exploits cross-plane correspondence within a single 3D case to provide adaptation signals for unlabeled target data.
  2. Task-compatible translation: CP-JEPA uses a task adapter to translate cross-plane predictive evidence into bounded segmentation corrections, addressing the gap between feature predictability and compatibility with the segmentation task.
Critical weaknesses
  1. Insufficient reliability gating under unlabeled conditions: Directional agreement between historical and current information does not necessarily translate into improved segmentation, and continual adaptation may degrade performance. Without labels, the gate still cannot reliably determine whether a correction is genuinely beneficial.
  2. Insufficient attribution of the proposed contributions through ablation studies: The control branches are explicitly encouraged during training to remain close to the source model’s predictions. Numerical inconsistencies in the reported results also need to be resolved, and stronger independent validation is required.
  3. Cross-case memory requires further validation: Residuals from different cases are mapped to a common grid through interpolation, but corresponding grid locations do not necessarily represent the same anatomical locations. The reliability of transferring historical residuals across variations in organ position, field of view, and body habitus remains unclear.
Questions for authors

Please pay attention to insufficient reliability gating under unlabeled conditions and the need for further validation of cross-case memory. If the author can address these two issues, I would be inclined to increase the score.

1 comment

We thank the reviewer for recognizing the value of cross-plane geometric supervision and task-compatible translation. Your suggestions motivated additional experiments to directly examine the benefits and harms of historical corrections, the independent contribution of cross-plane prediction, and the spatial and ordering sensitivity of cross-case memory. We respond point by point below.

1. Can unlabeled gating determine whether historical corrections are beneficial?

We agree with the reviewer’s distinction: directional agreement provides a basis for using historical information, but does not guarantee segmentation improvement. The original gate modulates historical corrections rather than serving as an oracle that determines their correctness. We will clarify this role and assess its reliability through observed benefits and harms, rather than relying solely on consistency scores.

To address this concern, we further explored a risk-aware readout trained and configured using source labels only. On four prostate cohorts, with a frozen DINOv3 + 3D U-Net backbone, fixed CP weights, and a fixed case order, the averages across three risk-readout training runs show:

  • Continual-mode Dice increased from 0.6924 to 0.6978.
  • Surface Dice increased from 0.4244 to 0.4299.
  • Harmful patient–condition instances decreased from 102 to 84.
  • Instances with degradation exceeding one Dice percentage point decreased from 60 to 42.
  • 98.3% of the original beneficial correction magnitude was retained.

These results indicate that historical information can be used more selectively to reduce larger degradations while preserving most of its benefits, supporting further improvement of continual adaptation.

2. Training constraints on control branches and contribution attribution

We thank the reviewer for identifying the confounding factor in the original control experiments. To more directly examine whether the gains arise solely from source-trained refinement, we added control branches that independently receive source-domain supervision without a penalty encouraging them to mimic the source prediction.

Using the same frozen DINOv3 + 3D U-Net source checkpoint and the same seven prostate target cohorts, three independent task-module training runs yield:

MethodMean Dice
Unadapted source model0.6830
Axial-context prediction0.7113 ± 0.0059
Shuffled cross-plane correspondence0.7147 ± 0.0049
Static three-view features + task adapter0.7160 ± 0.0126
Static 3D refinement module0.7202 ± 0.0104
Exact cross-plane prediction + task adapter0.7306 ± 0.0045

These controls are explicitly allowed to improve segmentation, addressing the concern that their lack of improvement was imposed by training. Exact cross-plane prediction improves over the static three-view control by approximately 1.46 Dice percentage points, supporting the additional information provided by exact geometric correspondence.

The stronger static 3D refinement module also performs well. CP-JEPA achieves a mean advantage of approximately 1.04 Dice percentage points over this baseline, providing further evidence of its effectiveness in these experiments.

3. Does a common grid imply anatomical correspondence across patients?

We appreciate this important distinction. A common grid does not establish precise anatomical correspondence across patients. Our method requires no inter-patient registration, and the memory stores task-correction residuals rather than reference segmentations for direct transfer. Its role is therefore best understood as a coarse correction prior modulated by the current case.

We added a comparison between spatial-grid memory and global memory. Across four prostate cohorts with memory enabled and three task-adapter pairings, mean Dice was 0.6702 with spatial memory, 0.6626 with global memory, and 0.6666 in single-volume mode. These results support the usefulness of spatial organization, while not establishing precise anatomical correspondence between grid locations.

We also examined different case orders and found that the benefits depend on both task-adapter quality and historical content. Among the 12 combinations of task-adapter pairing and case order, 58.3% yielded positive gains. One pairing improved performance under all four orders by 2.04–2.48 Dice percentage points; another also yielded positive gains, while the remaining pairing showed slight degradation. These findings reveal the dependence of continual-memory benefits on the task-adapter pairing.

These additional experiments extend the analysis beyond whether memory improves aggregate performance to the more informative questions of which historical information can benefit the current case and how to evaluate its benefits and risks. We thank the reviewer again for highlighting this direction, which will guide our future research.

Reviewer_2KrhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a new test-time adaptation method for medical image segmentation. The method exploits cross-plane correspondence within a single unlabeled volume. It predicts masked axial representations from orthogonal views and translates the findings into corrections attached to the source prediction. The study is conducted with several segmentation tasks and in almost all cases the results are slightly better than the state of the art models.

Critical strengths

The work addresses a practical challenge: domain adaptation for medical image segmentation given 3D volumetric data that is acquired differently compared to the training data. The overall idea of using volume-based registration is interesting, however not particularly novel.

Critical weaknesses

The proposed work is not particularly novel and the results only marginally advance SOTA performance. While performance metrics indicate improvements, their practical significance is questionable since there is no meaningful medical evaluation.

Questions for authors

The paper is very difficult to follow; a more clear explanation of the proposed method would have been greatly appreciated.

2 comments

We thank the reviewer for recognizing the practical deployment challenge addressed in this work. Your comments have helped us identify the need to distinguish more clearly among the methodological contributions, performance improvements, and clinical implications. We respond in three parts.

1. Clarifying the method: exploiting within-volume correspondence.

CP-JEPA does not estimate deformations or align different patients. The location of the same voxel across axial, coronal, and sagittal planes is directly determined by image coordinates. We use this correspondence to construct a predictive task within the current unlabeled volume.

Specifically, CP-JEPA first predicts axial features from orthogonal-view features. It then compares these predictions with those obtained from axial context to extract the additional information provided by the orthogonal views. A source-trained task adapter translates this evidence into bounded segmentation corrections, rather than directly replacing the original features. The continual setting further investigates which historical corrections can be used by the current case.

We will reorganize the method description around “prediction–evidence extraction–correction translation–memory retrieval” and explicitly distinguish source-domain training, test-time optimization, and states retained across cases to improve readability.

2. Novelty and improvement magnitude: additional independently trained controls to distinguish predictive learning from conventional refinement.

We agree that improvements in final Dice alone are insufficient to establish the contribution of each component. Accordingly, our additional experiments allow each control branch to receive source-domain segmentation supervision independently, without forcing it to reproduce the source prediction. Using the same prostate source checkpoint and seven target cohorts, three independent task-adapter training runs yield:

MethodMean Dice
Unadapted source model0.6830
Static three-view features + source-trained adapter0.7160 ± 0.0126
Static 3D refinement module0.7202 ± 0.0104
CP-JEPA with exact cross-plane prediction0.7306 ± 0.0045

These results show that source-trained refinement modules contribute to the gains, so the entire improvement should not be attributed to test-time prediction. CP-JEPA improves over the static three-view control by approximately 1.46 Dice percentage points and over the stronger static 3D refinement module by approximately 1.04 percentage points, demonstrating higher mean performance in these experiments. Our method also outperforms existing 3D medical TTA methods in most evaluated settings and achieves a strong advantage in average performance gains.

The contribution of this work is therefore more precisely framed as investigating when within-volume predictive supervision provides additional information and how to translate that information into task-compatible corrections, rather than simply performing three-view registration.

3. Medical relevance: distinguishing improved segmentation accuracy from clinical benefit.

We thank the reviewer for considering the medical applicability of our work. We agree that improvements in segmentation metrics do not directly establish benefits in clinical outcomes. The practical value of medical imaging methods also encompasses robustness across acquisition conditions, annotation requirements, and deployment resources, and should therefore be assessed beyond whether a clinical outcome study has been conducted.

This work addresses a specific deployment challenge: source models may deteriorate when encountering new scanners or acquisition protocols, while the current case typically lacks manual annotations. CP-JEPA uses the volume’s intrinsic geometric information to provide adaptation supervision, without requiring additional annotations for each test case. We also assess the consistency of its benefits through boundary metrics, case-level analyses of improvements and degradations, and paired statistical analyses.

The method also offers measurable advantages in adaptation resources. For the compact prostate implementation, under the same backbone and cached-feature conditions, it reduces the number of parameters optimized at test time by 29.2% and peak GPU memory usage by 38.5% relative to the decoder-update control. These results suggest potential value for deployment under resource constraints.

The contribution therefore extends beyond improving average Dice to exploring an adaptation mechanism that requires no annotations for the current case and reuses an existing source model. We will present these practical benefits more clearly, while treating clinical benefit validation as future work and keeping clinical conclusions separate from algorithmic performance metrics.

To further clarify, we exploit known geometric correspondence between orthogonal planes within the same volume, rather than registration across patients. The key is to obtain case-specific evidence through test-time prediction and translate it into task-compatible segmentation corrections, going beyond multi-view fusion alone. With the adapter weights held fixed, exact predictive evidence improves Dice by 1.45 percentage points over zero evidence, with a joint 95% confidence interval of [0.54, 2.65] percentage points, supporting the independent contribution of test-time prediction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.