- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a test-time medical image segmentation method, namely CP-JEPA. Instead of updating the segmentation network, the method trains predictors to estimate masked axial representations using coronal and sagittal views. A source-trained adapter then translates the predictive advantage of these cross-plane views into bounded segmentation corrections. The method also equips a memory grid to perform task-space corrections across volumes for continual adaptation. The method is evaluated on prostate, cardiac, and liver MRI under various adaptation protocols.
Critical strengths
-
Although there are many test-time adaptation objectives for segmentation tasks, leveraging the geometry of 3D medical scans, which is in this case, matching anatomical locations across orthogonal planes is a smart way to generate a test-time signal.
-
The authors conducted evaluations across multiple anatomies, MRI modalities, and adaptation setups. The results on the cardiac benchmark are also encouraging.
Critical weaknesses
-
The experiments don't distinguish the benefit of the test-time cross-plane prediction from the source-trained fully supervised adapter. During training, the control branches are penalized and forced to mimic the source prediction. They show no improvement by design indeed. It is also possible the gains come from just having a source-trained refinement module looking at orthogonal features, rather than test-time predictive learning itself. Also, the reported 79.4% cross-plane predictive advantage in Fig. 4 is measured on only 350 sampled voxels from a single prostate volume, far too thin to show it generalizes across subjects and shifts.
-
The paper reported a large number of TTA baselines, but Appendix A.5 acknowledges the baselines are not reproductions rather rough adaptations of original ones, and some like ASR coming from different fields like speech recognition. Also more relevant multi-view medical TTA paper (Joshi et al.) is completely skipped in the main accuracy tables, and just appeared only in a supplementary compute table. In addition, the method does not consistently dominate, for example standard TENT actually beats it in several setups. Finally, despite claiming to run three random seeds, there are no error bars or standard deviations for Dice improvements. There are also a few numerical contradictions between the text and tables, e.g., single-volume Dice of 0.7114, whereas Table 1 reports 0.7189.
Questions for authors
- Could the authors provide a baseline comparison that uses a source-trained 3D refinement module on the same cross-plane views (without any test-time JEPA optimization) and ideally with a fair setup of aligned compute and parameters?
- Why was the paper Joshi et al. excluded from the segmentation performance tables? Can you also provide standard deviations or paired confidence intervals for your main results, and clarify the results inconsistent CHAOS and TENT scores between the main text and appendix?
- Since the continual setup relies on task-specific residuals across patients, how sensitive is it to patient ordering differences? Please report performance across different patient permutations and detail how often the gating mechanism described in teh paper rejects historical corrections.
2 comments
4. How will standard deviations, confidence intervals, and numerical inconsistencies be addressed?
Space constraints prevented us from including standard deviations in Table 1. We now provide additional statistics from three independent training seeds and will incorporate these into the revised manuscript in the coming days.
| Dataset | Single-volume improvement over the frozen model, with 95% CI | Continual-volume improvement over the frozen model, with 95% CI |
|---|---|---|
| Prostate T2 | +3.34 [2.40, 4.58] percentage points | +4.47 [3.40, 5.74] percentage points |
| Cardiac M&Ms | +3.34 [2.86, 3.88] percentage points | +6.01 [5.16, 6.93] percentage points |
| CHAOS | +0.30 [0.14, 0.46] percentage points | +3.91 [−0.45, 8.24] percentage points |
The mean improvements on prostate and cardiac segmentation are supported by paired statistical analyses. The continual CHAOS result has greater uncertainty; we will explicitly retain this qualification.
We investigated the discrepancy between the CHAOS results of 0.7114 and 0.7189, as well as the TENT differences between the main text and appendix. During experimentation, we initially explored configurations using one seed and added two further seeds after finalizing the method to evaluate its stability. Our checks confirmed the reliability of the results reported in Table 1.
5. How sensitive is continual memory to case order, and how often does gating reject historical corrections?
We added experiments on four prostate cohorts with memory enabled, covering three adapter-seed combinations and four case orders.
The main adapter configuration improves over single-volume adaptation under all four orders, with gains ranging from 2.04 to 2.48 Dice percentage points. This indicates that its continual gains do not depend on a single favorable order. Across all 12 conditions, seven yield positive gains and five yield negative gains, with an average improvement of approximately 0.32 percentage points. Differences across adapters also indicate that the effectiveness of continual memory depends substantially on adapter quality.
In the evaluation across four target cohorts and three risk-model training seeds, there were 345 repeated case exposures eligible for historical correction. Of these, 261 benefited from historical corrections, while 42 exhibited substantial degradation. The strategy retained 98.3% of the original positive gain magnitude, and mean Dice increased from 0.6924 to 0.6978. These results provide encouraging evidence of an improved balance between benefits and risks.
We thank the reviewer for these suggestions. The additional experiments clarify that cross-plane prediction provides information beyond the controls, the source-supervised adapter translates that information into segmentation corrections, and continual memory can yield average gains across case orders while requiring careful attention to stability.
We thank the reviewer for recognizing the value of using orthogonal geometric correspondence to construct test-time supervision. Your comments helped us further distinguish the roles of test-time prediction, the source-supervised task adapter, and continual memory. We have added independently trained controls, cross-patient analyses, a MuVi comparison using the same source model, paired confidence intervals, and case-order experiments.
1. Do the gains primarily come from the source-supervised adapter rather than test-time prediction?
We added independently trained controls without requiring the control branches to remain close to the source prediction, allowing every branch to learn effective corrections through source-domain segmentation supervision.
All branches use the same frozen DINOv3 + 3D U-Net checkpoint, source training data, target cases, and 50 epochs of adapter training. Results across three independent adapter training seeds are:
Exact cross-plane prediction outperforms static three-view averaging by 1.46 Dice percentage points, with positive paired differences for all three seeds. After averaging across seeds, the patient-level paired 95% confidence interval is [1.04, 1.91] percentage points. Thus, the additional source-supervised adapter and static multi-view inputs do not fully explain the gains.
Following your suggestion, we also added a static 3D refinement module with a closely matched total parameter count: approximately 607,100 parameters, compared with approximately 606,800 parameters for the CP-JEPA adapter and temporary predictors combined. CP-JEPA achieves a mean advantage of 1.04 percentage points. These results support an additional contribution from test-time cross-plane prediction and show higher mean performance than static 3D refinement in this comparison.
2. Does the cross-plane predictive advantage generalize across patients?
We expanded the original single-case example to 248 independent patients across seven target cohorts.
In all seven cohorts, exact cross-plane prediction achieves a positive mean validation cosine-similarity advantage over axial prediction. When keeping the same adapter fixed, exact predictive evidence also provides a positive mean segmentation gain over zero evidence in every cohort.
These results show that cross-plane predictability is not an isolated observation from a single case. They support the additional predictive information provided by orthogonal geometry. However, this information requires task-compatible translation and should not be directly equated with segmentation improvement or a safety guarantee. This distinction is one of the central motivations of our work.
3. Why was MuVi omitted from the accuracy comparison, and are the other baselines fair?
We acknowledge that methods such as ASR are not well suited to this setting. We will therefore remove comparisons with ASR, IMSE, and similarly unsuitable methods from the revision.
We considered MuVi and attempted to use it in our earlier experiments. Under its default settings, we observed overfitting, with source-to-target Dice gaps reaching 20 percentage points on some datasets, and consequently omitted it from the main table. To provide a comparison under matched conditions, we have now added experiments based on the core of its official training code, using the same DINOv3 hybrid backbone, source checkpoint, and seven target cohorts as CP-JEPA:
Under this setting, CP-JEPA outperforms MuVi by 2.70 Dice percentage points. MuVi exhibits performance degradation, which may reflect our limited exploration of its adaptation hyperparameters under the available time constraints. We will continue investigating these settings during the remaining rebuttal period.
We also retain the results showing TENT’s advantages in some settings. We appreciate the reviewer’s correction and will remove any wording suggesting that our method outperforms existing approaches under every setting.