acceptodds
Under review as a conference paper at ICLR 2027

MANIFOLD-CONSTRAINED DISTILLATION FOR MISSING-MODALITY BRAIN TUMOR SEGMENTATION

Abstract

Magnetic Resonance Imaging (MRI)-based brain tumor segmentation is crucial for clinical diagnosis and treatment planning, yet its accuracy degrades severely when part of the imaging modalities is missing. Knowledge distillation (KD) alleviates this by using a full-modality teacher to guide a modality-constrained student. Existing methods mainly rely on feature-level distillation, which incurs additional memory and computational costs, especially for high-resolution med- ical images. In contrast, logit-only distillation provides a lightweight alternative without explicit feature alignment, yet remains unexplored in missing-modality brain tumor segmentation. However, native logit distillation may force the student toward teacher predictions outside its reachable prediction space, as the teacher exploits modality information unavailable to the student. In this paper, we refor- mulate logit distillation under missing modalities as a manifold-constrained tar- get estimation problem, and propose Manifold-Constrained Distillation (MCD), a novel manifold-aware logit distillation framework. MCD projects the teacher’s correction onto the student’s transferable space under the Fisher geometry, re- taining only the realizable component as the target. On BraTS 2018 and BraTS 2024 benchmarks, MCD consistently improves segmentation accuracy over strong baselines across diverse missing-modality scenarios. To further demonstrate the generalizability of our framework, we extend MCD to an additional task, multi- modal sentiment analysis, and again observe the superiority of our MCD model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Brain Tumor Segmentation, Missing Modality, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_B9yVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies logit distillation for brain-tumor segmentation when the student observes only a subset of MRI modalities and the teacher observes all modalities. It argues that part of the teacher–student logit correction is not predictable from the observed modalities. The proposed Manifold-Constrained Distillation (MCD) estimates an output-space basis from the student’s finite-difference responses to modality-specific, nominally label-preserving input perturbations. It then projects the teacher correction onto this basis under the local Fisher metric and uses the projected correction as the distillation target.

Critical strengths
  1. The paper targets an important practical setting with a lightweight inference path. Missing MRI modalities are common in practice, and avoiding feature-level alignment can reduce inference-time complexity. Evaluation across all 15 nonempty modality patterns on two BraTS versions is valuable.

  2. The empirical improvements and several controlled ablations are promising. MCD improves the most challenging ET metric by meaningful margins on BraTS 2018 and 2024.

Critical weaknesses
  1. The estimated space is not the student’s reachable prediction space. MCD uses input-transformation directions, whereas learnable corrections are governed by the parameter Jacobian. A parameter update can produce predictable corrections outside the input-tangent span. Therefore, MCD may remove realizable signal, and its guarantee depends on an unverified bias condition.

  2. The experiments do not validate the theory and contain reporting inconsistencies. Key theoretical quantities and uncertainty estimates are missing. Appendix Table 4 omits the stated standard deviations and norm-matched baseline; Section 5.4 reverses the reported trend; and Table 3 does not support the claim of consistent outperformance. These issues weaken both robustness and mechanism claims.

Questions for authors
  1. Please provide the omitted standard deviations and norm-matched row for Appendix Table 4, or clarify whether the caption and accompanying text are incorrect.

  2. Does the computational-cost comparison include the KK additional perturbed student forward passes per observed modality during training, or only inference FLOPs?

1 comment

W1: The estimated space is not the student's reachable prediction space

We sincerely thank the reviewer for this insightful comment. We agree that parameter-induced prediction variations can span a much larger space than input-induced tangent variations, particularly in overparameterized networks. However, parameter realizability does not necessarily imply learnability from the currently available modalities. A network may fit arbitrary prediction corrections through parameter updates, but such corrections are not necessarily predictable or generalizable from the observed modalities.

The objective of MCD is therefore not to characterize the entire parameter-reachable space, but to identify prediction variations supported by the currently available modalities. We use modality-specific input perturbations to estimate this input-observable subspace and retain teacher corrections within it. We acknowledge that this estimation may exclude some predictable corrections outside the estimated subspace, introducing a projection bias $\delta^2 $.

Δz⊥=(μ(xS)−zS)⊥+ϵ⊥,\Delta z^\perp=(\mu(x_S)-z_S)^\perp+\epsilon^\perp,

where ϵ⊥\epsilon^\perp represents conditionally unpredictable supervision, while (μ(xS)−zS)⊥(\mu(x_S)-z_S)^\perp may contain predictable and potentially learnable corrections. The latter introduces a projection bias quantified by δ2\delta^2.

According to Proposition 2, the projection is beneficial when:

σϵ,⊥2>δ2.\sigma_{\epsilon,\perp}^2>\delta^2.

To examine this condition, we conduct additional experiments on BraTS 2018. The results are shown in Table S1.

Table S1. Empirical verification of the theoretical improvement condition.

Available modalitiesδ2\delta^2σϵ,⊥2\sigma_{\epsilon,\perp}^2δ2\delta^2-σϵ,⊥2\sigma_{\epsilon,\perp}^2
10.14580.45020.3044
20.07870.22050.1418
30.05260.10670.0541

The positive differences indicate that, under the evaluated settings, the benefit of removing unpredictable supervision outweighs the induced projection bias.

W2.&Q1 Insufficient experimental validation and inconsistencies in result reporting

(1) Appendix Table 4: Standard deviations and norm-matched baseline

We report the mean and standard deviation across three independent random seeds and include the missing norm-matched correction baseline.

Table S2. Updated results for Appendix Table 4

MethodWT Dice (Mean ± Std)TC Dice (Mean ± Std)ET Dice (Mean ± Std)Average Dice
Original correction85.75±0.8176.92±0.6560.91±0.5774.53±0.68
Norm-matched86.03±0.6976.71±0.8261.86±0.3774.87±0.59
MCD86.53±0.7478.76±0.6164.11±0.5876.46±0.61

(2) Inconsistency between Section 5.4 and Figure 3(b) We identified a typo in the formula in Section 5.4: the reported values correspond to the reachable-space coverage, ∣Δz∥∣F2/∣Δz∣F2|\Delta z^\parallel|_F^2/|\Delta z|_F^2, rather than the unreachable fraction. We will correcte the formula and revise the description to clarify that reachable-space coverage decreases from 69% to 42% as more modalities are missing, consistent with Figure 3(b).

Table 3: Clarification of the consistent outperformance claim

As shown in Table 3, MCD achieves the best average Acc-2 and F1 scores (76.3% and 75.5%, respectively), outperforming the second-best method, UMDF (76.0% and 75.1%). Although MCD does not lead in every individual configuration, it achieves the best overall average performance across all seven modality combinations. We will revise the relevant wording in accordance with your suggestion.

(3) Additional theoretical quantities and uncertainty estimates

We supplement the empirical estimates of the projection-bias term δ2\delta^2 and the orthogonal residual energy σϵ,⊥2\sigma_{\epsilon,\perp}^2 in Table S1. As you suggested, we tried our best to conduct three additional runs with different random seeds on the BraTS2018 dataset in Table S2.

Q2: Computational overhead of tangent-space estimation

We clarify that the computational-cost comparison in Fig. 1(c) reports inference FLOPs. To provide a complete comparison, we additionally report the training overhead.

Table S3. Computational cost comparison

MetricConventional KDMCD
Training GFLOPs1491.21922.9
Training time per epoch106s147s
Inference GFLOPs257.3257.3
Inference time36.1min36.1min
Peak GPU memory22.6GB23.8GB
Reviewer_5p8yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue in missing-modality brain tumor segmentation where traditional knowledge distillation struggles to distinguish between transferable and non-transferable teacher knowledge. A manifold-constrained knowledge distillation method (MCD) is proposed, which guides the student in learning teacher corrections through subspace estimation and Fisher projection screening. Experiments demonstrate that this method achieves competitive segmentation performance across multiple datasets and modality-missing scenarios.

Critical strengths

1.The research motivation is relatively clear. The paper focuses on the problem of information asymmetry between teacher and student models under missing modality conditions and analyzes the impact of modality absence on knowledge distillation through preliminary experiments. 2.The method design is relatively concise. The paper constructs the distillation target via subspace estimation and Fisher projection, provides relevant theoretical analysis, and does not introduce additional inference-stage computational steps. 3.The experiments are relatively comprehensive. The paper evaluates multiple missing modality scenarios on two BraTS datasets and provides further empirical support through cross-task experiments and ablation studies.

Critical weaknesses

1.Insufficient justification for the cut-space approach. It is not fully demonstrated whether the prediction changes caused by manually introduced perturbations can represent knowledge that students can actually learn.
2.Lack of direct verification for the knowledge selection mechanism. It has not been validated whether the teacher corrections that were discarded are indeed unlearnable from the existing modalities.
3.Incomplete key control experiments. The results for the described equal-amplitude correction control are missing, making it impossible to fully distinguish between the benefits brought by directional selection and those from diminished distillation strength.

Questions for authors

1.Can the authors compare, through experiments, the predictability of teacher corrections that are retained versus discarded from existing modalities, to verify that the constructed subspace can indeed distinguish between transferable and non-transferable knowledge? 2.Can the authors supplement the appendix with the experimental results of the norm-matched correction described therein, and explain whether the proposed projection method still maintains a performance advantage when the correction magnitude is controlled?

2 comments

W1: Insufficient justification for the cut-space approach

We thank the reviewer for raising this important concern. To better justify the proposed cut-space construction, we provide a theoretical analysis showing that sufficiently rich label-preserving perturbations can recover the prediction space supported by the observed modalities.

Theoretical formulation. Let MS\mathcal M_S denote the local support manifold of the observed-modality data distribution. We assume that MS\mathcal M_S is an rr-dimensional C2C^2 manifold around xSx_S. Let USU_S be an orthonormal basis of its tangent space. We define the modality-supported prediction space as

SS∗=Range⁡(JSUS),\mathcal S_S^* = \operatorname{Range}(J_SU_S),

where JS=∂PS(xS;θ)/∂xSJ_S=\partial P_S(x_S;\theta)/\partial x_S is the student prediction Jacobian.

Consider a general label-preserving perturbation distribution QS,ϵQ_{S,\epsilon} supported on the local manifold neighborhood. We impose the following local coverage condition: its density with respect to the manifold volume measure is positive throughout the local neighborhood, and its local tangent-coordinate second moment satisfies

E[uu⊤∣∥u∥≤ϵ]⪰c0ϵ2Ir,\mathbb E[u u^\top \mid \|u\|\leq\epsilon] \succeq c_0\epsilon^2 I_r,

where uu denotes the local tangent coordinates and c0>0c_0>0 is independent of sufficiently small ϵ\epsilon.

For a perturbed sample x~S\tilde x_S, define the normalized prediction response

dϵ(x~S)=PS(x~S;θ)−PS(xS;θ)ϵ.d_\epsilon(\tilde x_S) = \frac{P_S(\tilde x_S;\theta)-P_S(x_S;\theta)} {\epsilon}.

The corresponding population response covariance is

Cϵ=EQS,ϵ[dϵdϵ⊤].C_\epsilon = \mathbb E_{Q_{S,\epsilon}} [d_\epsilon d_\epsilon^\top].

Since MS\mathcal M_S is locally smooth, its points admit the expansion

x~S=xS+USu+O(∥u∥2).\tilde x_S = x_S+U_Su+O(\|u\|^2).

By the local smoothness of the student prediction function,

PS(x~S)−PS(xS)=JSUSu+O(∥u∥2).P_S(\tilde x_S)-P_S(x_S) = J_SU_Su+O(\|u\|^2).

Consequently,

Cϵ=JSUSE[uu⊤]ϵ2US⊤JS⊤+O(ϵ).C_\epsilon = J_SU_S \frac{\mathbb E[uu^\top]}{\epsilon^2} U_S^\top J_S^\top +O(\epsilon).

Under the non-degenerate coverage condition, the leading-order covariance has the same range as JSUSJ_SU_S. Provided that JSUSJ_SU_S has stable rank ss and a positive minimum nonzero singular value, the leading ss-dimensional eigenspace of CϵC_\epsilon converges to the modality-supported prediction space:

lim⁡ϵ→0∥Πs(Cϵ)−ΠSS∗∥2=0.{ \lim_{\epsilon\rightarrow0} \left\| \Pi_s(C_\epsilon) - \Pi_{\mathcal S_S^*} \right\|_2 =0. }

Furthermore, with KK independently sampled perturbations, the empirical covariance converges to its population counterpart. Under bounded fourth moments and a positive eigengap, the Davis–Kahan theorem gives

∥Π^K,ϵ−ΠSS∗∥2=Op(K−1/2+ϵγ).{ \|\hat\Pi_{K,\epsilon} - \Pi_{\mathcal S_S^*}\|_2 = O_p\left( \frac{K^{-1/2}+\epsilon}{\gamma} \right). }

where γ>0\gamma>0 denotes the limiting spectral separation.

This result establishes that the proposed perturbation-based cut-space estimator can consistently recover the modality-supported prediction space, provided that the perturbation distribution sufficiently covers the local support of the observed-modality data distribution.

W2&Q1: Lack of direct verification for the knowledge selection mechanism

We thank the reviewer for this suggestion. We compare the predictability of retained and discarded teacher corrections using separate predictors with identical architectures, training configurations, and data splits on BraTS2018. Both predictors are trained exclusively on the student's observed modalities and evaluated on independent subjects.

Table S1. Missing-modality information rescue

Correction componentObserved modalities (R2R^2 ↑)All modalities (R2R^2 ↑)Average Dice (%) ↑
Δz∥\Delta z^\parallel0.550.7776.46
Δz⊥\Delta z^\perp.0.120.7973.23

W3&Q2: Incomplete key control experiments

We thank the reviewer for pointing out the missing norm-matched control experiment. We compare the original correction, the projected correction, and the norm-matched projected correction under identical training settings on BraTS2018.

Table S2. Comparison of original, projected, and norm-matched corrections

MethodWT Dice (Mean ± Std)TC Dice (Mean ± Std)ET Dice (Mean ± Std)Average Dice
Original correction85.75±0.8176.92±0.6560.91±0.5774.53±0.68
Norm-matched86.03±0.6976.71±0.8261.86±0.3774.87±0.59
MCD86.53±0.7478.76±0.6164.11±0.5876.46±0.61

Thank you for your reply. I will improve my score.

Reviewer_G9GvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a knowledge distillation framework called "Manifold-Constrained Distillation" (MCD), aimed at addressing brain tumor segmentation in scenarios where certain modalities are missing. Although the paper explores a problem of practical importance and offers a novel geometric perspective on knowledge distillation for cases with missing modalities, I have serious reservations regarding the theoretical rigor of the proposed formulation, the validity of the underlying geometric assumptions, the comprehensiveness of the experimental comparisons, and whether the reported performance gains are sufficient to substantiate the paper's core arguments.

Critical strengths
  1. Missing-modality brain tumor segmentation is an important problem because multimodal MRI acquisitions may be incomplete in real clinical settings.
  2. The paper identifies a meaningful limitation of conventional teacher–student distillation: a student with incomplete observations may be unable to reproduce all information contained in the teacher's predictions.
  3. The proposed geometric interpretation may be applicable to other multimodal learning problems beyond medical image segmentation.
Critical weaknesses
  1. The practical significance of the proposed geometric constraints has not been sufficiently justified. Authors should more clearly distinguish between information that is fundamentally unrecoverable from the observed modalities and information that is merely inaccessible within the proposed local approximation framework. Without this distinction, the practical implications of the proposed geometric constraints remain ambiguous.
  2. The assumptions underlying the definition of the "student-accessible manifold" require further justification. The validity of the construction of the student-accessible tangent space relies on several insufficiently tested assumptions: (1) whether the tangent space, induced by local input perturbations, fully encompasses all possible predictions that the student model can learn through optimization; (2) whether different perturbation strategies might lead to vastly different projections, thereby resulting in distinct distillation objectives; and (3) whether local linear approximation remains reliable when the student model’s prediction function is highly non-linear, or when a significant distribution shift occurs due to missing modalities. The authors should provide more robust theoretical or empirical evidence to demonstrate that the estimated tangent space effectively approximates the information accessible to the student model.
  3. The mathematical derivation underlying the error decomposition requires correction and clarification. There appears to be an inconsistency in the signs used for terms related to bias. Furthermore, the derivation should explicitly define the conditions under which the projection onto the Fisher metric holds, including the properties of the metric itself, the approach to local approximation, and the handling of potential degenerate directions. The authors should provide a complete derivation, clearly state all assumptions, and verify whether the theoretical conclusions remain valid after addressing the aforementioned inconsistencies.
  4. The experimental comparisons fail to sufficiently isolate the specific contribution of "manifold-constrained distillation." Although the paper reports performance superior to existing methods, the current experimental design is not compelling enough to demonstrate that these improvements are indeed attributable to the proposed manifold constraints. To clarify the contribution of the proposed mechanism, it is recommended to include comparisons with the following approaches: (1) traditional knowledge distillation using teacher model logits or probability distributions; (2) feature-based or relational knowledge distillation; (3) distillation methods based on confidence weighting or uncertainty awareness; (4) gradient-alignment-based distillation; (5) distillation using random or low-rank projections; (6) consistency regularization based on perturbations (excluding the proposed manifold projection); (7) a baseline using a "random projection" with matched dimensionality; and (8) a baseline based on "perturbation consistency." These comparisons would help determine whether the observed improvements stem from the proposed geometric interpretation or simply from reinforcing the stability of the student model's predictions under local perturbations.
  5. The experimental evaluation fails to sufficiently validate the model's robustness in scenarios where certain modalities are missing. The primary application of this study is brain tumor segmentation in cases where some MRI modalities are unavailable. Consequently, it is essential to examine the model's performance across various specific combinations of missing modalities. The authors should provide a more detailed analysis, including: (1) the model's performance for each specific combination of missing modalities; (2) how the model's performance changes as the number of available modalities decreases; (3) the model's segmentation performance across different tumor sub-regions; and (4) whether the proposed projection mechanism yields more significant gains as the information gap between the teacher and student models widens. Without such evidence, the logical link between the proposed geometric mechanism and the observed improvements in segmentation performance remains incomplete.
  6. The performance gains reported in the paper require more rigorous statistical and numerical analysis. The paper should include indicators of statistical uncertainty (such as standard deviations or confidence intervals) and, ideally, subject-level paired comparisons. Furthermore, the experimental tables and the descriptions of implementation details should be more consistent and comprehensive. Any missing data points or inconsistencies in the reporting of hyperparameters must be clarified to ensure that the experimental results can be independently reproduced.
  7. Evidence regarding qualitative assessment and underlying mechanisms is currently insufficient. The paper should provide direct evidence demonstrating that the proposed projection mechanism effectively separates "knowledge in the teacher model accessible to the student" from "knowledge inaccessible to the student." At a minimum, the following analyses should be added: (1) Use visualization or quantitative metrics to illustrate the teacher-student discrepancies (both post-projection and residual), in order to determine whether the projection operation successfully removes "supervisory information truly inaccessible to the student"; (2) Analyze the evolution of the projection rank, tangent-space stability, and the degree of teacher-student alignment throughout the training process.
Questions for authors
  1. How do the authors justify interpreting a tangent space induced by local input perturbations as the full set of predictions accessible to the student through learning?
  2. Can the authors provide a corrected and complete proof of Proposition 2, including the sign and interpretation of the bias-related term?
  3. How sensitive is MCD to the perturbation magnitude, perturbation distribution, and number of tangent directions?
  4. How does MCD compare with random projections, low-rank projections, and perturbation-consistency regularization under matched computational budgets?
  5. Does the benefit of MCD systematically increase as the information discrepancy between the complete-modality teacher and missing-modality student becomes larger?
  6. Are the reported improvements statistically significant across subjects, missing-modality configurations, and independent training runs?
  7. What is the computational overhead of tangent-space estimation relative to conventional knowledge distillation?
  8. Can the authors provide direct evidence that the projected distillation targets preserve useful teacher knowledge while suppressing genuinely inaccessible components?
4 comments

W1: Practical significance and validation of the knowledge selection mechanism

Thanks for your insightful comments. We clarify that fundamentally unrecoverable information and locally inaccessible information are related but distinct concepts. The former refers to statistically unpredictable teacher information, defined as ϵ=zT−μ(xS)\epsilon = z_T - \mu(x_S), where μ(xS)=E[zT∣xS]\mu(x_S) = \mathbb{E}[z_T \mid x_S] and E[ϵ∣xS]=0\mathbb{E}[\epsilon \mid x_S] = 0. The latter refers to the component of teacher correction outside the student's locally reachable tangent space:

Δz⊥=(μ−zS)⊥+ϵ⊥.\Delta z^\perp = (\mu - z_S)^\perp + \epsilon^\perp.

Our MCD employs Fisher-metric projection to retain the locally reachable correction Δz∥\Delta z^\parallel while removing Δz⊥\Delta z^\perp. This reduces non-transferable noise but introduces a projection bias δ2\delta^2. According to Proposition 2, improvement is achieved when:

σ2(ϵ⊥)−δ2>0.\sigma^2(\epsilon^\perp) - \delta^2 > 0.

To verify this condition empirically, we estimate both terms under different missing-modality settings on BraTS 2018. The results are presented in Table S1.

Table S1. Empirical verification of the theoretical improvement condition.

Available modalitiesδ2\delta^2σ2(ϵ⊥)\sigma^2(\epsilon^\perp)σ2(ϵ⊥)−δ2\sigma^2(\epsilon^\perp)-\delta^2
10.14580.45020.3044
20.07870.22050.1418
30.05260.10670.0541

All settings yield positive differences, indicating that the noise reduction outweighs the projection bias and providing empirical support for Proposition 2.

W2 & Q1, Q3: Assumptions underlying the Student-accessible manifold

Regarding the first concern, the student-accessible tangent space characterizes prediction directions supported by the observed modalities. Although parameter-driven reachability may be broader, the ability to fit a prediction direction does not imply that its associated information is predictable from the observed modalities. As established in Proposition 1, E[ϵ∣xS]=0E[\epsilon \mid x_S] = 0. Therefore, MCD constrains teacher supervision to observable information rather than the student's overall parameterization capacity.

To address the remaining concerns, we conduct additional experiments to evaluate the sensitivity of tangent-space estimation to perturbation strategies and its stability across missing-modality configurations. The results are presented in Tables S2 and S3.

Table S2. Sensitivity to perturbation strategies.

Perturbation strategyResponse coverage (%)Projection consistency (%)Average Dice
Bias field only37.8969.7476.15
Intensity transformation only31.5364.6375.70
Elastic deformation only35.2457.4175.93
Combined perturbations (MCD)56.14—76.46

Table S3. Stability across missing-modality configurations.

Available modalitiesResponse coverage (%)Average Dice
142.3051.14
257.2564.17
369.0573.22

The results show that combined perturbations achieve higher response coverage and Dice performance. Moreover, response coverage increases consistently with the number of available modalities, supporting the robustness of tangent-space estimation under different missing-modality configurations.

W3 & Q2: Mathematical derivation and Fisher metric conditions

We sincerely thank the reviewer for these valuable comments. We have carefully revisited Proposition 2 and the Fisher-metric derivation and provide the following clarifications.

Proof of Proposition 2. Let t=zS+ut=z_S+u, where u=u∥+u⊥u=u_{\parallel}+u_{\perp}. By Fisher orthogonality, the objective becomes

L=E[∥u∥−Δz∥∥F2]+E[∥u⊥−Δz⊥∥F2]+γE[∥u⊥∥F2].\mathcal{L}=\mathbb{E}\left[\left\Vert u_{\parallel}-\Delta z_{\parallel}\right\Vert_F^2\right]+\mathbb{E}\left[\left\Vert u_{\perp}-\Delta z_{\perp}\right\Vert_F^2\right]+\gamma\mathbb{E}\left[\left\Vert u_{\perp}\right\Vert_F^2\right].

Minimization gives u∥⋆=Δz∥u_{\parallel}^{\star}=\Delta z_{\parallel} and u⊥⋆=Δz⊥1+γu_{\perp}^{\star}=\frac{\Delta z_{\perp}}{1+\gamma}. Thus, as γ→∞\gamma\to\infty, t⋆=zS+Δz∥t^{\star}=z_S+\Delta z_{\parallel}.

Furthermore, since Δz=(μ−zS)+ϵ\Delta z=(\mu-z_S)+\epsilon,

t⋆−μ=−(μ−zS)⊥+ϵ∥.t^{\star}-\mu=-(\mu-z_S)_{\perp}+\epsilon_{\parallel}.

The two components are Fisher-orthogonal. Consequently,

E[∥t⋆−μ∥F2]=δ2+σϵ∥2.\mathbb{E}\left[\left\Vert t^{\star}-\mu\right\Vert_F^2\right]=\delta^2+\sigma_{\epsilon_{\parallel}}^2.

Combining this with E[∥zT−μ∥F2]=σϵ∥2+σϵ⊥2\mathbb{E}\left[\left\Vert z_T-\mu\right\Vert_F^2\right]=\sigma_{\epsilon_{\parallel}}^2+\sigma_{\epsilon_{\perp}}^2, we obtain

E[∥t⋆−μ∥F2]=E[∥zT−μ∥F2]−(σϵ⊥2−δ2).\mathbb{E}\left[\left\Vert t^{\star}-\mu\right\Vert_F^2\right]=\mathbb{E}\left[\left\Vert z_T-\mu\right\Vert_F^2\right]-(\sigma_{\epsilon_{\perp}}^2-\delta^2).

The signs in the original equation are therefore correct: σϵ⊥2\sigma_{\epsilon_{\perp}}^2 represents removed non-transferable noise, whereas δ2\delta^2 represents the additional projection-induced bias. Improvement occurs when σϵ⊥2>δ2\sigma_{\epsilon_{\perp}}^2>\delta^2.

Fisher metric and projection conditions. The Fisher metric is derived from the second-order Taylor expansion of the KL divergence:

DKL(p(z+Δz) ∥ p(z))=12τ2Δz⊤FΔz+O(∥Δz∥3),D_{\mathrm{KL}}\left(p(z+\Delta z)\,\|\,p(z)\right)=\frac{1}{2\tau^2}\Delta z^{\top}F\Delta z+O\left(\left\Vert\Delta z\right\Vert^3\right),

where F=diag⁡(p)−pp⊤F=\operatorname{diag}(p)-pp^{\top}.

Since the softmax Fisher matrix is positive semidefinite and has a null direction corresponding to common logit shifts, we define the projection on the centered logit subspace, where FF is positive definite. For a full-rank tangent basis BB satisfying B⊤FB≻0B^{\top}FB\succ0, the Fisher-orthogonal projector is

ΠF=B(B⊤FB)−1B⊤F.\Pi_F=B(B^{\top}FB)^{-1}B^{\top}F.

This guarantees the orthogonal decomposition and Pythagorean identity.

W4 & Q4 & Q7: Insufficient comparisons with alternative distillation methods

We sincerely thank the reviewer for this constructive suggestion. All additional experiments were conducted on the BraTS 2018 dataset, following the same experimental settings and training protocols in the main manuscript. Specifically, Table S4 isolates the contribution of the proposed manifold constraint against alternative distillation and projection baselines, while Table S5 evaluates the computational overhead of tangent-space estimation relative to conventional KD and compares efficiency under matched training settings. The experimental comparisons are summarized below.

Table S4. Comparison with alternative distillation methods

MethodWT DiceTC DiceET DiceAverage Dice
Logit-based KD85.7576.9260.9174.53
Feature-based KD86.0277.0461.2574.77
Confidence/Uncertainty-aware KD86.1377.5863.1875.63
Gradient-alignment KD86.1777.3662.3175.28
Random projection (matched dimension)85.8777.5161.9275.10
Low-rank projection (matched dimension)86.0877.3962.5875.35
Perturbation consistency (without projection)85.4176.5461.1874.37
MCD (Ours)86.5378.7664.1176.46

Table S5. Computational cost comparison

MethodTraining time/epochPeak GPU memoryInference time
Conventional KD106s22.6GB36.1min
Random projections113s23.3GB36.1min
Low-rank projections115s23.4GB36.1min
Perturbation-consistency regularization184s31.7GB36.1min
MCD (Ours)147s23.8GB36.1min

W5 & Q5 : Robustness under different missing-modality configurations

We evaluate MCD under different numbers of available modalities and report the average segmentation performance across all corresponding modality combinations. Furthermore, we quantify the average teacher-student discrepancy at each modality level and investigate its relationship with the performance gains of MCD.

Table S6. Segmentation performance under different numbers of available modalities

Number of available modalitiesBaseline WTBaseline TCBaseline ETMCD WTMCD TCMCD ET
181.3470.2646.9481.7572.1651.14
285.9177.5760.5887.0679.4864.17
388.4180.8168.6989.4782.7273.22
489.7383.0475.8590.6984.9579.33

Table S7. Relationship between teacher-student discrepancy and MCD performance gains

Number of available modalitiesAverage teacher-student discrepancyBaseline Average DiceMCD Average DiceImprovement
158%66.1868.35+2.17
243%74.6976.91+2.22
331%79.3081.80+2.50
40%82.8784.99+2.12

W6 & Q6: Statistical significance, reproducibility, and computational cost

As you suggested, we tried our best to conduct three additional runs with different random seeds on the BraTS2018 dataset and compared their mean and standard deviation with the original results.

Furthermore, we added the missing norm-matched baseline to Appendix Table 4. As shown below, MCD achieves an average Dice of 76.46%, outperforming the norm-matched baseline (74.87%) by 1.59 percentage points. This demonstrates that the performance gain comes not merely from reducing the correction magnitude, but also from selecting more appropriate correction directions.

Table S8. Statistical analysis of segmentation performance

MethodWT Dice (Mean ± Std)TC Dice (Mean ± Std)ET Dice (Mean ± Std)Average Dice
Original correction85.75±0.8176.92±0.6560.91±0.5774.53±0.68
Norm-matched86.03±0.6976.71±0.8261.86±0.3774.87±0.59
MCD86.53±0.7478.76±0.6164.11±0.5876.46±0.61

Regarding implementation details, we identified a typographical error in Appendix B.2 (Line 752). The reported step-size factors {0.025, 0.05, 0.1, 0.5} should be {0.025, 0.05, 0.1, 0.2}, consistent with the actual experiments reported in Table 5. We will correct this error to ensure consistency and reproducibility.

W7 & Q8: Qualitative assessment and validation of the knowledge selection mechanism

Table S9 compares the predictability of retained and discarded teacher corrections using separate predictors with identical architectures, training configurations, and data splits on BraTS2018. Table S10 tracks the evolution of projection rank, tangent-space stability, and teacher-student alignment during training, providing further evidence for the proposed projection mechanism.

Table S9. Quantitative analysis of projected and residual teacher corrections

Correction componentObserved modalities (R2R^2 ↑)All modalities (R2R^2 ↑)Average Dice (%) ↑
Δz∥\Delta z^\parallel0.550.7776.46
Δz⊥\Delta z^\perp.0.120.7973.23

Table S10. Evolution of geometric properties during training

Training epochProjection rankPrincipal Angles (°)Teacher-student alignment
Epoch 2013—0.92
Epoch 8015680.31
Epoch 15016590.18
Epoch 20016710.12
Final epoch16660.10

We hope that the above clarifications and additional experiments provided in our comments have sufficiently addressed your concerns. If you are satisfied with our responses, we would greatly appreciate it if you could consider updating your score to reflect the newly provided results and clarifications. We remain committed to addressing any further questions or concerns you may have during the discussion phase.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear PC, SAC, AC, and Reviewers,

We sincerely thank all reviewers for their time and constructive feedback. We briefly summarize our responses and additional work during the discussion period as follows:

  1. Theoretical justification: We provided additional theoretical justification for the perturbation-based cut-space construction, clarified the distinction between input-observable and parameter-reachable prediction spaces, and supplemented the complete proof of Proposition 2 and the conditions for Fisher-metric projection.

  2. Validation of the knowledge selection mechanism: We added experiments to verify the theoretical improvement condition, analyze the predictability of Δz∥\Delta z_{\parallel} and Δz⊥\Delta z_{\perp}, and track the evolution of projection rank and tangent-space properties during training.

  3. Additional experimental validation: We added comparisons with alternative distillation methods, perturbation sensitivity analyses, evaluations across different missing-modality configurations, and norm-matched controls.

  4. Statistical analysis and computational efficiency: We supplemented results across multiple random seeds, clarified experimental inconsistencies, and provided additional computational cost analyses.

Overall, we believe these additional analyses have substantially strengthened our work, and all additional results will be incorporated into the revised manuscript. We are grateful that two reviewers indicated their willingness to increase their scores, and appreciate Reviewer G9Gv's recognition of the broader potential of MCD's geometric perspective in multimodal learning. We believe MCD addresses a key challenge in logit-based knowledge distillation under missing-modality settings, where teacher logits may contain knowledge not supported by the student's observable modalities, while opening a new research direction for missing-modality learning.

We sincerely thank the Area Chair and all reviewers once again for their time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.