- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies logit distillation for brain-tumor segmentation when the student observes only a subset of MRI modalities and the teacher observes all modalities. It argues that part of the teacher–student logit correction is not predictable from the observed modalities. The proposed Manifold-Constrained Distillation (MCD) estimates an output-space basis from the student’s finite-difference responses to modality-specific, nominally label-preserving input perturbations. It then projects the teacher correction onto this basis under the local Fisher metric and uses the projected correction as the distillation target.
Critical strengths
-
The paper targets an important practical setting with a lightweight inference path. Missing MRI modalities are common in practice, and avoiding feature-level alignment can reduce inference-time complexity. Evaluation across all 15 nonempty modality patterns on two BraTS versions is valuable.
-
The empirical improvements and several controlled ablations are promising. MCD improves the most challenging ET metric by meaningful margins on BraTS 2018 and 2024.
Critical weaknesses
-
The estimated space is not the student’s reachable prediction space. MCD uses input-transformation directions, whereas learnable corrections are governed by the parameter Jacobian. A parameter update can produce predictable corrections outside the input-tangent span. Therefore, MCD may remove realizable signal, and its guarantee depends on an unverified bias condition.
-
The experiments do not validate the theory and contain reporting inconsistencies. Key theoretical quantities and uncertainty estimates are missing. Appendix Table 4 omits the stated standard deviations and norm-matched baseline; Section 5.4 reverses the reported trend; and Table 3 does not support the claim of consistent outperformance. These issues weaken both robustness and mechanism claims.
Questions for authors
-
Please provide the omitted standard deviations and norm-matched row for Appendix Table 4, or clarify whether the caption and accompanying text are incorrect.
-
Does the computational-cost comparison include the additional perturbed student forward passes per observed modality during training, or only inference FLOPs?
W1: The estimated space is not the student's reachable prediction space
We sincerely thank the reviewer for this insightful comment. We agree that parameter-induced prediction variations can span a much larger space than input-induced tangent variations, particularly in overparameterized networks. However, parameter realizability does not necessarily imply learnability from the currently available modalities. A network may fit arbitrary prediction corrections through parameter updates, but such corrections are not necessarily predictable or generalizable from the observed modalities.
The objective of MCD is therefore not to characterize the entire parameter-reachable space, but to identify prediction variations supported by the currently available modalities. We use modality-specific input perturbations to estimate this input-observable subspace and retain teacher corrections within it. We acknowledge that this estimation may exclude some predictable corrections outside the estimated subspace, introducing a projection bias $\delta^2 $.
Δz⊥=(μ(xS)−zS)⊥+ϵ⊥,
where ϵ⊥ represents conditionally unpredictable supervision, while (μ(xS)−zS)⊥ may contain predictable and potentially learnable corrections. The latter introduces a projection bias quantified by δ2.
According to Proposition 2, the projection is beneficial when:
σϵ,⊥2>δ2.
To examine this condition, we conduct additional experiments on BraTS 2018. The results are shown in Table S1.
Table S1. Empirical verification of the theoretical improvement condition.
The positive differences indicate that, under the evaluated settings, the benefit of removing unpredictable supervision outweighs the induced projection bias.
W2.&Q1 Insufficient experimental validation and inconsistencies in result reporting
(1) Appendix Table 4: Standard deviations and norm-matched baseline
We report the mean and standard deviation across three independent random seeds and include the missing norm-matched correction baseline.
Table S2. Updated results for Appendix Table 4
(2) Inconsistency between Section 5.4 and Figure 3(b) We identified a typo in the formula in Section 5.4: the reported values correspond to the reachable-space coverage, ∣Δz∥∣F2/∣Δz∣F2, rather than the unreachable fraction. We will correcte the formula and revise the description to clarify that reachable-space coverage decreases from 69% to 42% as more modalities are missing, consistent with Figure 3(b).
Table 3: Clarification of the consistent outperformance claim
As shown in Table 3, MCD achieves the best average Acc-2 and F1 scores (76.3% and 75.5%, respectively), outperforming the second-best method, UMDF (76.0% and 75.1%). Although MCD does not lead in every individual configuration, it achieves the best overall average performance across all seven modality combinations. We will revise the relevant wording in accordance with your suggestion.
(3) Additional theoretical quantities and uncertainty estimates
We supplement the empirical estimates of the projection-bias term δ2 and the orthogonal residual energy σϵ,⊥2 in Table S1. As you suggested, we tried our best to conduct three additional runs with different random seeds on the BraTS2018 dataset in Table S2.
Q2: Computational overhead of tangent-space estimation
We clarify that the computational-cost comparison in Fig. 1(c) reports inference FLOPs. To provide a complete comparison, we additionally report the training overhead.
Table S3. Computational cost comparison