- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HiPPO distills knowledge from heterogeneous foundation models into a compact, vision-only 3D student network. Guided by local student features, a policy samples components from a shared codebook to construct candidate representations, which are then scored by compatibility rewards computed in each visual teacher’s native representation space. The policy is optimized with group-relative advantages, while the difficulty of matching each candidate modulates the spatial distillation weights. In addition, a separate branch provides privileged semantic supervision using lesion-centered views available only during training. We further introduce MedBridge-Bench, which evaluates classification and segmentation across CT, MRI, and PET/CT. At inference time, only the compact student is retained.
Critical strengths
The problem addressed in this paper is practically relevant, and the evaluation is reasonably broad. Transferring knowledge from multiple training-time teachers into a vision-only student—without requiring additional deployment inputs or auxiliary modules—is a meaningful and well-motivated objective. The authors also report training overhead, which is helpful for assessing the practical cost of the approach. The paper goes beyond reporting final task performance and provides supporting evidence at the mechanism level. The ablations—covering deterministic projection, soft mixtures, random sampling, policy removal, and the spatial weighting scheme—along with feature probes and gradient-path analysis, together offer a solid basis for understanding why and how the method works.
Critical weaknesses
The novelty claim does not sufficiently engage with the most directly related work on visual multi-teacher distillation. AM-RADIO [1], Theia [2], and DUNE [3] already combine teachers with heterogeneous objectives and representation spaces via teacher-specific mappings, and MTKD-RL [4] has explored policy-driven multi-teacher distillation for visual recognition. Consequently, heterogeneous teacher fusion, native-space alignment, or the use of a policy, taken individually, cannot be the sole basis for the paper’s main novelty. To be fair, HiPPO does offer a genuine distinction in that it operates on local codebook compositions rather than on teacher weights. However, the paper neither clearly articulates nor empirically validates what specific problem this change in the action space solves. Ablations over simple variants, while informative, cannot substitute for head-to-head comparisons against these directly related methods. [1] Ranzinger M, Heinrich G, Kautz J, et al. Am-radio: Agglomerative vision foundation model reduce all domains into one[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 12490-12500. [2] Shang J, Schmeckpeper K, May B B, et al. Theia: Distilling diverse vision foundation models for robot learning[J]. arXiv preprint arXiv:2407.20179, 2024. [3] Sarıyıldız M B, Weinzaepfel P, Lucas T, et al. Dune: Distilling a universal encoder from heterogeneous 2d and 3d teachers[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 30084-30094. [4] Yang C, Yu X, Yang H, et al. Multi-teacher knowledge distillation with reinforcement learning for visual recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(9): 9148-9156.
The attribution of the core mechanism, and of the external-generalization gains in particular, remains ambiguous. The sampled representations are trained with teacher rewards, whereas the deterministic student learns primarily through a separate weighted feature loss; as such, the contribution of discrete representation exploration needs to be disentangled from that of multi-teacher supervision and spatial hard-example weighting. In Table 5, the AUC improvement over deterministic projection is only 0.64 percentage points, and no evidence from repeated training runs is provided to establish whether this difference is statistically meaningful. Furthermore, the external kidney cohort mixes positive public cancer cases with private non-contrast negative controls, which risks confounding disease status with data source or acquisition protocol. Finally, the kidney sensitivity and specificity in Table 8 imply a BAcc of approximately 88.88%, whereas Table 2 reports 90.52%; if different thresholds or sample subsets were used for the two tables, this should be explicitly stated.
Questions for authors
Under matched teachers, students, supervision, and budgets, how does HiPPO compare with AM-RADIO/Theia-style feature distillation and policy-driven teacher weighting? Which results specifically favor representation-composition actions over teacher-weight actions?
What if codebook sampling is removed and difficulty weights are derived directly from deterministic student–teacher distances, with all other terms unchanged? This would test whether sampling adds information beyond direct hard-example weighting.
Please provide repeated-seed results for the full method, deterministic projection, and soft mixture. Also, what happens when the semantic branch’s soft tokens are removed or shuffled?
Could the external evaluation include matched positive and negative cases from the same sources and acquisition protocols? Please also reconcile the kidney metrics in Tables 2 and 8 and specify the threshold-selection partition.
7 comments
Relationship to Visual Multi-Teacher Distillation and the Contribution of HiPPO (2)
(3) Same-Protocol Comparisons with AM-RADIO-Style and Teacher-Weighted Distillation
We compare HiPPO with AM-RADIO-style equal-weight distillation and teacher-weighted distillation under the same experimental protocol. All values are percentages.
| Model | mAUC | Macro-BAcc | Macro-F1 | Macro-Dice |
|---|---|---|---|---|
| AM-RADIO-style, equal weight | 88.62 | 82.01 | 56.41 | 49.92 |
| Teacher-weighted distillation | 89.28 | 82.62 | 60.10 | 51.74 |
| HiPPO (ours) | 90.33 | 83.54 | 61.29 | 52.41 |
HiPPO achieves the highest scores across all four metrics. Compared with AM-RADIO-style distillation, the gains are 1.71, 1.53, 4.88, and 2.49 percentage points in mAUC, Macro-BAcc, Macro-F1, and Macro-Dice, respectively. Against teacher-weighted distillation, the corresponding gains are 1.05, 0.92, 1.19, and 0.67 percentage points. These comparisons support the joint benefit of adaptive spatial supervision and learned prototype-subset selection beyond equal-weight and teacher-weighted distillation.
(4) Additional Experiments
We are currently running Theia and MTKD-RL comparisons that we had hoped to include in this rebuttal. Given the extensive additional experimental workload, we will report their results if they are completed before the rebuttal deadline. In our runs the MTKD-RL agent collapses to a uniform mixture (entropy , share ) after updates. Its released objective over the simplex is minimized uniquely at for any , and the scalar, teacher-agnostic reward cannot break this symmetry. The fused alignment term is also near-uniform here, as the three experts have nearly equal student–teacher alignment (disagreement ). MTKD-RL thus reduces to equal-weight averaging, whereas HiPPO composes the codebook representation scored by the experts and learns non-uniform, per-position compositions.
The Necessity and Effectiveness of Reinforcement Learning
We thank the reviewer for highlighting this question. We use paired DeLong tests for per-organ AUC and paired case-level bootstraps for classification metrics and Dice. Here are our analysis.
(1) A Matched Control Identifies the Value of Sampled Optimization
Soft Mixture retains the codebook and gate parameterizations, teacher-specific projectors, three frozen experts, privileged-information term, and training protocol. It replaces sampled subsets with their expected composition, . Alignment then differentiates through the gates, without rollouts; the exploration entropy term is inactive in this deterministic variant. Against this matched control, HiPPO improves mAUC by percentage points (pp), with a CI of and . Because both variants retain the codebook and input-conditioned gates, this gain supports the sampled, group-relative optimization formulation beyond the shared representation architecture.
(2) Complementary Controls Support Learned, Input-Conditioned Selection
The controls examine different aspects of the formulation. All differences are expressed as HiPPO minus the control.
| Control | Modification | mAUC difference (pp) | MacroF1 difference (pp) |
|---|---|---|---|
| Soft Mixture | Evaluates the expected codebook composition | ; CI ; | ; CI ; |
| Deterministic projection | Replaces the codebook and gates with a continuous projection | ; CI ; | ; CI ; |
| Uniform random rollout | Retains sampling with gate probabilities fixed at | ; | ; |
| Input-independent dropout | Retains sampling without input-conditioned gating | ; | ; |
HiPPO improves both mAUC and MacroF1 over random and input-independent sampling. These comparisons identify a contribution from learning which compositions to select, rather than stochastic feature variation alone. The deterministic-projection comparison provides complementary evidence, with its clearest gain in MacroF1 at the reported operating points.
(3) Benefits Extend to Lesion Segmentation
Relative to removing Codebook-GRPO, HiPPO gains MacroF1 pp () and macro Dice points ( CI , ), alongside a mAUC change (). The liver Dice gain is points ( CI , ). Against Deterministic projection, HiPPO improves macro Dice by points ( CI , ). Together, the matched Soft Mixture comparison and the classification and segmentation results support the practical benefit of the sampled codebook-policy formulation. Its contribution is particularly evident in MacroF1 and voxel-overlap performance.
(4) Composition-Level Credit Explains the Optimization Rationale
At a fixed student state, hold the codebook and expert readouts fixed. Let , , and denote the expert-based compatibility reward. The expected-reward objective has the exact gate derivative:
where .
An atom therefore receives credit according to its marginal reward contribution across the discrete contexts formed by the other selected atoms. Soft Mixture instead evaluates a directional derivative at one mean representation:
Under nonlinear expert scoring, these signals are generally different. Codebook-GRPO constructs group-relative feedback from actual candidate evaluations, enabling selection learning to respond to combination-dependent usefulness.
This explains the rationale for sampled optimization, while the matched empirical comparisons establish its practical benefit.
(5) Multi-Seed Experiments
We report the mean results across training seeds. HiPPO achieves the highest mean on each reported metric. We acknowledge that, across the variants, some metrics were lower in the additional-seed runs than in the original experiments, reflecting variability across training runs.
| Model | mAUC | MacroF1 | Macro-BAcc | Macro-Dice |
|---|---|---|---|---|
| HiPPO | 89.23 | 59.55 | 82.60 | 51.59 |
| Deterministic projection | 88.64 | 57.16 | 81.80 | 50.65 |
| Soft Mixture | 88.55 | 59.21 | 81.74 | 50.62 |
Relationship to Visual Multi-Teacher Distillation and the Contribution of HiPPO (1)
We thank the reviewer for this important pointer. AM-RADIO, Theia, DUNE and MTKD-RL are indeed the directly related work on the teacher side, and we will add them to Related Work in the revision. Our Related Work was organized around the two lenses through which we developed the method, on-policy distillation and privileged information distillation, and both are student-side lenses, so the agglomerative multi-teacher literature was never surfaced as a separate axis. We agree this is a gap in exposition and that expert-side work belongs in the discussion.
(1) Shared Foundations and Methodological Differences
Teacher-specific mappings are established components of multi-teacher distillation. AM-RADIO computes separate teacher-matching losses through teacher-specific heads and aggregates those scalar losses. Theia uses teacher-specific translators, while DUNE uses Transformer projectors to transfer heterogeneous global and patch-level knowledge. MTKD-RL learns sample-wise teacher weights for logit and feature distillation. HiPPO retains teacher-specific alignment but changes the object and spatial granularity of policy control.
The distinction is between controlling the importance of teacher losses and constructing local representation candidates whose compatibility is evaluated across teachers.
(2) What the Local Action Space Addresses
The action-space change addresses adaptive local composition in dense cross-dimensional transfer. Different regions of a medical volume can have different student deficiencies, structural characteristics, and lesion relevance. HiPPO makes both the local representation choice and the allocation of dense supervision explicit.
At position p, the policy constructs a candidate:
f^p(g)=∑k=1Mak,p(g)ek,ap(g)∈{0,1}M.
The same candidate is evaluated in each teacher's native feature space:
qi,p(g)=cos(hi(f^p(g)),ti,p),rp(g)=K1∑i=1Kqi,p(g).
The student-conditioned policy learns local codebook compositions through group-relative comparisons of sampled rewards, rather than assigning teacher weights. Under nonlinear expert scoring, evaluating sampled compositions generally differs from evaluating their mean. The same feedback guides teacher-specific spatial priorities through a position-wise maximum of alignment difficulty, structural salience, and segmentation error, followed by unit-mean normalization that preserves coefficient budgets. HiPPO therefore coordinates what to compose and where to concentrate dense supervision. A shared 3D encoder jointly supports lesion segmentation and multi-label presence prediction, with organ-guided decoding when annotations are available. A separate training-only privileged branch transfers annotation-localized medical-VLM supervision through student soft tokens, independently of the visual rollout reward.