acceptodds
Under review as a conference paper at ICLR 2027

OMT-Distill: Object-Measure Transport for Heterogeneous 3D Detection Distillation

Abstract

Knowledge distillation enables LiDAR-only 3D detectors to learn from multimodal teachers. However, differences in encoders, decoders, and detection heads complicate feature alignment and the association of teacher and student predictions. Predicted 3D boxes and class labels have common geometric and semantic meanings, allowing the two prediction sets to be represented as confidence-weighted object measures. We propose Object-Measure Transport (OMT), which combines teacher-location supervision with soft transport over native student objects. Teacher-location supervision extends quality and ordering guidance to responses omitted by student candidate selection. Unbalanced transport distributes supervision through many-to-many object associations, accommodating differences in prediction coverage and confidence. On the nuScenes validation set, OMT-Distill achieves 61.07 mAP and 67.92 NDS with a CenterPoint-Voxel student. OMT-Distill improves detection accuracy across four heterogeneous teacher–student pairs while retaining each student's original LiDAR-only inference architecture.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D object detection, knowledge distillation, cross-modal distillation, heterogeneous architectures, unbalanced optimal transport

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_nM1ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OMT-Distill, an object-level knowledge distillation framework for heterogeneous 3D object detection. The paper considers the setting where a multimodal teacher transfers knowledge to a LiDAR-only student with a different detector architecture, and argues that decoded 3D object predictions provide a common interface for knowledge transfer. The proposed Object-Measure Transport (OMT) consists of two main components: teacher-location supervision, which supervises student responses at teacher object locations, including locations outside the student’s native top-k candidates, and an unbalanced optimal transport formulation that establishes soft many-to-many correspondences between teacher and student predictions based on their confidence, spatial distance, box geometry, and class compatibility. OMT is further combined with feature, response, and candidate-level distillation to form OMT-Distill. Experiments on nuScenes across several heterogeneous teacher–student pairs show improvements over the considered distillation baselines. In the main BEVDilation-to-CenterPoint-Voxel setting, OMT-Distill achieves 61.07 mAP and 67.92 NDS, improving over the HRCA baseline by 1.01 mAP and 0.39 NDS. The results support the paper’s claim that decoded object-level supervision can provide an effective interface for distillation between heterogeneous 3D detectors.

Critical strengths
  1. Well-motivated object-level distillation formulation. The paper identifies the difficulty of distilling knowledge between heterogeneous 3D detectors with different encoders, decoders, and prediction heads, and uses decoded 3D objects as a shared semantic and geometric interface for knowledge transfer. This provides a clear motivation for the proposed framework.

  2. Teacher-location supervision addresses candidate coverage limitations. A key strength is that the method supervises student responses at teacher object locations, including locations outside the student’s native top-k candidates. This allows teacher knowledge to influence regions that would otherwise be excluded by the student’s own candidate selection.

  3. Flexible matching with encouraging empirical results. The unbalanced optimal transport formulation enables soft many-to-many associations between teacher and student predictions while accommodating differences in prediction coverage and confidence. Experiments across multiple heterogeneous teacher–student pairs show consistent improvements over the considered baselines, with a 1.01 mAP and 0.39 NDS gain over HRCA in the main setting.

Critical weaknesses
  1. The necessity of optimal transport is not sufficiently established. Although the paper attributes the flexibility of object-level matching to unbalanced optimal transport, it does not compare OMT against simpler matching strategies, such as nearest-neighbor, Hungarian, IoU/radius-based matching, or soft distance-based matching. It is therefore unclear whether the performance gains are specifically due to the OT formulation or more generally due to object-level matching and additional supervision.

  2. The contributions of individual OMT components are not fully disentangled. OMT combines teacher-location carrier supervision, ordering supervision, transport-based quality and geometry losses, and exclusion loss. The current ablations do not sufficiently isolate these components, making it difficult to determine which components are essential for the reported improvement. A more comprehensive component-wise or factorial ablation would strengthen the empirical justification of the method.

  3. The role of teacher-score calibration is not sufficiently investigated. The method converts teacher scores into class-relative empirical percentiles and then squares them before using them as quality targets and transport masses. This is a non-trivial design choice that may substantially affect the results. It would be useful to compare this formulation with raw teacher scores and other calibration strategies to determine how much of the improvement comes from score calibration rather than the proposed transport mechanism.

  4. The exclusion loss introduces a potentially important false-negative risk. The proposed transport-based exclusion mechanism suppresses native student responses that receive weak support from the teacher. However, teacher predictions are inevitably incomplete, so valid student detections that are missed by the teacher may also be suppressed. Although the paper acknowledges this issue, the experiments do not quantitatively characterize how often valid predictions are removed or how performance changes when the exclusion loss is removed or weakened.

  5. The statistical robustness of the main results is limited. The formal experiments are based on a single training seed, while the reported bootstrap analysis only measures variation across validation scenes at fixed model weights and does not capture training randomness. Since the main improvement over HRCA is relatively modest (1.01 mAP and 0.39 NDS), experiments with multiple training seeds would provide stronger evidence that the gains are reproducible.

  6. The scope of the generalization claim is not fully supported by the experiments. The paper presents OMT as a framework for heterogeneous 3D detection and emphasizes its architecture flexibility, but the experiments are conducted on nuScenes within a relatively specific LiDAR-based student and multimodal-teacher setting. More diverse datasets, modalities, or substantially different detector families would be needed to substantiate a broader claim of architecture-agnostic or general heterogeneous distillation.

Questions for authors
  1. Could the authors clarify how the teacher-relative quality score is computed in practice when the number of teacher predictions for a given class is small? In particular, how robust is the empirical percentile estimation in such cases?
  2. Could the authors clarify the motivation for detaching the optimal transport plan before computing the student gradients? It would be helpful to explain whether this choice is primarily for optimization stability or whether allowing gradients through the transport plan was empirically found to be less effective.
1 comment
Reviewer_nM1ZOpenReview ↗

Thanks for authors' response

The authors’ rebuttal addressed some of concerns. I maintain my original score.

Reviewer_qMvVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OMT-Distill, a knowledge distillation framework that transfers knowledge from multimodal 3D detectors to LiDAR-only students with different architectures. Instead of directly aligning intermediate features, the authors represent teacher and student predictions as confidence-weighted object measures. The method consists of two main components: teacher-location supervision, which provides quality and ranking guidance at teacher-identified locations, and Object-Measure Transport (OMT), which uses unbalanced optimal transport to establish soft many-to-many associations between teacher and student predictions. The framework keeps the student's original inference architecture unchanged. Experiments on nuScenes show improvements across four teacher–student pairs.

Critical strengths
  • The paper is generally clear and well organized. I find the diagnostic experiments in Section 3 particularly useful, as they provide some motivation for transferring knowledge through object predictions.

  • Using object predictions as a common interface between different teacher and student architectures is a reasonable idea. It avoids the need to directly align their internal representations or prediction heads.

  • The experimental evaluation is fairly detailed. The authors test different teacher–student configurations and provide additional analyses of transport associations, prediction coverage, and object-level changes. I also appreciate that the authors acknowledge the limitations of their uncertainty estimates.

Critical weaknesses
  1. Limited Novelty
  • My main concern is the novelty of the proposed method. Optimal transport has already been widely explored in knowledge distillation and object detection, including unbalanced OT for object assignment. The proposed formulation seems to build mainly on these existing techniques, with the main difference being its application to confidence-weighted 3D object predictions.

  • I also find some overlap between the proposed teacher-location supervision and the HRCA baseline in Section 4.2. HRCA already supervises student responses at teacher centers and includes candidate-ranking guidance. Although the proposed method introduces different quality targets and ordering constraints, I am not fully convinced that these changes constitute a substantial methodological contribution. I would like the authors to explain more clearly what distinguishes OMT from existing transport-based distillation and object-matching methods.

  1. Several design choices are not well justified:
  • Equation (4) combines center distance, confidence differences, geometric discrepancies, and a class-mismatch penalty to construct the transport cost. However, the weights of these terms seem to be manually chosen. How were these values determined, and how sensitive is the method to them?

  • The teacher-location supervision uses percentile-based quality targets and scene-level ordering constraints. While I understand the motivation, it is not clear why these transformations are preferable to directly using calibrated teacher confidence scores.

  1. Additional training cost
  • The authors emphasize that OMT does not introduce additional inference overhead, which is an advantage. However, the method still adds considerable complexity during training. OMT increases training-step time by around 13% over HRCA, while the complete framework increases it by nearly 55% compared with task-only training.

  • Given that OMT improves HRCA by approximately 1 mAP point, I would like to see a clearer discussion of whether the additional training cost is justified. A comparison with simpler object-level distillation methods would also help assess this trade-off.

Questions for authors
  • Could the authors clarify the main novelty of OMT compared with existing transport-based distillation methods? In particular, what is the key difference between the proposed teacher-location supervision and the teacher-center supervision already included in HRCA?

  • Could the authors provide separate ablations for teacher-location supervision and transport-based supervision? I would also like to see comparisons with simpler object-matching strategies to better justify the use of unbalanced OT.

  • How were the transport cost weights, OT regularization parameters, and percentile-based quality targets selected? Could the authors provide more justification or sensitivity analysis for these choices?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.