acceptodds
Under review as a conference paper at ICLR 2027

UBD-Bench: Benchmarking and Analyzing Unimodal Branch Degradation in Multimodal Perception for Autonomous Driving

Abstract

Multimodal perception improves autonomous-driving performance by combining complementary sensor information. However, stronger multimodal predictions do not necessarily imply stronger representations in each modality-specific branch. Such branch-level weaknesses can remain hidden in conventional output-level evaluation. We introduce UBD-Bench, a systematic benchmark for studying unimodal branch degradation (UBD) in multimodal perception. UBD-Bench uses a Paired Frozen-Branch Probe to compare the task-relevant representation quality of the same modality after independent unimodal training and multimodal joint training under matched decoding conditions. Across four autonomous-driving perception tasks and 10 models, we find that Camera branches consistently degrade after multimodal joint training, whereas LiDAR branches exhibit weaker and more model-dependent changes, revealing a pronounced modality asymmetry. Hierarchical probing further shows that Camera-branch degradation extends from the visual backbone through the Camera-to-BEV pathway and is most severe at the pre-fusion Camera BEV representation. Guided by this diagnosis, we introduce unimodal-to-multimodal distillation at the degraded representation. This intervention consistently improves both Camera-branch Probe performance and full-model performance across all four tasks. These results show that multimodal gains can coexist with degraded unimodal representations, highlighting branch-level representation quality as an important dimension for evaluating and improving multimodal perception.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Perception, Autonomous Driving, Representation Learning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DXG4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the camera and LiDAR branches of a jointly trained camera–LiDAR perception model learn representations as useful for the task as those of independently trained unimodal models (p. 1). To measure this it introduces a Paired Frozen-Branch Probe: the branch up to the pre-fusion BEV representation is frozen, once from a unimodal model and once from the fusion model, and a fresh task decoder with identical architecture, initialization, data and budget is trained on each; a lower probe score for the fusion branch is defined as unimodal branch degradation (Sec. 3.1, Eqs. 1–3). Across four tasks (3D detection, vectorized mapping, BEV segmentation, occupancy) and ten model–task entries, the camera branch degrades in every case, by up to 26 points, while the LiDAR branch changes little in either direction (Table 1). A hierarchical probe that moves the probing position upstream shows the gap present from the backbone output through the BEV encoder, largest at the pre-fusion BEV for three of the four tasks (Sec. 3.3, Table 2). Finally, distilling a frozen camera-only teacher's pre-fusion BEV into the fusion model's camera branch during joint training improves both the camera-branch probe and the full model on all four tasks (Sec. 3.4, Table 3), with ablations in the appendix (Tables 4–14).

Critical strengths
  1. The question is well posed and the measurement is simple and controlled. The paper separates the quality of a branch representation from the quality of the fused output, which output-level metrics cannot do (p. 1), and measures it with a paired probe in which the frozen extractor is the only difference between the two sides: same decoder, initialization, data and training budget (Sec. 3.1, Eqs. 1–3, Alg. A.1). This matters because the central quantity is operationally defined and architecture-agnostic, so the finding can be checked and the probe reused on other fusion models; my remaining concerns are about what differs between the source models and how faithful the probe is in detection, not about the protocol's design.

  2. The gap is clearly defined and well characterized. Rather than stopping at failure cases (Fig. 1), the paper turns the concern into a named, measurable quantity, Δ_C, and reports it for every entry next to its reference and end-to-end scores (Table 1). It then characterizes the gap: it persists across backbone families and capacities (Table 4), appears in all six segmentation classes for the three models in Table 7, is traced along the camera pathway from backbone to BEV encoder (Table 2), and is reduced by a targeted intervention (Table 3). This matters because, whatever its cause, the gap as defined is a concrete target that others can report and try to reduce; this is the paper's main contribution and the main consideration in favour of acceptance.

Critical weaknesses
  1. The headline claims — camera-branch degradation attributed to joint training, and the camera–LiDAR asymmetry — are not separated from how the source models were trained.

    • The probe controls only the decoder stage. The camera-only, LiDAR-only and fusion models are "fully trained" checkpoints whose initialization, schedule and pretraining are given nowhere (Sec. 3.1, p. 3 l. 154–156; Alg. A.1 takes the source checkpoints as inputs).
    • Table 1's BEVFusion scores and CMT's Full C and Full F match the public results (e.g., Full F 68.55 vs 68.52 and 70.30 vs 70.3 mAP), so public recipes may have been used. In the public BEVFusion detection recipe the fusion model starts from the LiDAR-only checkpoint and trains 6 epochs versus 20 for the camera-only model; this could explain F2L ≈ L2L (47.69 vs 47.70) and part of Δ_C. In the public BEVFusion segmentation and CMT recipes the camera branch has the same initialization and epochs in both models, yet Δ_C is −5.24 and −25.93.
    • The largest camera gaps are in detection, where the probe is least faithful to end-to-end performance, also for unimodal features (CMT F2F 53.24 vs Full F 70.30; BEVFusion L2L 47.69 vs Full L 64.68, Table 1), so part of Δ_C there may be decoder adaptation under a fixed budget rather than representation quality. For CMT, which has no explicit camera BEV, the published masked-modal model reaches 38.3 mAP from camera input alone (Yan et al., 2023, Table 3); if that is the Table 1 model, the probe interface may explain part of its F2C of 10.50.
    • Consequence: the asymmetry is clear for the evaluated checkpoints, but the paper's explanation (degradation "after multimodal joint training", abstract) cannot be checked while the source recipes are unreported. One recipe-matched pair, with the recipes reported, would test this and could change my assessment.
  2. The intervention is evaluated on one model per task (two for segmentation), without a teacher-free comparison, and its full-model gains are small where the probe gap nearly closes.

    • Full-model gains come from a single representative model per task (BEVFusion, MapTR, CONet; Table 3), plus DifFUSER for segmentation without a post-distillation probe (Table 11, p. 17; p. 18 l. 930–932), although a camera-only model is reported for every entry in Table 1 (Full C). Two of the four gains are below one point (+0.98 mAP, +0.62 mIoU).
    • The Table 3 baselines differ from Table 1's end-to-end scores without explanation (BEV segmentation 59.83 vs 62.94; the distilled model, 60.45, stays below Table 1's own fusion model), which is a provenance question (Q4), not evidence against the distillation.
    • Closing the probe gap carries over only weakly: for detection and segmentation the distilled F2C is within about one point of C2C (C2C from Table 1), yet the full model improves by only 0.98 mAP and 0.62 mIoU (Table 3). This weakens the "intervention-based evidence for the diagnosis" (p. 2 l. 106–107; p. 9 l. 481).
    • The method needs a separately trained camera-only teacher (Alg. A.3). A teacher-free alternative — modality dropout / masked-modal training, as in the CMT paper the submission evaluates (Yan et al., 2023) — is not compared. Table 1's CMT Full F (70.30/72.94) equals the published masked-modal CMT result, so that checkpoint may already use it, yet its Δ_C is the largest (−25.93); the submission does not say.
    • Consequence: the practical contribution claimed for the distillation (Sec. 3.4, Conclusion) rests on one model per task (plus DifFUSER, full model only) and no teacher-free comparison, so I cannot tell whether the gains generalize or whether a simpler training change would give them.
Questions for authors

Questions whose answers could change my assessment:

  1. Source-model recipes (W1). Because the probe controls only the decoder and the source models' training is not described, I cannot separate joint-training effects from recipe effects. Please give, for each Full C / Full L / Full F model in Table 1, the initialization (in particular whether the fusion model starts from the LiDAR-only or camera-only checkpoint), the schedule, and any pretraining, and state which are public checkpoints. Appendix B.1 trains both source models per backbone (p. 13); for one such pair, please report Δ_C and Δ_L when the fusion model uses the same initialization and the same number of camera-training steps as the camera-only model.

  2. A teacher-free baseline for the intervention (W2). The distillation needs a separately trained camera-only teacher, while modality dropout (masked-modal training, as in the CMT paper you evaluate) needs none. Did the CMT checkpoint in Table 1 use masked-modal training, and if so, why is its Δ_C still the largest? For one task — the BEVFusion segmentation setting of Table 10 — please add a row trained with modality dropout and report its Δ_C and full-model mIoU next to the distilled row.

  3. Generality of the intervention (W2). Table 3 reports one model per task, and the segmentation result on DifFUSER (Table 11) has no post-distillation probe. Since a camera-only model is reported for every entry in Table 1, please add the distillation result — full-model score and F2C — for at least one further model in detection (CMT) and in mapping (HIMap, MapTRv2 or MapQR), and the post-distillation F2C for DifFUSER. Consistent gains across models would carry more weight with me than the current one model per task.

Clarification requests needed to verify the reported numbers:

  1. Table 3 versus Table 1. The Table 3 baselines differ from the Table 1 end-to-end scores (segmentation 59.83 vs 62.94; MapTR 61.97 vs 63.14; CONet IoU 25.30 vs 26.50), BEVFusion's F2C values also differ (12.06 vs 11.58; 49.30 vs 51.20), yet MapTR's and CONet's F2C values are identical across the two tables. Which checkpoints produce each table, and why do they differ?

  2. Models in Figure 4 and probing positions. Figure 4 includes UVTR, which appears in no table, in the text, or in the references; its LiDAR drop (−3.48) would be the largest aggregate LiDAR drop in the study. Table 1 lists nine distinct methods: is UVTR the tenth of the "10 models", and where do its numbers come from? Also, CMT has no explicit camera BEV (Yan et al., 2023): what is its "pre-fusion Camera BEV" probing position? And since MapPrior's probe numbers nearly equal BEVFusion's (C2C 56.56 vs 56.44; L2L 47.69 vs 47.45, Table 1), do the two share the same branches?

  3. Probe and distillation settings. Please state the decoder training budget per task, the values of λ_t, the data split used as the "fixed validation set" (p. 4), and whether the probe configurations and checkpoints will be released.

Reviewer_vb8MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • This paper introduces UBD-Bench, a benchmark designed to investigate unimodal branch degradation in camera–LiDAR multimodal perception systems for autonomous driving.

  • The authors argue that while multimodal fusion generally improves overall perception performance, joint training may degrade the representation quality of individual modality-specific branches.

  • To investigate this phenomenon, the authors propose a Paired Frozen-Branch Probe that compares representations learned through unimodal training and multimodal joint training under matched decoding conditions.

  • Experiments across four perception tasks and ten models demonstrate consistent degradation in camera representations, while LiDAR representations exhibit relatively minor and model-dependent changes.

  • Based on these findings, the authors introduce a unimodal-to-multimodal knowledge distillation strategy, transferring knowledge from an independently trained camera model to the camera branch of a multimodal model.

  • Experimental results show improvements in both camera-branch representation quality and overall multimodal perception performance across all four tasks.

Critical strengths
  • The paper is clearly written.

-- The systematic approach to analyzing the imbalanced reliance on different modalities in multimodal perception is interesting.

  • The findings may encourage further research into the asymmetric behavior of multimodal fusion systems.

  • This work can also inspire the development of more robust multimodal fusion strategies.

Critical weaknesses
  • The main finding of this paper is not particularly surprising. The observed degradation of the camera branch can be reasonably expected, given that different modalities provide complementary information and may contribute asymmetrically to multimodal perception. In particular, since LiDAR sensors provide accurate 3D geometric measurements, multimodal fusion models may naturally rely more heavily on LiDAR features for 3D perception tasks. Consequently, the gradients associated with optimizing the overall perception performance may predominantly favor the LiDAR branch, potentially resulting in weaker learning signals for the camera branch.

  • Several strategies have already been explored to mitigate excessive reliance on a single modality, including modality masking, dropout, and input perturbation. The proposed knowledge distillation approach is conceptually aligned with these existing strategies.

  • While the paper provides a systematic empirical analysis of this phenomenon, the underlying problem of modality imbalance is already well recognized in the community. In my view, the experimental findings do not offer sufficiently novel or compelling and the proposed distillation strategy is relatively straightforward and offers limited methodological novelty.

  • Overall, although the paper is clearly written and presents a systematic evaluation, I find its conceptual and methodological contributions insufficiently significant for acceptance at ICLR.

Questions for authors

Please address the comments I provided in "Critical Weaknesses" Section.

Reviewer_5fi7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UBD-Bench to evaluate unimodal branch degradation in camera–LiDAR perception. Across four tasks and ten models, camera branches consistently degrade after multimodal training, while LiDAR changes vary by model. Hierarchical probing localizes the degradation, and distillation from a camera-only teacher improves both branch representations and overall performance.

Critical strengths
  1. The paired frozen-branch probing protocol provides a clear way to evaluate modality-specific representations, revealing weaknesses hidden by overall fusion performance.

  2. Experiments across four tasks and ten models provide broad empirical evidence, while hierarchical probing and distillation connect the diagnosis to practical improvements.

Critical weaknesses
  1. Lower standalone probe scores may reflect complementary specialization rather than harmful degradation. The paper does not sufficiently distinguish these interpretations.

  2. Training details are underspecified, and no multi-seed results or uncertainty estimates are reported, making reproducibility and the significance of smaller improvements difficult to assess.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.