- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether the camera and LiDAR branches of a jointly trained camera–LiDAR perception model learn representations as useful for the task as those of independently trained unimodal models (p. 1). To measure this it introduces a Paired Frozen-Branch Probe: the branch up to the pre-fusion BEV representation is frozen, once from a unimodal model and once from the fusion model, and a fresh task decoder with identical architecture, initialization, data and budget is trained on each; a lower probe score for the fusion branch is defined as unimodal branch degradation (Sec. 3.1, Eqs. 1–3). Across four tasks (3D detection, vectorized mapping, BEV segmentation, occupancy) and ten model–task entries, the camera branch degrades in every case, by up to 26 points, while the LiDAR branch changes little in either direction (Table 1). A hierarchical probe that moves the probing position upstream shows the gap present from the backbone output through the BEV encoder, largest at the pre-fusion BEV for three of the four tasks (Sec. 3.3, Table 2). Finally, distilling a frozen camera-only teacher's pre-fusion BEV into the fusion model's camera branch during joint training improves both the camera-branch probe and the full model on all four tasks (Sec. 3.4, Table 3), with ablations in the appendix (Tables 4–14).
Critical strengths
-
The question is well posed and the measurement is simple and controlled. The paper separates the quality of a branch representation from the quality of the fused output, which output-level metrics cannot do (p. 1), and measures it with a paired probe in which the frozen extractor is the only difference between the two sides: same decoder, initialization, data and training budget (Sec. 3.1, Eqs. 1–3, Alg. A.1). This matters because the central quantity is operationally defined and architecture-agnostic, so the finding can be checked and the probe reused on other fusion models; my remaining concerns are about what differs between the source models and how faithful the probe is in detection, not about the protocol's design.
-
The gap is clearly defined and well characterized. Rather than stopping at failure cases (Fig. 1), the paper turns the concern into a named, measurable quantity, Δ_C, and reports it for every entry next to its reference and end-to-end scores (Table 1). It then characterizes the gap: it persists across backbone families and capacities (Table 4), appears in all six segmentation classes for the three models in Table 7, is traced along the camera pathway from backbone to BEV encoder (Table 2), and is reduced by a targeted intervention (Table 3). This matters because, whatever its cause, the gap as defined is a concrete target that others can report and try to reduce; this is the paper's main contribution and the main consideration in favour of acceptance.
Critical weaknesses
-
The headline claims — camera-branch degradation attributed to joint training, and the camera–LiDAR asymmetry — are not separated from how the source models were trained.
- The probe controls only the decoder stage. The camera-only, LiDAR-only and fusion models are "fully trained" checkpoints whose initialization, schedule and pretraining are given nowhere (Sec. 3.1, p. 3 l. 154–156; Alg. A.1 takes the source checkpoints as inputs).
- Table 1's BEVFusion scores and CMT's Full C and Full F match the public results (e.g., Full F 68.55 vs 68.52 and 70.30 vs 70.3 mAP), so public recipes may have been used. In the public BEVFusion detection recipe the fusion model starts from the LiDAR-only checkpoint and trains 6 epochs versus 20 for the camera-only model; this could explain F2L ≈ L2L (47.69 vs 47.70) and part of Δ_C. In the public BEVFusion segmentation and CMT recipes the camera branch has the same initialization and epochs in both models, yet Δ_C is −5.24 and −25.93.
- The largest camera gaps are in detection, where the probe is least faithful to end-to-end performance, also for unimodal features (CMT F2F 53.24 vs Full F 70.30; BEVFusion L2L 47.69 vs Full L 64.68, Table 1), so part of Δ_C there may be decoder adaptation under a fixed budget rather than representation quality. For CMT, which has no explicit camera BEV, the published masked-modal model reaches 38.3 mAP from camera input alone (Yan et al., 2023, Table 3); if that is the Table 1 model, the probe interface may explain part of its F2C of 10.50.
- Consequence: the asymmetry is clear for the evaluated checkpoints, but the paper's explanation (degradation "after multimodal joint training", abstract) cannot be checked while the source recipes are unreported. One recipe-matched pair, with the recipes reported, would test this and could change my assessment.
-
The intervention is evaluated on one model per task (two for segmentation), without a teacher-free comparison, and its full-model gains are small where the probe gap nearly closes.
- Full-model gains come from a single representative model per task (BEVFusion, MapTR, CONet; Table 3), plus DifFUSER for segmentation without a post-distillation probe (Table 11, p. 17; p. 18 l. 930–932), although a camera-only model is reported for every entry in Table 1 (Full C). Two of the four gains are below one point (+0.98 mAP, +0.62 mIoU).
- The Table 3 baselines differ from Table 1's end-to-end scores without explanation (BEV segmentation 59.83 vs 62.94; the distilled model, 60.45, stays below Table 1's own fusion model), which is a provenance question (Q4), not evidence against the distillation.
- Closing the probe gap carries over only weakly: for detection and segmentation the distilled F2C is within about one point of C2C (C2C from Table 1), yet the full model improves by only 0.98 mAP and 0.62 mIoU (Table 3). This weakens the "intervention-based evidence for the diagnosis" (p. 2 l. 106–107; p. 9 l. 481).
- The method needs a separately trained camera-only teacher (Alg. A.3). A teacher-free alternative — modality dropout / masked-modal training, as in the CMT paper the submission evaluates (Yan et al., 2023) — is not compared. Table 1's CMT Full F (70.30/72.94) equals the published masked-modal CMT result, so that checkpoint may already use it, yet its Δ_C is the largest (−25.93); the submission does not say.
- Consequence: the practical contribution claimed for the distillation (Sec. 3.4, Conclusion) rests on one model per task (plus DifFUSER, full model only) and no teacher-free comparison, so I cannot tell whether the gains generalize or whether a simpler training change would give them.
Questions for authors
Questions whose answers could change my assessment:
-
Source-model recipes (W1). Because the probe controls only the decoder and the source models' training is not described, I cannot separate joint-training effects from recipe effects. Please give, for each Full C / Full L / Full F model in Table 1, the initialization (in particular whether the fusion model starts from the LiDAR-only or camera-only checkpoint), the schedule, and any pretraining, and state which are public checkpoints. Appendix B.1 trains both source models per backbone (p. 13); for one such pair, please report Δ_C and Δ_L when the fusion model uses the same initialization and the same number of camera-training steps as the camera-only model.
-
A teacher-free baseline for the intervention (W2). The distillation needs a separately trained camera-only teacher, while modality dropout (masked-modal training, as in the CMT paper you evaluate) needs none. Did the CMT checkpoint in Table 1 use masked-modal training, and if so, why is its Δ_C still the largest? For one task — the BEVFusion segmentation setting of Table 10 — please add a row trained with modality dropout and report its Δ_C and full-model mIoU next to the distilled row.
-
Generality of the intervention (W2). Table 3 reports one model per task, and the segmentation result on DifFUSER (Table 11) has no post-distillation probe. Since a camera-only model is reported for every entry in Table 1, please add the distillation result — full-model score and F2C — for at least one further model in detection (CMT) and in mapping (HIMap, MapTRv2 or MapQR), and the post-distillation F2C for DifFUSER. Consistent gains across models would carry more weight with me than the current one model per task.
Clarification requests needed to verify the reported numbers:
-
Table 3 versus Table 1. The Table 3 baselines differ from the Table 1 end-to-end scores (segmentation 59.83 vs 62.94; MapTR 61.97 vs 63.14; CONet IoU 25.30 vs 26.50), BEVFusion's F2C values also differ (12.06 vs 11.58; 49.30 vs 51.20), yet MapTR's and CONet's F2C values are identical across the two tables. Which checkpoints produce each table, and why do they differ?
-
Models in Figure 4 and probing positions. Figure 4 includes UVTR, which appears in no table, in the text, or in the references; its LiDAR drop (−3.48) would be the largest aggregate LiDAR drop in the study. Table 1 lists nine distinct methods: is UVTR the tenth of the "10 models", and where do its numbers come from? Also, CMT has no explicit camera BEV (Yan et al., 2023): what is its "pre-fusion Camera BEV" probing position? And since MapPrior's probe numbers nearly equal BEVFusion's (C2C 56.56 vs 56.44; L2L 47.69 vs 47.45, Table 1), do the two share the same branches?
-
Probe and distillation settings. Please state the decoder training budget per task, the values of λ_t, the data split used as the "fixed validation set" (p. 4), and whether the probe configurations and checkpoints will be released.