acceptodds
Withdrawn from ICLR 2027

X-MV3D: Benchmarking Cross-Dataset Generalization in Multi-View 3D Object Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite substantial progress in multi-view 3D object detection, existing detectors are still developed and evaluated largely within individual datasets, leaving their ability to transfer to newly collected datasets unclear. We introduce X-MV3D, a unified benchmark for cross-dataset scene-level 3D detection from multi-view captures. X-MV3D converts ARKitScenes, ScanNet, HyperSim, and 3RScan into a common gravity-aligned coordinate frame, with standardized scene records, category mappings, and 3D box annotations, enabling the four datasets to be trained and evaluated under the same protocol. We define three evaluation settings: single-source transfer, leave-one-dataset-out (LODO) transfer, and target-included mixed training, with LODO serving as the primary target-unseen setting. Across representative detector families, we find that current multi-view 3D detectors are strongly dataset-conditioned. Multi-source training improves over single-source transfer, but LODO remains far below target-included training, showing that multi-source training alone is insufficient for reliable new-dataset transfer. Failure decomposition further shows the transfer gap is dominated by degraded proposal coverage and box localization, suggesting that acquisition shifts may affect the camera-dependent feature lifting and multi-view aggregation. Motivated by this analysis, we propose CamGeoGate, a lightweight detector-agnostic baseline that injects explicit camera geometry into image features and trains detectors with stochastic view gating. CamGeoGate consistently improves LODO performance across three detector families, showing that camera-aware and view-robust training is a promising direction for reusable multi-view 3D detectors.

Primary area:
datasets and benchmarks
Keywords:
Multi-view 3D object detection, Cross-dataset generalization, Benchmarking.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_b1pgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduce X-MV3D trying to benchmark the generalize ability of Multi-View 3D Object Detector across datasets. The paper analyzes the main gap on cross-dataset generalize are camera-dependent feature lifting and multi-view aggregation under LODO and target-included settings. Then the paper proposes CamGeoGate which adds camera-geometry encodings to image features and randomly drops views during training.

Critical strengths
  1. Cross-dataset transfer of multi-view 3D detectors has received little systematic study. The three-level setup (best single-source, LODO, target-included) clearly shows how much source diversity helps, and how much is still gained once target data is included. Using LODO as the main target-unseen setting is a sensible choice. Releasing unified metadata for four datasets also lowers the cost of follow-up work in this direction.
  2. The zero-initialized residual adapter CamGeoGate and training-time view dropout add little overhead and need no target annotations. The method improves [email protected] for most detector and datasets.
Critical weaknesses
  1. The failure analysis attributes the gap to acquisition shift, but the experiments can hardly support this, as there are only two in-domain perturbation tests on ScanNet. CamGeoGate does improve the performance, but it does not show the potential to close the gap, since the improvement is quite small compared with the cross-dataset gap.
  2. The dataset merging is quite simple, which only performs coordinate alignment and category merging. However, the different annotation conventions are also important in the cross-dataset setting. EmbodiedScan[1] analyzes the bbox sizes across different datasets, which shows that datasets differ in how objects are split and which category they are assigned to, but the authors neither analyze this nor apply any handling. Therefore, the Common-9 experiment in Table 4 is also of limited meaning, since unprocessed training data will already cause conflicts for the model, especially when a category needs to be learned in one dataset but is not annotated in another, and this pattern lacks analysis.
  3. All three baselines are RGB-only models, so why is MVSDet not compared? And why is the input restricted to RGB, instead of designing a point cloud input setting, which all the datasets support and is more popular?
  4. Table 3 looks strange in this position. The input settings of the methods in Table 3 all differ from the RGB-only setting of the paper, and it is unclear what this table tries to claim.
Questions for authors
  1. The failure analysis on the camera setting and annotation conventions.
  2. MVSDet and other popular methods comparison would be helpful. As ImVoxelNet[2022] is quite earlier.
  3. Explain why Table 3 is important.
Reviewer_MAorOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a benchmark and a comparison study on cross dataset performance transfer in three different scenarios: dataset-to-dataset transfer, leave-one-dataset-out (LODO), and all datasets mixed together. The authors use four public datasets for the benchmark, that are rebuilt to follow the same dataset structure, have the same coordinate frames, and object classes mapped to the NYU40 vocabulary. Finally, the authors analyze the sources of the performance gap between the LODO and all-mixed-together cases and propose two method-agnostic updates that can help with reducing the gap: injecting the per-feature camera 3D geometry information and doing random drops of images in the sequence. The authors claim a gain in performance metrics as a result of these modifications.

Critical strengths
  1. Unifying the dataset structure is valuable.
  2. Cross-dataset generalization is interesting, and investigations in this can be valuable.
Critical weaknesses
  1. The main observations of the paper, as well as the proposed improvements, are fairly standard. The performance drop in cross-dataset generalization is pretty much expected behavior across most of ML. The proposed improvements are already widely used wherever applicable specifically for 3D geometry-related tasks.
  2. It is also not clear whether the improvement indeed comes from encoding 3D geometry in features or from the drop augs. Given that all three used methods have explicit voxel grids or 3D cost volumes, I would expect them not to struggle with the 3D geometry on a feature level, but it would be beneficial to understand if this is or is not the case.
  3. Error bars are missing to prove the validity of the improvements.
Questions for authors
  1. The ablation study in Section 5.1 reports only SGCDet; did the authors try ablating separate improvements on other methods as well?
  2. Something seems to be off with Hypersim numbers: on SGCDet in-domain is worse than LODO (0.115 in Table 18 vs 0.141 in Table 2) - are these numbers correct?
Reviewer_ku2vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AnyScene is a two-stage, occupancy-centric driving scene generator. A BEV-conditioned latent DiT first generates 12 Hz semantic occupancy from BEV layout sequences. Semantic and coordinate buffers rendered from the occupancy then condition a control branch on a frozen Wan2.1-14B video model, which supports reference-free generation and autoregressive view expansion to arbitrary camera rigs. The authors also build nuCraftv2, a 12 Hz dense occupancy dataset derived from nuScenes. On nuScenes, the video generator reports the best FVD and BEVFormer mIoU/mAP among the compared methods, and the extra generated views densify sparse-view reconstruction.

Critical strengths
  1. Conditioning the video model on geometry rendered from occupancy makes the camera rig something chosen at inference time. One model can generate surround views without a reference frame, expand from 6 to 24 cameras, and re-render a laterally shifted ego path. Fixed-rig generators cannot do this without retraining.
  2. The occupancy generator is well designed, and the ablations support the design. Joint attention over BEV and occupancy tokens clearly beats additive conditioning (19.01 vs. 13.59 mIoU), and teacher forcing is essential for stable autoregressive rollout. The rendered semantic and coordinate buffers also matter a lot for video quality (Table 4: FVD drops from 112.2 to 43.2).
  3. nuCraftv2 (12 Hz dense occupancy with per-instance assets) could be a useful resource for occupancy generation and world modeling, if it is released.
Critical weaknesses
  1. Controllability is not compared with prior work. Table 6 reports only AnyScene, although Appendix D.1 says the metrics follow X-Scene and UniScene. Neither method, nor any other BEV-to-occupancy method, is included in the table. The BEV-vs-layout score of 52.45 mIoU is also hard to interpret without the same score for ground-truth occupancy. The 3D scores are low (19.01 mIoU, 15.58 IoU). The paper explains this by sparse ground truth, but nuCraftv2 is presented as dense, and Fig. 11 cannot separate plausible completion from hallucination. Reporting mIoU only over observed voxels would help.
  2. Zero-shot generalization is claimed but not measured. The abstract and contributions state that experiments demonstrate zero-shot generation on nuPlan, WOMD, OpenStreetMap and user layouts. The evidence is only Figs. 1, 15 and 16. Appendix C.1 does not explain how these sources are mapped to the 15-channel layout, and it never lists the five static classes. For OSM, which has no agents and only partial lane-level geometry, the conversion matters a lot. Layout-consistency scores on each source would make this claim checkable.
  3. The comparison in Table 2 needs clarification. The text says all methods were re-evaluated, but several baseline rows match the values reported in GenieDrive. Two entries differ from that source: the MagicDrive-V2 clip length and the GenieDrive mAP. The methods also differ in clip length, frame rate, resolution, backbone size (1.3B vs. 14B) and conditioning, so the FVD gaps are not directly comparable. In addition, the mIoU/mAP listed in the 49-frame row are identical to the 16-frame results in Table 4.
Questions for authors
  1. Could you add occupancy-generation baselines (e.g., UniScene, X-Scene) and the BEV-vs-layout score of ground-truth occupancy?
  2. How are nuPlan, WOMD and OSM converted to the 15-channel layout, and where do agents come from for OSM? Can you report layout consistency on each source?
  3. What are the inference time and memory costs, especially for 24-view expansion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.