acceptodds
Under review as a conference paper at ICLR 2027

FreqLift: BEV Fusion and Cross-Frequency State Space Modeling for LiDAR-Camera 3D Object Detection

Abstract

In recent years, significant progress has been made in BEV-based 3D object detection. However, processes such as voxelization, viewpoint transformation, and downsampling can still compromise local structural details. Camera features, when projected into the BEV space, are affected by depth estimation and projection errors, and direct fusion may introduce these errors into the LiDAR-camera BEV representation. To address these issues, this paper proposes FreqLift, a LiDAR-camera 3D detection framework that combines BEV fusion with cross-frequency state-space modeling. FreqLift first refines the camera BEV using CameraBEVRefiner and fuses it with the LiDAR BEV. Subsequently, WaveLift uses multi-level wavelet decomposition and Wavelet Cross-Gate to model high- and low-frequency interactions; Freq-Mamba then uses low-frequency structural differences and BEV position coordinates to modulate the state-space model, thereby enhancing the structural modeling capabilities of the fused BEV. On the nuScenes validation set, FreqLift achieves 71.2% mAP and 73.1% NDS, whereas its LiDAR-only variant, FreqLift-L, achieves 67.1% mAP and 71.0% NDS. An ablation shows that this +4.1 mAP gain comes from direct camera fusion (+1.4 mAP) and CameraBEVRefiner (+2.7 mAP).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Computer Vision, Object Detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RddKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FreqLift, a BEV-based 3D object detection method for LiDAR--camera fusion. CameraBEVRefiner first refines camera BEV features generated by LSS using smooth and detail residual branches with a gating mechanism, before fusing them with LiDAR BEV features. WaveLift then performs cross-frequency interactions through a three-level Haar wavelet decomposition, while Freq-Mamba models spatial features using four-directional scanning and state-parameter modulation. The detection head follows a TransFusion-style design and incorporates high-frequency responses as a bias in the candidate heatmap.

Critical strengths

S1. The Refiner shows clear gains in the internal comparison. In Table 4, adding the Refiner increases mAP from 68.5 to 71.2 and NDS from 71.7 to 73.1 while sharing the LiDAR checkpoint, WaveLift, detection head, and training settings. This supports the value of refining camera BEV features before fusion.

S2. Effective use of complementary information from the camera modality. The method addresses feature uncertainty after camera view transformation and the multiscale structure of BEV features without relying on an external camera-only 3D detector. The camera branch's ResNet-50 still uses ImageNet pretraining.

Critical weaknesses

W1. The WaveLift/Freq-Mamba ablations are not sufficiently convincing. All ablations in Table 3 retain DWT, WCG, and Freq-Mamba, varying only the number of levels, heatmap bias, or Δ\Delta modulation. They are also conducted only in the LiDAR-only configuration. Without comparisons against no WaveLift, a capacity-matched CNN, and a standard Mamba without wavelets under the full fusion setting, the gains from cross-frequency modeling cannot be separated from those due to additional capacity or receptive field. Given precedents such as WaveMamba and VoxelMamba, this directly affects the assessment of the added contribution.

W2. The scope of applicability and cost-benefit trade-off remain unclear. The method is evaluated only on nuScenes validation and the current architecture, while the high-frequency heatmap bias may depend on a specific detection head. Could the authors test on other benchmarks and verify whether the modules can be integrated into another BEV detector? Latency/FPS, memory usage, and the overhead of the main modules should also be reported under the same hardware, batch size, and numerical precision. In Table 1, IS-Fusion (CVPR 2024) achieves 72.8 mAP / 74.0 NDS, compared with 71.2 / 73.1 for this method.

Questions for authors
  • What advantage does the Refiner offer over a residual convolutional module of equal capacity? Can results under controlled depth errors or extrinsic calibration perturbations support its intended noise-suppression role?
  • The paper does not report inference speed. Could the authors provide end-to-end latency or FPS under the same hardware, batch size, and inference precision, and compare against the baseline to quantify the speed overhead of the added modules?
  • Figure clarity and typesetting: The module hierarchy and connections in Figure 2 should be clearer. In Sec. 3.1.1 on page 3, MambaGu & Dao (2023) lacks a separator between the model name and the citation.
3 comments

As the discussion phase is drawing to a close, I would be very glad to discuss with the authors if they are willing to respond. If the authors do not reply, I will keep my recommendation score unchanged.

Thank you very much for your interest in our work; we are truly delighted by it.

Regarding the contribution of "bevrefiner," we have already provided a response within our overall official comments. Subsequent validation experiments indicate that the contribution of the "refiner" module could not be substantiated, which we find quite regrettable.

Although we initially considered not responding further, we decided to provide a reply out of respect.

Your other suggestions have been invaluable in enhancing the completeness of our work, and we have learned a great deal from them. We will address these issues in future versions.

Thank you once again.

Thanks to the authors for their candid response. Wish you all the best in your future research.

Reviewer_isHmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents FreqLift for LiDAR-camera 3D object detection. The main idea is to refine the projected camera BEV before fusion and then model the fused BEV at multiple frequency scales. CameraBEVRefiner separates smooth and detail features and predicts a gated residual correction. WaveLift performs three-level wavelet decomposition, uses Wavelet Cross-Gates to exchange low- and high-frequency information, and applies Freq-Mamba with cross-scale and positional modulation. A high-frequency response is also added to the detection heatmap. On the nuScenes validation set, FreqLift achieves 71.2% mAP and 73.1% NDS, compared with 67.1% and 71.0% for its LiDAR-only model.

Critical strengths

The overall design is intuitive and lightweight. Refining the camera BEV before fusion and modeling the fused BEV at separate frequency scales are reasonable responses to projection noise and local detail loss. The camera branch starts from an ImageNet-pretrained ResNet-50 without an external camera-only detector, CBGS, or test-time augmentation.

The matched fusion improvement is nontrivial. Under the matched setting in Table 4, direct camera fusion improves the LiDAR-only model by 1.4 mAP and CameraBEVRefiner adds a further 2.7 mAP, reaching 71.2% mAP and 73.1% NDS on nuScenes validation. The two proposed modules add 1.86M parameters, giving the method a relatively small parameter overhead.

Critical weaknesses

WaveLift, the main technical contribution, is not isolated. Every row in Table 3 retains DWT, WCG, and Freq-Mamba, only the number of levels, high-frequency bias, and Δ\Delta modulation change, producing 0.4–0.7 mAP differences from single runs. There is no matched model without WaveLift, standard-Mamba baseline, DWT-only or WCG-only variant, parameter-matched convolutional block, or ablation of the positional and directional mechanisms. The ablations are also LiDAR-only, although WaveLift is motivated for the fused BEV. The comparison with TransFusion-L is confounded by the BEVFix-style training recipe. Moreover, the cited LION-Mamba, VoxelMamba, and UniMamba report stronger nuScenes validation results, but are omitted from Table 1. While their protocols differ, they are necessary context for a Mamba-based detector. The experiments therefore do not show that cross-frequency state-space modeling contributes to either FreqLift-L or the full model. Resolving this would require a new controlled evaluation of both models rather than a clarification in the rebuttal.

The CameraBEVRefiner gain does not validate the claimed correction of projection errors. The refiner predicts its residual and gate only from the projected camera BEV, without a depth-error signal or LiDAR guidance. Table 4 shows that additional camera-BEV processing helps, but does not distinguish gated residual correction from simply adding an encoder: the paper does not clarify what processing follows LSS in the “w/o Refiner” baseline, and the same module raises camera-only mAP from 18.8 to 26.0. No parameter-matched residual CNN, gate/error visualization, alignment metric, or controlled depth and calibration perturbation is provided. The evaluation is further limited to one dataset and single runs, without per-class, distance-based, or qualitative results. The gain is credible, but its claimed mechanism and generality are not established.

Questions for authors
  1. Could you report both the LiDAR-only and full model without WaveLift under the same training recipe, and compare with standard Mamba, DWT only, WCG only, Freq-Mamba without positional and directional modulation, and a parameter-matched convolutional block? Please also include the SSM detectors cited in Section 2.3 and clarify protocol differences.
  2. Can the authors directly validate whether CameraBEVRefiner corrects projection errors? Please clarify what follows LSS in the “w/o Refiner” baseline, compare with a parameter-matched residual CNN or standard camera-BEV encoder, and visualize McM_c against projection-error regions. Could you also test controlled depth and calibration perturbations and a publicly pretrained camera branch?
Reviewer_5dQtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FreqLift, a LiDAR-camera 3D object detection framework with two main components. CameraBEVRefiner processes camera BEV features before fusion using smooth/detail residual blocks and a spatially gated residual correction. After concatenating the refined camera BEV with the LiDAR BEV, WaveLift recursively applies three levels of Haar wavelet decomposition and cross-frequency gating. Its Freq-Mamba blocks perform four-directional scans, use cross-scale low-frequency differences to modulate the Mamba step size, and add BEV-coordinate-dependent biases to the B and C parameters. Fine-scale high-frequency responses also bias category-heatmap-based query initialization.

On the nuScenes validation set, without CBGS or test-time augmentation, the complete model reports 71.2 mAP / 73.1 NDS, while its LiDAR-only variant reports 67.1 / 71.0. In the most informative internal comparison, direct camera fusion reaches 68.5 / 71.7, and CameraBEVRefiner further improves it to 71.2 / 73.1. Additional LiDAR-only ablations examine WaveLift depth, high-frequency heatmap bias, and cross-scale step-size modulation.

Critical strengths

1. CameraBEVRefiner has a substantial gain under a relatively well-controlled internal comparison.

Table 4 states that the fusion variants share the LiDAR checkpoint, WaveLift configuration, detection head, training schedule, and augmentation. Direct fusion improves mAP/NDS from 67.1/71.0 to 68.5/71.7, while refined fusion reaches 71.2/73.1. Refinement therefore adds 2.7 mAP and 1.4 NDS beyond simply introducing camera features. Camera-only performance also improves from 18.8/24.6 to 26.0/31.5. This is the paper’s strongest evidence because it demonstrates a substantial internal gain rather than relying only on heterogeneous cross-paper comparisons.

2. The framework provides a coherent way to route structural and detail information through the detector.

The architecture consistently uses low-frequency representations for multiscale structural modeling and high-frequency responses for cross-frequency interaction and candidate initialization. Table 3 provides some supporting evidence: removing the high-frequency bias reduces mAP from 67.1 to 66.5, removing step-size modulation reduces it to 66.7, and reducing the hierarchy from three to two levels reduces it to 66.4. The added parameter count is also modest, although parameter count alone does not establish runtime efficiency.

Critical weaknesses

1. The experiments do not establish the independent value of WaveLift’s central wavelet/state-space design.

All Table 3 variants retain wavelet decomposition, WCG, and Freq-Mamba. The experiments only vary the number of levels, high-frequency heatmap bias, and cross-scale step-size modulation. Consequently, they do not show the gain of WaveLift over an otherwise identical model without it, or whether parameter-matched multiscale convolution or standard spatial Mamba would provide a similar benefit.

WCG and the positional B/C biases are not independently ablated, and all WaveLift ablations are conducted in the LiDAR-only setting even though the paper emphasizes post-fusion cross-frequency modeling. The reported component effects are also relatively small—0.4 to 0.7 mAP—and lack variability across runs. This leaves the contribution emphasized in the title substantially under-identified.

2. CameraBEVRefiner’s utility is demonstrated, but its specific architecture and claimed error-correction mechanism are not isolated.

Table 4 compares the full refiner with no refiner, without a parameter-matched residual-convolution alternative or ablations of the smooth/detail decomposition and spatial gate. It therefore establishes that learned camera-BEV refinement is useful, but does not isolate the added value of the proposed SDR design over a comparably sized conventional refiner.

Moreover, the paper motivates the module as mitigating depth-estimation and projection errors, yet provides no alignment diagnostic, controlled depth/calibration perturbation, or gate/residual visualization connecting the gain to those errors. The module may instead function as a general feature-enhancement block. Finally, the heterogeneous comparisons in Table 2 cannot establish the broader claim that camera-BEV reliability is more important than camera-detector strength.

Questions for authors

Assessment-changing experimental evidence

  1. WaveLift attribution: Under exactly the same training protocol, what are the results after removing WaveLift and after replacing it with parameter-matched multiscale convolution or standard four-directional Mamba? Can WCG and the positional B/C biases be isolated, and can variability across runs be reported for the 0.4–0.7 mAP differences in Table 3? Does WaveLift remain beneficial in the complete LiDAR-camera model?

  2. CameraBEVRefiner attribution: How does the full refiner compare with a parameter-matched ordinary residual-convolution refiner, and what are the separate contributions of the smooth/detail split and spatial gate? Is there direct evidence that its gain specifically comes from mitigating projection/depth errors? Does the gain persist with a stronger camera branch under an otherwise matched setup?

Clarification of the current results and implementation

  1. Training controls: Please specify the epoch counts and frozen/trainable modules in each fusion stage, as well as the initialization of the camera and LiDAR branches. Does the LiDAR-only row in Table 4 receive an optimization budget comparable to the complete fusion model, including any additional fine-tuning? This affects whether the full 4.1 mAP difference can be attributed solely to the addition and refinement of camera information.

  2. Module definitions and gradient path: Please specify the SDR and WCG computations, directional aggregation, and coarse-to-fine routing. Are adjacent-scale low-frequency coefficients normalized before the difference in Eq. (8)? With a standard orthonormal 2D Haar transform, coefficient magnitudes vary across levels, so an unnormalized difference may encode scale as well as structural change. For the high-frequency heatmap bias, is the modified heatmap included in the existing heatmap loss, or used only for discrete Top-K candidate selection? Please clarify how the high-frequency projection branch receives training gradients, and also clarify the DAL-Base NDS value in Table 1.

These questions concern interpretation and reproducibility of the current contribution rather than suggestions for unrelated future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reply to the question about the Bevrefiner module

First, we would like to thank the three reviewers for their careful and constructive feedback. Before the discussion period concludes, we wish to clarify a point raised by all three reviewers regarding the contribution of the "BEVRefiner" module. Upon receiving the reviewers' comments, we immediately conducted experiments to verify the effectiveness of this module. Due to limited computational resources, these experiments took several days to complete, and the results did not align with our initial expectations.

Upon further investigation, we discovered that the introduction of the "refiner" was accompanied by modifications to other components of the camera branch. This led us to mistakenly attribute the performance gains solely to the "refiner"—an error in our work that we must acknowledge. This finding is a significant setback for us, as it effectively invalidates about half of the contributions we had claimed.

Although we hesitated regarding how to proceed, out of respect for the reviewers and the Area Chair, we have decided to report these findings transparently. Furthermore, we will correct this issue in a subsequent version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.