acceptodds
Under review as a conference paper at ICLR 2027

LMF-3DDet: Efficient LiDAR–Camera Detection via Response-Guided Sampling and Linear Fusion

Abstract

LiDAR–camera fusion enhances 3D object detection by combining geometric and semantic information, but redundant inputs and costly cross-modal interactions hinder real-time deployment. Aggressive input reduction can also discard informative observations, making it difficult to improve efficiency while maintaining detection accuracy. To address this challenge, we propose LMF-3DDet, a lightweight framework that jointly addresses input selection, feature fusion, and knowledge transfer. First, response-guided sampling uses image detection heatmaps and a geometric dilation margin to select informative LiDAR observations while retaining nearby geometric context. Second, kernelized linear cross-attention fuses the selected point tokens with image features without constructing a dense pairwise attention matrix, reducing the cost of cross-modal interaction. Third, uncertainty-aware distillation guides the lightweight student using a full-input teacher, emphasizing difficult examples without adding teacher computation at inference. Experiments on nuScenes and Waymo demonstrate a favorable accuracy–efficiency trade-off with real-time throughput, while ablation studies validate the benefits of the sampling, fusion, and distillation components.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D object detection, point cloud, image-point fusion, light-weight design

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U9k9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The core idea of this paper is to improve the efficiency of LiDAR–camera 3D object detection by concentrating computation on informative LiDAR observations and reducing the cost of cross-modal interaction, rather than proposing a fundamentally new detection paradigm. Specifically, it introduces response-guided sampling, which uses image detection heatmaps and a 3D geometric dilation margin to select a small but informative subset of LiDAR points while preserving nearby geometric context; then it applies kernelized linear cross-attention with an ELU+1 kernel, where image features are first summarized into a compact d×dd \times d global representation so that LiDAR queries can retrieve image information without constructing a dense pairwise attention matrix, reducing the interaction cost from (O(hMLd)) to O(h(M+L)d2)O(h(M+L)d^2). Finally, during training, a frozen full-input TransFusion teacher with an uncertainty head supervises a lightweight student through uncertainty-aware distillation, giving more weight to difficult examples while removing the teacher at inference. The overall idea is therefore a coordinated system design that combines image-guided input reduction, efficient linear fusion, and training-only knowledge transfer to achieve a favorable accuracy–efficiency trade-off for real-time LiDAR–camera 3D detection.

Critical strengths

This paper presents LMF-3DDet, a lightweight LiDAR–camera 3D detection framework that jointly addresses input selection, cross-modal fusion, and knowledge transfer. Its response-guided sampling effectively reduces the LiDAR token budget while preserving detection accuracy, and the kernelized linear cross-attention lowers the cost of image-to-LiDAR interaction from quadratic to linear scaling in token count, yielding a favorable accuracy–efficiency trade-off. The uncertainty-aware distillation further improves the lightweight student without adding inference cost, and experiments on nuScenes and Waymo demonstrate real-time throughput of around 45 FPS with ablations validating the sampling, fusion, and distillation components. The system design is practically relevant for efficient autonomous-driving perception.

Critical weaknesses

The work is largely incremental, as it combines existing ideas—image-guided sampling, linear attention with an ELU+1 kernel, and uncertainty-aware distillation—without introducing a fundamentally new viewpoint or technical breakthrough. The reported accuracy remains below state-of-the-art multi-camera fusion methods, and the evaluation is limited to a single-camera setting. RGS depends on image responses and calibration, so degraded images or projection errors may discard useful LiDAR observations, and the uncertainty weighting may emphasize unreliable teacher targets. The comparisons with random sampling change both the selection policy and token budget, making it difficult to isolate the effect of the sampling strategy. Speed comparisons are not based on a uniform benchmark, and the paper lacks repeated-seed evaluations, embedded-device latency, energy measurements, and class-specific robustness analysis. Thus, while the engineering integration is useful, the novelty and empirical rigor are limited.

Questions for authors

See Weakness.

3 comments
AuthorsOpenReview ↗

Clarifying the System Contribution and Strengthening Experimental Evidence

We thank the reviewer for recognizing the practical relevance and accuracy–efficiency trade-off of LMF-3DDet. We understand the concerns about incremental novelty and experimental controls. We clarify the contribution and distinguish demonstrated results from limitations below.

  1. Contribution Beyond Component Aggregation. As stated in Sections 1 and 3.3, we do not claim a new detection paradigm or a new linear-attention identity. Our contribution concerns how computation is allocated across input selection, cross-modal interaction, and training supervision in a lightweight detector. These components address different constraints: RGS reduces the number of LiDAR queries; kernelized attention reduces interaction cost at a given query count; and KD improves supervision without retaining the teacher at inference. RGS and linear attention therefore act on complementary sources of computation, while distillation supports the reduced-capacity student. This motivates their joint design, although it does not by itself prove an empirical synergy. The existing experiments provide concrete operating points. Table 3 reports 69.7 mAP at 44.5 FPS with 11.5k tokens, compared with 68.5 mAP at 32.1 FPS for 30k random sampling. Table 4 shows a throughput increase from 32.5 to 44.5 FPS with a 0.2-point mAP decrease. We agree that these results do not fully isolate every component. We will sharpen the contribution statement and add a configuration matrix separating individual and combined changes, with a fixed student and training protocol wherever applicable.

  2. Accuracy, Camera Configuration, and Speed Comparisons. We acknowledge that our reported accuracy is below recent multi-camera fusion methods. Section 4.2 explicitly reports this gap and presents our single-camera model as a lightweight operating point, rather than the highest-accuracy detector. A single-camera configuration does not establish superiority under matched inputs. We will annotate modality, camera count, temporal inputs, and timing conditions in the comparison tables. Published FPS values will remain contextual references, not evidence of a controlled speed ranking. Additional matched-setting comparisons will be distinguished from literature results.

  3. Sampling Controls and Robustness. We agree that Table 3 changes both selection policy and token budget, as already acknowledged in Section 4.3 and the Limitations section. To isolate selection quality, we will add random and farthest-point sampling controls matched to the RGS retained-point count per frame, alongside a full-input baseline and a no-dilation variant. Raw point counts and encoded token counts will be distinguished. The paper also acknowledges that dilation cannot guarantee foreground coverage under degraded images or calibration errors. We will quantify object-level point retention by distance and available visibility annotations, reporting objects with no original LiDAR points separately. Coverage alone will not be treated as detection robustness; class- and distance-specific detection results are also needed. Controlled image degradation and calibration perturbations would further test this limitation, and we will distinguish measured evidence from unresolved robustness questions.

  4. Uncertainty Weighting and Statistical Reliability. Section 3.5 explicitly describes the uncertainty weight as hard-example emphasis, not teacher-confidence filtering. Its range, from 0.5 toward approximately 0.9933, limits relative weighting to less than twofold, but does not guarantee that uncertain targets are reliable. We will clarify this trade-off and use same-architecture no-KD, uniform-KD, and uncertainty-KD controls, including a uniform control matched to average effective KD strength. Table 5 contains configuration differences, so its 2.2-point improvement cannot be attributed to KD alone. We also agree that single-run differences do not establish statistical reliability. Repeated-seed results are particularly important for interpreting the 0.2-point attention difference and the 0.6-point distillation difference. We will report means and variability for the principal controlled comparisons rather than imply that these differences are established beyond training variation.

  5. Deployment Scope and Presentation. Our current measurements establish RTX 4090 throughput, not embedded-device latency or energy efficiency. Fewer tokens and lower interaction complexity do not, by themselves, demonstrate lower energy consumption. We will limit deployment claims accordingly and identify embedded and energy evaluation as remaining work unless measured results are available.

To improve readability, we will add a compact configuration table and more explicitly connect each design choice to its supporting experiment and limitation. Essential clarifications will appear in the manuscript, with supporting configurations, measurement procedures, and available supplementary results provided in an anonymous repository.

Reviewer_U9k9OpenReview ↗

I think this paper is indeed a solid piece of work, but for ICLR the acceptance bar is indeed quite high. I do not think the core motivation of this paper merits a score of 3; for a lightweight work without particularly interesting points, to me I can only give it a 2.

I think this paper is indeed a solid piece of work, but for ICLR the acceptance bar is indeed quite high. I do not think the core motivation of this paper merits a score of 3; for a lightweight work without particularly interesting points, to me I can only give it a 2.

AuthorsOpenReview ↗

Clarifying the Contribution and Novelty of LMF-3DDet

Thank you for your candid assessment and for recognizing that this is a solid piece of work. We understand your concern that, despite being technically sound, the paper may not offer the level of conceptual novelty expected at ICLR.

We would like to clarify the scope of our claim. We do not present LMF-3DDet as a new detection paradigm, nor do we claim novelty for linear attention or knowledge distillation in isolation. Our contribution is narrower: we study how to allocate computation across LiDAR input selection, cross-modal interaction, and training-time supervision in a lightweight single-camera detector. We agree that combining established components is not, by itself, sufficient evidence of a meaningful contribution. The key question is whether the proposed design provides a measurable benefit under controlled comparisons.

In the revision, we will make this positioning more explicit and strengthen the evidence with matched-budget sampling controls, a configuration matrix that separates the effects of the individual components, and repeated-seed results for the principal comparisons. We will also clearly distinguish measured results from claims that remain unverified, and limit our efficiency claims to the tested hardware and camera setting.

We appreciate your point that these clarifications may not change your assessment of the paper’s novelty. We nevertheless hope that the more precise scope and controlled evidence will help you judge the contribution on its intended terms. If, after considering these clarifications, you still find that the work does not meet ICLR’s novelty bar, we understand and respect your assessment.

Reviewer_yGdHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes LMF-3DDet, a lightweight framework that jointly addresses input selection, feature fusion, and knowledge transfer to improve efficiency while maintaining or improving detection accuracy. The framework consists of three components: response-guided sampling to select informative LiDAR observations while preserving local geometric context, kernelized linear cross-attention for efficient feature fusion, and uncertainty-aware distillation to transfer knowledge from a teacher model to a lightweight student for efficient inference. The authors evaluate the framework on nuScenes and Waymo and provide ablation studies on sampling, attention type, and distillation, along with a module-level latency breakdown.

Critical strengths
  1. LMF-3DDet has a clear modular design, with each component addressing a specific challenge. Reducing the number of LiDAR tokens before cross-modal interaction and using the teacher only during training help improve inference efficiency while preserving detection accuracy.

  2. The analysis of computational cost and latency is thorough. Section 3.3 carefully distinguishes which components scale linearly and which do not. The module-level latency breakdown in Table 6 is informative, and reporting both FPS and mAP in the main results facilitates assessment of the trade-off between efficiency and accuracy.

  3. The experimental evaluation is comprehensive, with comparisons against multiple baselines across different datasets. The component studies also provide useful evidence of the contributions of individual design choices.

Critical weaknesses
  1. The comparison tables include methods with different input modalities and camera configurations. For example, BEVFormer is a camera-only method, while some baselines use multiple camera views and the proposed method uses only a single view. These differences make direct comparisons difficult to interpret. The authors should clearly document each method’s input configuration and either evaluate baselines under a unified protocol or group results by modality and camera setup.

  2. Although inference is efficient, the approach requires training both a teacher and a student, increasing the overall training cost and implementation burden. Moreover, knowledge distillation could also improve competing methods. Applying a comparable distillation strategy to relevant baselines would help distinguish the benefits of the proposed architecture from those of teacher supervision.

  3. The paper lacks comparisons with recent methods, such as Detecting As Labeling (ECCV 2024) and Adaptive LiDAR Scanning (2025), where applicable. Such comparisons would help establish the method’s competitiveness in terms of both detection accuracy and efficiency.

Questions for authors
  1. Point-cloud sampling may remove all points associated with distant or partially visible objects, but the paper does not quantify this risk. Could the authors report the proportion of ground-truth objects that retain at least one LiDAR point after RGS, ideally broken down by distance and visibility?

  2. Could the authors add the following comparisons to Table 3: (i) random sampling and farthest point sampling under the same 11.5k-point, and (ii) a baseline using the full point cloud without sampling?

  3. Could other methods also achieve lower inference latency while preserving detection accuracy if equipped with a comparable teacher–student distillation framework? Such a comparison would help clarify whether the gains stem from the proposed architecture or from distillation more generally.

3 comments
AuthorsOpenReview ↗

Clarifying Fair Comparisons, Training Costs, and Sampling Coverage

We thank the reviewer for recognizing the modular design, computational analysis, and experimental evaluation of our framework. We agree that clearer comparison settings and additional controls would strengthen the evidence.

  1. Input Configurations and Fair Comparisons. We agree that methods using different modalities and camera configurations should not be interpreted as directly controlled comparisons. We will reorganize the main tables by input modality and annotate the camera count, temporal inputs, and whether each result is published or independently reproduced. Camera-only BEVFormer will be separated from LiDAR–camera methods. Our single-camera configuration represents a distinct operating point, not evidence of architectural superiority under identical inputs. As already noted in Section 4.2, published FPS values use different timing protocols and do not establish a controlled speed ranking. We will make this qualification more prominent and distinguish literature comparisons from experiments conducted under matched input and timing settings.

  2. Training Cost and Distillation Controls. We agree that inference efficiency does not imply low total training cost. We will document teacher preparation, student training, and teacher-target generation, including GPU-hours, peak memory, checkpoint reuse, and whether targets are generated online or cached. KD can also benefit competing methods. Our contribution is the coordinated sampling–fusion–supervision design, not an exclusive ability to use KD. Distillation improves the training signal; the deployed architecture and input processing determine inference computation. To distinguish these effects, a controlled comparison evaluates both our student and a representative lightweight baseline with and without comparable KD. The teacher checkpoint, data split, training budget, target construction, and KD-strength selection procedure should be matched, with architecture-specific alignment documented. Table 5 contains configuration differences, so it does not establish the architecture-only or KD-only gain. We will revise the attribution claims and distinguish new controlled results from existing evidence.

  3. Object Coverage After RGS. We agree that aggregate mAP does not quantify the risk of removing all points from an object. We will add object-level coverage statistics using the same evaluation-eligible ground-truth boxes before and after sampling. For objects containing at least one input LiDAR point, we will report the proportion retaining at least one point after RGS. Objects with zero input points will be reported separately to distinguish original sensor sparsity from sampling-induced loss. Results will be grouped by distance and, where annotations permit, visibility or occlusion, with group definitions and object counts specified. Since one retained point does not guarantee detectability, retained-point counts and distance-stratified detection performance will complement this measure. Inside-/outside-FOV coverage is also relevant to our single-camera setting.

  4. Matched-Budget and Full-Input Comparisons. We agree with the requested additions to Table 3. We will clarify whether 11.5k denotes retained raw points or encoded tokens, which Section 3.1 distinguishes. Because RGS produces scene-dependent counts, matching random sampling and farthest-point sampling to its retained-point count per frame is more controlled than matching only the dataset average. We will add this comparison and a full-point-cloud baseline without RGS selection or dilation, while preserving standard preprocessing and range limits. The student architecture, attention, supervision, and training schedule will remain fixed across these controls, and timing will include sampling overhead. This will help separate selection quality from input-budget reduction.

  5. Recent Related Methods Both suggested methods are relevant. Detecting As Labeling (DAL) directly studies LiDAR–camera detection and the accuracy–speed trade-off. Its separation of modality contributions to classification and regression differs from our focus on point selection, efficient interaction, and teacher supervision. We will include DAL in the related work and add compatible benchmark results with explicit configuration annotations. Adaptive LiDAR Scanning addresses the related problem of allocating a limited sensing budget. It uses temporal information to guide subsequent scanning, whereas RGS selects points from an already acquired scan using current-image responses. We will discuss this relationship and compare compatible detection results, while distinguishing acquisition-stage energy savings from post-acquisition computational savings. We do not claim that RGS demonstrates reduced LiDAR acquisition energy.

We will incorporate essential clarifications into the manuscript and provide supporting configurations, measurement procedures, and available supplementary results in an anonymous repository.

Thank you for addressing my questions in the rebuttal. The paper is clearly written and presents strong experimental results. The paper offers few new insights beyond the empirical gains. I therefore keep my original rating of 3.

AuthorsOpenReview ↗

Thank You for Your Follow-up and Assessment

Thank you for your follow-up and for acknowledging the clarity of the paper and the strength of the experimental results. We appreciate your assessment and will make the scope of our contribution and the supporting evidence more explicit in the revision.

Reviewer_hfSZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the accuracy-efficiency trade-off in LiDAR-camera 3D detection. It uses a frozen image-response branch to select LiDAR points, expands the selected set with a 1 m geometric neighborhood, replaces dense cross-modal attention with normalized ELU-based kernel attention, and trains the student with uncertainty-weighted supervision from a full-input TransFusion teacher. The claimed contribution is the coordinated system rather than a new linear-attention identity.

Critical strengths

The design targets distinct sources of deployment cost: input processing, feature interaction, and student capacity. Removing the teacher at inference is a sensible way to improve training supervision without retaining the teacher's deployment cost. The attention formulation also clearly states its normalization and dimensions, and correctly distinguishes the ELU kernel from softmax attention.

The component results provide useful operating points. In Table 3, RGS uses 11.5k tokens and reports 69.7 mAP at 44.5 FPS, versus 30k tokens, 68.5 mAP, and 32.1 FPS for random sampling. Table 4 reports 44.5 versus 32.5 FPS for linear versus standard attention, with a 0.2-point mAP decrease. The 22.1 ms runtime breakdown is arithmetically consistent with approximately 45.3 FPS. These are meaningful reported trade-offs, subject to the controls below.

Critical weaknesses

First, the relationship between single-camera sampling and the benchmark evaluation region is unclear. Equation 2 retains only image-response seeds and points within 1 m of those seeds. Points far from all valid seeds are therefore discarded, including points outside the selected camera's coverage. The paper explicitly states that it does not aggregate responses across views, but does not explain an outside-FOV fallback, the camera used, or any filtering of ground truth and evaluation regions. This does not prove that the reported accuracy is impossible; it prevents determining whether the comparisons use the same detection task.

Second, the distillation study does not isolate distillation. Table 5 and Section 4.1 explicitly allow different model components across rows. The no-distillation row runs at 32.5 FPS, whereas the two distillation rows run at about 45 FPS. Consequently, the 2.2-point full-model improvement cannot be attributed to KD alone. Even the uniform-versus-uncertainty difference requires a configuration matrix. The unnormalized weighting in Eq. 8 also changes average KD strength, so a matched-strength uniform control would help identify the value of uncertainty ranking.

Third, at least one baseline entry conflicts with its cited source. Table 1 lists BEVFormer at 68.5 mAP and 71.0 NDS on nuScenes test, whereas the original BEVFormer v2 Table 1 reports 48.1/56.9 for V2-99 and 44.5/53.5 for R101. If this is an adapted or rerun model, its configuration and provenance need to be identified; otherwise the entry should be corrected. This is a result-provenance concern, not an inference about intent.

Finally, the supplied 11-page PDF does not contain the referenced supplementary Sections A/B. Important missing definitions include the teacher-student correspondence set, aggregation of regression variances into an instance uncertainty, response-branch pretraining, and point-encoder/BEV configurations. Table 3 also lacks an equal-token-budget control and a no-dilation variant.

Questions for authors
  • Please specify the exact point and ground-truth filtering, the official evaluation configuration, and inside/outside-FOV coverage and AP.
  • Please provide same-architecture no-KD, uniform-KD, and uncertainty-KD results, including matched average KD strength.
  • Please trace the main-table results to their original configurations and provide the missing supplementary definitions.
  • An equal-budget sampling study and explicit mapping of all modules into the runtime total would also be valuable.
2 comments
AuthorsOpenReview ↗

Clarifying Evaluation Coverage, Distillation Attribution, and Result Provenance

We thank the reviewer for recognizing the practical motivation and coordinated design of our framework. We appreciate the concerns regarding evaluation coverage, component attribution, baseline provenance, and reproducibility. We address them below.

  1. Camera selection and evaluation coverage. We use fixed front-facing streams: CAM_FRONT for nuScenes and CameraName.FRONT for Waymo. The channel remains fixed across training and evaluation, without response-based view selection or cross-view aggregation. Projection uses official dataset calibration and accounts for image resizing and cropping. Following Eq. (2), seeds must have positive camera depth, valid image coordinates, and a maximum class response above 0.3. We retain these seeds and all existing LiDAR points within 1 m of any seed. Outside-FOV points are retained only through this neighborhood rule; no additional fallback is applied. Dilation therefore preserves local context, not full-scene coverage. Ground-truth boxes are not additionally filtered by camera FOV or retained points. We use the official nuScenes and Waymo 3D detection protocols, reporting LEVEL_1 results for Waymo, without camera-based restrictions on the evaluation region. We will document these settings explicitly. Aggregate scores alone do not resolve the coverage concern; coverage statistics and inside-/outside-FOV AP are needed to characterize it directly.

  2. Distillation attribution and weighting. Sections 4.1 and 4.3 and the Table 5 caption already state that FPS differences arise from model configurations, not the training-only KD loss. We do not attribute inference acceleration to KD. Nevertheless, Table 5 does not isolate KD under an identical architecture. The 2.2-point difference is a configuration-level improvement, not an isolated KD gain. The 0.6-point uniform-versus-uncertainty difference also requires an explicit configuration matrix before it can establish the contribution of uncertainty weighting. We will revise these claims accordingly. Section 3.5 describes the weighting as bounded hard-example emphasis, not teacher-confidence filtering. Eq. (9) gives weights from 0.5 toward approximately 0.9933. This bound does not remove the loss-scale confound. A controlled comparison must fix the student architecture, sampling, attention, and training settings across no-KD, uniform-KD, and uncertainty-KD runs, and include a uniform control matched to the average effective KD weight. We will distinguish the current evidence from any additional controlled results.

  3. Baseline provenance. We confirm that Table 1 of the cited ECCV 2022 BEVFormer paper reports 48.1 mAP / 56.9 NDS for V2-99 and 44.5 / 53.5 for R101 on nuScenes test. Our 68.5 / 71.0 entry is not supported as a published result of that reference. We will correct this entry and its attribution, audit the Waymo entry and remaining baselines, and distinguish published results from independently rerun or adapted configurations. We will also verify FPS provenance. As noted in the manuscript, literature speeds provide context rather than a controlled speed ranking.

  4. Reproducibility, sampling controls, and runtime. The supplied PDF does not contain the referenced supplementary Sections A/B. We will provide the teacher-student correspondence algorithm, regression-variance aggregation into instance uncertainty, response-branch pretraining and freezing, and point-encoder/BEV configurations. A symbolic correspondence set alone does not specify its construction. Section 4.3 and the Limitations section acknowledge that Table 3 changes both sampling policy and token budget. These results demonstrate operating points but do not isolate response guidance, token count, and dilation. Equal-budget and no-dilation controls are needed for that purpose; we will limit the current claims accordingly. The 22.1 ms total in Table 6 is consistent with approximately 45.3 FPS. We will clarify how response prediction, point encoding, BEV processing, and all remaining inference operations map to the reported stages, together with the timing scope and measurement procedure.

We will incorporate essential clarifications into the revised manuscript and provide additional implementation details, configurations, evaluation protocols, and available supplementary results in an anonymous GitHub repository. Our contribution remains the coordinated accuracy-efficiency design; component-level conclusions will reflect the available controls, and any protocol or provenance corrections will be reported transparently.

Thanks for your rebuttal. Most of concerns are resolved. I'd like to raise my ratings to 3 weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.