acceptodds
Under review as a conference paper at ICLR 2027

MoE4D: Mixture of Spatio-Temporal Experts with Frequency-Modulated Router for LiDAR Novel View Synthesis

Abstract

Although recent progress in 4D scene representations significantly advances LiDAR novel view synthesis, it still remains inconclusive which spatio-temporal representation performs better across diverse regions and time-variant motion patterns. NeRF-based methods encode continuous 4D implicit field with superior parameter efficiency, while struggling in long-term dynamics and occlusion scenarios. In contrast, Gaussian Splatting-based methods enable fast real-time rendering but commonly require complicated ray-object intersection modeling, dense trajectory labels of dynamics, and explicit storage of Gaussian primitives. In this paper, we design a novel differential LiDAR simulation method based on Mixture-of-Experts, called MoE4D, to effectively combine various spatio-temporal cues within a unified rendering pipeline. Specifically, multi-planar, hash grid, and object-centric GS serve as main expert candidates, providing multi-granularity geometry and deformation priors. We also observe that previous works mostly overlook spectral discrepancies within LiDAR sequences. To mitigate this issue, a frequency-modulated context router is designed based on the Fourier analysis of input 4D signals. Furthermore, a dense-to-sparse expert distillation strategy is developed to maintain high-fidelity reconstruction quality with significantly reduced computational budgets. Extensive experiments on KITTI-360 and nuScenes demonstrate state-of-the-art performance of our MoE4D. Codes will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LiDAR novel view synthesis, Mixture-of-Experts, Frequency modulation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_SHDgOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MoE4D, a Mixture-of-Experts (MoE)-based method that combines heterogeneous spatio-temporal representations for LiDAR novel view synthesis (NVS). Existing NeRF- and Gaussian Splatting (GS)-based methods have different strengths and limitations depending on spatial regions and motion patterns. To address this, the proposed method employs five experts: static and dynamic K-planes, static and dynamic hash grids, and object-centric Gaussians. K-planes are used to model low-frequency geometry, hash grids capture high-frequency details, and the Gaussian expert models dynamic objects. To effectively combine these experts, the paper introduces a frequency-modulated context router. This router uses spatial features, temporal embeddings, and Fourier-based spectral descriptors to select the appropriate top-k experts for each ray sample. The selected features are then processed through a unified volume rendering pipeline to predict depth, intensity, and ray-drop. Finally, the method employs dense-to-sparse distillation to transfer knowledge from a teacher model using more experts to a student model. It also adaptively allocates more experts to samples with higher routing ambiguity, balancing reconstruction quality and computational efficiency. Experimental results demonstrate that the proposed method generally achieves superior reconstruction quality compared to existing methods on KITTI-360 and nuScenes, while reducing inference latency through sparse routing.

Critical strengths

The paper effectively leverages the complementary properties of different spatio-temporal representations. In particular, the design of K-planes, hash grids, and Gaussians for low-frequency geometry, high-frequency details, and dynamic objects, respectively, is well motivated. Combining these representations within a unified rendering pipeline is also a reasonable design choice. The idea of selecting appropriate experts based on spatio-temporal queries is particularly interesting for unordered and sparse LiDAR point clouds.

The frequency-modulated context router is also well designed. Instead of relying solely on spatial coordinates, it incorporates temporal context and spectral descriptors for expert selection. In particular, the use of FiLM to modulate Fourier-based descriptors is well aligned with the goal of handling different levels of geometric complexity.

The paper considers not only reconstruction quality but also computational efficiency. The use of dense-to-sparse distillation and adaptive routing to allocate more experts to samples that require them is reasonable. The authors also conduct various ablation studies and backbone compatibility experiments, and the method demonstrates generally strong reconstruction quality.

Critical weaknesses

One of the main strengths of this paper is its ability to select appropriate experts based on spatial structures and temporal characteristics. However, the paper lacks sufficient analysis of which geometric characteristics actually influence the router's expert selection. Although Figure 1 provides some visualizations, the relationship between frequency characteristics and expert selection is not thoroughly validated. Therefore, it remains unclear whether each expert has learned its intended specialization.

Although the paper provides ablation studies that remove individual experts and routing features, these experiments do not sufficiently disentangle the benefits of combining heterogeneous representations from those of learned routing. In particular, comparisons with simple feature concatenation or conventional learned gating using the same experts are missing. Such comparisons would help clarify the effectiveness of the proposed frequency-modulated router.

The object-centric Gaussian expert requires tracked bounding boxes, but the sensitivity of the method to the accuracy of this information is not sufficiently analyzed. Furthermore, the experiments mainly focus on held-out frame reconstruction in relatively short sequences, leaving the robustness of the method under large viewpoint changes, long-term motion, and occlusions unclear.

Finally, MAC-NeRF is included in the qualitative comparison and compatibility evaluation but is excluded from the main quantitative comparisons. Additionally, computational efficiency is primarily evaluated across the method's own top-k settings. A comparison of computational costs with existing LiDAR NVS methods would further strengthen the experimental evaluation.

Questions for authors
  1. Expert Specialization: The key idea of this paper is to select appropriate experts based on spatial regions and motion patterns. However, it is unclear whether the router actually learns the intended expert specialization. Could the authors analyze the selection probabilities of each expert for planar surfaces, sharp boundaries, fine-grained geometry, and dynamic objects? I would also be interested in understanding the relationship between geometric complexity or motion magnitude and the resulting routing decisions.

  2. Effectiveness of the Frequency Descriptor: In Eq. (2), Fourier coefficients are predicted from spatial features to compute spectral descriptors. However, it is unclear how closely these descriptors correspond to the actual geometric frequencies or temporal motion characteristics of the scene. Could the authors analyze how spectral centroid, entropy, and band power respond to different geometric structures and motion patterns? In particular, I would like to understand how this frequency information actually influences expert selection.

  3. Heterogeneous Expert Fusion: The paper demonstrates strong performance by combining multiple representations. However, it is difficult to distinguish whether the improvements come from incorporating additional experts or from the learned routing mechanism. Could the authors compare the proposed method with simple concatenation, dense weighted fusion, and a conventional learned router using the same set of experts? Comparisons under similar model capacities and computational budgets would help better isolate the benefits of the proposed routing mechanism.

  4. Object-Centric Gaussian Expert: The Gaussian expert models dynamic objects using tracked bounding boxes. Are these bounding boxes obtained from ground-truth annotations or from a separate tracking model? Additionally, how are object positions determined at novel timestamps? I would also like to understand how sensitive the reconstruction quality is to bounding box errors or tracking failures.

  5. Novel View Extrapolation: The current experiments mainly focus on reconstructing held-out frames from existing sequences. However, an important application of LiDAR NVS is sensor simulation from viewpoints that differ from the original trajectory. Could the authors provide spatial extrapolation experiments by applying lateral offsets or rotations to the original sensor trajectory?

  6. Ambiguity in Adaptive Routing: The paper defines reconstruction ambiguity using the probability mass outside the selected top-k experts. However, similar selection probabilities across multiple experts do not necessarily indicate that a sample is difficult to reconstruct. Could the authors analyze the correlation between the proposed ambiguity score and actual reconstruction error? It would also be helpful to visualize whether samples with high ambiguity scores are concentrated around object boundaries, occluded regions, or dynamic regions.

  7. Baseline Comparison: MAC-NeRF is included in Table 5 but excluded from the main quantitative comparisons in Tables 1 and 3. Is there a specific reason for this omission? Additionally, could the authors compare inference latency against existing GS-based LiDAR NVS methods under the same hardware and evaluation settings?

6 comments

Thanks for your constructive review and recognition of our novelty, motivation, and strong experimental results. We revise our paper accordingly as follows:

Q1: Expert Specialization

A1: Thank you for your advice. We supplement expert specialization experiments and the analysis of geometric complexity and motion magnitude.

(i) Region-wise routing analysis For each ray with a valid LiDAR return, we calculate and report the mean selection probabilities based on the dense five-expert setting. The first three rows are from static background points outside Gaussian support boxes. Ground-truth geometry and object tracks are used only to define evaluation groups. The results are as below:

RegionPlane-staticPlane-dynamicHash-staticHash-dynamicGaussian
Planar surfaces93.6%3.1%1.9%1.4%0%
Sharp boundaries2.5%0.1%92.8%4.6%0%
Fine-grained geometry4.7%1.0%94.1%0.2%0%
Dynamic objects0.4%10.6%0.2%11.8%77.0%

From the table, static plane receives the largest probability on planar surfaces, sharp boundaries and fine-grained geometry are mostly allocated to static hash, and Gaussian expert has the highest probability in dynamic objects. These results support the intended routing preferences for smooth structure, geometric detail, and object motion, showing that multiple experts contribute within each region.

(ii) Routing decision w.r.t. geometric complexity We also further evaluate the routing probabilities with different geometric complexity. We quantify geometric complexity using local surface variation computed within a fixed-radius neighborhood. We restrict the analysis to static background regions away from prominent boundaries and partition the samples into low-, medium-, and high-complexity levels.

Geometric complexityPlane-staticPlane-dynamicHash-staticHash-dynamicGaussian
low91.2%2.6%5.7%0.5%0%
medium43.1%3.5%48.8%4.6%0%
high7.1%1.4%90.9%0.6%0%

For regions with low geometry complexity, most of samples have the highest preference for plane representation. As the geometric complexity increases, more samples will be assigned to a hash-based representation, which can capture more complex geometry details.

(iii) Routing decision w.r.t. motion magnitude We estimate foreground object's speed as the displacement of its bounding-box center between consecutive frames in world coordinates divided by the frame interval. We then group the samples into low-, medium-, and high-speed categories.

MagnitudePlane-staticPlane-dynamicHash-staticHash-dynamicGaussian
low3.9%13.1%6.4%12.8%63.8%
medium0.6%12.7%1.3%14.6%71.6%
high0.4%8.3%0.2%9.8%81.3%

As the motion magnitude becomes larger, the Gaussian expert will be assigned more routing probabilities, and the proportion of static experts becomes smaller.

Q2: Effectiveness of the Frequency Descriptor

A2: Thank you for raising the question. We supplement two experiments that examine how the spectral descriptor relates to scene characteristics and influences expert selection.

(i) Analysis of how the spectral descriptor responds to geometry/motion To examine how the learned spectral descriptors respond to geometric structure, we evaluate the frequency distributions across different geometry complexities with the same setting as Q1. Normalized band powers, spectral centroid, and entropy are computed as:

Geometric complexityπ0\pi_{0}π1\pi_{1}π2\pi_{2}π3\pi_{3}CentroidEntropy
low0.55670.18700.18080.07550.25840.8252
medium0.44400.18410.19290.17900.36900.9359
high0.35870.17600.19330.27200.45950.9705

From the table, as the geometric complexity increases, low-frequency DC band power decreases, while high-frequency band power increases, resulting in a higher spectral centroid. This result is consistent with our motivation, where higher-frequency components commonly relate to complex geometry details. The accompanying increase in the entropy of the mean spectrum indicates a more balanced energy distribution across frequency bands, consistent with richer learned spectral content in geometrically complex regions.

Furthermore, we also analyze the spectral distributions in terms of various motion magnitudes.

Motion magnitudeπ0\pi_{0}π1\pi_{1}π2\pi_{2}π3\pi_{3}CentroidEntropy
low0.51380.17420.21560.09640.29820.8677
medium0.50720.17330.21470.10480.30570.8763
high0.48640.17060.22370.11930.32530.8951

From the results, we observe that with higher motion magnitude, there is larger dominance in high-frequency band powers, with higher centroid and entropy as well. This is partly because dynamics with large motion may be associated with more rapid temporal variations in local patterns.

(ii) Analysis of how frequency influences expert selection Analyzing the above tables together, we can get the following:

Geometric complexityPlane-staticPlane-dynamicHash-staticHash-dynamicπ0\pi_{0}π1\pi_{1}π2\pi_{2}π3\pi_{3}CentroidEntropy
low91.2%2.6%5.7%0.5%0.55670.18700.18080.07550.25840.8252
medium43.1%3.5%48.8%4.6%0.44400.18410.19290.17900.36900.9359
high7.1%1.4%90.9%0.6%0.35870.17600.19330.27200.45950.9705

Typically, for regions with low geometic complexity, most of the band powers are low-frequency, which motivates the router to assign more samples to the plane-based representation. In contrast, a higher frequency may indicate positions with more complex geometric details, in which hash grids are more specialized.

Q3: Heterogeneous Expert Fusion

A3: Thank you for your insightful idea. Accordingly, we compare our router-based fusion with three counterparts:

  • Simple concatenation: directly concatenates the five representation features.
  • Dense weighted fusion: combines representation features using globally learned softmax weights shared across all queries.
  • Conventional learned gating: uses a gating network to predict query-dependent softmax weights from normalized features, without spectral conditioning.

In particular, the same five expert representations are used for a fair comparison. The comparison results are as follows:

SettingsPoint Cloud CD↓Point Cloud F-score↑Depth RMSE↓Depth MedAE↓Depth LPIPS↓Depth SSIM↑Depth PSNR↑Intensity RMSE↓Intensity MedAE↓Intensity LPIPS↓Intensity SSIM↑Intensity PSNR↑
(i) simple concatenation0.10360.92983.74100.04030.28260.699726.92070.12780.03600.28640.454817.9486
(ii) dense weighted fusion0.10240.92953.77430.04210.30160.690326.82870.12860.03670.29440.442317.8833
(iii) conventional learned gating0.09930.93273.48790.03650.27190.738927.18640.11860.03420.25350.460019.1644
MoE-based routing0.09470.93913.07200.02490.07040.878028.74960.09890.02610.14680.660320.1314

From the table, our MoE routing achieves much better performance across all 12 metrics. Specifically, (i) simple concatenation has 9.4% higher point cloud CD, 21.78% higher depth RMSE, and 29.22% higher intensity RMSE. This indicates that simply combining heterogeneous representations may inadequately reconcile redundant or even conflicting cues across the five experts, whereas MoE adaptively weights their contributions to exploit complementary information at each query location. (ii) Compared to direct concatenation, dense weighted fusion induces even higher reconstruction errors in terms of most metrics. This is partly because one set of weights is shared across all queries, which may overemphasize representations that are less useful in particular regions. (iii) Compared to simple concatenation and dense weighted fusion, the learned gating network has better fusion results by adaptively weighting representations to each query. However, the conventional gating method relies solely on content features, with 4.86% higher point cloud CD, 13.54% higher depth RMSE, and 19.92% higher intensity RMSE errors than MoE-based gating.

We agree that these comparisons will help to better isolate the benefits of the proposed routing mechanism. Thanks for your insight and we have included these comparison results and analysis in Table 4 (c) of the main paper.

Q4: Object-Centric Gaussian Expert

A4: Following LiDAR-RT (CVPR'25), 3D bounding boxes are obtained from ground truth annotations in the official KITTI-360 or nuScenes datasets. We associate annotated bounding boxes across training frames by object identity and use their poses to position each object's Gaussians. At novel timestamps, object positions are estimated by linearly interpolating bounding-box centers between valid neighboring training frames, while orientations are interpolated using quaternion spherical linear interpolation. Furthermore, we also evaluate the sensitiveness of our reconstruction quality to bounding box errors.

(a) Adding Gaussian Noise to Box Positions We perturb training-box centers with independent zero-mean Gaussian noise along each world-coordinate axis, keeping box orientations and dimensions fixed. Held-out object positions are then interpolated from the perturbed observations.

SettingsPoint Cloud CD↓Depth RMSE↓Intensity RMSE↓
GT bouding box0.09473.07200.0989
Noise (σ\sigma=0.1m)0.0948(+0.106%↑)3.07200.0990(+0.101%↑)
Noise (σ\sigma=0.5m)0.0951(+0.422%↑)3.0722(+0.007%↑)0.0994(+0.506%↑)

(b) Tracking Interruptions We remove each affected object’s bounding boxes over consecutive training frames, with gap length controlling interruption severity.

SettingsPoint Cloud CD↓Depth RMSE↓Intensity RMSE↓
GT bouding box0.09473.07200.0989
Remove 5 frames0.0952(+0.528%↑)3.0726(+0.020%↑)0.0994(+0.506%↑)
Remove 10 frames0.0956(+0.950%↑)3.0731(+0.036%↑)0.0998(+0.910%↑)

(c) Analysis about the Robustness From the results above, our overall reconstruction quality is quite robust to bounding box errors,** with no more than 1% fluctuations** in reconstructed point, depth, and intensity metrics. The reasons are maily two-folds: (1) In autonomous driving scenes, dynamic points typically account for a very small proportion of all the points. For example, in the KITTI-360 evaluation data, the averaged proportion of dynamic points with tracked bounding boxes is only 1.65%. These points can hardly have severe influence on the overall reconstruction quality. (2) We combine multiple deformation representations, reducing the dependence on the specific expert. Even if the bounding box is not accurate enough and the GS expert becomes inactive, the dynamic hash and dynamic plane experts can also model the dynamic objects.

Q5: Novel View Extrapolation

A5: Thank you for your great suggestion. We have added novel spatial extrapolation results in Section 4.5 of the main paper. Apart from just reconstructing held-out existing frames, our MoE4D also supports LiDAR re-simulation with novel viewpoints or sensor configurations. As in Fig.5, we manipulate LiDAR sensors and generate extrapolated LiDAR points under four re-simulation settings, including lateral shift, beam reduction, field-of-view (FOV) adjustment, and sensor rotation.

Q6: Ambiguity in Adaptive Routing

A6: Thank you for your question. We would like to clarify this from four aspects:

(i) Per-sample ambiguity socre v.s. per-ray LiDAR reconstruction We need to point out that our designed ambiguity score is defined at each individual volumetric sampling location, whereas reconstruction error is defined on per-ray rendered LiDAR returns. A sampling location does not have an independently corresponding GT depth or intensity to evaluate reconstruction quality. The attributes of the sampling locations are obtained by integrating all the prediction values along the entire ray. Therefore, ambiguity score need not exhibit a monotonic correlation with the reconstruction error.

(ii) Post-allocation reconstruction error cannot reflect the correlation Secondly, the final reconstruction error is measured after adaptive expert allocation, which means that within each ray, high-ambiguity samples have already received additional experts. The measured reconstruction error therefore already reflects this corrective intervention. Therefore, the post-allocation reconstruction error can not reflect the real correlation or contribution of ambiguity.

For both reasons above, the correlation with the final reconstruction error alone is insufficient to assess the effectiveness of the proposed mechanism.

Reviewer_M68bOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on LiDAR-based novel view synthesis. This work first proposes using Mixture of Experts to integrate information from different encodings including static/dynamic hashgrid, static/dynamic K-planes and their designed Gaussian features. Then this work proposes further distilling the trained model to improve synthesis efficiency. The effectiveness of the proposed method is verified on two pubic datasets KITTI360 and nuScenes.

Critical strengths

The use of Mixture of Experts (MOE) is novel in LiDAR-based novel view synthesis and the paper shows that this integration of different encoded information using MOE can be extended to other backbones.

Critical weaknesses

Although the use of MOE is novel, the experts are all based on existing methods. Hence, I focus more on the design of router. However, the illustration of router is not clear enough and several questions (below) require further clarification.

In addition, the motivation for proposing the distillation strategy seems incremental. As the work uses multiple experts, which introduces more computation, distillation is hence a natural design choice.

Questions for authors
  1. Regarding Lightweight Spatial and Temporal Routing Context, how is v(X,t) constructed? What does ``summarizing temporal variations" mean in Line 193?
  2. Regarding Fourier-based Spectral Descriptor, the motivation behind is ``low frequency spectrum corresponds to smooth planes or coarse geometry structures and high frequency spectrum captures geometric details or rapidly-moving dynamic" (Line94-95), however, the Fourier coefficients are predicted using spatial context g only, not including temporal information, then how does this spectral descriptor handle dynamics?
  3. For distillation, is distillation performed after training the teacher network? If yes, is Equation 6 for training the student network?
  4. Line 200, wrong writing? {a_b(X),b_b(X)} instead of {a_b(X,b_b(X)}?
4 comments

Thank you for your recognition of our paper's novelty and universality across different backbones. We appreciate your questions and will strengthen our writing as follows.

Q1: Clarifications on several router details

A1: We agree that some detailed descriptions in the router are missing. Thank you for your suggestion, and we have included them in the main paper now.

  • Q1.1: How v(x,t) is computed?

  • A1.1: We construct v(x,t) from multi-frame LiDAR sequences using a shared 3D voxel grid. For each training frame, we count valid returns in each voxel, normalize the counts by the number of sampled rays, and apply logarithmic compression and local spatial averaging. Afterwards, we compute the mean and standard deviation of this normalized return-density signal at each voxel. In Line 193, “summarizing temporal variations” specifically refers to the standard deviation across frames, which measures temporal fluctuations in observed LiDAR returns at each spatial location. At query time, we obtain v(X,t) by trilinear interpolation of the precomputed grid.

  • Q1.2: How does the spectral descriptor handle dynamics?

  • A1.2: The ability of our spectral descriptor to handle dynamics comes mainly from: (1) Frequency coefficients correspond to different temporal components in the Fourier parameterization.

As shown in Eq. (2), we construct

u(X,t)=a0(X)+∑b=1B[ab(X)cos⁡(2πνbt)+bb(X)sin⁡(2πνbt)].\mathbf{u}(X,t)=\mathbf{a}_{0}(X)+\sum_{b=1}^{B}\left[\mathbf{a}_{b}(X)\cos(2\pi\nu_{b}t)+\mathbf{b}_{b}(X)\sin(2\pi\nu_{b}t)\right].

The predicted coefficients are associated with temporal sine and cosine basis functions at predefined frequencies. Their band powers are used to construct a learned Fourier-parameterized spectral descriptor. For example, a1(X)\mathbf{a}_{1}(X) associated with cos⁡(2πt)\cos(2\pi t) corresponds to a lower-frequency temporal basis than a2(X)\mathbf{a}_{2}(X) associated with cos⁡(4πt)\cos(4\pi t).

(2) The coefficient prediction head is supervised by multi-frame reconstruction losses. Although the coefficient prediction head takes only the spatial context g(X), its outputs are trained using reconstruction losses from multiple timestamps. Gradients can therefore flow from the reconstruction loss through the expert weights and the spectral descriptor to the coefficient head, allowing the network to adaptively learn which relative band powers are useful to depict high-frequency dynamics.

(3) Furthermore, the obtained frequency descriptor is subsequently leveraged to modulate spatio-temporal features c(x,t)c(x,t) through FiLM. The temporal context c(X,t)c(X,t) influenced by the frequency descriptor will then be output to the expert scoring. For example, at a road location that a vehicle crosses, the temporal context can differ before and during vehicle passage. The same location-specific spectral descriptor may be combined with these different temporal contexts, allowing the router to assign different routing probabilities to static and dynamic experts.

(4) We further prove the temporally-aware spectral descriptor claim by classifying different levels of dynamics and separately computing their corresponding band powers. Specifically, we analyze the spectral distributions in terms of diverse motion magnitudes as:

Motion magnitudeπ0\pi_{0}π1\pi_{1}π2\pi_{2}π3\pi_{3}CentroidEntropy
low0.51380.17420.21560.09640.29820.8677
medium0.50720.17330.21470.10480.30570.8763
high0.48640.17060.22370.11930.32530.8951

Here, we estimate the foreground object's speed as the displacement of its bounding-box center between consecutive frames in world coordinates divided by the frame interval. We then group the samples into low-, medium-, and high-speed categories. From the results, we observe that with a higher motion magnitude, there is a larger dominance in the high-frequency band powers. This is partly because dynamics with large motion may be associated with more rapid temporal variations in local patterns.

  • Q1.3: Is distillation performed after training the teacher network?

  • A1.3: Yes. Distillation is performed after the higher-budget teacher is trained. The two training stages use the following objectives.

  • Teacher training. We train the teacher with a fixed top-kdk_d routing budget only using the ground-truth reconstruction losses defined in Eq. (6), Lteacher=λdLd+λinLin+λrayLray.L_{{teacher}}=\lambda_{d}L_{d}+\lambda_{{in}}L_{{in}}+\lambda_{{ray}}L_{{ray}}.

  • Student training. We initialize the lower-budget student from the pretrained teacher and optimize the full objective in Eq. (7), Lstudent=λdLd+λinLin+λrayLray+λKDLKD.L_{{student}}=\lambda_{d}L_{d}+\lambda_{{in}}L_{{in}}+\lambda_{{ray}}L_{{ray}}+\lambda_{{KD}}L_{{KD}}.

The student retains ground-truth reconstruction supervision and additionally matches the teacher’s fused features and predicted densities. We have clarified the distinction more clearly in the revised manuscript.

  • Q1.4: Wrong notation on line 200
  • A1.4: Thank you for pointing out this. Yes, we missed a ')' in Line 200, and we have added it to the main paper now.

Q2: The motivation and design of distillation

A2: Because the MoE architecture is already quite compute-efficient with sparse routing (please refer to Table 6, fixed routing), our main focus of distillation is to retain sufficient reconstruction priors, rather than reducing computational costs. We are the first work which introduces knowledge distillation in LiDAR NVS from dense high-budget experts to sparse low-budget ones. This can also help retain near-teacher reconstruction quality while operating at low latency. Furthermore, we are not just trivially adopting distillation here. Instead, we make two additional novel designs specially for the task:

  • (1) Hybrid feature-density distillation: To enforce both representation and geometry consistency between teacher and student networks, we match both features and densities over sampled queries with KD loss as in Eq.(5), where feature-level alignment and density discrepancies are considered.

  • (2) Ambiguity-guided adaptive routing: Budgets of dense experts at Teacher-level and sparse experts at student-level are adaptively allocated to different samples, which is quantified by the probability mass in the routing process. This strategy can enable the network to adaptively maintain sufficient reconstruction quality if needed while reducing the inference budget.

Thank you for the detailed clarifications. They have addressed my concerns, and I tend to increase my score.

Thank you so much for your recognition of our work and raising your score! We really appreciate your insightful suggestions on the router design, and the current version with addtional clarifications further strengthens our work. Thank you for your time and effort in reviewing our paper.

Reviewer_psUvOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a novel MoE-based LiDAR novel view synthesis pipeline. Within a unified volume rendering paradigm, five spatial-temporal experts including multi-planar, hash grid, and Gaussian-based representations are designed to combine various 4D priors. The paper also develops dense-to-sparse distillation module, which can retain sufficient geometric and motion cues from dense experts while keeping high efficiency. Experiments are extensive, where the proposed method achieves state-of-the-art accuracy in both KITTI-360 and nuScenes.

Critical strengths
  1. The paper’s contributions are novel to the LiDAR novel view synthesis task. It is the first work introducing the MoE architecture to this field, and integrating different deformation priors for better 4D representation learning is reasonable.
  2. The proposed method is well-motivated and designed appropriately. The authors introduce muti-planar expert for low-frequency regions, hash grids for high-frequency details, and object-level Gaussian expert for high dynamics.
  3. The integration of spatio-temporal and frequency cues in the MoE router is also interesting. The frequency analysis of LiDAR sequences is consistent with the dispatch for high- and low- frequency experts.
  4. Dense-to-sparse distillation can maintain accurate 4D reconstruction accuracy while improving inference efficiency.
  5. Experiments are sufficient to support the main claim. The method achieves the state-of-the-art performance on both KITTI-360 and nuScenes datasets.
Critical weaknesses

Overall, this paper is a great work with sufficient novelty and superior experiment results, which has no severe weaknesses. I only have one question about the frequency descriptor. The paper adopts the basic Fourier analysis of input 4D sequences, while additional ablations of other frequency-based design alternatives can be compared, such as FFT transformation.

Questions for authors

The paper adopts the basic Fourier analysis of input 4D sequences, while additional ablations of other frequency-based design alternatives can be compared, such as FFT transformation. Also, in Table 5 what is the detailed adaptation process based on the MAC-NeRF baseline?

3 comments

Thank you so much for your acknowledgment of the novelty, design, motivation, and strong results of our paper. We will supplement the additional comparisons and descriptions as follows.

Q1: Comparison with FFT-based frequency design

A1: Thank you for your question. We replace the Fourier-based frequency design with the FFT-based one, and the comparison results are as:

SettingsPoint Cloud CD↓Point Cloud F-score↑Depth RMSE↓Depth MedAE↓Depth LPIPS↓Depth SSIM↑Depth PSNR↑Intensity RMSE↓Intensity MedAE↓Intensity LPIPS↓Intensity SSIM↑Intensity PSNR↑
FFT0.10350.93363.07380.02630.07160.877928.71980.09910.02660.15820.656020.1162
Fourier (Ours)0.09470.93913.07200.02490.07040.878028.74960.09890.02610.14680.660320.1314

From the table, our designed Fourier-based frequency outperforms FFT-based method across all metrics, which proves that Fourier analysis may better capture basic frequency attributes in 4D LiDAR sequences. Thank you for your suggestion. We have included this comparison in the main paper Table 4 (a).

Q2: Clarification of detailed MAC-NeRF adaptation

A2: We follow similar adaptation process as that on LiDAR4D (main results). Typically, MAC-NeRF was established on LiDAR4D, where hybrid NeRF-based representations in LiDAR4D were retained and additional curriculum learning strategy with rectified temporal consistency loss was introduced. When adapting MAC-NeRF, we keep the hash grid and plane experts and curriculum learning unchanged, adding object-centric Gaussian representation as another expert. Similarly, we also develop spatial-temporal cues and frequency information for router. Thank you for your question. We have incorporated the detailed adaptation process into the description of the experiments.

Thank the author for their detailed responses to my question. The additional experiments and clarifications further strengthen their claim and contribution. Overall, this is a very interesting paper with solid novelty and insight for the LiDAR NVS field. Therefore, I am confident it should be accepted.

We are grateful for your acknowledgement of our paper's novelty and insight. Also, thank you for your time and effort in reviewing our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.