- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MoE4D, a Mixture-of-Experts (MoE)-based method that combines heterogeneous spatio-temporal representations for LiDAR novel view synthesis (NVS). Existing NeRF- and Gaussian Splatting (GS)-based methods have different strengths and limitations depending on spatial regions and motion patterns. To address this, the proposed method employs five experts: static and dynamic K-planes, static and dynamic hash grids, and object-centric Gaussians. K-planes are used to model low-frequency geometry, hash grids capture high-frequency details, and the Gaussian expert models dynamic objects. To effectively combine these experts, the paper introduces a frequency-modulated context router. This router uses spatial features, temporal embeddings, and Fourier-based spectral descriptors to select the appropriate top-k experts for each ray sample. The selected features are then processed through a unified volume rendering pipeline to predict depth, intensity, and ray-drop. Finally, the method employs dense-to-sparse distillation to transfer knowledge from a teacher model using more experts to a student model. It also adaptively allocates more experts to samples with higher routing ambiguity, balancing reconstruction quality and computational efficiency. Experimental results demonstrate that the proposed method generally achieves superior reconstruction quality compared to existing methods on KITTI-360 and nuScenes, while reducing inference latency through sparse routing.
Critical strengths
The paper effectively leverages the complementary properties of different spatio-temporal representations. In particular, the design of K-planes, hash grids, and Gaussians for low-frequency geometry, high-frequency details, and dynamic objects, respectively, is well motivated. Combining these representations within a unified rendering pipeline is also a reasonable design choice. The idea of selecting appropriate experts based on spatio-temporal queries is particularly interesting for unordered and sparse LiDAR point clouds.
The frequency-modulated context router is also well designed. Instead of relying solely on spatial coordinates, it incorporates temporal context and spectral descriptors for expert selection. In particular, the use of FiLM to modulate Fourier-based descriptors is well aligned with the goal of handling different levels of geometric complexity.
The paper considers not only reconstruction quality but also computational efficiency. The use of dense-to-sparse distillation and adaptive routing to allocate more experts to samples that require them is reasonable. The authors also conduct various ablation studies and backbone compatibility experiments, and the method demonstrates generally strong reconstruction quality.
Critical weaknesses
One of the main strengths of this paper is its ability to select appropriate experts based on spatial structures and temporal characteristics. However, the paper lacks sufficient analysis of which geometric characteristics actually influence the router's expert selection. Although Figure 1 provides some visualizations, the relationship between frequency characteristics and expert selection is not thoroughly validated. Therefore, it remains unclear whether each expert has learned its intended specialization.
Although the paper provides ablation studies that remove individual experts and routing features, these experiments do not sufficiently disentangle the benefits of combining heterogeneous representations from those of learned routing. In particular, comparisons with simple feature concatenation or conventional learned gating using the same experts are missing. Such comparisons would help clarify the effectiveness of the proposed frequency-modulated router.
The object-centric Gaussian expert requires tracked bounding boxes, but the sensitivity of the method to the accuracy of this information is not sufficiently analyzed. Furthermore, the experiments mainly focus on held-out frame reconstruction in relatively short sequences, leaving the robustness of the method under large viewpoint changes, long-term motion, and occlusions unclear.
Finally, MAC-NeRF is included in the qualitative comparison and compatibility evaluation but is excluded from the main quantitative comparisons. Additionally, computational efficiency is primarily evaluated across the method's own top-k settings. A comparison of computational costs with existing LiDAR NVS methods would further strengthen the experimental evaluation.
Questions for authors
-
Expert Specialization: The key idea of this paper is to select appropriate experts based on spatial regions and motion patterns. However, it is unclear whether the router actually learns the intended expert specialization. Could the authors analyze the selection probabilities of each expert for planar surfaces, sharp boundaries, fine-grained geometry, and dynamic objects? I would also be interested in understanding the relationship between geometric complexity or motion magnitude and the resulting routing decisions.
-
Effectiveness of the Frequency Descriptor: In Eq. (2), Fourier coefficients are predicted from spatial features to compute spectral descriptors. However, it is unclear how closely these descriptors correspond to the actual geometric frequencies or temporal motion characteristics of the scene. Could the authors analyze how spectral centroid, entropy, and band power respond to different geometric structures and motion patterns? In particular, I would like to understand how this frequency information actually influences expert selection.
-
Heterogeneous Expert Fusion: The paper demonstrates strong performance by combining multiple representations. However, it is difficult to distinguish whether the improvements come from incorporating additional experts or from the learned routing mechanism. Could the authors compare the proposed method with simple concatenation, dense weighted fusion, and a conventional learned router using the same set of experts? Comparisons under similar model capacities and computational budgets would help better isolate the benefits of the proposed routing mechanism.
-
Object-Centric Gaussian Expert: The Gaussian expert models dynamic objects using tracked bounding boxes. Are these bounding boxes obtained from ground-truth annotations or from a separate tracking model? Additionally, how are object positions determined at novel timestamps? I would also like to understand how sensitive the reconstruction quality is to bounding box errors or tracking failures.
-
Novel View Extrapolation: The current experiments mainly focus on reconstructing held-out frames from existing sequences. However, an important application of LiDAR NVS is sensor simulation from viewpoints that differ from the original trajectory. Could the authors provide spatial extrapolation experiments by applying lateral offsets or rotations to the original sensor trajectory?
-
Ambiguity in Adaptive Routing: The paper defines reconstruction ambiguity using the probability mass outside the selected top-k experts. However, similar selection probabilities across multiple experts do not necessarily indicate that a sample is difficult to reconstruct. Could the authors analyze the correlation between the proposed ambiguity score and actual reconstruction error? It would also be helpful to visualize whether samples with high ambiguity scores are concentrated around object boundaries, occluded regions, or dynamic regions.
-
Baseline Comparison: MAC-NeRF is included in Table 5 but excluded from the main quantitative comparisons in Tables 1 and 3. Is there a specific reason for this omission? Additionally, could the authors compare inference latency against existing GS-based LiDAR NVS methods under the same hardware and evaluation settings?
6 comments
Furthermore, we also analyze the spectral distributions in terms of various motion magnitudes.
| Motion magnitude | Centroid | Entropy | ||||
|---|---|---|---|---|---|---|
| low | 0.5138 | 0.1742 | 0.2156 | 0.0964 | 0.2982 | 0.8677 |
| medium | 0.5072 | 0.1733 | 0.2147 | 0.1048 | 0.3057 | 0.8763 |
| high | 0.4864 | 0.1706 | 0.2237 | 0.1193 | 0.3253 | 0.8951 |
From the results, we observe that with higher motion magnitude, there is larger dominance in high-frequency band powers, with higher centroid and entropy as well. This is partly because dynamics with large motion may be associated with more rapid temporal variations in local patterns.
(ii) Analysis of how frequency influences expert selection Analyzing the above tables together, we can get the following:
| Geometric complexity | Plane-static | Plane-dynamic | Hash-static | Hash-dynamic | Centroid | Entropy | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| low | 91.2% | 2.6% | 5.7% | 0.5% | 0.5567 | 0.1870 | 0.1808 | 0.0755 | 0.2584 | 0.8252 |
| medium | 43.1% | 3.5% | 48.8% | 4.6% | 0.4440 | 0.1841 | 0.1929 | 0.1790 | 0.3690 | 0.9359 |
| high | 7.1% | 1.4% | 90.9% | 0.6% | 0.3587 | 0.1760 | 0.1933 | 0.2720 | 0.4595 | 0.9705 |
Typically, for regions with low geometic complexity, most of the band powers are low-frequency, which motivates the router to assign more samples to the plane-based representation. In contrast, a higher frequency may indicate positions with more complex geometric details, in which hash grids are more specialized.
Q3: Heterogeneous Expert Fusion
A3: Thank you for your insightful idea. Accordingly, we compare our router-based fusion with three counterparts:
- Simple concatenation: directly concatenates the five representation features.
- Dense weighted fusion: combines representation features using globally learned softmax weights shared across all queries.
- Conventional learned gating: uses a gating network to predict query-dependent softmax weights from normalized features, without spectral conditioning.
In particular, the same five expert representations are used for a fair comparison. The comparison results are as follows:
| Settings | Point Cloud CD↓ | Point Cloud F-score↑ | Depth RMSE↓ | Depth MedAE↓ | Depth LPIPS↓ | Depth SSIM↑ | Depth PSNR↑ | Intensity RMSE↓ | Intensity MedAE↓ | Intensity LPIPS↓ | Intensity SSIM↑ | Intensity PSNR↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (i) simple concatenation | 0.1036 | 0.9298 | 3.7410 | 0.0403 | 0.2826 | 0.6997 | 26.9207 | 0.1278 | 0.0360 | 0.2864 | 0.4548 | 17.9486 |
| (ii) dense weighted fusion | 0.1024 | 0.9295 | 3.7743 | 0.0421 | 0.3016 | 0.6903 | 26.8287 | 0.1286 | 0.0367 | 0.2944 | 0.4423 | 17.8833 |
| (iii) conventional learned gating | 0.0993 | 0.9327 | 3.4879 | 0.0365 | 0.2719 | 0.7389 | 27.1864 | 0.1186 | 0.0342 | 0.2535 | 0.4600 | 19.1644 |
| MoE-based routing | 0.0947 | 0.9391 | 3.0720 | 0.0249 | 0.0704 | 0.8780 | 28.7496 | 0.0989 | 0.0261 | 0.1468 | 0.6603 | 20.1314 |
From the table, our MoE routing achieves much better performance across all 12 metrics. Specifically, (i) simple concatenation has 9.4% higher point cloud CD, 21.78% higher depth RMSE, and 29.22% higher intensity RMSE. This indicates that simply combining heterogeneous representations may inadequately reconcile redundant or even conflicting cues across the five experts, whereas MoE adaptively weights their contributions to exploit complementary information at each query location. (ii) Compared to direct concatenation, dense weighted fusion induces even higher reconstruction errors in terms of most metrics. This is partly because one set of weights is shared across all queries, which may overemphasize representations that are less useful in particular regions. (iii) Compared to simple concatenation and dense weighted fusion, the learned gating network has better fusion results by adaptively weighting representations to each query. However, the conventional gating method relies solely on content features, with 4.86% higher point cloud CD, 13.54% higher depth RMSE, and 19.92% higher intensity RMSE errors than MoE-based gating.
We agree that these comparisons will help to better isolate the benefits of the proposed routing mechanism. Thanks for your insight and we have included these comparison results and analysis in Table 4 (c) of the main paper.
Q4: Object-Centric Gaussian Expert
A4: Following LiDAR-RT (CVPR'25), 3D bounding boxes are obtained from ground truth annotations in the official KITTI-360 or nuScenes datasets. We associate annotated bounding boxes across training frames by object identity and use their poses to position each object's Gaussians. At novel timestamps, object positions are estimated by linearly interpolating bounding-box centers between valid neighboring training frames, while orientations are interpolated using quaternion spherical linear interpolation. Furthermore, we also evaluate the sensitiveness of our reconstruction quality to bounding box errors.
(a) Adding Gaussian Noise to Box Positions We perturb training-box centers with independent zero-mean Gaussian noise along each world-coordinate axis, keeping box orientations and dimensions fixed. Held-out object positions are then interpolated from the perturbed observations.
| Settings | Point Cloud CD↓ | Depth RMSE↓ | Intensity RMSE↓ |
|---|---|---|---|
| GT bouding box | 0.0947 | 3.0720 | 0.0989 |
| Noise (=0.1m) | 0.0948(+0.106%↑) | 3.0720 | 0.0990(+0.101%↑) |
| Noise (=0.5m) | 0.0951(+0.422%↑) | 3.0722(+0.007%↑) | 0.0994(+0.506%↑) |
(b) Tracking Interruptions We remove each affected object’s bounding boxes over consecutive training frames, with gap length controlling interruption severity.
| Settings | Point Cloud CD↓ | Depth RMSE↓ | Intensity RMSE↓ |
|---|---|---|---|
| GT bouding box | 0.0947 | 3.0720 | 0.0989 |
| Remove 5 frames | 0.0952(+0.528%↑) | 3.0726(+0.020%↑) | 0.0994(+0.506%↑) |
| Remove 10 frames | 0.0956(+0.950%↑) | 3.0731(+0.036%↑) | 0.0998(+0.910%↑) |
(c) Analysis about the Robustness From the results above, our overall reconstruction quality is quite robust to bounding box errors,** with no more than 1% fluctuations** in reconstructed point, depth, and intensity metrics. The reasons are maily two-folds: (1) In autonomous driving scenes, dynamic points typically account for a very small proportion of all the points. For example, in the KITTI-360 evaluation data, the averaged proportion of dynamic points with tracked bounding boxes is only 1.65%. These points can hardly have severe influence on the overall reconstruction quality. (2) We combine multiple deformation representations, reducing the dependence on the specific expert. Even if the bounding box is not accurate enough and the GS expert becomes inactive, the dynamic hash and dynamic plane experts can also model the dynamic objects.
Q5: Novel View Extrapolation
A5: Thank you for your great suggestion. We have added novel spatial extrapolation results in Section 4.5 of the main paper. Apart from just reconstructing held-out existing frames, our MoE4D also supports LiDAR re-simulation with novel viewpoints or sensor configurations. As in Fig.5, we manipulate LiDAR sensors and generate extrapolated LiDAR points under four re-simulation settings, including lateral shift, beam reduction, field-of-view (FOV) adjustment, and sensor rotation.
Q6: Ambiguity in Adaptive Routing
A6: Thank you for your question. We would like to clarify this from four aspects:
(i) Per-sample ambiguity socre v.s. per-ray LiDAR reconstruction We need to point out that our designed ambiguity score is defined at each individual volumetric sampling location, whereas reconstruction error is defined on per-ray rendered LiDAR returns. A sampling location does not have an independently corresponding GT depth or intensity to evaluate reconstruction quality. The attributes of the sampling locations are obtained by integrating all the prediction values along the entire ray. Therefore, ambiguity score need not exhibit a monotonic correlation with the reconstruction error.
(ii) Post-allocation reconstruction error cannot reflect the correlation Secondly, the final reconstruction error is measured after adaptive expert allocation, which means that within each ray, high-ambiguity samples have already received additional experts. The measured reconstruction error therefore already reflects this corrective intervention. Therefore, the post-allocation reconstruction error can not reflect the real correlation or contribution of ambiguity.
For both reasons above, the correlation with the final reconstruction error alone is insufficient to assess the effectiveness of the proposed mechanism.
Thanks for your constructive review and recognition of our novelty, motivation, and strong experimental results. We revise our paper accordingly as follows:
Q1: Expert Specialization
A1: Thank you for your advice. We supplement expert specialization experiments and the analysis of geometric complexity and motion magnitude.
(i) Region-wise routing analysis For each ray with a valid LiDAR return, we calculate and report the mean selection probabilities based on the dense five-expert setting. The first three rows are from static background points outside Gaussian support boxes. Ground-truth geometry and object tracks are used only to define evaluation groups. The results are as below:
From the table, static plane receives the largest probability on planar surfaces, sharp boundaries and fine-grained geometry are mostly allocated to static hash, and Gaussian expert has the highest probability in dynamic objects. These results support the intended routing preferences for smooth structure, geometric detail, and object motion, showing that multiple experts contribute within each region.
(ii) Routing decision w.r.t. geometric complexity We also further evaluate the routing probabilities with different geometric complexity. We quantify geometric complexity using local surface variation computed within a fixed-radius neighborhood. We restrict the analysis to static background regions away from prominent boundaries and partition the samples into low-, medium-, and high-complexity levels.
For regions with low geometry complexity, most of samples have the highest preference for plane representation. As the geometric complexity increases, more samples will be assigned to a hash-based representation, which can capture more complex geometry details.
(iii) Routing decision w.r.t. motion magnitude We estimate foreground object's speed as the displacement of its bounding-box center between consecutive frames in world coordinates divided by the frame interval. We then group the samples into low-, medium-, and high-speed categories.
As the motion magnitude becomes larger, the Gaussian expert will be assigned more routing probabilities, and the proportion of static experts becomes smaller.
Q2: Effectiveness of the Frequency Descriptor
A2: Thank you for raising the question. We supplement two experiments that examine how the spectral descriptor relates to scene characteristics and influences expert selection.
(i) Analysis of how the spectral descriptor responds to geometry/motion To examine how the learned spectral descriptors respond to geometric structure, we evaluate the frequency distributions across different geometry complexities with the same setting as Q1. Normalized band powers, spectral centroid, and entropy are computed as:
From the table, as the geometric complexity increases, low-frequency DC band power decreases, while high-frequency band power increases, resulting in a higher spectral centroid. This result is consistent with our motivation, where higher-frequency components commonly relate to complex geometry details. The accompanying increase in the entropy of the mean spectrum indicates a more balanced energy distribution across frequency bands, consistent with richer learned spectral content in geometrically complex regions.