acceptodds
Under review as a conference paper at ICLR 2027

Triplex4DGS: High-Speed 4D Gaussian Splatting with Complementary Vision Sensors

Abstract

High-speed dynamic scene 4D reconstruction is crucial for real-world applications. Conventional RGB-based reconstruction suffers from limited temporal resolution to record fast-moving objects, while high-speed RGB cameras incur prohibitive bandwidth and data overhead. As an alternative, the complementary vision sensor (CVS) simultaneously captures synchronized 30-fps RGB frames alongside high-frequency, sparse and multi-bit Temporal Difference (TD) and Spatial Difference (SD) signals, enabling high-speed imaging with much lower bandwidth than traditional sensors. However, directly reconstructing RGB frames from TD/SD may introduce temporal inconsistencies that propagate into subsequent 4D reconstruction. Moreover, TD and SD are monochromatic and follow photometric characteristics distinct from RGB pixels, making joint optimization challenging. Meanwhile, existing 4D Gaussian Splatting frameworks lack differentiable rendering models for TD and SD supervision, so such data still fails to yield improvements for high-speed 4D reconstruction. To tackle the above challenges, we propose Triplex4DGS, a two-stage framework that fully exploits CVS' three complementary data modalities, achieving the capability of ultra-high-speed 4D scene reconstruction with extremely low hardware overhead. In the first stage of Triplex4DGS, a continuous dynamic Gaussian representation is initialized from sparse RGB keyframes. The second stage builds unified supervision within the TD–SD response space via a differentiable spatio-temporal difference simulator, enabling hybrid observation-guided optimization, where SD refines fine-grained spatial geometry, TD regularizes temporal motion trajectories, and RGB frames maintain consistent visual appearance. Furthermore, we build a multi-CVS data acquisition system and construct the first simulated and real-world CVS 4D reconstruction benchmark. Extensive experiments on both datasets demonstrate state-of-the-art performance, validating the effectiveness of Triplex4DGS with CVS for high-speed 4D reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Gaussian Splatting, Dynamic Scene Reconstruction, Complementary Vision Sensor

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_btT1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Triplex4DGS, a two-stage 4D Gaussian reconstruction framework for Tianmou's complementary vision sensor (CVS), which is used for high-speed dynamic scene reconstruction. Tianmou simultaneously provides approximately 30 FPS of RGB observations and 757 FPS of TD/SD differential signals.

Critical strengths

High-speed dynamic 4D reconstruction is limited by the RGB frame rate, and event cameras require additional RGB and calibration. Tianmu's RGB + TD/SD synchronous observation provides a potential alternative solution. The paper introduces CVS into 4DGS, which has good exploration value.

Critical weaknesses

(1) The paper mainly compares RGB-only 4DGS and event-based E-D3DGS. The RGB-only baseline cannot utilize TD/SD, so the comparison can only indicate that "the additional high-frequency signals are useful", but it cannot demonstrate that "the proposed simulator and two-stage strategy are the best way to utilize TD/SD". (2) Under high-speed movement, sparse viewpoints, and 30 FPS RGB, the pose estimation of COLMAP may be unstable. The synchronization accuracy between cameras, can also affect 4D reconstruction. (3) How does the model address the issues of minor artifacts and temporal fluctuations?

Questions for authors

See the weakness section.

2 comments

We thank the reviewer for the constructive comments. We address each concern below and highlight the corresponding revisions in the updated manuscript.

W1: We thank the reviewer for this important comment. We agree that additional comparative experiments help demonstrate the effectiveness of our method. Our experiments have already considered two intuitive TD/SD integration strategies:

(1) two-stage reconstruction (2D reconstruction followed by 4DGS). We include CBRDM + 4DGaussians, which first reconstructs intermediate 2D images from RGB and TD/SD signals and then uses them to supervise 4DGaussians. In contrast, our method directly supervises dynamic Gaussians through a differentiable TD/SD simulator, achieving better reconstruction quality on synthetic data. CBRDM requires approximately 26 h 32 min for 2D preprocessing alone, whereas our method completes the entire 4D reconstruction in 1 h 49 min, reducing runtime by 93.2%. The former excludes subsequent 4DGaussians training, further highlighting our efficiency advantage.

(2) Direct joint optimization. We compare direct joint training with our two-stage progressive optimization in the appendix. Both use the same dynamic representation and TD/SD supervision, differing mainly in the optimization procedure. Our strategy reduces Gaussian primitives from 333,069 to 128,676 and model size from 78.78 MB to 30.43 MB, representing approximately 61.4% reductions in both metrics.

These comparisons demonstrate the advantages of our framework in reconstruction quality, computational efficiency, and model compactness under our experimental settings.

W2: We agree that camera pose errors and temporal misalignment may affect high-speed 4D reconstruction. However, these issues mainly concern data acquisition and preprocessing rather than inherent limitations of TD/SD supervision.

Our sparse-view setup is constrained by the available camera number and aims to validate practical feasibility rather than deliberately simulate extreme sparsity. Similar limited-camera settings are common in previous high-speed reconstruction studies [1–3]. We acknowledge that motion blur may degrade COLMAP feature matching and pose estimation. Recent CVS studies demonstrate TD/SD-assisted motion deblurring [4] and extreme high-speed reconstruction [5]. Integrating these techniques could provide clearer RGB images, improving pose estimation and subsequent reconstruction. We discuss this in the appendix’s Limitations section.

For temporal synchronization, we use a flashing light as a common reference and align its recorded responses across cameras, controlling inter-camera TD/SD alignment errors within 1 ms (1/2 TD/SD sampling interval). Details are provided in the appendix section Multi-Camera Temporal Synchronization.

W3: The artifacts and temporal fluctuations discussed in our paper mainly concern pipelines that reconstruct intermediate 2D images before 4DGS optimization.

Learning-based 2D reconstruction may introduce texture artifacts, brightness deviations, and inconsistencies across independently reconstructed frames due to the lack of unified 3D constraints. When these images supervise 4DGS, their errors may propagate into the dynamic scene representation, causing temporal discontinuities or flickering.

In contrast, our method directly optimizes a unified dynamic Gaussian representation using RGB and TD/SD observations. Renderings across timestamps share scene parameters and continuous temporal modeling, providing consistent spatiotemporal constraints rather than relying on independently reconstructed 2D frames. This helps maintain structural and appearance consistency while avoiding error propagation from intermediate 2D reconstruction.

Thus, instead of introducing an additional artifact removal module, our method mitigates these potential artifacts and temporal fluctuations through direct, unified 4DGS optimization.

Refrence:

[1] BulletTime4D: Towards High Spatio-Temporal Resolution Dynamic Scene Rendering via Spike-Guided Stereo Vision[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(4): 2743-2751.

[2] Spike4DGS: Towards High-Speed Dynamic Scene Rendering with 4D Gaussian Splatting via a Spike Camera Array[J]. Advances in Neural Information Processing Systems, 2026, 38: 28707-28727.

[3] E-4dgs: High-fidelity dynamic reconstruction from the multi-view event cameras[C]//Proceedings of the 33rd ACM International Conference on Multimedia. 2025: 7356-7365.

[4] Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2026: 37496-37506.

[5] Diffusion-based extreme high-speed scenes reconstruction with the complementary vision sensor[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 5701-5710.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Responses.

Dear Reviewer btT1,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable feedback.

As the discussion deadline approaches, we would be grateful if you could review our responses and share any further feedback. Please let us know if you have any remaining concerns or questions. We would be happy to provide additional clarification.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_QRVuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper propose a two-stage 4D Gaussian reconstruction framework (Triplex4DGS) tailored for Tianmouc complementary vision sensor and introduce differentiable sensor simulation module. The paper also presents two dynamic scene datasets containing synthetic and real-world sensor data. Triplex4DGS compares in quality to five other methods, outperforming them in only two of the three metrics. Only one of those methods addresses a problem of a high speed motion capture, but uses a different type of sensor.

Critical strengths

Gaussian reconstruction of a high-speed motion captured with novel sensors is an active research problem. High-level approach of direct differentiable supervision from raw sensor data is a clean and sensible approach.

Critical weaknesses

Evaluation strategy doesn't conclusively demonstrate method's performance:

  • Quantitative comparison with other Gaussian reconstruction methods is performed only on synthetic dataset and inconclusive:
    • Synthetic dataset construction has a potential domain gap: main focus of the paper is reconstruction of a high speed motion, but synthetic dataset is based on conventional regular motion video, which might have less blur than true high speed motion video. Authors don't demonstrate that synthetic dataset matches real data.
    • In Table 1, CBRDM+4DGaussians baseline, being one of the closest to the proposed method, shows a quality significantly lower then 4DGaussians alone, which authors attribute it to amplified temporal fluctuation without sufficient analysis.
  • Quantitative comparison of reconstruction quality with other gaussian reconstruction methods on real data is not performed at all. Other methods are only compared qualitatively or by processing time.
Questions for authors

Can you provide quantitative comparison on real data? How much does the observed performance improvement depend on the choice of evaluation frames and scenes?

2 comments

We thank the reviewer for the constructive comments. We address each concern below and highlight the revisions in the manuscript.

W1: Using synthetic datasets to simulate high-speed motion is common in Gaussian Splatting research [1–4], as acquiring real-world high-speed multi-view data and ground-truth (GT) references is challenging. Such datasets enable controlled evaluation of high-frequency visual supervision.

Our goal is to investigate the effectiveness of TD/SD supervision for high-speed dynamic 4D reconstruction. Motion blur depends on both object velocity and exposure time; shorter exposures can mitigate blur even during fast motion. If longer exposures are demanded, the TD/SD-assisted motion deblurring [5] method can provide a sharp RGB frame compatible with our pipeline, while retaining TD/SD signals for spatiotemporal constraints. Thus, blur becomes a secondary challenge with this complementary sensing architecture. We discuss this in the appendix’s Limitations section.

We acknowledge the domain gap between synthetic and real data in sensor responses, noise, and imaging conditions. To validate our difference response simulator, we have generated simulated SD signals from real RGB images and compare them with actual SD measurements at nearby timestamps. As shown in Fig. 7, the simulated signals reproduce the main spatial structures and response characteristics of real SD measurements. Although this does not eliminate the domain gap, it supports the simulator’s ability to approximate real SD responses. Details are provided in the appendix section Validation of the Difference Response Simulator.

Regarding the degradation of CBRDM + 4DGaussians, we believe domain mismatch is a possible explanation. CBRDM was trained on high-speed RGB datasets (GoPro, X4K1000FPS, and SportsSlomo), paired with TD/SD measurements captured using a DMD-based optical setup. In contrast, our N3V-based benchmark uses mathematically simulated TD/SD signals. Differences in sensor responses, noise, hardware effects, scene content, and motion distributions may affect generalization. We use the official pretrained CBRDM weights without fine-tuning, as the relatively small N3V dataset is insufficient for reliable diffusion-model adaptation. Such domain discrepancies may degrade CBRDM’s predictions and subsequent 4DGaussians reconstruction. We clarify this potential explanation in Evaluation Results.

W2/Q1: We thank the reviewer for this suggestion. We have added quantitative comparisons with Ex4DGS and 4DGaussians on two real-world scenes, Flower and Ball, using PSNR, SSIM, and LPIPS. CBRDM is also included as a 2D reconstruction reference. Since our scene contains a substantial static component that contributes little to evaluation, we adopt a region-of-interest (ROI) covering the full motion extent of moving objects. This ROI-based evaluation amplifies the differences in dynamic reconstruction performance of our 4DGS method.

Table 1. Comparison on Flower.

MethodPSNR ↑SSIM ↑LPIPS ↓
Ex4DGS17.77990.43580.3280
4DGaussians15.84670.40990.2954
Ours18.20770.58200.3140
————
CBRDM (2D)19.47130.65850.2567

Table 2. Comparison on Ball.

MethodPSNR ↑SSIM ↑LPIPS ↓
Ex4DGS20.00920.58010.3322
4DGaussians14.85840.46180.2097
Ours20.24240.74110.2649
————
CBRDM (2D)21.52980.78610.2439

The results show that Triplex4DGS achieves the highest PSNR and SSIM among Gaussian-based methods on both scenes, while ranking second in LPIPS. These results support the effectiveness of TD/SD supervision for high-speed dynamic reconstruction.

Although CBRDM achieves higher PSNR and SSIM through diffusion-based 2D reconstruction, it does not directly produce a 4D scene representation and requires subsequent Gaussian-based reconstruction. For a synthetic scene with 20 views and 6,000 TD/SD frames, CBRDM requires 26 h 32 min for 2D preprocessing alone, excluding 4D reconstruction. In contrast, Triplex4DGS completes the entire reconstruction in 1 h 49 min, reducing runtime by 93.2% with at least a 14.6× speedup, demonstrating the efficiency of direct TD/SD-based 4D reconstruction.

We added these results and analyses to the Real-World Implementation section.

Refrence:

[1] Spike4DGS: Towards High-Speed Dynamic Scene Rendering with 4D Gaussian Splatting via a Spike Camera Array. NeurIPS 2026.

[2] Event-Boosted Deformable 3D Gaussians for Dynamic Scene Reconstruction. ICCV 2025.

[3] FastEventDGS: Deformable Gaussian Splatting for Fast Dynamic Scenes from a Single Event Camera. CVPR 2026.

[4] E-4DGS: High-Fidelity Dynamic Reconstruction from the Multi-View Event Cameras. ACM MM 2025.

[5] Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor. CVPR 2026.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Responses.

Dear Reviewer QRVu,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable feedback.

As the discussion deadline approaches, we would be grateful if you could review our responses and share any further feedback. Please let us know if you have any remaining concerns or questions. We would be happy to provide additional clarification.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_un5gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses normal RGB images and high-frequency spatiotemporal differential signals to achieve high-speed dynamic 4D gaussian splatting reconstruction. Although high speed cameras can provide rich high frequency information, they also bring high overhead. To address this, the authors use high-frequency TD/SD signals combined with normal-speed rgb images for reconstruction. The method includes RGB-guided dynamic initialization and differentiable TD/SD simulator, then a progressive optimization is introduced. The authors construct a simulated dataset for evaluation, and also performed experiments in real world.

Critical strengths

The authors provide a new way for high-speed dynamic scene reconstruciton through the fusion of RGB and TD/SD.

The authors construct real-world system and evaluate their method, which improves the practical applicability.

Critical weaknesses

The authors accelerated 30fps data to construct a 750fps sequence. Although this acceleration boosts the speed of the video, it does not produce the effects of high speed, such as motion blur.

Questions for authors

TD and SD improve the reconstruction metrics, can they also provide greater accuracy in 3D geometry and 3D motion trajectories?

In ablation, why RGB is better than RGB + SD? Why does this degradation disappear when TD is added?

2 comments

We thank the reviewer for the constructive comments and address each concern below.

W1: Synthetic high-speed datasets are common in Gaussian Splatting research [1–4], as acquiring real-world high-speed multi-view data with ground truth (GT) is challenging. They enable controlled evaluation of high-frequency supervision.

Our goal is to investigate the effectiveness of TD/SD supervision for high-speed dynamic 4D reconstruction. Motion blur depends on both object velocity and exposure time; shorter exposures can mitigate blur even during fast motion. If longer exposures are demanded, the TD/SD-assisted motion deblurring [5] method can provide a sharp RGB frame compatible with our pipeline, while retaining TD/SD signals for spatiotemporal constraints. Therefore, when adopting this complementary sensing architecture, blur itself becomes a secondary challenge. We discuss this limitation and potential solutions in the appendix’s "Limitations".

Q1: Quantitative evaluation of geometry and motion remains challenging because commonly used 4D reconstruction benchmarks lack ground-truth 3D geometry and motion trajectories. We conduct two additional experiments to assess the contribution of TD/SD supervision to 3D geometry and motion trajectory reconstruction.

(1) 3D Geometry Reconstruction. Without GT depth, we train 4DGS on original complete RGB sequences and use its rendered depth maps as pseudo-GT to evaluate depth consistency using Absolute Relative Error. The results show comparable errors of 0.1436 (w/o TD/SD) and 0.1477 (w/ TD/SD). This may be attributed to our two-stage design: the initial geometry is established from sparse RGB observations, while TD/SD signals provide high-frequency spatiotemporal constraints without introducing additional camera viewpoints. Consequently, their impact on depth accuracy is limited.

(2) 3D Motion Trajectory Evaluation. Since ground-truth 3D trajectories are unavailable, we evaluate motion reconstruction indirectly at the temporal midpoints between consecutive RGB observations, where the temporal distance to the nearest RGB frame is maximized. With TD/SD supervision, PSNR improves by 1.73 dB and SSIM increases from 0.9157 to 0.9256, while LPIPS remains comparable. These results indicate improved intermediate motion-state reconstruction, providing indirect evidence of more accurate 3D motion recovery.

These experiments are added in the appendix section Additional Evaluation of Geometry and Motion Reconstruction.

Q2: The degradation under RGB + SD supervision may arise from photometric ambiguity in spatial differential measurements. Consider a discrete TD/SD sequence sampled at timestamps tkt_k, where k=0,…,Kk=0,\ldots,K. The intensity maps I(x,t0)I(x,t_0) and I(x,tK)I(x,t_K) are aligned with the RGB keyframes RGB0RGB_0 and RGB1RGB_1, respectively.

When only SD signals are used, the intensity at an intermediate timestamp can be expressed as:

I(x,tk)=I(x0,tk)+∫x0x∇I(ξ,tk)⋅dξ.I(x,t_k)=I(x_0,t_k)+\int_{x_0}^{x}\nabla I(\xi,t_k)\cdot d\xi.

However, the reference intensity I(x0,tk)I(x_0,t_k) is unknown for intermediate timestamps k=1,…,K−1k=1,\ldots,K-1. Consequently, spatial differential measurements alone cannot uniquely determine absolute intensity, potentially introducing brightness inconsistencies during optimization. Previous studies have highlighted the importance of absolute brightness constraints for differential measurements [6] and the potential interference between differential brightness supervision and Gaussian color optimization [7].

In contrast, TD signals provide temporal brightness constraints:

I(x,tk)=I(x,t0)+∫t0tk∂I(x,τ)∂τ dτ.I(x,t_k)=I(x,t_0)+\int_{t_0}^{t_k}\frac{\partial I(x,\tau)}{\partial \tau}\,d\tau.

By linking intermediate timestamps to RGB keyframes, TD supervision helps reduce photometric ambiguity and maintain appearance consistency. SD signals can then provide complementary spatial structural constraints with less interference with RGB appearance supervision. Thus, RGB, TD, and SD provide complementary constraints: RGB anchors appearance, TD constrains temporal brightness changes, and SD captures spatial gradients. This may explain why RGB + SD degrades performance, whereas incorporating TD enables more effective utilization of SD information.

Reference:

[1] Spike4DGS: Towards High-Speed Dynamic Scene Rendering with 4D Gaussian Splatting via a Spike Camera Array. NeurIPS, 2025.

[2] Event-Boosted Deformable 3D Gaussians for Dynamic Scene Reconstruction. ICCV, 2025.

[3] FastEventDGS: Deformable Gaussian Splatting for Fast Dynamic Scenes from a Single Event Camera. CVPR, 2026.

[4] E-4DGS: High-Fidelity Dynamic Reconstruction from Multi-View Event Cameras. ACM MM, 2025.

[5] Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor. CVPR, 2026.

[6] Recovering Scene Geometry and Material from Event Streams with 3D Gaussian Splatting. Sensors, 2026.

[7] EdMCGS: Event-Driven Markov Chain Gaussian Splatting for Extreme-Low-Frame-Rate Dynamic Scene Reconstruction. arXiv, 2026.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Responses.

Dear Reviewer un5g,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable feedback.

As the discussion deadline approaches, we would be grateful if you could review our responses and share any further feedback. Please let us know if you have any remaining concerns or questions. We would be happy to provide additional clarification.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.