- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper tackles the problem of feedforward 3dgs compression and studies how to send a 3dgs scene built by a feed forward model to a weak device (which is memory constrained) using as few bits as possible. Sending the input images directly forces the receiver to run the whole feedforward network whereas sending the full set of gaussians is far too costly in terms of the communication bits. The authors main contribution is to send the networks intermediate features instead and they do it in 2 rounds. In the first round the sender compresses the features and decodes them itself just as the receiver would and then it builds the 3d scene from this decoded version and renders it back at the input views so it can compare the renders with the real images. This lets the second round send only what the first round missed and the recevier combines the two decoded pieces and turns them into gaussians with a small network. Across 3 standard scene datasets the method matches or slightly beats prior compression approaches.
Critical strengths
-
The paper frames feed forward 3dgs compression in terms of the split of computation between sender and receiver -> It compares image compression and gaussian compression and feature compression under one setup (Fig 1 and Table 1) and also evaluates zeroshot transfer to ACID. Under this framing the decoder is very lightweight. It has only 3.45M parameters and requires no intermediate rendering since the feedback loop runs entirely at the encoder (Sec 3.3). This is about 13x smaller than the CodecSplat decoder and far smaller than running DepthSplat or ReSplat at the receiver. This is a meaningful advantage in a one2many setting where a single sender serves many resource constrained receivers.
-
The component ablation (Fig 5 left) directly tests the core design choice -> When the second stage extractor receives only the input image without the rendered view it loses about ~0.6 dB PSNR at high rates. This indicates that the rendering feedback carries information beyond what a second encoding pass of the same image provides.
Critical weaknesses
-
The better RD claim holds only partly -> On RE10K and ACID FeCoSplat nearly ties CodecSplat in PSNR and SSIM. On DL3DV it is below CodecSplat at every rate and only FeCoSplat* wins by using CodecSplat's frozen backbone and 150K extra fine-tuning steps. Elsewhere the backbones differ (DepthSplat small vs base). The LPIPS gains may partly come from the LPIPS training loss and there is no BD rate or variance to judge the gain. The claim should be "a much lighter receiver at similar RD" but not "better RD".
-
The novelty of the feedback stage is overstated -> Decoding at the encoder and then coding only what the reconstruction still misses is a long standing idea in learned compression. Toderici et al. (ICLR 2016 and CVPR 2017) already code residuals iteratively with RNNs whose recurrent decoder state is updated by each new bitstream which is essentially the S0 -> S1 -> S2 state with two stages. Progressive codecs such as DPICT (CVPR 2022) and DCVC style conditional coding on decoded context (NeurIPS 2021) follow the same pattern and none of this work is cited.
Questions for authors
-
Please report BD rate in PSNR and LPIPS against CodecSplat on all three datasets and run the CodecSplat backbone variant on RE10K and ACID as well. This would show whether the RD gain holds beyond DL3DV and beyond the backbone change, and the claims in the abstract and Sec 4.3 should then be aligned with the results.
-
Please show the two stage results on top of a strong single-stage model (for example the FeCoSplat* backbone with one stage) so that the feedback is not measured against a weak base.
-
Please discuss the closed-loop residual and conditional coding literature (Toderici et al. 2016 and 2017, DPICT, DCVC) and state clearly what is new relative to it.
5 comments
W1&Q1: Rate--Distortion Performance and Model Comparability (Continued from the Previous Comment).
We also provide a controlled CodecSplat w/ Feedback experiment in Appendix B.2.2. It demonstrates that rendering-based feedback improves BD-PSNR by +0.08 dB on RE10K and +0.19 dB on DL3DV compared with the original CodecSplat, with larger gains under zero-shot settings (+4.02 dB for view-count generalization and +1.01 dB for resolution generalization). This provides additional evidence that the improvements are not solely attributable to differences in feature extraction backbones or training strategies.
Table 2: Effectiveness of rendering-based feedback on a frozen pretrained CodecSplat baseline. Due to the noticeable differences in PSNR between the two methods, we report BD-PSNR instead of BD-Rate, which measures the average PSNR gain at equivalent bitrates. BD-PSNR improvements are measured relative to the original CodecSplat, with higher value indicating a stronger performance.
| Method | RE10K BD-PSNR ↑ | DL3DV BD-PSNR ↑ | DL3DV (2 ctx) BD-PSNR ↑ | DL3DV (256×256) BD-PSNR ↑ | Sender Params (M) ↓ | Receiver Params (M) ↓ | Sender Latency (s) ↓ | Receiver Latency (s) ↓ |
|---|---|---|---|---|---|---|---|---|
| CodecSplat | 0.00 | 0.00 | 0.00 | 0.00 | 181.73 | 24.90 | 0.189 | 0.138 |
| CodecSplat w/ Feedback | +0.08 | +0.19 | +4.02 | +1.01 | 183.98 | 27.18 | 0.273 | 0.195 |
W2&Q3: Novelty of Rendering-Based Feedback.
We thank the reviewer for highlighting the connection to progressive compression. We acknowledge that multi-stage coding and recurrent state updates are established techniques, and have expanded Appendix C.2 to clarify the distinctions between our approach and related methods in progressive image and video compression.
Our contribution lies in task-driven progressive feature compression through rendering-based feedback for 3D scene reconstruction. Unlike conventional progressive coding methods that progressively refine image reconstructions or latent representations toward predefined reconstruction targets, feed-forward 3DGS reconstruction relies on indirect supervision through rendered 2D images, as ground-truth Gaussian primitives are generally unavailable and the learned multi-view features lack predefined reconstruction target. Inspired by this observation, FeCoSplat reconstructs intermediate 3D Gaussians and renders them to guide the extraction of additional scene information, incorporating this rendering-based feedback into the second-stage coding process.
To validate this distinction, we introduce a residual feature coding baseline (Appendix B.2.1, Figure A3, and Table A3), which extends the residual coding strategy of the residual autoencoder mentioned in Toderici et al. (ICLR 2016) [1] to multi-view features in our framework:
Table 3: Comparison of rendering-based feedback and residual feature coding. Due to the noticeable differences in PSNR between the two stages, we report BD-PSNR instead of BD-Rate, which measures the average PSNR gain at equivalent bitrates. BD-PSNR is measured relative to Stage 1 of feedback-guided coding, with higher value indicating a stronger performance.
| Method | Stage 1 BD-PSNR (dB) ↑ | Stage 2 BD-PSNR (dB) ↑ | Sender Params (M) ↓ | Receiver Params (M) ↓ |
|---|---|---|---|---|
| Feedback-Guided Coding (Ours) | 0.00 | +0.74 | 37.28 | 3.45 |
| Residual Feature Coding | -0.31 | -0.22 | 35.49 | 4.46 |
Rendering-based feedback achieves a higher improvement from Stage 1 to Stage 2 compared with residual feature coding.
Together with the feedback-enhanced CodecSplat, these experiments support the effectiveness of rendering-based feedback in progressively improving 3D scene reconstruction, while distinguishing our contribution from conventional progressive feature refinement.
[1] Toderici et al., “Variable Rate Image Compression with Recurrent Neural Networks,” ICLR 2016.
Q2: Effectiveness of Two-Stage Coding with a Strong Backbone.
We agree that feedback should be evaluated on top of a strong single-stage model. We have added stage-wise rate--distortion comparisons for FeCoSplat-B/L, which share CodecSplat's pretrained feature extraction backbone (Appendix B.3, Figure A6). Both variants improve PSNR, SSIM, and LPIPS from Stage 1 to Stage 2 on both context and target views, with similar improvements under zero-shot generalization from eight to two input views (Figure A7). Moreover, the CodecSplat w/ Feedback mentioned above demonstrates that adding rendering-based feedback to a strong pretrained single-stage model yields further improvements.
W1&Q1: Rate--Distortion Performance and Model Comparability.
We thank the reviewer for raising concerns about RD comparisons and backbone differences. To further investigate the impact of model capacity, we conducted additional experiments with three FeCoSplat variants (S, B, and L) at different parameter scales. For each variant, we trained separate models on RE10K and DL3DV for 300K iterations each. We evaluated the RE10K-trained models on ACID without fine-tuning. It should be noted that FeCoSplat-B/L follow the same training strategy as CodecSplat: the pretrained multi-view feature extractor is frozen, while the remaining modules are trained for 300K iterations, consistent with the training schedule specified in the official CodecSplat GitHub repository. This corresponds to the second stage of CodecSplat's two-stage training strategy described in the original paper. Importantly, FeCoSplat-B/L are trained using the pretrained multi-view feature extractor rather than fine-tuned from a fully trained CodecSplat model. We have also added detailed RD results and PSNR-/LPIPS-based BD-Rate comparisons in Appendices B.1 and B.6.
Table 1: Comparison of rate–distortion performance and model complexity. BD-Rate is computed relative to CodecSplat and negative values indicate better compression efficiency.
FeCoSplat-B/L achieve competitive rate–distortion performance compared with CodecSplat while requiring fewer model parameters, demonstrating favorable compression performance across different model capacities. Gaussian-compression baselines are excluded from BD-Rate calculations because their higher operating bitrates provide insufficient RD-curve overlap, making extrapolation unreliable. Nevertheless, the RD curves and complexity comparisons show that FeCoSplat-S achieves better compression efficiency with lower overall model complexity and coding latency than these Gaussian-compression-based pipelines. We have also revised the relevant discussions of our experimental results to better reflect the trade-offs among rate–distortion performance and model complexity, providing a more comprehensive analysis from a rate–distortion–complexity (RDC) tradeoff.
Regarding perceptual supervision, FeCoSplat uses the same pixel-wise and LPIPS reconstruction losses as CodecSplat. Thus, the improvements cannot be attributed solely to differences in training objectives. Furthermore, FeCoSplat demonstrates stronger zero-shot generalization across input-view counts, spatial resolutions, and datasets (Section 4.4, Appendix B.7), supporting its effectiveness beyond standard evaluation settings.