acceptodds
Under review as a conference paper at ICLR 2027

GR4DGS: Sparse-View 4D Gaussian Compression via Temporal Grid Prediction

Abstract

4D Gaussian representations enable free-viewpoint video, but their frame-wise scene updates incur substantial storage and transmission costs. We investigate this compression challenge under sparse-view capture. In our three-view evaluation of a grid-based streaming codec, motion grids account for about 99% of P-frame payloads, although consecutive quantized grids remain highly correlated. Independent per-frame grid coding therefore repeatedly transmits information that is predictable from the decoder's preceding state. We propose GR4DGS to exploit this redundancy through temporal motion-grid prediction. GR4DGS transmits integer differences from the previously decoded grid, recovering the current quantized grid exactly while retaining the other rendering inputs. To further reduce payloads, GR4DGS-E2E combines residual-domain rate optimization with occupancy-aware initialization: unqueried cells inherit the preceding decoded values, while queried cells restart from a shared warm-up. Across six Neural 3D Video scenes of 300 frames each under three-view training, lossless prediction reduces P-frame payloads by 65–72% relative to the default independently coded 4DGC baseline. The separately trained GR4DGS-E2E variant reduces them by 88–93%, with per-scene mean PSNR differences within 0.03 dB. Including the shared uncompressed keyframe once, E2E reduces sequence payloads by 81–86%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Gaussian compression, temporal prediction, sparse-view reconstruction, entropy coding, rate-distortion optimization, free-viewpoint video

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CFNAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies compression of streaming 4D Gaussian (4DGS) representations for free-viewpoint video under sparse-view capture. Building on the 4DGC codec, the authors first measure that motion grids constitute ~99% of P-frame payloads in their three-view reproduction, with consecutive decoded grids correlating at 0.92–0.95 and ~95–96% of integer residuals being zero (§3.2). They then propose GR4DGS, a lossless post-training recoding that transmits integer differences against the previously decoded grid (Δ_t = S_t − S_{t−1}, Eq. 2), achieving bit-exact recovery with 65–72% P-frame payload reduction, plus an optional causal context entropy coder (§4.2). A second variant, GR4DGS-E2E (§4.4), trains in the residual domain with a residual-rate objective (Eq. 5) and occupancy-aware initialization (Eq. 6), reaching 88–93% P-frame reduction with per-scene mean PSNR differences within 0.03 dB. Including the shared uncompressed keyframe, sequence payloads fall by 81–86%. Validation covers six Neural 3D Video scenes (300 frames each) at three/six/nine training views, with ablations on initialization strategies, occupancy pruning, and rate weights (Tables 1–3, 5, 9).

Critical strengths
  1. Well-motivated, reproducible diagnostics. The empirical decomposition (motion grids ≈ 99% of P-frame payload; consecutive-grid correlation 0.92–0.95; 41.5% cell occupancy, Table 8) precisely identifies the dominant compression bottleneck, and the lossless variant is verified bit-exact on every decoded stream with transparent byte accounting (frequency tables, fp32 MLP, keyframe listed separately).

  2. Clean and honest experimental practice. The ablation panel (Table 5: rate-only / plain temporal / occupancy-aware / GOP-10 initialization) cleanly isolates the contribution of residual-domain training, the drift failure modes are disclosed, and limitations (seeking, packet loss, single E2E runs) are stated rather than hidden.

Critical weaknesses
  1. Missing rate–distortion evaluation; the central claim is not established on equal terms. All headline savings are reported against a single default operating point (λ_grid = 10⁻⁵). Table 9 shows that merely tuning the baseline rate weight makes 4DGC competitive or better: on cook spinach, 4DGC at λ_grid = 10⁻⁴ achieves 0.020 MiB/frame at 21.325 dB, versus GR4DGS-E2E's 0.084 MiB/frame at 21.387 dB — 4× fewer bits at essentially equal quality. The authors acknowledge the same phenomenon at six views but provide no R–D curves or BD-rate. It is therefore impossible to separate "our method compresses better" from "the baseline was simply not tuned."

  2. Weak baseline and no comparison or citation of 2025 state-of-the-art methods. The comparison is confined to the 4DGC family and 2024-or-earlier methods (QUEEN, TeTriRF, 3DGStream, GC-4DGS). Directly competing 2025 works are neither compared nor cited. Examples include [1] "Compressing streamable free-viewpoint videos to 0.1 MB per frame" (AAAI 2025), [2] GIFStream (CVPR 2025), [3] Motion matters (NeurIPS 2025), [4] 4D Gaussian videos with motion layering (TOG 2025), and [5] EvolvingGS (SIGGRAPH Asia 2025). The paper never establishes its position relative to the current state of the art, and its absolute bitrate is not shown to be competitive under a unified accounting convention.

Questions for authors
  1. It is recommended that the authors provide sequence-level R–D curves and BD-rate for 4DGC, GR4DGS, and GR4DGS-E2E across 3–4 rate weights, and reconcile Table 9, where tuned 4DGC (0.020 MiB/frame at 21.325 dB on cook spinach) uses 4× fewer bits than GR4DGS-E2E (0.084 MiB/frame at 21.387 dB) at nearly the same PSNR.

  2. It is recommended that the authors compare against the recent works mentioned above under a unified accounting convention (MLP, keyframe, container overhead). If reproducing them is infeasible, it is recommended that they state the specific technical difference that keeps their method competitive.

  3. It is recommended that the authors clarify how sparse-view capture specifically affects the temporal redundancy of motion grids, and why three-view training is the appropriate evaluation protocol rather than a general streaming setting.

  4. It is recommended that the authors discuss keyframe compression, since the uncompressed keyframe is >40% of the E2E sequence payload (Table 2), and state how the claimed sequence-level savings would change.

1 comment

Thank you for these questions.

Q1. How much of the gain survives tuning 4DGC?

A1. Tuning 4DGC closes most of the original gap. E2E still saves about one fifth at the middle weight. 4DGC penalizes the current grid. E2E penalizes its change from the decoder’s previous grid.

Table 10 and Fig. 8 (p. 25) report the three paired cook weights, all using one regenerated warm-up. At 3×10⁻⁵, losslessly recoding tuned 4DGC increased the cook sequence from 35.404 to 37.495 MiB. Training E2E gave 28.557 MiB, saving 19.34% at −0.00134 dB. On sear it saved 23.89% (31.204 to 23.748 MiB, +0.0315 dB). Totals count one keyframe and 299 updates.

The original 0.020 vs. 0.084 MiB/update used 10⁻⁴ for 4DGC and 10⁻⁵ for E2E (Tables 9 and 1, pp. 24 and 7). At a shared 10⁻⁴ we measured 0.0193 vs. 0.0195. E2E was 0.25% larger overall. Table 9 also reports 4DGC at 3×10⁻⁴. Cook PSNR BD-rates vs. 4DGC were −14.84% for E2E and −3.23% for lossless recoding (PCHIP), or −17.70% and −6.52% (linear). The PSNR overlap for E2E is 0.02216 dB. Each setting ran once. Lossless recoding saved 61.46% at the default cook weight but cost bytes at both higher weights.

Q2. Where does GR4DGS stand relative to the five recent codecs?

A2. On three-view cook, E2E used fewer bytes at higher PSNR than iFVC. GIFStream had the smallest stream and highest PSNR in Table 12 (p. 26). Lossless GR4DGS also recodes an existing 4DGC sequence with exact grid recovery.

  • iFVC used 73.203 MiB at 21.1986 dB, including 52.570 MiB for its two motion streams. E2E used 28.557 MiB at 21.3667 dB (60.99% fewer bytes). iFVC totals 73.204 MiB with its 1,408-byte configuration. iFVC, AAAI 2025
  • GIFStream used 20.366 MiB at 21.5148 dB vs. 22.133 MiB at 21.3449 dB for high-compression E2E. Packing each complete sequence, including decoder metadata, with tar/xz preset 6 gave 17.033 vs. 19.081 MiB. GIFStream, CVPR 2025

All methods share the images, cameras and point cloud. Sizes cover all 300 frames and quality covers frames 1–299. We independently decoded one point per external codec, each trained on its own schedule. E2E used saved-model renders and separate grid checks. Native bytes exclude transport framing. We counted E2E’s keyframe, grids, tables, residual attributes and motion MLP, and all five GIFStream groups with metadata.

The remaining comparisons use published work (p. 26). 4DGV codes deformation offsets and reports 34.008 dB at 24.87 MB on cook with 17 training cameras. It cannot be ranked directly against our three-view runs. ComGS propagates motion from sparse keypoints. EvolvingGS deforms Gaussians and adds or removes points. 4DGV, ACM TOG 2025 · ComGS, NeurIPS 2025 · EvolvingGS

Q3. Why study compression with only three input cameras?

A3. A small rig lowers the cost of filming a moving scene from several directions. Its fitted motion field can still be expensive to stream. 4C4D recorded Dyn4Cam with four cameras at a total rig cost below USD 1,500. RF-DeRF discusses motion ambiguity with few views. We use three spaced cameras for depth estimation (one reference, two source views) and a held-out camera for evaluation. On cook we vary the training views on the same recording (§1, p. 1 and Appendix D, p. 19). 4C4D, CVPR 2026 · RF-DeRF, SIGGRAPH 2024

Reducing six views to three lowered default 4DGC’s held-out PSNR from 29.585 to 21.365 dB, yet mean update size stayed near 0.7 MiB (0.723 vs. 0.708, Table 9, p. 24). Motion grids account for about 99% of default three-view update bytes (§3.2, p. 3). Across 298 cook transitions, 94.75%, 95.52% and 95.80% of finest-grid differences were zero at 3, 6 and 9 views. Coding those differences saved 65.0%, 67.1% and 68.6% of update bytes with exact recovery (Table 4, p. 15 and Appendix G, p. 25). Redundancy persisted at all three view counts on cook. The other five scenes were tested only at three views. Over 130 six-view cook updates, default E2E used 0.0653 MiB/update at 29.627 dB vs. tuned 4DGC’s 0.0174 at 29.512 dB (p. 16).

Q4. Does the saving survive keyframe compression?

A4. Yes. Compressing the common keyframe subtracts the same bytes from both totals. Lossless xz (preset 6) reduced it from 17,099,168 to 13,911,640 bytes on cook. At the middle weight, 4DGC and E2E then used 32.364 and 25.518 MiB, increasing the saving from 19.34% to 21.16%. At 10⁻⁴ the totals were (19.037 vs. 19.093 MiB). At the default weight the saving rose from 82.46% to 83.58% (p. 25). Table 2’s 81.9% is from the original batch (p. 8).

Reviewer_hEezOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies compression of the motion grids used in three-view dynamic Gaussian video. Building on 4DGC, GR4DGS encodes the difference between the quantized integer grids of adjacent frames and reconstructs the quantized grid exactly by integer addition. GR4DGS-E2E, referred to below as E2E, further applies a rate penalty to the temporal residual and combines it with occupancy-aware initialization, in which unqueried cells inherit the preceding decoded values while queried cells restart from a shared warm-up. On six scenes from Neural 3D Video, the two variants reduce P-frame payload by 65 to 72% and 88 to 93% relative to the default three-view 4DGC. After the shared keyframe is counted once, the reductions in sequence payload are 58 to 66% and 81 to 86%. The per-scene mean PSNR of the E2E variant differs from the baseline by at most 0.03 dB. The paper also reports a context-coding probe, an initialization ablation, and a partial sweep of the grid rate weight.

Critical strengths
  1. The exact-recovery mechanism is clear, and the default-point comparison is strictly paired. Eq. 2 on p. 4 uses the previously decoded integer grid as the prediction state, so exact recovery follows from integer arithmetic rather than from an additional learned assumption. The authors report decoding every main-result stream and comparing the reconstructed grids bit for bit. Table 1 reports per-frame payloads, and Table 2 reports sequence totals that include each actual shared keyframe once; the two are consistent at the displayed precision. This provides credible support for the practical value of lossless re-encoding at the default operating point. The claim is lossless with respect to the existing quantized representation and the retained rendering inputs, not with respect to the original video.
  2. The diagnostic experiments reveal useful design trade-offs, and negative results are disclosed. Table 7 compares pruning and prediction on the same decoded grids and shows that pruning does not necessarily improve temporal compression. Table 5 reports clear quality degradation for plain temporal initialization in two configurations and provides rate-only and GOP-refresh controls. The appendix also discloses the tuned baseline, the single E2E runs, the future-frame access in preprocessing, and the unfinished context decoder. This transparency helps delimit the claims.
Critical weaknesses
  1. The additional rate-distortion advantage over a tuned 4DGC baseline remains unestablished. In Table 9, varying only λ_grid from 10⁻⁵ to 3 × 10⁻⁵ lowers the cook_spinach payload from 0.708 to 0.069 MiB per frame, with PSNR moving from 21.365 to 21.350 dB and LPIPS from 0.2245 to 0.2248; the main E2E result is 0.084 MiB per frame at 21.387 dB and 0.2243 LPIPS. The E2E result is not strictly dominated, since its quality is slightly higher, but the tuned baseline already achieves about a 90.3% payload reduction, and its payload is lower than that of the main E2E method. For sear_steak, the tuned baseline is similarly close: 0.048 versus 0.047 MiB per frame and 21.895 versus 21.887 dB. Table 5 further shows that the plain E2E variant reaches 0.041 MiB per frame at essentially the same quality on full cook_spinach. The occupancy-aware variant used in the main table costs 0.084 MiB per frame, while the plain variant drifts in flame_salmon_1 and six-view cook_spinach. Without complete rate-distortion curves for both sides, it is not possible to determine whether the training mechanism improves the achievable rate-distortion boundary.
  2. The external comparison is not conducted under a unified protocol. Appendix D discloses that QUEEN's MiDaS depth initialization and depth loss were disabled and that its adaptive per-frame schedule was not enabled. TeTriRF was trained on the first 100 frames, whereas this paper's initialization masks use all 300 frames. The tables also mix actual coded bytes, entropy estimates, and uncompressed model sizes. These adaptations may be reasonable under a shared-prior protocol, and the authors label them clearly, but their effect on the comparison is not quantified. The reported competitiveness against these baselines is therefore not established.
  3. The paper omits two direct and recent related methods. It does not cite or discuss GIFStream (CVPR 2025) or iFVC (AAAI 2025), both of which have substantive official implementations addressing, respectively, temporal compression of dynamic features and rate-distortion optimization of a motion cache. Because both operate on the same problem of compressing dynamic scene updates, their absence leaves the claimed technical boundary and novelty incomplete.
  4. The occupancy-aware training design has not been independently attributed. Integer differencing, causal context coding, and residual-rate optimization all have established precedents, so the necessity of the new hybrid initialization strategy is particularly important for judging the contribution. Appendix B states that plain temporal initialization inherits the grid, MLP, and entropy model, whereas the occupancy-aware variant inherits only unqueried cells and resets the MLP and entropy model. Differences between the two variants therefore cannot be attributed to the occupancy rule alone.
  5. The rate-only control weakens the case for occupancy-aware initialization. Table 5 shows that the large drift largely disappears without occupancy-aware initialization, although the rate-only variant's salmon LPIPS is 0.010 worse than that of the occupancy-aware variant. For ‘flame salmon 1’, the rate-only and occupancy-aware results are 17.768 and 17.830 dB. For six-view cook_spinach, they are 29.643 and 29.627 dB, and the occupancy-aware variant mainly reduces payload. A control that fixes the MLP and entropy-model initialization and varies only the grid inheritance rule is still needed to establish the mechanism's independent benefit. Because each E2E configuration is evaluated only once, differences of 0.01 to 0.03 dB should not be interpreted as reliable quality differences.
Questions for authors
  1. On full cook_spinach, the plain E2E variant uses 0.041 MiB per frame and the occupancy-aware variant uses 0.084 MiB per frame. When the plain variant does not drift, which configuration do the authors recommend as the main result, and what rate-robustness trade-off should the reader use when comparing it with 4DGC?
  2. Appendix B lists three differences between the plain and occupancy-aware variants: grid inheritance, MLP reset, and entropy-model reset. Which factor do the authors consider responsible for the observed drift prevention, and what evidence currently supports that attribution?
  3. Why were QUEEN's depth modules disabled, and how should the reader weigh TeTriRF's shorter temporal access when comparing the reported numbers?
1 comment

Thank you for distinguishing the rate comparison from the question of what prevents drift.

Q1. When would you use plain inheritance, and what gain remains over tuned 4DGC?

A1. We would use plain inheritance on a sequence where it has proved stable. We report occupancy-aware E2E across all six scenes. With that policy, E2E still saves 19.34% on cook and 23.89% on sear against tuned 4DGC at the shared middle weight. Plain inheritance carries the grids and networks forward. This can keep updates small, but an inherited motion error can carry over as well. Occupancy-aware E2E keeps the previous values in unread cells, so their initial difference is zero. Queried cells and both networks start from the shared warm-up. The reference Gaussian positions determine which cells are queried.

On full cook at the default weight, plain inheritance used 0.041 MiB/update and occupancy-aware inheritance used 0.084, both at 21.387 dB to the displayed precision (Table 5, pp. 17–18). On salmon and six-view cook, plain inheritance fell 1.75 and 0.65 dB below its matched 4DGC baseline over 130 updates (p. 17). This is why Table 1 uses the common policy even on cook, where plain is cheaper.

For your sear example, 0.048 vs. 0.047 MiB/update compares tuned 4DGC with default E2E (Tables 9 and 1, pp. 24 and 7). At 3×10⁻⁵ for both, occupancy-aware E2E saved 23.89% on sear (31.204 to 23.748 MiB, +0.0315 dB) and 19.34% on cook (35.404 to 28.557 MiB, −0.00134 dB). At 10⁻⁴ on cook, the sequence totals converged (22.077 vs. 22.133 MiB). All totals include the keyframe and 299 updates. Each setting was trained once (Table 10, p. 25).

Q2. Did the occupancy rule itself prevent drift on salmon?

A2. Resetting the motion MLP and entropy model together improved quality even when we inherited every grid cell. With both networks reset, changing only the grid rule moved PSNR by 0.0173 dB on salmon. Section 5.5 (pp. 8–9) now reports this distinction in the main text. All three runs share a new keyframe, warm-up, cameras, rate weight and 130 updates (Table 11, p. 26).

  • Keep full grid inheritance and change only the two networks’ initialization. Resetting both raised PSNR from 17.4370 to 17.8544 dB (+0.4175 dB), lowered LPIPS from 0.41736 to 0.35334, and reduced the sequence from 32.772 to 31.720 MiB.
  • Keep both networks reset and change only which cells inherit. Full inheritance used 10.32% fewer bytes than occupancy-aware inheritance (31.720 vs. 35.369 MiB). Its PSNR was 0.0173 dB lower, while SSIM and LPIPS were slightly better.

The rate-only run you highlighted also avoided the large drift when all parameters started from the warm-up (Table 5, p. 18). The 1.75 dB loss in Table 5 compares plain inheritance to its historical 4DGC baseline. The +0.4175 dB compares the joint reset with plain inheritance within Table 11. Each setting was trained once. The contribution of either network alone remains open.

Q3. What can the adapted QUEEN and TeTriRF comparisons establish?

A3. QUEEN and 4DGC started from the same point cloud, which required adapting QUEEN. TeTriRF and E2E were scored on the same frames, but E2E’s initialization had access to a longer sequence. Neither result establishes how the methods would rank with their native inputs and schedules.

For QUEEN we disabled MiDaS initialization and depth loss to start from the same fused point cloud as 4DGC. We used fixed epochs because the adaptive schedule requires precomputed frame differences. E2E had lower LPIPS on all six scenes under these settings. QUEEN’s size is an entropy estimate excluding frame 0. The effect of restoring its native settings is unmeasured (Table 1, p. 7 and Appendix D, p. 20). QUEEN, NeurIPS 2024

TeTriRF trained on 100 frames, while our initialization used masks from all 300. Both were scored on frames 1–99. E2E’s LPIPS was 0.12–0.18 lower under that protocol. We use LPIPS because the PSNR aggregation differs. TeTriRF’s video bytes exclude its group MLPs (about 0.005 MiB/frame, Table 6, p. 21). TeTriRF, CVPR 2024

For a direct comparison of coded bytes, Table 12 (p. 26) uses the same three-view cook inputs, evaluation frames and metrics. Against iFVC’s per-update motion cache, middle-weight E2E used 60.99% fewer native bytes at +0.1681 dB. GIFStream had higher PSNR and a smaller package than high-compression E2E (17.033 vs. 19.081 MiB with identical whole-sequence tar/xz packing). We independently decoded one operating point for each external codec. iFVC, AAAI 2025 · GIFStream, CVPR 2025

Reviewer_4bXfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GR4DGS for streamable 4D Gaussian compression. It exploits temporal redundancy in quantized motion grids by encoding integer differences from the previously decoded grid. The post-training variant reconstructs the original quantized grids exactly while retaining the other rendering inputs. A separately trained E2E variant combines residual-domain rate optimization with occupancy-aware initialization.

Critical strengths
  1. The method is simple, well motivated, and practically useful. The analysis identifies substantial redundancy in the motion-grid payload, and the proposed prediction directly addresses it without changing the underlying representation. Exact recovery of the existing quantized grids is particularly appealing, as it provides compression savings without retraining or introducing additional distortion into those grids.
  2. The reported savings are substantial in the evaluated setting. Across six three-view scenes, lossless prediction reduces P-frame payloads by 65–72% relative to the default independently coded 4DGC baseline. The separately trained E2E variant reports 88–93% reductions, with per-scene mean PSNR differences within 0.03 dB. These P-frame results should be distinguished from whole-sequence savings, which also include the shared keyframe. The occupancy-aware initialization also has a reasonable motivation: preserve unqueried cells while allowing queried cells to adapt without inheriting the entire previous motion state.
Critical weaknesses
  1. The relationship to existing predictive compression methods could be clarified. Temporal differencing is established, and prior work such as “Rate-aware Compression for NeRF-based Volumetric Video” already combines prediction from decoded feature grids with residual-domain rate optimization. The contribution here appears more specifically to concern quantized Gaussian motion grids, exact post-training recoding, and occupancy-aware initialization. An explicit comparison of these mechanisms would make the novelty boundary clearer. I do not view the use of established predictive-coding principles as a problem by itself.
  2. The emphasis on sparse views seems narrower than the method itself requires. The six- and nine-view experiments suggest that prediction remains effective with more input cameras, while the six-view reconstruction improves substantially over the three-view result. However, these additional view-count experiments focus on cook_spinach. Broader cross-scene validation would help establish how generally the observed redundancy and compression gains hold, particularly when more training views are available.

Overall assessment

Weak accept. I find the method reasonable and its lossless compression capability practically valuable. The core idea builds on established predictive coding, but the concrete implementation and measured gains constitute a useful contribution. Clearer positioning and broader view-count validation would strengthen the paper.

Questions for authors
  1. Which properties of the proposed method specifically depend on sparse-view capture, beyond the chosen evaluation setting?
  2. Can the authors clarify the distinction from previous feature-grid residual compression and provide additional higher-view-count results across scenes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.