acceptodds
Under review as a conference paper at ICLR 2027

What RoR Can Write under Privileged Supervision: Decoder Expressivity for Continuous-Time 3D Gaussians

Abstract

Open-loop continuous-time 3D Gaussian extrapolators forecast well under continuous observation, yet under intermittent reveals they still need a correctable write of off-screen geometric change. We run a controlled write-path diagnostic on NVFi Visible–Disappear–Reappear: GT-render reveals, constant-rate hidden change (translation or centroid rotation), and dense 3D attribute supervision—isolating which components of an image-space Reconcile-on-Reobservation (RoR) stack can record the change, not claiming a transferable real-camera world model. A co-evolving latent drift-rate is inert; a frozen translation-only head fails under in-place rotation (fan). Unfreezing the Stage-2 decoder so RoR can express quaternion updates accounts for most of the rotational sign flip vs. an honest static-memory baseline ( RoR-only at occ10 over training seeds); gated typed on-manifold residuals add only a small, seed-sensitive margin at this operating point (typed ), but that margin triples as the hidden rotation grows along either occlusion length or rotation rate ( pp), so the typed channel is operating-point dependent rather than negligible. The same residual family specialized to translation remains useful for relocation ( inj-seed mean on bat @ occ10), and adding the SO(3) channel converts fan's relocation deficit from to , confirming that its residual error there is rotational. Photometric-only finetuning (state weight ) collapses the flip ( at occ10), and freezing the reconcile path localizes that collapse to a single operator while recovering a positive write (); a weak state weight () suffices to learn the write from scratch with no privileged initialization ( vs. for full supervision), so image-only calibration of the reveal reconcile is the specific open problem, not privilege in the attribute signal.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
continuous-time 3D Gaussian Splatting, Reconcile-on-Reobservation, world model memory, SE(3) residual write, decoder expressivity, privileged supervision diagnostic, occlusion, NVFi

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1E7cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how continuous-time 3D Gaussian models can incorporate newly observed geometric changes after periods without observations. It constructs a controlled Visible–Disappear–Reappear diagnostic using rendered observations, synthetic geometric perturbations, and dense Gaussian-attribute supervision. The model combines a continuous-time latent prior, an image-conditioned reconciliation operator (RoR), and optional typed translation, rotation, and scale residual rates.

Critical strengths

The paper asks a useful diagnostic question and evaluates geometric errors rather than relying only on image quality. It reports negative results and acknowledges the limitations of its controlled setting. The main rotation experiment includes useful component ablations.

Critical weaknesses
  1. The paper writing and presentation are significantly poor. The text is repetitive and terminology-heavy. It is hard to track the initialization, losses, and trainable components for each experiment..
  2. No technical contributions. The main causal claim is not fully supported. The results show that decoder fine-tuning helps, but do not establish that limited decoder expressivity is the cause of failure. A frozen decoder may still produce different quaternions when its latent input changes. Moreover, the RoR-only result in Table 5 uses a checkpoint trained with typed residuals, rather than a separately trained RoR-only model.
  3. The main closed-loop evaluation lacks comparisons with relevant existing methods. Most comparisons are against the authors’ own variants or a static-memory baseline. The open-loop comparisons in Tables 12–13 do not establish the advantage of the proposed reconciliation mechanism. The authors should evaluate relevant prior methods under matched observation and supervision conditions, or explain why they cannot be meaningfully adapted to this setting.
Questions for authors
  1. What prevents the frozen decoder from producing useful quaternion corrections, beyond difficulty optimizing the latent update?
  2. Which modules and losses are active in every row of Tables 4–5?
  3. How does the method compare with simple extrapolation from the same reconciled observations?
1 comment

Thank you for a critical and precise review. Your three substantive points, the unsupported decoder claim, the RoR-only control and the missing closed-loop comparisons, were correct. We ran the experiments they called for; the general response summarizes the outcome. Point by point:

W2 / Q1 — The decoder-expressivity claim; what prevents the frozen decoder from producing useful quaternion corrections. You were right, and the answer is that nothing does. With the decoder frozen, directly optimizing the latent at each reveal against the ground-truth state (an oracle, 1000 steps) reaches 0.44 rad geodesic error, below static memory (0.55) and below learned RoR (≈0.50). Unfreezing lowers the oracle to 0.21–0.24 rad in all three seeds and the linear residual of the quaternion Jacobian from 0.54 to 0.38 (Appendix H). Both decoders can represent much better states than RoR reaches, so the bottleneck is the latent update, not decoder expressivity. We withdrew the claim. Appendix I adds a related diagnosis: photometric-only training de-calibrates the correction operator itself (anchor error 0.55 → 0.99 rad).

W2 — RoR-only reused a typed checkpoint. We trained an independent RoR-only model (same initialization, epochs and loss, no rate extrapolation; 3 training × 3 injection seeds). It reaches +5.0/+8.7% on fan rotation, better than the learned residual rate (+1.7/+4.7%, Table 5). The typed residual is therefore no longer a contribution.

W3 / Q3 — Comparison with existing methods and with simple extrapolation from the same reconciled observations.

Simple extrapolation from the same anchors: per-Gaussian and rigid CV, correction CV, the RoR rollout and a learned rate all start from the same RoR anchors (Sec. 4.3). RoR+CV is the most accurate state predictor and our state operating point (Table 1). Prior continuous-time predictors under matched observations: latent ODE (ODE-GS style), CfC, GRU and cubic spline, plain and velocity-augmented, trained on the same frozen Stage-1 and split and re-encoded from our RoR anchor at every reveal, so observations and anchors are identical. RoR+CV has a lower mean state error than the best of them on every NVFi scene and occlusion length; the intervals exclude zero on 3 of 6 (Table 3). Ground-truth re-encoding is far better than RoR re-encoding (e.g. latent ODE +53% vs. +8% on bat), so with any of these dynamics models the reveal-time state estimate is the bottleneck (Appendix E). Test-time photometric fitting: a learning-free tracker refits 8 rigid SE(3) parts to each reveal image, close in spirit to Dynamic 3D Gaussians. It renders well but has state error 8–824% above static memory, because single-view rigid fits slide along textureless surfaces (Sec. 5.3). Not adapted: physics-based forecasters (NVFi, FreeGave, TRACE) fit per-scene physical parameters; re-optimizing them photometrically at every reveal amounts to the tracker above (Sec. 2). The open-loop HyperNeRF tables now only check the backbone, and the text says so (Appendix L). W1 — Writing; initialization, losses and trainable components per experiment. We rewrote the paper around one question (what carries closed-loop accuracy) and removed repeated claims. Table 7 lists every trained model with initialization, trainable modules, losses, data and epochs, followed by a terminology paragraph. Sec. 3 defines the protocol and metrics in one place.

Q2 — Active modules and losses in each row of Tables 4–5 (submission). Those tables were replaced. In the revision every trained model is a row of Table 7. The supervision table (Table 14) states in its caption that all rows finetune the same rotation model and differ only in w_state and w_img. The rotation table (Table 5) uses one rotation model, with predictors that are inference-only.

Technical contribution. We accept that the submission's mechanism claims did not survive. What the revision contributes is a closed-loop state estimator whose anchors make simple extrapolation more accurate than published continuous-time predictors given the same observations; a demonstration that the anchor, not the extrapolator, carries the gain; and a protocol with paired no-change controls and intervals showing where image and state metrics disagree. We would be grateful if you would assess the revised paper on these terms.

Reviewer_z2KbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates geometric state updates in 3D Gaussian models under intermittent observations. It introduces a reconcile-on-reobservation (RoR) mechanism and typed residual memory to update and predict geometric states. Experiments on controlled dynamic scenes analyze the effects of decoder expressivity, residual memory, and privileged supervision.

Critical strengths

The paper addresses an interesting problem of updating geometric states under intermittent observations. The experiments and ablation studies explain the roles of different components. The authors also provide a relatively detailed analysis of failure modes and acknowledge the limitations of the proposed approach.

Critical weaknesses
  1. Insufficient evaluations and baselines. The experiments are limited to 4 NVFi scenes with synthetically injected constant-rate translation and rotation. Although such controlled settings help isolate specific mechanisms, the paper provides insufficient justification for focusing on these simple motions, without evaluating more complex dynamics. Moreover, these motions can potentially be handled by basic geometric extrapolation, yet comparisons with constant-velocity or constant-angular-velocity predictors are missing.

  2. Limited contribution of the proposed module. The experiments suggest that most of the improvement comes from unfreezing the decoder rather than the proposed typed residual memory. On the main rotation benchmark, it provides limited improvement over RoR-only on average.

  3. Limited improvements in rendering quality. Although the paper reports improvements in geometric state estimation, the gains in rendering quality remain relatively limited in some scenes. For example, the PSNR improvements over Static Memory are only 0.43 dB on falling ball and 0.53 dB on fan (Table 9).

Questions for authors
  1. The proposed framework is built upon Deformable 3DGS. Could the authors elaborate on the motivation behind this choice? Do the authors see potential for extending the framework to frontier 4D feed-forward foundation models, such as VGGT-Ω?

  2. The paper reports significant performance degradation under photometric-only supervision. Could the authors provide more failure cases and further discuss the underlying causes? Is there potential to achieve reliable state updates without relying on privileged 3D supervision? Because it is hard to get accurate 3D supervision in the real world.

  3. Could the authors provide more details on the training configuration and computational resources, including GPU specifications, training time, and memory requirements? These details would help assess the reproducibility of the proposed method.

1 comment

Thank you for the review. Your two central points, the missing constant-velocity baseline and the limited contribution of the typed residual, were both correct. Addressing them changed the paper; the general response summarizes how. Point by point:

W1b — Constant-velocity / constant-angular-velocity baselines. We added per-Gaussian CV (position and quaternion, Sec. 4.3), a rigid variant (centroid velocity plus one Kabsch rotation rate) and correction CV, each run from three state sources. The result is the main finding of the revision. From RoR anchors, CV is the most accurate state predictor we tested, reducing static-memory error by 14–28% on bat/whale and 8–21% on D-NeRF (Table 1). From raw context frames it is 13–351% worse than static memory, and from a learning-free photometric tracker 8–824% worse (Sec. 5.3). Simple extrapolation is competitive only once the reveal has been reconciled by RoR, and RoR+CV is now our state operating point.

W1a — Only constant-rate translation and rotation. Every result is now averaged over four drift profiles (constant, accelerating, varying-rate, oscillating out and back), three injection seeds and two occlusion lengths, on two datasets (NVFi and D-NeRF). Table 12 breaks the results down by profile. CV from anchors is best for monotone change (e.g. bat accelerating +47.3%), but under oscillation at L=10 every extrapolator is worse than static memory on bat/whale (CV −3.0/−9.3%) while the RoR rollout stays positive. We report this as a limitation (Sec. 6). We keep geometric changes because they make the ground-truth state error measurable, and Sec. 6 states that the change is synthetic and geometric.

W2 — Most of the gain comes from unfreezing the decoder, not typed residuals. Agreed, and the new experiments go further. An independently trained RoR-only model reaches +5.0/+8.7% on fan rotation, against +1.7/+4.7% for the learned residual rate (Table 5, Appendix H), so the typed rate adds nothing under rotation. A decoder oracle (Appendix H) shows that even the frozen decoder can represent better states than RoR reaches, so decoder expressivity was not the cause either. We withdrew both claims. The contribution is now the RoR anchor as a closed-loop state estimate, with simple predictors on top, compared against published continuous-time predictors re-encoded from the same anchors (Table 3).

W3 — Limited rendering gains (fallingball +0.43 dB, fan +0.53 dB). Two changes. First, photometric selection among predictors (RoR-Sel) raises fan to 26.66/26.17 dB against 23.37/22.60 for static memory and +3.3/+3.6 dB over every re-encoded continuous-time predictor (Tables 2–3). Inside the dynamic-target mask the fan gain is 4.1 dB (Table 16). Second, we state where gains stay small: on fallingball RoR-Sel is within 0.3 dB of static memory, and it is tied with the re-encoded spline on bat, whale and fallingball; dynamic-region PSNR shows this is not just background dilution. An optional reveal-time refinement matches or exceeds a learning-free tracker's PSNR in all 12 held-out settings (Table 6), at a reported state cost.

Q1 — Why Deformable 3DGS; extension to 4D feed-forward models (e.g. VGGT-Ω). Deformable 3DGS gives explicit per-Gaussian trajectories (dense state targets, measurable state error) and is the frozen Stage-1 of ODE-GS, so published predictors run on the same Gaussians. RoR needs only a decodable Gaussian state and a renderer, so a feed-forward 4D reconstructor could replace Stage-1 or supply pseudo-3D targets; our noisy-target result (Sec. 5.8) suggests this is tolerable, but it is untested (Sec. 6).

Q2 — Photometric-only failure cases; avoiding privileged 3D supervision. Appendix I diagnoses the failure. Under photometric-only finetuning the static-memory error, which uses the corrected anchors, rises from 0.55 to 0.99 rad: image gradients de-calibrate the correction operator itself. Freezing the correction restores the anchors exactly but leaves the rate write under-calibrated at every occlusion length. Upweighting changed pixels 20× makes both variants worse, so the failure is not background dilution. On fan, rotations that differ in state render almost alike (Sec. 5.5), so the image does not determine the state. Toward weaker supervision, RoR trained with noisy (5%/10°) or 10%-sparse targets stays within 2.1 points of clean supervision (Sec. 5.8). These targets still come from ground truth, and we do not claim an image-only estimator.

Q3 — Training configuration and compute. Table 7 lists every trained model; Table 8 gives times and memory on NVIDIA A40 (46 GB). Stage-1 14–71 min; backbone ≤75 min, 8.4 GB; RoR head 8–13 min, ≤1.3 GiB; rotation/RoR-only models 13–15 min, about 2 GiB; one evaluation cell about 2 min (<14 GB; fallingball about 28 GB). Scripts and the table generator are in the code supplement.

We would be grateful if you would reconsider your assessment in light of the new baselines.

Reviewer_HqRXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how continuous-time 3D Gaussian models can record geometric changes that occur while a target is unobserved and use a subsequent observation to improve later predictions. It introduces a controlled Visible-Disappear-Reappear diagnostic with ground-truth-rendered observations, injected translation or centroid rotation, and dense 3D attribute supervision. The analysis compares latent-rate updates, decoder adaptation, and typed residual updates against a static-memory baseline that shares the same corrections at reobservation. On the tested NVFi scenes, decoder adaptation enables most of the rotational improvement; typed residuals provide a smaller, setting-dependent additional gain. Photometric-only and initialization controls further characterize the supervision requirements. The HyperNeRF results evaluate open-loop extrapolation rather than the central closed-loop mechanism.

Critical strengths
  1. Useful diagnostic and baseline. The paper separates correction when the object becomes visible again from prediction while it remains hidden. Since the static-memory baseline receives the same corrections, the comparison helps assess the benefit of dynamic prediction beyond reconstruction quality alone.
  2. Useful controls and transparent reporting. The paper reports failed extensions and small gains that vary across random seeds. The photometric-only, freezing, and from-scratch controls help explain the method’s strengths and limitations within its restricted, privileged setting.
Critical weaknesses
  1. The paper already includes comparisons such as RoR-only, ungated residuals, and gated residuals. Please clarify which modules are retained or disabled in each configuration and whether they share the same dynamics prior and observation-correction procedure. Please also explain how residual rates are used to predict subsequent states when gating is disabled.

  2. The visualizations in Figure 2 use geometric localization to protect the static background, whereas the quantitative results in Table 9 use an unlocalized configuration. Table 10 also shows that localization affects image quality and background-related metrics differently. Please describe the localization method and the information it requires, and provide visualizations using the same configuration as the quantitative experiments.

Questions for authors
  1. Table 4 and the from-scratch experiments show that effective updates can still be learned with a lower weight on the 3D state loss, which is a valuable finding. However, reducing the loss weight does not reduce the available 3D supervision. Please clarify which conclusions these experiments support and avoid overinterpretation.

  2. The main closed-loop experiments cover four NVFi scenes. Although HyperNeRF results are provided, they evaluate only open-loop prediction. A representative closed-loop experiment on another dataset would help assess whether the main findings depend on the current data and perturbation protocol.

  3. The relative improvements in Tables 4 and 7 show small fluctuations. Please replace the relevant claims of “monotonic increase” with “an overall increasing trend” to remain consistent with the reported data.

  4. The meaning and units of the rotation parameter need clarification. The text calls it a “rotation rate,” but the tables report it as an angle. Please clarify whether it represents a total rotation angle or an angular velocity, and use consistent terminology and units throughout.

1 comment

Thank you for the careful review and for recognizing the value of the shared-anchor static-memory baseline and the controls. Following the reviews we ran a broader closed-loop study; the general response summarizes how it changed the paper. Point by point:

W1 — Which modules are retained in each configuration; how residual rates predict when gating is disabled. Table 7 (Appendix A) now lists every trained model with its initialization, trainable modules, loss, data and epochs. All RoR variants keep Stage-1 and the CfC prior frozen and share the same backbone and the same correction operator (Eq. 1); predictors, selection, the twist and the tracker have no trained parameters and act on the same anchors. Sec. 4.3 gives the rate predictor explicitly: the correction is converted into rates u_p = (x⁺ − x⁻)/τ and u_ω = Log(q⁺ q⁻⁻¹)/τ, scaled by learned per-Gaussian gates γ ∈ (0,1) and added to the prior rollout, x̂_t = x_t^prior + γ_p u_p (t − t_k). The ungated variant fixes γ ≡ 1, i.e. the full rate is added. The RoR-only control is now an independently trained model (same initialization, epochs and loss), not the typed checkpoint with typed channels disabled.

W2 — Localization in Figure 2 vs. the unlocalized Table 9. Thank you for catching this inconsistency. Localization selected a tight core of the target by Stage-1 path length, which in the submission was computed over the full horizon including future frames, i.e. privileged information. The revision does not use localization anywhere: all figures, including the qualitative Figure 3, are rendered with exactly the configuration of the quantitative tables (Appendix K). For completeness: selecting the core from context frames only gives the same state results as the full-horizon version (fallingball +8.7 vs. +9.2% at L=6, +23.9 vs. +24.6% at L=10) but lowers PSNR from 19.7 to 17.7 dB, which is why we dropped it.

Q1 — Lower state-loss weight does not reduce available 3D supervision. Agreed. Appendix I now states that every row still sees full 3D targets, so Table 14 only shows that the correction can be learned with a smaller state weight. To test reduced supervision instead, Sec. 5.8 retrains RoR from the frozen backbone with noisy targets (3%/5° and 5%/10°, half per-window bias, half per-frame jitter) or with only a fixed 10% subset of supervised Gaussians. At L=10 the learned rate stays within 2.1 points of clean supervision on bat and whale and does not degrade on fan rotation. We state explicitly that this supports robustness to imprecise targets, not training without 3D targets, and the limitations section says we do not claim an image-only estimator.

Q2 — Closed-loop experiment on another dataset. The closed loop now runs on D-NeRF (bouncingballs, jumpingjacks; every third frame, giving the same 2–3 reveals per sequence as NVFi) with the same protocol. RoR+CV reduces static-memory error by 8–21% there and is best in three of four columns (Table 1); it is 17–84 points ahead of the re-encoded spline (Table 3). We report two limitations: the available D-NeRF latent-ODE/CfC/GRU checkpoints work in their native evaluation but fail inside our closed loop even when re-encoded from ground truth, so we report them as n/a (Appendix E), and on real HyperNeRF captures we only test the open-loop backbone (Appendix L).

Q3 — "Monotonic increase". Thank you. The revision contains no monotonicity claim; the tables they referred to were replaced, and the remaining occlusion-length results (Table 14) are described without trend language.

Q4 — Meaning and units of the rotation parameter. It is the total angle applied over the prediction horizon, not a rate. Sec. 3 now reads: in-place rotation rotates the target about its centroid by a total angle of π/2 rad over the horizon; the average angular rate is this angle divided by the horizon length. Table 7 uses the same wording.

We hope the clarified configurations, the unlocalized figures and the new D-NeRF closed loop address your concerns, and we would be glad to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.