acceptodds
Under review as a conference paper at ICLR 2027

Motion Is Not Projection: View-Consistent Attribution in Dynamic Gaussian Splatting

Abstract

Dynamic novel-view synthesis reconstructs time-varying scenes from videos and renders novel views at unobserved viewpoints. Existing Gaussian optimization entangles geometry with training-view information, rendering them non-identifiable from 2D projection errors alone. In this paper, we investigate a fundamental question: can image-space errors be attributed to 3D Gaussian geometry in an identifiable and viewpoint-invariant manner? We first prove that position and shape respond differently to viewpoint changes, inducing view-dependent projection bias that can be compensated by normal-direction shrinkage. Subsequently, we propose MoProGS (Motion-Projection Factorized Gaussian Splatting), which factorizes dynamic deformation from view-dependent projection. Specifically, we separately model the geometric responses of Gaussian position and covariance, and derive a geometry-view coupling factor to boundedly adjust the projected support before rasterization. This balances the relative contributions of geometric parameters to forward projection and backward gradients, yielding viewpoint-consistent geometric error attribution. Experiments on D-NeRF, HyperNeRF, and Neu3D show that MoProGS consistently improves novel-view synthesis quality. Controlled studies further verify the distinct projection sensitivities of position and shape parameters, and clarify the roles of camera conditioning and individual components.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Dynamic scene reconstruction.4d gaussian splatting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tGYSOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper shows that changes in a Gaussian's position and normal-direction thickness have a greater impact on the rendered image at larger viewing angles, with the effect of thickness increasing more rapidly. In monocular video reconstruction, this imbalance can contribute to view-dependent reconstruction errors being incorrectly learned as geometry deformation over time. MoProGS mitigates this issue by reducing deformed Gaussians' normal-direction thickness at grazing incidence. Controlled studies validate the analysis.

Critical strengths
  1. The paper presents an interesting view-dependent approach supported by theoretical analysis.
  2. Controlled experiments show that the proposed method reduces view-specific fitting in the learned deformation.
Critical weaknesses
  1. The experimental evaluation could be further strengthened to better support the paper's claims (see Q1 and Q2 below).
    • I highly recommend including video comparisons for a more comprehensive evaluation.
Questions for authors
  1. Could the authors compare their method with learned view-dependent splatting kernels [1], which pursue a similar goal, in the setting of static scene reconstruction without a deformation network?
    • Since the fundamental question (L16–18) is not limited to dynamic scenes, excluding the deformation network would clarify the effect of the proposed approach.
  2. Could the authors quantify whether the adapters actually reduce coupling between the position and covariance branches, and show that the improvement comes from the reduced coupling rather than the increased model capacity?
  3. Could the authors clarify the Neu3D evaluation protocol and results?
    • Why is the flame salmon scene excluded, unlike evaluations in prior work such as 4D-GS?
    • PSNR and SSIM improve marginally from 32.18 to 32.27 dB and from 0.946 to 0.947, while LPIPS improves substantially, from 0.146 to 0.089, compared to FAGS. How can this discrepancy be explained?
      • Could it be explained by the different number of Gaussians?
  4. Could the authors extend Table 12 and move it to the main paper?
    • The table supports the camera–time leakage problem (L043–045). Since prior methods already use camera-blind deformation (L162), I suggest moving Fig. 6 to the appendix and placing Table 12 in the introduction (please cite the table in the text as well).
    • Moreover, the table does not directly show whether the proposed method mitigates leakage under the same experimental setting. Extending Table 12 such as follows would better support the paper's main contribution. |Method|Time|Assigned|Recombined|Held-out view| |-|-|-|-|-| |w/o view-dependent approach|shared|32.980|-|xx.xxx| |w/o view-dependent approach|unique|34.117|31.214|xx.xxx| |w/ view-dependent approach|shared|xx.xxx|-|xx.xxx| |w/ view-dependent approach|unique|xx.xxx|xx.xxx|xx.xxx|
    • This would help readers better understand the paper's motivation and contribution.

[1] Ding et al. "Learning View-Dependent Splatting Kernels." SIGGRAPH 2026.

7 comments

We thank the reviewer for the positive assessment. Following the suggestions, we extended Tab. 12, measured the coupling between the two branches and unified the LPIPS protocol.

W1/Q1: Compare with learned view-dependent splatting kernels [1] in static scene reconstruction without a deformation network.

We agree that this comparison isolates the operator, and we are running it.

  • Analysis. The projection effect itself does not need a deformation network. On our controlled sheet, which has no time variable, current-incidence support changes the slope of the center-to-support ratio from −1.534 to −0.007. A constant contraction leaves it at −1.515 (Fig. 4, Tab. 7). In a static multi-view scene no temporal field can absorb a view-dependent residual. We therefore expect the attribution to become consistent there as well, with a smaller PSNR gain than under monocular dynamic capture.

  • Relation to [1]. [1] learns the kernel profile, decoded per view into a radially symmetric 2D kernel. MoProGS keeps the Gaussian kernel and changes only the extent along the normal axis, by an analytic factor of the current incidence. [1] makes every primitive more expressive and targets reconstruction quality. Our operator adds one scalar per scene and targets the attribution, so the two are complementary. We will add this discussion to Sec. 2.

  • Experiment. We apply our support operator to static 3DGS without any deformation network, on the benchmarks used in [1]. We will report PSNR next to the numbers of [1], together with the within-Gaussian attribution slope of Tab. 9 with the support on and off. We will post the results here as soon as the runs finish and add them to the paper.

W1/Q2: Do the adapters actually reduce coupling between the two branches? Does the improvement come from reduced coupling rather than increased capacity?

Yes to both. The adapters lower the coupling toward the chance level, and at equal capacity PSNR follows the coupling (see Tab. R6 below).

Table R6. Coupling between the position and covariance branches on Neu3D. Cook Spinach and Cut Roasted Beef, two seeds each. Chance is the level of random directions.

VariantAdded paramsCoupling, median absolute cosineAbove chance
No adapters00.085–0.0890.014–0.018
Independent adapters, ours33,0240.075–0.0800.004–0.009
Shared adapters, same parameters33,0240.21–0.250.14–0.18
Random directions, for referencen/a0.0710
  • Coupling. We measure the median absolute cosine between the sensitivities of the position and covariance outputs to the shared temporal feature. Random directions give 0.071. Without adapters the coupling is 0.014 to 0.018 above this level, and with independent adapters 0.004 to 0.009. Sharing the same adapters raises it to 0.14 to 0.18.

  • Coupling or capacity. Sharing the same adapters between the two branches keeps the 33,024 parameters and the function class, and changes only the coupling. PSNR then drops by 0.37 ± 0.13 dB, in all four paired runs.

  • D-NeRF. Here the backbone ties both branches to one shared weight, and the decomposition unties them. The correlation between the two weights then falls from 1 to 0.58 on Hook. This is where Tab. 4 measures the 0.20 dB. A shared response with the same 2,570 extra parameters and depth stays 0.12 ± 0.03 dB below independent weights on Hook (Tab. R2 in our response to Reviewer 6q62).

  • Leakage test. In Tab. R7 the full model reaches a higher held-out PSNR with a lower training PSNR, which added capacity would not produce.

W2: Video comparisons are highly recommended.

We are preparing side-by-side videos on D-NeRF, HyperNeRF and Neu3D, including cameras that are never used in training. We will upload them to the supplementary material as soon as they are ready.

Q3: Clarify the Neu3D protocol and results. Why is Flame Salmon excluded? Why does LPIPS improve much more than PSNR and SSIM? Could it be the number of Gaussians?

  • Protocol. cam00 is held out for all 300 frames. We train on the remaining cameras at 1352 × 1014 with the released schedule.

  • Flame Salmon. The baseline rows of Tab. 2 are taken from FAGS, which reports the other five scenes. Flame Salmon is also the only sequence with 1,200 frames. We are training MoProGS on its first 300 frames under the same protocol and will post the result here.

  • LPIPS gap. It was a protocol mismatch on our side. Our LPIPS used AlexNet with inputs in [−1, 1], while the baseline rows come from a code base that evaluates with VGG on inputs in [0, 1]. Under one protocol MoProGS obtains 0.134 against 0.146 for FAGS, which is 8% lower (Tab. R4 in our response to Reviewer crMu). The LPIPS margin is then in line with the PSNR and SSIM margins. All three are smaller on Neu3D than on the monocular benchmarks, as the analysis predicts when many synchronized cameras constrain every state.

  • Number of Gaussians. It does not explain the gap, which came from the protocol. MoProGS leaves densification and pruning unchanged and uses 210k to 330k Gaussians per scene on Neu3D. Our FAGS re-implementation uses 220k to 340k on the same scenes.

Q4: Extend Tab. 12 to show whether the method mitigates leakage under the same setting, and move it to the main paper.

We ran the table exactly as proposed, with three seeds per row and the same initialization, training budget and evaluation (see Tab. R7 below). The “w/” rows are the full MoProGS model, with both the branch responses and the projection support.

Table R7. Frozen-scene intervention on Cook Spinach. PSNR in dB, mean ± s.d. over three seeds. Held-out views are the 12 rig cameras not used in training plus cam00.

MethodTimeAssignedRecombinedHeld-out view
w/o view-dependent approachshared33.34 ± 0.02n/a32.14 ± 0.05
w/o view-dependent approachunique34.51 ± 0.0131.46 ± 0.0231.24 ± 0.01
w/ view-dependent approach, full MoProGSshared33.09 ± 0.05n/a32.64 ± 0.06
w/ view-dependent approach, full MoProGSunique34.33 ± 0.1231.87 ± 0.1031.62 ± 0.10
  • MoProGS mitigates the leakage. Held-out PSNR rises by 0.50 ± 0.07 dB with shared times and by 0.38 ± 0.09 dB with unique times. PSNR on unseen camera-time recombinations rises by 0.41 ± 0.09 dB. The gap between assigned and recombined pairs narrows from 3.05 to 2.46 dB, a reduction of 19%. Every difference has the same sign in all three seeds.

  • The gain is not a better training fit. PSNR on the assigned pairs changes by −0.25 dB and −0.18 dB. The spurious temporal variation of the Gaussian centers falls from 0.055 to 0.034 scene units, in every seed.

  • Revision. As suggested, the table will be placed in Sec. 1 and cited next to the leakage motivation (L043–045), and Fig. 6 moves to the appendix. All four rows are retrained with three seeds, and this table replaces Tab. 12.

All changes above will be made in one revision of the manuscript once the remaining experiments are complete.

Thank you for the response. I will review it once the revisions are completed.

In the meantime, I have a few brief comments:

  1. This discrepancy in the LPIPS evaluation protocol raises concerns about the reliability of the experiments.
    • Please carefully verify that all experiments have been conducted and evaluated fairly and correctly.
  2. It seems that the Tables R6 and R7 may have been swapped.
    • If so, I wonder whether a PSNR over 40 dB is possible for the held-out (test) views of the cook spinach scene.
  3. I could not find the stated "Held-out PSNR rises by 0.50 ± 0.07 dB" in the provided table.
Reviewer_crMuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies deformation-based 4D Gaussian Splatting. It shows that for a flat Gaussian, the projected center shift scales with incidence pp, while the thickness contribution to the projected covariance scales with p2p^2, so the gradient split between position and thickness depends on the camera. MoProGS therefore keeps the deformation camera-blind and contracts the normal-axis scale according to the current view before rasterization. Experiments on D-NeRF, HyperNeRF, and Neu3D report gains over FAGS.

Critical strengths

1. Clear analysis. The different responses of position and thickness are well derived and verified on a controlled Gaussian sheet.

2. Simple operator. The support operator adds a single learned scalar and can be readily integrated into existing pipelines.

Critical weaknesses

1. Inconsistent baseline values. Three FAGS entries in Table 1 differ from the published FAGS paper, and each of these published values matches the corresponding MoProGS value:

MetricFAGS in Table 1Published FAGS
Mutant PSNR44.0944.16
Bouncing Balls SSIM0.9950.996
Jumping Jacks LPIPS0.0070.006

With the published values, MoProGS ties FAGS on Mutant rather than strictly outperforming it on all seven scenes. Table 2 likewise reuses published baseline values without specifying the LPIPS network.

2. Unclear source of the gain. The backbone is not clearly specified. The support-off training comparison is limited to Cook Spinach (L375–377; Supplement Sec. E.2), and the 0.1–0.2 dB gaps in Table 4 are reported without variance estimates. Moreover, the operator renders flat Gaussians about 2–5× thinner at grazing views, so the rendered geometry does follow the camera, contrary to the title. The paper neither discusses nor compares against recent view-conditioned dynamic Gaussian methods [1].

Minor writing and cross-reference issues. L046: "solutons" → "solutions"; L074: "In above solutions" → "In the above solutions"; L389: use "center" instead of "centre" for spelling consistency; L1262: "Eq. 40" → "Eq. 39".

Questions for authors
  1. Clarification: Please clarify the source of the FAGS values in Table 1 and specify the LPIPS network used.

  2. Additional evidence: Please extend the ablation without the support operator beyond Cook Spinach, report results across several seeds, and include a geometry metric such as Chamfer distance [3], as well as an evaluation on views far from the training trajectory [2].

References

[1] 7DGS: Unified Spatial-Temporal-Angular Gaussian Splatting. ICCV 2025.
[2] Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction. ICLR 2026.
[3] 4DSurf: High-Fidelity Dynamic Scene Surface Reconstruction. CVPR 2026.

5 comments

We thank the reviewer for carefully checking our numbers, and apologize for the errors in Tab. 1. We have corrected them, unified the LPIPS protocol and extended the ablations as requested.

W1/Q1: Three FAGS entries in Tab. 1 differ from the published values. Please clarify their source and specify the LPIPS network.

The three entries are transcription errors on our side. We corrected them to the published values: Mutant PSNR 44.16, Bouncing Balls SSIM 0.996 and Jumping Jacks LPIPS 0.006. To verify the whole row, we also re-ran FAGS on all seven scenes with three seeds, using our re-implementation (see Tab. R3 below). The re-runs are within 0.15 dB of the published values on every scene and 0.06 dB lower on average, so Tab. 1 keeps the published values.

Table R3. PSNR on D-NeRF in dB. Re-runs report mean ± s.d. over three seeds.

SceneFAGS, publishedFAGS, our re-runMoProGS, paperMoProGS, three seeds
Bouncing Balls43.3243.22 ± 0.0943.6043.58 ± 0.08
Hell Warrior43.1342.98 ± 0.1243.2743.30 ± 0.11
Hook40.1840.09 ± 0.0740.4840.46 ± 0.06
Jumping Jacks40.5240.55 ± 0.1041.3841.41 ± 0.09
Mutant44.1644.15 ± 0.0544.1644.17 ± 0.05
Standup47.3047.32 ± 0.0847.5747.55 ± 0.07
T-Rex40.6940.56 ± 0.0841.4641.47 ± 0.07
Average42.7642.70 ± 0.0443.1343.13 ± 0.04
  • Comparison with the corrected values. With the published values MoProGS ties FAGS on these three entries. It keeps the best PSNR on six scenes and ties on Mutant. Its average PSNR is 43.13 dB against 42.76 dB, a margin of +0.37 dB. The largest gains are on Jumping Jacks and T-Rex, at +0.86 and +0.77 dB. Against our three-seed re-runs the margin is +0.43 dB. Sec. 4.2 will read “the best or tied PSNR on all seven scenes”, and the Mutant margin in Supp. F.1 becomes 0.00 dB.

  • LPIPS network. Tab. 2 mixed two protocols. Our values used AlexNet v0.1 with inputs in [−1, 1], the default of the official LPIPS package. The baseline rows are taken from FAGS. FAGS does not state its network, but it builds on the Grid4D and 4D-GS code base, which evaluates with VGG on inputs in [0, 1]. Our FAGS re-runs confirm this. They give 0.149 under the VGG protocol, close to the published 0.146, and 0.100 under the AlexNet protocol. We re-scored the same MoProGS renders under the VGG protocol, without retraining (see Tab. R4 below).

Table R4. LPIPS on Neu3D. The first two rows use the same protocol, VGG with inputs in [0, 1].

MethodCoffee MartiniCook SpinachCut Roasted BeefFlame SteakSear SteakAverage
FAGS, published0.1790.1450.1480.1320.1250.146
MoProGS, unified protocol0.1720.1320.1320.1090.1260.134
MoProGS, as submitted with AlexNet0.1150.0880.0880.0730.0840.089
  • Comparison under one protocol. MoProGS obtains 0.134 against 0.146 for FAGS. This is 8% lower on average and lower on four of the five scenes. MoProGS thus keeps the best average PSNR, SSIM and LPIPS on Neu3D. The margins are smaller there than on monocular D-NeRF and HyperNeRF, as the analysis predicts when many synchronized cameras constrain every state. Tab. 2 and Sec. 4.2 will use the unified protocol.

  • Source of the FAGS numbers. The FAGS entries of Tab. 1 to Tab. 3 are the published values. The FAGS images in the qualitative figures come from our re-implementation. The revised captions will state the source and the LPIPS protocol of every row.

W2: The backbone is not clearly specified.

MoProGS retains a standard deformation-based pipeline: canonical Gaussians, a camera-blind temporal encoder, prediction heads for position and for rotation and scale, and the 3DGS rasterizer. It adds a branch-specific response in the deformation stage and the incidence-conditioned normal support before rasterization. The base model disables both, and every paired comparison keeps everything else fixed. Each addition is measured by these paired ablations: the response decomposition in Tab. 4 and Tab. R2, the support in Tab. R5 and Tab. R8, and the full model in Tab. R7. Tab. R2 and Tab. R7 are in our responses to Reviewers 6q62 and tGYS. The parameterizations and parameter counts are listed in our response to Reviewer 6q62 (W2/Q2). We will add them to Sec. 4.1 and Supp. G.

W2/Q2: The support-off ablation is limited to Cook Spinach. Extend it and report several seeds.

We extended it to D-NeRF and to three Neu3D scenes.

  • D-NeRF. We removed the support on all seven scenes with three seeds. The mean PSNR falls by 0.15 ± 0.04 dB, in every seed (see Tab. R8 below).

  • Neu3D. We trained three scenes with one camera per time step, with support on and off, and used three seeds on Cook Spinach (see Tab. R5 below). This setting couples the observer with time and allows testing on cameras that are never used in training.

Table R5. PSNR gain from the support under one-camera-per-time training, in dB.

SceneSeedsAll held-out viewsUnseen cameras
Cook Spinach3+0.27 ± 0.33+0.32 ± 0.30, positive in 3 of 3
Cut Roasted Beef1−0.18−0.06
Coffee Martini1+0.13+0.14

The support improves unseen cameras on Cook Spinach in all three seeds and on Coffee Martini. On Cut Roasted Beef the change is within the seed variation measured on Cook Spinach. Seed repeats for Cut Roasted Beef and Coffee Martini are running and will be posted here. The support also acts on attribution and on the stored geometry. It reduces the within-Gaussian slope by 32% to 65% (Tab. 9) and halves the spurious thickness variation in the frozen scene (Tab. 16).

W2: The 0.1–0.2 dB gaps in Tab. 4 are reported without variance estimates.

Tab. 4 reports seven-scene means of single runs. We repeated all five rows with three seeds and added the row without the support (see Tab. R8 below). Every gap keeps its sign in all three seeds. For the largest gap, the response decomposition, we added a capacity-matched control with paired seeds on Neu3D. Sharing the response costs 0.37 ± 0.13 dB, in all four paired runs (Tab. R2 in our response to Reviewer 6q62).

Table R8. Tab. 4 repeated with three seeds, with an added row without the support. Seven-scene mean PSNR on D-NeRF in dB, mean ± s.d. over seeds.

VariantPSNRΔ PSNRSeeds with the same sign
MoProGS, full43.13 ± 0.04reference
w/o response decomposition42.93 ± 0.04−0.20 ± 0.033 of 3
w/o projection support, added42.98 ± 0.04−0.15 ± 0.043 of 3
w/o flatness confidence q43.01 ± 0.04−0.12 ± 0.033 of 3
w/o detached routing42.96 ± 0.05−0.17 ± 0.043 of 3
Routing from canonical state43.03 ± 0.04−0.10 ± 0.033 of 3

W2: The rendered geometry follows the camera, contrary to the title.

The title refers to the reconstructed scene state. The canonical Gaussians and the deformation output never receive the camera, and at a fixed time their variation across cameras is exactly zero (Tab. 17). The camera enters only at rasterization, through one bounded analytic factor on the normal-axis scale with a single learned strength κ per scene. The thinning at grazing views is thus an intended rendering-stage effect, like the screen-space filters of EWA splatting and Mip-Splatting. To make this explicit, the abstract and Sec. 1 will state that “geometry” means the scene state before support.

W2: The paper neither discusses nor compares against view-conditioned dynamic Gaussian methods [1].

Thank you for pointing us to 7DGS, the closest view-conditioned method. We will discuss it in Sec. 2, and the comparison is running.

  • Analysis. 7DGS represents each primitive as a 7D Gaussian over position, time and viewing direction, so the view enters the learned geometry itself. MoProGS keeps the learned geometry independent of the view and applies one analytic factor at rendering. With many views per state, a view-conditioned geometry is well constrained. With one view per time, viewing direction and time are fully correlated in training, and such a geometry can absorb camera-specific fitting. Our pipeline shows this when the camera is fed into the deformation. The scene state then varies across views by 0.011 in center, 0.063 in log-scale and 0.046 in rotation (Tab. 17). Even the indirect route through time is costly. In the frozen scene it costs 3.05 dB on unseen camera-time pairs, and MoProGS narrows this gap to 2.46 dB (Tab. R7 in our response to Reviewer tGYS).

  • Experiment. Its published results average 35.13 dB on the seven D-NeRF scenes we share, at half resolution (its Tab. 1). MoProGS reaches 43.13 dB at full resolution. We are running 7DGS under our protocol, together with the camera-time recombination test of Tab. R7 in our response to Reviewer tGYS. We will post the numbers here as soon as the runs finish and add them to the paper.

Q2: Include a geometry metric such as Chamfer distance [3].

Chamfer distance needs ground-truth surfaces and an extracted surface. D-NeRF, HyperNeRF and Neu3D provide no ground-truth surfaces, and MoProGS, like the methods we compare with, does not extract one. 4DSurf itself reports the metric on Hi4D and CMU Panoptic. We therefore measure geometry where the ground truth is exact. In the frozen-scene test nothing moves, so any temporal variation of a Gaussian is an error. MoProGS reduces the center variation from 0.055 ± 0.001 to 0.034 ± 0.015 scene units, in every seed (see Q4 in our response to Reviewer tGYS). The support alone halves the thickness variation, from 0.0997 to 0.0500 (Tab. 16). We will cite 4DSurf [3] in Sec. 2.

Q2: Evaluate on views far from the training trajectory [2].

  • D-NeRF. The standard test views are unseen camera-time pairs, the case our analysis targets. On Hook the camera of the nearest training time is a median of 71° from the test camera, with a range of 19° to 147°. A training camera within 17° exists only at other times, so the learned state must transfer across cameras.

  • Neu3D. The unseen cameras of Tab. R5 are never used in training and lie 0.5° to 35° from the nearest training camera. The support improves them on Cook Spinach in all three seeds.

We will cite USplat4D [2] in Sec. 2.

Minor: writing and cross-reference issues.

All four are corrected: “solutions” in L046, “In the above solutions” in L074, “center” in L389 and in all other occurrences, and “Eq. 39” in L1262.

All changes above will be made in one revision of the manuscript once the remaining experiments are complete.

Reviewer_6q62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoProGS addresses view-dependent differences between position and thickness sensitivities in dynamic Gaussian Splatting. It combines separate response adapters with bounded, incidence-conditioned normal-scale contraction before rasterization. Controlled experiments, frozen-model analyses, and three reconstruction benchmarks support reduced sensitivity dependence and improved image quality, although the calibration objective and architectural contribution remain insufficiently isolated.

Critical strengths

1.Good writing and organization. The paper presents its motivation, analysis, method, and experiments in a coherent sequence. 2.Controlled evidence of practical benefit. With data, initialization, optimizer, training budget, and response architecture fixed, enabling projection support improves held-out Cook Spinach PSNR from 31.569 to 31.774 dB and reduces L1 by 3.1%

Critical weaknesses
  1. The calibration objective lacks sufficient validation. A flatter sensitivity ratio does not necessarily imply better geometric updates. Current versus mean incidence improves PSNR by only 0.020 dB on one scene, without training-run variability estimates (Appendix C.2, p.23).
  2. Response decomposition is confounded with added capacity. The baseline already uses separate prediction heads, while adapters add parameters (Appendix G.3, p.35). Without controls matched for capacity and depth, the 0.20 dB gain cannot be attributed specifically to decoupling (Table 4, p.6).
Questions for authors
  1. Targeted validation: Does calibration reduce incorrect-direction updates for both center- and thickness-generated residuals in the existing known-cause experiments?
  2. What precisely changes in “w/o response decomposition”? Please specify the adapter architecture and parameter count, and compare against expanded prediction heads matched for capacity and depth, with projection support and training conditions fixed.
4 comments

We thank the reviewer for the insightful comments, which led us to analyze the known-cause experiment for both residual types and to add capacity- and depth-matched controls.

W1/Q1: The calibration objective lacks validation. Does calibration reduce incorrect-direction updates for both center- and thickness-generated residuals?

Not at the same view, and no reweighting of the two responses can achieve this (Eq. 26). The share falls for center-generated residuals and rises for thickness-generated ones. What calibration reduces for both is the camera dependence of this share, which is the target of our objective (see Tab. R1 below).

Table R1. Known-cause residuals on the controlled sheet (Supp. B.7), with current-incidence support against no support. The wrong-direction share is the fraction of the initial gradient that goes to the non-generating direction (Eq. 27).

Cause of the residualWrong-direction shareIts cross-view s.d.
Center displacementfalls−23%
Thickness changerises−40%
  • Why both cannot fall at one view. Let cc be the overlap of the two image Jacobians (the absolute cosine between JμJ_{μ} and JsJ_{s}) and R=‖Js‖/‖Jμ‖R = ‖J_{s}‖ / ‖J_{μ}‖ their response ratio. By Eq. 26, a center-generated residual is credited to thickness with ratio χμ→s=c⋅Rχ_{μ→s} = c · R, and a thickness-generated one to position with χs→μ=c/Rχ_{s→μ} = c / R. Their product c2c^{2} does not depend on RR. The support rescales RR by ξn2ξ_{n}^{2} (Eq. 8), so one ratio falls and the other rises. This holds for any reweighting of the two responses.

  • When both can fall. Only when the overlap cc drops by more than RR changes, that is c′/c<min(R′/R,R/R′)c′/c < min(R′/R, R/R′). Reweighting cannot do this, because cc is set by what the view observes. It is 0.12 to 0.13 over the full image and 0.52 to 0.59 on one-sided boundaries, and the support changes it by less than 10%. What the support changes is the incidence slope of RR, from 1.30 to 0.35 (Tab. 8).

  • Role of the current observer. A constant contraction reduces the thickness-generated variability by only 8%, against 40% with current incidence. The consistency thus comes from matching the current observer.

  • Effect on geometric updates. Under monocular capture the camera changes with time, so a camera-dependent split is stored as motion. We agree that this needs a direct test, and the frozen scene provides one. Nothing moves there, so any temporal change of a Gaussian is an error. The support alone halves the spurious thickness variation, from 0.0997 to 0.0500, and leaves the positional variation unchanged, with a median ratio of 1.01 (Tab. 16). Repeated with three seeds, the thickness variation falls from 0.099 ± 0.004 to 0.051 ± 0.005. The larger share credited to position thus does not turn into spurious motion. The full model lowers the spurious center variation from 0.055 to 0.034 scene units (see Q4 in our response to Reviewer tGYS).

  • The 0.020 dB gap. The gap is small because frame-mean incidence is already a strong control when 20 synchronized cameras observe every state. It is systematic: current incidence lowers the error on 96.3% of the 300 held-out frames (Tab. 10). We also retrained both variants with one camera per time and three seeds. The PSNR difference stays within seed variation there, so we do not rest the case on PSNR. The case for current incidence is attributional. Only current incidence reduces the within-Gaussian slope, by 32% to 65%, while frame-mean and shuffled incidence leave it unchanged (Tab. 9). We will state this in Supp. C.2.

W2/Q2: Response decomposition is confounded with added capacity. What changes in “w/o response decomposition”? What are the adapter architecture and parameter count? How does it compare with heads matched for capacity and depth?

The gain comes from separating the two responses and not from the added parameters. We added controls matched for capacity and depth (see Tab. R2 below).

  • What “w/o response decomposition” changes. Only the branch-specific response is removed, so both branches share one response again. The encoder, the heads, the support operator and the training schedule are unchanged.

  • Architecture and parameter count. The decomposition is placed where the retained deformation network exposes the response (see the table below). Both forms start from the same forward pass as the model without decomposition. On Neu3D each adapter is a linear layer in front of a linear layer, so the function class is unchanged. We will add the D-NeRF form and both parameter counts to Supp. G.3.

ConfigurationResponse in the retained networkWhat MoProGS addsAdded parameters
D-NeRFOne scalar weight per deformation step, shared by the center and covariance residualsA separate weight head for the covariance, Linear(256, 1) per step2,570, which is 0.11% of the 2.40M-parameter deformation network. No added depth
Neu3DSeparate position and covariance heads on a shared 128-d featureTwo linear adapters, 128 → 128, one in front of each head33,024. One linear layer per branch
  • Controls matched for capacity and depth. On D-NeRF we keep the shared weight and add a second weight head with the same 2,570 parameters, summed into it. On Neu3D we keep both adapters but let the two branches share them. The training schedule and the support setting are fixed within each comparison, so only the response pathway differs. The support is disabled in all D-NeRF arms and enabled in all Neu3D arms.

Table R2. Response decomposition against capacity-matched controls. Δ PSNR is relative to independent responses and paired by scene and seed.

SettingVariantAdded paramsΔ PSNR (dB)
D-NeRF, Hook, 3 seedsShared weight, as in the "w/o" variant0−0.09 ± 0.03, lower in 3 of 3
D-NeRF, Hook, 3 seedsShared weight plus extra head, matched capacity and depth2,570−0.12 ± 0.03, lower in 3 of 3
D-NeRF, Hook, 3 seedsIndependent weights, ours2,570reference
Neu3D, 2 scenes × 2 seedsShared adapters, same parameters33,024−0.37 ± 0.13, lower in 4 of 4
Neu3D, 2 scenes × 2 seedsIndependent adapters, ours33,024reference
  • Analysis. At equal parameters and function class, sharing the response costs 0.37 ± 0.13 dB on Neu3D, in all four paired runs. The D-NeRF control agrees. With the same 2,570 parameters and depth but one shared response, the model stays 0.12 ± 0.03 dB below independent weights, in all three seeds. Extra parameters without separate responses therefore give no gain.

All changes above will be made in one revision of the manuscript once the remaining experiments are complete.

Thank you again for your comments. The revision is uploaded, with Tab. R1 in Supp. B.7 and Tab. R2 as Tab. 7 in Sec. 4.3. Since our first reply, we ran new experiments on eight static NeRF Synthetic scenes, without a deformation network, and completed the seed repeats. Both bear directly on W1.

Summary of the new results. Gaussian kernel, eight scenes. Tab. R9, R10 and R12 in our reply to Reviewer tGYS list every scene.

QuantityResult
Slope β without support1.36 to 1.62
Change of β under mean incidenceat most 0.02
Change of β under current incidence0.87 to 2.85
Zero crossing κ*, measured0.71 to 0.81, mean 0.76
PSNR change from the support0.00 dB on average

W1. The calibration objective is validated on eight NeRF Synthetic scenes. Our objective is a slope of zero under the current observer. At fixed κ, the slope crosses zero in all eight scenes, at κ* from 0.71 to 0.81 with a mean of 0.76. Eq. 31 places the crossing at κqp = 1/3, so κ* scales as 1/q. The trained Gaussians are flatter than the controlled sheet, and their crossing lies below its 0.87 to 0.88, as this scaling implies. The objective and its balance point thus hold beyond the controlled sheet.

W1. Current versus mean incidence, beyond one scene. You noted that the two were compared on one scene only. On all eight scenes, mean incidence leaves the slope within 0.02, while current incidence moves it by 0.87 to 2.85. Matching the current observer thus matters on every scene.

W1. When calibration improves geometric updates. You noted that a flatter ratio need not give better updates. The static scenes provide the control. Every state is shared by 100 views, so a camera dependent split cannot be stored. There the support changes the attribution, and PSNR stays unchanged. The benefit thus comes from the mechanism we target, a split stored in separate states, and not from a general change of rendering.

W1. Training-run variability. You noted that the 0.020 dB gap between current and mean incidence came without variability estimates. We retrained both variants with one camera per time over three seeds.

Table R14. PSNR of current minus mean incidence, in dB. Values over seeds are mean ± s.d.

Training and test viewsSeedsCurrent − mean
Synchronized cameras, cam00, Supp. C.21+0.020
One camera per time, all held-out views3+0.08 ± 0.26, positive in 2 of 3
One camera per time, unseen cameras3+0.09 ± 0.33, positive in 2 of 3

Current incidence stays ahead on average, and the difference lies within the seed variation. The case for current incidence therefore rests on attribution, where only current incidence moves the slope, on all eight scenes above. The support itself improves unseen cameras in 6 of 9 runs over three scenes and three seeds, by +0.13 dB on average (Tab. R5 in our reply to Reviewer crMu).

We hope these results address your concerns. We would be grateful if you would take them into account in your final assessment.

Thank you for the detailed response and the additional experiments. Your rebuttal addressed most of my concerns. I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.