acceptodds
Under review as a conference paper at ICLR 2027

PhysGraph: Point Graphs for Physical Scene Understanding

Abstract

Real-world videos mix rigid bodies, cloth, and liquids – yet existing dynamic-reconstruction methods commit to a single motion model. We present PhysGraph, a temporal interpolation and extrapolation pipeline that uses a one-shot, foundation-model-predicted material phase as a prior for selecting one of three classical motion models on a shared per-object kNN point graph: Procrustes-style SE(3) for solids, thin-plate-spline residuals for deformables, and optimal transport for liquids. The graph is reused across branches as a computational primitive – for neighborhood-aware correspondence filtering (rigid, elastic), as the elastic-kernel subgraph (TPS), and as the rewireable bipartite source-to-target structure that OT solves over (liquid). Composed with bidirectional Gaussian splatting, the resulting motions synthesize novel frames up to faster than per-scene-optimized baselines. On six synthetic scenes PhysGraph reduces depth error by over linear 3D interpolation; on 90 DAVIS sequences it has the highest dynamic-region recall at every stride with zero failures (\vs up to 69% for per-scene optimizers), and it leads dynamic-region recall in extrapolation as well. PhysGraph targets large temporal gaps, where per-scene optimizers become ill-conditioned and fail. The source code will be made public.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D scene understanding, physical scene understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z5fsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

PhysGraph interpolates and extrapolates sparsely sampled monocular videos by using predicted material categories to select rigid SE(3), TPS deformation, or optimal-transport motion models on per-object point graphs. It combines existing perception models with bidirectional Gaussian splatting and evaluates the framework on six synthetic scenes and 90 DAVIS sequences.

Critical strengths
  1. The material-conditioned framework provides an interpretable way to handle heterogeneous motion, combining rigid, deformable, and liquid models within a shared per-object point-graph representation.
  2. The evaluation covers six synthetic scenes and 90 DAVIS sequences. The end-to-end DAVIS results include errors from estimated geometry, correspondences, and segmentation, and demonstrate practical robustness and efficiency relative to the tested per-scene optimization methods.
Critical weaknesses
  1. The method evolves surface points observed in the endpoint frames rather than a complete object representation. Bidirectional fusion can recover surfaces visible at either endpoint, but no mechanism is described for surfaces absent from both endpoints that become visible during interpolation or extrapolation. The evaluation does not establish reliable behavior under substantial occlusion and disocclusion, limiting support for the broader physical scene understanding claim.
  2. The experiments primarily assess temporal interpolation and extrapolation, yet the baselines largely target dynamic 3D reconstruction. These comparisons are informative, but restricting reconstruction methods to sparse keyframes does not establish competitiveness against dedicated learning-based frame interpolation and video prediction methods.
  3. The method assigns motion models through a fixed noun-to-phase mapping, although objects within the same category may exhibit different motion structures. If this simplification fails to preserve the underlying motion structure, it remains unclear what physical understanding the semantic mapping provides. Meanwhile, the reported pass rates at PSNR ≥15 dB and ≥20 dB do not by themselves establish convincing visual fidelity or temporal consistency. Together with the acknowledged physically incorrect motion in articulated objects, these results do not sufficiently establish either physical validity or visual quality.
Questions for authors
  1. How are newly visible surfaces handled when they are absent from both endpoint point clouds? Could the authors evaluate performance specifically in regions undergoing occlusion and disocclusion?
  2. Could the authors compare against dedicated learning-based frame interpolation and video prediction methods under matched input and temporal-gap settings, to establish the benefit of the proposed geometry and material-processing stages?
  3. Given the acknowledged physically incorrect motion in articulated objects, could the authors clarify what physical understanding the fixed noun-to-phase mapping provides and what evidence supports its benefits for physical plausibility or visual quality?
3 comments

We revised the paper to narrow scope to material-conditioned motion modeling (Secs. 1, 3.7), isolate splatting from infill across visibility strata (Table 15), benchmark VFI, diffusion, and 4D models (Tables 9-11b, Fig. 8), and ablate phase dispatch across 414 runs (Tables 4, 14).

1. Doubly Occluded Surfaces, Disocclusions, and Extrapolation (W1, Q1)

PhysGraph advects endpoint Gaussians without hallucinating unseen 3D interiors (Sec. 3.7): single-endpoint disocclusions (A⊕BA\oplus B) are recovered in 3D by the bidirectional Gaussian union, while doubly occluded holes (A∪B‾\overline{A\cup B}, unseen in both A,BA,B) composite over (1−α)IA+αIB(1-\alpha)\mathbf{I}_A+\alpha\mathbf{I}_B in interpolation and pull past static points plus Telea inpainting in extrapolation (α>1\alpha>1).

Table 15 isolates Forward-Only, Bidir (no inf.), and Bidir + Infill across 3D visibility strata (4 mesh scenes) and 18 DAVIS clips (s=5/10/25s=5/10/25), plus causal extrapolation (Table 3):

Region (s=5/10/25s=5/10/25)ShareForward-OnlyBidir (no inf.)Bidir + Infill
Synth Mutual (A∩BA\cap B)88.4/78.6/58.3%26.51/24.93/21.0926.66/25.25/21.3026.67/25.27/21.34
Synth 1-Endpt (A⊕BA\oplus B)11.5/21.1/39.9%17.15/16.75/16.2924.55/23.00/21.1924.61/23.33/21.25
Synth Doubly (A∪B‾\overline{A\cup B})0.11/0.35/1.77%15.07/14.21/13.1315.43/14.64/13.2817.80/17.77/17.64
DAVIS-18 Full PSNR100%16.97/15.50/13.6818.18/17.20/15.7819.33/18.22/16.72
DAVIS-18 Dyn. dPSNRDyn.15.95/14.49/12.8316.29/14.91/13.1216.41/14.97/13.35
Extrap. dR@15 (s=3/5/10s=3/5/10)DAVIS-9034.4/27.8/22.9% (RepeatKF)30.0/23.6/19.7% (GP)36.7/32.2/22.9%

Two conclusions follow: (i) doubly occluded pixels form under 2% of interpolation frames, while bidirectional union sharply lifts single-endpoint disocclusions over forward-only splatting (and beats 2D cross-dissolve on moving objects at every stride), with infill boosting doubly occluded holes and handheld borders; and (ii) in causal extrapolation, explicit 3D motion models beat both GaussianPrediction (at 3x lower runtime) and copying the last keyframe.

2. Learning-Based VFI, Diffusion, and 4D Baselines (W2, Q2)

We evaluated FILM, MoMo, RIFE, SVD-XT GenInb, MoVieS, UFO-4D, and 3D Linear under matched keyframes across all 90 DAVIS clips and checkerboard synthetic renders (Tables 9-11a):

MethodSynth Tex. (s=5/10/25s=5/10/25)DAVIS-90 Mean/Med (s=5/10/25s=5/10/25)s=25s=25 dR@15SVD-103DExtrap.
3D Linear19.33/18.97/15.69------YesYes
FILM (2D)19.99/16.79/14.2015.82/15.11, 13.67/12.67, 12.16/11.5213.3%--NoNo
MoMo (2D)24.07/19.48/15.5017.29/16.79, 14.86/13.77, 12.95/12.0820.0%--NoNo
RIFE (2D)24.34/19.79/16.1917.81/17.12, 15.23/14.29, 13.31/12.6024.4%--NoNo
SVD-XT------13.16 (25m)NoNo
MoVieS (4D)--14.52/14.05, 13.67/12.94, 12.60/11.9015.6%--YesNo
UFO-4D (4D)--16.52/15.82, 14.85/14.02, 13.19/12.5124.4%--YesNo
PhysGraph19.77/19.33/16.1416.29/15.25, 14.64/13.99, 13.04/12.3424.4%14.31 (1.4m)YesYes

At dense strides, 2D warping avoids 1-px splat discretization on checkerboards; at wide gaps (s=25s=25), PhysGraph overtakes 3D Linear, FILM, and MoMo and matches RIFE with metric depth. On DAVIS, RIFE and UFO-4D hold a slight mean-dPSNR lead by bypassing/absorbing MonST3R noise, yet neither extrapolates (α>1\alpha>1) and RIFE lacks 3D output. PhysGraph ties both for #1 sparse recall, beats FILM, MoVieS, and MoMo at large gaps, and beats SVD-XT on 10/10 clips (18x faster).

3. Phase Routing and Visual/Temporal Fidelity (W3, Q3)

  • Phase routing (Tables 4, 14): PhysGraph is, to our knowledge, the first dynamic monocular video method to classify solid, deformable, and liquid objects from images (Table 12) to select link rigidity (SE(3), SE(3)+TPS, OT) rather than causality. Across 414 fixed-frontend runs, zero-shot routing matches human oracle labels, cuts clean-3D depth error by over a third, halves centroid drift vs. All-Liquid, and beats all single-branch policies on multi-phase water-plus-foreground DAVIS clips (All-TPS absorbs MonST3R drift on dry-road vehicles; whole-body masks limit articulated limbs, Sec. 5).
  • Fidelity beyond PSNR recall (Table 11b, Fig. 8):
Method (n=90n=90, s=3/5/10/25s=3/5/10/25)dLPIPS ↓\downarrowSharpness ↑\uparrowtOF ↓\downarrow
MoVieS (4D).322/.323/.346/.394.383/.379/.377/.39215.2/13.0/11.9/11.8
UFO-4D (4D).186/.223/.288/.388.421/.413/.387/.3699.35/9.37/9.42/9.48
RIFE (2D).109/.154/.237/.343.691/.644/.669/.5889.03/9.61/10.3/9.73
PhysGraph (Ours).264/.295/.350/.421.505/.525/.553/.60012.5/11.9/11.9/12.1

PhysGraph keeps sharper dynamic edges than 4D baselines (avoiding MoVieS erasure and UFO-4D ghosting, Fig. 8), beats MoVieS on dLPIPS/tOF at moderate strides, and holds flat tOF across wide gaps.

Thank you for the additional experiments. However, my main concerns remain unresolved.

  1. Bidirectional splatting only addresses interpolation. Extrapolation still relies on reconstructed points and image inpainting, without resolving newly visible surfaces.

  2. The new results show no clear visual advantages over dedicated interpolation methods, leaving the necessity of the additional 3D reconstruction and motion-model complexity insufficiently justified.

  3. The noun-to-phase mapping remains a simple heuristic. The new ablations demonstrate reconstruction improvements but provide no convincing evidence of physically plausible motion.

Therefore, I maintain my original rating for now.

Thank you for clarifying the remaining concerns. Below we address each.

1. Extrapolation and Newly Visible Surfaces

In causal extrapolation (α>1\alpha > 1, from keyframes 0…K0 \dots K), newly exposed pixels split into two regimes:

  • Previously observed disocclusions (visible in 0…K−10 \dots K-1 before occlusion at KK, e.g., background behind an advancing boat or car): PhysGraph resolves these directly in 3D without 2D inpainting by aggregating unoccluded static 3D points across past keyframes (Sec. 3.7). In Table A, observed 3D geometry covers over nine-tenths of extrapolated pixels on synthetic and DAVIS clips, and past-keyframe 3D aggregation substantially lifts disoccluded-wake accuracy before any 2D infill.
  • Surfaces never observed in 0…K0 \dots K (e.g., unseen backs or new frustum borders): no observation-grounded method, 2D or 3D, can recover geometry never seen by the camera. Synthesizing it requires generative priors, whereas PhysGraph stays observation-grounded to avoid inventing fake geometry (and beats SVD-XT diffusion in accuracy and speed). Telea infill is strictly a fallback on this small never-seen fraction.

2. Necessity of 3D Reconstruction and Motion Models vs. 2D VFI

For fixed-camera 2D interpolation alone, dedicated 2D warpers like RIFE are simpler and avoid monocular depth noise because fixed-ray projection hides 3D depth error along the viewing ray (added for W2/Q2).

PhysGraph instead solves sparse-keyframe 4D dynamic scene modeling, delivering outputs 2D interpolators structurally lack: (i) metric 3D geometry and trajectories (Table 1); (ii) free-viewpoint 3D rendering from novel cameras at held-out times, where 2D VFI cannot run and 3D depth/motion errors appear as parallax misprojections, giving PhysGraph a clear lead over 3D Linear and All-Rigid (Table A); and (iii) causal extrapolation (α>1\alpha > 1), where RIFE, FILM, MoMo, and UFO-4D cannot run (Table 3). Within 4D methods (MoSca, SoM, GP, MoVieS, UFO-4D), PhysGraph eliminates crashes (Fig. 3), runs much faster than 4DGS optimizers, and avoids MoVieS erasure and UFO-4D ghosting (Fig. 8).

Table A: Causal Extrapolation 3D Coverage, Pure-3D Wake Recovery, and Novel-View 4D

SettingMetricBaseline / ComparisonPhysGraph
Synth Extrap (s=3/5/10s=3/5/10)Causal 3D observed (0…K0\dots K)4.5/5.9/8.6% never seen95.6/94.1/91.4% in 3D
DAVIS-18 Extrap (s=5s=5)Direct 3D splats (α3D≥.1\alpha_\text{3D}\ge .1)7.2% never-seen fallback92.8% direct 3D splats
DAVIS-18 Wake (s=5s=5, no infill)Wake PSNR (pure 3D)12.25 dB (KF-KK 3D only)13.49 dB (+Past-KF 3D)
Novel-View 4D (12∘12^\circ, s=5/10s=5/10)Moving-object PSNR2D VFI: N/A | Lin: 23.07/19.30 | Rig: 24.92/20.9228.62 / 24.16 dB

3. Beyond Noun Lookup and Direct Evidence of Physically Plausible Motion

  • Open-ended visual reasoning (no noun table required): Appendix C.3 (Table 12, Fig. 7) already implements direct image-based VLM classification on class-agnostic masks with zero noun list, judging phase from visual context (e.g., water vs. ice/reflections). On our 18-clip DAVIS benchmark (Table B), Qwen2.5-VL-7B visual labels strongly agree with oracle labels and match SAM 3 lookup and oracle routing. SAM 3 is simply a fast default.
  • Direct 3D physical invariants (not PSNR): Table B evaluates 3D physical invariants and simulator errors against the Blender rigid, cloth/FEM, and fluid engines:
    • Solids: Linear shrinks rotating bodies along chords, TPS bends under noise, and OT scrambles vertices (nearly doubling DAVIS drift); SE(3) keeps exact zero isometric strain and lowest 3D rigid error.
    • Deformables: Rigid freezes bending while Linear/OT scramble neighborhoods; SE(3)+TPS preserves local topology with lowest 3D cloth/soft-body error.
    • Liquids: Fixed-graph models stretch pouring fluids into mid-air sheets; OT conserves 100% marginal mass and cuts 3D free-surface error over two- to three-fold.

Table B: 3D Physical Invariants (s∈{5,10,25}s\in\{5,10,25\}, σ=15\sigma=15 mm) and VLM Routing (DAVIS-18)

Policy / Phase SourceRig StrainRig EPEDef kNNDef EPELiq ChamferDAVIS-18 dPSNR / CE
All-Linear5.31%67.1 mm8.50%58.8 mm86.7 mm15.07 / 31.9 px
All-Rigid (SE(3))0.00%62.4 mm0.03% (frozen)60.4 mm134.9 mm14.66 / 30.7 px
All-TPS (SE(3)+TPS)1.88%65.6 mm6.31%58.2 mm149.4 mm15.13 / 30.2 px
All-Liquid (OT)9.42%92.3 mm10.80%90.1 mm40.9 mm14.23 / 54.8 px
PhysGraph (SAM 3)0.00%62.4 mm6.31%58.2 mm40.9 mm14.91 / 30.1 px
PhysGraph (Qwen2.5-VL, 0 nouns)----------14.93 / 30.3 px (91.8% Orac.)
PhysGraph (Oracle)0.00%62.4 mm6.31%58.2 mm40.9 mm14.92 / 30.1 px
Reviewer_PmViOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PhysGraph, a material-conditioned framework for temporal interpolation and extrapolation of dynamic monocular scenes. Given sparsely sampled video keyframes, the method first obtains 3D geometry, correspondences, and object segments using off-the-shelf perception models. It then predicts a coarse physical phase for each dynamic object and dispatches it to one of three classical motion models: SE(3) motion for rigid solids, thin-plate-spline residual deformation for deformable objects, and entropic optimal transport for liquids. These branches are organized around a shared per-object point-graph representation, and the estimated motions are rendered with bidirectional Gaussian splatting to synthesize missing or future frames.

The method is evaluated on six synthetic scenes and all 90 DAVIS 2017 sequences under different temporal strides. The reported results show improved dynamic-region robustness under sparse temporal observations, zero failures on DAVIS in the tested setting, lower runtime than several per-scene optimization-based methods, and support for short-horizon extrapolation.

Critical strengths

1. Simple and interpretable material-conditioned motion modeling.

The paper identifies a meaningful limitation of applying a single generic motion model to physically heterogeneous objects. Routing rigid, deformable, and liquid regions to motion models with different inductive biases is simple but well motivated, especially under sparse temporal observations where highly flexible per-scene optimization can become unstable. The resulting motion models are explicit and naturally support interpolation and extrapolation.

2. Broad real-world evaluation and strong robustness characterization.

The evaluation on all 90 DAVIS sequences, multiple temporal strides, runtime, failure rates, extrapolation, fallback analysis, and explicit failure cases is a notable strength. In particular, the reported zero-failure behavior and strong dynamic-region recall under large temporal gaps provide meaningful evidence that the proposed design is useful in the sparse-input regime targeted by the paper.

Critical weaknesses

1. Limited methodological novelty and an overly broad physical-scene-understanding framing.

The main motion components—robust SE(3) fitting, TPS deformation, and entropic optimal transport—are established techniques, and the material phase is obtained through foundation-model perception followed by a predefined mapping to motion categories. The primary novelty therefore lies in system-level routing and composition rather than in a new motion model or physical reasoning mechanism. Moreover, the title and high-level framing around “physical scene understanding” appear broader than the actual technical scope. The method mainly uses coarse material phase information to support temporal interpolation and extrapolation; it does not explicitly reason about contact, support, forces, collisions, or physical causality. The contribution is more accurately characterized as material-aware dynamic motion modeling than general physical scene understanding.

2. The central material-routing hypothesis is not isolated as directly as expected.

Since the main claim is that material phase should determine the motion model, the most important ablation would directly compare predicted phase routing against oracle phase routing and against applying a single motion model to all dynamic objects under an otherwise identical frontend and renderer. The current ablations extensively study deformable solvers, fallback mechanisms, and continuous physical parameters, but they do not isolate as clearly how much of the end-to-end improvement is specifically caused by the discrete material-conditioned dispatch. This is important because the dispatch mechanism is the central methodological contribution of the paper.

Questions for authors

Q1. Can the authors provide a direct end-to-end ablation of the material-conditioned dispatch? In particular, how does the proposed predicted-phase routing compare with oracle phase labels and with using a single motion branch for all dynamic objects (e.g., all-rigid, all-TPS, or another generic deformation baseline) under exactly the same perception frontend and renderer? This comparison would materially affect my assessment of whether the central gain comes from material-aware routing itself.

Q2. How should the comparison with globally optimized methods such as MoSca and Shape of Motion be interpreted? These methods are primarily designed to exploit a full observed video for globally consistent 4D reconstruction, whereas PhysGraph is evaluated under increasingly sparse keyframe sampling. Is the intended claim specifically superiority in the sparse-keyframe / large-temporal-gap regime rather than general superiority in dynamic 4D reconstruction? A complementary dense-input comparison would help clarify this point.

Q3. Could the authors additionally report mean and median dynamic-region PSNR, or the corresponding quality distributions, in addition to thresholded recall? The recall metrics are useful for capturing catastrophic failures, but they make it difficult to distinguish robustness from absolute reconstruction quality on successful sequences.

2 comments

We revised the paper to narrow scope, highlight our phase-routing novelty, isolate end-to-end dispatch across 414 runs (Tables 4, 14), clarify dense vs. sparse regimes, and report 90-clip distributions (Tables 2, 10).

1. Narrowed Kinematic Scope and Core Novelty (W1)

We retitled the paper "PhysGraph: Point Graphs for Material-Aware Dynamic Motion Modeling" and revised the Abstract and Secs. 1, 2, 3.3, and 5 around material-aware dynamic motion modeling - surface kinematics rather than contact, support, forces, collisions, or causality (Sec. 1).

On novelty, prior physics-3DGS works simulate forward MPM on static multi-view assets (PhysGaussian, OmniPhysGS) or require studio views with manual material classes (PAC-NeRF), whereas dynamic monocular 3D/4D and VFI methods (MoSca, SoM, MoVieS, UFO-4D, RIFE) force one material-blind motion model onto the whole scene. To our knowledge, PhysGraph is the first dynamic monocular video method to automatically classify objects from images into solid, deformable, and liquid phases (Table 12) and route each segment to a matched solver on one shared kNN graph: locked edge lengths (Solid, GNC SE(3)), elastic residuals (Deformable, SE(3)+TPS), and bipartite transport with 2D signed-distance clipping (Liquid, Sinkhorn OT + SDT, Sec. 3.7).

2. Controlled End-to-End Phase-Dispatch Ablation (W2, Q1)

Holding MonST3R, CoTracker, SAM 3, and the 3DGS renderer strictly fixed, we varied only dynamic routing across s∈{5,10,25}s\in\{5,10,25\} on 6 synthetic scenes (90 runs, GT depth/poses) and 18 stratified DAVIS clips (324 runs across rigid, deformable, liquid+deformable, and articulated categories; Tables 4, 14):

PolicymovPSNRdPSNRMAE(m)Liquid(2)Liq+Def(3)Deform(6)R@15%CE(px)
All-Linear20.1322.560.007314.8718.8015.5587.031.9
All-Rigid21.3822.600.007515.3018.1015.1883.330.7
All-TPS21.4422.600.007315.2018.5815.6785.230.2
All-Liquid23.1623.450.006921.5518.6015.2387.054.8
PhysGraph (Pred.)23.5223.650.004621.5518.8315.6787.030.1
PhysGraph (Oracle)23.5223.650.004621.5518.9715.6787.030.1
  1. Clean-3D specialization: No single branch handles heterogeneous scenes - rigid and TPS solvers break on liquids where point identity is lost, while optimal transport degrades solids/deformables and doubles drift. Routing by phase wins every synthetic metric and cuts 3D depth error by over a third.
  2. Predicted matches Oracle: Across all strides on DAVIS-18, zero-shot routing matches human oracle labels almost identically, as near-rigid misclassifications collapse their TPS residual to rigid motion.
  3. Real multi-phase gains: On multi-phase liquid-plus-deformable DAVIS scenes (where foreground masks exclude water), PhysGraph beats every single-branch baseline, beats All-Rigid on deformables, leads recall, and halves centroid drift vs. All-Liquid (on dry-road vehicles, All-TPS absorbs MonST3R depth-scale drift via its non-rigid residual, Table 14b).

3. Operating Regime vs. Global 4D Optimizers (Q2)

We claim zero-failure robustness, low runtime, and α>1\alpha>1 extrapolation at sparse strides, not superiority over converged 4D optimizers on dense video (Sec. 4.2). On dense frames (s=2s=2, Table 10), 2D warpers and converged 4D optimizers achieve higher pixelwise PSNR by bypassing or amortizing MonST3R noise across many views. As gaps widen (s=3→25s=3\to 25), per-scene 4D optimization becomes ill-conditioned and fails on up to two-thirds of DAVIS clips at over half an hour per clip, whereas PhysGraph solves each pair in one pass with zero failures and an order-of-magnitude speedup.

4. Unified 90-Sequence Distributions on DAVIS (Q3)

Table 10 reports full distributions alongside MoVieS and optimization failure/runtime stats across all 90 DAVIS clips (UFO-4D and 2D VFI also in Table 10):

StridePhysGraph Mean/MedIQRdR@15/@18/@20/@22 (%)MoVieSMoSca/SoM FailTime (Ours vs MoSca)
s=2s=218.39/17.7616.16-20.1288.9/42.2/25.6/16.7------
s=3s=317.49/16.5015.02-19.3175.6/36.7/20.0/12.214.98/14.3811%/17%2.9 vs 52.2m
s=5s=516.29/15.2513.78-18.0351.1/27.8/16.7/6.714.52/14.0511%/17%2.2 vs 48.5m
s=10s=1014.64/13.9912.25-16.5140.0/16.7/6.7/4.413.67/12.9437%/18%1.7 vs 46.2m
s=25s=2513.04/12.3410.99-14.6424.4/6.7/3.3/2.212.60/11.9069%/29%1.4 vs 36.1m

Tight, unimodal IQRs confirm consistent dynamic quality across clips rather than threshold artifacts, paired with metric depth and α>1\alpha>1 extrapolation (Table 3).

Thank you for your detailed and thoughtful response. I appreciate the effort you have put into addressing my comments and concerns. Your clarifications have satisfactorily addressed my previous questions and helped me better understand the proposed approach. After carefully considering your rebuttal, I have decided to maintain my original rating. Thank you again for your thorough response and constructive discussion.

Reviewer_pYMaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes PhysGraph which try to address temporal interpolation and extrapolation of monocular videos captured at large frame gaps using a training-free workflow. It includes 3 models, MonST3R works for depth estimation and camera pose estimation, CoTracker works for dense correspondences, SAM3 works for segmentation and tracking. Each noun maps to a type of material and type of material determines which kind of motion model will be used. The evaluation contains 6 blender synthetic scenes and 90 sequences of DAVIS.

Critical strengths
  1. The motivation and idea is clear and easy to understand. It is intuitive and addresses a real limitation of existing dynamic reconstruction methods which use a single motion model for the whole scene.

  2. The pipeline uses no per-scene optimization, so it has zero failures on all 90 DAVIS sequences. In comparison, MoSca fails on 11–69% of sequences and Shape of Motion on 17–29%, and both get worse as the stride grows. And also this method is pretty efficient than Mosca.

Critical weaknesses
  1. The visualization of Figure 1 is terrible. I feel like this figure is generated by AI. And the correspondence lines, segmentation masks, and kNN graph are inconsistent with the method described (the graph is drawn on the water rather than the object; the swan is labeled deformable yet assigned E=2.5 GPa), and several icons contain pseudo-text. Please replace it with a figure built from real intermediate results.

  2. And I feel the novelty is weak for ICLR venue. Every perception module is off-the-shelf and the physical properties estimated by the VLM have almost no effect.

  3. The main claim rests on recall at a 15 dB PSNR threshold, which is very lenient. At 20 dB the numbers are weak.

Questions for authors

Please take a look of weaknesses.

If possible, could you please show some video demos in sup_mat or project page?

3 comments

We revised the paper to rebuild Fig. 1 from real outputs, isolate phase routing with perception held fixed (Tables 4, 14), report PSNR distributions and @20 dB comparisons across 2D, 3D, and 4D baselines (Tables 2, 3, 10), and add temporal strips (Fig. 8).

1. New Figure 1 from Real Intermediate Results (W1)

We replaced Fig. 1 with a vector diagram built from real outputs on DAVIS blackswan and boat:

  • Consistent perception: Real MonST3R depth, CoTracker correspondence lines, ego-motion residuals, and SAM 3 masks replace all AI icons and pseudo-text.
  • Graphs on objects, no E=2.5 GPaE=2.5\text{ GPa}: Separate 3D kNN graphs sit directly on the objects (boat and swan) rather than on the water and Col. 5 shows actual rendered interpolation and extrapolation frames.

2. Novelty Beyond Off-the-Shelf Perception: Image-Driven Phase Routing (W2)

Perception is modular by design, and we moved VLM physical-property estimation out of the main pipeline (App. D-E.5). Prior dynamic 3D/4D and VFI methods apply one material-blind motion model to the whole scene, while physics-3DGS works only simulate forward MPM on static multi-view assets (PhysGaussian, OmniPhysGS) or require studio views with manual material classes (PAC-NeRF). To our knowledge, PhysGraph is the first dynamic monocular video method to classify objects from images into solid, deformable, and liquid phases (Table 12) and route each object to a matched 3D solver on one shared kNN graph, reused for outlier filtering, elastic kernels, and fluid transport.

Holding MonST3R, CoTracker, SAM 3, and the 3DGS renderer strictly fixed across 414 runs proves the gain comes from phase routing itself (Tables 4, 14):

PolicymovPSNRdPSNRMAE(m)Liquid(2)Liq+Def(3)Deform(6)R@15%CE(px)
All-Linear20.1322.560.007314.8718.8015.5587.031.9
All-Rigid21.3822.600.007515.3018.1015.1883.330.7
All-TPS21.4422.600.007315.2018.5815.6785.230.2
All-Liquid23.1623.450.006921.5518.6015.2387.054.8
PhysGraph (Pred.)23.5223.650.004621.5518.8315.6787.030.1
PhysGraph (Oracle)23.5223.650.004621.5518.9715.6787.030.1

Three conclusions follow: (i) no single motion model handles heterogeneous scenes - rigid and TPS solvers fail on liquids where point identity breaks, whereas optimal transport degrades structured objects and doubles tracking drift; (ii) routing by phase captures the best behavior in each regime, cutting clean-3D depth error by over a third and beating every single-branch baseline on multi-phase water-plus-foreground DAVIS clips; and (iii) zero-shot classification matches human oracle labels, as near-rigid misclassifications collapse their TPS residual to rigid motion. `

3. Addressing the 15 dB vs. 20 dB PSNR Thresholds (W3)

Within dynamic 3DGS (Table 2), PhysGraph already ties MoSca and GaussianPrediction for #1 dPSNR@20 at s=3s=3 (20.0%) and leads all dynamic 3DGS baselines in dPSNR@20 at s=25s=25. Across wide gaps (s=10s=10-2525), large non-linear motion means even a small phase offset drops PSNR below 20 dB despite sharp boundaries (Table 11b). Comparing all 2D, 3D, and 4D paradigms at s=25s=25 shows dR@20 is near zero for every method, making 15-18 dB and full PSNR distributions the informative regime (Tables 2, 10):

Method (s=25s=25)dR@20dR@15Fail3DExtrap.
MoSca / SoM (4D opt.)0.0 / 0.0%11.4 / 13.3%69 / 29%YesYes/No
MoVieS (4D feed-fwd)2.2%15.6%0%YesNo
GaussianPred. (4D opt.)2.9%19.0%0%YesYes
FILM / MoMo (2D VFI)2.2 / 4.4%13.3 / 20.0%0%NoNo
UFO-4D (4D feed-fwd)3.3%24.4%0%YesNo
RIFE (2D VFI)5.6%24.4%0%NoNo
PhysGraph (Ours)3.3%24.4%0%YesYes

Tables 3 and 10 also report Mean/Median dPSNR, IQR, @15/@18/@20 recall, and extrapolation across all 90 DAVIS clips:

StrideMean/Med dPSNRIQRdR@15/@18/@20Extrap dR@15 (Ours vs GP/RepeatKF)
s=2s=218.39/17.7616.16-20.1288.9/42.2/25.6%--
s=3s=317.49/16.5015.02-19.3175.6/36.7/20.0%36.7% vs 30.0/34.4%
s=5s=516.29/15.2513.78-18.0351.1/27.8/16.7%32.2% vs 23.6/27.8%
s=10s=1014.64/13.9912.25-16.5140.0/16.7/6.7%22.9% vs 19.7/22.9%
s=25s=2513.04/12.3410.99-14.6424.4/6.7/3.3%--

4. Video Demos in Supplement / Project Page (Q)

External URLs are disallowed under double-blind review; we will host video demos on our project page upon acceptance. In the PDF supplement, we added frame-by-frame temporal strips across interpolation and extrapolation (App. C, Fig. 8; Fig. 1 Col. 5), showing PhysGraph keeping sharp objects where MoVieS erases fast motion and UFO-4D double-ghosts.

Dear Reviewer,

Did we manage to address your concerns? Please let us know if you need further details.

Thanks,
the Authors

Thank you for the detailed rebuttal and the revised Figure 1. However, the new ablations show that phase routing gives little gain on real videos (e.g., All-TPS and All-Linear match or exceed PhysGraph in dPSNR on DAVIS-18), and the method still trails dedicated interpolation methods such as RIFE in dPSNR@20, mean dPSNR, and dLPIPS. My concerns about novelty and visual quality therefore remain, and I will keep my score of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.