- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
PhysGraph interpolates and extrapolates sparsely sampled monocular videos by using predicted material categories to select rigid SE(3), TPS deformation, or optimal-transport motion models on per-object point graphs. It combines existing perception models with bidirectional Gaussian splatting and evaluates the framework on six synthetic scenes and 90 DAVIS sequences.
Critical strengths
- The material-conditioned framework provides an interpretable way to handle heterogeneous motion, combining rigid, deformable, and liquid models within a shared per-object point-graph representation.
- The evaluation covers six synthetic scenes and 90 DAVIS sequences. The end-to-end DAVIS results include errors from estimated geometry, correspondences, and segmentation, and demonstrate practical robustness and efficiency relative to the tested per-scene optimization methods.
Critical weaknesses
- The method evolves surface points observed in the endpoint frames rather than a complete object representation. Bidirectional fusion can recover surfaces visible at either endpoint, but no mechanism is described for surfaces absent from both endpoints that become visible during interpolation or extrapolation. The evaluation does not establish reliable behavior under substantial occlusion and disocclusion, limiting support for the broader physical scene understanding claim.
- The experiments primarily assess temporal interpolation and extrapolation, yet the baselines largely target dynamic 3D reconstruction. These comparisons are informative, but restricting reconstruction methods to sparse keyframes does not establish competitiveness against dedicated learning-based frame interpolation and video prediction methods.
- The method assigns motion models through a fixed noun-to-phase mapping, although objects within the same category may exhibit different motion structures. If this simplification fails to preserve the underlying motion structure, it remains unclear what physical understanding the semantic mapping provides. Meanwhile, the reported pass rates at PSNR ≥15 dB and ≥20 dB do not by themselves establish convincing visual fidelity or temporal consistency. Together with the acknowledged physically incorrect motion in articulated objects, these results do not sufficiently establish either physical validity or visual quality.
Questions for authors
- How are newly visible surfaces handled when they are absent from both endpoint point clouds? Could the authors evaluate performance specifically in regions undergoing occlusion and disocclusion?
- Could the authors compare against dedicated learning-based frame interpolation and video prediction methods under matched input and temporal-gap settings, to establish the benefit of the proposed geometry and material-processing stages?
- Given the acknowledged physically incorrect motion in articulated objects, could the authors clarify what physical understanding the fixed noun-to-phase mapping provides and what evidence supports its benefits for physical plausibility or visual quality?
3 comments
Thank you for the additional experiments. However, my main concerns remain unresolved.
-
Bidirectional splatting only addresses interpolation. Extrapolation still relies on reconstructed points and image inpainting, without resolving newly visible surfaces.
-
The new results show no clear visual advantages over dedicated interpolation methods, leaving the necessity of the additional 3D reconstruction and motion-model complexity insufficiently justified.
-
The noun-to-phase mapping remains a simple heuristic. The new ablations demonstrate reconstruction improvements but provide no convincing evidence of physically plausible motion.
Therefore, I maintain my original rating for now.
Thank you for clarifying the remaining concerns. Below we address each.
1. Extrapolation and Newly Visible Surfaces
In causal extrapolation (, from keyframes ), newly exposed pixels split into two regimes:
- Previously observed disocclusions (visible in before occlusion at , e.g., background behind an advancing boat or car): PhysGraph resolves these directly in 3D without 2D inpainting by aggregating unoccluded static 3D points across past keyframes (Sec. 3.7). In Table A, observed 3D geometry covers over nine-tenths of extrapolated pixels on synthetic and DAVIS clips, and past-keyframe 3D aggregation substantially lifts disoccluded-wake accuracy before any 2D infill.
- Surfaces never observed in (e.g., unseen backs or new frustum borders): no observation-grounded method, 2D or 3D, can recover geometry never seen by the camera. Synthesizing it requires generative priors, whereas PhysGraph stays observation-grounded to avoid inventing fake geometry (and beats SVD-XT diffusion in accuracy and speed). Telea infill is strictly a fallback on this small never-seen fraction.
2. Necessity of 3D Reconstruction and Motion Models vs. 2D VFI
For fixed-camera 2D interpolation alone, dedicated 2D warpers like RIFE are simpler and avoid monocular depth noise because fixed-ray projection hides 3D depth error along the viewing ray (added for W2/Q2).
PhysGraph instead solves sparse-keyframe 4D dynamic scene modeling, delivering outputs 2D interpolators structurally lack: (i) metric 3D geometry and trajectories (Table 1); (ii) free-viewpoint 3D rendering from novel cameras at held-out times, where 2D VFI cannot run and 3D depth/motion errors appear as parallax misprojections, giving PhysGraph a clear lead over 3D Linear and All-Rigid (Table A); and (iii) causal extrapolation (), where RIFE, FILM, MoMo, and UFO-4D cannot run (Table 3). Within 4D methods (MoSca, SoM, GP, MoVieS, UFO-4D), PhysGraph eliminates crashes (Fig. 3), runs much faster than 4DGS optimizers, and avoids MoVieS erasure and UFO-4D ghosting (Fig. 8).
Table A: Causal Extrapolation 3D Coverage, Pure-3D Wake Recovery, and Novel-View 4D
| Setting | Metric | Baseline / Comparison | PhysGraph |
|---|---|---|---|
| Synth Extrap () | Causal 3D observed () | 4.5/5.9/8.6% never seen | 95.6/94.1/91.4% in 3D |
| DAVIS-18 Extrap () | Direct 3D splats () | 7.2% never-seen fallback | 92.8% direct 3D splats |
| DAVIS-18 Wake (, no infill) | Wake PSNR (pure 3D) | 12.25 dB (KF- 3D only) | 13.49 dB (+Past-KF 3D) |
| Novel-View 4D (, ) | Moving-object PSNR | 2D VFI: N/A | Lin: 23.07/19.30 | Rig: 24.92/20.92 | 28.62 / 24.16 dB |
3. Beyond Noun Lookup and Direct Evidence of Physically Plausible Motion
- Open-ended visual reasoning (no noun table required): Appendix C.3 (Table 12, Fig. 7) already implements direct image-based VLM classification on class-agnostic masks with zero noun list, judging phase from visual context (e.g., water vs. ice/reflections). On our 18-clip DAVIS benchmark (Table B), Qwen2.5-VL-7B visual labels strongly agree with oracle labels and match SAM 3 lookup and oracle routing. SAM 3 is simply a fast default.
- Direct 3D physical invariants (not PSNR): Table B evaluates 3D physical invariants and simulator errors against the Blender rigid, cloth/FEM, and fluid engines:
- Solids: Linear shrinks rotating bodies along chords, TPS bends under noise, and OT scrambles vertices (nearly doubling DAVIS drift); SE(3) keeps exact zero isometric strain and lowest 3D rigid error.
- Deformables: Rigid freezes bending while Linear/OT scramble neighborhoods; SE(3)+TPS preserves local topology with lowest 3D cloth/soft-body error.
- Liquids: Fixed-graph models stretch pouring fluids into mid-air sheets; OT conserves 100% marginal mass and cuts 3D free-surface error over two- to three-fold.
Table B: 3D Physical Invariants (, mm) and VLM Routing (DAVIS-18)
| Policy / Phase Source | Rig Strain | Rig EPE | Def kNN | Def EPE | Liq Chamfer | DAVIS-18 dPSNR / CE |
|---|---|---|---|---|---|---|
| All-Linear | 5.31% | 67.1 mm | 8.50% | 58.8 mm | 86.7 mm | 15.07 / 31.9 px |
| All-Rigid (SE(3)) | 0.00% | 62.4 mm | 0.03% (frozen) | 60.4 mm | 134.9 mm | 14.66 / 30.7 px |
| All-TPS (SE(3)+TPS) | 1.88% | 65.6 mm | 6.31% | 58.2 mm | 149.4 mm | 15.13 / 30.2 px |
| All-Liquid (OT) | 9.42% | 92.3 mm | 10.80% | 90.1 mm | 40.9 mm | 14.23 / 54.8 px |
| PhysGraph (SAM 3) | 0.00% | 62.4 mm | 6.31% | 58.2 mm | 40.9 mm | 14.91 / 30.1 px |
| PhysGraph (Qwen2.5-VL, 0 nouns) | -- | -- | -- | -- | -- | 14.93 / 30.3 px (91.8% Orac.) |
| PhysGraph (Oracle) | 0.00% | 62.4 mm | 6.31% | 58.2 mm | 40.9 mm | 14.92 / 30.1 px |
We revised the paper to narrow scope to material-conditioned motion modeling (Secs. 1, 3.7), isolate splatting from infill across visibility strata (Table 15), benchmark VFI, diffusion, and 4D models (Tables 9-11b, Fig. 8), and ablate phase dispatch across 414 runs (Tables 4, 14).
1. Doubly Occluded Surfaces, Disocclusions, and Extrapolation (W1, Q1)
PhysGraph advects endpoint Gaussians without hallucinating unseen 3D interiors (Sec. 3.7): single-endpoint disocclusions (A⊕B) are recovered in 3D by the bidirectional Gaussian union, while doubly occluded holes (A∪B, unseen in both A,B) composite over (1−α)IA+αIB in interpolation and pull past static points plus Telea inpainting in extrapolation (α>1).
Table 15 isolates Forward-Only, Bidir (no inf.), and Bidir + Infill across 3D visibility strata (4 mesh scenes) and 18 DAVIS clips (s=5/10/25), plus causal extrapolation (Table 3):
Two conclusions follow: (i) doubly occluded pixels form under 2% of interpolation frames, while bidirectional union sharply lifts single-endpoint disocclusions over forward-only splatting (and beats 2D cross-dissolve on moving objects at every stride), with infill boosting doubly occluded holes and handheld borders; and (ii) in causal extrapolation, explicit 3D motion models beat both GaussianPrediction (at 3x lower runtime) and copying the last keyframe.
2. Learning-Based VFI, Diffusion, and 4D Baselines (W2, Q2)
We evaluated FILM, MoMo, RIFE, SVD-XT GenInb, MoVieS, UFO-4D, and 3D Linear under matched keyframes across all 90 DAVIS clips and checkerboard synthetic renders (Tables 9-11a):
At dense strides, 2D warping avoids 1-px splat discretization on checkerboards; at wide gaps (s=25), PhysGraph overtakes 3D Linear, FILM, and MoMo and matches RIFE with metric depth. On DAVIS, RIFE and UFO-4D hold a slight mean-dPSNR lead by bypassing/absorbing MonST3R noise, yet neither extrapolates (α>1) and RIFE lacks 3D output. PhysGraph ties both for #1 sparse recall, beats FILM, MoVieS, and MoMo at large gaps, and beats SVD-XT on 10/10 clips (18x faster).
3. Phase Routing and Visual/Temporal Fidelity (W3, Q3)
PhysGraph keeps sharper dynamic edges than 4D baselines (avoiding MoVieS erasure and UFO-4D ghosting, Fig. 8), beats MoVieS on dLPIPS/tOF at moderate strides, and holds flat tOF across wide gaps.