acceptodds
Under review as a conference paper at ICLR 2027

Chronofy: Pose-Aware Generative 4D Reconstruction from Videos

Abstract

Recent generative reconstruction methods effectively reconstruct static 3D content but often exhibit pronounced inter-frame jitter when applied to dynamic content. We present Chronofy, a pose-aware 4D generative reconstruction framework that jointly predicts complete dynamic objects and unified camera-space poses from single-view videos. Built upon SAM3D, Chronofy transfers its powerful generative reconstruction priors to the 4D domain through three key designs: (i) a pose-aware 4D representation that encodes frame-dependent motion in spatiotemporal structured latents and uses a unified pose for sequence-level placement; (ii) a generative reconstruction model that utilizes a spatiotemporal mixture of transformers for temporal modeling and frame-wise correspondence between shape and pose; and (iii) a dynamic Gaussian decoder that combines temporal latent interactions with time-varying Gaussian attributes to reduce inter-frame flickering. To preserve occlusion robustness with limited 4D training data, we use temporal masking augmentation, while layout test-time optimization refines multi-object spatial alignment. Experiments on Motion-80, Kubric-4D, and our simulated real-world Chrono-64 datasets demonstrate that Chronofy substantially improves spatiotemporal consistency in both 4D object generation and 4D scene reconstruction.

Primary area:
generative models
Keywords:
4D generation, 4D reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XhNTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Chronofy, a framework for joint 4D object reconstruction and camera-space pose estimation from monocular videos. Building upon SAM3D, the authors introduce a sequence-level unified pose representation, spatiotemporal transformers, and a dynamic Gaussian decoder to improve temporal consistency. Temporal masking augmentation and test-time pose refinement are further used to handle occlusions and improve scene alignment. Experiments on Motion-80, Kubric-4D, and Chrono-64 demonstrate improved temporal consistency and competitive reconstruction quality compared to existing methods.

Critical strengths
  1. The problem is meaningful. Existing video-to-4D generation and 4D reconstruction methods do not directly produce complete 4D objects with camera-space poses for scene composition. The proposed sequence-level normalization with a unified pose is a simple but effective way to address the shape-pose ambiguity in frame-wise SAM3D, and Table 4 provides a useful ablation.

  2. The improvement in temporal consistency is substantial and clearly visible in the supplementary videos. The authors also report results where their method performs worse (Kubric ADD-S in Table 7) and discuss failure cases.

Critical weaknesses
  1. The comparisons are not entirely fair. In Table 2, Chronofy uses Layout TTO while SAM3D does not, even though TTO builds on SAM3D's pose refinement. When both use TTO, SAM3D performs better on 3D IoU, [email protected], and 2D IoU (Table 7). Similarly, Chronofy and SAM3D use GT pointmaps in Table 1, while other baselines only use RGB. With DA3 pointmaps (Table 6), Chronofy's advantage over MORPHOS becomes much smaller (LPIPS 0.180 vs. 0.176, FVD 302 vs. 308). The paper also lacks pose accuracy results with estimated depth, which is more realistic. These issues make it difficult to assess the claimed improvements in object quality and pose accuracy.

  2. The ablations do not fully support the architectural choices. Table 3 lacks a baseline that fine-tunes SAM3D frame-wise using the same 18K training samples and augmentation. Much of the per-frame improvement also appears to come from augmentation (row d). Furthermore, adding the dynamic Gaussian decoder worsens all metrics except FVD in the full system (Table 3, b→c), despite improving results with GT latents (Table 5). This discrepancy deserves more discussion. Reporting variance across different seeds would also help establish the reliability of the results.

Questions for authors
  1. Please include SAM3D w/ TTO in Table 2 for both datasets, and clarify whether Chronofy uses TTO on Kubric-4D.
  2. Please report the results in Tables 1 and 2 using DA3 pointmaps for both Chronofy and SAM3D, including pose accuracy.
  3. Please add a frame-wise fine-tuned SAM3D baseline to Table 3, trained with the same data and augmentation.
  4. Is the dynamic Gaussian decoder trained only on GT latents? Why does it hurt most metrics in Table 3 (b→c) but help in Table 5?
  5. The supplementary material states that Mesh4D uses 6 frames. How is FVD computed against the 16-frame ground truth?
5 comments
AuthorsOpenReview ↗

Response to Reviewer XhNT (Part 1 / 3)

We really appreciate the reviewer for the detailed and constructive feedback. We appreciate your recognition of the importance of jointly reconstructing complete dynamic objects and their camera-space poses, as well as the temporal-consistency gains shown in our supplementary videos.

In response, we provide additional control experiments and evaluation details on pointmap inputs, layout TTO, and frame-wise adaptation versus temporal modeling. We also clarify the decoder ablations and Mesh4D’s FVD evaluation protocol. Below, we address each concern in detail.


Q1: The comparisons are not entirely fair. In Table 2, Chronofy uses Layout TTO while SAM3D does not, even though TTO builds on SAM3D's pose refinement. When both use TTO, SAM3D performs better on 3D IoU, [email protected], and 2D IoU (Table 7).

A1: Thank you for pointing this out. Scene reconstruction should indeed be evaluated under matched TTO settings. We report both methods with and without layout TTO on Chrono-64 and Kubric-4D in R3-Table1 and R3-Table2, and will update the revised manuscript accordingly.

R3-Table1: Quantitative comparison of 4D scene reconstruction on Chrono-64 dataset.

Method3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
Video + Masks
COM4D————
Video + Masks + GT Pointmaps
ActionMesh + MegaPose0.549616.24010.11320.7441
ActionMesh + FoundationPose0.615014.78530.05780.8877
Chronofy + FoundationPose0.601413.57890.06130.8496
SAM3D (w/o layout TTO)0.586213.99370.04770.9414
Chronofy (w/o layout TTO)0.584112.72630.04820.9395
SAM3D (w/ layout TTO)0.603612.85370.04700.9502
Chronofy (w/ layout TTO)0.597712.59250.04610.9463
Video + Masks + DA3 Pointmaps
SAM3D0.561414.62720.05280.9209
Chronofy0.576213.25180.05070.9287

Under identical GT pointmaps and matched TTO settings, Chronofy and SAM3D trade leads across metrics. These results confirm that Chronofy preserves competitive camera-space pose accuracy while substantially improving temporal consistency in dynamic object generation.

R3-Table2: Quantitative comparison of 4D scene reconstruction on Kubric-4D test set.

Method3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
Video + Masks
COM4D0.061239.25161.47470.0245
Video + Masks + GT Pointmaps
ActionMesh + MegaPose0.469432.28340.16250.5439
ActionMesh + FoundationPose0.471330.72620.14310.6107
Chronofy + FoundationPose0.452226.38550.13740.6756
SAM3D (w/o layout TTO)0.577616.56720.05920.8728
Chronofy (w/o layout TTO)0.571415.93820.08150.8546
SAM3D (w/ layout TTO)0.580316.02470.05980.8745
Chronofy (w/ layout TTO)0.581715.49260.07710.8649
Video + Masks + DA3 Pointmaps
SAM3D0.548217.29180.07160.8697
Chronofy0.559715.87410.09640.8470

Remaining gaps on certain pose metrics partly reflect the limited scale of our 4D training data. To verify this, we synthesized 25K additional 4D samples following SAM3D’s Flying Occlusion protocol and continued training. As shown in R3-Table3, scaling 4D training improves Chronofy’s pose accuracy on Kubric-4D under identical inputs and TTO settings, surpassing SAM3D in 3D IoU, ICP-Rot., and ADD-S @ 0.1.

R3-Table3: Effect of additional Flying Occlusion training on Chronofy’s 4D scene reconstruction on the Kubric-4D test set. All methods use GT pointmaps and are evaluated without layout TTO.

Method3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
SAM3D0.577616.56720.05920.8728
Chronofy0.571415.93820.08150.8546
Chronofy (+25K Flying Occlusion 4D samples)0.584215.17390.06070.8735

AuthorsOpenReview ↗

Response to Reviewer XhNT (Part 2 / 3)

Q2: Similarly, Chronofy and SAM3D use GT pointmaps in Table 1, while other baselines only use RGB. With DA3 pointmaps (Table 6), Chronofy's advantage over MORPHOS becomes much smaller. The paper also lacks pose accuracy results with estimated depth, which is more realistic.

A2: Evaluating with predicted pointmaps indeed better reflects practical scenarios. We have added 4D object generation results for Chronofy and SAM3D under identical DA3-predicted pointmaps on Chrono-64 and Motion-80 in R3-Table4 and R3-Table5.

R3-Table4: Quantitative comparison of 4D object generation on Chrono-64 dataset.

MethodLPIPS ↓CLIP-S ↑PSNR ↑SSIM ↑FVD ↓
Video + Masks
L4GM0.18850.715811.13400.6555345.53
GVFDiffusion0.25300.59289.19480.6140858.50
SS4D0.19320.736510.03020.6688342.23
MORPHOS0.17580.790210.63260.6592307.93
Mesh4D0.32010.61559.30200.6125584.96
Motion3240.31840.65479.19810.5986388.23
Video + Masks + GT Pointmaps
SAM3D0.20750.839213.19020.7725724.04
Chronofy0.16350.851213.94880.7647263.19
Video + Masks + DA3 Pointmaps
SAM3D0.22990.806711.88830.7583783.37
Chronofy0.17980.824112.50630.7596301.68

Compared with MORPHOS on Chrono-64, using DA3 pointmaps narrows Chronofy’s FVD lead and yields slightly worse LPIPS, while PSNR and SSIM remain clearly higher. More importantly, under identical DA3 conditioning, Chronofy still outperforms SAM3D by a wide margin across both datasets, confirming that our 4D extension remains effective without GT geometry.

R3-Table5: Quantitative comparison of 4D object generation on Motion-80 dataset.

MethodLPIPS ↓CLIP-S ↑PSNR ↑SSIM ↑FVD ↓
Video + Masks
L4GM0.16020.837012.52250.7276275.19
GVFDiffusion0.24260.726312.44250.6962436.67
SS4D0.14580.864114.93830.7125267.74
MORPHOS0.17610.831613.65910.7051285.31
Mesh4D0.21530.781912.52810.6877563.01
Motion3240.22310.803110.21870.6894308.07
Video + Masks + GT Pointmaps
SAM3D0.19370.843514.73620.7543784.31
Chronofy0.15130.877615.68360.7726243.46
Video + Masks + DA3 Pointmaps
SAM3D0.21840.820912.35800.7136861.40
Chronofy0.16700.842714.15140.7375275.52

For pose accuracy with estimated depth, R3-Table1 and R3-Table2 also report DA3-conditioned scene reconstruction on Chrono-64 and Kubric-4D, where predictions are transformed into the GT coordinate frame and inevitably incur geometry and alignment errors. Despite the expected drop relative to GT pointmaps, Chronofy maintains competitive camera-space pose accuracy alongside SAM3D, complementing its temporal-consistency gains in dynamic object generation.


Q3: The ablations do not fully support the architectural choices. Table 3 lacks a baseline that fine-tunes SAM3D frame-wise using the same 18K training samples and augmentation.

A3: We have added two controls: (a) frame-wise SAM3D fine-tuned on the same 18K samples with the same augmentation, and (b) training-free SAM3D with shared noise and the same Markley pose averaging in R3-Table6. Both fine-tuned models are evaluated at 20K training steps.

R3-Table6: Controlled comparisons for 4D object generation on Chrono-64.

MethodLPIPS ↓CLIP-S ↑PSNR ↑SSIM ↑FVD ↓
(a)0.18460.845913.42150.7764526.72
(b)0.21180.835413.43650.7519698.36
Chronofy0.18170.847613.58420.7582347.45

While frame-wise fine-tuning improves SAM3D’s per-frame metrics, Chronofy achieves the best temporal consistency across these variants. These results support the benefits of learned cross-frame modeling beyond data adaptation and augmentation alone.


AuthorsOpenReview ↗

Response to Reviewer XhNT (Part 3 / 3)

Q4: Much of the per-frame improvement also appears to come from augmentation (row d). Furthermore, adding the dynamic Gaussian decoder worsens all metrics except FVD in the full system (Table 3, b→c), despite improving results with GT latents (Table 5). This discrepancy deserves more discussion.

A4: This discrepancy stems from a train-inference input mismatch: the dynamic Gaussian decoder is trained exclusively on structured latents encoded from GT assets, whereas at inference it decodes predicted latents from the generative model. Table 5 isolates decoding on clean GT latents, while Table 3 evaluates the full generative pipeline.

On clean GT latents in Table 5, the decoder improves both spatial and temporal quality, raising SSIM from 0.9480 to 0.9628 and cutting FVD from 473.82 to 157.66. In Table 3 (b→c), however, 4D adaptation without occlusion augmentation yields incomplete predictions in occluded regions, where the dynamic decoder introduces artifacts that lower per-frame scores. Even so, the decoder still reduces FVD from 403.62 to 289.36.

We acknowledge that occlusion augmentation provides the largest incremental PSNR gain (+1.08 dB), alleviating these occlusion artifacts to restore SSIM from 0.6927 to 0.7647 and further lower FVD to 263.19. Temporal modeling and occlusion augmentation therefore play complementary roles: temporal modeling drives motion coherence and video quality, while occlusion augmentation preserves single-frame fidelity under occlusion.


Q5: The supplementary material states that Mesh4D uses 6 frames. How is FVD computed against the 16-frame ground truth?

A5: The originally reported FVD (584.96) uses forward–reverse repetition to extend both Mesh4D’s six-frame predictions and the corresponding six GT frames to 16 frames, matching the evaluation length used for other methods. R3-Table7 additionally reports evaluation using Mesh4D’s original six-frame output and the first six GT frames, yielding an FVD of 516.63.

R3-Table7: Effect of temporal extension on Mesh4D evaluation on Chrono-64 dataset.

MethodsFVD ↓
Mesh4D (forward–reverse repetition)584.96
Mesh4D (original 6 frames)516.63

We sincerely thank the reviewer for the detailed and constructive feedback. If any concerns remain unaddressed, we would be glad to provide further clarification.

Reviewer_qpDaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Chronofy extends SAM3D, which jointly generates an object’s shape and camera-space pose from a single image, to single-view videos. One shared bounding box normalizes all frames, so the per-frame structured latents carry all of the motion and a single unified pose places the whole sequence in camera space. The model temporally inflates SAM3D’s transformers and restricts each frame’s pose token to that frame’s shape tokens. It decodes dynamic Gaussians with lifespan and velocity, and adds occlusion augmentation and layout test-time optimization (TTO) for multi-object scenes. On Motion-80, Kubric-4D and the new Chrono-64, it reports much lower FVD than frame-wise SAM3D and 4D generators while keeping layout accuracy close to SAM3D.

Critical strengths
  1. Relevant, well-framed problem. The paper targets complete 4D objects placed in camera space, between geometry-based 4D reconstruction (aligned but incomplete) and 4D generation (complete but unplaced).
  2. Rich experiments. These include seven video-to-4D baselines, pose-estimation pipelines, COM4D, Lift4D, MoVieS, NeoVerse, ablations, a user study and runtimes. Negative results and failure cases are reported candidly.
Critical weaknesses
  1. The experimental results do not isolate the claimed gains.

    • In Table 1, Chronofy and SAM3D receive GT pointmaps (A.4). With DA3 pointmaps (Table 6), the FVD and LPIPS lead over MORPHOS nearly vanishes (301.68 vs. 307.93, 0.1798 vs. 0.1758), and LPIPS falls behind it (0.1798 vs. 0.1758). Alignment by four yaw angles with GT-alpha crops likely penalizes baselines rendered in their own canonical frame.
    • In Table 2, Chronofy uses layout TTO and SAM3D does not (cf. Table 7). With TTO for both, SAM3D is better on 3D IoU, [email protected] and 2D IoU. Chronofy is also worse on Kubric-4D ADD-S (0.0771 vs. 0.0592). All layout metrics use GT pointmaps, and 2D IoU is the criterion TTO itself uses to accept a refinement (A.6). This supports parity with SAM3D, not improvement.
    • In Table 3, occlusion augmentation gives the largest per-frame gain (+1.08 dB PSNR), while the MoT and decoder lower SSIM (0.7725 → 0.6927). No per-frame fine-tuned SAM3D or training-free temporal SAM3D baseline is reported.
  2. Scope weakness in large-range dynamics, long horizons and dynamic cameras. These are more general limitations in current problem setting

    • One bounding box must contain the whole trajectory, so the object’s share of the fixed-resolution grid shrinks as motion range, deformation extent or clip length grows. The method therefore suits short clips (16 training frames, at most 32 at inference; A.7) without large-range motion or deformation. Similarly, MORPHOS uses the same normalization and drops training sequences in which the object would occupy too few voxels.
    • The unified pose is defined in one camera frame, so the formulation assumes a static camera, as do the training data, Chrono-64 and Kubric-4D. I see no easy extension to a single moving camera, let alone several. That would need world-frame inputs, camera conditioning and moving-camera training data, as in feed-forward models that take posed video (4DGT, BTimer), or an explicit camera-motion term (DreamScene4D).
  3. Missing baselines and attribution.

    • DreamScene4D (NeurIPS 2024) is not cited nor compared. It generates complete multi-object 4D scenes from monocular video, reports results on DAVIS and Kubric, and factorizes motion into object-centric deformation, object-to-world transformation and camera motion. It warrants at least a qualitative comparison
    • The paper names 4DGT (NeurIPS 2025) as the decoder’s inspiration, and it is a natural Table 12 baseline. BTimer, V2M4 and ShapeGen4D should also be discussed.
    • The sequence normalization is identical to MORPHOS’s. Sec. 3.1 and A.3 use the same T-SLAT normalization: a union bounding box over all frames, one center and scale, and a [−0.5, 0.5]³ canonical space. As for temporal machinery, SS4D already folds time into the token axis, alternates shifted temporal windows, adds temporal RoPE, temporally aligns its decoder, and masks conditioning frames.
Questions for authors
  1. I recommend report Table 1 with DA3 pointmaps for all pointmap-conditioned methods on both datasets. Please also state each baseline’s exact alignment (scale, translation, elevation), ideally with metrics computed after a per-method similarity alignment.
  2. I recommend report Table 2 with matched TTO settings, and state whether the Kubric-4D Chronofy row uses TTO. Please also add layout metrics with DA3 pointmaps.
  3. Consider adding two controls: SAM3D fine-tuned per frame on the same data and augmentation, and SAM3D with shared noise plus the same Markley pose averaging.
  4. Why is design (c) better than rigid alignment (b)?
    • Fig. 3 plots adjacent-frame L2 as a ratio to (c), so (c) equals 1 by construction, and the measured quantity is undefined.
    • If pose is included, (c)’s variation is zero by definition, so the plot mainly reflects per-frame pose stability.
    • If normalized latents are measured, dividing by the full trajectory extent alone explains the ordering.
    • Table 4 trains for only 4K steps on 2K sequences, tests on Chrono-64 only, and presumably lets (c) average 16 predictions of one pose.
    • The failures arise under rigid or fast motion (Kubric ADD-S, Fig. 12b). That is where (b) should help, and prior work handles object-to-world motion separately from deformation (DreamScene4D, V2M4).

Additional feedback (not part of my assessment)

  • Eq. 3 lets pose tokens attend to all T frames, but Table 13 lists an 8-frame MoT window.
  • The Fig. 11 text claims the best visual quality throughout, yet in Table 12 SAM3D has higher CLIP-S and NeoVerse higher PSNR.
  • P denotes pointmaps, the unified pose and pose tokens, and t denotes both time and translation.
6 comments
AuthorsOpenReview ↗

Response to Reviewer qpDa (Part 1 / 5)

We sincerely appreciate your insightful and professional comments. We are encouraged that you found our focus on reconstructing complete 4D objects in camera space to be relevant and meaningful. We are also truly grateful for your remarks on our extensive experiments.

In addition to your acknowledgement of our paper’s main contributions, we have carefully considered your constructive feedback and questions on gain isolation, dynamic cameras, and missing baselines. To fully address your concerns, we provide detailed discussions and experiments below.


Q1: In Table 1, Chronofy and SAM3D receive GT pointmaps (A.4). With DA3 pointmaps (Table 6), the FVD and LPIPS lead over MORPHOS nearly vanishes, and LPIPS falls behind it. Alignment by four yaw angles with GT-alpha crops likely penalizes baselines rendered in their own canonical frame.

A1: We have added results for all pointmap-conditioned baselines using predicted DA3 pointmaps on Chrono-64 and Motion-80, and explicitly labeled the input conditions for every method in R2-Table1 and R2-Table2.

R2-Table1: Quantitative comparison of 4D object generation on Chrono-64 dataset.

MethodLPIPS ↓CLIP-S ↑PSNR ↑SSIM ↑FVD ↓
Video + Masks
L4GM0.18850.715811.13400.6555345.53
GVFDiffusion0.25300.59289.19480.6140858.50
SS4D0.19320.736510.03020.6688342.23
MORPHOS0.17580.790210.63260.6592307.93
Mesh4D0.32010.61559.30200.6125584.96
Motion3240.31840.65479.19810.5986388.23
Video + Masks + GT Pointmaps
SAM3D0.20750.839213.19020.7725724.04
Chronofy0.16350.851213.94880.7647263.19
Video + Masks + DA3 Pointmaps
SAM3D0.22990.806711.88830.7583783.37
Chronofy0.17980.824112.50630.7596301.68

R2-Table2: Quantitative comparison of 4D object generation on Motion-80 dataset.

MethodLPIPS ↓CLIP-S ↑PSNR ↑SSIM ↑FVD ↓
Video + Masks
L4GM0.16020.837012.52250.7276275.19
GVFDiffusion0.24260.726312.44250.6962436.67
SS4D0.14580.864114.93830.7125267.74
MORPHOS0.17610.831613.65910.7051285.31
Mesh4D0.21530.781912.52810.6877563.01
Motion3240.22310.803110.21870.6894308.07
Video + Masks + GT Pointmaps
SAM3D0.19370.843514.73620.7543784.31
Chronofy0.15130.877615.68360.7726243.46
Video + Masks + DA3 Pointmaps
SAM3D0.21840.820912.35800.7136861.40
Chronofy0.16700.842714.15140.7375275.52

Performance with predicted pointmaps: Under DA3 pointmaps on Chrono-64, Chronofy’s FVD lead over MORPHOS narrows and its LPIPS is slightly worse, though it still achieves clearly higher PSNR and SSIM. Crucially, under identical DA3 inputs, Chronofy outperforms SAM3D by a large margin in both FVD and LPIPS across both Chrono-64 and Motion-80, confirming the effectiveness of our 4D extension without GT geometry.

Evaluation protocol fairness: We apologize for the brief description in the original paper and will clarify the full protocol in the revision. To account for differences in native position, scale, and azimuth across canonical-frame baselines, we re-normalize each predicted 4D object using a single translation and isotropic scale from its sequence-wide union bounding box, and render all methods under shared camera intrinsics and distance. Following MORPHOS, we choose the yaw in {0∘,90∘,180∘,270∘}\{0^\circ, 90^\circ, 180^\circ, 270^\circ\} minimizing first-frame L2 error on the input view, keep it fixed across the sequence, and evaluate only the three novel views. Following SS4D, image-level metrics apply an identical GT bounding-box crop to reduce background bias, whereas FVD is computed on full, uncropped frames and is unaffected by cropping. We will clarify these details in the revision. Our supplementary material further illustrates Chronofy’s temporal-consistency improvements, while our matched scene reconstruction evaluations support competitive camera-space pose accuracy.


AuthorsOpenReview ↗

Response to Reviewer qpDa (Part 2 / 5)

Q2: In Table 2, Chronofy uses layout TTO and SAM3D does not (cf. Table 7). With TTO for both, SAM3D is better on 3D IoU, [email protected] and 2D IoU. Chronofy is also worse on Kubric-4D ADD-S. All layout metrics use GT pointmaps, and 2D IoU is the criterion TTO itself uses to accept a refinement (A.6). This supports parity with SAM3D, not improvement.

A2: To clarify the pointmap sources and layout TTO usage in Table 2, we provide a clearer breakdown across both datasets in R2-Table3 and R2-Table4, explicitly disentangling the effects of pointmap conditions and layout TTO.

R2-Table3: Quantitative comparison of 4D scene reconstruction on Chrono-64 dataset.

Method3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
Video + Masks
COM4D————
Video + Masks + GT Pointmaps
ActionMesh + MegaPose0.549616.24010.11320.7441
ActionMesh + FoundationPose0.615014.78530.05780.8877
Chronofy + FoundationPose0.601413.57890.06130.8496
SAM3D (w/o layout TTO)0.586213.99370.04770.9414
Chronofy (w/o layout TTO)0.584112.72630.04820.9395
SAM3D (w/ layout TTO)0.603612.85370.04700.9502
Chronofy (w/ layout TTO)0.597712.59250.04610.9463
Video + Masks + DA3 Pointmaps
SAM3D0.561414.62720.05280.9209
Chronofy0.576213.25180.05070.9287

R2-Table4: Quantitative comparison of 4D scene reconstruction on Kubric-4D test set.

Method3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
Video + Masks
COM4D0.061239.25161.47470.0245
Video + Masks + GT Pointmaps
ActionMesh + MegaPose0.469432.28340.16250.5439
ActionMesh + FoundationPose0.471330.72620.14310.6107
Chronofy + FoundationPose0.452226.38550.13740.6756
SAM3D (w/o layout TTO)0.577616.56720.05920.8728
Chronofy (w/o layout TTO)0.571415.93820.08150.8546
SAM3D (w/ layout TTO)0.580316.02470.05980.8745
Chronofy (w/ layout TTO)0.581715.49260.07710.8649
Video + Masks + DA3 Pointmaps
SAM3D0.548217.29180.07160.8697
Chronofy0.559715.87410.09640.8470

Even without layout TTO, Chronofy performs on par with SAM3D across both datasets and consistently achieves lower ICP-Rot, with minor gaps in 3D IoU and ADD-S attributable to smaller training data scale. This confirms that Chronofy’s spatial accuracy stems from our unified sequence-level pose modeling rather than extra TTO refinement. Likewise, under predicted DA3 pointmaps, where predictions are transformed into the GT coordinate frame for evaluation and inevitably incur geometry and alignment errors, Chronofy still retains competitive camera-space layout accuracy relative to SAM3D.

Overall, these controlled comparisons align directly with our central claim: extending SAM3D into a temporally coherent 4D generative reconstruction framework while preserving competitive camera-space layout accuracy.


Q3: In Table 3, occlusion augmentation gives the largest per-frame gain (+1.08 dB PSNR), while the MoT and decoder lower SSIM (0.7725 → 0.6927). No per-frame fine-tuned SAM3D or training-free temporal SAM3D baseline is reported.

A3: 4D adaptation without occlusion augmentation weakens SAM3D’s completion prior in occluded regions, causing the dynamic Gaussian decoder to introduce artifacts that lower intermediate SSIM. Adding occlusion augmentation resolves this issue, yielding the largest PSNR gain (+1.08 dB) and restoring SSIM from 0.6927 to 0.7647 (close to SAM3D’s 0.7725). Importantly, even before augmentation, adding MoT and the dynamic decoder already reduces FVD from 724.04 to 403.62 and 289.36, with augmentation further lowering it to 263.19.

Following your suggestion, we evaluate both requested controls on Chrono-64 in R2-Table5: (a) per-frame fine-tuned SAM3D, evaluated alongside Chronofy at 20K steps under identical training data and occlusion augmentation to isolate cross-frame modeling from data adaptation, and (b) training-free SAM3D with shared noise and identical Markley pose averaging.

R2-Table5: Additional ablation of 4D object generation on Chrono-64 dataset.

MethodLPIPS ↓CLIP-S ↑PSNR ↑SSIM ↑FVD ↓
(a)0.18460.845913.42150.7764526.72
(b)0.21180.835413.43650.7519698.36
Chronofy0.18170.847613.58420.7582347.45
AuthorsOpenReview ↗

Response to Reviewer qpDa (Part 3 / 5)

Frame-wise fine-tuning improves SAM3D’s per-frame metrics, but Chronofy still achieves the best temporal consistency among the compared variants. This advantage also holds over SAM3D with shared noise and Markley pose averaging. These controls support the benefit of cross-frame modeling, while Table 3 shows that occlusion augmentation helps recover per-frame quality during 4D adaptation.


Q4: One bounding box must contain the whole trajectory, so the object’s share of the fixed-resolution grid shrinks as motion range, deformation extent or clip length grows. The method therefore suits short clips without large-range motion or deformation.

A4: Our pose-aware 4D representation is well suited to pronounced non-rigid motion, such as articulation and body bending, within a compact spatial region while recovering a unified camera-space pose. By jointly modeling deformation and displacement in a shared space, it reduces inconsistencies between frame-wise shape and pose predictions, as illustrated in Appendix Table 4 and Figures 13 and 15.

While large translations indeed enlarge the sequence bounding box and leave fewer voxels for object details, complex deformation does not necessarily imply a large spatial footprint. The key constraint is spatial extent relative to object size rather than deformation complexity alone. Deformations that substantially expand the bounding box face the same resolution limit, whereas rapid pose changes introduce separate reconstruction challenges discussed in Appendix A.7.

Similarly, sequence length alone does not determine spatial resolution, since repeated motion within the same region need not enlarge the bounding box. Trained on 16-frame clips, our model currently supports up to 32 frames, beyond which higher computational cost and long-term temporal consistency become the primary bottlenecks.


Q5: The unified pose is defined in one camera frame, so the formulation assumes a static camera, as do the training data, Chrono-64 and Kubric-4D. I see no easy extension to a single moving camera, let alone several.

A5: Our current formulation indeed defines the unified pose in a fixed camera frame, with training and main evaluations conducted on static-camera sequences. For modest camera motion, however, the model can still yield useful approximate reconstructions, as shown in Appendix A.2 and Figure 15, where we use DA3-estimated poses and pointmaps to align per-frame outputs into a shared world frame.

That said, handling large camera motions requires explicitly decoupling camera and object motion. Because our current model does not learn this decomposition, our initial qualitative examples do not yet establish robustness to large viewpoint changes.

A natural extension to a single moving camera is to define the unified pose from normalized space to world coordinates while using per-frame camera transforms to model viewpoint changes. This would involve transforming input pointmaps into a consistent world frame, adding camera-pose conditioning, and training on dynamic camera videos, thereby preserving our shared object-dynamics representation alongside explicit camera motion. Multi-camera setups would further require inter-camera calibration and cross view fusion, which we leave as valuable directions for future validation.


Q6: DreamScene4D (NeurIPS 2024) is not cited nor compared. It generates complete multi-object 4D scenes from monocular video, reports results on DAVIS and Kubric, and factorizes motion into object-centric deformation, object-to-world transformation and camera motion. It warrants at least a qualitative comparison.

A6: Thank you for pointing out DreamScene4D [1], which is closely related in monocular multi-object 4D scene generation and motion factorization. We will cite DreamScene4D and discuss its relationship to Chronofy in the revised manuscript.

Following your suggestion, we have added qualitative comparisons on Chrono-64 and DAVIS in Appendix A.8. Across most examples, Chronofy achieves better visual quality and substantially faster reconstruction, as our learned generative inference avoids DreamScene4D’s per-video optimization. Furthermore, its explicit separation of object and camera motion offers valuable guidance for our future extension to moving cameras.


Reviewer_KqkfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Chronofy, a framework for generative 4D reconstruction from videos. Built on SAM3D, it extends frame-wise reconstruction to video sequences to improve temporal consistency. The framework combines: (1) a unified representation that encodes per-frame shapes and motion in a shared normalized space; (2) a spatiotemporal mixture of transformers for joint shape and pose generation; and (3) a dynamic Gaussian decoder for temporally coherent reconstruction.

Critical strengths
  1. The core ideas are reasonable. Intuitively, encoding per-frame object shapes and motion in a shared normalized space and training on this representation should improve temporal consistency. Temporal interactions in both the generator and decoder also provide a sensible approach to reducing inconsistencies introduced by independent frame-wise reconstruction.

  2. The experiments provide meaningful evidence of improved reconstruction quality. Chronofy achieves the lowest FVD on both datasets in Table 1 while remaining competitive on image-quality metrics. These results support the effectiveness of the proposed method in improving the temporal quality of reconstructed videos.

Critical weaknesses
  1. The fairness of the evaluation requires clarification. The main results in Tables 1 and 2 rely on ground-truth pointmaps. This raises concerns about comparisons with methods that do not receive ground-truth geometric inputs. The conclusions would be more convincing if the authors clearly specified the input information available to each baseline and demonstrated that the improvements hold under comparable input conditions and evaluation protocols.

  2. The quality of 4D scene reconstruction requires further discussion. Unlike Table 1, Table 2 provides limited evidence of improvements in scene reconstruction. The proposed method performs similarly to SAM3D on some metrics, such as ADD-S on Chrono64 (0.0461 vs. 0.0477), and worse on others, such as [email protected] on Kubric4D (0.8649 vs. 0.8728). The paper offers possible explanations, but these are not sufficiently supported by experiments. Since temporal modeling is intended to improve reconstruction consistency, the authors should provide a more thorough analysis of why these gains do not consistently translate into better scene reconstruction and whether temporal modeling contributes to the observed performance degradation.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Kqkf (Part 1 / 2)

We deeply appreciate Reviewer Kqkf for the positive and thoughtful review. We are encouraged that you recognized the value of encoding shape and motion in a shared normalized space, as well as the resulting gains in temporal consistency.

We take this feedback very seriously and provide detailed responses below, covering fair and thorough comparisons, scene reconstruction analysis and temporal modeling ablations.


Q1: The fairness of the evaluation requires clarification. The main results in Tables 1 and 2 rely on ground-truth pointmaps. This raises concerns about comparisons with methods that do not receive ground-truth geometric inputs.

A1: Evaluating without ground-truth (GT) pointmaps is indeed essential for demonstrating real-world practicality. In Appendix A.1 and A.2 of our original submission, we already provided results with predicted Depth Anything 3 (DA3) pointmaps on Chrono-64 and DAVIS, showing that Chronofy maintains strong generation quality and temporal consistency. For a more complete evaluation, we have now extended these DA3 experiments to Motion-80 and 4D scene reconstruction, and revised Tables 1 and 2 to specify the input modalities across all baselines, which will be updated in the revised PDF.

R1-Table1: Quantitative comparison of 4D object generation on Chrono-64 dataset.

MethodLPIPS ↓CLIP-S ↑PSNR ↑SSIM ↑FVD ↓
Video + Masks
L4GM0.18850.715811.13400.6555345.53
GVFDiffusion0.25300.59289.19480.6140858.50
SS4D0.19320.736510.03020.6688342.23
MORPHOS0.17580.790210.63260.6592307.93
Mesh4D0.32010.61559.30200.6125584.96
Motion3240.31840.65479.19810.5986388.23
Video + Masks + GT Pointmaps
SAM3D0.20750.839213.19020.7725724.04
Chronofy0.16350.851213.94880.7647263.19
Video + Masks + DA3 Pointmaps
SAM3D0.22990.806711.88830.7583783.37
Chronofy0.17980.824112.50630.7596301.68

R1-Table2: Quantitative comparison of 4D object generation on Motion-80 dataset.

MethodLPIPS ↓CLIP-S ↑PSNR ↑SSIM ↑FVD ↓
Video + Masks
L4GM0.16020.837012.52250.7276275.19
GVFDiffusion0.24260.726312.44250.6962436.67
SS4D0.14580.864114.93830.7125267.74
MORPHOS0.17610.831613.65910.7051285.31
Mesh4D0.21530.781912.52810.6877563.01
Motion3240.22310.803110.21870.6894308.07
Video + Masks + GT Pointmaps
SAM3D0.19370.843514.73620.7543784.31
Chronofy0.15130.877615.68360.7726243.46
Video + Masks + DA3 Pointmaps
SAM3D0.21840.820912.35800.7136861.40
Chronofy0.16700.842714.15140.7375275.52

For 4D object generation, only SAM3D and Chronofy require pointmap inputs. When replacing GT pointmaps with predicted DA3 pointmaps, Chronofy still maintains competitive visual quality. Although DA3’s inaccurate pointmap estimates on Motion-80’s white-background videos degrade both methods, Chronofy preserves its temporal-consistency advantage, achieving 68% lower FVD than SAM3D under identical DA3 conditioning.

R1-Table3: Quantitative comparison of 4D scene reconstruction on Chrono-64 dataset.

Method3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
Video + Masks
COM4D————
Video + Masks + GT Pointmaps
ActionMesh + MegaPose0.549616.24010.11320.7441
ActionMesh + FoundationPose0.615014.78530.05780.8877
Chronofy + FoundationPose0.601413.57890.06130.8496
SAM3D0.586213.99370.04770.9414
Chronofy0.597712.59250.04610.9463
Video + Masks + DA3 Pointmaps
SAM3D0.561414.62720.05280.9209
Chronofy0.576213.25180.05070.9287
AuthorsOpenReview ↗

Response to Reviewer Kqkf (Part 2 / 2)

R1-Table4: Quantitative comparison of 4D scene reconstruction on Kubric-4D test set.

Method3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
Video + Masks
COM4D0.061239.25161.47470.0245
Video + Masks + GT Pointmaps
ActionMesh + MegaPose0.469432.28340.16250.5439
ActionMesh + FoundationPose0.471330.72620.14310.6107
Chronofy + FoundationPose0.452226.38550.13740.6756
SAM3D0.577616.56720.05920.8728
Chronofy0.581715.49260.07710.8649
Video + Masks + DA3 Pointmaps
SAM3D0.548217.29180.07160.8697
Chronofy0.559715.87410.09640.8470

For scene reconstruction, all methods except COM4D use pointmap conditioning. DA3-conditioned predictions are transformed into the GT coordinate frame for evaluation, where errors in predicted geometry or alignment may affect pose metrics. Chronofy maintains competitive scene reconstruction performance relative to SAM3D under both GT and DA3 conditioning.


Q2: Since temporal modeling is intended to improve reconstruction consistency, the authors should provide a more thorough analysis of why these gains do not consistently translate into better scene reconstruction.

A2: To clarify our primary design objective: Chronofy focuses on resolving temporal flickering and achieving sequence-level coherence while maintaining camera-space layout accuracy on par with SAM3D, rather than solely optimizing per-frame pose metrics.

The temporal consistency gains in Table 1 do not directly translate into higher scene reconstruction scores in Table 2 due to two factors:

  1. Decoder mismatch: Table 1 evaluates outputs from our dynamic Gaussian decoder, which drives the temporal coherence gains, whereas Table 2 evaluates meshes produced by the SAM3D mesh decoder, which cannot directly inherit these Gaussian-level improvements.
  2. Global pose sensitivity and training scale: Chronofy registers the entire sequence to the camera frame via a single unified transformation. While this enforces sequence-level consistency, any residual global pose error propagates across all frames, penalizing coordinate-sensitive metrics such as ADD-S. Furthermore, Chronofy was trained on far less data than SAM3D, which may partly benefit from million-scale training and multi-round post-training.

To examine the role of data scale, we synthesized 25K additional 4D sequences following SAM3D’s Flying Occlusion protocol for further fine-tuning. As shown in R1-Table5, scaling up training data substantially improves scene reconstruction, while ADD-S remains slightly higher than SAM3D. These results support the benefit of additional 4D training within our framework.

R1-Table5: Effect of additional Flying Occlusion training on Chronofy’s 4D scene reconstruction on the Kubric-4D test set. All methods use GT pointmaps and are evaluated without layout TTO.

Method3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
SAM3D0.577616.56720.05920.8728
Chronofy0.571415.93820.08150.8546
Chronofy (+25K Flying Occlusion 4D data)0.584215.17390.06070.8735

Q3: whether temporal modeling contributes to the observed performance degradation.

A3: As shown in R1-Table6, cross-frame attention preserves comparable pose accuracy with modest gains across most metrics, indicating that temporal modeling is not the main source of the layout degradation. Because sequence-level normalization alters SAM3D’s original per-frame representation, temporal context may help the model adapt by jointly modeling shape and motion. This ablation uses an earlier training checkpoint than the main comparisons.

R1-Table6: Ablation study of temporally inflating on Kubric-4D test set.

Method (w/o Layout TTO)3D IoU ↑ICP-Rot. ↓ADD-S ↓ADD-S @ 0.1 ↑
Chronofy (w/o temporally inflate)0.569816.73150.08210.8535
Chronofy (w/ temporally inflate)0.572615.81420.08260.8539

We hope these responses address your concerns. If any concerns remain unaddressed, we would be glad to provide further clarification.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer Kqkf,

Thanks again for your efforts and suggestions for this paper. As the deadline for the author-reviewer discussion phase is nearing, we would like to courteously inquire if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Rebuttal Summary

We thank the reviewers for their detailed and constructive feedback. We also appreciate their recognition of Chronofy's contributions to improving temporal consistency in 4D generative reconstruction.

Chronofy jointly generates complete dynamic objects and their unified camera-space poses from monocular videos. We introduce a pose-aware 4D representation and a Spatiotemporal Mixture of Transformers, together with dynamic Gaussian decoding. These designs improve temporal consistency while maintaining competitive pose accuracy. The reviewers' main concerns focus on the fairness of comparisons, the contributions of individual components, and the distinctions from prior work. Our responses include the following experiments and clarifications.


1. Input conditions and layout TTO. We now report SAM3D and Chronofy using the same DA3 pointmaps for object generation and scene reconstruction. We also compare both methods with and without layout TTO under GT pointmaps in R2-Tables1-4. Chronofy consistently achieves the best temporal consistency while maintaining competitive pose accuracy across these settings.

2. Temporal modeling and augmentation. We add two SAM3D controls: (i) frame-wise fine-tuning, (ii) shared noise, with Markley pose averaging without training. Chronofy achieves lower FVD than both controls in R3-Table6. Additionally, a separate attention ablation shows comparable pose accuracy with and without cross-frame interaction in R1-Table6. We explain how occlusion augmentation recovers per-frame quality during 4D adaptation.

3. Representation and training scale. We extend the representation comparison to 10K sequences and 20K steps. Our design achieves the highest voxel and silhouette IoU among the three variants in R2-Table8. Training on an additional 25K sequences synthesized using the Flying Occlusion protocol in R1-Table5, demonstrates the scalability of our pose-aware 4D representation.

4. Related methods and scope. Additional quantitative comparisons with 4DGT and V2M4 are provided in R2-Tables6–7. Qualitative comparisons with DreamScene4D are also included on Chrono-64 and DAVIS. We further clarify our contributions relative to MORPHOS and SS4D. Finally, we discuss the remaining challenges of large motions, long sequences, and moving cameras.


We have incorporated these results into the revised manuscript. Section 2 clarifies related work, while Section 5.2 adds matched-input comparisons. Appendix A.1 expands the representation and layout TTO analyses, and Appendix A.2 adds baseline comparisons. Appendix A.4 clarifies evaluation settings, while Appendix A.8 introduces DreamScene4D comparisons. Revisions are marked in red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.