- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Chronofy, a framework for joint 4D object reconstruction and camera-space pose estimation from monocular videos. Building upon SAM3D, the authors introduce a sequence-level unified pose representation, spatiotemporal transformers, and a dynamic Gaussian decoder to improve temporal consistency. Temporal masking augmentation and test-time pose refinement are further used to handle occlusions and improve scene alignment. Experiments on Motion-80, Kubric-4D, and Chrono-64 demonstrate improved temporal consistency and competitive reconstruction quality compared to existing methods.
Critical strengths
-
The problem is meaningful. Existing video-to-4D generation and 4D reconstruction methods do not directly produce complete 4D objects with camera-space poses for scene composition. The proposed sequence-level normalization with a unified pose is a simple but effective way to address the shape-pose ambiguity in frame-wise SAM3D, and Table 4 provides a useful ablation.
-
The improvement in temporal consistency is substantial and clearly visible in the supplementary videos. The authors also report results where their method performs worse (Kubric ADD-S in Table 7) and discuss failure cases.
Critical weaknesses
-
The comparisons are not entirely fair. In Table 2, Chronofy uses Layout TTO while SAM3D does not, even though TTO builds on SAM3D's pose refinement. When both use TTO, SAM3D performs better on 3D IoU, [email protected], and 2D IoU (Table 7). Similarly, Chronofy and SAM3D use GT pointmaps in Table 1, while other baselines only use RGB. With DA3 pointmaps (Table 6), Chronofy's advantage over MORPHOS becomes much smaller (LPIPS 0.180 vs. 0.176, FVD 302 vs. 308). The paper also lacks pose accuracy results with estimated depth, which is more realistic. These issues make it difficult to assess the claimed improvements in object quality and pose accuracy.
-
The ablations do not fully support the architectural choices. Table 3 lacks a baseline that fine-tunes SAM3D frame-wise using the same 18K training samples and augmentation. Much of the per-frame improvement also appears to come from augmentation (row d). Furthermore, adding the dynamic Gaussian decoder worsens all metrics except FVD in the full system (Table 3, b→c), despite improving results with GT latents (Table 5). This discrepancy deserves more discussion. Reporting variance across different seeds would also help establish the reliability of the results.
Questions for authors
- Please include SAM3D w/ TTO in Table 2 for both datasets, and clarify whether Chronofy uses TTO on Kubric-4D.
- Please report the results in Tables 1 and 2 using DA3 pointmaps for both Chronofy and SAM3D, including pose accuracy.
- Please add a frame-wise fine-tuned SAM3D baseline to Table 3, trained with the same data and augmentation.
- Is the dynamic Gaussian decoder trained only on GT latents? Why does it hurt most metrics in Table 3 (b→c) but help in Table 5?
- The supplementary material states that Mesh4D uses 6 frames. How is FVD computed against the 16-frame ground truth?
5 comments
Response to Reviewer XhNT (Part 2 / 3)
Q2: Similarly, Chronofy and SAM3D use GT pointmaps in Table 1, while other baselines only use RGB. With DA3 pointmaps (Table 6), Chronofy's advantage over MORPHOS becomes much smaller. The paper also lacks pose accuracy results with estimated depth, which is more realistic.
A2: Evaluating with predicted pointmaps indeed better reflects practical scenarios. We have added 4D object generation results for Chronofy and SAM3D under identical DA3-predicted pointmaps on Chrono-64 and Motion-80 in R3-Table4 and R3-Table5.
R3-Table4: Quantitative comparison of 4D object generation on Chrono-64 dataset.
| Method | LPIPS ↓ | CLIP-S ↑ | PSNR ↑ | SSIM ↑ | FVD ↓ |
|---|---|---|---|---|---|
| Video + Masks | |||||
| L4GM | 0.1885 | 0.7158 | 11.1340 | 0.6555 | 345.53 |
| GVFDiffusion | 0.2530 | 0.5928 | 9.1948 | 0.6140 | 858.50 |
| SS4D | 0.1932 | 0.7365 | 10.0302 | 0.6688 | 342.23 |
| MORPHOS | 0.1758 | 0.7902 | 10.6326 | 0.6592 | 307.93 |
| Mesh4D | 0.3201 | 0.6155 | 9.3020 | 0.6125 | 584.96 |
| Motion324 | 0.3184 | 0.6547 | 9.1981 | 0.5986 | 388.23 |
| Video + Masks + GT Pointmaps | |||||
| SAM3D | 0.2075 | 0.8392 | 13.1902 | 0.7725 | 724.04 |
| Chronofy | 0.1635 | 0.8512 | 13.9488 | 0.7647 | 263.19 |
| Video + Masks + DA3 Pointmaps | |||||
| SAM3D | 0.2299 | 0.8067 | 11.8883 | 0.7583 | 783.37 |
| Chronofy | 0.1798 | 0.8241 | 12.5063 | 0.7596 | 301.68 |
Compared with MORPHOS on Chrono-64, using DA3 pointmaps narrows Chronofy’s FVD lead and yields slightly worse LPIPS, while PSNR and SSIM remain clearly higher. More importantly, under identical DA3 conditioning, Chronofy still outperforms SAM3D by a wide margin across both datasets, confirming that our 4D extension remains effective without GT geometry.
R3-Table5: Quantitative comparison of 4D object generation on Motion-80 dataset.
| Method | LPIPS ↓ | CLIP-S ↑ | PSNR ↑ | SSIM ↑ | FVD ↓ |
|---|---|---|---|---|---|
| Video + Masks | |||||
| L4GM | 0.1602 | 0.8370 | 12.5225 | 0.7276 | 275.19 |
| GVFDiffusion | 0.2426 | 0.7263 | 12.4425 | 0.6962 | 436.67 |
| SS4D | 0.1458 | 0.8641 | 14.9383 | 0.7125 | 267.74 |
| MORPHOS | 0.1761 | 0.8316 | 13.6591 | 0.7051 | 285.31 |
| Mesh4D | 0.2153 | 0.7819 | 12.5281 | 0.6877 | 563.01 |
| Motion324 | 0.2231 | 0.8031 | 10.2187 | 0.6894 | 308.07 |
| Video + Masks + GT Pointmaps | |||||
| SAM3D | 0.1937 | 0.8435 | 14.7362 | 0.7543 | 784.31 |
| Chronofy | 0.1513 | 0.8776 | 15.6836 | 0.7726 | 243.46 |
| Video + Masks + DA3 Pointmaps | |||||
| SAM3D | 0.2184 | 0.8209 | 12.3580 | 0.7136 | 861.40 |
| Chronofy | 0.1670 | 0.8427 | 14.1514 | 0.7375 | 275.52 |
For pose accuracy with estimated depth, R3-Table1 and R3-Table2 also report DA3-conditioned scene reconstruction on Chrono-64 and Kubric-4D, where predictions are transformed into the GT coordinate frame and inevitably incur geometry and alignment errors. Despite the expected drop relative to GT pointmaps, Chronofy maintains competitive camera-space pose accuracy alongside SAM3D, complementing its temporal-consistency gains in dynamic object generation.
Q3: The ablations do not fully support the architectural choices. Table 3 lacks a baseline that fine-tunes SAM3D frame-wise using the same 18K training samples and augmentation.
A3: We have added two controls: (a) frame-wise SAM3D fine-tuned on the same 18K samples with the same augmentation, and (b) training-free SAM3D with shared noise and the same Markley pose averaging in R3-Table6. Both fine-tuned models are evaluated at 20K training steps.
R3-Table6: Controlled comparisons for 4D object generation on Chrono-64.
| Method | LPIPS ↓ | CLIP-S ↑ | PSNR ↑ | SSIM ↑ | FVD ↓ |
|---|---|---|---|---|---|
| (a) | 0.1846 | 0.8459 | 13.4215 | 0.7764 | 526.72 |
| (b) | 0.2118 | 0.8354 | 13.4365 | 0.7519 | 698.36 |
| Chronofy | 0.1817 | 0.8476 | 13.5842 | 0.7582 | 347.45 |
While frame-wise fine-tuning improves SAM3D’s per-frame metrics, Chronofy achieves the best temporal consistency across these variants. These results support the benefits of learned cross-frame modeling beyond data adaptation and augmentation alone.
Response to Reviewer XhNT (Part 3 / 3)
Q4: Much of the per-frame improvement also appears to come from augmentation (row d). Furthermore, adding the dynamic Gaussian decoder worsens all metrics except FVD in the full system (Table 3, b→c), despite improving results with GT latents (Table 5). This discrepancy deserves more discussion.
A4: This discrepancy stems from a train-inference input mismatch: the dynamic Gaussian decoder is trained exclusively on structured latents encoded from GT assets, whereas at inference it decodes predicted latents from the generative model. Table 5 isolates decoding on clean GT latents, while Table 3 evaluates the full generative pipeline.
On clean GT latents in Table 5, the decoder improves both spatial and temporal quality, raising SSIM from 0.9480 to 0.9628 and cutting FVD from 473.82 to 157.66. In Table 3 (b→c), however, 4D adaptation without occlusion augmentation yields incomplete predictions in occluded regions, where the dynamic decoder introduces artifacts that lower per-frame scores. Even so, the decoder still reduces FVD from 403.62 to 289.36.
We acknowledge that occlusion augmentation provides the largest incremental PSNR gain (+1.08 dB), alleviating these occlusion artifacts to restore SSIM from 0.6927 to 0.7647 and further lower FVD to 263.19. Temporal modeling and occlusion augmentation therefore play complementary roles: temporal modeling drives motion coherence and video quality, while occlusion augmentation preserves single-frame fidelity under occlusion.
Q5: The supplementary material states that Mesh4D uses 6 frames. How is FVD computed against the 16-frame ground truth?
A5: The originally reported FVD (584.96) uses forward–reverse repetition to extend both Mesh4D’s six-frame predictions and the corresponding six GT frames to 16 frames, matching the evaluation length used for other methods. R3-Table7 additionally reports evaluation using Mesh4D’s original six-frame output and the first six GT frames, yielding an FVD of 516.63.
R3-Table7: Effect of temporal extension on Mesh4D evaluation on Chrono-64 dataset.
| Methods | FVD ↓ |
|---|---|
| Mesh4D (forward–reverse repetition) | 584.96 |
| Mesh4D (original 6 frames) | 516.63 |
We sincerely thank the reviewer for the detailed and constructive feedback. If any concerns remain unaddressed, we would be glad to provide further clarification.
Response to Reviewer XhNT (Part 1 / 3)
We really appreciate the reviewer for the detailed and constructive feedback. We appreciate your recognition of the importance of jointly reconstructing complete dynamic objects and their camera-space poses, as well as the temporal-consistency gains shown in our supplementary videos.
In response, we provide additional control experiments and evaluation details on pointmap inputs, layout TTO, and frame-wise adaptation versus temporal modeling. We also clarify the decoder ablations and Mesh4D’s FVD evaluation protocol. Below, we address each concern in detail.
Q1: The comparisons are not entirely fair. In Table 2, Chronofy uses Layout TTO while SAM3D does not, even though TTO builds on SAM3D's pose refinement. When both use TTO, SAM3D performs better on 3D IoU, [email protected], and 2D IoU (Table 7).
A1: Thank you for pointing this out. Scene reconstruction should indeed be evaluated under matched TTO settings. We report both methods with and without layout TTO on Chrono-64 and Kubric-4D in R3-Table1 and R3-Table2, and will update the revised manuscript accordingly.
Under identical GT pointmaps and matched TTO settings, Chronofy and SAM3D trade leads across metrics. These results confirm that Chronofy preserves competitive camera-space pose accuracy while substantially improving temporal consistency in dynamic object generation.
Remaining gaps on certain pose metrics partly reflect the limited scale of our 4D training data. To verify this, we synthesized 25K additional 4D samples following SAM3D’s Flying Occlusion protocol and continued training. As shown in R3-Table3, scaling 4D training improves Chronofy’s pose accuracy on Kubric-4D under identical inputs and TTO settings, surpassing SAM3D in 3D IoU, ICP-Rot., and ADD-S @ 0.1.