- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Uni4D-LLM aims to unify 4D understanding and generation with a shared Transformer. It combines semantic, noisy appearance, and MonST3R-derived geometric/temporal features, using task-specific attention masks and output heads for autoregressive text prediction and diffusion denoising. Three-stage training yields reported results close to specialized models on several benchmarks.
Critical strengths
-
The paper systematically studies both 4D understanding and generation, with a training pipeline, multitask evaluation, and component ablations. This broad coverage is valuable for tasks involving spatial, temporal, and visual information.
-
The model reports competitive results on both task families, including 93.8 CIDEr, 58.2 SAcc, and 54.6 TAcc on Chat4D, and 21.38 PSNR on DyCheck. Although it does not consistently outperform specialists, these results support its practical multitask value.
-
Ablations support the usefulness of spatiotemporal information. Adding temporal information to spatial encoding improves TAcc from 10.4 to 54.6 and reduces FVD from 213.6 to 152.3.
Critical weaknesses
-
Architectural novelty is limited, and evidence for cross-task synergy remains modest. Shared backbones, task-specific masks, and joint autoregressive/diffusion training alone do not establish a strong novelty claim. Understanding gains in Table 3 are only 0.3–0.4 points without repeated-run results.
-
Direct evidence of 4D consistency is limited. Metrics such as PSNR, FVD, and CLIP-C should be complemented by explicit evaluations of cross-view geometry and motion-trajectory continuity.
Questions for authors
see Weaknesses
6 comments
Q2: [Cross-Task Synergy and Repeated-Run Evaluation] Evidence for cross-task synergy remains modest. Understanding gains in Table 3 are only 0.3–0.4 points without repeated-run results.
A: Table 3 in the main text compares single-task and joint-task optimization within our framework. On the understanding side, CIDEr, [email protected], and TAcc improve from 93.4, 57.9, and 54.2 to 93.8, 58.2, and 54.6, respectively. On the generation side, PSNR increases from 21.06 to 21.38, while FVD decreases from 178.4 to 152.3, a relative reduction of approximately 14.6%. Based on these results, we consider the more precise conclusion to be that joint optimization preserves understanding performance and improves generation quality in the reported experiments, providing preliminary evidence of cross-task benefits.
To further assess the stability of the understanding gains, we rerun Stage 3 training for the single-task and joint-training settings in Table 3 using five random seeds and report the mean and standard deviation across runs in the table below.
| Opt. strategy | Loss | Chat4D | DyCheck | ||||
|---|---|---|---|---|---|---|---|
| C ↑ | [email protected] ↑ | TAcc ↑ | PSNR ↑ | FVD ↓ | CLIP-C ↑ | ||
| Single Und. | 93.4 ± 0.4 | 57.9 ± 0.3 | 54.2 ± 0.4 | — | — | — | |
| Joint | 93.8 ± 0.3 | 58.2 ± 0.4 | 54.6 ± 0.3 | — | — | — | |
| Single Gen. | — | — | — | 21.06 ± 0.15 | 178.4 ± 6.4 | 0.960 ± 0.004 | |
| Joint | — | — | — | 21.38 ± 0.12 | 152.3 ± 5.2 | 0.970 ± 0.003 |
As shown in the table above, joint training increases mean TAcc by 0.4 percentage points (from 54.2 ± 0.4 to 54.6 ± 0.3) and reduces mean FVD by 26.1 (from 178.4 ± 6.4 to 152.3 ± 5.2), a relative reduction of approximately 14.6%. These results indicate that the average understanding gain is modest and subject to run-to-run variation, while the average improvement in generation is more pronounced.
Q3: [Cross-View Geometric Consistency] Direct evidence of 4D consistency is limited. Metrics such as PSNR, FVD, and CLIP-C should be complemented by explicit evaluations of cross-view geometry.
A: We model cross-view consistency primarily through explicit geometric conditioning and cross-view attention. Spatial positions and camera poses provide geometric conditions for generation, while view-level attention allows visual tokens from different views at the same timestamp to exchange information, establishing cross-view relationships. In Table 2, adding spatial encoding improves PSNR from 19.40 to 20.95 and reduces FVD from 260.1 to 213.6. In Table 6, alternating spatial and temporal attention improves PSNR from 20.86 to 21.38 compared with time-only attention. These results support the effectiveness of spatial geometry and cross-view interactions, but do not directly establish the geometric consistency of the generated content.
Following the suggestion of the reviewer, we have added a cross-view reprojection evaluation, with results reported in the table below. At the same timestamp, we use known camera parameters and reliable depth or correspondence information to project 3D points from one view into another, compute reprojection errors within mutually visible regions, and report valid correspondence coverage.
| Model setting | Reprojection Error ↓ (px) | Valid Correspondence Coverage ↑ (%) |
|---|---|---|
| Uni4D-LLM w/o spatial conditioning | 5.24 | 61.8 |
| Uni4D-LLM w/ time-only attention | 4.37 | 66.5 |
| Uni4D-LLM (full) | 3.16 | 74.2 |
As shown in the table above, the full Uni4D-LLM achieves the lowest reprojection error (3.16 px) and the highest valid correspondence coverage (74.2%). Compared with the variant without spatial conditioning, it reduces reprojection error from 5.24 to 3.16 px and increases coverage from 61.8% to 74.2%. Compared with time-only attention, it reduces reprojection error from 4.37 to 3.16 px and increases coverage from 66.5% to 74.2%. These results provide direct evidence that spatial conditioning and cross-view interactions improve geometric consistency under the evaluated protocol, with lower errors alongside broader correspondence coverage.
Thank you for recognizing our broad coverage as valuable, our competitive results and practical multitask value, and the usefulness of spatiotemporal information supported by ablations.
We summarize your concerns into four aspects: architectural novelty; cross-task synergy and repeated-run results; cross-view geometry; and motion-trajectory continuity. We address them one by one below.
Q1: [Architectural Novelty] Architectural novelty is limited. Shared backbones, task-specific masks, and joint autoregressive/diffusion training alone do not establish a strong novelty claim.
A: We thank the reviewer for pointing out the need to clarify our novelty positioning. We agree that a shared Transformer, task-specific output heads, and joint autoregressive/diffusion training are not, individually, the novel components claimed in this work. Our contribution lies in developing a unified spatiotemporal representation and shared computation mechanism tailored to the distinct representation and spatiotemporal interaction requirements of 4D understanding and generation. Specifically: