- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper unifies physical-world understanding, free-viewpoint spatial simulation, and 3D world generation/reconstruction within a single multimodal model built on three "native world states": physics (gravity field, latitude), geometry (depth), and appearance (RGB). Its Omni-Camera representation concatenates a gravity-anchored absolute field (up-vector, latitude) with a relative ray field. Physics propagation rotates the reference-view gravity direction to each future view via the known relative rotation. Appearance and depth share one VAE (depth encoded as an invertible Hilbert-color image) and are generated jointly under a shared flow-matching objective with asymmetric attention. The authors build Puffin-16M (15M vision-language-camera triplets plus 1M trajectories) and annotate 28 public datasets (~44.5M images). Results include state-of-the-art camera-to-world understanding on four benchmarks, camera-controllable generation on Puffin-Cam-Bench (FID 75.93), and 3D world generation on RealEstate10K and Puffin-Traj-Bench.
Critical strengths
- A well-motivated, unified camera representation. Combining gravity-anchored absolute orientation with a relative ray field, and using it to drive single-view control, multi-view synthesis, and reconstruction, is a coherent idea supported by strong camera-understanding results.
- A substantial potential community resource. Puffin-16M and the annotations for 28 public datasets (~44.5M images) could support camera-grounded modeling beyond this system. Their scale and coverage are valuable, contingent on release and on documenting annotation quality.
Critical weaknesses
- Lack of an independent controllability evaluation. Using the model's own perception branch to score all generated images leaves open whether the camera-control margins depend on the chosen evaluator. Re-scoring the existing outputs independently would address this central claim without changing the generation experiment.
- Architectural and conditioning effects are not fully isolated. The comparison with Puffin changes both the architecture and the training data. Table A5 is a useful propagation ablation, but the absolute-camera information available to its baseline needs to be specified. Matched-data camera-injection comparisons and a clear conditioning definition would distinguish architectural improvements from data scaling or additional camera information.
Questions for authors
- Independent controllability. Please re-score the same generated images (Tables 2 and 3) with independent estimators such as GeoCalib and AnyCalib, to test whether the camera-control margins survive evaluator choice.
- Table A5 baseline. What does the no-propagation baseline receive—relative rays only, a repeated initial field, or per-view absolute fields? Please specify the conditioning and whether weights, data, and trajectories are matched.
- Matched-data control. With backbone, training data, and update budget fixed, how does replacing the new camera-injection path with Puffin-style Perspective-Field conditioning change camera errors and FID?
5 comments
Response to Reviewer ioPk (part 2/3)
Table R3-3. Physics-propagation ablation
| Setting | Motion | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Roll error ↓ | Pitch error ↓ |
|---|---|---|---|---|---|---|
| Baseline | Roll | 22.97 | 0.69 | 0.12 | 2.35 / | 6.73 / |
| Pitch | 18.53 | 0.60 | 0.26 | 2.17 / | 5.60 / | |
| Yaw | 18.12 | 0.59 | 0.30 | 1.31 / | 3.10 / | |
| Average | 19.87 | 0.62 | 0.23 | 1.94 / | 5.14 / | |
| Baseline with PP | Roll | 24.02 | 0.74 | 0.10 | 2.04 / | 6.31 / |
| Pitch | 19.18 | 0.65 | 0.24 | 2.03 / | 5.28 / | |
| Yaw | 18.42 | 0.62 | 0.30 | 1.22 / | 2.67 / | |
| Average | 20.54 | 0.67 | 0.21 | 1.76 / | 4.75 / |
From the re-evaluated results, we have three observations:
-
Camera-controllable image generation. On Puffin-Cam-Bench, Puffin-World retains the lowest errors across all six camera metrics. With GeoCalib, its mean up-vector, latitude, and gravity errors are , , and , compared with , , and for the strongest baseline, Puffin.
-
Multi-view generation. On Puffin-Traj-Bench, Puffin-World retains the lowest median roll, pitch, and FoV errors: , , and . It also achieves the highest or joint-highest roll/pitch AUC at every reported threshold, tying with SEVA at pitch AUC@10° ().
-
Physics propagation. GeoCalib also confirms the average benefit of PP: roll error decreases from to , and pitch error from to . Across individual motion categories, the camera errors either improve or remain unchanged at the reported precision.
Although absolute scores and some baseline orderings vary between evaluators, the overall performance trends and our main comparative conclusions remain consistent. This independent check supports the credibility of using our physics-perception branch to compare camera-controlled spatial simulation across the tested methods, and shows that our main conclusions are not specific to the original evaluator. We have included both sets of results in the revised manuscript and clarified the evaluator dependence of individual scores and performance margins.
[1] Veicht, Alexander, et al. "Geocalib: Learning single-image calibration with geometric optimization." European Conference on Computer Vision, 2024.
Response to Reviewer ioPk (part 3/3)
W2. Separating architectural effects, data scaling, and the absolute information in the PP baseline.
Thank you for highlighting this point. The primary goal of our camera interface is to support scalable, unified generation across tasks while preserving single-view quality. We clarify the comparison and conditioning protocol below.
-
Matched-backbone and matched-data comparison. We compare Puffin-style camera-token injection with our latent-space fusion using the same SD3.5-Large MMDiT backbone and training data. For this shared single-view task, both interfaces encode the target absolute camera configuration, while the relative ray component of Omni-Camera is held constant. Thus, the comparison does not introduce additional inter-view motion information.
Table R3-4. Camera-interface comparison under a shared backbone and training dataset.
Metric or capability Puffin-style injection Puffin-World fusion FID ↓ 78.12 75.93 Gravity median error ↓ 1.22 0.79 Inference time ↓ 44.5 s 19.5 s Compute per image ↓ 7.0 PFLOPs 2.9 PFLOPs Additional camera tokens ✓ ✗ Camera-controllable text-to-image ✓ ✓ Joint multi-view image-to-3D / text-to-3D ✗ ✓ Native RGB-D generation ✗ ✓ The two interfaces achieve comparable FID and camera metrics, while our fusion reduces inference time and computation substantially. This comparison controls the backbone and training data and evaluates the camera interface as a whole, including its representation and injection mechanism. We therefore do not attribute the difference to latent-space fusion alone.
-
Explicit conditioning definition for Table A5. The no-propagation baseline retains the relative ray maps but replaces the absolute Perspective Field with constant values. It therefore receives no informative absolute gravity field through this conditioning component. With physics propagation, the Perspective Field is instead constructed from the reference view’s estimated absolute camera state and the commanded relative rotations. Both settings use the same checkpoint, reference inputs, and camera trajectories; only the Perspective Field changes. During training, we randomly set this field with probability to constant while retaining the relative rays, so the model encounters both informative and constant-field conditions. Table A5 therefore evaluates the benefit of propagated gravity conditioning, separately from the camera-interface comparison above.
-
The scalability advantage. Puffin’s interface VAE-encodes a three-channel Perspective Field as additional camera tokens. Our fusion module accommodates the nine-channel Omni-Camera representation and view-role mask within existing latent positions, providing a shared interface for absolute camera grounding, relative motion, and different view roles. This provides a practical basis for unified text-to-image, image-to-3D, and text-to-3D generation, including joint multi-view RGB-D outputs.
We have revised Section 2.2 to emphasize this scalability contribution, identify the original Puffin comparison as system-level, and explicitly document the matched comparison and Table A5 conditioning protocol in the manuscript.
Q1. Re-scoring the camera metrics.
Please refer to our response to W1 for more details. The overall performance trends and our main comparative conclusions remain consistent.
Q2. What does the no-propagation baseline receive, and what is held fixed?
The no-propagation baseline receives relative ray maps, with the absolute Perspective Field replaced by constant values; it uses neither a repeated initial field nor per-view absolute fields. Both settings use the same checkpoint, training data, reference inputs, and camera trajectories. The only change at inference is whether the Perspective Field is constant or constructed through physics propagation. Please refer to our response to W2 for the training augmentation and detailed conditioning protocol.
Q3. Matched-data comparison with Puffin-style camera injection.
Please refer to our response to W2 and Table R3-4 for the comparison protocol and results. Our primary objective is efficient camera conditioning that applies across tasks, rather than improving a single-task score alone. As emphasized in W2, our interface integrates absolute camera grounding, relative motion, and view-role information without introducing additional camera tokens. This significantly reduces computational cost while providing a shared conditioning formulation for text-to-image, image-to-3D, and text-to-3D generation, including joint multi-view RGB-D outputs. Its central advantage is therefore the combination of comparable single-view quality, greater efficiency, and broader task applicability within one scalable unified model.
Response to Reviewer ioPk (part 1/3)
Thank you for this valuable suggestion. We re-evaluated the generated outputs in Sections 3.3, 3.4, and Appendix A0.5 using GeoCalib [1], an independent camera estimator with the state-of-the art performance. The tables below report both evaluators. Specifically, each paired camera-metric cell reports Puffin-World evaluator / underlined GeoCalib evaluator. All angular errors are in degrees; AUC values retain the original 0–1 scale.
Table R3-1. Camera-controllable generation on Puffin-Cam-Bench
Table R3-2. Camera-control evaluation on Puffin-Traj-Bench