- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Zebra3D, a practical method that converts pretrained 3D reconstruction transformers into hybrids of linear and tile-sparse attention for effiicient reconstruction. Zebra3D assigns linear attention to stages that tolerate replacement, while sensitive stages retain selected token-level interactions and landmark summaries of omitted tiles. Then a centered affine correction fitted on calibration data enables conversion without gradient updates. The paper also presents optional refinement that combines task supervision with distillation. The authors provide extensive experiments in the paper across several benchmarks and compare the method with wide baselines. The paper largely retain the accuracy of the dense-model, while an aggressive configurations provide larger acceleration.
Critical strengths
-
The core contribution of Zebra3D is a framework that demonstrates useful and efficient conversion across multiple pretrained backbones and measurable end-to-end acceleration. he conservative configurations retain near-dense pose accuracy across three backbones, while a separate refined VGGT configuration achieves 5.60× speedup with comparable reconstruction results, supporting a practical quality–efficiency trade-off.
-
The paper provide detailed experiments and ablations that validate the proposed architecture. The sensitivity-guided layer allocation is supported by controlled comparisons, providing direct evidence that where the operators are placed affects reconstruction performance. The explanations and discussions also provide thorough analysis of the gains and is informative for evaluating where the gains come from.
-
The initialization and context-preservation mechanisms are well-motivated and supported by strong evidence in the experiments sections. The paper breakdowns the effectiveness of adding landmarks to tile-sparse attention and shows that the components contribute more than implementation convenience. This further proves that the strategy is practical and highly reuseable for feed-forward dense backbones deployments.
Critical weaknesses
-
The specific contribution of relational distillation remains insufficiently isolated in the ablation. In Tables 3(c), the authors do not include the complete objective with only the relational term removed, while the dense GT-FT control omits all distillation. This makes it difficult to distinguish relational preservation from the other feature and output matching losses. I suggest adding this experiments in the experiments section.
-
The current efficiency presentation leaves the exact practical operating costs unclear. The results of efficiency in the paper lack absolute latency and peak-memory measurements. I suggest adding paired measurements directly to make the reported accuracy–runtime trade-offs easier to assess and reproduce.
-
Long-sequence accuracy is not established as broadly as long-sequence runtime. Figure 5 reports speedups at 1,024 views for all three benchmarks, whereas Table 2 provides quantitative long-sequence geometry only for refined VGGT variants. I suggest adding the corresponding results since short-sequence pose results cannot establish equivalent quality retention at these larger scales. This would help readers to better understand the quality-efficiency tradeoff of the proposed methods.
-
The paper mainly provides metrics in the tables to prove the effectiveness, I would suggest adding more qualitative comparisons with its inference efficiency for better paper presentation. In addition, I would also suggest the authors include more in-the-wild examples either in the main paper or the appendix for a more comprehensive comparisons.
-
There are some misalignments in the papers: a) A small numerical discrepancy should be corrected. For Pi3 at 0.5L/0.8S, RE10K AUC-30 is 0.866 in Table 1 and 7 but 0.867 in Table 12 b) The calibration-set reporting needs clarification. Figure 8 states that results are averaged over 11 held-out scenes, whereas Appendix B.6 describes 13 scenes for reference profiling and fitting. c) A minor typo exists in section 4.3, “Therefor” should be “Therefore.”
Questions for authors
The paper already validates the architecture effectiveness on 3 different backbones, I wonder how does the proposed method perform if using VGGT-Omega as a backbone. In addition, as mentioned in the weakness, how does the complete refinement objective perform when only the relational loss is removed, keeping all other losses, data, and optimization settings unchanged?
5 comments
Official Comment by Authors
[Answer 2] Inference cost and peak GPU memory (W2)
We added peak GPU memory measurements for all three backbones across 100–1,000 views, comparing Zebra3D with dense inference, A3D, and BlockSparse3D.
Table R19: Peak GPU memory in GiB.
| Backbone | Method | 100 views | 300 views | 500 views | 700 views | 1,000 views |
|---|---|---|---|---|---|---|
| VGGT | Dense | 13.4 | 29.6 | 45.8 | 62.0 | 84.2 |
| A3D | 13.4 | 29.7 | 45.9 | 62.1 | 84.3 | |
| BlockSparse3D | 13.6 | 29.8 | 47.6 | 75.3 | 128.0 | |
| Zebra3D (0.5L/0.6S) | 13.5 | 29.7 | 45.9 | 62.1 | 88.0 | |
| Zebra3D (0.7L/0.8S) | 13.5 | 29.8 | 46.0 | 62.2 | 84.4 | |
| HYWM | Dense | 16.3 | 37.9 | 59.4 | 80.9 | 110.4 |
| A3D | 16.4 | 37.9 | 59.5 | 81.0 | 110.5 | |
| BlockSparse3D | 16.5 | 38.1 | 59.6 | 81.2 | 136.1 | |
| Zebra3D (0.5L/0.6S) | 16.4 | 38.0 | 59.5 | 81.0 | 110.5 | |
| Zebra3D (0.7L/0.8S) | 16.5 | 38.0 | 59.6 | 81.1 | 110.5 | |
| π³ | Dense | 8.2 | 17.1 | 26.0 | 34.8 | 48.1 |
| A3D | 7.8 | 17.1 | 26.0 | 34.9 | 48.2 | |
| BlockSparse3D | 8.6 | 19.9 | 38.1 | 62.3 | 109.7 | |
| Zebra3D (0.5L/0.6S) | 7.3 | 14.6 | 22.8 | 31.9 | 46.7 | |
| Zebra3D (0.7L/0.8S) | 7.2 | 14.4 | 22.1 | 30.3 | 43.6 |
Zebra3D uses approximately the same memory as dense inference on VGGT and HYWM, and less memory on π³ (Table R19). Zebra3D also uses substantially less memory than BlockSparse3D at larger view counts.
We will add the reported speedups, using the same configurations and input lengths.
[Answer 3] Long-sequence reconstruction on HYWM and π³ (W3)
We have added ScanNet-50 reconstruction results for HYWM and π³ across 100–1,000 frames, reporting quality and speed for the same configurations.
Table R20: ScanNet-50 long-sequence reconstruction with HYWM and π³.
| Backbone | Method | 1,000 frames CD ↓ | Speedup ↑ | 500 frames CD ↓ | Speedup ↑ | 300 frames CD ↓ | Speedup ↑ | 100 frames CD ↓ | Speedup ↑ |
|---|---|---|---|---|---|---|---|---|---|
| HYWM | Dense | 0.456 | 1.00× | 0.449 | 1.00× | 0.432 | 1.00× | 0.431 | 1.00× |
| Zebra3D (0.5L/0.8S) | 0.453 | 3.26× | 0.441 | 2.86× | 0.430 | 2.54× | 0.428 | 1.81× | |
| Zebra3D-KD (0.7L/0.8S) | 0.459 | 5.21× | 0.459 | 4.11× | 0.439 | 3.17× | 0.431 | 2.01× | |
| π³ | Dense | 0.608 | 1.00× | 0.604 | 1.00× | 0.608 | 1.00× | 0.611 | 1.00× |
| Zebra3D (0.5L/0.8S) | 0.609 | 4.11× | 0.610 | 3.72× | 0.621 | 1.83× | 0.618 | 1.65× | |
| Zebra3D-KD (0.7L/0.8S) | 0.604 | 7.04× | 0.603 | 6.05× | 0.608 | 2.89× | 0.604 | 2.25× |
The new results show substantial acceleration without refinement. Training-free Zebra3D improves HYWM’s reconstruction quality at every evaluated length. On π³, its error is nearly identical to dense inference at 1,000 frames. These configurations run 3.26× and 4.11× faster, respectively (Table R20).
The more aggressive refined configuration provides further acceleration. On π³, it matches or improves dense reconstruction quality at every evaluated length and reaches 7.04× speedup. On HYWM, it reaches 5.21× speedup with a small increase in reconstruction error.
We will add this table so that the long-sequence runtime comparisons are accompanied by geometry measurements for all three backbones.
[Answer 4] Qualitative comparisons and in-the-wild examples (W4)
The manuscript includes long-sequence reconstruction comparisons in Figure 6 and Appendix Figure 11, along with KITTI and NRGBD examples in Appendix Figure 12.
We will add more qualitative comparisons in the appendix, including in-the-wild captures.
[Answer 5] Reporting corrections and typos (W5)
Thank you for catching these points. We will address each in the revision:
- π³ RE10K result: We will check the underlying evaluation output and use the same correctly rounded value throughout manuscript Tables 1, 7, and 12.
- Calibration scene counts: We will clarify that 13 scenes were used for the Figure 8 averages and for reference profiling and fitting in Appendix B.6, and correct any inconsistent reporting.
- Typo: We will change “Therefor” to “Therefore” in Section 4.3.
Official Comment by Authors
[Answer 6] Results on VGGT-Omega and Fast3R (Q1)
(a) VGGT-Omega. We have evaluated Zebra3D on VGGT-Omega and compared it with the dense model, A3D, and BlockSparse3D.
Table R21: VGGT-Omega. Sparsity is reported at 300 and 10 views.
| Method | Sparsity, 300 / 10 views | 7S Acc. ↓ | 7S Comp. ↓ | NRGBD Acc. ↓ | NRGBD Comp. ↓ | RE10K AUC-30 ↑ | RE10K RTA-5 ↑ | CO3Dv2 AUC-3 ↑ | CO3Dv2 AUC-30 ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Dense | 0% / 0% | 0.013 | 0.013 | 0.025 | 0.010 | 0.878 | 0.801 | 0.481 | 0.881 |
| A3D | 73% / 66% | 0.013 | 0.013 | 0.025 | 0.011 | 0.869 | 0.783 | 0.381 | 0.855 |
| BlockSparse3D | 70% / 70% | 0.013 | 0.013 | 0.034 | 0.013 | 0.863 | 0.771 | 0.367 | 0.855 |
| Zebra3D (0.4L/0.7S) | 78% / 72% | 0.013 | 0.013 | 0.024 | 0.010 | 0.876 | 0.797 | 0.450 | 0.874 |
Zebra3D matches or improves every reported quality metric relative to both efficient baselines while using greater sparsity (Table R21). It also stays close to dense VGGT-Omega, matching its 7Scenes reconstruction errors and remaining within 0.2 percentage points on RE10K AUC@30.
(b) Fast3R. We also test Fast3R, whose global-attention architecture provides a different test from VGGT-Omega. We select its layer allocation using its own sensitivity measurements.
Table R22: Fast3R reconstruction and pose estimation.
| Method | Mean sparsity at 300 views | 7Scenes Acc. ↓ | NRGBD Acc. ↓ | RE10K AUC-30 ↑ | CO3Dv2 AUC-30 ↑ |
|---|---|---|---|---|---|
| Dense Fast3R | 0% | 0.036 | 0.072 | 0.605 | 0.767 |
| A3D | 80.6% | 0.115 | 0.245 | 0.132 | 0.006 |
| Zebra3D (0.5L/0.6S) | 79.5% | 0.052 | 0.109 | 0.450 | 0.549 |
At similar sparsity, Zebra3D outperforms our A3D port on all four metrics, reducing reconstruction accuracy errors by approximately 55% on both datasets (Table R22). A gap to dense Fast3R remains, but Zebra3D preserves much more of its reconstruction and pose accuracy.
These additional experiments bring the evaluation to five backbones and show that the method also applies beyond architectures that alternate frame-wise and global attention. We will add both comparisons to the revised manuscript.
We sincerely thank the reviewer for the careful assessment and for recognizing Zebra3D’s practical conversion framework, sensitivity-guided allocation, and context-preservation mechanisms. In response, we ran the requested experiments, and they support our original claims. ** [Answer 1]** presents a matched ablation of the relational loss. [Answer 2] reports peak GPU memory for all backbones and methods up to 1,000 views. [Answer 3] extends the ScanNet-50 evaluation to HYWM and π³. ** [Answer 4]** clarifies the visual comparisons, and [Answer 5] corrects the reporting inconsistencies and typo. [Answer 6] evaluate Zebra3D on two additional backbones, VGGT-Omega and Fast3R. We hope these additional experiments and clarifications address the reviewer’s concerns.
We trained the requested control by removing only (Lrel), keeping the remaining losses, their weights, the training data, and the optimization settings unchanged. The loss weights are not renormalized.
The full feature objective is Lfeat=0.25L1+0.50Lcos+0.25Lrel while the control uses Lfeatw/o rel=0.25L1+0.50Lcos. Both retain the same task and output-distillation losses.
(a) Multi-view reconstruction and pose (Table R17). Without the relational loss, reconstruction degrades on 7Scenes and on ScanNet-50 at every tested sequence length (100 to 1000 frames). On NRGBD and RE10K the variant without it is slightly better, so the gain is not uniform across benchmarks, but it is consistent across all lengths of the long-sequence benchmark. The relational loss preserves the teacher's similarity structure between tokens of different views, and the resulting features are used for every view pair in a long sequence; its effect is visible on ScanNet-50 at all four lengths. A scene-level reconstruction score, however, averages over many view relations, which compresses per-pair improvements into a small aggregate gain. We therefore measure the per-pair effect directly with two-view pose estimation (Table R18), where the model must relate a single pair of views.
Table R17: Full objective versus removing only the relational term. VGGT, 0.5L/0.6S.
(b) Two-view pose estimation. To measure the per-pair effect behind the long-sequence gain, we evaluated MegaDepth-1500 [1] and ScanNet-1500 [2], where the model must relate a single pair of views directly. Here the full objective outperforms the variant without the relational loss on all eight metrics, with significant gains on six (Table R18). On MegaDepth, it even slightly exceeds the dense teacher at the three larger angular thresholds, while the control remains below the teacher.
Table R18: Two-view relative pose estimation. Zebra3D-VGGT(0.5L/0.6S). An asterisk indicates a statistically significant improvement over training without the relational loss (95% paired-bootstrap confidence interval excludes zero). Bold compares the student variants.
Summary. The matched ablation shows that the relational loss consistently improves long-sequences reconstruction (Table R17). Two-view pose estimation, which removes the compensation from intermediate views, isolates the underlying effect and shows it clearly (Table R18). This confirms that the long-sequence gain stems from better-preserved cross-view token relations, as the loss is designed to do. We will add these experiments to the revised manuscript.
[1] Li, Zhengqi, and Noah Snavely. "Megadepth: Learning single-view depth prediction from internet photos." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.
[2] Dai, Angela, et al. "Scannet: Richly-annotated 3d reconstructions of indoor scenes." 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, 2017.