- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The work addresses the problem of inducing ‘shared geometric reasoning’ across features of corresponding pixels across multiple images in recent feed forward reconstruction.
Critical strengths
The work introduces a simple architecture to achieve results similar to AMB3R that addresses the explicit geometric reasoning in feed-forward reconstruction models.
Critical weaknesses
- Lack of novelty and the motivation paper, AMB3R, does and solves something different: Although the work is inspired by AMB3R, AMB3R solves the issue of making the geometric reasoning across features of corresponding pixels explicit rather than implicit. For this they introduce ‘space compactness’ that enforces the ‘explicit geometric reasoning’. This work, however, tries to achieve that via ‘no explicit’ measure and does so implicitly by extra tunable weights. This purpose is thus different from AMB3R that introduces explicit geometric reasoning, and it is not clear to me why a VGGT-like model with just extra transformer block (with alternating attention mechanism) is not enough to achieve what this work is doing. Also, since VGGT’s cross attention blocks attends to all other tokens of different images to achieve an implicit ‘geometric understanding’, how is the proposed work different from just adding extra transformer blocks? Just adding extra weights, whose global attention mechanisms have been widely used in literature, is not novel enough. Therefore, my conclusion is that this work is not solving geometric reasoning as done by it’s peer AMB3R and introducing significant novelty.
- Missing experiment: As said in weakness (1), a relevant study is that of adding the VGGT-style transformer block and just training more because I am not sure if the improvement is coming from just extra training. Furthermore, the experiment for Camera Pose Estimation task in this work utilizes only 10 random frames per-scene for evaluation which is not enough to say how well a method performs on a task. Therefore, for such task it has a very limited scope.
- Results are less convincing compared with VGGT: Although the work achieves performance similar to AMB3R’s reproduced version by the authors themselves, there is a good gap between the AMB3R’s publicly reported results. Moreover, I see very little convincing improvements compared to VGGT, a method whose geometric reasoning this work pursues to advance. For example on Multi-view reconstruction’s DTU and ETH3D benchmark VGGT performs 0.84 and 6.04 in ‘rel’ whereas the proposed work achieves 0.79 and 6.54. Specially, on ETH3D the performance is worse than VGGT itself. Therefore, it is less convincing that the proposed methodology performs a ‘good geometric reasoning.’
- Narrow Claim: For ICLR, the claim of the paper and the novelty proposed is very narrow as mentioned in the discussion section of the paper itself L463.
Questions for authors
N/A
2 comments
Dear Reviewer pQp2,
Thank you for reviewing our paper and for your thoughtful feedback. With the rebuttal period closing soon, we would sincerely appreciate any further feedback on our additional experiments and clarifications.
We believe that our additional experiments and responses have adequately addressed the concerns raised in your original review. Nevertheless, if you still consider they are not appropriate, we would be grateful if you could clarify which concerns remain most significant to your assessment in light of our additional evidence. We will do our best to address any remaining concerns within the discussion period.
Thank you again for your time and consideration.
We thank the reviewer for the valuable feedback and address each question below.
We resolved the ETH3D gap in our reproduction of AMB3R by retraining both backends with updated training data (see question (2) in our reply to Reviewer UrkK). The results in this response come from these retrained models unless stated otherwise.
Adding Transformer blocks to VGGT does not reach the gains of our backend on 7-Scenes and ETH3D. We thank the reviewer for suggesting this control. We inserted six pairs of trainable VGGT blocks (per-view and global attention, 151M parameters, compared with 155M trainable parameters in our model) into the frozen VGGT, one pair every four layers. Each block is a copy of the preceding pretrained block with zero-initialized output projections. They are trained with the same data and schedule as our backend, with a learning rate of 10−6 for 30 epochs.
Our backend is better than the added blocks on 7-Scenes and ETH3D (4.68 vs. 6.18 and 4.29 vs. 4.57), while the added blocks are better on DTU (0.75 vs. 0.78). Both add trainable weights to the frozen VGGT, so the gains do not come from added capacity alone but from a separate backend whose output is injected into VGGT, which does not need to be a voxel backend.
The reviewer is right that our backend does not build the explicit shared 3D state of AMB3R, in which observations of the same location are fused into one voxel. The question we study is whether that spatially compact representation is necessary for the gains of a trainable backend. Our backend has no such representation and still has a lower rel than the reproduced voxel backend on all three datasets.
The most direct test is the ablation in Table 10 of our paper, which adds AMB3R-style voxel pooling and KNN interpolation to the same Transformer backend. With the retrained configuration, the pooling makes all three datasets worse.
This is the RealEstate10K protocol of VGGT and AMB3R, which sample 10 random frames per scene and report AUC@30, and we use it to keep the numbers comparable.
After retraining with the updated training data, our backend has a lower rel than VGGT on all three reconstruction datasets (0.84 to 0.78 on DTU, 5.42 to 4.68 on 7-Scenes, and 6.04 to 4.29 on ETH3D). The degradation on ETH3D in the submission was mainly caused by training the backends on Waymo with VGGT as the frontend, and it affected both backends. The comparison with the voxel backend does not depend on this change. With the submitted data, our backend also has a lower rel than the voxel backend on all three datasets (Table 4 of the submission).
The claim tests the explanation that a recent system gives for the gains of its backend. AMB3R states that "this spatial compactness leads to state-of-the-art performance". Under matched training, a backend without a voxel grid, serialization or KNN interpolation obtains the comparable gains.
The revision states the claim for the evaluated setting with a frozen frontend, where spatially compact voxel fusion is not necessary for a trainable backend to match the gains of the voxel backend (Sections 1 and 5). We removed wording that suggests our model reproduces the explicit shared 3D state of AMB3R, and Section 1 now states that our backend does not build such a state. We also added the baseline suggested by the reviewer (Table 6f of the revision).