acceptodds
Under review as a conference paper at ICLR 2027

A Simple Transformer Backend for Feed-Forward 3D Reconstruction

Abstract

Recent feed-forward 3D reconstruction methods such as DUSt3R and VGGT predict pointmaps and camera poses directly from images, yet their internal representations remain image-aligned and lack explicit cross-view geometric reasoning. AMB3R closes this gap by appending a sparse-voxel 3D backend to a frozen frontend, attributing its accuracy gains to the spatial compactness of voxelization. We hypothesize that a much simpler ingredient suffices: aggregating multi-view geometric features and injecting the refined features back into the frontend. We accordingly replace the voxel backend with a deliberately minimal alternative: a single pass of global self-attention over per-pixel geometric tokens, injected back into the frozen decoder through zero-initialized adapters, with no voxel grid, no space-filling-curve serialization, and no -nearest-neighbor interpolation. Despite its simplicity, the proposed backend is comparable to a voxel backend trained under matched conditions across zero-shot monocular depth, camera pose, multi-view depth, multi-view 3D reconstruction, and video depth. Controlled ablations on 3D positional encoding, the backend-to-frontend injection operator, and iterative refinement suggest that a single feed-forward pass with minimal design machinery is sufficient. Finally, on three additional frontends (, Déjà View, and Depth Anything 3), our backend performs comparably to or better than a voxel backend trained on the same frontend, suggesting that the mechanism studied in this paper transfers beyond VGGT.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D reconstruction, feed-forward 3D reconstruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pQp2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The work addresses the problem of inducing ‘shared geometric reasoning’ across features of corresponding pixels across multiple images in recent feed forward reconstruction.

Critical strengths

The work introduces a simple architecture to achieve results similar to AMB3R that addresses the explicit geometric reasoning in feed-forward reconstruction models.

Critical weaknesses
  1. Lack of novelty and the motivation paper, AMB3R, does and solves something different: Although the work is inspired by AMB3R, AMB3R solves the issue of making the geometric reasoning across features of corresponding pixels explicit rather than implicit. For this they introduce ‘space compactness’ that enforces the ‘explicit geometric reasoning’. This work, however, tries to achieve that via ‘no explicit’ measure and does so implicitly by extra tunable weights. This purpose is thus different from AMB3R that introduces explicit geometric reasoning, and it is not clear to me why a VGGT-like model with just extra transformer block (with alternating attention mechanism) is not enough to achieve what this work is doing. Also, since VGGT’s cross attention blocks attends to all other tokens of different images to achieve an implicit ‘geometric understanding’, how is the proposed work different from just adding extra transformer blocks? Just adding extra weights, whose global attention mechanisms have been widely used in literature, is not novel enough. Therefore, my conclusion is that this work is not solving geometric reasoning as done by it’s peer AMB3R and introducing significant novelty.
  2. Missing experiment: As said in weakness (1), a relevant study is that of adding the VGGT-style transformer block and just training more because I am not sure if the improvement is coming from just extra training. Furthermore, the experiment for Camera Pose Estimation task in this work utilizes only 10 random frames per-scene for evaluation which is not enough to say how well a method performs on a task. Therefore, for such task it has a very limited scope.
  3. Results are less convincing compared with VGGT: Although the work achieves performance similar to AMB3R’s reproduced version by the authors themselves, there is a good gap between the AMB3R’s publicly reported results. Moreover, I see very little convincing improvements compared to VGGT, a method whose geometric reasoning this work pursues to advance. For example on Multi-view reconstruction’s DTU and ETH3D benchmark VGGT performs 0.84 and 6.04 in ‘rel’ whereas the proposed work achieves 0.79 and 6.54. Specially, on ETH3D the performance is worse than VGGT itself. Therefore, it is less convincing that the proposed methodology performs a ‘good geometric reasoning.’
  4. Narrow Claim: For ICLR, the claim of the paper and the novelty proposed is very narrow as mentioned in the discussion section of the paper itself L463.
Questions for authors

N/A

2 comments

We thank the reviewer for the valuable feedback and address each question below.

We resolved the ETH3D gap in our reproduction of AMB3R by retraining both backends with updated training data (see question (2) in our reply to Reviewer UrkK). The results in this response come from these retrained models unless stated otherwise.

it is not clear to me why a VGGT-like model with just extra transformer block (with alternating attention mechanism) is not enough to achieve what this work is doing.

Adding Transformer blocks to VGGT does not reach the gains of our backend on 7-Scenes and ETH3D. We thank the reviewer for suggesting this control. We inserted six pairs of trainable VGGT blocks (per-view and global attention, 151M parameters, compared with 155M trainable parameters in our model) into the frozen VGGT, one pair every four layers. Each block is a copy of the preceding pretrained block with zero-initialized output projections. They are trained with the same data and schedule as our backend, with a learning rate of 10−610^{-6} for 30 epochs.

MethodDTU rel ↓7-Scenes rel ↓ETH3D rel ↓
VGGT0.845.426.04
AMB3R-reproduced0.834.954.44
VGGT + extra VGGT blocks0.756.184.57
Ours0.784.684.29

Our backend is better than the added blocks on 7-Scenes and ETH3D (4.68 vs. 6.18 and 4.29 vs. 4.57), while the added blocks are better on DTU (0.75 vs. 0.78). Both add trainable weights to the frozen VGGT, so the gains do not come from added capacity alone but from a separate backend whose output is injected into VGGT, which does not need to be a voxel backend.

The reviewer is right that our backend does not build the explicit shared 3D state of AMB3R, in which observations of the same location are fused into one voxel. The question we study is whether that spatially compact representation is necessary for the gains of a trainable backend. Our backend has no such representation and still has a lower rel than the reproduced voxel backend on all three datasets.

The most direct test is the ablation in Table 10 of our paper, which adds AMB3R-style voxel pooling and KNN interpolation to the same Transformer backend. With the retrained configuration, the pooling makes all three datasets worse.

MethodDTU rel ↓7-Scenes rel ↓ETH3D rel ↓
Ours0.784.684.29
Ours + voxel pooling0.804.834.66

Furthermore, the experiment for Camera Pose Estimation task in this work utilizes only 10 random frames per-scene for evaluation which is not enough to say how well a method performs on a task.

This is the RealEstate10K protocol of VGGT and AMB3R, which sample 10 random frames per scene and report AUC@30, and we use it to keep the numbers comparable.

Results are less convincing compared with VGGT

After retraining with the updated training data, our backend has a lower rel than VGGT on all three reconstruction datasets (0.84 to 0.78 on DTU, 5.42 to 4.68 on 7-Scenes, and 6.04 to 4.29 on ETH3D). The degradation on ETH3D in the submission was mainly caused by training the backends on Waymo with VGGT as the frontend, and it affected both backends. The comparison with the voxel backend does not depend on this change. With the submitted data, our backend also has a lower rel than the voxel backend on all three datasets (Table 4 of the submission).

Narrow Claim: For ICLR, the claim of the paper and the novelty proposed is very narrow

The claim tests the explanation that a recent system gives for the gains of its backend. AMB3R states that "this spatial compactness leads to state-of-the-art performance". Under matched training, a backend without a voxel grid, serialization or KNN interpolation obtains the comparable gains.

The revision states the claim for the evaluated setting with a frozen frontend, where spatially compact voxel fusion is not necessary for a trainable backend to match the gains of the voxel backend (Sections 1 and 5). We removed wording that suggests our model reproduces the explicit shared 3D state of AMB3R, and Section 1 now states that our backend does not build such a state. We also added the baseline suggested by the reviewer (Table 6f of the revision).

Dear Reviewer pQp2,

Thank you for reviewing our paper and for your thoughtful feedback. With the rebuttal period closing soon, we would sincerely appreciate any further feedback on our additional experiments and clarifications.

We believe that our additional experiments and responses have adequately addressed the concerns raised in your original review. Nevertheless, if you still consider they are not appropriate, we would be grateful if you could clarify which concerns remain most significant to your assessment in light of our additional evidence. We will do our best to address any remaining concerns within the discussion period.

Thank you again for your time and consideration.

Reviewer_8doXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper revisits the architectural choices of AMB3R, a feed-forward 3D reconstruction approach that improves a frozen pretrained model (e.g., VGGT) through a trainable sparse-voxel backend for aggregating multi-view geometric features. The authors question whether explicit voxel-based aggregation is necessary and propose replacing it with a lightweight transformer operating directly on image-aligned geometric features, followed by injection into the frozen frontend decoder. Experiments across several reconstruction and depth estimation benchmarks show performance largely comparable to a reproduced AMB3R baseline. The authors further ablate positional encodings, injection mechanisms, and iterative refinement, and evaluate the approach with several additional frontends.

Critical strengths
  • Clear research question and narrative. The paper is well focused on understanding which components of the AMB3R backend are actually necessary, rather than introducing additional architectural complexity.

  • Systematic ablations. The experiments examine several architectural choices under largely controlled conditions, providing useful evidence that voxelization, explicit 3D positional encoding, and iterative refinement are not essential in the considered setting.

  • Simplicity. The proposed backend is substantially smaller than the voxel alternative (approximately 8M versus 153M parameters), while achieving comparable reconstruction quality.

Critical weaknesses
  • Limited scope and novelty. The paper is largely an architectural simplification and ablation of AMB3R. While this is useful, the contribution feels relatively narrow. In particular, the original AMB3R paper already evaluates a 2D attention-based backend (Table 14), which seems like a particularly relevant baseline.

  • Limited efficiency analysis. Despite the substantially smaller parameter count (~8M versus ~153M), the proposed method does not consistently improve inference time. In fact, it is slightly slower than AMB3R on the longer 7-Scenes sequences (2.42 versus 2.25 seconds, including evaluation overhead). Given the quadratic complexity of global attention, a systematic evaluation of runtime and peak memory as a function of the number of input views would strengthen the practical motivation.

Questions for authors
  1. Could the authors compare their approach against the 2D attention-based backend evaluated in the original AMB3R paper (Table 14)? This seems particularly relevant for establishing the novelty and advantages of the proposed architecture.

  2. Could the authors provide a more systematic analysis of inference time and peak memory consumption as a function of the number of input views? Given the quadratic complexity of global attention, it would be useful to understand how the proposed backend scales relative to AMB3R and whether its substantially smaller parameter count translates into practical efficiency gains.

1 comment

We appreciate the constructive comments and questions from the reviewer. Our responses are below.

We resolved the ETH3D gap in our reproduction of AMB3R by retraining both backends with updated training data (see question (2) in our reply to Reviewer UrkK). The results in this response come from these retrained models unless stated otherwise.

Limited scope and novelty. The paper is largely an architectural simplification and ablation of AMB3R. While this is useful, the contribution feels relatively narrow.

Our contribution is to show that spatially compact voxel fusion, which AMB3R regards as the source of its gains, is not necessary for a trainable backend to obtain them. AMB3R states that "this spatial compactness leads to state-of-the-art performance" and supports this with the gap to its 2D backend in Table 14. We test this claim directly. Adding AMB3R-style voxel pooling to our Transformer backend does not improve DTU, 7-Scenes or ETH3D (Table 10), and with the retrained configuration it makes all three worse (see our reply to Reviewer pQp2). Under matched training, our backend also has a lower rel than the voxel backend on all three datasets.

Could the authors compare their approach against the 2D attention-based backend evaluated in the original AMB3R paper (Table 14)?

We thank the reviewer for pointing to this ablation. AMB3R describes this baseline only as a "2D backend implemented by alternating attention", and to our knowledge its implementation is not part of the released code. We therefore cannot reproduce it exactly. As a closely related control, we retrained our backend with its four blocks alternating between per-view and global attention (Table 6e of the revision). The data, schedule and number of parameters are unchanged.

MethodDTU rel ↓7-Scenes rel ↓ETH3D rel ↓
VGGT0.845.426.04
AMB3R-reproduced0.834.954.44
Ours (alternating attention)0.764.704.32
Ours (global attention)0.784.684.29

After 30 epochs, the alternating variant performs on par with global attention (7-Scenes rel 4.70 vs. 4.68, ETH3D 4.32 vs. 4.29, DTU 0.76 vs. 0.78). In AMB3R's Table 14, the 2D backend is behind the 3D backend on all three datasets. Under matched training we do not observe this gap. Both voxel-free variants have a lower rel than the reproduced voxel backend on all three datasets.

AMB3R's Table 14 leaves open whether the gap between the 2D and the 3D backends shows that compact fusion is required. The voxel-pooling ablation of Table 10 addresses this more directly, because it adds the pooling to the same backend and keeps the rest of the backend unchanged.

Could the authors provide a more systematic analysis of inference time and peak memory consumption as a function of the number of input views?

We measured the full models for 8 to 128 input views of one 7-Scenes sequence on a single H100 GPU. Time is the median of three forward passes after a warm-up, and memory is the peak allocated by PyTorch.

ViewsAMB3R time (s)Ours (global attention) time (s)Per-view attention time (s)AMB3R memory (GiB)Ours memory (GiB)
80.550.480.449.046.57
161.081.120.9711.857.64
322.492.962.3613.2810.07
646.669.096.5016.2616.27
12820.7632.5220.5928.5528.67

Our model uses less memory than AMB3R up to 32 views and the same memory beyond. It is about as fast up to 16 views, but for longer inputs the quadratic cost of global attention makes it slower (by 57% at 128 views). For reference, VGGT alone takes 0.22, 0.47, 1.14, 3.17 and 10.13 s and uses 6.19, 6.85, 8.47, 13.03 and 22.16 GiB for 8 to 128 views. VGGT itself applies global attention over all views, so its cost already grows quadratically with the number of views, and it is best suited to short inputs, the regime in which our model uses less memory than AMB3R and is about as fast (Appendix D of the revision). The per-view attention variant of Table 6e, which performs comparably to global attention after 30 epochs, avoids the quadratic cost of the backend. It is at least as fast as AMB3R for 8 to 128 views and uses about the same memory as our default model (Appendix D of the revision). We keep global attention as the default because it has an advantage with fewer training samples (Section 4.3 of the revision).

Reviewer_UrkKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a simple voxel-free Transformer backend for improving pretrained feed-forward 3D reconstruction models. The work revisits the central hypothesis of AMB3R that spatially compact voxel representations are essential for effective geometric refinement. The authors argue that the improvements of voxel-based backends may instead arise primarily from cross-view feature exchange and subsequent feature injection into the pretrained reconstruction frontend.

To investigate this hypothesis, the authors replace AMB3R's voxelization, feature pooling, spatial serialization, and KNN interpolation with a standard Transformer operating directly on flattened multi-view geometric tokens. The refined features are injected into the frozen frontend decoder through zero-initialized adapters. The framework also investigates several architectural choices, including absolute and relative 3D positional encoding, different injection mechanisms, and recurrent or stacked refinement. The proposed backend is trained using the same reconstruction supervision as AMB3R, while keeping the pretrained frontend frozen.

Extensive experiments cover monocular depth estimation, camera pose estimation, multi-view depth estimation, 3D reconstruction, and video depth estimation. The proposed method achieves performance comparable to a reproduced AMB3R voxel backend under matched training conditions, while using substantially fewer backend parameters. Additional experiments with π3, Déjà View, and Depth Anything 3 demonstrate the applicability of the approach to different reconstruction frontends. The authors conclude that explicit voxel-based spatial compactness is not necessary to obtain backend refinement gains in the evaluated setting.

Critical strengths

(1) The paper investigates whether the architectural complexity of existing voxel-based backends is actually necessary for improving feed-forward 3D reconstruction. This is an interesting question that could help simplify future reconstruction systems. Rather than introducing another increasingly complex reconstruction architecture, the authors revisit an important assumption underlying AMB3R and attempt to isolate the mechanisms responsible for its improvements. This investigation provides useful empirical insights into the role of spatial representations and cross-view feature exchange, potentially informing the design of simpler reconstruction systems. (2) The proposed framework replaces several specialized geometric processing operations with standard self-attention while achieving competitive reconstruction performance. The reduction in backend parameters from approximately 153M to 8M is particularly appealing. Moreover, the architecture requires relatively few specialized components and can be integrated with pretrained reconstruction models through a lightweight feature-injection interface, making it a potentially practical alternative to more complicated geometric backends. (3) The paper evaluates multiple geometric tasks and reconstruction backbones, providing evidence that the proposed design is not restricted to VGGT. The controlled comparisons and additional voxel-pooling ablation also support the investigation of spatial compactness.

Critical weaknesses

(1) The central conclusion is that cross-view geometric feature exchange followed by frontend injection is sufficient to recover the benefits of voxel-based refinement. However, the proposed Transformer performs global self-attention over all image tokens, without explicitly distinguishing within-view refinement from cross-view interactions. Since pretrained frontends such as VGGT already incorporate substantial multi-view geometric information, the improvements could potentially arise from additional feature processing capacity rather than explicit cross-view reasoning. The paper lacks controlled comparisons with per-view attention or simpler feature-refinement modules, making it difficult to identify the actual mechanism responsible for the improvements. (2) Although the authors reproduce AMB3R under matched training conditions, the reproduced voxel backend performs substantially worse than the publicly released checkpoint on certain benchmarks. For example, on ETH3D, the reproduced AMB3R obtains a reconstruction relative error of 7.39, compared with 4.63 for the public checkpoint, while the proposed method achieves 6.54. The authors acknowledge possible differences in data preprocessing and sampling, but their relative contributions remain unclear. Although the matched comparison is useful for isolating architectural differences, this reproduction gap limits how confidently the findings can be generalized to the original AMB3R training regime. (3) The architectural ablations in Table 6 are conducted using a shorter six-epoch training schedule, whereas the main results use thirty epochs with a different learning-rate configuration. It is therefore unclear whether the conclusions regarding positional encoding, injection mechanisms, and iterative refinement remain valid under the full training setup. Furthermore, recurrent refinement repeatedly applies a model trained for a single refinement pass, rather than explicitly training it to handle iterative updates. The observed degradation may therefore reflect a mismatch between training and inference rather than demonstrate that iterative refinement is inherently unnecessary.

Questions for authors

(1) Could the authors provide a controlled comparison between global cross-view attention and attention restricted to individual views, while keeping the same backend architecture and training configuration? Additionally, how does the proposed approach compare with a lightweight per-token refinement module without self-attention? (2) Could the authors clarify which differences between the reproduced AMB3R training setup and the original training configuration are most likely responsible for the substantial performance gap on ETH3D? In particular, is there evidence that the relative advantage of the voxel-free backend remains consistent under alternative data mixtures or a training configuration closer to the public AMB3R checkpoint? (3) Could the authors clarify whether the conclusions from the six-epoch architectural ablations remain consistent with the full thirty-epoch training configuration? In particular, has iterative refinement been evaluated with an explicitly trained recurrent model rather than repeatedly applying a single-pass model at inference?

2 comments

We thank the reviewer for the detailed comments and questions, which led to several new experiments. Our responses are below.

We resolved the ETH3D gap in our reproduction of AMB3R by retraining both backends with updated training data, as described under (2). The results in this response come from these retrained models unless stated otherwise.

(2) Could the authors clarify which differences between the reproduced AMB3R training setup and the original training configuration are most likely responsible for the substantial performance gap on ETH3D? In particular, is there evidence that the relative advantage of the voxel-free backend remains consistent under alternative data mixtures or a training configuration closer to the public AMB3R checkpoint?

We found that training on Waymo was the main cause of the ETH3D degradation in our setup. On our Waymo training samples, the public checkpoint of AMB3R leaves the predictions of VGGT almost unchanged in depth, points and cameras, whereas our reproduction changes them strongly. We therefore removed Waymo from the training to follow the behavior of the public checkpoint, as AMB3R itself excludes Co3D because "VGGT might overfit on certain patterns on Co3D" (Sec. 8.2 of AMB3R). With the proportions of Appendix A.2 of the submission, the ETH3D rel of our backend is 4.69 without Waymo, compared with 6.54 in Table 4 of the submission. We also split the indoor samples equally among the four indoor datasets and raised VKITTI2 from 30 to 300 samples, instead of these proportions. Each epoch now has 1,700 samples (Appendix A.2 of the revision), with 175 each from ScanNet, ScanNet++, Hypersim and Aria (indoor), 245 from WildRGBD and 55 from OmniObject3D (object-centric), 300 from VKITTI2 (driving), and 235 from MapFree, 110 from GTA-SfM and 55 from MVS-Synth (outdoor). Both backends were retrained from scratch with these data.

Training dataMethodDTU rel ↓7-Scenes rel ↓ETH3D rel ↓
VGGT0.845.426.04
AMB3R-public0.814.524.63
SubmittedAMB3R-reproduced0.845.557.39
SubmittedOurs0.794.486.54
RetrainedAMB3R-reproduced0.834.954.44
RetrainedOurs0.784.684.29

All rows except the retrained ones are the values of Table 4 of the submission. After retraining, the reproduced AMB3R reaches 4.44 on ETH3D, compared with 4.63 for the public checkpoint. Both backends have a lower rel than VGGT on all three datasets. The retrained models also close the gaps to the public checkpoint that the submission had on KITTI and camera pose, and our retrained model stays within 0.5 rel of its submitted values on the other datasets.

Training dataMethodMonocular KITTI rel ↓Multi-view KITTI rel ↓Video KITTI rel ↓Pose AUC@30 ↑
AMB3R-public7.22.84.186.2
SubmittedAMB3R-reproduced8.03.45.486.0
SubmittedOurs8.43.45.785.6
RetrainedAMB3R-reproduced7.32.83.886.7
RetrainedOurs7.22.94.186.4

For the second part of the question, the reconstruction table contains two matched configurations, the submitted one and the retrained one. In both, our backend has a lower rel than AMB3R-reproduced on all three datasets. We have updated all results with VGGT as the frontend in the revision.

(1) Could the authors provide a controlled comparison between global cross-view attention and attention restricted to individual views, while keeping the same backend architecture and training configuration? Additionally, how does the proposed approach compare with a lightweight per-token refinement module without self-attention?

We ran both controls with the same training configuration as our backend and added them to Table 6e of the revision. In the first, the self-attention of the backend is restricted to tokens of the same view. In the second, the attention layers are removed, so that each block is a per-token MLP.

MethodDTU rel ↓7-Scenes rel ↓ETH3D rel ↓
VGGT0.845.426.04
AMB3R-reproduced0.834.954.44
Ours (global attention)0.784.684.29
Per-view attention0.774.684.22
Per-token MLP (no attention)0.764.684.62

After 30 epochs, the three voxel-free variants reach almost the same rel on DTU and 7-Scenes, while the per-token MLP is behind on ETH3D (4.62 vs. 4.29). The variants differ more earlier in training. At epoch 6 of the same 30-epoch runs, global attention is clearly ahead of both controls on 7-Scenes and ETH3D.

Method (epoch 6)DTU rel ↓7-Scenes rel ↓ETH3D rel ↓
Ours (global attention)0.825.314.13
Per-view attention0.825.685.03
Per-token MLP (no attention)0.805.774.32

(continued in the next comment)

Thus, after 30 epochs, per-view attention performs as well as global attention, and even the per-token MLP without attention performs similarly on DTU and 7-Scenes. A likely reason for both is that the frozen VGGT performs cross-view reasoning after the injection. That even these simpler backends obtain similar gains further supports our claim that the gains do not require an elaborate backend design, i.e., sparse voxel fusion. Adding trainable blocks inside VGGT is not enough, however, as the control in our reply to Reviewer pQp2 shows (7-Scenes 6.18, ETH3D 4.57). The gains come from a separate backend whose output is injected into VGGT, not from the attention design inside the backend. Global attention in the backend is still ahead with fewer training samples. We have revised the abstract, contribution (iii) and the discussion accordingly, removed the statement that global cross-view exchange in the backend is the source of the gain, and instead stated that spatially compact voxel fusion is not necessary for a trainable backend to match the gains of the voxel backend. These experiments support this claim, since all three voxel-free variants have a lower rel than the reproduced voxel backend on DTU and 7-Scenes, and the two attention variants are also lower on ETH3D. Adding voxel pooling to our backend also makes all three datasets worse after retraining (see our reply to Reviewer pQp2).

(3) Could the authors clarify whether the conclusions from the six-epoch architectural ablations remain consistent with the full thirty-epoch training configuration? In particular, has iterative refinement been evaluated with an explicitly trained recurrent model rather than repeatedly applying a single-pass model at inference?

We retrained the variants of Table 6 of the submission for 30 epochs with the same training data as the retrained models in (2) and report them in Table 6a–6d of the revision. The 2- and 3-stage variants add a second and a third backend on top of the frozen 30-epoch default and 2-stage models, respectively.

VariantDTU rel ↓7-Scenes rel ↓ETH3D rel ↓
Default (3D RoPE/ZeroConv/1 stage)0.784.684.29
No PE0.774.664.28
Absolute PE0.775.114.21
AdaLN-Zero0.844.874.40
2 stages (extra parameters)0.775.054.39
3 stages (extra parameters)0.775.114.47

Removing the positional encoding (no PE) still has no measurable effect. Absolute positional encoding is slightly better than 3D RoPE on DTU and ETH3D and worse on 7-Scenes, so neither positional encoding is consistently better. Additional stages (2/3 stages) are slightly better on DTU (0.77 vs. 0.78) but worse on 7-Scenes and ETH3D, and three stages are worse than two on both.

For AdaLN-Zero, the conclusion changes on DTU. It is similar to zero-convolution on 7-Scenes and ETH3D but worse on DTU (0.84 vs. 0.78), whereas the two were similar on DTU in Table 6 of the submission.

For recurrent inference (no extra parameters), we repeated the experiment of Table 6 of the submission with the 30-epoch model, applying the single-pass model for several rounds.

RoundsDTU rel ↓7-Scenes rel ↓ETH3D rel ↓
1 round0.784.684.29
2 rounds0.966.286.38
3 rounds1.747.099.18

The error grows with the number of rounds, as on 7-Scenes in Table 6 of the submission, and now also on DTU.

Overall, the conclusions of Table 6 of the submission remain unchanged with the 30-epoch configuration, except for AdaLN-Zero on DTU. The gains are not explained by explicit 3D positional signals, since removing the positional encoding has no measurable effect. This also supports our main claim, because a backend that does not use 3D positions in its attention obtains the same gains, so grouping the tokens by their 3D positions, as voxel fusion does, is not required. Adding a second or third backend or applying the trained single-pass backend repeatedly adds cost without a consistent improvement over a single pass.

For the second part of the question, we trained a recurrent model (Appendix F of the revision). During training, the same backend is applied for 1 to 4 rounds, chosen at random at each step, and the loss is computed on the output of the last round. We evaluate it with 1 to 8 rounds.

RoundsDTU rel ↓7-Scenes rel ↓ETH3D rel ↓
1 round0.764.794.81
2 rounds0.764.924.81
4 rounds0.764.934.93
8 rounds0.764.934.96

With more rounds, DTU stays at 0.76, and 7-Scenes and ETH3D become slightly worse. Recurrent training therefore does not change the conclusion of Table 6 that additional rounds add cost without improving over a single pass.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in the Revision

Dear AC and Reviewers,

Thank you for the careful reviews and for the concrete suggestions, which helped us improve the paper. We have uploaded a revised version in which the main changes are marked in blue. We summarize the changes below.

  • Retrained models (Appendix A.2 and A.3). We retrained both our backend and the reproduced AMB3R backend with the updated training data described in our reply to Reviewer UrkK. All results with VGGT as the frontend in Tables 1–6 and 10 now use this configuration. Both backends now improve over VGGT on all three reconstruction benchmarks, and our backend keeps a lower rel than the reproduced voxel backend on all three.
  • Revised claims (Abstract, Sections 1, 2, 3.2, 4.3, 4.4, 5 and 6). Following our replies to Reviewers UrkK and pQp2, the paper states that spatially compact voxel fusion is not necessary for the backend gains, and it no longer presents cross-view exchange inside the backend as the source of the gains. Section 1 also states that our backend does not build the explicit shared 3D state of AMB3R.
  • New controls on the backend (Section 4.3, Table 6e and 6f). As requested by Reviewer UrkK, we added a backend with attention restricted to each view and a per-token MLP without attention. As a closely related control for the 2D backend of AMB3R raised by Reviewer 8doX, we added a backend that alternates per-view and global attention. As suggested by Reviewer pQp2, we inserted six pairs of trainable VGGT blocks into the frozen VGGT instead of a separate backend, with a similar number of trainable parameters (151M vs. 155M).
  • Ablations with the full training schedule (Section 4.3, Table 6a–6d). As requested by Reviewer UrkK, all ablations are now trained for 30 epochs with the retrained configuration and are also evaluated on ETH3D.
  • Recurrent training (Appendix F, Table 11). As requested by Reviewer UrkK, we trained a backend for repeated rounds and evaluated it with up to eight rounds.
  • Inference time and memory (Appendix D, Tables 8 and 9). As requested by Reviewer 8doX, we report the inference time and peak memory for 8 to 128 input views. We also report the per-view attention variant of Table 6e, which performs comparably to global attention after full training and is at least as fast as AMB3R for every number of views. We keep global attention as the default because it has an advantage with fewer training samples (Section 4.3).
  • Voxel pooling (Appendix E, Table 10). The ablation that adds AMB3R-style voxel pooling to our backend was retrained with the new configuration.
  • Other changes. The definition of AdaLN-Zero moved to Appendix A.4, and the former appendix on the ETH3D degradation was removed, since this degradation no longer occurs with the retrained configuration.
  • Other frontends (Table 7). The results with π3\pi^3, Déjà View and Depth Anything 3 still use the earlier training setup, as noted in the caption of Table 7. We will update them with the retrained configuration in the final version.

We hope that these revisions adequately address the concerns raised in the reviews. We are truly grateful for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.