acceptodds
Withdrawn from ICLR 2027

Anderson Acceleration of Test-Time Training for Feed-Forward 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed-forward 3D reconstruction predicts a scene representation from a few input views in a single forward pass, enabling efficient reconstruction without scene-specific optimization. Despite their efficiency, these methods often suffer from limited reconstruction quality and generalization to unseen scenes. To address this limitation, recent approaches introduce test-time training (TTT), which leverages supervision from test-time observations to adapt the model to each target scene. However, this scene-specific adaptation substantially increases inference time and diminishes the efficiency advantage of feed-forward reconstruction. To this end, we propose AA-TTT, a test-time training framework that formulates TTT as a fixed-point iteration and accelerates the process via Anderson acceleration. By directly exploiting the TTT trajectory, AA-TTT is applicable to a wide range of feed-forward 3D reconstruction systems. Extensive experiments on four representative TTT-based reconstruction systems demonstrate that AA-TTT consistently preserves reconstruction quality while achieving an average 1.4 test-time training speed across diverse reconstruction frameworks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Feed-Forward 3D Reconstruction, Anderson Acceleration, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_t4LkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents AA-TTT, a framework for accelerating the convergence of test-time training for 3D reconstruction. AA-TTT formulates test-time optimization as a fixed-point problem and uses Anderson acceleration to improve its convergence speed. The method presented can be applied to different reconstruction architectures and optimization variables.

Critical strengths

The paper provides a formulation of test-time training that is independent of the architecture and of the variables being optimized. This general-purpose framework can be applied to optimize Gaussian parameters, implicit tokens, feature maps, or network adapters.

The paper provides consistent speed-up in all the evaluations reported without any loss in quality of the reconstructed outputs

Critical weaknesses

The paper does not report enough details of the evaluation setup for the experiments. It does not specify what datasets and scenes each method is evaluated on. It also does not specify what scenes the learning rate is tuned on and whether they are part of the evaluation set.

The paper only compares against other first-order optimizers. Since Anderson acceleration can be interpreted as a quasi-Newton method, it would be useful to compare against second-order baselines like 3DGS-LM [1], and quasi-Newton baselines like L-BFGS. Additionally, the paper should discuss learned acceleration methods like ReSplat [2].

[1] 3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt, Hollen et al., ICCV 2025

[2] ReSplat: Learning Recurrent Gaussian Splatting, Xu et al., ECCV 2026

The paper does not provide any visualizations of the method outputs.

Questions for authors

Can the authors provide the missing details for the experiments? Additionally, the 40 scenes subset used in Table 3 seems too small to draw general conclusions, can the authors provide results on the full evaluation set?

Can the authors include a comparison against L-BFGS in Table 3?

Reviewer_GyKKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AA-TTT, a method that uses Anderson Accelerator for the optimization step after feedforward 3D reconstruction. Specifically, instead of using step-by-step gradient descent, the authors use Anderson accelerator, which combines the differences from several recent steps and find an optimal step that has smaller residual and reduces number of optimization steps. It also includes safeguards to avoid unreliable updates. The authors did experiments by applying AA-TTT to TokenGS and other methods.

Critical strengths
  • The authors applied AA-TTT for test time optimization, which is a simple formulation and might reduce the optimization budget.
  • The method is plug-and-play and compatible with different baselines.
  • The paper is easy to follow and understand.
Critical weaknesses
  • The paper directly apply AA to test-time optimization without much new things, it is also unclear why the author chooses 3D reconstruction as the task.
  • This paper doesn’t contain any visual results or pipeline figure. As its task is novel view synthesis, figures are generally required. Lack of visual results make the reader unable to understand where the gains come from.
  • The improvement over the baseline is quite small, i.e., 0.1PSNR, which can come from same method runs using different seeds. It is hard to believe that whether AA-TTT is really effective.
Questions for authors
  • What datasets and numbers of scenes were used for each of the four systems in Tables 1 and 2?
  • How the number of history depth M is selected?
Reviewer_MwbFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper applies Anderson acceleration to the optimization performed after a feed-forward 3D reconstruction. It uses recent states and update residuals to predict an accelerated step, checks the TTT loss, and falls back to the native update when the prediction is rejected. Experiments on TCO, QuerySplat, C3G, and TokenGS report an average 1.4× speedup in the TTT stage, with equal or better reported reconstruction quality.

Critical strengths

The idea is tested on 4 different optimization variables. TCO updates adapters, QuerySplat updates features, C3G updates explicit Gaussians, and TokenGS updates latent tokens. The result shows improvement in every listed metric, showing the effectiveness and generalizability of the proposed method.

Critical weaknesses
  1. Lack of comparison setting details. Section 6 does not identify the datasets, view settings, or prior configurations behind the main tables. It also does not explain how the iteration budgets were chosen to reach “comparable reconstruction quality.” The original methods have multiple settings, so referring to their defaults does not settle this. Quality-versus-time curves and repeated timings would show whether that advantage holds beyond the selected points. The times should include coefficient solving, safeguard checks, and rollback.
  2. What was required to apply AA to these optimizers? Section 4 calls T the complete native update, but leaves out the optimizer state and sampling rules. TCO uses Adam and randomly chooses a source view at each iteration, what happened to Adam's state after an AA prediction or rollback? And for C3G, it describes densification during TTO. If this is retained, the Gaussian set can change, and the paper needs to explain what happens to the stored AA history.

Although “test-time training” has a broader history, “per-scene test-time optimization” would describe this setting more clearly. Please distinguish it from the TTT memory layers used in ZipMap, Scal3R, and LoGeR, etc.

Questions for authors

I lean toward weak rejection because the paper does not make the quality-matched timing comparison reproducible or explain how AA interacts with the native optimization procedures. Since the contribution is largely the application of established AA machinery, these details are central to my assessment. The following clarifications could change my rating:

  1. Quality matching. Which datasets and view/prior settings were used for Tables 1–3? How were the quality targets and iteration budgets selected, and were they fixed before evaluation? Please state which metrics and tolerances define a match.
  2. The tuned TokenGS comparison. Does AA retain its advantage over tuned AdamW across a useful range of reconstruction quality? Curves for default TokenGS, tuned AdamW, and AA-TTT, including all acceleration costs, would answer this.
  3. Optimizer state and sampling. What is preserved or reset after an AA prediction or rollback? For TCO's stochastic updates, are the safeguard losses computed using the same observations?
  4. Changing the Gaussian set. Is C3G's densification retained? If so, how are states and residuals in the AA history handled when the set changes? If not, does the baseline use the same modification?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.