acceptodds
Withdrawn from ICLR 2027

AdaVGGT: Self-Supervised Test-Time Adaptation for Video-based 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

45 views

Abstract

Feed-forward geometry models reconstruct 3D scenes in a single pass, but repeatedly applying frozen weights to long videos leaves scene-specific errors uncorrected. Conversely, traditional per-scene optimization aligns observations to a target scene but lacks general priors and incurs heavy computation. We bridge these two paradigms with AdaVGGT, a self-supervised online test-time adaptation method that specializes pretrained predictors to a test video on the fly. We freeze the backbone of VGGT and jointly optimize lightweight low-rank (LoRA) corrections on its depth and camera pose heads during inference. Our objective requires no offline training, auxiliary datasets, or external optical flow; instead, it refines predictions directly on current observations via multi-view photometric reprojection and cross-frame depth consistency. To process long sequences online, adaptation proceeds causal-sequentially on overlapping temporal chunks, where each chunk inherits the optimized parameters and optimizer momentum from its predecessor. Across five real-world and synthetic outdoor, driving, and indoor video benchmarks, AdaVGGT consistently improves camera tracking and depth estimation over the unadapted foundation model. Notably, on KITTI Odometry, adaptation cuts the mean tracking error of VGGT from to — a reduction — outperforming both classical vSLAM baselines and existing test-time optimization methods. These results demonstrate how strong general priors and scene-specific self-supervision can work together to enable robust, pixel-aligned 3D reconstruction at deployment. Code and models will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Test-Time Adaptation, Feed-Forward Geometry Models, 3D Foundation Models, Self-Supervised Learning, Video 3D Reconstruction, Camera Tracking and Depth Estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q2qeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a self supervised online testing adaptation (TTA) framework called AdaVGGT to address the pain point of accumulated errors in long video reconstruction caused by fixed weights and the huge computational cost of traditional Per scene Optimization based on forward geometry large models (such as VGGT). Its core contributions and methods include: low rank adaptation (LoRA) head fine-tuning: freezing the feature extraction backbone network of VGGT, injecting lightweight LoRA adapters only on the deep prediction head and camera pose head, achieving low-cost and efficient online parameter updates. Without the need for offline training, annotated data, or external optical flow, the joint optimization of depth and pose prediction can be achieved by utilizing the multi view photometric reprojection residuals between current video frames and the cross frame depth consistency loss. We use sliding overlapping time blocks (Chunks) for online processing, and seamlessly transfer the LoRA parameters learned in the previous block and the momentum state (moment cache) of the Adam optimizer to the next block, achieving forward migration and fast preheating. Tests were conducted on five indoor and outdoor real and synthetic video benchmarks, including KITTI Odometry, VKITTI2, KITTI Depth, ScanNet++, and 7Scenes. Among them, the trajectory mean absolute error (ATE) was significantly reduced by 66% on KITTI Odometry.

Critical strengths
  1. Efficient online adaptive paradigm design: ingeniously combining the strong universal prior of the forward base model with the scene alignment ability optimized during testing. By fine-tuning only the lightweight LoRA header and caching the backbone features, single step updates only account for about 7% of the forward overhead, balancing accuracy and efficiency.

  2. Rigorous and solid causal state transmission mechanism: The ablation experiment proves that not only transmitting LoRA weight parameters, but also inheriting the momentum state (Full Carry) of Adam optimizer is crucial for smooth trajectory and convergence speed across blocks, effectively avoiding independent overfitting between blocks.

Critical weaknesses
  1. The ablation experiment (as shown in Table 9) showed that when the precise calibration parameters were removed and the model was used for prediction, the post TTA ATE on KITTI sequence 09 deteriorated sharply from 6.44 m to 21.52 m. This exposed that the method may face performance collapse in generalized "wild" video scenes lacking high-quality camera parameters.

2.Although overlapping partitioning and Sim (3) rigid alignment are used, the cumulative error between partitions in ultra long open-loop sequences may still lead to long-distance drift, and the method itself acknowledges that it still relies on non streaming sliding segmentation, which has limited support for true real-time infinite length streaming processing.

  1. Self supervised photometric loss ($l1 $+SSIM) depends on the assumption of Lambertian body reflection and constant lighting, but in real outdoor driving videos such as KITTI, it is often accompanied by dynamic changes in shadows, strong light reflection, water surface or glass reflection. Despite the introduction of composite masks (SAM3, etc.), it may still introduce erroneous gradients under complex lighting conditions.

4.Although the single step update time is relatively short (0.42 seconds), the entire framework has not undergone a comprehensive and in-depth benchmark evaluation of the peak memory and overall time overhead when dealing with sliding windows containing feature caching and multiple iterative optimizations (such as $N_j=70 $), which raises doubts about the deployment friendliness of edge or resource limited devices.

  1. There are significant differences in learning rate, window size ($w, o ),updatesteps(), update steps (N_j ),andlossweight(), and loss weight (\ lambda=0.5 $) between different datasets (such as indoor ScanNet++and outdoor KITTI) (see Table 7), indicating that the method lacks the ability to adaptively adjust hyperparameters in unknown scenes.
Questions for authors

1.Given the decisive impact of internal parameters on method performance (Table 9), what is the extent of performance degradation of AdaVGGT in daily mobile phone videos without camera calibration, requiring joint prediction or rough estimation of internal parameters?

  1. Do LoRA's update steps $N_j $or learning rate need to be manually re tuned for scenes with different motion speeds or a large number of dynamic objects? Have you considered designing a dynamic stopping strategy based on residual convergence?
Reviewer_rYGkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AdaVGGT adapts VGGT on the test video using calibrated intrinsics, photometric reprojection, and cross-frame depth consistency. It freezes the backbone to cache features, updates LoRA adapters in the depth and camera heads, and carries parameters and Adam state across overlapping windows before stitching their reconstructions. Evaluation covers five benchmarks spanning outdoor driving, synthetic environments, and indoor scenes. Across all eleven KITTI Odometry sequences, mean post-loop ATE decreases from 18.09 to 6.15 m relative to unadapted VGGT under the same intrinsics and stitching protocol.

Critical strengths
  1. The gains over unadapted VGGT are substantial. Across all eleven KITTI sequences, mean ATE improves from 31.78m to 8.87m before loop closure and from 18.09 to 6.15 m afterward. Both versions use the same intrinsics, windows, and stitching, making this a meaningful internal comparison.
  2. Carrying adaptation state is useful. Section 4.5 reports post-loop ATE of 8.25m for independent per-window adaptation, 6.70m for parameter-only carry, and 6.15 m for carrying both parameters and optimizer state.
  3. The lightweight design and existing ablations are useful. Freezing the backbone enables feature caching. The paper also studies adapter rank, masks, intrinsics, and update count, and reports a combination with Free Geometry.
Critical weaknesses
  1. The distinct contribution beyond existing adaptation methods is unclear. VGGT-Align already adapts models with photometric/geometric objectives and accumulates updates across windows; TCO and Free Geometry also use LoRA for test-time adaptation [R1–R3]. The plausible additions are head-only joint depth/camera adaptation and Adam-state carry. The paper should explain why these are substantive advances rather than another configuration of existing frameworks. Improvement over an unadapted model establishes effectiveness, not novelty by itself.
  2. Accurate intrinsics are a strong prior that substantially affects adaptation. Table 9 provides direct evidence on KITTI sequence 09: | Intrinsics | Pre-loop ATE: unadapted → adapted | Post-loop ATE: unadapted → adapted | | --- | --- | --- | | Calibrated | 29.19 → 7.48 m | 17.92 → 6.44 m | | VGGT-predicted | 39.27 → 59.11 m | 21.53 → 21.52 m |

With predicted intrinsics, adaptation worsens pre-loop error and provides almost no post-loop gain on this sequence. The internal comparison remains fair because both versions receive the same intrinsics. However, comparisons against methods without supplied calibration use different inputs. The authors should match the intrinsics or explicitly limit the claims to calibrated videos. This ablation covers only sequence 09, not all sequences.

  1. Better carry performance does not yet establish transfer to new frames. Section 4.5 interprets lower starting photometric residuals as transfer to unseen content; the steps-to-target diagnostic in Figure 7(c) uses sequence 09. However, with 60-frame windows and 30-frame overlap, half of the next window was already used for adaptation. Some gains in starting loss may reflect reuse on previously seen content. A cleaner test would evaluate only newly added frames before adapting the next window. This concerns the forward-transfer interpretation, not the practical carry gains observed across all eleven sequences.
  2. The total cost and the head-only accuracy–efficiency trade-off need stronger evidence. The paper reports 5.75 s for a raw forward pass and 0.42 s per adaptation update, with 70 updates per KITTI window. The component estimate is 5.75 + 70 × 0.42 ≈ 35.15 s/window, about 6.1 times the raw forward cost. This is not measured end-to-end latency, but a cheap update does not imply a cheap full pipeline. Figure 7(a) also shows fewer updates remain useful: carry achieves 7.61 m at 30 updates and 6.15 m at 70. A matched-time comparison should report accuracy, total runtime, and peak memory against methods with comparable outputs to clarify the accuracy–cost trade-off. A small comparison of head-only, partial-backbone, and joint adaptation would also clarify the trade-off. The Free Geometry combination in Table 6 changes initialization and adaptation history, so it does not isolate adaptation location. Updating more layers is not guaranteed to help; the comparison should also credit head-only feature caching.

[R1] Zhang et al. VGGT-Align: Bridging Local Reconstruction and Global Consistency for Long-Sequence 3D Reconstruction. arXiv:2608.15260v1.

[R2] Zhou et al. Learning 3D Reconstruction with Priors in Test Time (TCO). arXiv:2604.03878v1.

[R3] Dai and Yang. Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself. arXiv:2604.14048v1.

Questions for authors
  1. What is the distinct contribution beyond the closest adaptation methods?
  2. Can the authors provide a matched-intrinsics comparison or calibration-sensitivity results on more sequences?
  3. Do the forward-transfer diagnostics evaluate only newly added frames?
  4. Under equal time budgets, how does head-only adaptation compare with partial-backbone or joint adaptation, and how does the full pipeline compare with other methods?
Reviewer_z2ufOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a self-supervised online test-time adaptation method that specializes pretrained predictors to a test video on the fly. The authors combine scene-specific correction with general priors, and jointly optimize lightweight LoRA corrections on depth and pose at inference. For long sequences, overlapping temporal chunks sequentially inherit optimized parameters and optimizer momentum. The method improves performance on multiple datasets.

Critical strengths

This paper combine pre-trained general priors from VGGT with scene-specific optimization.

The authors freeze the VGGT backbone and apply LoRA adaptation only to the depth and pose heads.

Critical weaknesses

Ablations on depth/pose prediction heads and different loss design are required.

The paper does not provide a supplementary video. It's difficult to verify the claimed reconstruction quality and temporal consistency as reported in the paper.

Questions for authors

Compared to frame-by-frame online reconstruction methods (e.g., StreamVGGT), what is the advantage of the chunk-based adaptation AdaVGGT?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.