- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a self supervised online testing adaptation (TTA) framework called AdaVGGT to address the pain point of accumulated errors in long video reconstruction caused by fixed weights and the huge computational cost of traditional Per scene Optimization based on forward geometry large models (such as VGGT). Its core contributions and methods include: low rank adaptation (LoRA) head fine-tuning: freezing the feature extraction backbone network of VGGT, injecting lightweight LoRA adapters only on the deep prediction head and camera pose head, achieving low-cost and efficient online parameter updates. Without the need for offline training, annotated data, or external optical flow, the joint optimization of depth and pose prediction can be achieved by utilizing the multi view photometric reprojection residuals between current video frames and the cross frame depth consistency loss. We use sliding overlapping time blocks (Chunks) for online processing, and seamlessly transfer the LoRA parameters learned in the previous block and the momentum state (moment cache) of the Adam optimizer to the next block, achieving forward migration and fast preheating. Tests were conducted on five indoor and outdoor real and synthetic video benchmarks, including KITTI Odometry, VKITTI2, KITTI Depth, ScanNet++, and 7Scenes. Among them, the trajectory mean absolute error (ATE) was significantly reduced by 66% on KITTI Odometry.
Critical strengths
-
Efficient online adaptive paradigm design: ingeniously combining the strong universal prior of the forward base model with the scene alignment ability optimized during testing. By fine-tuning only the lightweight LoRA header and caching the backbone features, single step updates only account for about 7% of the forward overhead, balancing accuracy and efficiency.
-
Rigorous and solid causal state transmission mechanism: The ablation experiment proves that not only transmitting LoRA weight parameters, but also inheriting the momentum state (Full Carry) of Adam optimizer is crucial for smooth trajectory and convergence speed across blocks, effectively avoiding independent overfitting between blocks.
Critical weaknesses
- The ablation experiment (as shown in Table 9) showed that when the precise calibration parameters were removed and the model was used for prediction, the post TTA ATE on KITTI sequence 09 deteriorated sharply from 6.44 m to 21.52 m. This exposed that the method may face performance collapse in generalized "wild" video scenes lacking high-quality camera parameters.
2.Although overlapping partitioning and Sim (3) rigid alignment are used, the cumulative error between partitions in ultra long open-loop sequences may still lead to long-distance drift, and the method itself acknowledges that it still relies on non streaming sliding segmentation, which has limited support for true real-time infinite length streaming processing.
- Self supervised photometric loss ($l1 $+SSIM) depends on the assumption of Lambertian body reflection and constant lighting, but in real outdoor driving videos such as KITTI, it is often accompanied by dynamic changes in shadows, strong light reflection, water surface or glass reflection. Despite the introduction of composite masks (SAM3, etc.), it may still introduce erroneous gradients under complex lighting conditions.
4.Although the single step update time is relatively short (0.42 seconds), the entire framework has not undergone a comprehensive and in-depth benchmark evaluation of the peak memory and overall time overhead when dealing with sliding windows containing feature caching and multiple iterative optimizations (such as $N_j=70 $), which raises doubts about the deployment friendliness of edge or resource limited devices.
- There are significant differences in learning rate, window size ($w, o N_j \ lambda=0.5 $) between different datasets (such as indoor ScanNet++and outdoor KITTI) (see Table 7), indicating that the method lacks the ability to adaptively adjust hyperparameters in unknown scenes.
Questions for authors
1.Given the decisive impact of internal parameters on method performance (Table 9), what is the extent of performance degradation of AdaVGGT in daily mobile phone videos without camera calibration, requiring joint prediction or rough estimation of internal parameters?
- Do LoRA's update steps $N_j $or learning rate need to be manually re tuned for scenes with different motion speeds or a large number of dynamic objects? Have you considered designing a dynamic stopping strategy based on residual convergence?