acceptodds
Under review as a conference paper at ICLR 2027

AUDITING TEST-TIME TRAINING IN STREAMING 3D RECONSTRUCTION: SCALE-INVARIANT UPDATES, LARGE WEIGHT CHANGES, SMALL POSE EFFECTS

Abstract

Streaming 3D reconstruction transformers increasingly incorporate test-time training (TTT) branches whose fast weights are updated online and are intended to retain long-horizon scene information. We audit this mechanism in the released Scal3R model under its published evaluation configuration on TUM RGB-D and 7-Scenes, explicitly tracing the implemented update rule and loss and evaluating interventions as paired differences relative to variation induced by window placement. We find that the deployed TTT update, inherited from large-chunk TTT and based on Muon-style orthogonalisation, is effectively scale-invariant: changing the base learning rate by 8× alters absolute trajectory error (ATE) by at most 0.023%, while the approximately 4× variation in per-token learning rates primarily reweights update directions. Although block gradients are highly redundant, orthogonalisation substantially magnifies their differences. Replacing the full gradient with that of a single block yields gradient cosine similarities of 0.77–0.99 but orthogonalised-update similarities of only 0.54–0.81, producing final fast weights that differ by 5–16%. Despite these large parameter-space differences, ATE remains within 0.14% of the full update across all tested 300-frame windows. More broadly, the fast weights undergo substantial adaptation, corresponding to an estimated 20°–34° parameter rotation, while their effect in trajectory space is small: removing the TTT branch or freezing its fast weights changes ATE by approximately 1% or less, and blockwise adaptation without retraining degrades ATE on two of three sequences, by as much as 26%. In contrast, a larger share of test-time variability originates elsewhere in the inference pipeline. Averaging predictions across two window offsets improves ATE on every sequence of both benchmarks, while ground-truth-free selection of the alignment back-end through self-consistency provides additional, though less universal, gains, reducing mean ATE by 9.9% on TUM and 8.3% on held-out 7-Scenes. These results reveal a pronounced mismatch between adaptation in parameter space and adaptation in reconstruction accuracy, and show that inference-window and alignment choices can dominate the observable contribution of the deployed TTT memory.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time training, streaming 3D reconstruction, fast weights, online adaptation, scale-invariant optimization, Newton-Schulz orthogonalization, block-gradient redundancy, test-time adaptation, camera trajectory estimation, mechanistic analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s7fwOpenReview ↗
Rating
1(−1)
Confidence
4(+2)
Informativeness
2
Presentation
1(−1)
Soundness
1
Read review
Paper summary

The paper audits the test-time training mechanism in the released Scal3R model. It analyzes the scale invariance of the implemented update and reports that substantial differences in fast weights have little effect on ATE under several interventions on TUM RGB-D and 7-Scenes. It also evaluates window averaging and self-consistency-based selection of the alignment back-end as ways to improve trajectory accuracy.

Critical strengths
  1. The audit distinguishes removing the TTT branch from freezing its fast weights, helping separate the branch’s contribution from the benefit of test-time adaptation. Paired comparisons on the same windows and the compute-matched zero control strengthen this analysis.

  2. The gradient probes distinguish gradient alignment from update similarity. First-block and full updates produce fast weights differing by 5–16%, despite small differences in ATE (Figure 2 and Section 6.2), providing an interesting observation about the deployed model.

Critical weaknesses
  1. The main contribution appears limited to an empirical observation that the tested Scal3R checkpoint is relatively insensitive to several TTT interventions in terms of ATE. The scale-invariance property is inherited from LaCT (Zhang et al., 2025) and Muon (Jordan et al., 2024), as the authors acknowledge. The study does not yet explain why the observed weight changes produce little improvement, or establish when this behavior should be expected. The window-averaging and back-end-selection results offer practical improvements, but provide limited additional insight into test-time adaptation.

  2. The interpretation of the ATE results seems stronger than the measurements support. Similar ATE values do not imply similar predicted trajectories, particularly after separate Sim(3) alignment. Tables 1 and 2 measure changes in reconstruction error rather than changes in the predictions themselves. Moreover, the depth evaluation in Appendix G only considers frame duplication, leaving the geometric effects of branch removal, frozen weights, and block subsampling unresolved.

  3. The reported 20°–34° weight rotation is estimated using random gradient directions rather than measured during actual scene adaptation (Section 6.3). The rotation after norm restoration depends on the update’s direction relative to the pretrained weights, so its magnitude cannot be established from normalization alone. The measured 5–16% weight differences compare first-block and full updates, which is a different comparison from the change induced by adaptation relative to the pretrained weights.

  4. The evaluation provides limited evidence about the long-range memory behavior motivating the study. Most experiments use 300-frame windows from nine indoor sequences, while the 1,200-frame evaluation covers only one sequence (Table 19). These results establish a useful observation in the tested settings, but leave unclear whether the same behavior persists in large-scale scenes where cross-block context may play a more substantial role.

Questions for authors

See weaknesses.

Reviewer_Et1tOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper audits the test-time training (TTT) mechanism in the released Scal3R model. Starting from the implementation, the authors analyze the scale invariance of its Newton–Schulz update and study the practical effect of TTT on camera trajectories through interventions including removing the TTT branch, freezing the fast weights, subsampling block gradients, and performing online updates. The paper argues that, although the fast weights may change substantially in parameter space, their effect on ATE is typically around 1% or less. In contrast, window placement and back-end alignment choices have a larger impact on the results.

Critical strengths
  1. The paper addresses a practically important question. Distinguishing the contribution of the TTT branch itself from that of test-time adaptation helps avoid conflating the two effects in conventional branch-removal ablations.
  2. The mechanistic analysis is detailed. The authors connect the update equations to the actual implementation and analyze gradient scale, token-wise rates, block-gradient geometry, and the Newton–Schulz update.
  3. The experimental process is transparent. The Branch, Zero, and Frozen controls are reasonably designed, while the Correction Log and Data Provenance sections improve the traceability of the reported results.
Critical weaknesses
  1. The experiments do not adequately cover the long-sequence settings central to Scal3R. Most experiments use 300-frame windows, corresponding to only about nine blocks under the evaluated configuration. The only 1,200-frame experiment contains a single sequence at a single offset. The main conclusions are not evaluated on Scal3R’s core long-sequence benchmarks, such as KITTI, Oxford Spires, and ScanNet++.
  2. The evaluation is largely limited to camera trajectories. ATE is appropriate for the paper’s focus on pose effects, but it does not establish how TTT affects depth or point-cloud reconstruction quality. The conclusions should therefore not be directly extended to the complete 3D reconstruction system.
  3. The dataset coverage and evaluation protocol are insufficient to establish robust conclusions. The paper evaluates only five TUM sequences and four 7-Scenes scenes, with only one offset for each 7-Scenes scene. Most results therefore lack sufficient independent sequences and repeated evaluations. In addition, the paper uses a chunk size/overlap configuration of 60/30 on TUM, whereas the original Scal3R paper reports 120/60, reducing direct comparability with the published evaluation setting.
Questions for authors
  1. Do the conclusions regarding branch removal, frozen fast weights, and block-gradient subsampling still hold on complete long sequences and a larger number of independent sequences? Could the authors repeat the main experiments on KITTI, Oxford Spires, and ScanNet++?
  2. Are the conclusions intended to be limited to camera-trajectory ATE? Could TTT affect depth or point-cloud reconstruction quality even when its effect on ATE is small?
  3. Can the main conclusions be reproduced using the 120/60 configuration officially reported for Scal3R on TUM-RGBD?
Reviewer_KpzVOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether the TTT branch in Scal3R improves camera accuracy during inference. The authors intervene at different stages along the chain "LR -> block gradients -> orthogonalized updates -> fast weights -> camera poses." They find that normalization largely cancels the effect of scaling the base LR. Although block gradients are highly similar, different fast weights with little change in trajectory error. Removing the TTT branch or freezing its fast weights also has only a small effect in the tested settings. Larger improvements come from averaging predictions across shifted inference windows and selecting the alignment backend based on trajecotry self-consistency.

Critical strengths
  1. The experiments seperate the branch's contribution from the effect of updating it.

  2. The analysis follows the actual implementation and checks each stage of the update.

  3. The paper identifies practical ways to improve inference results.

Critical weaknesses

The reason for the small trajectory effect remains unclear. Large changes in fast weights have little influence on trajectories, but is not fully explained. Measurements of intermediate activations and geometric predictions before backend processing would help identify the cause.

Questions for authors

Please address the concern mentioned above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.