acceptodds
Withdrawn from ICLR 2027

How Far Can Vision Test Time Training Models Adapt at Test Time?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time training (TTT) lets a model adapt its computation to every input through an inner parameter update, and how far that update can move before adaptation stops working has not been examined. We find that vision TTT adapts over a narrow range. Modest changes in update magnitude can erase the benefit of adaptation, and the preferred magnitude shifts as the input geometry changes. We show that this range is learned during training, and we test this with a minimal change to the training recipe, sampling the magnitude of the inner update on each forward pass and sharing it across all TTT blocks, so that the model learns from a continuum of adapted states while each update keeps its sample-specific direction. Range-trained models remain effective across a much broader span of update magnitudes at a small cost at the standard setting, their learned computation still depends strongly on sample-specific update directions, and the learned range persists under distribution shift, changes in input resolution and downstream fine-tuning. Vision TTT can thus learn how far to adapt at inference, which makes adaptation range a trainable property of the model.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
test-time training, adaptation range, inner update magnitude

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fZNXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the sensitivity of vision test-time training (TTT) models to the magnitude of their inner updates. It finds that existing models perform well only within a narrow range around the training magnitude, and attributes this fragility to updates compensating for harmful outputs at the initial weights. The authors propose η-ViT³, which samples a shared update magnitude from U(0, 2) during each training forward pass. Experiments on classification, segmentation, and image generation show that this simple change broadens the usable adaptation range with little impact on standard-setting performance and no additional inference cost. The increased tolerance persists under distribution shifts, resolution changes, and downstream fine-tuning, while retaining dependence on input-specific update directions.

Critical strengths
  1. An insightful diagnosis of adaptation fragility. The paper shows that vision TTT models can be highly sensitive to update magnitude, with skipping adaptation sometimes causing more damage than removing the TTT branch entirely. Its output decomposition and intervention experiments provide evidence that adaptation compensates for harmful initial-weight outputs. This analysis goes beyond reporting sensitivity and offers a useful explanation of why adaptation fails, making it a substantive contribution to understanding TTT models.

  2. A simple intervention with convincing empirical support. Sampling the update magnitude during training substantially broadens the usable adaptation range without adding parameters or inference computation. For the tiny classifier, accuracy drops by at most 0.5 points over magnitudes from 1/8 to 2, compared with up to 16.8 points for the matched baseline. The persistence of this benefit across model sizes, resolution changes, distribution shifts, and downstream fine-tuning strengthens its practical significance, while direction-replacement experiments support that the model still uses input-specific adaptation.

Critical weaknesses

The practical value of a wider adaptation range is unclear. What practical benefit does changing the update magnitude provide? In Table 3, all three models perform worse than their standard-training baselines at the default setting. What useful capability does varying the update magnitude provide that justifies this loss in performance?

Questions for authors

Could the authors provide a concrete use case where varying the update magnitude offers a measurable benefit over the standard fixed-magnitude baseline, sufficient to justify the lower default-setting performance reported in Table 3?

Reviewer_oTmtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper seems to be analyzing how to choose the the learning rate for the shared feature extractor weights in TTT. The contribution of the paper is unclear but I believe the authors' main contribution is to randomly sample a learning rate parameter instead of using a fixed one. For experiments, authors use model sizes up to ViT-B and they use one of image classification, image segmentation, image generation and text generation tasks only. This is inadequate to come to any conclusions for the usefulness of the method.

This paper contains a lot of LLM-generated text full of pseudo-terminology starting with the first sentence of the paper --- in which a parameter becomes an "inner parameter", training becomes "adapting its computation", magnitudes become "how far an update moves". This is not a quirk or simple style choice, it makes understanding the paper much more difficult for a human. There are also factual errors as would be expected from an LLM generated submission, e.g. immediately in the first paragraph of the introduction, the authors/LLM make a factually wrong statement: "First developed for language modeling, TTT..." but TTT was introduced in 2020 using ResNets on CIFAR10.

Critical strengths

None.

Critical weaknesses

The paper is unreadable.

Contributions are unclear or very small. Varying learning rates during training, if this is indeed the proposed method, is a small contribution.

Experiments are insufficient with only smaller architectures from a single model family and on very few datasets.

Questions for authors

None

Reviewer_49hxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I thank the authors for their hard work and submitting the paper to ICLR.

The paper argues for a procedure to run TTT layers where learning rate could adapts dynamically given a particular input image. The core use of such a procedure is that one could (in theory) run ttt layers with larger learning rates, and hence converge faster with minimal or no loss of performance.

Critical strengths

Several interesting points are noted :

- LLMS can run on wider learning rate range than VLMs

- The resolution of the input seems to have an effect on how sensitive  a ttt layer is to a given choice of learning rate,
Critical weaknesses

There are several things which are not clear.

  • Fig 2: left and right are identical except : the usage of n.

    • training is uniform sample between ranges of n~[0,2]. that might be contributing to robustness.

    • The network itself has no understanding of what optimal n is for each test sample. I believe that a constraint which predicts what learning rate is needed based on contents of input sample needs to be accomodated.

  • Equation 3, learning rate is a scalar. It might be beneficial to experiment with different learning rates for each entries in G_{hat}.

Rest i will cover in questions.

Questions for authors

Questions

  • Line 53: What does skipping the update mean?

  • Line 80: conceptually, why do llms tolerate a wider span of magnitudes?

  • There are already methods like TPT which update on a input sample in just one gradient iteration. so why one might use this method?

  • what is the reason for setting learning rate as 1 during inference.

  • I was just curious, do the authors think it is possible to eliminate equation 3 altogether, and do some other kind of update rule? It is just a question, would love to know their thoughts :-)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.