acceptodds
Withdrawn from ICLR 2027

Position Aware Layer Queries for Test Time Training In Vision Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-Time Training (TTT) adapts models to incoming test samples (e.g. out-of-distribution, OOD) when conventional fine-tuning is infeasible. Existing TTT methods for Vision-Language Models (VLMs) create supervision from several augmented views, each requiring forward (and often backward) passes through the entire VLM, incurring substantial computational cost. We observe that one forward pass with all the intermediate layer outputs already yields far more signal than the final embedding from all augmentations. We introduce **Layer Query Network (LQN)**, a lightweight approach that can adapt a frozen VLM (teacher) in a single forward pass of the VLM via a small model (student). LQN uses **Position-Aware Distillation (PAD)** to mimic the teacher VLM’s intermediate-layer spatial tokens by querying spatial coordinates of intermediate tokens. LQN additionally relies on **Location Consistency Regularization (LCR)**, a self-supervision technique, replacing expensive O(HxW) image augmentation with O(1) coordinate sampling. Integrating these, LQN i) Adapts and improves zero-shot CLIP ViT-B/16 by 9.8% Top-1 on OOD ImageNet, ii) Outperforms the previous best GS-Bias on fine-grained classification by 3.9% Top-1, iii) Faster convergence than TPS for CLIP ResNet-50 (47 mins vs 55 mins) iv) Generalizes adaptation to VLMs like SigLIP, EVA-CLIP, and CoCa, and lightweight students like MLP, ResNet, VGG, and v) Extends to panoptic, instance, and semantic segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Test Time Training, Position Aware Queries

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vp2iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies single-sample test-time training (TTT) for vision-language models (VLMs) under distribution shift.

The authors note that existing TTT methods such as TPT build supervision from many augmented views of each test image, and each view requires a pass through the VLM. They argue instead that a single forward pass already provides sufficient signal through the teacher's intermediate-layer tokens.

They propose Layer Query Network (LQN), a small student trained from scratch on each test image with two objectives:

  • Position-Aware Distillation (PAD): the student regresses the frozen teacher's token at sampled (i,j,l)(i,j,l) coordinates.
  • Location Consistency Regularization (LCR): the student keeps its predictions consistent across randomly sampled src coordinates. This is presented as an O(1)O(1) substitute for image augmentation.
Critical strengths

1. Architecture-agnostic distillation. Querying a single shared student by (i,j,l)(i,j,l) removes the need for layer-to-layer correspondence between teacher and student. As a result, the approach applies to a range of students and teachers. This is the most clearly motivated design choice in the paper.

2. A practical problem with a clean protocol. The paper targets a real cost of single-sample TTT. The protocol is clean:

  • the VLM runs once per sample;
  • the student is reset after each sample;
  • no other test samples are accessed (Sec. 3.2, Alg. 1).

This keeps the comparison with prior TTT methods fair while substantially reducing compute.

Critical weaknesses

1. The paper does not explain why intermediate-layer features are a good adaptation signal. The premise is that intermediate tokens from one pass carry "far more signal" than augmented final embeddings (Abstract). Two supports are offered:

  • the number of tokens a single pass produces (Sec. 1);
  • prior work showing that intermediate-layer distillation helps compress large teachers into small students (Sec. 3.2.2).

Neither explains why these features help adapt to a single OOD sample. A token count is not a measure of information. The distillation literature concerns compression, not test-time adaptation.

Suppose intermediate distillation does help the student imitate the teacher more faithfully. The paper should still explain what makes it uniquely effective at test time, that is, why imitating the teacher's intermediate layers should correct the teacher's own errors on a shifted input.

2. The paper gives no rationale for coordinate consistency (LCR), nor for this particular design. LCR is the component that replaces augmentation. By the paper's own ablation, it is also the component behind most of the gain. Yet the paper never explains why it should work.

  • The stated core reasons are speculative. The paper's justifications are phrased as intuitions or beliefs: "Intuitively, LCR encourages…"; "We believe this is likely because…"; "potentially acting as a denoiser". No explanation, example, or conceptual illustration helps the reader see why the method works.
  • The core idea is not explained. src is never given a conceptual meaning, and it contains no image information. It is therefore unclear what invariance the student acquires by being consistent across src.
  • Alternatives are not discussed. The paper does not argue why this formulation is appropriate, let alone preferable, compared with natural alternatives. Considering the method's contribution, this missing rationale for it makes the method hard to understand, trust or build upon.
Questions for authors

Please refer to the weaknesses

Reviewer_3muoOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes LQN, a single-sample test-time training method for frozen VLMs. The teacher runs once on the test image, and its patch tokens from all layers are cached. A small MLP student, re-initialized for each sample, then learns to reproduce these tokens. Its input is the image compressed by a single-channel strided convolution, concatenated with RoPE-encoded source and destination coordinates (i,j,l)(i,j,l). It is trained for about 15 steps with two losses: 1) PAD regresses the teacher token at a sampled destination, with the source fixed at (0,0,0)(0,0,0). 2) LCR makes predictions for the same destination consistent across two random sources. A −log⁡∥wsrc∥-\log\lVert w_{src}\rVert term prevents the student from ignoring the source. At inference, the student’s predicted last-layer tokens are averaged and scored against the teacher’s text embeddings. Experiments cover 14 classification benchmarks, 5 segmentation settings with EoMT, and several teachers and students. The reported gains are satisfactory.

Critical strengths

S1. Efficiency. TPT-style methods need dozens of teacher forward passes per sample. LQN runs the teacher once and moves all optimization into a small proxy. The FLOP comparison (160.5 vs. 1312 GFLOPs) and the wall-clock study make this concrete. If the accuracy results hold, this is a practical recipe for single-sample TTT.

S2. Layer-indexed querying. Conditioning the student on (i,j,l)(i,j,l) removes the need for block-to-block correspondence in intermediate-feature distillation. Any student architecture can therefore learn from all teacher layers through one shared network. This idea could be useful beyond TTT.

Critical weaknesses

W1. LCR does not do what it claims. LCR is meant to make the student’s output invariant to the source coordinate. Yet replacing the fixed source with random sources drops Food-101 accuracy from 89.2 to 59.1 (Appendix D.1). A source-invariant student would barely be affected by this change. The appendix’s own explanation, that destinations are decoded “relative to an anchor”, contradicts the invariance LCR is said to enforce. It is therefore unclear why LCR adds 17.4 points over PAD alone (Table 5).

The anti-collapse term is also ill-posed. If −log⁡∥wsrc∥-\log\lVert w_{src}\rVert is clamped at 0.7, its gradient vanishes below ∥wsrc∥≈0.5\lVert w_{src}\rVert\approx 0.5, which is exactly the collapse regime it should prevent. Above that point, the term is unbounded below.

W2. Large in-distribution gains need much stronger evidence. EoMT was trained on COCO, ADE20K and Cityscapes, so the segmentation evaluations are in-distribution. Yet LQN reports +3.3 to +6.4 over EoMT, and 64.5 PQ on COCO panoptic would, to my knowledge, exceed any published result. The supplement reports similar gains for supervised Vision Mamba classifiers on ImageNet val (78.3 → 80.9). In Fig. 4 (right), features predicted for layers 21 to 24, which were never seen in training, give higher PQ (60.3) than the teacher’s real layer-24 features (58.3). It is also unclear where EoMT’s query tokens come from, since LQN predicts only patch tokens.

W3. Missing controls and unfair comparisons.

  • Readout baseline. LQN’s readout averages last-layer patch tokens, while CLIP uses the projected CLS token. The teacher’s own patch tokens with the same readout is therefore a necessary baseline.
  • Prompt ensemble. LQN uses an 80-prompt ensemble on the ImageNet benchmarks (Appendix C.1), but the headline +9.8 is measured against single-prompt CLIP. Against the Ensemble row in Table 1, the gain is 7.5. It is unclear which prompts the baselines use.
  • Tuning on test data. N=15N=15 is the value that maximizes test accuracy in Table 15, and α\alpha is tuned separately on each test set (Table 14). The baselines do not receive this tuning.
  • Pretrained variants. LQN (Pretrained) distills DINOv2 on ImageNet, and the CNN students are ImageNet-pretrained. This contradicts the paper’s claims of random initialization and no dataset-specific pre-training, so these rows are not comparable to zero-shot baselines.
  • CNN students. Only an input projector is trained, yet a frozen ResNet-34 reaches 87.2 on Food-101, above its CLIP teacher (83.7).

W4. Reported compute and implementation numbers are inconsistent.

  • Parameter count. The main text states a 25M-parameter student, but the layers in Table 12 add up to at least 35M parameters.
  • Triplet sampling. Appendix B says 5% of all (src1,src2,dest)(src_1, src_2, dest) triplets are sampled per image. Even for CLIP ViT-B/16 at 224 px (n=2352n=2352 tokens), this is about 10810^8 triplets. That is incompatible with 15 iterations at 10 to 24.5 GFLOPs each and with 47 minutes for 50,000 images.
  • Table 16 totals. The totals do not add up. 4146+15×24.5=45144146 + 15\times24.5 = 4514 GFLOPs, not 4384, and 0.5+15×0.68=10.70.5 + 15\times0.68 = 10.7 s on Orin, not 8.9 s.
  • Speed relative to the teacher. Table 16 also shows LQN is about 36× slower than its teacher per image (2.88 s vs. 0.08 s). The efficiency claims should state this.
  • Iteration count. NN is 14 in the main text and 15 in the supplement.

W5. Poor notation and writing quality. The notation is inconsistent and often undefined, which makes the method hard to follow.

  • YimgY_{img} is the final embedding in Eq. (1) but the stack of all intermediate tokens afterwards.
  • In Eq. (4), wsrcw_{src} and ⟨⋅⟩\langle\cdot\rangle are undefined, and ϵ\epsilon sits outside the logarithm, where it has no effect. Also, there is inconsistent use of bold font in S_{img}.
  • src and dest are used both as coordinates and as encoded vectors. The text samples a batch of coordinates, while Eqs. (3) and (4) and Algorithm 1 use a single one.
  • Fig. 3 -> "ImageNet"
  • The writing also needs substantial editing. "TTA" means both test-time adaptation and test-time augmentation, and Table 11 is titled “Test Time Augmentation” although it reports methods that use test history. Please fix all the typos and grammar errors.

W6. Limited novelty over APM. APM (Modi & Rawat, NeurIPS 2024) already distills CLIP into a location-conditioned student at test time, from one forward pass and without augmentation. The supplement adds more comparisons with APM. However, no ablation shows which change (layer indices, RoPE or LCR) explains the gains.

Questions for authors

The following clarifications could change my assessment.

Q1. Architecture. How is RoPE used as an input vector to an MLP? What do lsl_s and wsrcw_{src} refer to, and which quantity is clamped at 0.7? What is the actual parameter count of the student?

Q2. Readout and resolution. How are the 768-d student outputs mapped to the 512-d text space of CLIP ViT-B/16? Is the teacher run at 448 px (Table 13) while the zero-shot baselines use 224 px?

Q3. Controls. What do Tables 1 and 2 show for (i) the frozen teacher with LQN’s readout and (ii) PAD on the last layer only? Do all baselines use the same 80-prompt ensemble as LQN?

Q4. LCR and tuning. Why does LCR add 17.4 points if the trained student is not source-invariant (Appendix D.1)? What are the results with a single α\alpha and NN chosen on held-out data rather than on the test sets? Please also report standard deviations in the main tables.

Q5. Segmentation. Where do EoMT’s query tokens come from at inference? Are the results on the standard validation splits with the public checkpoints?

Q6. Pretraining and CNN students. How much of the gain remains without DINOv2 or ImageNet pretraining, and what does LQN (Pretrained) achieve at N=0N=0? For the CNN students, which parameters are trained, and how is the output mapped to teacher tokens? Is the 90.8 MLP result in Table 6 the pretrained variant?

Q7. Compute. How many coordinate triplets are used per iteration, and how does this relate to the 5% figure in Appendix B? Please give the full per-sample FLOPs, including inference queries, and correct the totals in Table 16.

Reviewer_G6xgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Layer Query Network (LQN), a test-time training framework for vision-language models. LQN uses a lightweight student network to predict the frozen teacher's multi-layer spatial features through position-conditioned queries. It combines Position-Aware Distillation (PAD) and Location Consistency Regularization (LCR) to adapt the student to each test sample. Experiments on classification and segmentation benchmarks demonstrate improvements over existing test-time adaptation methods.

Critical strengths
  • LQN exploits multi-layer spatial features rather than relying solely on global image representations, potentially improving adaptation under distribution shifts.

  • By using coordinate-conditioned queries and consistency regularization, LQN avoids the need for multiple image augmentations during test-time training.

  • Experiments cover diverse classification and segmentation benchmarks, multiple CLIP backbones, and a pretrained student variant.

Critical weaknesses
  • Unclear motivation for LCR. LCR enforces consistency across different source coordinates, although the source is fixed during inference and changing it does not alter the input image. Moreover, preventing the student from ignoring the source coordinate seems inconsistent with requiring output invariance to it.

  • Potential memory overhead. Despite reducing augmentation costs, LQN still requires performing repeated backpropagation for each test sample. Peak GPU memory consumption should be reported.

Questions for authors
  • Can the authors provide a direct efficiency comparison between LQN and GS-Bias in terms of inference latency, computational cost, and peak GPU memory consumption?

  • How sensitive is LQN's performance to the depth of the student network? Would a shallower student achieve a better efficiency–accuracy trade-off?

  • Could the authors investigate whether a staged optimization strategy, applying PAD followed by LCR or vice versa, could improve performance compared with jointly optimizing both objectives throughout test-time training?

  • How does LQN perform on corruption benchmarks such as ImageNet-C? Under severe corruptions, teacher spatial features may become less reliable or harder to predict, potentially limiting the effectiveness of feature distillation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.