acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Batching in Visual Test-Time Learning

Abstract

In visual test-time learning, batching can change what a model learns from an image. An unrelated batch mate selects extra padding, and these tokens enter the target's inner training set even though the images never share features. Same-image interventions isolate this effect from numerical noise and ordinary padding effects. Removing all padding is not the right correction: it changes the checkpoint's singleton computation and lowers average precision (AP) by 0.592 points on a preregistered Small pilot. Canonical-Support Inner Training (CSIT) instead retains the canvas each image would occupy alone and excludes only batch-excess support from the inner loss. It requires no retraining or learned parameters and preserves the steady-state singleton output tensor-exactly. On all 5,000 Common Objects in Context (COCO) validation images, random-batch matched-object drift falls by 68–76% across two released detector scales and batch sizes 4–8. For reference scores near the decision threshold, flip rates fall from 0.30–0.32 to 0.08–0.11. When a batch may be split, exact-shape grouping is more faithful and faster in our measured workload; when a heterogeneous forward must be retained, CSIT repairs its support. AP changes are small and mixed. These findings establish inner training support as part of an explicit execution contract for inference-time learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
test-time training, object detection, batch dependence, canonical support, execution contracts, inference consistency

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_T9wSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how heterogeneous batch padding alters the inner optimization support of visual test-time learners, causing prediction drift even without explicit cross-image interaction. It proposes Canonical-Support Inner Training (CSIT), which excludes batch-induced excess padding from the inner loss while preserving singleton-equivalent support. Experiments on two scales of a ViT3 detector demonstrate reduced prediction drift. Additional analyses examine batching strategies, execution constraints, and other test-time adaptation methods. However, downstream AP improvements are small and inconsistent, while exact-shape grouping achieves better singleton consistency and efficiency when batch splitting is allowed.

Critical strengths

The paper identifies an interesting and overlooked source of batch dependence in test-time learning: padding can alter the effective inner training set rather than merely affect numerical execution. The fixed-canvas and duplicate controls help distinguish this effect from numerical nondeterminism and direct cross-image interaction. The distinction between reference consistency and task accuracy is also valuable.

Critical weaknesses
  1. In essence, CSIT is a support-masking correction technique for heterogeneous batching, rather than a new learning method. The strongest evidence for it is derived from two scale configurations within a single class of ‘fast-weight" detector architectures. It lacks independent replication or validation across other similar architectures. Therefore, framing it within the broader, more ambitious concept of an execution contract exceeds what its actual contributions can support.

  2. Cross-family experiments do not directly reproduce the core mechanism. Some studies focus on different adaptation or implementation effects. But only supporting the transfer to TTTIR fails. Moreover, the change of AP is small and mixed. Precise shape grouping can also achieve better single instance consistency and reduce running time in the reported workload. Therefore, CSIT seems only to solve a specific implementation problem.

Questions for authors
  1. Is CSIT a specific architecture revision for heterogeneous batch processing?

  2. Any evidence in here that can prove the universality of CSIT?

  3. What practical scenarios need heterogeneous batch processing instead of precise shape grouping?

  4. The manuscript covers loosely related topics, including filling support, contract execution, state life cycle, scheduling, buffering and external adaptation methods. It is difficult to understand despite careful reading.

  5. It is interesting and useful to find that batch filling can change the effective optimization support of learners during testing. However, it is still limited to develop it into a universal learning method.

  6. These problems are not easy to solve. The experimental evidence fails to fully support the broad proposition of the paper. Positive results only exist in one detector family. Cross-family verification is inconclusive. Only supporting migration to TTTIR also failed.

1 comment

Q1. Architecture-specific correction or general principle? CSIT as implemented is a correction for one detector family. Tiny and Small are two scales, not independent architectures. What we think applies more broadly is a question one can test: if spatial positions are examples for an inference-time learner, does batching change the examples used to fit the target's weights? The answer need not be yes in every model, and when it is yes the same mask need not fix it.

For a detector block, let CC be the aligned canvas the checkpoint uses for the image alone and EE the positions added by a batch mate. On an unweighted grid, with features and initial weights fixed, the mean gradient before clipping is gU=ρgC+(1−ρ)gEg_U=\rho g_C+(1-\rho)g_E, where ρ=∣C∣/(∣C∣+∣E∣)\rho=|C|/(|C|+|E|). Restricting the loss to CC recovers the singleton gradient if the canonical features are unchanged. If they have changed, or if the downstream detector responds directly to the larger canvas, recovering the loss support need not recover the output. The all-padding mask is also different: it removes positions from CC and changed the checkpoint's singleton AP in our Small pilot. These are conditions on a local argument, not a proof of invariant detection or improved accuracy.

Q2. What does the failed TTTIR transfer establish? The negative result motivated a test with a falsifiable prediction. TTTIR learns per-input convolution weights from spatial and frequency targets. Its frequency generator normalizes bands across the canvas. A loss restricted to the original region can therefore still contain different target values. We varied the loss domain and the domain used to compute band statistics separately. After exploratory work on 15 LOL-v1 pairs, we fixed the joint repair, the primary 64-pixel extension and the endpoints, then evaluated all 100 LOL-v2-Real test pairs with the released checkpoint.

TTTIR intervention                  RGB drift to native    PSNR vs. GT (dB)
Native (no extension)                          0.00000              30.794
Padded, uniform                                0.00650              30.208
Canonical loss only                            0.01219              29.897
Canonical band statistics only                 0.01563              29.245
Both together                                  0.00180              30.745
Crop the entire state transition               0.00133              30.737

Neither single-factor repair helped. Together they reduced drift by 72.4% [61.0%, 80.7%] and recovered 0.536 dB [0.171, 0.998] against padded uniform. These are paired 97.5% intervals adjusted for the two primary endpoints, keeping inputs with identical ground-truth images together (43 groups). SSIM was inconclusive. The full-state crop gave lower drift; neither repair exceeded native PSNR. This tests the mechanism in an independent fast-weight restoration architecture under externally added padding. It does not establish an unchanged CSIT transfer or a second native variable-size detector. The original loss-only failure is part of the result.

Q3. Why not use exact-shape grouping? We would use it when singleton fidelity is required. We have no production trace showing that splitting must be forbidden. We therefore kept splitting legal. On the original full workload, exact grouping was both faster and more faithful. On the 206 held-out heterogeneous four-image groups with at most 10% mean excess area, cold CSIT took 131.8 / 211.8 ms (Tiny / Small) versus 140.9 / 232.6 ms for the faster tested exact policy. High-padding controls favored exact on both time and fidelity. Replaying the fixed rule over all 1,250 groups gave only 1.24% / 1.42% lower recorded service time than always splitting sequentially, before routing overhead. This is a conditional tradeoff on one GPU.

Accuracy and scope. At singleton-selected thresholds, ground-truth F1 was inconclusive for CSIT versus uniform on Tiny and favored uniform on Small. A separate question is whether score retuning can recover the same objects. On the frozen detections, even choosing uniform's best scalar cutoff separately for each class and evaluation fold, with access to the singleton answers, left 721 / 653 status disagreements (Tiny / Small). CSIT at the original thresholds left 222 / 181. We therefore limit the claim to identifying changed training support, demonstrating its consequence for reference behavior, and showing why loss-only transfer can fail. State lifetime, scheduling and general task accuracy are not coequal positive claims.

Reviewer_WF74OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper identifies that in visual test-time learning, padding introduced by unrelated batch mates contaminates the target image's inner update and causes prediction score drift. The authors propose Canonical-Support Inner Training (CSIT), which masks out batch-excess padding while retaining each image's native singleton canvas. Experiments on ViT3ViT^3-based detectors show that CSIT reduces random-batch score drift while preserving tensor-exact single-image outputs.

Critical strengths
  1. The paper is thorough, providing extensive experimental details and comprehensive ablation studies across multiple architectures and execution strategies.

  2. The authors demonstrate commendable intellectual honesty by transparently reporting negative results.

Critical weaknesses
  1. Instead of a systematic academic study, this paper reads more like a course project summary; the core method lacks substantial novelty and is essentially an engineering trick over-packaged into a lengthy manuscript.

  2. The writing reflects inexperience, such as referencing figures from several pages later without prior context in the introduction, presenting crude and low-information Figures 1 and 4, and relying on excessive defensive AI-generated phrasing. The authors are advised to read more literature to develop their academic proficiency before submitting to ICLR.

  3. The CSIT algorithm is validated exclusively within a single fast-weight detector family, lacking generalizability and offering limited contribution to the broader community.

Questions for authors

See the weaknesses above.

1 comment

The introduction asked readers to follow too many diagnostics before it had explained what changes for a single image. Here is the example we use to explain the result and distinguish it from ordinary padding masks.

A COCO zebra image is 800 by 1,067 pixels after resizing. Alone, the detector pads it to 800 by 1,088. Put it next to a taller image and its canvas becomes 960 by 1,088. The zebra pixels have not changed; the images never attend to one another. Yet the zebra score goes from 0.544 to 0.458, so the box disappears at a display threshold of 0.5. The distinction is physical: the 21 padded columns were there when the detector ran alone; the 160 extra rows were not. The model fits temporary weights using its spatial tokens. The taller image has supplied no features to the zebra, but it has changed which tokens train those weights.

Why not just ignore padding? We tried the obvious mask: ignore all padded positions. This reduced batch dependence but also changed single-image behavior, costing 0.592 singleton AP on the Small pilot. CSIT keeps the checkpoint's native aligned canvas and normalizes the inner loss over that canvas. In steady-state singleton execution it follows the original path tensor for tensor. The contribution is the distinction between an invalid image pixel and an extra training example. They do not have the same boundary.

The zebra was selected to illustrate a threshold crossing. To check how widespread such changes are, we compared the identities of detected COCO objects in all 5,000 validation images, using thresholds selected out of fold on singleton detections. On Small, ordinary batching changed the detected/missed status of 929 of 36,335 eligible objects across 686 images. CSIT changed 181 objects across 171 images; exact-shape grouping changed zero. These changes go both ways: some objects lost under one execution are recovered under another. We do not interpret this as 929 additional detection errors or as an AP gain.

What we changed in the prepared manuscript. The opening example gives the dimensions and the two scores before referring to a figure. In Figure 1, the blue outline encloses the image and its native alignment padding; orange denotes only the mate-added area. The caption says exactly which portion enters each loss. Figure 4 now asks whether score calibration explains the effect. Even fitting separate calibrators to ordinary batching and CSIT leaves held-out mean drift of 0.02249 versus 0.00681 on Tiny and 0.01970 versus 0.00500 on Small. The main argument follows the image intervention, the failed all-padding mask, then the comparison with regrouping. It does not require accepting a general framework for state and scheduling.

What carries beyond this detector? The detection evidence is from two scales of one family. In a different fast-weight model, TTTIR for low-light restoration, copying the loss mask actually made things worse. On all 100 LOL-v2-Real test pairs, with a controlled 64-pixel canvas extension, loss restriction increased RGB drift from 0.00650 to 0.01219. This architecture uses frequency statistics to generate its training targets. Restricting those statistics as well as the loss reduced drift to 0.00180. It does not give us a second native detector replication. It does give another researcher a concrete test: when an image's canvas changes, check both which terms enter the inner loss and whether the values of those terms have already changed.

Reviewer_ttMyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates how batch-dependent padding changes the inner updates and predictions of visual test-time learners. It proposes CSIT to exclude padding introduced by batching and renormalize the inner update while preserving single-image behavior. Experiments on ViT³-based detectors show substantial reductions in prediction drift, with small and mixed AP changes, alongside additional studies of alternative execution strategies and applicability.

Critical strengths

The experimental evaluation is extensive, covering multiple datasets, two detector scales, different batching settings, and simple execution baselines. Controlled comparisons help distinguish the effect of padding on the inner update from ordinary padding effects and numerical variation. The authors also report failed transfers and mixed AP changes, making the limitations explicit.

Critical weaknesses
  1. The comprehensive analysis is useful and informative, but the major contiribution of CSIT lies o an engineering correction to how an existing TTT update handles padding, with marginal methodological novelty. As shown in Table 2 and Appendix I, simple exact-shape grouping achieves better consistency and higher throughput in the reported workload. The practical need to retain a single heterogeneous forward therefore requires stronger justification.
  2. The evaluation relies on absolute changes in uncalibrated confidence scores. Since drift magnitude may depend on score scaling, it is unclear whether the reported reductions persist after calibration or improve decision reliability. Although the authors distinguish consistency from accuracy, the mixed AP changes leave the practical benefit beyond consistency insufficiently demonstrated. A calibration baseline and evaluation of confidence reliability would strengthen the evidence.
Questions for authors

See weakness.

1 comment

Your review asks two distinct questions: do the reported changes survive calibration, and are the resulting detections any better? We tested both. Calibration does not remove the disagreement with singleton execution. On the other hand, CSIT does not generally improve ground-truth F1 or confidence reliability.

Which objects change? We evaluated all 5,000 COCO validation images in five held-out folds, keeping each four-image batch intact. On the four training folds we chose the singleton threshold that maximized detection F1; on the fifth fold we used that same threshold for every execution. We then checked the IDs of all 36,335 eligible ground-truth objects under class-aware COCO box matching at IoU 0.5. A switch means an object was detected in one execution and missed in the other. Both directions count.

GT objects whose status differs from singleton                Tiny    Small
Uniform batching, singleton threshold                        1,028      929
Uniform batching, best possible threshold per class/fold       721      653
CSIT, unchanged singleton threshold                            222      181
Exact shape, unchanged singleton threshold                       0        0

The second row deliberately favors threshold adjustment. We let uniform batching see the evaluation-fold singleton answers, ignored the cost of false positives, and exhaustively chose the post-NMS cutoff for each class and fold that minimized switches. Thus 653 vs. 181 on Small is not a comparison against a poorly tuned threshold: even the best scalar cutoffs on these frozen detections did not recover the original decisions. This is a retrospective bound, not a deployable calibration method. A separate check did not require choosing a threshold at all: among within-image same-class object pairs matched in all arms, uniform / CSIT reversed the singleton order for 3.83% / 0.93% of pairs on Tiny and 3.39% / 0.80% on Small. A strictly increasing classwise map cannot undo a reversal.

Do the score differences survive calibration? We fitted positive-slope logistic calibrators on the training folds. The shared map was fitted to singleton scores; separate maps let uniform and CSIT each fit their own scores. Mean held-out matched-object drift to the corresponding singleton reference was:

                            Tiny: uniform / CSIT    Small: uniform / CSIT
Raw scores                       0.01972 / 0.00623       0.01742 / 0.00453
Shared calibrator                0.02168 / 0.00679       0.01911 / 0.00494
Separate calibrators             0.02249 / 0.00681       0.01970 / 0.00500

All images entered the evaluation; 4,952 had eligible objects for mean score drift. With separate maps, 15-bin expected calibration error for uniform / CSIT was 0.00474 / 0.00551 on Tiny and 0.00493 / 0.00468 on Small. Brier score favored uniform on both scales on each arm's own emitted detections. These metrics do not establish better calibration from CSIT.

Does reference agreement improve the decisions against ground truth? On Small, uniform lost 258 singleton detections and gained 671 others: 929 switches, but 413 fewer misses overall. It also made 509 more false positives. Its F1 rose from 72.399% (singleton) to 72.630%; CSIT gave 72.382%. The CSIT-minus-uniform F1 difference was -0.016 percentage points on Tiny (97.5% interval [-0.207, 0.168]) and -0.248 on Small ([-0.420, -0.086]). Retuning each arm for F1 gave no CSIT advantage either. What changed with the batch was the operating behavior validated on singleton images; whether that change helps accuracy is a different question.

Why not regroup? Exact shape was faster and more faithful on the original full workload. On all 206 held-out heterogeneous groups with at most 10% mean excess area, uncached CSIT took 131.8 / 211.8 ms per group (Tiny / Small), against 140.9 / 232.6 ms for the faster tested exact implementation. High-padding controls reversed that ordering. Our methodological claim is the distinction between native alignment padding, which already trains the singleton model, and training positions added by a batch mate. This gives a way to preserve the former when keeping the batch intact is worthwhile; it does not justify replacing exact grouping everywhere.

Reviewer_MQzDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors address how batching changes what a model learns at test time in visual test time training layers. The study was done on Vision TestTime Training (ViT^3) (Han et al., 2026) tiny and small Mask R-CNN detectors. The main observation is that when variable-size images are batches, each one is padded to the largest shape of the batch and because the learner has no validity mask, the padded areas enter the training loss of the target image. The paper shows that the effect of this is beyond numerical noise. The paper proposes Canonical-Support Inner Training (CSIT), which retains the canvas each image would occupy alone and excludes only batch-excess support from the inner loss. This scheme does not require any retraining or learned parameters.

The results on full COCO show that CSIT curs the random batch score drift by 68-76%. Moreover, decision that near the threshold change from 0.30-0.32 to 0.08-0.11. The paper shows comparisons to exact-shape grouping, aspect ratio buffering and fixed canvases in square shapes.

Critical strengths

[S1] The paper shows clearly a real failure mode in visual test-time learning.

[S2] The problem is well motivated in the introduction.

[S3] Showing that removing the padding is not a good solution and it harms the AP performance of the models is useful.

[S4] The paper shows a very simple, yet effective fix in visual test-time training.

[S5] I appreciate the careful presentation of the results, showing subgroup losses, mixed AP changes, images where the performance gets worse and alternatives that can sometimes beat the proposed method.

Critical weaknesses

[W1] The results seem to reveal that the proposed method is not so useful in practice. On full COCO, the exact-shape grouping reproduces singleton outputs to numerical precision. The paper also reports that this is faster than the proposed method for both Tiny and Small models (Table 10 results in Appendix I). Therefore, it seems that CSIT is only useful when a single heterogeneous forward pass must be retained but I did not manage to find in the paper a description of a setting where this restriction binds. makes

[W2] The proposed fix is a practically a padding mask with limited technical novelty. This can be a useful engineering finding, but I feel that alone it is very limited for an ICLR paper. On top of that, the results show that the AP is essentially unchanged.

[W3] I feel that the presentation of the paper can be vastly improved. The main idea is simple but I found several parts in the main text that were hard to follow with lots of caveats and acronyms. The paper fails to focus on the main parts and explain them plainly and it mixes the method with the story of how the research was done. The paper would be much clearer if it focused on the main problem, the fix, the comparison on COCO with simpler alternatives (grouping images with same size and buffering) and discussion the limitations.

[W4] The proposed framework has four components (µ, s, Δ, ω), but it really one studies one of them in a proper way since the other three are only brief and minor experiments.

Questions for authors

[Q1] Can you describe a concrete deployment in which a single heterogeneous forward must be retained, and measure the CSIT’s gains there over exact-shape grouping and singleton execution? In the current experiments, the alternative baselines match the singleton reference exactly, and exact-shape grouping is also the fastest approach.

[Q2] How much of the CSIT's remaining drift is mediated by the inner update?

[Q3] Does the effect of CSIT hold over to another fast-weight architecture outside of the object detection task?

2 comments

Thank you for pointing out that exact-shape grouping can split an arriving batch without waiting for another image. We agree: a no-wait constraint alone does not justify CSIT. We tested the three questions in your review directly.

Q1. When is it worth retaining the heterogeneous forward? We do not have a production trace in which splitting is forbidden. We instead left splitting available and measured when it costs more. An exploratory timing run suggested testing four-image batches whose aligned shapes differ but whose mean excess canvas area is at most 10%. After fixing this rule and excluding every discovery image, we evaluated all 206 remaining qualifying groups (824 COCO images).

Mean milliseconds per group of four       Tiny    Small
Uniform heterogeneous forward             129.2   209.5
CSIT, including support construction      131.8   211.8
Exact shape, sequential                   141.0   233.5
Exact shape, concurrent CUDA streams      140.9   232.6
Four singleton forwards                   171.8   300.6

On these groups, uncached CSIT took 6.4% / 9.0% less time than the faster exact implementation (Tiny / Small). Mean matched-object drift fell from 0.00474 to 0.00223 on Tiny and from 0.00380 to 0.00239 on Small relative to uniform. Exact shape remained the most faithful (drift below 10−610^{-6}); on 16 high-padding control groups it also won on time. These are synchronized FP32 measurements on one RTX 5090, from preprocessing through prediction, with two randomized repetitions. They exclude decoding, networking and queueing.

How much does this help on the whole workload? In a separate replay of the original 1,250 groups, the fixed rule used CSIT for 218 groups and sequential exact shape for the other 1,032. Mean recorded time per group changed from 156.28 to 154.35 ms on Tiny and from 252.74 to 249.15 ms on Small. That saves only 1.24% / 1.42% before live routing overhead, with mean drift of 0.00040 / 0.00041 instead of numerical precision. We would use exact grouping when strict fidelity is required. The full trace does not contain concurrent exact timings; the replay is not a service benchmark.

Q2. How much of the remaining drift depends on the update? We recorded the post-clipping singleton gradients at every block and replayed them in the heterogeneous forward. This fixes the adapted weights to their singleton values while leaving the batched features and downstream operations intact. On 128 predetermined COCO images per scale:

Mean score drift to singleton    Uniform      CSIT      Singleton-update replay
Tiny                             0.01912      0.00541   0.00199
Small                            0.02230      0.00290   0.00177

The paired 95% interval for CSIT minus replay was [0.00162, 0.00545] on Tiny and [-0.00037, 0.00243] on Small. Fixing the updates removed further drift on Tiny; this sample did not resolve that contrast on Small. Both retained some drift. A percentage attributed to the update would be misleading because features and adapted weights interact before a detection score is produced.

Q3. Does another fast-weight model exhibit the effect? In TTTIR, a restoration model, canvas expansion changes the statistics used to generate the inner training targets. After exploration on LOL-v1 we fixed the intervention and tested all 100 LOL-v2-Real test pairs. With a 64-pixel extension, mean RGB drift from native was 0.00650 for uniform execution, 0.01219 for loss restriction alone, and 0.00180 after restricting both the target statistics and the loss. PSNR was 30.208 dB for padded uniform, 30.745 dB for the joint repair and 30.794 dB for native. This is an independent fast-weight architecture under a controlled canvas extension, not a second native variable-size detector.

We have narrowed the presentation around the result your review singled out: the important choice is which positions count as training examples. Masking all padding removes positions used by the released singleton model (Small pilot: -0.592 singleton AP). CSIT preserves those positions. The mask is simple, detection AP is mixed, and the evidence for support membership is stronger than for the other proposed axes.

I appreciate the response and the new experiments. The response partly answered my questions but didn't fully covered my main concerns.

The singleton update replay study answers and addresses my Q2. Q3 is also addressed, maybe only partially since it seems that a model-specific fix was need and it was tested only with artificially added padding. So, it seems that the idea could transfer to another task but the method itself does not. Regarding Q1/W1, my concern still remains since it seems that there is no setting where a heterogeneous forward must be kept. CSIT is clearly faster on groups with low padding but on the full workload it saves only 1.2%-1.4% of the time.

W3 and W4 are not addressed. The authors describe changes but they did not update the submitted PDF during the discussion paper, so we can't check any updates or improvements in the presentation.

For these reasons, I tend to maintain my original score of 2. In case of a rejection, I suggest that the authors update the paper to address all the concerns before submitting it for another review cycle.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.