acceptodds
Under review as a conference paper at ICLR 2027

When Does Adaptive Test-Time Augmentation Pay Off? A Matched Execution Study in Dense Vision

Abstract

Adaptive test-time augmentation (TTA) can allocate computation usefully and run faster without improving quality–runtime utility. We develop a comparator-aware procedure that measures complete systems and translates utility deficits into improvement requirements. A frozen SegFormer-B2 router on 400 held-out ADE20K images exhibits two effects across RTX 3090, T4 and A100 environments. At 0.8 cross-entropy (CE)/second, A100 batching increases percentage savings over multiscale from 24.7% to 28.3%, but reduces absolute savings from 19.24 to 10.12 ms/image, below the 17.48 ms required by its quality penalty. On T4, routing beats multiscale in utility but loses to native inference, the best static at that preference. The recorded batching reversal on RTX 3090 and A100 holds throughout CE/second, not only at the historical price. Removing observation/head computation alone leaves substantial deficits; a 2,000-image cross-fitted evaluation retains negative batch-8 mean surplus. Prospectively frozen ACDC calibration changes 30.0% of decisions, but runtime absorbs 85.2% of the CE gain, leaving superiority unresolved. Matched execution identifies the relevant comparators and the improvement needed to outperform them.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
adaptive inference, test-time augmentation, inference efficiency, hardware-aware evaluation, batched execution, quality–runtime trade-offs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q93mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an evaluation protocol for assessing the cost-effectiveness (accuracy versus runtime) of adaptive test-time augmentation (TTA). It introduces a surplus metric, defined as the difference in utility (negative cross-entropy minus priced complete runtime) between an adaptive router and the best-performing static policy, to determine whether the router outperforms the most cost-effective static alternative. Experiments under varied conditions, including execution environment (RTX 3090, T4, A100) and batch size, show that the router does not necessarily outperform the best static policies.

Critical strengths
  • S1. This paper does not propose a new TTA method; instead, it focuses on evaluating the cost-effectiveness of TTA methods. In practical deployment, it is important to consider computational cost as well as accuracy.
Critical weaknesses
  • W1. Clarity. The paper is difficult to read. It provides little explanation of existing adaptive TTA methods. It also introduces many paper-specific terms, and their definitions often begin before the motivation for introducing them is sufficiently explained.

  • W2. Limited novelty. The main novelty of the paper is the proposed evaluation protocol; it introduces neither a new technique nor particularly novel findings. The scope of the empirical claims is also limited. The router is designed by the authors and differs from existing adaptive TTA methods. Only three actions are evaluated, which is considerably fewer than in existing adaptive TTA work (e.g., Kim et al., 2020 use 12 augmentation types). The main experiments are restricted to a single architecture (SegFormer-B2), a single task (semantic segmentation), and two datasets (ADE20K and ACDC). Therefore, while the reported findings may hold within this limited setting, it is unclear whether they generalize. Furthermore, the paper does not apply the proposed protocol to any existing adaptive TTA method, so it remains unclear what the protocol would reveal about existing methods. Consequently, there is limited evidence that the protocol itself is important.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer q93m

Thank you for your direct and helpful review. We totally agree with both main points. The paper was hard to read, and a protocol must show what it finds exactly on existing methods.

W1. Clarity.

The main text is rewritten in plain and easy words. It opens with the result (Figure 1) and collects its terms in Table 1. Then it explains the existing methods (Section 2), works through one example before the corollary (Section 3), and ends with a four-step test (Section 6).

W2. Novelty and scope.

  • What the protocol reveals about an existing method (Section 5.7). We applied it to five-crop AdapTTA with its threshold of 0.8, on its own task (ImageNetV2, two classifiers), on a desktop CPU in place of its embedded target. (a) Confidence does pick the right images: on EfficientNet-B0 every early stop kept the five-crop label, against 97.5% of images when the same stopping depths go to random images. (b) Its saving in crops carries over only in part: it computes 27% and 36% fewer crops but saves 1% (8% in a first test) and 27% of the time of five crops in one call, near the forecasts saved before the test. (c) Counting forward calls made it 13 to 15% faster. (d) Against static crop sets chosen on validation data it has no clear advantage: on EfficientNet-B0 two crops match its top-1 (−0.1 points, interval −0.9 to +0.7) in half the time; on ResNet-50 it wins over a range of prices, though not firmly.

  • More than three actions. We timed a twelve-view menu with three selectors, one a loss predictor trained by the recipe of Kim et al. (Appendix D.10). In CE, each of the 61 adaptive choices has a static competitor that is faster and better. In mIoU the trained predictor wins sixteen comparisons; a fixed three-view order, timed later, was level with it and faster. This tests their recipe on our views only, not their own results.

  • One architecture, our own router. Timing now covers SegFormer, Mask2Former with its own router (Appendix D.8) and two ImageNet classifiers. The router is ours, which limits what its results say about published methods; the arithmetic of Corollary 1 carries over to other rules.

What we learned is set out in the general response. In short, static timings, with one untimed pass of a rule, tell where it cannot win (formally, a static policy won in all 20 settings outside the band). A saving in views is a saving in time only if a forward call goes. The searched static subsets are the bar, although a label oracle shows 1.3 to 1.4 mIoU points of room.

We narrowed the claims to what was run (Section 6). Thank you again for pushing us to make the paper readable.

Thank you for the rebuttal.

W1. The paper appears to have been substantially revised. The Author Guidelines state that reviewers are not required to look at every revision. I therefore checked only the introduction. It is more readable than the previous version, but the paper remains difficult to read. For example, in the “The measurements” paragraph, the price is described only as “a fixed exchange rate between cross-entropy (CE) and time”; its meaning (how much CE one second per image is worth) becomes clear only in Table 1, which is referenced at the end of the paragraph. The paragraph also does not explain why CE is used as the quality metric. Furthermore, 0.8 CE/s is described as “the value that favours our router most,” without stating by which metric and on which data, and it is unclear what the statement that 0.069 CE is worth 86 ms is meant to show.

W2. The application to an existing method (AdapTTA) addresses my request. However, the newly presented results do not contain novel findings. They only show that the paper’s claim, that an adaptive TTA method does not necessarily outperform the best static TTA policy in terms of accuracy and runtime, holds in extended experimental settings, which is intuitively expected.

Since my concerns are not resolved, I will keep my score.

Thank you for reading the revised introduction and for the concrete examples.

W1. All four points come from one paragraph that says too little. It should read:

We time every policy end-to-end on the system it runs on. Quality is the cross-entropy (CE) of the output. We define CE for each image, and train our routers on it; we pool mIoU over all images, so we compare it separately at equal latency (Section 5.6). To weigh accuracy against time we fix a price λ: the CE that one second per image is worth. We use 0.8 CE per second. On validation images, with each view charged the time it takes alone, this is the price at which our router’s lead over the best static policy was largest, so the comparison starts in the router’s favor. At this price, multiscale’s gain of 0.069 CE over native pays for up to 86 ms per image. Run alone, its extra views cost 96 ms, so running them on every image does not pay, and choosing images looks worthwhile. In a batch of eight on the RTX 3090 they cost 38 ms, so running them on every image does pay.

The last three sentences are what the 86 ms was meant to show, and they are the paper’s message in small. We will replace the paragraph with this text.

I understand this is a small piece of the whole manuscript. I will revise them if we find further missing points.

W2. We agree that “an adaptive method does not always beat the best static policy” is expected, and we do not claim it as new. Our contribution is a procedure for checking such methods, and three places it found where the usual way of reporting misleads, each with its size and its cause:

The reported cost is not the deployed cost, and the difference changes the verdict. Papers count views or time one device. Counted that way, 83 of our 105 tuned rules beat the best static policy; timed end to end, 19 do. AdapTTA computes 27% fewer crops on EfficientNet-B0 and is 1% faster than five crops in one call.

The usual baseline is too weak. Against the three usual static policies, 88 of the 105 rules win at some price for time. Against static subsets found by a simple search, none does.

The verdict can be forecast before the adaptive method is run, from static timings alone. With forecasts saved first, a static policy won in all 22 settings where the criterion said it would. For AdapTTA the forecast said that its time saving would vanish on one classifier (−3%) and stay on the other (23%); we measured 1% and 27%. Intuition says a rule may lose. It does not say on which model, or by how much.

The same accounting also said what to change: a schedule that counts forward calls made the published rule 13 to 15% faster with identical predictions.

Thank you again for the time you have given this paper.

Reviewer_r463OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper evaluates whether adaptive test-time augmentation improves the quality-runtime trade-off relative to the best static policy in a specified menu. Using mainly a frozen SegFormer-B2 router, it measures complete execution across platforms and batch sizes, showing that fewer inference passes or shorter runtime need not yield higher utility. An ACDC experiment also compares selection using proxy costs with selection using complete costs.

Critical strengths
  1. The complete-workload measurements account for routing and output assembly, and the fixed-action intervention quantifies the cost of the observation/head path.

  2. The reporting is transparent about uncertainty and scope, including the unresolved benefit of complete-cost calibration on ACDC.

Critical weaknesses

[W1] - The central observations are straightforward, with limited new insight

The distinction between fewer forward passes and better overall efficiency is familiar. Once quality and runtime enter a common utility, a faster but less accurate policy may lose to a static alternative, and the choice of comparator naturally matters. The main accounting results organize these comparisons, but largely follow from the utility definition; converting a utility deficit into a required time saving does not provide a new predictive criterion.

A measurement study can contribute without proposing a new algorithm. Here, however, the specific measurements do not establish a sufficiently new empirical insight beyond the need to measure complete execution and choose appropriate comparators.

[W2] - The execution conclusions need validation under an optimized deployment configuration

The complete PyTorch timings are useful, but the paper does not establish whether the utility ordering persists under an optimized execution stack. CUDA Graph and compilation settings are not clearly documented. These choices matter because fixed augmentation paths and adaptive sub-batches may benefit differently from reduced launch overhead and graph capture. Dynamic sub-batch sizes can also affect utilization and the cost of handling changing shapes.

I would like to see at least one deployment-oriented comparison, using an optimized PyTorch configuration or a backend such as TensorRT, with appropriate optimization applied to both adaptive and static policies. A timeline profile would help separate host dispatch and synchronization from GPU execution. The current stage profiles combine dispatch with additional-view forwards, so they do not isolate scheduling losses. The authors acknowledge that improved execution may change the ordering; such a comparison would clarify which findings persist beyond the current implementation.

Questions for authors
  1. What new, transferable empirical conclusion follows beyond measuring complete costs and comparing against the best static policy?

  2. Do the utility reversals persist with optimized execution, and how do CUDA Graph settings and dynamic sub-batching affect the comparison?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer r463

Thank you for your review and for the precise questions on execution.

W1, Q1. What new, transferable conclusion follows?

A predictive criterion. The break-even calculation did not give this criterion and two measurements behind it.

We give that Corollary 1 is an identity. When one factor κ rescales the cost of every extra view and the router's own cost is not negative, the router can win only while κ lies in its band. What is empirical finding is that these conditions held on systems the router had never been timed on. With forecasts saved first (Section 5.5, Table 5), a static policy won in all 20 new settings with κ outside the band; inside it the router won four of seven. Always guessing "static" is right in 23 of 27. At batch size 1, where either side can win, the surplus forecast named all nine winners (a constant guess: five).

Second, the rest of a router's cost is mostly a fixed host cost during one forward call, so skipping a view saves little if it does not remove a call. On Mask2Former, 20% of images require an extra view, yet the mirror call still runs in 81% of batches of eight. With this cost per call, 43 of 48 batched twelve-view margins were forecast within 1.5 ms per image (17 with each view priced at its own deployed cost).

The same cost accounting carried over to a published method on another task and a desktop CPU (Section 5.7). For five-crop AdapTTA, forecasts saved before the test said that its saving of 27% and 36% of the crops would be −3% and 23% of the time of five crops in one call. Measured: 1% and 27%.

W2, Q2. Optimized execution.

CUDA graphs and dynamic-shape compilation applied to every policy on two GPUs with BF16. The reversals persist when data is batched, and they grow. At batch size 8 the router's deficit increased from 1.7 and 6.2 in the FP32 reruns (in 10⁻³ CE, RTX 3090 and A100) to 6.8 and 10.5 in BF16, and to 8.5 and 10.8 under CUDA graphs. Every router win is at batch size 1 without CUDA graphs, and none is distinct from a tie over resampled images. Sub-batches cost the router a fixed host cost per call without graphs (Appendix D.9). Padding with them took 3.67 ms per image on the RTX 3090 at batch size 8 (Table A16; a traced decision is in Figure A5).

TensorRT (Appendix D.6). We ran it last, on an L4, a GPU the study had not used, with FP16 engines shared by every policy and the forecasts saved before the router was timed. TensorRT brought κ from 0.88 to 0.90 (FP32 PyTorch on that GPU) down to 0.37 and 0.43, below the band, so every forecast named a static policy, and multiscale won at batch sizes 1 and 8: the router's surplus was −3.8 and −1.7 (forecast −6.1 and −1.8, in 10⁻³ CE). The losses are small, ties over resampled images. The router would also lose with its whole pipeline cost removed, as in all 18 batched settings before (Appendix D.9): κ decides, and static timings give it. Batch size 32 did not fit the L4's memory.

Thank you very much again for these helpful questions.

Thank you for the thoughtful response. I still see limited evidence that it can identify when adaptation offers a meaningful gain, since the adaptive wins were close to ties. This leaves me with some reservations about the novelty of the findings, so I’ll keep my original score.

Thank you for reading our response and for saying what is still missing. You are right about: every win of our routers is small and, over resampled images, not distinct from a tie.

We may not have made clear enough what kind of paper this is. It proposes no method and does not claim that adaptation gives a meaningful gain. It is an evaluation procedure, and its findings are about where the usual way of reporting adaptive test-time augmentation overstates. That the wins shrink to ties once everything is timed is one of those findings. There are three.

  1. Counting views overstates the saving. With each view charged its stand-alone time, 83 of our 105 tuned rules beat the best static policy; timed end to end, 19 do. The published AdapTTA rule computes 27% fewer crops on EfficientNet-B0 and is 1% faster than five crops in one call.

  2. The usual static baselines are too weak. When timed, 88 of the 105 rules beat native, flip and multiscale at some price for time. With three static subsets found by search in the menu, none does at any price.

  3. Where a rule cannot win can be told before it is built. In all 22 settings where the cost drift lay outside the band, a static policy won, with forecasts saved first. This now includes the TensorRT test you asked for, run since our first response on an L4 (Appendix D.6): the drift fell to 0.37 and 0.43, and a static policy won at batch sizes 1 and 8, as forecast.

On your question itself: the forecast also gives the size. It put the router’s four wins between +0.3 and +5.3 (in 10⁻³ CE) before they were timed at +1.1 to +6.6 (Table 4), so “close to a tie” was the forecast. And where a gain was large, the same accounting said so in advance: for AdapTTA on ResNet-50, a forecast time saving of 23% against a measured 27% (Section 5.7).

We agree that the evidence for large gains is one published rule on a CPU, and the paper states this as a limit. Thank you again for the careful reading, and for asking for TensorRT.

Reviewer_ka1GOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether choosing test-time augmentations for each image improves accuracy enough to justify the inference time. It measures routing costs as well as model execution, showing why fewer passes or faster processing may still lose to a fixed inference strategy.

Critical strengths
  1. Complete-workload measurements include routing, transfers, preprocessing, and output assembly, addressing a common weakness of efficiency comparisons.
  2. The experimental design of the paper is highly comprehensive; it includes all data from the actual reasoning process and verifies the consistency between the actual decisions made and the final outputs.
  3. The paper carefully distinguishes descriptive findings, sampling uncertainty, and unverified superiority rather than overstating its negative results.
Critical weaknesses
  1. Most runtime experiments use one fixed SegFormer-B2 model with native, flip, and multiscale processing. More efficient batching or other adaptive architectures could change the conclusions.
  2. While Cross-Entropy (CE) is a valid optimization objective, semantic segmentation typically prioritizes mIoU; the experimental results indicate that the proposed approach does not offer an advantage in terms of either mIoU or runtime.
Questions for authors
  1. If data augmentation is handled more efficiently—for instance, through batch processing or other adaptive models—which conclusions remain valid?
  2. When the focus is on mIoU or when there is a fixed latency budget, how should one strike a balance between accuracy and computational time?
1 comment
AuthorsOpenReview ↗

Reply to Reviewer ka1G

Thank you for your thoughtful and helpful review. You are right that our router gives no mIoU advantage over well-chosen static policies at equal runtime. The revision says so plainly and explains it.

W1, Q1. Do the conclusions hold with faster augmentation or other adaptive designs?

At our price for time, yes when views are batched. At batch size 1, or at a higher price, either side can win. Faster execution makes each extra view cheaper, and then running all views pays more than choosing among them (Corollary 1). In advance tests with BF16, CUDA graphs and compilation on two GPUs, the router lost by more in all ten new batched settings, as forecast. At batch size 1 it won in three of six, also as forecast, though within noise over images (Section 5.5). For other designs, we tuned 105 rules from five designs, two of them after AdapTTA and Kim et al. Once three static subsets found by search are in the menu, none wins at any price for time (Section 5.6). A published rule on its own task shows the same effect on a desktop CPU: five-crop AdapTTA computes 27% fewer crops on EfficientNet-B0 but is only 1% faster than five crops in one call, 8% in a first test (Section 5.7).

W2. mIoU.

We agree, and we added it. On all 2,000 ADE20K images the router reaches 46.15 mIoU at 31.6 ms, where a mix of static policies reaches 46.23 (−0.08, 95% interval −0.22 to +0.08). A label oracle lies 1.3 to 1.4 points higher, but the routers do not reach that room. Only on ACDC was ours ahead of multiscale, by 0.1 points and faster, in one comparison without an interval.

Q2. Balancing accuracy and time.

We recommend evaluating the computational cost of static policies and various adaptive rules within the deployment environment. Based on a predefined time limit, these can then be filtered using the mIoU metric calculated on a validation set. In a subsequent analysis of 21 cases, adaptive rules achieved a higher mIoU than three conventional static policies in 14 instances; however, when three additional static subsets—derived via search—are included in the comparison, the adaptive rules fail to outperform them in any case (with static policies leading by an average of 0.20 percentage points; see Appendix H.2).

Under TensorRT on an L4, run last, the router also lost at batch sizes 1 and 8, as forecast and by small margins (Appendix D.6). Fused multi-action execution remains untested (Section 6).

Thank you again very much for your helpful questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response

We greatly thank the three reviewers for their careful reading and advice. All three reviewers wrote that they learned little from the submission, and that point was fair: it measured carefully but did not say what the measurements teach. We reorganized the manuscript first. The revision now states three findings, tests the first two in advance, and applies the procedure to a published method.

What is new.

  1. A criterion from static timings (Corollary 1, Section 5.5). Deployment rescales the cost of extra views of similar size by about one factor, the cost drift κ, which timing the static policies gives. A rule tuned at stand-alone costs can win only while κ lies in the middle band. When the router utilized stored predicted values ​​before operating under a new configuration, the "static" policy outperformed the alternative in all 22 cases where the κ value fell outside the designated band, and in 4 out of 7 cases where it fell within the band. In other words, the "static" policy proved correct in 25 out of 29 instances; thus, this paper also validates the performance margin.

  2. The cost is per call. Since most of the overhead cuased by the router, beyond processing the views themselves, is a fixed cost per forward call, skipping views only saves time when the full set of calls would have otherwise proceeded. In Mask2Former, 20% of images trigger a request for additional views, yet the mirrored calls are still executed in 81% of cases within a batch of eight images.

  3. The bar is a searched static policy (Section 5.6). At stand-alone costs, 83 of our 105 tuned rules beat the three usual static policies; timed, 19 do; with three static subsets found by search in the menu, none wins at any price for time. A label oracle still gains 1.3 to 1.4 mIoU points, so prediction is the limit.

A published method on its own task (Section 5.7). Five-crop AdapTTA on ImageNetV2, run on a desktop CPU in place of its embedded target, computes 27% and 36% fewer crops but saves 1% and 27% of the time of five crops in one call, near the forecasts saved before the test (−3% and 23%); counting forward calls made it 13 to 15% faster.

Limits. Two segmentation models and one classification test on one CPU. Only five-crop AdapTTA runs its published rule; the other designs are ours.

We answer each reviewer below and would be glad to clarify anything further.

As only 24 hours remain in the author rebuttal period, I sincerely invite the reviewers to review the author's response and promptly raise any remaining questions or indicate whether you believe the author has addressed the issues you have raised.

AuthorsOpenReview ↗

Summary for the Area Chair and reviewers

We thank the reviewers and the area chair for the discussion. We add a short summary because the first version of the paper did not make its claim easy to see. That was our doing.

What the paper is. It proposes no method. It is a procedure for checking adaptive test-time augmentation at deployed cost, and three findings from applying it:

  1. Reported cost is not deployed cost. We tuned 105 adaptive rules on SegFormer-B2: five designs (two confidence gates, a stopping rule after AdapTTA, a loss predictor after Kim et al., and our router), each at three latency budgets, in seven settings on an RTX 3090 (FP32 and BF16 at batch sizes 1, 8 and 32; CUDA graphs at batch size 1). With each view charged the time it takes alone, 83 of the 105 beat the best static policy. Timed end to end, 19 do.
  2. The usual baselines are too weak. Against three static view subsets found by a simple greedy search, none of the 105 rules wins at any price for time.
  3. The outcome can be forecast from static timings alone. We saved forecasts before timing the adaptive policy in 29 new settings: two models (SegFormer-B2, Mask2Former), three GPUs (RTX 3090, A100, L4), five software stacks (FP32, BF16, CUDA graphs, dynamic-shape compilation, TensorRT) and batch sizes from 1 to 32. The criterion said the adaptive policy could not win in 22 of them, and a static policy won in all 22. In the other seven, the adaptive policy won four times.

What we do not claim. That adaptive methods cannot work (a label oracle shows room), or that "an adaptive method may lose" is new.

What the discussion added. A published method on its own task: five-crop AdapTTA on ImageNetV2 with two classifiers on a CPU computes 27% and 36% fewer crops, and is 1% and 27% faster than five crops in one call. We also added the TensorRT test and rewrote the main text.

How each count is made up

CountMade of
105 rules5 designs × 3 latency budgets × 7 settings on an RTX 3090 (15 rules per setting)
7 settingsFP32 at batch 1, 8, 32; BF16 at batch 1, 8, 32; CUDA graphs at batch 1
83 win at stand-alone cost12, 12, 12, 11, 12, 12, 12 of 15, in the order of the seven settings
19 win when timed5, 2, 2, 10, 0, 0, 0 of 15, in the same order
29 new settings16 + 7 + 4 + 2 (the next four rows)
16: SegFormer, faster stacks2 GPUs (RTX 3090, A100) × (BF16 at batch 1, 8, 32; CUDA graphs at batch 1, 8; dynamic shapes at batch 1, 8, 32); 12 ruled out by the criterion
7: Mask2FormerRTX 3090: FP32 and BF16 at batch 1, 8, 32; CUDA graphs at batch 1; 6 ruled out
4: Mask2Former, other batch sizesRTX 3090: FP32 and BF16 at batch 4 and 16; 2 ruled out
2: SegFormer, TensorRTL4 at batch 1 and 8; 2 ruled out
22 ruled out12 + 6 + 2 + 2; a static policy won in all 22
7 not ruled out4 + 1 + 2 + 0; the adaptive policy won 3 + 1 + 0 + 0 = 4

"Ruled out" means that the cost ratio measured from the static policies' timings lay outside the range in which the adaptive policy can win. The forecasts were saved before the adaptive policy was timed.

We would be grateful if the paper were judged as an analysis evaluation study on these claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.