- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes an evaluation protocol for assessing the cost-effectiveness (accuracy versus runtime) of adaptive test-time augmentation (TTA). It introduces a surplus metric, defined as the difference in utility (negative cross-entropy minus priced complete runtime) between an adaptive router and the best-performing static policy, to determine whether the router outperforms the most cost-effective static alternative. Experiments under varied conditions, including execution environment (RTX 3090, T4, A100) and batch size, show that the router does not necessarily outperform the best static policies.
Critical strengths
- S1. This paper does not propose a new TTA method; instead, it focuses on evaluating the cost-effectiveness of TTA methods. In practical deployment, it is important to consider computational cost as well as accuracy.
Critical weaknesses
-
W1. Clarity. The paper is difficult to read. It provides little explanation of existing adaptive TTA methods. It also introduces many paper-specific terms, and their definitions often begin before the motivation for introducing them is sufficiently explained.
-
W2. Limited novelty. The main novelty of the paper is the proposed evaluation protocol; it introduces neither a new technique nor particularly novel findings. The scope of the empirical claims is also limited. The router is designed by the authors and differs from existing adaptive TTA methods. Only three actions are evaluated, which is considerably fewer than in existing adaptive TTA work (e.g., Kim et al., 2020 use 12 augmentation types). The main experiments are restricted to a single architecture (SegFormer-B2), a single task (semantic segmentation), and two datasets (ADE20K and ACDC). Therefore, while the reported findings may hold within this limited setting, it is unclear whether they generalize. Furthermore, the paper does not apply the proposed protocol to any existing adaptive TTA method, so it remains unclear what the protocol would reveal about existing methods. Consequently, there is limited evidence that the protocol itself is important.
Questions for authors
See Weaknesses.
3 comments
Thank you for the rebuttal.
W1. The paper appears to have been substantially revised. The Author Guidelines state that reviewers are not required to look at every revision. I therefore checked only the introduction. It is more readable than the previous version, but the paper remains difficult to read. For example, in the “The measurements” paragraph, the price is described only as “a fixed exchange rate between cross-entropy (CE) and time”; its meaning (how much CE one second per image is worth) becomes clear only in Table 1, which is referenced at the end of the paragraph. The paragraph also does not explain why CE is used as the quality metric. Furthermore, 0.8 CE/s is described as “the value that favours our router most,” without stating by which metric and on which data, and it is unclear what the statement that 0.069 CE is worth 86 ms is meant to show.
W2. The application to an existing method (AdapTTA) addresses my request. However, the newly presented results do not contain novel findings. They only show that the paper’s claim, that an adaptive TTA method does not necessarily outperform the best static TTA policy in terms of accuracy and runtime, holds in extended experimental settings, which is intuitively expected.
Since my concerns are not resolved, I will keep my score.
Thank you for reading the revised introduction and for the concrete examples.
W1. All four points come from one paragraph that says too little. It should read:
We time every policy end-to-end on the system it runs on. Quality is the cross-entropy (CE) of the output. We define CE for each image, and train our routers on it; we pool mIoU over all images, so we compare it separately at equal latency (Section 5.6). To weigh accuracy against time we fix a price λ: the CE that one second per image is worth. We use 0.8 CE per second. On validation images, with each view charged the time it takes alone, this is the price at which our router’s lead over the best static policy was largest, so the comparison starts in the router’s favor. At this price, multiscale’s gain of 0.069 CE over native pays for up to 86 ms per image. Run alone, its extra views cost 96 ms, so running them on every image does not pay, and choosing images looks worthwhile. In a batch of eight on the RTX 3090 they cost 38 ms, so running them on every image does pay.
The last three sentences are what the 86 ms was meant to show, and they are the paper’s message in small. We will replace the paragraph with this text.
I understand this is a small piece of the whole manuscript. I will revise them if we find further missing points.
W2. We agree that “an adaptive method does not always beat the best static policy” is expected, and we do not claim it as new. Our contribution is a procedure for checking such methods, and three places it found where the usual way of reporting misleads, each with its size and its cause:
The reported cost is not the deployed cost, and the difference changes the verdict. Papers count views or time one device. Counted that way, 83 of our 105 tuned rules beat the best static policy; timed end to end, 19 do. AdapTTA computes 27% fewer crops on EfficientNet-B0 and is 1% faster than five crops in one call.
The usual baseline is too weak. Against the three usual static policies, 88 of the 105 rules win at some price for time. Against static subsets found by a simple search, none does.
The verdict can be forecast before the adaptive method is run, from static timings alone. With forecasts saved first, a static policy won in all 22 settings where the criterion said it would. For AdapTTA the forecast said that its time saving would vanish on one classifier (−3%) and stay on the other (23%); we measured 1% and 27%. Intuition says a rule may lose. It does not say on which model, or by how much.
The same accounting also said what to change: a schedule that counts forward calls made the published rule 13 to 15% faster with identical predictions.
Thank you again for the time you have given this paper.
Reply to Reviewer q93m
Thank you for your direct and helpful review. We totally agree with both main points. The paper was hard to read, and a protocol must show what it finds exactly on existing methods.
W1. Clarity.
The main text is rewritten in plain and easy words. It opens with the result (Figure 1) and collects its terms in Table 1. Then it explains the existing methods (Section 2), works through one example before the corollary (Section 3), and ends with a four-step test (Section 6).
W2. Novelty and scope.
What the protocol reveals about an existing method (Section 5.7). We applied it to five-crop AdapTTA with its threshold of 0.8, on its own task (ImageNetV2, two classifiers), on a desktop CPU in place of its embedded target. (a) Confidence does pick the right images: on EfficientNet-B0 every early stop kept the five-crop label, against 97.5% of images when the same stopping depths go to random images. (b) Its saving in crops carries over only in part: it computes 27% and 36% fewer crops but saves 1% (8% in a first test) and 27% of the time of five crops in one call, near the forecasts saved before the test. (c) Counting forward calls made it 13 to 15% faster. (d) Against static crop sets chosen on validation data it has no clear advantage: on EfficientNet-B0 two crops match its top-1 (−0.1 points, interval −0.9 to +0.7) in half the time; on ResNet-50 it wins over a range of prices, though not firmly.
More than three actions. We timed a twelve-view menu with three selectors, one a loss predictor trained by the recipe of Kim et al. (Appendix D.10). In CE, each of the 61 adaptive choices has a static competitor that is faster and better. In mIoU the trained predictor wins sixteen comparisons; a fixed three-view order, timed later, was level with it and faster. This tests their recipe on our views only, not their own results.
One architecture, our own router. Timing now covers SegFormer, Mask2Former with its own router (Appendix D.8) and two ImageNet classifiers. The router is ours, which limits what its results say about published methods; the arithmetic of Corollary 1 carries over to other rules.
What we learned is set out in the general response. In short, static timings, with one untimed pass of a rule, tell where it cannot win (formally, a static policy won in all 20 settings outside the band). A saving in views is a saving in time only if a forward call goes. The searched static subsets are the bar, although a label oracle shows 1.3 to 1.4 mIoU points of room.
We narrowed the claims to what was run (Section 6). Thank you again for pushing us to make the paper readable.