- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies a cost-controlled test-time adaptation setting where different adaptation methods are provided a fixed cost for adaptation (either per adaptation stream, or per fixed window). The paper proposes ECO-TTA, an orthogonal direction that can be combined with any TTA method for better budget utilization. Experiments are carried out on ImageNet-{C, R, S} showing that ECO-TTA outperforms random and greedy budget assignments.
Critical strengths
The main strengths of this paper are:
-
The orthogonality of ECO-TTA. It can be combined with any TTA method.
-
Under r=1, ECO-TTA shows performance improvement. This means that indeed ECO-TTA has a better assignment of budget during adaptation. Equivalently in this setting, ECO-TTA can be leveraged as a data-filtration process for adaptation
-
The paper is generally well written and easy to follow.
Critical weaknesses
The main weaknesses of this work are:
-
The main motivation of this work is unclear for me. While indeed at edge cases, compute can be limited, but that does not grant a fixed compute budget per window. However, the update budget is dictated by the available compute and stream speed. These two factors decide on the maximum number of updates that can be done per received batch. The current formulation sums this budget over the total number of batches to decide on the total budget per adaptation, which is not practical (if one skips one batch, that does not give extra budget when receiving the next batch). I need a proper justification on why is the presented setting relevant and when is it practical?
-
The entire formulation of ECO-TTA assumes the knowledge of T a-priori. However, during testing. This dis-validates the formulation presented.
-
Different TTA methods have different per-update costs. The paper completely ignores this critical practical consideration. A proper discussion and empirical validation with this taken into account is necessary.
-
I am still not convinced about how practical is the setting of this problem setup. I think there is merit in analyzing the cost of updates, but isn't the problem defined by how fast is the stream during testing in releasing new batches to the model? For instance, if the stream is too fast, one cannot do more x updates even if ECO-TTA advises to.
-
I think a better narrative for the proposed method is to be used as a data-filtration mechanism (similar to the ones in ETA/SAR). Unlike the methods in the literature, ECO-TTA can decide on the batch as a whole to skip/update. For example, the results with in Table 4 hint for the success of ECO-TTA in this task.
Questions for authors
For the greedy strategy, what does "as early as possible" mean? Does it mean that the adaptation method consumes all updates on the first batch? or does it consume a single update per batch until it runs out of budget?
5 comments
Response to f7y8 (part II)
W4: How practical is the budget formulation when stream arrival rates limit the number of updates? For instance, if the stream is too fast, one cannot do more x updates even if ECO-TTA advises to.
A4: Thank you for this question. We supply a practical deployment experiment in R-Table 15, where the fast stream limits more updates in ECO-TTA. We incorporate this constraint into ECO-TTA by first excluding actions that exceed either the remaining budget or the per-batch processing-time allowance, then selecting among feasible actions using predicted gains and the dynamic price. Under these constraints, ECO-TTA consistently outperforms Uniform and Greedy across all four corruption groups (R-Table 16). We have included this example and evaluation in Appendix D.4 of the revised manuscript.
R-Table 15. Shared-GPU deployment example with measured runtimes and resource constraints.
| Item | Setting |
|---|---|
| Batch arrival rate | 2 batches/s, 120 batches/min |
| Inference time per batch | 66 ms |
| Time per adaptation step | 138 ms, including gain estimation |
| GPU time required by background workloads | 45 s/min |
| Maximum updates per batch | |
| Update budget per 120-batch window |
R-Table 16. Accuracy (%) on ImageNet-C under label shifts and the stream constraints in R-Table 15.
| Scheduler | Noise | Blur | Weather | Digital | Avg. |
|---|---|---|---|---|---|
| Greedy | 52.13 | 57.50 | 69.82 | 64.59 | 61.60 |
| Uniform | 52.77 | 57.89 | 69.69 | 65.87 | 62.14 |
| ECO-TTA | 57.58 | 58.03 | 70.83 | 67.91 | 63.99 |
W5: Could ECO-TTA be better understood as a data-filtration mechanism, similar to ETA or SAR?
A5: We agree that batch-level adaptation filtering is an intuitive interpretation of ECO-TTA. It selects which batches to update and how many steps to allocate, while retaining the base method's adaptation objective and sample-selection rules. At the same budget allowance (), selective scheduling improves all reported methods over full adaptation on ImageNet-C and ImageNet-R (R-Table 17). We have made this filtering perspective explicit in the revised manuscript.
R-Table 17. Accuracy (%) comparison with original method. ECO-TTA uses ; Full denotes the original method updating every batch.
| TTA method | ImageNet-C: Full | ImageNet-C: ECO-TTA | ImageNet-R: Full | ImageNet-R: ECO-TTA |
|---|---|---|---|---|
| Tent | 49.28 | 61.07 | 44.39 | 59.46 |
| SAR | 54.32 | 64.07 | 51.66 | 61.96 |
| ReCAP | 64.21 | 67.96 | 60.92 | 68.42 |
| ZeroSiam | 59.93 | 63.84 | 54.08 | 62.44 |
Q1: For the greedy strategy, what does "as early as possible" mean?
A6: Sorry for the confusion. Greedy performs one update per batch whenever budget is available. Under a global budget, it updates the earliest batches until the total budget is exhausted. Under a sliding-window budget, it skips updates when the quota is exhausted and resumes when quota becomes available as the window advances. We have clarified this definition in the revised manuscript.
Look forward to your post-rebuttal feedback!
Dear Reviewer f7y8,
Thank you again for your thoughtful suggestions and comments. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need.
In our previous response, we carefully addressed your feedback and made the following updates and additions:
- Explained the practical motivation for global and sliding-window budgets.
- Conducted additional experiments to validate ECO-TTA's operation without a known stream length.
- Conducted additional experiments under a compute-time budget to account for method-specific update costs.
- Conducted additional experiments in a fast-stream setting.
- Compared ECO-TTA with full adaptation to assess the benefits of batch-level adaptation filtering.
- Clarified how Greedy scheduling operates under both budget protocols.
We hope these explanations and new experiments have clarified the contributions and strengthened our submission.
Please do not hesitate to reach out if there are any further clarifications or analyses we can provide.
Thank you for your time and careful consideration!
Best, Authors
Response to f7y8 (part I)
Thank you for your careful review and for recognizing ECO-TTA’s compatibility with different TTA methods, clear presentation, and improved budget allocation. We also appreciate your insight into its value as a data-filtering mechanism for adaptation. Your constructive suggestions have helped us clarify its deployment scenarios and better demonstrate its practical value. Below, we address each concern with clarifications and additional experiments.
A1: Thank you for this valuable suggestion. Our setting models cumulative resource limits imposed on adaptation by deployment costs or resource-allocation policies. Two practical examples are:
We have provided a practical deployment example in Appendix D.4. We have clarified the practical sources of these budgets and their applicable deployment scenarios in Section 4.1 of the revised manuscript.
A2: Thank you for this feedback. ECO-TTA can operate without prior knowledge of T. We have added a sliding-window formulation for unknown or unbounded streams (Sec 3.1, Eq. (4)). In this setting, ECO-TTA can directly set ρB=B/W or estimate online from available adaptation time and measured update costs (Sec 3.3, Eq. (8)). Neither option requires T.
We have also added experiments in which neither setting provides T to the scheduler (Appendix C.3, Table 17). As summarized in R-Table 13, fixed and online-estimated rates yield comparable accuracy under stable resources. Under varying resources, online estimation improves accuracy from 62.42% to 62.91%, a gain of 0.49%.
R-Table 13. Fixed versus online-estimated spending rates with ReCAP under label shifts and window budgets.
A3: Thank you for this constructive suggestion. We additionally evaluate all methods on the same hardware under a common 180 s compute-time budget, using measured runtimes to account for method-specific update costs. Under this setting, higher per-update costs translate into fewer available updates. R-Table 14 shows that ECO-TTA consistently outperforms Uniform and Greedy across all six methods, with average accuracy gains of +7.32%. These improvements hold for methods with both faster updates, such as Tent (+10.67%), and slower updates, such as SAR (+10.30%). We have included this discussion and empirical validation in Section 4.2 of the revised manuscript.
R-Table 14. Accuracy (%) on ImageNet-C under label shifts with a 180 s compute-time budget, averaged over 15 corruptions. Average inference time (without adaptation) is 65.7 ms.