acceptodds
Under review as a conference paper at ICLR 2027

Is Adaptation Worth the Cost? Economics-Guided Test-Time Adaptation under Budget Constraints

Abstract

Test-Time Adaptation (TTA) enables models to dynamically adjust to non-stationary test distributions through online updates during inference. However, when computational resources are limited, existing methods with pre-defined update schedules may waste budget on low-value or unreliable batches, leading to suboptimal overall utility. This raises a critical yet overlooked question: under a limited test-time budget, which adaptations are truly worth performing? To address this issue, we propose **ECO-TTA**, an **ECO**nomics-guided **TTA** framework that reformulates adaptation as a cost-aware investment process. Specifically, ECO-TTA defines a multi-level action space and treats each candidate action as a purchasable asset, whose price is its adaptation cost and value is its performance payoff. To estimate this payoff without labels, ECO-TTA derives an efficient value estimator that transfers realized adaptation gains from statistically similar historical batches to predict the expected return of the current asset. It further designs an adaptive budget controller based on the shadow price, which quantifies the opportunity cost of current budget consumption and adjusts the cost penalty along budget consumption trajectories. Together, these designs establish a unified cost-benefit criterion for online decision-making, enabling an adaptive trade-off between limited budgets and high-value updates. Our extensive experiments demonstrate the consistent superiority of ECO-TTA across diverse shift types, TTA scenarios, and budget settings. Code will be publicly available.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-Time Adaptation, Transfer Learning, Dynamic Resource Allocation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f7y8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a cost-controlled test-time adaptation setting where different adaptation methods are provided a fixed cost for adaptation (either per adaptation stream, or per fixed window). The paper proposes ECO-TTA, an orthogonal direction that can be combined with any TTA method for better budget utilization. Experiments are carried out on ImageNet-{C, R, S} showing that ECO-TTA outperforms random and greedy budget assignments.

Critical strengths

The main strengths of this paper are:

  1. The orthogonality of ECO-TTA. It can be combined with any TTA method.

  2. Under r=1, ECO-TTA shows performance improvement. This means that indeed ECO-TTA has a better assignment of budget during adaptation. Equivalently in this setting, ECO-TTA can be leveraged as a data-filtration process for adaptation

  3. The paper is generally well written and easy to follow.

Critical weaknesses

The main weaknesses of this work are:

  1. The main motivation of this work is unclear for me. While indeed at edge cases, compute can be limited, but that does not grant a fixed compute budget per window. However, the update budget is dictated by the available compute and stream speed. These two factors decide on the maximum number of updates that can be done per received batch. The current formulation sums this budget over the total number of batches to decide on the total budget per adaptation, which is not practical (if one skips one batch, that does not give extra budget when receiving the next batch). I need a proper justification on why is the presented setting relevant and when is it practical?

  2. The entire formulation of ECO-TTA assumes the knowledge of T a-priori. However, T→∞T\rightarrow \infty during testing. This dis-validates the formulation presented.

  3. Different TTA methods have different per-update costs. The paper completely ignores this critical practical consideration. A proper discussion and empirical validation with this taken into account is necessary.

  4. I am still not convinced about how practical is the setting of this problem setup. I think there is merit in analyzing the cost of updates, but isn't the problem defined by how fast is the stream during testing in releasing new batches to the model? For instance, if the stream is too fast, one cannot do more x updates even if ECO-TTA advises to.

  5. I think a better narrative for the proposed method is to be used as a data-filtration mechanism (similar to the ones in ETA/SAR). Unlike the methods in the literature, ECO-TTA can decide on the batch as a whole to skip/update. For example, the results with r=1r=1 in Table 4 hint for the success of ECO-TTA in this task.

Questions for authors

For the greedy strategy, what does "as early as possible" mean? Does it mean that the adaptation method consumes all updates on the first batch? or does it consume a single update per batch until it runs out of budget?

5 comments
AuthorsOpenReview ↗

Response to f7y8 (part I)

Thank you for your careful review and for recognizing ECO-TTA’s compatibility with different TTA methods, clear presentation, and improved budget allocation. We also appreciate your insight into its value as a data-filtering mechanism for adaptation. Your constructive suggestions have helped us clarify its deployment scenarios and better demonstrate its practical value. Below, we address each concern with clarifications and additional experiments.


W1: I need a proper justification on why is the presented setting relevant and when is it practical?

A1: Thank you for this valuable suggestion. Our setting models cumulative resource limits imposed on adaptation by deployment costs or resource-allocation policies. Two practical examples are:

  1. Usage-based billing. For a cloud inference service billed by GPU execution time, the operator sets a fixed spending cap for adaptation over a service session, corresponding to our global budget.
  2. Shared-GPU inference. On a GPU shared by online inference and background jobs, adaptation receives a GPU-time quota per rolling window to preserve resources for other workloads, corresponding to our window budget.

We have provided a practical deployment example in Appendix D.4. We have clarified the practical sources of these budgets and their applicable deployment scenarios in Section 4.1 of the revised manuscript.


W2: The formulation of ECO-TTA assumes T a-priori, but test streams can be unbounded during testing.

A2: Thank you for this feedback. ECO-TTA can operate without prior knowledge of TT. We have added a sliding-window formulation for unknown or unbounded streams (Sec 3.1, Eq. (4)). In this setting, ECO-TTA can directly set ρB=B/W\rho_B=B/W or estimate online from available adaptation time and measured update costs (Sec 3.3, Eq. (8)). Neither option requires TT.

We have also added experiments in which neither setting provides TT to the scheduler (Appendix C.3, Table 17). As summarized in R-Table 13, fixed and online-estimated rates yield comparable accuracy under stable resources. Under varying resources, online estimation improves accuracy from 62.42% to 62.91%, a gain of 0.49%.

R-Table 13. Fixed versus online-estimated spending rates with ReCAP under label shifts and window budgets.

Resource availabilityRate settingAccuracy (%)
Stable, r=0.2r=0.2Fixed ρB=0.2\rho_B=0.262.42
Stable, r=0.2r=0.2Estimated online62.35
Varying, r=0.2→0.3→0.1r=0.2\rightarrow0.3\rightarrow0.1Fixed ρB=0.2\rho_B=0.262.42
Varying, r=0.2→0.3→0.1r=0.2\rightarrow0.3\rightarrow0.1Estimated online62.91

W3: Different TTA methods have different per-update costs. A proper discussion and empirical validation with this taken into account is necessary.

A3: Thank you for this constructive suggestion. We additionally evaluate all methods on the same hardware under a common 180 s compute-time budget, using measured runtimes to account for method-specific update costs. Under this setting, higher per-update costs translate into fewer available updates. R-Table 14 shows that ECO-TTA consistently outperforms Uniform and Greedy across all six methods, with average accuracy gains of +7.32%. These improvements hold for methods with both faster updates, such as Tent (+10.67%), and slower updates, such as SAR (+10.30%). We have included this discussion and empirical validation in Section 4.2 of the revised manuscript.

R-Table 14. Accuracy (%) on ImageNet-C under label shifts with a 180 s compute-time budget, averaged over 15 corruptions. Average inference time (without adaptation) is 65.7 ms.

TTA methodOne-step time (ms)UniformGreedyECO-TTAAvailable updates
Tent150.451.7351.2862.17912
SAR302.551.8752.0362.25326
ActMAD210.70.330.189.56533
CMF278.766.5765.3369.76362
ReCAP203.362.2761.6265.99561
ZeroSiam151.957.7059.9664.58896
AuthorsOpenReview ↗

Response to f7y8 (part II)

W4: How practical is the budget formulation when stream arrival rates limit the number of updates? For instance, if the stream is too fast, one cannot do more x updates even if ECO-TTA advises to.

A4: Thank you for this question. We supply a practical deployment experiment in R-Table 15, where the fast stream limits more updates in ECO-TTA. We incorporate this constraint into ECO-TTA by first excluding actions that exceed either the remaining budget or the per-batch processing-time allowance, then selecting among feasible actions using predicted gains and the dynamic price. Under these constraints, ECO-TTA consistently outperforms Uniform and Greedy across all four corruption groups (R-Table 16). We have included this example and evaluation in Appendix D.4 of the revised manuscript.

R-Table 15. Shared-GPU deployment example with measured runtimes and resource constraints.

ItemSetting
Batch arrival rate2 batches/s, 120 batches/min
Inference time per batch66 ms
Time per adaptation step138 ms, including gain estimation
GPU time required by background workloads45 s/min
Maximum updates per batch⌊500−66138⌋=3\left\lfloor \frac{500-66}{138} \right\rfloor=3
Update budget per 120-batch window⌊15,000−120×66138⌋=51\left\lfloor \frac{15{,}000-120\times66}{138} \right\rfloor=51

R-Table 16. Accuracy (%) on ImageNet-C under label shifts and the stream constraints in R-Table 15.

SchedulerNoiseBlurWeatherDigitalAvg.
Greedy52.1357.5069.8264.5961.60
Uniform52.7757.8969.6965.8762.14
ECO-TTA57.5858.0370.8367.9163.99

W5: Could ECO-TTA be better understood as a data-filtration mechanism, similar to ETA or SAR?

A5: We agree that batch-level adaptation filtering is an intuitive interpretation of ECO-TTA. It selects which batches to update and how many steps to allocate, while retaining the base method's adaptation objective and sample-selection rules. At the same budget allowance (r=1r=1), selective scheduling improves all reported methods over full adaptation on ImageNet-C and ImageNet-R (R-Table 17). We have made this filtering perspective explicit in the revised manuscript.

R-Table 17. Accuracy (%) comparison with original method. ECO-TTA uses r=1r=1; Full denotes the original method updating every batch.

TTA methodImageNet-C: FullImageNet-C: ECO-TTAImageNet-R: FullImageNet-R: ECO-TTA
Tent49.2861.0744.3959.46
SAR54.3264.0751.6661.96
ReCAP64.2167.9660.9268.42
ZeroSiam59.9363.8454.0862.44

Q1: For the greedy strategy, what does "as early as possible" mean?

A6: Sorry for the confusion. Greedy performs one update per batch whenever budget is available. Under a global budget, it updates the earliest batches until the total budget is exhausted. Under a sliding-window budget, it skips updates when the quota is exhausted and resumes when quota becomes available as the window advances. We have clarified this definition in the revised manuscript.

AuthorsOpenReview ↗

Look forward to your post-rebuttal feedback!

Dear Reviewer f7y8,

Thank you again for your thoughtful suggestions and comments. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need.

In our previous response, we carefully addressed your feedback and made the following updates and additions:

  • Explained the practical motivation for global and sliding-window budgets.
  • Conducted additional experiments to validate ECO-TTA's operation without a known stream length.
  • Conducted additional experiments under a compute-time budget to account for method-specific update costs.
  • Conducted additional experiments in a fast-stream setting.
  • Compared ECO-TTA with full adaptation to assess the benefits of batch-level adaptation filtering.
  • Clarified how Greedy scheduling operates under both budget protocols.

We hope these explanations and new experiments have clarified the contributions and strengthened our submission.

Please do not hesitate to reach out if there are any further clarifications or analyses we can provide.

Thank you for your time and careful consideration!

Best, Authors

Reviewer_DgheOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ECO-TTA, a framework for allocating a limited adaptation budget during online test-time adaptation. At each batch, it can skip adaptation, perform one or multiple update steps, or reset the model. It estimates adaptation value using a label-free disagreement proxy and outcomes from similar historical batches, while a dynamic shadow price regulates spending. Experiments across several base methods and distribution shifts report improvements over Uniform and Greedy scheduling.

Critical strengths
  1. The paper address a practical problem of budget-limited adaptation with an inspiring framework.
  2. The paper evaluates multiple base methods, distribution shifts, and budget settings, with component ablations and hyperparameter studies.
Critical weaknesses
  1. Some experimental details are insufficient or unclear.
  • In main results (Tab 1,2), only results under budget are provided, while the compared baseline with limited budget is only Uniform, which is weak and naive. Improvements upon Uniform may not be persuasive. The results of original method (without budget limits) are also not provided, making it unclear how the budget-limited setting affects the performance.
  • In ablation study, what does Base. mean? Meanwhile, when using Proxy or multi-step action and dynamic price update, the results are worse than Base. Why?
  1. The action-value estimator relies on assumptions that are insufficiently validated. Similar historical batch statistics do not necessarily imply similar gains, since the model is evolving. Also, in Eq(7) the dynamic prices indicate that when accumulated updates exceed the budget, future updates will be penalized. Could this lead to lack of update on potential future data that may be more informative?
Questions for authors
  1. Please respond to the points raised in Weaknesses
  2. Could the authors provide a case example to demonstrate how the proposed method outperform the traditional setting (as shown in Figure 1)? For example, in what cases (or when processing which kinds of images) the update is multi-step or skipped? Why?
  3. Could the authors validate the exponential multi-step gain model against actual gains over successive update steps, including cases where further updates become harmful?
5 comments
AuthorsOpenReview ↗

Response to Dghe (part I)

Thank you for your careful review and for recognizing ECO-TTA as an inspiring framework for the practical problem of budget-limited adaptation. We also appreciate your positive assessment of the breadth of our evaluation. Your constructive suggestions help us strengthen the experimental comparisons and further validate the action-value estimator. We address each concern below with clarifications and additional results.


W1.1: The main results (Tab 1,2) compare only against Uniform scheduling. Could the authors include other scheduling baselines and the original methods without budget limits? It's unclear how the budget-limited setting affects the performance.

A1: Thank you for this suggestion. Due to space constraints, Tables 1 and 2 include only Uniform as the scheduling baseline; comparisons with Greedy are provided in Appendix Tables 9–11. R-Table 5 summarizes these results: ECO-TTA outperforms both schedulers across all six base methods under the same budget.

R-Table 6 quantifies the effect of budget constraints relative to full adaptation (r=1.0r=1.0), where the original methods update every batch. Progressively reducing rr from 1.0 to 0.1 consistently lowers accuracy for both methods across all reported settings. Table 4 of the main paper further shows that ECO-TTA can outperform full adaptation using only 10% of its adaptation budget. We have added cross-references to all these comparisons in Section 4.2 of the revised manuscript.

R-Table 5. Accuracy (%) on ImageNet-C under label shifts with a global budget of r=0.5r=0.5.

TTA methodUniformGreedyECO-TTA
Tent50.9250.7461.87
SAR55.1854.7964.18
ActMAD0.710.269.02
CMF77.4375.8078.75
ReCAP63.4262.2666.22
ZeroSiam57.1059.6364.29

R-Table 6. Accuracy (%) with Uniform scheduling across budget ratios. Bold indicates the best result within each method and setting.

TTA methodBudget ratioLS: NoiseLS: BlurLS: WeatherLS: DigitalLS: Avg.BS1: NoiseBS1: BlurBS1: WeatherBS1: DigitalBS1: Avg.Mixed shift: Avg.
ReCAP0.143.7950.7063.5762.3855.8627.4141.5055.4554.8945.9756.20
ReCAP0.248.5054.7767.3665.4759.7334.3146.3560.2759.1351.0657.87
ReCAP0.553.1958.7770.3668.7963.4244.5453.5466.0765.0258.1459.35
ReCAP1.056.2060.2271.2069.5364.8355.7062.2672.3870.1965.7659.44
ZeroSiam0.131.3741.2254.4154.3946.2827.0438.0350.7351.1942.7350.46
ZeroSiam0.238.4346.6959.4458.0351.4632.6342.0555.7554.8747.2453.03
ZeroSiam0.545.2552.5064.5963.0857.1040.6548.4561.6259.4353.3355.13
ZeroSiam1.052.7860.1371.1669.6764.1552.4759.8970.9169.5863.9357.12

W1.2: What does Base. mean in the ablation study, and why do some variants perform worse than Base.?

A2: Sorry for the confusion. Base. denotes ReCAP with Uniform scheduling under the same budget. R-Table 7 clarifies the configurations in the ablation study. Some variants perform worse than Uniform because the alternatives used to replace key components fail to provide reliable guidance for budget allocation.

When Proxy is disabled, prediction entropy serves as the alternative gain signal. Because entropy reduction does not necessarily imply improved accuracy, it can misdirect budget allocation even with multi-step actions and dynamic pricing, resulting in performance below Uniform.

When dynamic pricing is disabled, we use a fixed price at median λ=0.5\lambda=0.5. This price cannot adjust to actual budget consumption and may reject useful updates, resulting in performance below Base. We have clarified these configurations and explanations in the revised manuscript (Section 5.3).

R-Table 7. Component ablations with ReCAP under label shifts and global budgets.

Configurationr=0.1r=0.1r=0.5r=0.5
Base.: ReCAP + Uniform55.963.4
Entropy + multi-step actions + dynamic pricing53.962.4
Proxy + single-step actions + fixed pricing55.163.1
Proxy + multi-step actions + fixed pricing56.963.5
Proxy + single-step actions + dynamic pricing59.565.4
Full ECO-TTA61.666.2
AuthorsOpenReview ↗

Response to Dghe (part II)

W2.1: Similar historical batch statistics do not necessarily imply similar gains, since the model is evolving.

A3: Thank you for this valuable question. We agree that model evolution can make older gain records less informative. ECO-TTA therefore combines similarity-weighted retrieval with a FIFO memory that retains the most recent records, rather than the entire history.

R-Table 8 shows that, with the default memory, predicted and true gains retain Spearman correlations of 0.835 and 0.857 in the final third of the stream. This indicates that recent history preserves useful gain information despite model evolution. Across all tested capacities (K=10K=10–200200), FIFO improves both correlation and full-stream accuracy over retaining all records. We have included this analysis and clarified the role of recent memory in the revised manuscript.

R-Table 8. Memory retention on a long mixed-shift stream with ReCAP. Spearman is evaluated over the final third of the stream; accuracy is evaluated over the full stream.

Memory strategyr=0.1r=0.1: Spearmanr=0.1r=0.1: Accuracyr=0.5r=0.5: Spearmanr=0.5r=0.5: Accuracy
Retain all records0.70157.170.71458.68
FIFO, K=10K=100.81257.790.83259.23
FIFO, K=50K=50 (default)0.83559.040.85761.41
FIFO, K=100K=1000.84359.220.87561.53
FIFO, K=200K=2000.84859.290.88361.56

W2.2: Future updates may be penalized by dynamic prices. Could this suppress updates on informative future batches?

A4: Thank you for this question. Dynamic pricing balances adaptation gains and costs: future updates can still be selected when their estimated gains outweigh the price penalty. To directly assess potential suppression, we measure price-induced update rates among budget-feasible high-gain batches, defined post hoc as the top 30% by true accuracy gain, over the first, middle, and final thirds of the stream. R-Table 9 shows a final-third update rate of 93.85%, indicating that most informative batches are not suppressed by the price penalty.

R-Table 9. Price-induced update rates among budget-feasible high-gain batches, evaluated with ReCAP under mixed shifts.

Stream segmentPrice-induced update rate (%) ↑
First third95.93
Middle third94.12
Final third93.85

Q2: Could the authors provide a case example to demonstrate how the proposed method outperform the traditional setting (as shown in Figure 1)?

A5: Appendix Fig. 6 visualizes the adaptation process under different schedulers. Uniform updates at fixed intervals, including updates with near-zero gains. Greedy updates continuously during the first half of the stream, including some negative-gain updates, and then exhausts its budget. ECO-TTA allocates updates according to their predicted value, preserving resources for later opportunities. For example, around batches 800–1200, it continues performing positive-gain updates after Greedy has stopped adapting.

R-Table 10 quantifies this behavior: ECO-TTA performs an average of 2.05 steps on selected beneficial batches and skips 97.74% of harmful batches, exceeding both baselines on these measures. We have highlighted Fig. 6 and added this quantitative analysis in Appendix D.3 of the revised manuscript.

R-Table 10. Allocation behavior with ReCAP under label shifts and a global budget of r=0.5r=0.5.

SchedulerMean gain per action (%)Mean steps on beneficial batchesHarmful-batch skip rate (%)
Uniform1.141.0048.71
Greedy1.031.0049.62
ECO-TTA1.772.0597.74

Q3: Could the authors validate the exponential multi-step gain model against actual gains over successive update steps, including cases where further updates become harmful?

A6: Thank you for this constructive suggestion. We compare predicted cumulative gains with true accuracy gains for executed actions with four steps as the configured maximum. The multi-step gain model accurately predicts cumulative accuracy gains for executed actions of 1–4 steps, with an average MAE of 1.46-2.09% (R-Table 11). We also report action-selection frequencies and harmful-action rates, defining an action as harmful if any update step reduces accuracy relative to the preceding step. Among executed four-step actions, 1.38% contain such harmful step (R-Table 12), indicating that stepwise accuracy degradation is uncommon among these selected actions. We have included this analysis in the revised manuscript (Appendix C.4).

R-Table 11. MAE between predicted cumulative gains and true accuracy gains for executed actions (%).

Update stepsNoiseBlurWeatherDigitalAvg.
11.581.901.570.821.46
21.652.121.431.211.60
31.892.091.991.711.92
41.992.362.051.922.09

R-Table 12. Selection frequencies and harmful-action rates by action length.

Update stepsSelection frequency (%)Harmful-action rate (%)
140.590.63
228.251.03
316.451.22
414.701.38
AuthorsOpenReview ↗

Look forward to your post-rebuttal feedback!

Dear Reviewer Dghe,

Thank you again for your constructive suggestions on the experimental comparisons and reliability of ECO-TTA’s gain estimates. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need.

We would like to briefly summarize the additional evidence provided in our rebuttal:

  • Consolidated comparisons with Uniform, Greedy, and full adaptation.
  • Conducted additional experiments to quantify how tighter adaptation budgets affect the original TTA methods.
  • Clarified the ablation baseline and replacement configurations.
  • Conducted additional experiments to evaluate historical gain reliability under model evolution.
  • Examined the effect of dynamic pricing on informative batches.
  • Supplemented the allocation visualization with quantitative statistics.
  • Conducted additional experiments to validate cumulative gain predictions.

We hope these results clarify the benefits of value-aware scheduling and provide stronger empirical support for the historical and multi-step gain estimates.

Please do not hesitate to reach out if there are any further clarifications or analyses we can provide.

Thank you for your time and thoughtful review!

Best, Authors

Reviewer_TPCDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the problem of Test-Time Adaptation (TTA) under limited computational budget and proposes ECO-TTA, an economics-inspired framework for dynamically allocating adaptation resources across incoming test batches. Unlike traditional TTA methods, which update models at a fixed frequency, this approach models TTA as an online investment problem. Based on the expected returns and computational costs of different adaptation operations, it dynamically determines whether to perform model updates and the number of update steps, thereby improving the utilization efficiency of limited computational resources.

Specifically, ECO-TTA comprises two main components. First, the authors propose a historical experience-based adaptation benefit estimator that constructs an accuracy-gain proxy metric—which does not require ground-truth labels—by performing a closed-form approximation of the probability of prediction inconsistency under dropout perturbations. This metric is then combined with the statistical similarity of features across historical test batches to predict the benefit of the current adaptation operation, while the diminishing returns hypothesis is used to estimate the effectiveness of multi-step updates. Second, the authors design a budget controller based on dynamic shadow prices, which dynamically adjusts the computational cost penalty according to the deviation between actual budget consumption and the preset consumption rate, and selects the action with the maximum expected net gain via a Lagrangian utility criterion.

The authors conducted experiments on the ImageNet datasets, covering various distribution drift scenarios, computational budget settings, model architectures, and six basic TTA methods. The experimental results show that, compared to uniform allocation and greedy scheduling strategies, ECO-TTA can allocate limited adaptation budgets more effectively, improve the model’s classification accuracy under distribution drift, and achieve better performance in some experiments with computational overhead significantly lower than that of full adaptation.

Critical strengths

1.The paper uses an economics-inspired approach to improve TTA; the idea is highly innovative. The paper is very well structured, allowing readers to intuitively understand the method, and it convinces readers through rigorous theory and extensive experimental results. 2.presenting a comprehensive methodology and theoretical framework, the author also conducted extensive sensitivity analyses, which added to the persuasiveness of the study.

Critical weaknesses

The authors use “disagreement” as an approximation of “accuracy”, but there is still a certain gap between the two,the authors do not provide a theoretical explanation for this. Additionally, the union bound provided by the authors in Proposition 1 is relatively loose.

Questions for authors

1.Can this method be extended and implemented for scenarios where T is unknown or has no upper bound? 2.Can the accuracy of the Gaussian approximation used be verified in the sensitivity analysis section? 3.If the author uses the same batch for both adaptation and estimating the gain, could this lead to estimation bias?

5 comments
AuthorsOpenReview ↗

Response to TPCD (part I)

Thank you for your careful review and positive assessment of our work. We especially appreciate your recognition of ECO-TTA’s highly innovative idea, clear presentation, and rigorous theoretical and experimental analyses. Your constructive suggestions help us clarify the theoretical foundations of our gain estimator and further evaluate its reliability and practical applicability. We address your concerns below with additional explanations and supplementary experimental results.


W1: The authors use “disagreement” as an approximation of “accuracy”, suggest to provide a theoretical explanation for this.

A1: Thank you for your constructive suggestion. The relationship between disagreement and accuracy is supported by the following proposition:

Proposition (Generalization–Disagreement Equality). Let PP and QQ denote the source and target distributions, respectively, allowing P≠QP\neq Q. Let Π\Pi be a distribution over classifiers induced by randomized training or stochastic perturbations. Let f,f′∼i.i.d.Πf,f'\overset{\mathrm{i.i.d.}}{\sim}\Pi, with their sampling randomness independent of the target example (X,Y)∼Q(X,Y)\sim Q. Define the ensemble prediction probabilities as πk(x)=Pr⁡f∼Π[f(x)=k],k=1,…,C.\pi_k(x)=\Pr_{f\sim\Pi}[f(x)=k], k=1,\ldots,C. With a well-calibrated ensemble on the target distribution QQ, in the class-wise sense that $ E_Q \left[\mathbf{1}_{\lbrace Y=k\rbrace} \mid \pi_k(X)\right] =\pi_k(X), \forall k. $ Then the ensemble-average error rate equals the average pairwise disagreement:

E_{f\sim\Pi} \Pr_{(X,Y)\sim Q}[f(X)\neq Y] }_{\overline{R}_Q}=\underbrace{ E_{f,f'\sim\Pi} \Pr_{X\sim Q_X}[f(X)\neq f'(X)] }_{\overline{D}_Q},\quad i.e. \quad E_{f\sim\Pi}[\operatorname{Acc}_Q(f)] =1-\overline{D}_Q,$$ where $Q_X$ is the input marginal of $Q$. **A Sketch Proof:** 1. By the tower property and class-wise calibration on $Q$: \$ E\_Q[\pi\_k(X)\mathbf{1}\_{\lbrace Y=k\rbrace}] =E\_Q\left[\pi\_k(X)E\_Q[\mathbf{1}\_{\lbrace Y=k\rbrace}\mid \pi\_k(X)]\right] =E\_Q[\pi\_k(X)^2]. \$ Thus, averaging over classifier randomness gives: \$ \overline{R}\_Q =1-\sum\_{k=1}^{C}E\_Q[\pi\_k(X)\mathbf{1}\_{\lbrace Y=k\rbrace}] =1-E\_{Q\_X}\left[\sum\_{k=1}^{C}\pi\_k(X)^2\right]. \$ 2. For any fixed input $x$, the independent draws $f,f'\sim\Pi$ yield: \$ \Pr[f(x)=f'(x)] =\sum\_{k=1}^{C}\Pr[f(x)=k]\Pr[f'(x)=k] =\sum\_{k=1}^{C}\pi\_k(x)^2. \$ 3. Taking the complement and averaging over target inputs gives: \$ \overline{D}\_Q =1-E\_{Q\_X}\left[\sum\_{k=1}^{C}\pi\_k(X)^2\right] =\overline{R}\_Q. \$ Consequently, \$ E\_{f\sim\Pi}[\operatorname{Acc}\_Q(f)] =1-\overline{D}\_Q. \$ This establishes a theoretical connection between disagreement and accuracy. Motivated by this connection, we derive a closed-form approximation that avoids explicit dropout sampling. We have provided this explanation in Section 3.2 and included the proposition and the complete proof in Appendix A.1. --- > **W2: The union bound provided in the proof of Proposition 1 is relatively loose.** **A2:** We appreciate your valuable comment. The union bound can be loose when multiple class-exceedance events overlap. To assess its empirical impact, we compare the union-bound estimate against Monte Carlo estimates using 4,096 dropouts per sample (R-Table 1). Multi-class exceedance occurs in only about 3% dropout samples. The union-bound estimate exhibits only 1.7-2.0% relative overestimated value, while maintaining a strong correlation with Monte Carlo disagreement (Pearson: 0.916–0.942). These results support its use for tracking disagreement despite its looseness. We have included this analysis in the revised Section 5.1. **R-Table 1: Empirical evaluation of the union-bound estimation.** | TTA method | Multi-class exceedance rate (%) | Relative overestimation (%) | Pearson correlation | | :--- | :---: | :---: | :---: | | Tent | 3.20 | 2.01 | 0.921 | | CMF | 2.13 | 1.76 | 0.942 | | ReCAP | 3.05 | 1.93 | 0.916 |
AuthorsOpenReview ↗

Response to TPCD (part II)

Q1: Can this method be extended and implemented for scenarios where TT is unknown or has no upper bound?

A3: Yes. ECO-TTA uses TT only to set the controller’s target spending rate ρB=B/T\rho_B=B/T. Specifying ρB\rho_B directly therefore enables operation on streams of unknown or unbounded length. This rate can be specified by ρB=B/W\rho_B=B/W or estimated online as ρ^B=(Tbatch−Tforward)/Tadapt\widehat{\rho}_B=(T_{batch}-T_{forward})/T_{adapt}, i.e., the available adaptation time per batch divided by the estimated time per adaptation step.

We have also added experiments in which neither setting provides TT to the scheduler (Appendix C.3, Table 17). As summarized in R-Table 2, it shows comparable accuracy under stable resources. When resources vary, online estimation adjusts the target rate accordingly, improving accuracy by +0.49%. We have clarified both ways of setting ρ^B\widehat{\rho}_B for streams of unknown or unbounded length in the revised manuscript.

R-Table 2. Fixed versus online estimated rates using ReCAP under label shift with sliding-window budgets.

Resource availabilityTarget-rate settingAccuracy (%) ↑
Stable (r=0.2r=0.2)Fixed ρB=0.2\rho_B=0.262.42
Stable (r=0.2r=0.2)Estimated online62.35
Varying (r=0.2→0.3→0.1r=0.2\rightarrow0.3\rightarrow0.1)Fixed ρB=0.2\rho_B=0.262.42
Varying (r=0.2→0.3→0.1r=0.2\rightarrow0.3\rightarrow0.1)Estimated online62.91

Q2: Can the accuracy of the Gaussian approximation used be verified?

A4: We appreciate this suggestion. We validate the approximation by comparing 1) Empirical logit-gap densities estimated from dropout samples using Kernel Density Estimation (KDE); and 2) Gaussian distributions parameterized by the theoretical means and variances derived in the paper. As shown in R-Table 3, the mean KL divergence is 0.0012 at the default keep probability and 0.0050 averaged across the tested probabilities, indicating a close fit to the empirical densities. We have included this validation in Section 5 of the revised manuscript.

R-Table 3. Mean KL divergence between the KDE-estimated logit-gap densities and Gaussian approximations.

Dropout keep probability0.10.50.7Avg.
Mean KL divergence ↓0.01140.00120.00230.0050

Q3: If the author uses the same batch for both adaptation and estimating the gain, could this lead to estimation bias?

A5: Thank you for this question. ECO-TTA uses recorded adaptation gains only to inform decisions on subsequent batches. To assess potential bias, we keep the adaptation process unchanged and compare gain estimates on the adaptation batch and a disjoint held-out batch from the same distribution, using the same pair of pre- and post-update models. R-Table 4 reports mean absolute differences below 3% for all three methods, suggesting the limited effect on estimation bias. We have included this analysis in the revised manuscript (Appendix C.4).

R-Table 4. Mean absolute differences (%) between gains estimated on adaptation and held-out batches under label shifts.

TTA MethodNoiseBlurWeatherDigitalAvg.
TENT1.682.012.141.881.94
ReCAP1.482.081.340.921.45
ZeroSiam1.561.972.171.811.90
AuthorsOpenReview ↗

Look forward to your post-rebuttal feedback!

Dear Reviewer TPCD,

Thank you again for your insightful suggestions on the theoretical foundation and reliability of our gain estimator. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need.

In our rebuttal, we provided the following explanations and additional analyses:

  • Provided a theoretical foundation for disagreement-based accuracy estimation.
  • Conducted additional experiments to validate tightness of the union bound.
  • Compared the Gaussian approximation with the dropout-perturbed logit gaps.
  • Conducted additional experiments to evaluate scheduling without a known stream length.
  • Examined potential batch-reuse bias by comparing gain estimates.

We hope these additions clarify the theoretical motivation, approximation quality, and empirical reliability of the estimator, as well as its applicability to streams of unknown length.

Please do not hesitate to reach out if there are any further clarifications or analyses we can provide.

Thank you for your time and valuable feedback!

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.