- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DTS-META extends GOLD's classifier-sensitive subspace adaptation with a fast pathway with a prototype-residual covariance based slow-fast paradigm and batch-conditioned parameter generation and a first-order support-query update. The slow pathway controls the overall feature abstraction, while the fast pathway quantifies current-batch-specific feature corrections. It evaluate three classifier benchmarks in standard, recurring, gradual and long-horizon streams as well as in a semantic segmentation setting. The proposed approach is not a new formulation of CTTA but an augmentation of the feature-subspace adaptation paradigm.
Critical strengths
-
The amount of classification improvement is small but consistent. DTS-META consistently outperforms GOLD for all three standard Evaluation benchmarks, with absolute error reductions of between 0.59 and 0.72. The five-seed results give this bit of progress more credence than one would have with a single best run -which is critical to a positive evaluation since individual improvements were not large.
-
The paper has fairly useful controls for its main additions. Both Forward and Full have the same Meta Net architecture, so there is a fairly good picture of the episodic update. In addition, the separate refresh-schedule experiment does not compare architectures with different capacities, but keeps the full model the same. These controls suggest that the optimization and schedule learning does something in excess of just adding an additional adapter.
Critical weaknesses
-
Address the discrepancy in the previously published results of GOLD. The original Table 1 of GOLD reports a 14.1% error for GOLD on CIFAR-10-C, while this paper reports 15.04% for GOLD and 14.45% for DTS-META. Therefore, DTS-META is better than the reproduced baseline but does not beat the results originally published in GOLD in terms of numbers; the reproduction gap between the results and the original is around 0.94, which is larger than the reported 0.59.
-
Episodic meta-learning gains only 0.27, 0.21, 0.23 over Forward (with the same Meta Net structure) in Table 3, and both default dual-timescale schedules only achieve 0.28 gains over frequently refreshing both pathways in Table 10. Though these are good controls, each table does not report an uncertainty estimate for these differences.
Questions for authors
-
The reason for the difference between GOLD's initial CIFAR-10-C result and the one shown here could be variations in checkpoint selection, order of the samples, hyperparameters, augmentations, and when predictions are made. It will be interesting to look into whether adjusting any of these factors affects the ranking relative to other methods.
-
For the comparison of Forward vs. Full, and for the refresh-schedule ablation, the authors should give results for each seed. A matched comparison showing that the prototype-residual covariance is likely to exceed the raw-feature covariance independently of extra capacity would also be helpful.
4 comments
Thank you for your time and for the additional experiments. My concerns regarding uncertainty and controlled comparisons have been largely addressed. The five-seed comparison between Forward/Full and Raw-cov, as well as the ablation study on the refresh schedule, directly respond to my requests.
However, the discrepancy regarding GOLD has only been partially resolved. While using the official implementation and reporting paired results does strengthen the comparison against your reproduced baseline, it does not explain why GOLD achieved a result of 15.04% instead of the published 14.1%.
Response to Reviewer 44sJ
Thank you for your follow-up response and for clarifying the remaining concern. We agree that the paired comparison against our reproduced GOLD baseline does not explain the difference from the published 14.1% result. We have not identified the cause of this discrepancy.
We checked our CIFAR-10-C configuration and source-model loading path against the official release. The source model is the Standard WideResNet-28-10 selected by GOLD’s loading code. We retained the released augmentation, prediction timing, and corruption order. Evaluation used batch size 200, with all 15 corruption types processed sequentially at severity 5. After loading the warmup checkpoint, we verified that Adam used learning rate 0.001, betas (0.9, 0.999), and weight decay 0. For the reported GOLD baseline, we retained the released architecture and hyperparameters.
We also repeated the evaluation at seed 1 after regenerating our locally computed source warmup checkpoint and class prototypes. These additional checks yielded errors close to 15.0% and did not recover the published result.
Across the five matched seeds, GOLD yields 15.04 ± 0.10% error and DTS-Meta yields 14.45 ± 0.09%, a mean reduction of 0.59 percentage points. This supports an improvement over our reproduced baseline under the recorded evaluation conditions. Our CIFAR-10-C error remains above GOLD’s published result.
We will report GOLD’s published result and our reproduced result separately in the revised manuscript. The reported 0.59-percentage-point reduction is based on paired runs against reproduced GOLD under the same evaluation protocol.
Dear Reviewer 44sJ,
Thank you very much for your active engagement in the discussion and for acknowledging our previous responses. We have made every effort to address your remaining concern in our latest reply. As the discussion period is nearing its conclusion, could we kindly inquire if our follow-up explanation has resolved this final question? Thanks for your time and efforts in reviewing our work, and we sincerely look forward to your reply.
Best regards,
Authors
Response to Reviewer 44sJ
Thank you for taking the time to review our paper and providing valuable feedback. We appreciate your recognition of the consistent improvements and the controlled comparisons of our design choices. In response to your Weaknesses (W) and Questions (Q), we provide the following Explanations (E).
W1 & Q1: GOLD reproduction and the published-result discrepancy
E1: We used GOLD's official implementation and released CIFAR-10-C configuration without additional hyperparameter tuning or structural changes. The relevant settings are:
Table G1 of our Common Response reports the individual results for seeds 1–5 and their paired uncertainty. Reproduced GOLD has an error of 15.04 ± 0.10% on CIFAR-10-C, compared with 14.1% in the original paper. DTS-META reaches 14.45 ± 0.09% and has lower error in every matched run. The mean reduction is 0.59 pp. We will distinguish the originally reported and reproduced GOLD values in the revised manuscript.
More broadly, all baselines in our benchmark were faithfully reproduced strictly following their officially released codebases and default hyperparameters without modification. Across all evaluated benchmarks, our reproduced results closely align with those reported in the literature, ensuring that all methods are evaluated on an identical, fair, and rigorous playing field.
W2 & Q2: Seed-Wise Uncertainty Estimates and Controlled Subspace Comparisons
E2: Table G2 of our Common Response summarizes the Forward, Raw-cov, and Full results. Forward omits the temporary inner update; Raw-cov replaces only the covariance used to estimate the fast basis. Both retain Full's parameter capacity. Table 1 below provides their individual seed errors. The matched Full errors are already reported in Table G1 of the Common Response.
Table 1: Individual-seed error rates (%) for Forward, Raw-cov, and Full.
Full has lower error than each control in every seed on all three benchmarks. Table 2 gives the paired gains and their pointwise 95% confidence intervals. There are five pairs per dataset; the 15 dataset–seed comparisons are not pooled into one test.
Table 2: Paired error reductions (pp) relative to the two controls.
These comparisons support the temporary support update and residual-covariance basis estimation at fixed parameter capacity in the tested settings. They do not attribute the effects to additional parameters or imply that their contributions are independent.
Table G3 of our Common Response reports the five-seed results for the ImageNet-C CDC schedules, and Table 3 below provides the paired comparisons. The default schedule, Ts=50,Tf=5, has lower error than Both Frequent, Both Infrequent, and Inverted in every matched seed.
Table 3: Paired gains (pp) from Default on ImageNet-C under CDC.
This consistent ordering supports the benefit of timescale coordination across the tested random stream initializations.