acceptodds
Under review as a conference paper at ICLR 2027

Evolving Replay Hulls for Geometrically Supported Continual Domain Shift Learning

Abstract

Unsupervised continual domain shift learning (UCDSL) requires continual adaptation to sequentially arriving unlabeled domains while retaining previously acquired knowledge and generalizing to future domains. However, its reliance on self-training makes continual adaptation vulnerable to unreliable pseudo-labels, whose errors can accumulate across domain stages. We propose a reliable UCDSL framework that estimates pseudo-label confidence from class-wise replay geometry. Replay samples of each class define a convex hull in representation space, and target reliability is measured by relative distances to class-wise hulls. The framework maintains informative replay regions throughout continual adaptation, allowing this confidence criterion to remain effective as new domains are encountered. We theoretically show how the coverage of class-wise replay hulls and sample distances to them relate to classification risk on unseen domains. Empirically, the proposed framework improves adaptation, generalization, and forgetting alleviation across multiple benchmark datasets, while achieving better predictive calibration than existing UCDSL methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Continual Learning, Unsupervised Domain Adaptation, Domain Generalization, Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9s2rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACE (Target Reliability Assessment via Convex Evidence), a framework for unsupervised continual domain shift learning (UCDSL) that aims to improve pseudo-label reliability during sequential adaptation to unlabeled target domains. The key idea is to estimate pseudo-label confidence using class-wise convex hulls constructed from replay exemplars, rather than relying solely on classifier predictions. Specifically, TRACE measures the relative distances between target representations and the convex hulls of the predicted and competing classes. To maintain informative replay geometry across domains, the framework incorporates geometry-preserving exemplar selection and a hull-matching objective, together with an EMA-based reference encoder for stable pseudo-label generation. The paper also provides a theoretical analysis relating replay-hull coverage and sample residuals to classification risk on unseen domains under certain assumptions. Experiments on Digits-5, PACS, and DomainNet demonstrate improvements in overall classification performance across adaptation, generalization, and forgetting metrics, as well as better predictive calibration measured by ECE, Brier score, and NLL compared with existing UCDSL baselines.

Critical strengths
  1. A well-motivated geometric perspective on pseudo-label reliability. The paper proposes using class-wise replay convex hulls to estimate pseudo-label confidence in UCDSL. Unlike prediction-based confidence measures, this approach considers the relative distances between target representations and class-wise replay regions. The framework also introduces geometry-preserving replay selection and hull matching to maintain these regions across domain shifts. This provides an alternative way to assess pseudo-label reliability using information retained from previous domains.

  2. Empirical improvements across multiple evaluation metrics. The experiments cover three benchmark datasets and evaluate target domain adaptation, generalization, forgetting alleviation, and predictive calibration. TRACE achieves the highest overall classification accuracy on all three datasets and lower ECE, Brier score, and NLL than the compared baselines. These results provide empirical evidence for the effectiveness of the proposed framework across several evaluation criteria.

Critical weaknesses
  1. Insufficient comparison with alternative geometric confidence estimation methods. Although TRACE achieves improvements over existing UCDSL methods, the experiments do not sufficiently establish whether the convex-hull-based confidence measure provides advantages over simpler geometric alternatives, such as prototype distances, nearest-exemplar distances, or Mahalanobis distances. Since geometric confidence estimation is a central component of the proposed framework, these comparisons are necessary to better isolate its contribution and determine whether the observed improvements specifically result from the use of convex hulls.

  2. Limited evaluation of computational efficiency and scalability. The proposed confidence estimation requires computing distances between target representations and class-wise replay convex hulls, introducing additional computational overhead. Although the authors adopt PCA dimensionality reduction and use 30 Frank–Wolfe iterations to approximate these distances, the paper does not provide sufficient runtime analysis or comparisons of computational costs. Furthermore, the experiments use a fixed replay buffer of 200 samples, and the DomainNet experiments are conducted on a 10-class subset. It therefore remains unclear how the method scales to larger numbers of classes or different memory budgets. Additional runtime comparisons and scalability experiments would help establish the practical applicability of the approach.

Questions for authors
  1. Comparison with alternative geometric confidence measures. Could the authors provide comparisons between the proposed convex-hull-based confidence measure and simpler geometric alternatives, such as prototype-based distances, nearest-exemplar distances, or Mahalanobis distances, under the same training framework? Such comparisons would help clarify whether the performance improvements can be attributed specifically to the convex-hull formulation.

  2. Computational overhead. Could the authors report the additional training time or computational overhead introduced by replay-hull confidence estimation and hull matching compared with the existing UCDSL baselines? This would help assess the computational cost associated with the reported performance improvements.

  3. Scalability with class number and replay memory size. How does the proposed method perform as the number of classes or the replay buffer size changes? Since the current experiments use a fixed buffer of 200 samples and a 10-class subset of DomainNet, results under different settings would help clarify the scalability of the approach, particularly when fewer replay exemplars are available per class.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9s2r — Weakness #1 / Question #1

Thank you for suggesting these comparisons. We conducted two complementary experiments to assess the contribution of hull-based confidence.

Experiment1.

We replaced only the distance measure within TRACE, keeping the remaining training procedure unchanged. Given the substantial training time required (see our response to Reviewer Shu5’s Weakness 2 & Question 3), we evaluated four PACS domain orders (1, 4, 6, 9), covering all four source domains, and report their average results below. Once all experiments are complete, we will revise the manuscript to include the full results and the analysis presented below.

Distance measureTDATDGFAAllCost per sample
Prototype88.3272.2688.1782.91O(Cr)O(Cr)
Nearest exemplar88.4872.0688.6283.05O(Cmr)O(Cmr)
Mahalanobis88.9672.9888.6483.52O(r2+Cr)O(r^2+Cr)
Convex hull (TRACE)89.0872.4688.8883.48O(TCmr)O(TCmr)

Hull-based confidence achieves higher mean TDA and FA than all alternatives and comparable overall performance to Mahalanobis. We acknowledge that iterative hull projection incurs a higher distance-computation cost. However, it operates in a low-dimensional PCA space (r=C)(r=C) with a limited number of replay anchors and a fixed iteration budget (T=30)(T=30), keeping the computation manageable in our setting. Here, CC, rr, mm, and TT denote the number of classes, PCA dimension, anchors per class, and Frank-Wolfe iterations, respectively. Mahalanobis additionally requires O(Cmr2+r3)O(Cmr^2+r^3) preparation per reference update, which grows with the number of classes under our choice of r=Cr=C.

Experiment2.

Using the same protocol as in our original calibration evaluation, we compared all four distance measures across 10 PACS orders and three seeds at every target stage.

Distance measureECEBrierNLLAURC
Prototype0.13170.14260.43870.1088
Nearest exemplar0.12230.14280.43260.1098
Mahalanobis0.13610.14410.44530.1128
Convex hull (TRACE)0.11020.14180.42450.1081

Hull-based confidence achieves the lowest values on all four metrics. Together, these experiments show that the hull formulation contributes beyond downstream accuracy by providing better-calibrated confidence. This directly supports its use for confidence-based sample selection and weighting in TRACE, while retaining the connection to our hull-residual theoretical analysis.

AuthorsOpenReview ↗

Response to Reviewer 9s2r — Weakness #2 / Question #2, #3

We agree that the manuscript should discuss TRACE’s computational cost and scalability in more detail. We designed three experiments to measure the cost of replay-hull operations, assess scalability with varing replay budget, and compare end-to-end training time with existing UCDSL baselines.

Experiment3.

Cost of replay-hull operations (Appendix C.7, Table 13). To quantify how much each operation contributes to training time, we profiled TRACE with the default 200-sample buffer.

ComponentTime (s)Share
Replay-hull distance5.60.7%
Hull matching162.820.1%
Entire target stage810.1100%

Replay-hull distance computation takes up only a small fraction of training time. The larger cost of hull matching reflects its role as a training loss, whose measured runtime includes encoder forward pass over queued target and replay images and the associated backward pass.

Experiment4.

Scalability with replay budget (Appendix C.8, Tables 14–15). To test the effect of having fewer exemplars per class while holding the task fixed, we varied the replay budget across 35, 70, 140, 200, and 700 samples on seven-class PACS, using all 10 domain orders. We assessed scalability across these budgets in terms of both predictive performance and computational cost.

BudgetTDATDGFAAllΔAll
3586.7969.5285.5580.62−1.54
7087.6370.2987.7881.90−0.27
14087.6170.1988.1281.98−0.18
20087.9170.2788.3082.160.00
70087.7770.3388.6782.26+0.10
BudgetDistance (ms/call)Matching (ms/call)Memory (MiB)
354.0122.616,076
704.3128.220,886
1404.1141.321,134
2004.6179.221,299
7004.2355.021,422

TRACE maintains comparable overall performance across a wide range of replay budgets, including substantially smaller buffers. Performance drops more noticeably at the smallest budget, with the largest decline in FA, suggesting that retention is particularly sensitive to very limited replay. As the buffer grows, distance latency stays nearly constant, while hull-matching time increases less than proportionally over the tested range.

Experiment5.

Training and inference cost compared with UCDSL baselines.

Training time (minutes/sequence)

MethodPACSDigits-5DomainNet
RaTP68171337
CoDAG104337244
MPM5769110
TRACE3519889

TRACE achieves relatively low training costs, with the shortest training times on PACS and DomainNet. On Digits-5, training takes longer primarily due to the computational cost of KNN on a larger number of samples. Nevertheless, TRACE performs adaptation and generalization within a single training process using a single prediction model. In contrast, CoDAG trains separate DA and DG models, while MPM maintains multiple domain-specific prototype classifiers, requiring additional model capacity.

Inference time (ms/sample)

MethodPACSDigits-5DomainNet
RaTP1.640.252.26
CoDAG1.530.191.99
MPM, with BiGE2.831.202.68
TRACE1.740.182.42

RaTP, CoDAG, and TRACE use a standard model forward pass at inference and have broadly similar runtimes. MPM additionally combines and refines predictions from its domain-specific prototype classifiers through BiGE, introducing graph-based computation. TRACE’s replay-hull operations are used during training and add no computation to its inference procedure.

Thank you for the detailed response and the additional experiments on computational overhead and replay budgets. These results have largely addressed my previous concerns regarding efficiency and scalability. Nevertheless, I remain uncertain about the necessity of the method’s core design. Since the entire training mechanism is built around convex hulls, their unique benefits over simpler geometric measures should be thoroughly validated. However, in the newly added replacement experiments, different distance measures yield very similar accuracy, while the calibration advantages of convex hulls are also relatively limited. Based on the current empirical evidence, it remains unclear whether the core component is necessary or offer substantial practical advantages. I would also like to see whether the concerns raised by the other reviewers have been adequately addressed.

Thank you for clarifying your remaining concern. We agree that similar average accuracy alone does not fully establish the benefits of choosing convex hulls. To investigate when this choice matters, we have added further analysis in Appendix C.9, “Comparison of Alternative Distance Measures.”

First, we compare convex hull, Mahalanobis, nearest-exemplar, and prototype distances on synthetic data with controlled replay coverage. These experiments separately examine limited numbers of replay anchors and insufficient source variation along directions that vary in the target domain.

Second, we compare convex-hull and Mahalanobis distances within TRACE across replay budgets on PACS, averaging over all 10 domain orders. By reducing the replay budget, we examine whether the effects of sample scarcity observed in the synthetic study also arise on a real-image benchmark.

Both experiments show benefits of convex hull distance under limited replay coverage. It provides more reliable confidence estimates in the synthetic study and improves overall performance on PACS, with the largest gain at the smallest replay budget. Together, these results highlight the practical advantages of convex hull confidence and support its reliability under varying replay coverage. We hope these additional analyses help address your concern about the choice of convex hulls and its practical value in TRACE.

Reviewer_Shu5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACE, a framework for unsupervised continual domain shift learning that uses class-wise replay geometry to assess pseudo-label reliability. An exponential-moving-average reference encoder provides more stable representations, while distances to predicted-class and competing-class replay hulls are used to construct confidence scores for pseudo-label supervision. The method further incorporates knowledge distillation, a directional hull-matching objective, a representation-repulsion loss, and a replay-selection strategy designed to preserve both representative and geometrically extensive samples. The paper also presents a margin-based theoretical analysis connecting replay-hull coverage to classification risk on unseen domains. Experiments on Digits-5, PACS, and DomainNet show improvements in overall continual-learning performance and several calibration metrics compared with existing UCDSL methods.

Critical strengths
  1. The paper addresses an important weakness of pseudo-label-based continual adaptation: classifier confidence alone can be unreliable under distribution shift, and incorrect pseudo-labels can affect subsequent adaptation stages. Using replay samples as class-conditional geometric evidence is a reasonable alternative to relying exclusively on softmax confidence. The framework also considers how the replay geometry should be maintained as new domains arrive.

  2. The empirical evaluation covers three benchmark datasets and multiple domain orders and random seeds. In addition to target-domain adaptation, the authors evaluate generalization to unseen domains, forgetting alleviation, and predictive calibration. The incremental ablation study suggests that replay-hull confidence is the largest contributor to the improvement in target-domain adaptation, while delayed adaptation and the other geometric components provide additional gains.

Critical weaknesses
  1. The theoretical guarantee is not clearly connected to TRACE’s confidence score. The theory relies on a small absolute residual to the correct-class hull, whereas the algorithm uses a relative distance ratio, which may remain high even when a sample is far from all hulls. Moreover, the theory uses the original representation space, while the implementation applies PCA. The authors should clarify how these differences affect the claimed guarantee.

  2. The results do not fully establish the robustness of all components. Accuracy gains are modest on Digits-5 and DomainNet, and MPM achieves better AURC on both. Ablations are limited to Digits-5, while sensitivity analysis uses a single seed. Additional ablations on PACS and DomainNet, along with uncertainty estimates across seeds and domain orders, would better support the method’s generality.

Questions for authors
  1. How does the relative-distance confidence score provide a reliable measure of absolute class-wise support? In particular, could a target sample far from all replay hulls still receive a high score if one hull is relatively closer than the others? Have the authors evaluated confidence as a function of both the relative distance ratio and the absolute residual to the nearest hull?

  2. The theoretical analysis defines replay hulls and residuals in the unprojected representation space, whereas the algorithm constructs them after PCA projection. Could the authors explain how the classification-margin guarantee transfers to the projected space? More generally, how do the assumptions in the theoretical results relate to pseudo-labels and replay samples available in the unlabeled target-domain setting?

  3. Could the authors report additional component ablations on PACS and DomainNet, as well as variability across random seeds and domain orders? It would also be useful to explain why AURC does not consistently improve over the strongest baseline despite the improvements in ECE, Brier score, and NLL.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Shu5 — Weakness #1 / Question #1, #2

Regarding relative distance ratio.

Thank you for raising important questions. We acknowledge that the ratio can be high even when the absolute residual is large if the nearest competing hull is much farther away. However, the same absolute residual can indicate different levels of support as hull sizes and relative positions vary across datasets and stages. Using the nearest competing hull as a reference gives TRACE a bounded confidence score ([0,1])([0,1]) that reflects the local replay geometry and can directly weight pseudo-label supervision.

DatasetMedian Spearman (rs)(r_s)25th–75th percentile (rs)(r_s)
PACS0.9530.613–0.978
DomainNet0.8960.827–0.957
Digits-50.9880.982–0.994

Also, we compare the rankings induced by qq and −dpred-d_{\mathrm{pred}} at initial pseudo-label generation. The correlations are generally high. The lowest values occur on PACS Sketch and DomainNet Quickdraw, where only 0.4% and 2.5% of initial pseudo-labels meet the confidence threshold, respectively. Consequently, these low-correlation cases have limited influence on the overall training process.

We provide further analysis in Appendix C.5 of the revised manuscript (Figure 8 and Table 13).

Regarding PCA space.

While PCA projection does not necessarily preserve the original classification margins exactly, the same margin-based reasoning can be extended to the projected space by accounting for the discriminative information discarded during projection. In particular, the margin distortion is bounded by

\leq |(I-UU^\top)(w_c-w_k)|_2 \cdot |(I-UU^\top)(z-\mu)|_2,$$ where $U$ is the PCA basis, $\mu$ is the mean source representation used for PCA, and $\tilde z$ is the PCA-reconstructed representation. Thus, although projection may reduce the classification margin, the underlying reasoning of the theorem remains applicable with an adjusted margin, potentially yielding a smaller certified region. To assess this connection empirically, we analyzed the alignment between the source PCA subspace and the frozen classifier's discriminative directions. Figure 9 shows that approximately $C-1$ principal components preserve nearly all classifier weight information across all three datasets. Further details are provided in Appendix C.6 and our response to Reviewer vJQE's Weakness 6. ### **Regarding theoretical assumptions and pseudo-labels.** Our theoretical results provide conditional guarantees under Assumption 1, rather than guaranteeing their validity for arbitrary pseudo-labeled replay samples. In TRACE, replay hulls are initially constructed using ground-truth source labels and subsequently updated using pseudo-labeled target samples. To encourage the preservation of label-consistent replay geometry, TRACE prioritizes high-confidence target samples for replay updates, reducing the risk of incorporating incorrectly labeled samples. While this design does not guarantee perfect label consistency, it is motivated by the conditions underlying our theoretical analysis. We further evaluated these conditions empirically across all three datasets, 10 domain orders, and three seeds. As detailed in our response to Reviewer vJQE’s Weakness 4, at least 99.88% of replay anchors and 98.33% of class-wise hulls satisfy the positive margin condition during target adaptation. These results provide empirical support that the theoretical assumptions are largely maintained throughout continual adaptation.
AuthorsOpenReview ↗

Response to Reviewer Shu5 — Weakness #2 / Question #3

Regarding sensitivity analysis with single seed.

We reran the Digits-5 sensitivity analysis across all 10 domain orders and all three random seeds and updated Appendix C.4. The results remained close to those in the submitted single-seed analysis. Averaged over the tested β\beta and γ\gamma settings, the overall score changed from 78.45% to 79.00% at τ\tau = 0.5 and from 81.76% to 81.71% at τ\tau = 0.8. The higher confidence threshold continued to yield a higher average score.

Regarding additional dataset for ablation and sensitivity analysis.

We focused the full sensitivity and ablation studies on Digits-5 because each configuration requires 30 runs per dataset (10 domain orders × 3 seeds). The incremental ablation evaluates six configurations, including full TRACE, and the sensitivity analysis evaluates 18 configurations. The table shows the mean elapsed time per run on an NVIDIA H200 and the estimated cumulative runtime for one configuration.

DatasetMean time per run (h)Estimated cumulative time for 30 runs (h)
Digits-53.52105.6
PACS0.7021.0
DomainNet2.1664.8

Evaluating all configurations from scratch on both PACS and DomainNet would require 1,440 runs. Based on the mean runtimes above, this amounts to approximately 2,060 hours of cumulative H200 runtime. Given the computational cost, we hope the reviewer will understand that it is difficult to complete the full ablation and sensitivity analysis on PACS and DomainNet within the short rebuttal period.

Regarding AURC.

MPM achieves lower AURC on Digits-5 (0.0743 vs. 0.0790) and DomainNet (0.2648 vs. 0.2913). AURC depends on both pseudo-label accuracy and where errors fall in the confidence ranking. As described in Appendix B.3, MPM generates the evaluated pseudo-labels after adapting its domain model to the current target data with SHOT. This adaptation may improve pseudo-label accuracy and ranking quality. TRACE nevertheless remains competitive with the evaluated baselines in AURC across all three datasets.

Thank you for the detailed response and the additional experiments. While the new empirical results are encouraging, I remain concerned that the relative-distance score does not necessarily reflect absolute class-wise support, leaving the gap between the theoretical guarantee and the implemented algorithm unresolved. Therefore, I will maintain my original rating.

Thank you for your continued engagement in the discussion. We would like to clarify a point that may have been misleading in our presentation.

First, our theoretical analysis is intended to motivate the use of replay-hull geometry for confidence estimation, rather than to provide a direct guarantee for the specific confidence formulation. This intention is also reflected in Section 4.5, where the theoretical results are discussed as motivation for TRACE's design choices. Accordingly, we do not claim that the proposed confidence score is mathematically equivalent to the absolute residual appearing in the risk bound.

Next, in TRACE, confidence is used to weight pseudo-label supervision, not to directly minimize the absolute residual. Therefore, the confidence score need not exactly correspond to the quantity appearing in the risk bound. We adopt a relative-distance ratio because it provides an interpretable, bounded score while avoiding dependence on a fixed absolute-distance scale, which may vary across datasets and adaptation stages.

The additional correlation analysis we provided in Appendix C.5 was intended to offer complementary empirical evidence that the two measures generally exhibit similar rankings, rather than to establish their theoretical equivalence.

We hope this clarifies that the theoretical analysis and the proposed confidence formulation serve distinct but complementary roles in TRACE.

Reviewer_vJQEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses Unsupervised Continual Domain Shift Learning (UCDSL), where a model sequentially adapts to unlabeled target domains while trying to retain performance on past domains and generalize to unseen future domains. The major novelty lies in Replay-Hull Confidence Estimation, representing reference feature as convex hulls in a low-dim space and computing target confidence based on distances regarding the hulls.

Critical strengths
  • the paper is easy to follow
  • theoretical justification is provided
Critical weaknesses
  • the authors should better justify the practical motivation of UCDSL over CTTA; for now, it feels like a rigged setting with less practical meaning.
  • the vertex selection, hull matching, and repulsion losses—which occupy a large portion of the methodology section—collectively add marginal performance gain.
  • too many hyper-parameters to select and some can be sensitive
  • assumption 1 seems strong and left unverified.
  • under severe domain shifts, W1W_1 in feature space can be very large unless the encoder is 1-Lipschitz, making the theoretical risk bound loose in practice.
  • the low-dim pca projection makes euclidean distance meaningful; however, it will trigger a significant information loss (2048D to 10D).
  • furthermore, since the pca projection is fixed, it cannot capture novel variance directions that emerge as the encoder adapts to target domains.
Questions for authors

see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vJQE — Weakness #1, #2, #3

Weakness 1.

Thank you for highlighting the need to clarify the practical motivation for UCDSL and its distinction from CTTA. The key distinction from standard CTTA evaluation is that UCDSL separately measures performance on unseen future domains before any adaptation to those domains. Following RaTP (Liu et al., 2023), UCDSL addresses the “unfamiliar period,” when predictions are required before a model has sufficiently adapted to a new domain.

For example, consider an autonomous driving system deployed in a new city: immediately after arrival, it may encounter unfamiliar road layouts, traffic signs, weather conditions, and camera characteristics, yet it must still make safe decisions. At the same time, adapting to this new environment should not cause the system to forget how to operate reliably in cities it has already encountered. We have clarified the distinction from CTTA in the Related Work section of the revised manuscript.

Weakness 2.

We agree that the combined average gain from vertex/support selection, hull matching, and repulsion is modest. However, this average masks substantial variation across source domains.

We have expanded the ablation analysis in the revised manuscript (Figure 5 in Appendix C.3). The analysis shows that smaller initial source replay-hull extent is associated with larger gains. Specifically, across all 10 domain orders and three seeds on Digits-5, the combined gain averages +1.76 points for USPS-source orders, compared with −0.30 points for SVHN-source orders.

These results highlight the role of these components in maintaining informative replay geometry and supporting robust continual adaptation, particularly when the initial source replay geometry is compact.

Weakness 3.

Most training settings, including the model architecture, learning rate, rtopr_{\mathrm{top}}, rtop′r'_{\mathrm{top}}, optimizer settings, and augmentation protocol, follow prior baselines. We therefore focused our sensitivity analyses on τ\tau, β\beta, and γ\gamma, which directly control confidence-based sample selection and the contributions of the two additional loss terms in TRACE as reported in Appendix C.4. Within the tested ranges, performance is more sensitive to τ\tau, which controls the selection of reliable target samples, while varying β\beta and γ\gamma has comparatively little effect. This observation is consistent with our motivation to account for pseudo-label confidence. In addition, the PCA dimension is set to the number of classes across all datasets, rather than tuned separately; we discuss this choice further in our response to Weaknesses 6–7.

AuthorsOpenReview ↗

Response to Reviewer vJQE — Weakness #4, #5

Weakness 4.

Thank you for raising this concern. For our frozen linear classifier, the Lipschitz condition holds with Lck=∥wc−wk∥2L_{ck}=\|w_c-w_k\|_2. The margin condition can be verified at the replay anchors because pairwise logit margins are affine. Thus, both conditions can be directly checked in the trained models, allowing us to empirically assess whether Assumption 1 holds.

We evaluated Assumption 1 across all three datasets, covering 10 domain orders and three seeds at every stage.

DatasetSource: anchors with γc>0\gamma_c>0Source: hulls with κc>0\kappa_c>0Target stages: anchors with γc>0\gamma_c>0Target stages: hulls with κc>0\kappa_c>0
PACS100%100%99.95–100%98.57–100%
DomainNet96.53%86.67%99.88–100%98.33–100%
Digits-596.43%84.00%100%100%

This summarizes stage-wise satisfaction rates aggregated over all orders and seeds. Although some source-stage hulls do not satisfy the margin condition, predominantly due to the exemplars misclassified by the source model, satisfaction reaches at least 99.88% of anchors and 98.33% of class hulls during target adaptation. These results show that the margin condition holds for nearly all replay hulls during continual adaptation, providing empirical support for Assumption 1 in our experiments.

Weakness 5.

We agree that the bound can become less informative when the class-conditional representation discrepancy is large. However, we view this explicit dependence as a strength of our analysis, as it characterizes how the guarantee deteriorates with increasing distribution mismatch, identifies when the bound becomes uninformative, and explains how improved replay coverage can extend its applicability.

This dependence is not unique to our analysis. Standard discrepancy-based bounds in domain adaptation and generalization exhibit similar behavior, including Wasserstein-based bounds [1, 2] and bounds involving source–target or target-to-source-mixture discrepancies [3, 4, 5]. These results provide conditional guarantees rather than uniformly tight guarantees under arbitrary domain shifts.

For example, Shen et al. [2] use an encoder fgf_g to map the original source and target inputs xsx^s and xtx^t to feature representations hs=fg(xs)h^s=f_g(x^s) and ht=fg(xt)h^t=f_g(x^t), respectively. In Eq. (3), they measure the Wasserstein distance between the resulting feature distributions:

W1(Phs,Pht)=sup⁡∥fw∥L≤1(EPhs[fw(h)]−EPht[fw(h)])W_1(\mathbb P_{h^s},\mathbb P_{h^t}) = \sup_{\lVert f_w\rVert_L \leq 1} \left(\mathbb E_{\mathbb P_{h^s}}[f_w(h)] - \mathbb E_{\mathbb P_{h^t}}[f_w(h)]\right).

The 1-Lipschitz condition in this expression applies to the domain critic fwf_w, rather than the encoder or classifier. In our analysis, the representation scale and classifier sensitivity are controlled separately: we L2-normalize encoder representations and use a linear classifier without a bias term, which is frozen after source training, as described in Section 4.5.

[1] Redko, Ievgen, et al. "Theoretical Analysis of Domain Adaptation with Optimal Transport." ECML PKDD (2017).

[2] Shen, Jian, et al. "Wasserstein distance guided representation learning for domain adaptation." AAAI (2018).

[3] Ben-David, Shai, et al. "A theory of learning from different domains." Machine learning 79.1 (2010).

[4] Albuquerque, Isabela, et al. "Generalizing to unseen domains via distribution matching." arXiv preprint arXiv:1911.00804 (2019).

[5] Sicilia, Anthony, et al. "Domain adversarial neural networks for domain generalization: When it works and how to improve." Machine Learning 112.7 (2023).

AuthorsOpenReview ↗

Response to Reviewer vJQE — Weakness #6, #7

Weakness 6.

To examine potential information loss from the fixed PCA projection, we have expanded Appendix C.6 with an analysis of classifier-PCA alignment in the manuscript, complementing the existing stage-wise variance-preservation analysis.

Across all three datasets, the first C−1C-1 source PCs retain 98.2–98.8% of the source variance and closely align with the classifier’s discriminative directions, supporting our choice of r=Cr=C. This is because of the linear classifier. Class predictions depend on the projection of the features onto the subspace spanned by differences between class weight vectors. This subspace has at most C−1C-1 dimensions. This structure helps explain why the leading source PCs align with the classifier’s discriminative directions in our experiments.

The stage-wise analysis further shows high variance retention throughout adaptation, including on unseen domains. These results support our design of adapting the encoder with a fixed source classifier, as in SHOT [6], while using a fixed PCA basis to provide consistent coordinates for comparing target and replay representations across stages.

[6] Liang, et al. "Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation." ICML (2020).

Weakness 7.

We agree that a fixed PCA projection cannot capture new variance directions outside the source subspace. As discussed in our response to W6, this subspace closely aligns with the frozen classifier’s discriminative directions, so it serves as a geometric reference for the feature-label relationship learned on the source domain.

Similar to the covariate shift assumption in domain adaptation, where PS(Y∣X)=PT(Y∣X)P_S(Y\mid X)=P_T(Y\mid X) despite changes in P(X)P(X), our design assumes that the feature-label relationship learned on the source domain remains sufficiently stable across target domains. Under this assumption, the fixed PCA subspace provides a consistent geometric reference for continual adaptation without target labels.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Additional Analyses

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback and discussion. We would like to briefly summarize our main contributions and how we have addressed the key concerns raised during the review process.

Core contribution. TRACE leverages class-wise replay geometry to estimate pseudo-label reliability and maintains informative replay regions throughout continual adaptation. Our theoretical analysis connects replay-hull coverage to unseen-domain classification risk, while experiments demonstrate the highest overall UCDSL performance and lowest ECE, Brier score, and NLL across all three benchmarks.

1. Theoretical justification and confidence estimation (vJQE, Shu5)

  • As originally described in Section 4.5, our theory motivates geometry-based confidence estimation rather than directly guaranteeing the relative-distance score. We further clarified this distinction and showed additional analysis that high correlations between relative confidence and absolute hull residuals--median Spearman: 0.896–0.988 across datasets (Appendix C.5).
  • We also empirically verified the theoretical assumptions across all three datasets, finding that at least 98.33% of replay hulls satisfy Assumption 1 during target adaptation. These results provide empirical support that the key condition underlying our theoretical analysis is largely maintained throughout continual adaptation.

2. PCA projection and component contributions (vJQE, Shu5)

  • We demonstrated that the leading C−1C-1 principal components preserve 98.2–98.8% of source variance and nearly all classifier-discriminative information, with high variance retention across subsequent stages (Appendix C.6).
  • We also expanded the ablation analysis, showing that geometry-maintenance components provide greater benefits when initial replay-hull coverage is limited (Appendix C.3).

3. Convex-hull advantages and computational efficiency (9s2r)

  • We conducted additional synthetic and PACS experiments demonstrating the advantages of convex-hull confidence under limited replay coverage. In the limited-anchor setting, convex-hull confidence achieved an AUROC of 0.806, compared with 0.555 for Mahalanobis. On PACS, TRACE also achieved a 1.87 percentage-point improvement in overall performance over Mahalanobis at the smallest replay budget (Appendix C.9).
  • Further analyses demonstrated manageable computational overhead, with hull-distance computation accounting for only 0.7% of training time, relatively stable performance across replay budgets, and no additional hull computation at inference (Appendices C.7–C.8).

We believe these additional experiments and clarifications have substantially addressed the main concerns and provided stronger support for TRACE's design choices and practical effectiveness. We hope they will be taken into consideration in the final assessment.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.