- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TRACE (Target Reliability Assessment via Convex Evidence), a framework for unsupervised continual domain shift learning (UCDSL) that aims to improve pseudo-label reliability during sequential adaptation to unlabeled target domains. The key idea is to estimate pseudo-label confidence using class-wise convex hulls constructed from replay exemplars, rather than relying solely on classifier predictions. Specifically, TRACE measures the relative distances between target representations and the convex hulls of the predicted and competing classes. To maintain informative replay geometry across domains, the framework incorporates geometry-preserving exemplar selection and a hull-matching objective, together with an EMA-based reference encoder for stable pseudo-label generation. The paper also provides a theoretical analysis relating replay-hull coverage and sample residuals to classification risk on unseen domains under certain assumptions. Experiments on Digits-5, PACS, and DomainNet demonstrate improvements in overall classification performance across adaptation, generalization, and forgetting metrics, as well as better predictive calibration measured by ECE, Brier score, and NLL compared with existing UCDSL baselines.
Critical strengths
-
A well-motivated geometric perspective on pseudo-label reliability. The paper proposes using class-wise replay convex hulls to estimate pseudo-label confidence in UCDSL. Unlike prediction-based confidence measures, this approach considers the relative distances between target representations and class-wise replay regions. The framework also introduces geometry-preserving replay selection and hull matching to maintain these regions across domain shifts. This provides an alternative way to assess pseudo-label reliability using information retained from previous domains.
-
Empirical improvements across multiple evaluation metrics. The experiments cover three benchmark datasets and evaluate target domain adaptation, generalization, forgetting alleviation, and predictive calibration. TRACE achieves the highest overall classification accuracy on all three datasets and lower ECE, Brier score, and NLL than the compared baselines. These results provide empirical evidence for the effectiveness of the proposed framework across several evaluation criteria.
Critical weaknesses
-
Insufficient comparison with alternative geometric confidence estimation methods. Although TRACE achieves improvements over existing UCDSL methods, the experiments do not sufficiently establish whether the convex-hull-based confidence measure provides advantages over simpler geometric alternatives, such as prototype distances, nearest-exemplar distances, or Mahalanobis distances. Since geometric confidence estimation is a central component of the proposed framework, these comparisons are necessary to better isolate its contribution and determine whether the observed improvements specifically result from the use of convex hulls.
-
Limited evaluation of computational efficiency and scalability. The proposed confidence estimation requires computing distances between target representations and class-wise replay convex hulls, introducing additional computational overhead. Although the authors adopt PCA dimensionality reduction and use 30 Frank–Wolfe iterations to approximate these distances, the paper does not provide sufficient runtime analysis or comparisons of computational costs. Furthermore, the experiments use a fixed replay buffer of 200 samples, and the DomainNet experiments are conducted on a 10-class subset. It therefore remains unclear how the method scales to larger numbers of classes or different memory budgets. Additional runtime comparisons and scalability experiments would help establish the practical applicability of the approach.
Questions for authors
-
Comparison with alternative geometric confidence measures. Could the authors provide comparisons between the proposed convex-hull-based confidence measure and simpler geometric alternatives, such as prototype-based distances, nearest-exemplar distances, or Mahalanobis distances, under the same training framework? Such comparisons would help clarify whether the performance improvements can be attributed specifically to the convex-hull formulation.
-
Computational overhead. Could the authors report the additional training time or computational overhead introduced by replay-hull confidence estimation and hull matching compared with the existing UCDSL baselines? This would help assess the computational cost associated with the reported performance improvements.
-
Scalability with class number and replay memory size. How does the proposed method perform as the number of classes or the replay buffer size changes? Since the current experiments use a fixed buffer of 200 samples and a 10-class subset of DomainNet, results under different settings would help clarify the scalability of the approach, particularly when fewer replay exemplars are available per class.
4 comments
Response to Reviewer 9s2r — Weakness #2 / Question #2, #3
We agree that the manuscript should discuss TRACE’s computational cost and scalability in more detail. We designed three experiments to measure the cost of replay-hull operations, assess scalability with varing replay budget, and compare end-to-end training time with existing UCDSL baselines.
Experiment3.
Cost of replay-hull operations (Appendix C.7, Table 13). To quantify how much each operation contributes to training time, we profiled TRACE with the default 200-sample buffer.
| Component | Time (s) | Share |
|---|---|---|
| Replay-hull distance | 5.6 | 0.7% |
| Hull matching | 162.8 | 20.1% |
| Entire target stage | 810.1 | 100% |
Replay-hull distance computation takes up only a small fraction of training time. The larger cost of hull matching reflects its role as a training loss, whose measured runtime includes encoder forward pass over queued target and replay images and the associated backward pass.
Experiment4.
Scalability with replay budget (Appendix C.8, Tables 14–15). To test the effect of having fewer exemplars per class while holding the task fixed, we varied the replay budget across 35, 70, 140, 200, and 700 samples on seven-class PACS, using all 10 domain orders. We assessed scalability across these budgets in terms of both predictive performance and computational cost.
| Budget | TDA | TDG | FA | All | ΔAll |
|---|---|---|---|---|---|
| 35 | 86.79 | 69.52 | 85.55 | 80.62 | −1.54 |
| 70 | 87.63 | 70.29 | 87.78 | 81.90 | −0.27 |
| 140 | 87.61 | 70.19 | 88.12 | 81.98 | −0.18 |
| 200 | 87.91 | 70.27 | 88.30 | 82.16 | 0.00 |
| 700 | 87.77 | 70.33 | 88.67 | 82.26 | +0.10 |
| Budget | Distance (ms/call) | Matching (ms/call) | Memory (MiB) |
|---|---|---|---|
| 35 | 4.0 | 122.6 | 16,076 |
| 70 | 4.3 | 128.2 | 20,886 |
| 140 | 4.1 | 141.3 | 21,134 |
| 200 | 4.6 | 179.2 | 21,299 |
| 700 | 4.2 | 355.0 | 21,422 |
TRACE maintains comparable overall performance across a wide range of replay budgets, including substantially smaller buffers. Performance drops more noticeably at the smallest budget, with the largest decline in FA, suggesting that retention is particularly sensitive to very limited replay. As the buffer grows, distance latency stays nearly constant, while hull-matching time increases less than proportionally over the tested range.
Experiment5.
Training and inference cost compared with UCDSL baselines.
Training time (minutes/sequence)
| Method | PACS | Digits-5 | DomainNet |
|---|---|---|---|
| RaTP | 68 | 171 | 337 |
| CoDAG | 104 | 337 | 244 |
| MPM | 57 | 69 | 110 |
| TRACE | 35 | 198 | 89 |
TRACE achieves relatively low training costs, with the shortest training times on PACS and DomainNet. On Digits-5, training takes longer primarily due to the computational cost of KNN on a larger number of samples. Nevertheless, TRACE performs adaptation and generalization within a single training process using a single prediction model. In contrast, CoDAG trains separate DA and DG models, while MPM maintains multiple domain-specific prototype classifiers, requiring additional model capacity.
Inference time (ms/sample)
| Method | PACS | Digits-5 | DomainNet |
|---|---|---|---|
| RaTP | 1.64 | 0.25 | 2.26 |
| CoDAG | 1.53 | 0.19 | 1.99 |
| MPM, with BiGE | 2.83 | 1.20 | 2.68 |
| TRACE | 1.74 | 0.18 | 2.42 |
RaTP, CoDAG, and TRACE use a standard model forward pass at inference and have broadly similar runtimes. MPM additionally combines and refines predictions from its domain-specific prototype classifiers through BiGE, introducing graph-based computation. TRACE’s replay-hull operations are used during training and add no computation to its inference procedure.
Thank you for the detailed response and the additional experiments on computational overhead and replay budgets. These results have largely addressed my previous concerns regarding efficiency and scalability. Nevertheless, I remain uncertain about the necessity of the method’s core design. Since the entire training mechanism is built around convex hulls, their unique benefits over simpler geometric measures should be thoroughly validated. However, in the newly added replacement experiments, different distance measures yield very similar accuracy, while the calibration advantages of convex hulls are also relatively limited. Based on the current empirical evidence, it remains unclear whether the core component is necessary or offer substantial practical advantages. I would also like to see whether the concerns raised by the other reviewers have been adequately addressed.
Thank you for clarifying your remaining concern. We agree that similar average accuracy alone does not fully establish the benefits of choosing convex hulls. To investigate when this choice matters, we have added further analysis in Appendix C.9, “Comparison of Alternative Distance Measures.”
First, we compare convex hull, Mahalanobis, nearest-exemplar, and prototype distances on synthetic data with controlled replay coverage. These experiments separately examine limited numbers of replay anchors and insufficient source variation along directions that vary in the target domain.
Second, we compare convex-hull and Mahalanobis distances within TRACE across replay budgets on PACS, averaging over all 10 domain orders. By reducing the replay budget, we examine whether the effects of sample scarcity observed in the synthetic study also arise on a real-image benchmark.
Both experiments show benefits of convex hull distance under limited replay coverage. It provides more reliable confidence estimates in the synthetic study and improves overall performance on PACS, with the largest gain at the smallest replay budget. Together, these results highlight the practical advantages of convex hull confidence and support its reliability under varying replay coverage. We hope these additional analyses help address your concern about the choice of convex hulls and its practical value in TRACE.
Response to Reviewer 9s2r — Weakness #1 / Question #1
Thank you for suggesting these comparisons. We conducted two complementary experiments to assess the contribution of hull-based confidence.
Experiment1.
We replaced only the distance measure within TRACE, keeping the remaining training procedure unchanged. Given the substantial training time required (see our response to Reviewer Shu5’s Weakness 2 & Question 3), we evaluated four PACS domain orders (1, 4, 6, 9), covering all four source domains, and report their average results below. Once all experiments are complete, we will revise the manuscript to include the full results and the analysis presented below.
Hull-based confidence achieves higher mean TDA and FA than all alternatives and comparable overall performance to Mahalanobis. We acknowledge that iterative hull projection incurs a higher distance-computation cost. However, it operates in a low-dimensional PCA space (r=C) with a limited number of replay anchors and a fixed iteration budget (T=30), keeping the computation manageable in our setting. Here, C, r, m, and T denote the number of classes, PCA dimension, anchors per class, and Frank-Wolfe iterations, respectively. Mahalanobis additionally requires O(Cmr2+r3) preparation per reference update, which grows with the number of classes under our choice of r=C.
Experiment2.
Using the same protocol as in our original calibration evaluation, we compared all four distance measures across 10 PACS orders and three seeds at every target stage.
Hull-based confidence achieves the lowest values on all four metrics. Together, these experiments show that the hull formulation contributes beyond downstream accuracy by providing better-calibrated confidence. This directly supports its use for confidence-based sample selection and weighting in TRACE, while retaining the connection to our hull-residual theoretical analysis.