- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a task-agnostic method for active domain adaptation. The proposed RTW (Regularized Transport Weighting) jointly selects target samples to label and learns training weights for source and queried samples. Its objective combines transport cost with a penalty on weight concentration. The authors give theoretical evidence for the proposed method with generalization bounds and a greedy approximation guarantee. Also, they evaluate the method on OfficeHome-RSUT and mini-DomainNet.
Critical strengths
- The authors propose a single objective that connects query selection and training weights. They give theoretical evidence by a generalization analysis and a greedy approximation.
- The proposed method shows strong results at small query budgets on mini-DomainNet.
Critical weaknesses
- The impact of query selection of RTW itself is unclear. Comparisons that control for training weight are helpful to assess how much the proposed query selection contributes. For example, a comparison with random or K-medoids selection using the same RTW weighting helps to clarify this.
- The effect of the weight regularizer is also not directly shown. A comparison with RTW without the regularizer (λ = 0) and an analysis of sensitivity to λ will help clarify its contribution. The K-medoids baselines use different training weights, so they do not provide this comparison.
Questions for authors
How does RTW’s query selection time compare with K-medoids?
3 comments
Thank you for your rebuttal. Changing λ affects both which samples are selected and how they are weighted. Could you compare RTW and K-medoids for sample selection, using the same positive λ to compute the final RTW weights in both cases?
Following up on the runtime question, how long does each method take to select samples on the same hardware and with the same query budget?
Response to Reviewer 67od
We thank the reviewer for engaging meaningfully with our work. Below, we have attached the requested results, comparing k-medoids selection with RTW reweighting to RTW selection with RTW reweighting.
We would also like to mention that the reason we did not include this comparison in our original submission, is that we cannot establish theoretical guarantees for the algorithm that chooses the queries via -medoids, and then reweights the chosen queries with the RTW weights. This is because the -medoids selection strategy does not account for the potential variance in the labels . The main contribution of our work is to establish a theoretical framework, that allows us to derive the RTW weighting in the first place. Once we have this, it is possible that it could be incorporated into a variety of efficient practical heuristics. Evaluating these various methods in practical settings is an interesting direction that we do not pursue.
For the running time comparison, at each greedy round for OfficeHome-RSUT, RTW takes ~10-15 seconds to identify the point to query, whereas k-medoids takes less than a second.
In addition to this, we also present some synthetic experiments, which concretely demonstrate situations where k-medoids can fail (even if it uses RTW reweighting).
OfficeHome-RSUT
Held-out target accuracy (%) at 100 queries, with λ = 10.
| Task | K-medoids + RTW | RTW + RTW |
|---|---|---|
| Clipart → Product | 71.79 ± 2.02 | 72.85 ± 1.35 |
| Clipart → Real World | 72.35 ± 0.92 | 71.39 ± 1.55 |
| Product → Clipart | 44.51 ± 1.81 | 43.53 ± 2.41 |
| Product → Real World | 77.69 ± 0.93 | 76.73 ± 2.61 |
| Real World → Clipart | 44.90 ± 2.15 | 45.29 ± 2.47 |
| Real World → Product | 78.04 ± 2.59 | 77.78 ± 1.55 |
mini-DomainNet
Held-out target accuracy (%) at 100 queries, with λ = 1.
| Task | K-medoids + RTW | RTW + RTW |
|---|---|---|
| Clipart → Real | 68.55 ± 0.28 | 67.70 ± 0.75 |
| Painting → Real | 73.32 ± 0.23 | 73.04 ± 0.30 |
| Sketch → Real | 64.31 ± 0.34 | 63.40 ± 0.62 |
Synthetic Experiments
As mentioned before, our main focus of our work is principled approach involving reweighting, and an algorithm with rigorous guarantees. We now present simple synthetic experiments that demonstrate the failure point of the k-mediod based query strategy (even if we later reweight using our RTW objective with ). We also compare it to our greedy RTW method with (which comes with a guarantee).
For each problem, we ran 30 paired trials in 40 dimensions. Both methods used the same 200 labeled source examples and selected queries from the same unlabeled target pool using features alone. We used for RTW selection and computed the final training weights for both query sets using RTW with . We used the same model and training settings within each comparison.
Noisy midpoint regression. The source distribution is a mixture of three clusters: component with mixing weight 0.1 is a standard Gaussian with center on the x-axis at -12 while the other two clusters and are point masses with centers at and on the x-axis. The target is a mixture of from , from , and from . The Gaussian component has response zero, while the observations for cluster and have responses with means and respectively with variance . We used a pool of 1,000 target examples, fit weighted affine ridge regression with penalty , and evaluated exact target population mean squared error (MSE).
The experiment results are given below. RTW selection reduced mean MSE from 0.943 to 0.776 under the common weighting rule, a 17.8% reduction. RTW selected six and five observations at the two noisy locations, respectively. K-medoids selected one at each, which already gives zero distance at that location. We saw a similar result with for RTW selection and both final weight computations (0.753 versus 0.939 MSE). While we described a specific setting for concreteness, these failure models for k-mediod query selection are natural in settings with very hetergeneous populations with clusters of different radii e.g., a mixture of Gaussians with different covariances (it becomes worse in high dimensions), as k-mediod picks at most (or very few points) from the tighter clusters even if there is variation in the labels. We will describe this synthetic experiment as an illustrative example in the final version.
Regression: target population MSE at 30 queries.
| λ | K-medoids + RTW | RTW + RTW | Relative MSE reduction |
|---|---|---|---|
| 1 | 0.943 ± 0.044 | 0.776 ± 0.012 | 17.8% |
| 10 | 0.939 ± 0.044 | 0.753 ± 0.008 | 19.8% |
Response to Reviewer
We thank the reviewer for the opportunity to address any questions they may have. Below we answer some of the specific concerns raised.
Thanks for the question about the effect of query selection vs reweighting. In this work, we focus on implementing the RTW method with the same fixed choice of λ for both the query selection, and the regularized reweighting. We chose to focus on this setting because this is the method for which we have theoretical guarantees. As a heuristic, it is very reasonable to consider two different choices of λ used for query selection and the final reweighting. For example, k-medoids followed by our reweighting would correspond to λ=0 at the query selection stage, and increasing λ at the end for reweighting. We leave the evaluation of these hybrid methods to future work. The k-medoids method does not admit a theoretical guarantee by itself when the label is not deterministic. As a bad example, consider the case when the target distribution has some target mass p at a point or tight cluster, that is not covered by the source. The k-medoids method will query one example from this cluster, and then consider the entire cluster labeled. However, if there is some variance in the labels in the cluster, the single query is not enough to estimate the average label at that point with high accuracy. Our RTW method accounts for this uncertainty during selection, and trades-off being close to the target feature distribution (in a geometric notion of distance), with increasing the effective sample size of the reweighted distribution.
Thanks for the suggestion to compare with RTW with λ=0. Since the k-medoids baseline in our plots uses 50:50 balanced weights, we additionally ran RTW with λ=0 (or equivalently, k-medoids selection with nearest-neighbor weights) on both benchmarks, and we will include these curves in the revision. First, on mini-DomainNet, nearest neighbor weights completely removes the accuracy drop of k-medoid when using 50:50 weights, and achieves a much higher accuracy overall. This confirms that transport-based weighting itself substantially improves over heuristic domain balancing. Second, comparing λ>0 with λ=0 matches the bias–variance trade-off intuition in Theorem 3.1: on OfficeHome-RSUT (the smaller dataset), λ=0 would concentrate weight on fewer source samples, and the λ=10 regularization consistently improves over λ=0 across most tasks and budgets (by up to ~3 percentage points). On mini-DomainNet where more samples are present, λ=0 and λ=1 perform very similarly overall.
Thanks for the question about the runtime of our method. Our method RTW does require more computation per candidate than k-medoids. This is because it requires a regularized transport subproblem, as opposed to k-medoids that only needs to compute nearest neighbor distances. In our experiments we implement a series of optimizations that maintain our theoretical guarantees. Subsampling the candidates, dual screening, warm starts, and batched GPU optimizations help us bring down the computational cost in our implementation, while preserving the theoretical guarantees. It is possible that there are further optimizations that can bring down the computational cost even further, or efficient and effective heuristics that combine aspects of RTW with other methods such as k-medoids. This would be an interesting direction for future work. Moreover, we focus on settings where acquiring labels is costly, such as astronomy surveys, and therefore prioritize selecting queries that yield the most information.