- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to reduce the number of supervised evaluations needed to choose a combination of source-data pools for a target task. Candidates are shortlisted with target-weighted A-optimal design on frozen-feature Gram matrices, fit on source labels, and the final pick is made on a separate target validation set. Authors provided a Bayesian linear-regression interpretation and separate screening omission from validation-selection error.
Critical strengths
- The paper demonstrates a concrete reduction in supervised evaluation costs.
- The evaluation carefully separates screening quality, final selection, and computation costs.
Critical weaknesses
-
Table 1 is where I have the major concern with the case for A-opt. As it shows, second-moment MMD achieves the same primary recall and final loss, gives better recall at small budgets, and is also cheaper. Then why to use A-opt? Domain-level moment matching also works reasonably well after repartitioning. There is evidence here that matching to the target helps, but the benefit of inverse-covariance weighting is unclear. An experiment with and without that weighting, keeping everything else fixed, would help settle this. ALso, I do not think the theory resolves the issue either. The paper identifies the score as the regularized target-variance criterion from transductive experimental design, so the contribution seems to lie in its use for screening. The Bayesian regression interpretation follows from the assumed model. As for the omission/selection bound, I may have missed something, but it seems obtainable from a uniform validation-error bound and says little about classification screening in particular.
-
I am also unsure how much of the reported savings would survive a very minor change of setup. The equal-cost experiments all draw from one DomainNet inventory, with source pools shared across the six targets. More readouts and partitions are useful. Moreover, considering the practical compute resource limitations, I would not ask for multi-seed experiments. The remaining results do not really settle this for me. In the ridge study, every candidate is within 5% regret, so there is little to learn from which candidates pass that threshold because they all do. The gains on PACS and Office-Home are below what authors themselves set as practically meaningful. Logistic regression does give a more useful test, since the candidates actually differ there. Still, the budget comparisons are exploratory. And finally under ridge, the image-cost savings go the other way, which makes me hard to read these results as evidence for savings beyond this particular setup.
Questions for authors
-
What substantive contribution does Target A-opt provide beyond applying existing transductive experimental design to pool unions? In which empirically identifiable settings should it be preferred to the cheaper MMD comparator?
-
Can the authors replicate the quality-cost advantage on an independent dataset using a protocol and shortlist budget fixed before outcome evaluation?
-
Do the conclusions persist with calibrated ridge probabilities, validation-selected readout regularization, and larger feature dimensions? Given ridge Brier values near 0.987 for 100 classes, comparison with the uniform-probability baseline of 0.99 would help contextualize absolute predictive quality.
-
How does the quality–cost tradeoff change as the number of pools and candidate combinations grows? Can the authors establish a practical rule for choosing the shortlist budget without relying on exhaustive test auditing?
Contribution. We have narrowed the claim to how a limited fitting budget is allocated over fixed source-pool combinations. Because candidates share pools, the validation loss of one fitted model can inform which other combinations are worth trying. We estimate additive pool effects from early losses and use them to choose the remaining fits. The revision develops this selection procedure and tests when the approximation preserves model quality with fewer fits and when it fails.
Inverse weighting and MMD. We agree that MMD is a strong, cheaper baseline. At the primary DomainNet budget, it selected the same models as Target A-opt. Holding the candidates, features and 50-fit budget fixed, mean test Brier was 0.864104 for Target A-opt, 0.896467 after removing inverse weighting, and 0.863951 for MMD. Inverse weighting therefore helped relative to target energy alone. The clearest case for preferring A-opt was calibrated ridge at 50 fits, where its mean Brier gap to full validation was −0.000150, compared with +0.002461 for MMD.
Ridge calibration and robustness. As you noted, every original ridge candidate met the 5% regret threshold. We now report absolute scores: calibrating the same ridge models reduced test Brier from 0.986695 to 0.743338, against 0.99 for uniform prediction. The feedback results were sensitive to readout settings. With fixed regularization, the 25-fit policy exceeded a +0.005 gap in 0/6 domains at 128 dimensions and 3/6 at 256. Tuning regularization changed those counts to 1/6 and 2/6 and required 75 fits.
Independent data and budget. On the independent Optdigits dataset, we fixed the 25-fit budget in advance. With hash pools, mean test Brier across four seeds was 0.304332 for A-opt versus 0.299730 for full validation. For one hash-seed configuration with one fitting worker, median decision time over three runs was 0.614 versus 9.632 seconds from in-memory features. With PC1 pools, both scored 0.339867. Appendix E also reports exact top-50 recovery among 4,060 candidates. A retrospective stopping rule saved fits for MMD but used more for A-opt, so choosing the budget prospectively remains open.