acceptodds
Under review as a conference paper at ICLR 2027

Feature Geometry for Data-Pool Screening: Utility, Stability, and Limits

Abstract

Selecting source-data pools for a target task can require training and validating a predictor for many combinations of pools. We propose a geometry-guided framework that shortlists combinations using unlabeled source and target features from frozen encoders, then trains predictors on source labels and selects one by target validation. Its core method, Target A-opt, applies target-weighted A-optimal design to pooled Gram matrices and the target second moment. Under a shared linear model, it favors combinations that reduce uncertainty about prediction coefficients, weighting feature directions by their energy in target data. A separate error decomposition distinguishes discarding good candidates from losing quality when validation chooses among those retained. On equal-sample-cost DomainNet combinations, an exploratory budget analysis with logistic regression finds lower mean test Brier from evaluating 50 Target A-opt candidates than 227 random candidates. In the original primary study, second-moment matching (MMD) matches Target A-opt on recall and final loss; it has higher mean recall at smaller budgets. Independent timing shows savings with cached features; adding measured feature-extraction costs preserves the logistic-regression saving but reverses it for ridge. These results support using target-aware geometry to reduce supervised data-pool evaluations, with savings depending on readout and feature preparation costs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Data-pool screening, Data selection, Frozen representations, Optimal experimental design, Distribution shift, Evaluation reliability.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HauMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to reduce the number of supervised evaluations needed to choose a combination of source-data pools for a target task. Candidates are shortlisted with target-weighted A-optimal design on frozen-feature Gram matrices, fit on source labels, and the final pick is made on a separate target validation set. Authors provided a Bayesian linear-regression interpretation and separate screening omission from validation-selection error.

Critical strengths
  • The paper demonstrates a concrete reduction in supervised evaluation costs.
  • The evaluation carefully separates screening quality, final selection, and computation costs.
Critical weaknesses
  • Table 1 is where I have the major concern with the case for A-opt. As it shows, second-moment MMD achieves the same primary recall and final loss, gives better recall at small budgets, and is also cheaper. Then why to use A-opt? Domain-level moment matching also works reasonably well after repartitioning. There is evidence here that matching to the target helps, but the benefit of inverse-covariance weighting is unclear. An experiment with and without that weighting, keeping everything else fixed, would help settle this. ALso, I do not think the theory resolves the issue either. The paper identifies the score as the regularized target-variance criterion from transductive experimental design, so the contribution seems to lie in its use for screening. The Bayesian regression interpretation follows from the assumed model. As for the omission/selection bound, I may have missed something, but it seems obtainable from a uniform validation-error bound and says little about classification screening in particular.

  • I am also unsure how much of the reported savings would survive a very minor change of setup. The equal-cost experiments all draw from one DomainNet inventory, with source pools shared across the six targets. More readouts and partitions are useful. Moreover, considering the practical compute resource limitations, I would not ask for multi-seed experiments. The remaining results do not really settle this for me. In the ridge study, every candidate is within 5% regret, so there is little to learn from which candidates pass that threshold because they all do. The gains on PACS and Office-Home are below what authors themselves set as practically meaningful. Logistic regression does give a more useful test, since the candidates actually differ there. Still, the budget comparisons are exploratory. And finally under ridge, the image-cost savings go the other way, which makes me hard to read these results as evidence for savings beyond this particular setup.

Questions for authors
  • What substantive contribution does Target A-opt provide beyond applying existing transductive experimental design to pool unions? In which empirically identifiable settings should it be preferred to the cheaper MMD comparator?

  • Can the authors replicate the quality-cost advantage on an independent dataset using a protocol and shortlist budget fixed before outcome evaluation?

  • Do the conclusions persist with calibrated ridge probabilities, validation-selected readout regularization, and larger feature dimensions? Given ridge Brier values near 0.987 for 100 classes, comparison with the uniform-probability baseline of 0.99 would help contextualize absolute predictive quality.

  • How does the quality–cost tradeoff change as the number of pools and candidate combinations grows? Can the authors establish a practical rule for choosing the shortlist budget without relying on exhaustive test auditing?

1 comment

Contribution. We have narrowed the claim to how a limited fitting budget is allocated over fixed source-pool combinations. Because candidates share pools, the validation loss of one fitted model can inform which other combinations are worth trying. We estimate additive pool effects from early losses and use them to choose the remaining fits. The revision develops this selection procedure and tests when the approximation preserves model quality with fewer fits and when it fails.

Inverse weighting and MMD. We agree that MMD is a strong, cheaper baseline. At the primary DomainNet budget, it selected the same models as Target A-opt. Holding the candidates, features and 50-fit budget fixed, mean test Brier was 0.864104 for Target A-opt, 0.896467 after removing inverse weighting, and 0.863951 for MMD. Inverse weighting therefore helped relative to target energy alone. The clearest case for preferring A-opt was calibrated ridge at 50 fits, where its mean Brier gap to full validation was −0.000150, compared with +0.002461 for MMD.

Ridge calibration and robustness. As you noted, every original ridge candidate met the 5% regret threshold. We now report absolute scores: calibrating the same ridge models reduced test Brier from 0.986695 to 0.743338, against 0.99 for uniform prediction. The feedback results were sensitive to readout settings. With fixed regularization, the 25-fit policy exceeded a +0.005 gap in 0/6 domains at 128 dimensions and 3/6 at 256. Tuning regularization changed those counts to 1/6 and 2/6 and required 75 fits.

Independent data and budget. On the independent Optdigits dataset, we fixed the 25-fit budget in advance. With hash pools, mean test Brier across four seeds was 0.304332 for A-opt versus 0.299730 for full validation. For one hash-seed configuration with one fitting worker, median decision time over three runs was 0.614 versus 9.632 seconds from in-memory features. With PC1 pools, both scored 0.339867. Appendix E also reports exact top-50 recovery among 4,060 candidates. A retrospective stopping rule saved fits for MMD but used more for A-opt, so choosing the budget prospectively remains open.

Reviewer_wsqfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on reducing the number of training and validation cycles required when selecting a combination of source data pools for a target task. The authors propose a method that first scores all candidate combinations using unlabeled features extracted from a frozen encoder, and then only the top-ranked combinations are subjected to supervised training, with the final selection made based on a target validation set. The paper also introduces an error decomposition that distinguishes between "discarding a good candidate" and "selecting a bad candidate during validation." Experiments conducted on the DomainNet dataset (involving 455 combinations of equal-sized data pools, two types of encoders, and two linear readout architectures) demonstrate that both the proposed A-opt approach and a simpler baseline based on the second-order moment of MMD can retain approximately 98% of the optimal combinations while filtering out half of the candidates. Overall, the MMD-based approach performs on par with, or even better than, the proposed A-opt.

Critical strengths
  1. The problem addressed in this paper has practical significance.

  2. The experiments in this paper are highly reproducible, and the appendix provides a detailed description.

Critical weaknesses
  1. My primary concern is that the method proposed in this paper lacks novelty and offers no significant performance improvement over existing approaches. Specifically, the proposed Target A-opt method is essentially an application of the transductive experimental design criterion (originally proposed by Yu et al. (2006)) to a fixed pool. On lines 76-77, the authors also say "Target A-opt follows the target-variance principle, but ranks fixed pool unions for subsequent supervised evaluation." Consequently, I think the novelty of this method is limited.

  2. For experiments and performance of the proposed method, one concern is that some empirical results (shown in Table 1 on page 7) show that the simple baseline method, such as MMD, achieves comparable performance or even slightly better than the proposed A-opt. Furthermore, the paper lacks comparisons with state-of-the-art data selection methods. As shown in Table 1 and Table 2, all compared methods are limited to simple geometric baselines. Although the authors mention the latest methods such as TSDS, TAROT, SkMM, and SIFT in the Related Work section, there are no experimental comparison results. So I cannot evaluate the performance of the proposed method more directly.

  3. The paper is also somewhat difficult to read and lacks logical coherence. For instance:

3.1. Terms such as "original study," "original primary study," and "follow-up" appear frequently throughout the main text, which may lead me to mistakenly assume that there are multiple parallel experimental tracks. Therefore, this phenomenon confused me and made it not easy to read and follow this paper.

3.2. In the experimental tables, such as Table 1 and Table 2, I suggest the authors use \textbf{bold} text to highlight the best-performing results, so that we can see an immediate assessment of the performance of the proposed method.

3.3. Additionally, in Table 1, for the metric Brier score, I suggest the authors include a measure of uncertainty (such as standard deviation using different seeds); otherwise, it is unclear whether the observed improvements are due to random fluctuations or the advantage of the proposed method.

3.4. I also suggest that the authors refine the layout of the paper. For example, there is a large, empty space on page 2.

Questions for authors

Please see weaknesses.

1 comment

Contribution and prior work. We have narrowed the contribution to using pool overlap and observed validation losses to allocate subsequent fits. A fixed shortlist commits the remaining trials before those scores are available. Our policy models each candidate by its pool membership, estimates pool effects from the first fits, and uses them to decide what to fit next. Target A-opt applies Yu et al.'s transductive design criterion, and the query formulation builds on linear experimental design and best-arm identification. We evaluate the value of this formulation through shared initialization, matched fitting budgets and failure cases.

Evidence for feedback. On DomainNet, 25 sequential fits give mean test Brier of 0.916072, close to 0.916078 from full validation with 455 fits. With the first 15 fits held fixed, choosing the last ten by estimated pool effects lowers mean test Brier by 0.003599 relative to continuing the MMD ranking.

Baselines and presentation. At 25 fits, mean DomainNet test Brier was 0.916072 for sequential querying, 0.916716 for D-design and 0.916678 for adapted LinGapE. These differences are small, and XY-design performed better on Optdigits with PC1 pools. For static screening, raw and whitened sliced Wasserstein gave 0.866406 and 0.865342 at 50 fits, compared with 0.863951 for MMD. Direct comparisons with TSDS, TAROT, SkMM and SIFT remain open. Table 1 now reports the standard deviation across four seed means and highlights the lowest mean at the shared budget. We also distinguish archived replays, newly fitted models and independent timing runs in the text, and revised the layout on page 2 to address the gap you identified.

Reviewer_hbbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript describes a data selection method, with a particular emphasis on the role of geometric information, and provides experimental validation across a variety of settings.

Critical strengths

The issue of data selection addressed in this research is both important and interesting; furthermore, the experimental section includes a discussion on several questions of interest to readers.

Critical weaknesses
  1. The relationship between this manuscript and related work is not sufficiently clear. Specifically, it is not well explained under what conditions current methods fail, nor how and why the proposed approach addresses this gap. The authors do not clearly define what they mean by "geometry", nor do they explain how it functions or why this role cannot be fulfilled by other means.
  2. The experimental evaluation appears inadequate; it lacks comparisons with a broader range of existing methods and lacks sufficiently comprehensive ablation studies. There is an overabundance of descriptive text relative to the experimental results or theoretical derivations, which makes the content appear somewhat thin.
Questions for authors

The questions that concern me most are: what specific problem, left unaddressed by existing works, does the methodology in this work solve? What is the underlying principle, and how does "geometry" play a role in it?

1 comment

Problem and idea. We have narrowed the task to selecting a model from fixed batches under a fitting budget. Choosing three of 15 batches gives 455 possible training sets. Our idea is to let early trials guide the rest. Shared pools allow the loss of a fitted combination to inform predictions for untried combinations, so the same evaluation can guide several later choices. The query policy uses these predictions to allocate the remaining fits.

Role of geometry. Before any model is fitted, the source Gram matrix tells us which feature directions a candidate covers, and the target second moment tells us how much target feature energy lies in those directions. Appendix A gives an example with equal sample counts and equal Gram traces in which exact moment matching selects the candidate with higher target prediction risk under a shared linear model.

How feedback works. Once models have been fitted, validation losses offer another way to guide selection. If two triples share two pools, their loss difference helps estimate the effect of swapping the third. We begin with 15 combinations whose pool-membership vectors have full rank. This makes the estimated pool effects unique. Under exact additivity and exact observations, these scores identify all candidate losses. We tested the practical value of feedback by holding the first 15 fits fixed. Using validation feedback to choose the final ten lowered mean DomainNet test Brier by 0.003599 relative to following the MMD ranking. Removing the exploration bonus preserved 47 of 48 winners. Appendix E compares D-design, XY-design and adapted LinGapE at the same budget.

Boundary of the claim. The limitation is visible on Optdigits. With PC1 pools, MMD matched full validation at 25 fits, while sequential querying scored 0.066802 worse. That partition changes class coverage as well as possible interactions between pools, so the gap cannot be attributed to either factor alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.