- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper filters a large unlabeled pool with a small sample from a target distribution. Under bounded features and approximate linear separation, it bounds the total variation distance between the filtered and target distributions, with complexity set by the margin rather than an ambient dimension. A lower bound matches the dependence on the positive sample size and the margin up to logarithmic factors, even with unlimited unlabeled data. Unknown margin is handled by selecting among candidates with independent validation samples. Experiments show downstream gains on synthetic data and target-source enrichment in a Wikipedia and C4 mixture.
Critical strengths
-
Control of filtering error without labeled negatives. The false-positive argument is the strongest part of the upper-bound analysis. It uses the constrained objective to connect errors on the two mixture components, then obtains a TV guarantee with explicit dependence on sample sizes, target proportion, and overlap. This explains why the filtering strategy works under the stated geometry.
-
A matching lower bound and a principled treatment of unknown margin. Keeping the mixture fixed across different targets isolates the information supplied by the positive sample. The resulting lower bound shows that the dependence is necessary, up to logarithmic factors. The margin-selection analysis adds a useful extension, with explicit control of the validation and selection errors.
Critical weaknesses
- The guarantee does not directly cover the filter the experiments run. Theorem 7 bounds the distance achieved by the solution of the constrained problem in (1), which keeps margin violations on the positive sample under a threshold while minimizing acceptance on the mixture. The synthetic experiment trains a weighted linear SVM and the text experiment an L2-regularised logistic regression, but neither is shown to solve (1) or inherit its guarantee. The conclusion lists robustness to inexact solutions as an open problem, and Section 6 still describes the experiments as a confirmation of the approach. A reader cannot tell how much of the reported gain the theory accounts for.
- The numbers in Table 1 and Figure 1 have no stated protocol. Table 1 reports false negative and false positive rates without saying which paragraphs they are computed on. The mixture holds 2,020 Wikipedia paragraphs and Table 1 reports 2,000 positive training examples, and the paper never says whether the positives come from that same pool. If they do, at most 20 target paragraphs stay outside training and the reported 0.180 and 0.173 are close to training error, so the enrichment from about 1 percent to roughly 9 and 18 percent target share cannot be checked. Figure 1 has the same gap, since the distance estimate is described only as histogram binning, with no word on which variables are binned or which samples are counted.
Questions for authors
- For the weighted SVM and the logistic regression used in the experiments, what is the margin violation rate on the positive sample, and how does the resulting filter compare with the solution of the constrained problem in (1)?
- Are the 2,000 positive training examples drawn from the same 2,020 Wikipedia paragraphs that make up the target component of the mixture? If so, which target paragraphs are left out when the rates in Table 1 are computed, and if not, how were the two sets kept apart?
- Which variables or projections are histogrammed for the distance estimate in Figure 1, how are the bins chosen, and which samples enter the calculation? Does the estimate describe the full feature distribution or a projection of it?
2 comments
I thank the authors for addressing my questions. I have no further questions and have decided to maintain my original score of 3, which I believe is a fair assessment.
Response
Thank you for your detailed and thoughtful comments.
Question 1: Our experiments do not use the exact solution to (1) because we show that it is NP-hard to solve this problem exactly (see Appendix G). This is why we instead use a simple convex relaxation — analogous to the standard ML practice of using convex relaxations such as hinge of logistic loss in ordinary binary classification settings. We provide details of additional simulations in the response to reviewer TDuq, where we also provide numbers on the percentage of points in the small target sample that violate the margin, as you requested. They are uniformly quite small.
For the MiniLM/BGE experiments, we do not know the appropriate margin, so it is impossible to say how many points violate it. This will be true in any real-world scenario.
Question 2: The 2000 positive training samples are distinct from the 2020 samples in the mixture.
Question 3: For Figure 1’s histogram, each trial compared the first coordinate x1 of the 200 positive training examples with x1 of the filtered examples, because this is the only coordinate that is relevant for separating S and O. Doing a full density estimation is difficult because of the curse of dimensionality.
We used 50 equal-width bins shared by both groups, spanning their pooled minimum minus 0.5 to their pooled maximum plus 0.5. After normalizing each histogram to sum to one, we computed 21∑j∣p^j−q^j∣. This measures discrepancy along the separating coordinate. The downstream accuracy used 5,000 fresh examples from S.
We will incorporate these points in the revision.