- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes WASSAL, an active learning method built on a class-conditional Wasserstein distance. For each class, the method learns weights over the unlabeled samples so that the weighted unlabeled distribution matches the labeled data distribution of that class. Samples with low weights across all classes are queried for expert labeling, while high-weight samples are assigned pseudo-labels and incorporated into training through an additional weighted loss.
Critical strengths
- The overall motivation is reasonable: using the distribution of the existing labeled examples to guide sample selection is an interesting direction.
- The paper compares against a reasonably broad set of baselines covering uncertainty-based, gradient-based, and representation-based active learning methods, and reports results both with and without the proposed weighted soft loss.
Critical weaknesses
-
I am not convinced of the novelty of the Wasserstein based weighting formulation, as the proposed class conditional weighting appears closely related to conventional nearest neighbor or kernel based similarity. In Eq. (4), the unlabeled sample weights are optimized over the probability simplex without any additional constraints. For the unregularized Wasserstein objective, this optimization appears to reduce to assigning each labeled example to its nearest unlabeled point. Under the plan entropy regularization in Eq. (3), it seems to yield a corresponding distance based softmax assignment. This raises the question of what the Wasserstein optimization captures beyond simpler similarity based weighting. Since the paper does not compare against nearest neighbor or RBF kernel based weighting, it is difficult to assess whether the additional optimization is necessary.
-
The interpretation of the learned weights is also not sufficiently justified. The paper assumes that samples with low weights across all classes are confusing or informative, whereas high weight samples are reliable representatives of their classes, but I do not think these interpretations follow directly from the optimization objective. A sample may receive low weights because it lies far from all existing labeled examples rather than because it is close to a decision boundary. Such samples could be outliers or examples from poorly represented regions, and it is unclear whether selecting them necessarily improves classification.
Similarly, the class wise weights are normalized across unlabeled samples rather than across classes for each sample, so they cannot be directly interpreted as class probabilities or classification confidence. This matters because high weight samples are used to generate pseudo labels, but the paper reports neither pseudo label accuracy nor any analysis of how the learned weights relate to pseudo label correctness. Without such evidence, it is difficult to judge whether the weights provide a reliable signal for either acquisition or pseudo labeling.
In addition, the paper does not clearly report how many pseudo labeled samples are used during training. Eq. (6) selects the "top few" samples for each class, but the selection size or ratio is not specified. Since the weighted soft loss is a main contribution, I think the authors should report the pseudo label coverage, the ratio of pseudo labeled to expert labeled samples, and how these quantities vary across datasets and acquisition rounds. Some analysis would also help clarify whether the improvement comes from reliable Wasserstein based pseudo labeling or simply from using additional unlabeled examples.
Questions for authors
Q1. Relation to similarity based weighting
Which regularized OT objective is used to solve Eq. (4) in practice (e.g., the plan-entropy form in Eq. (3), a KL penalty with respect to the product of the marginals, or a debiased Sinkhorn divergence), and with what regularization strength? Given that the optimal weights appear to reduce to a nearest neighbor assignment without regularization and to a distance based softmax assignment with plan-entropy regularization, could the authors compare WASSAL against nearest neighbor and RBF kernel weighting baselines that use the same selection and pseudo labeling rules?
Q2. Characteristics of the queried samples
Why should low weights across all classes indicate informative samples rather than outliers or samples far from all labeled examples? Could the authors analyze the queried samples, for example by reporting their distance to the nearest labeled example and their overlap with the samples selected by CoreSet?
Q3. Quantity and quality of pseudo labels
- How many pseudo labeled samples are selected per class in Eq. (6), and what is the ratio of pseudo labeled to expert labeled samples across datasets and acquisition rounds?
- Could the authors report pseudo label accuracy as a function of the learned weights or of the number of selected samples?
- How are samples that receive high weights for multiple classes handled?
Q4. Source of the soft loss gains
How does WASSALWithSoft compare with standard confidence based pseudo labeling or self-training under the same annotation budget and training protocol? In Appendix B, are the pseudo labels for the other query rules also obtained from the Wasserstein weights in Eq. (4)? Finally, what explains the decrease in average accuracy on four of the five datasets when the soft loss is added?
We thank the reviewer. The additional experiments clarify the OT objective, query behaviour, and source of improvement.
Q1. OT objective; NN/RBF comparisons. Our implementation uses debiased Sinkhorn divergence (ε=0.0025), with 10 projected Adam steps initialized from uniform weights. We agree that the exact LP yields 1-NN assignments, while entropic transport yields distance-softmax weights. We compared these alternatives using identical query, S2 selection, and loss rules (Eqs. 5–7).
Caltech-101, budget 100, 3 seeds:
Sinkhorn gains 1.49 points over 1-NN and 2.66 over RBF. Although NN/RBF yield slightly more accurate pseudo-labels, Sinkhorn selects a 2–4× larger soft subset covering all classes. We will clarify that the transport step constructs a large, class-covering training subset.
Q2. Query characteristics. At budget 100, queried points lie at the 87th percentile of nearest-labeled-point distance on Caltech-101 and 64th percentile on STL-10. They cover relatively distant regions but are not simply the most distant points. Overlap with CoreSet queries is 24% and 6%, respectively.
Without soft loss, the query rule performs comparably to Random (Caltech-101: 18.00 vs. 19.38; STL-10: 26.97 vs. 25.71). We agree that low weights should not be interpreted as proximity to decision boundaries. We will instead describe the rule as coverage-oriented.
Q3. Pseudo-label quantity and quality. S2 includes top-weight points covering up to 80% of each class's mass, capped at 30/class (Caltech-101) or 400/class (STL-10). Points selected by multiple classes are assigned to their highest-weight class only.
The S2/labeled ratio is 2.1 on Caltech-101 (2.4 in round 1; 1.5 in round 7) and 3.7 on STL-10.
Pseudo-label accuracy is 14.2% on Caltech-101 (chance 1%; model 10.6%) and 25.7% on STL-10 (chance 10%; model 23.8%). It increases across rounds: 10.6→15.6% and 20.0→29.7%, respectively.
Accuracy by weight quartile (low→high):
Higher weights correlate with correctness on STL-10 but not Caltech-101. Hence within-class weight magnitude is not a reliable confidence measure across datasets. The class-level S2 selection carries the useful signal.
Q4. Source of soft-loss gains. We confirm that all
+softrows in Appendix B use the same Wasserstein S2; only the query rule changes.Self-training control: With identical annotation budgets, confidence-based self-training scores 11.48 vs. 24.04 on Caltech-101 and 25.50 vs. 28.65 on STL-10. Transport selection is particularly useful when model predictions are weak.
Fixed-S2 reweighting: Replacing Wasserstein weights with confidence or uniform weights yields comparable performance (Reviewer 1 response). Thus S2 selection, rather than specific weight values, explains most of the improvement.
Loss-scaling control: On Caltech-101, scaling the hard loss improves accuracy from 18.00 to 22.26, versus 24.04 with transport pseudo-labels (+1.78). On STL-10, gradient clipping limits scaling effects (27.16 vs. 26.97), while WASSALwithSoft achieves 28.65. The improvement cannot be attributed solely to loss magnitude.
We acknowledge that soft loss lowers average accuracy on four of five datasets. Its benefit is most pronounced when transport pseudo-labels outperform model predictions, especially in the many-class, few-label regime. We will explicitly restrict our claims.
Revisions. We will (i) specify the Sinkhorn objective, ε, solver, nonzero filtering, S2 rule and β; (ii) correct the imbalance description and Caltech-101 pool size; (iii) report mean±std; (iv) add learning curves, AULC, balanced accuracy, β sensitivity and S2 statistics; (v) include 1-NN, RBF, self-training, fixed-S2 and scaling controls; and (vi) remove the decision-boundary interpretation, emphasizing transport-guided soft-example selection instead.