- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AL-ERO for training a classifier for threshold-based auto-labeling: maximize empirical coverage at a target error rate using frozen foundation-model representations, with an auxiliary-variable reformulation of the selection indicator solved by an exact penalty method plus a confidence–selection alignment regularizer. Experiments on two image and two text datasets report higher mean coverage than the evaluated baselines at 5% and 10% error targets.
Critical strengths
- Standard classifier training does not directly target high coverage at a specified error level, and optimizing prediction and confidence on the same labeled set is a reasonable approach when labels are limited. The auxiliary-variable formulation also avoids choosing the sharpness of a sigmoid surrogate.
- The results are encouraging: AL-ERO achieves the highest mean coverage in all eight settings in Table 1 with mean error below target, e.g. coverage on PubMed at 5% error rising from 31.61% (AURC) to 37.22%.
Critical weaknesses
- My main concern is the gap between the paper's exactness claim and the objective that is actually optimized. Definition 3.2 and Theorem 3.3 refer to non-singular points, but Eq. (3.6) does not restrict its optimization domain accordingly; at the equality admits selections the original threshold rule cannot express. Appendix A.3's equivalence of non-singular feasible sets does not establish equivalence of global optima if singular points remain admissible. The alignment regularizer discourages singularity but changes the objective and does not rule it out, and deployment checkpoints are selected by coverage on without evidence that they satisfy the original hard training constraint. Without this, the theoretical distinction from surrogate optimization is not supported in practice.
- The experiments do not isolate the benefit of exact reformulation. The surrogate+EP ablation in Table 7 differs from AL-ERO not only in the indicator treatment but also in the alignment regularizer and the optimizer's batch size (full-batch for AL-ERO vs mini-batch for surrogate+EP in the released code), and all released surrogate+EP configurations use , which keeps the soft selection weights within roughly [0.378, 0.622] over the nominal score range. With Table 7 coverage gains of only 0.14 and 0.09 percentage points on two datasets, it is unclear how much of the gain comes from the reformulation rather than confidence shaping or optimization conditions.
- Some implementation details differ from the paper's description: the code uses a ratio-based error penalty rather than Algorithm 1's stated form, adaptively increases the alignment weight rather than treating as a fixed input, and decreases some consistency weights. These may be reasonable choices, but the paper and code should agree on which procedure produced the results.
- Fixing the threshold at 0.5 relies on a linear bias absorbing the threshold shift, but that argument does not transfer to maximum softmax probability in a multiclass classifier: a common logit shift leaves probabilities unchanged, while class-specific shifts can change predictions.
Questions for authors
- Theorem 3.3 explicitly restricts the candidate solutions to non-singular points; is global optimality also taken over a non-singular feasible domain? Please make this explicit and report, for the deployed checkpoints, the hard-constraint diagnostics the code already computes, including coverage, error, residuals, and selection discrepancies.
- Could you compare AL-ERO with a matched full-batch surrogate+EP baseline, a wider sharpness search, and a separate ablation of the alignment regularizer?
- Please reconcile Algorithm 1 and Eq. (3.10) with the released code, and state what the values in the tables represent.
- How does the fixed 0.5 threshold interact with maximum softmax probability in the multiclass setting, and does a learned-threshold version perform comparably?
5 comments
Weakness 1 / Question 1 — Hard-Constraint Diagnostics
As requested, we report the hard-constraint diagnostics on the training set of the first TBAL round, at both the deployed (i.e., selected) checkpoint and, for reference, the last checkpoint (the final outer iteration):
- coverage and error of the hard-threshold selection on training set at the training threshold , from which the residual of the error constraint can be read directly;
- selection discrepancy, i.e., the proportion of training samples whose auxiliary selection () differs from the hard-threshold selection (), which reflects the violation of the indicator constraints;
- residual of the indicator constraints, reported as the proportion of training samples whose residual is strictly smaller than .
Each entry is mean ± SD over five seeds; coverage and error are in %.
| Dataset (Target Error) | Coverage / Error (Selected Checkpoint) | Coverage / Error (Last Checkpoint) |
|---|---|---|
| OrganACSMNIST (5%) | 94.73 ± 0.16 / 6.16 ± 0.15 | 93.94 ± 0.09 / 4.31 ± 0.03 |
| OrganACSMNIST (10%) | 98.75 ± 0.95 / 6.93 ± 0.52 | 99.75 ± 0.03 / 7.44 ± 0.10 |
| PathMNIST (5%) | 98.19 ± 0.32 / 3.95 ± 0.13 | 99.17 ± 0.03 / 4.02 ± 0.01 |
| PathMNIST (10%) | 98.77 ± 0.85 / 3.01 ± 0.34 | 100.00 ± 0.01 / 3.41 ± 0.03 |
| PubMed 200k RCT (5%) | 92.31 ± 0.06 / 9.90 ± 0.10 | 92.80 ± 0.08 / 8.90 ± 0.12 |
| PubMed 200k RCT (10%) | 91.79 ± 2.97 / 10.93 ± 1.60 | 94.92 ± 0.09 / 9.09 ± 0.09 |
| 20 Newsgroups (5%) | 74.44 ± 0.23 / 5.59 ± 0.28 | 73.85 ± 0.11 / 4.77 ± 0.08 |
| 20 Newsgroups (10%) | 83.78 ± 0.34 / 9.49 ± 0.17 | 83.30 ± 0.06 / 8.82 ± 0.04 |
| Dataset (Target Error) | Mismatch Rate (Selected Checkpoint) | Mismatch Rate (Last Checkpoint) |
|---|---|---|
| OrganACSMNIST (5%) | 4.022% ± 0.389% | 0.274% ± 0.027% |
| OrganACSMNIST (10%) | 1.010% ± 0.910% | 0.016% ± 0.005% |
| PathMNIST (5%) | 1.368% ± 0.290% | 0.004% ± 0.009% |
| PathMNIST (10%) | 1.252% ± 0.861% | 0.000% ± 0.000% |
| PubMed 200k RCT (5%) | 9.388% ± 0.112% | 5.460% ± 0.138% |
| PubMed 200k RCT (10%) | 5.992% ± 4.809% | 0.456% ± 0.103% |
| 20 Newsgroups (5%) | 1.572% ± 0.829% | 0.180% ± 0.020% |
| 20 Newsgroups (10%) | 1.540% ± 0.152% | 0.204% ± 0.036% |
| Dataset (Target Error) | Residual (Selected Checkpoint) | Residual (Last Checkpoint) |
|---|---|---|
| OrganACSMNIST (5%) | 92.766% ± 0.505% | 99.680% ± 0.027% |
| OrganACSMNIST (10%) | 95.304% ± 4.889% | 99.798% ± 0.008% |
| PathMNIST (5%) | 96.204% ± 0.543% | 99.982% ± 0.019% |
| PathMNIST (10%) | 82.890% ± 13.218% | 99.984% ± 0.005% |
| PubMed 200k RCT (5%) | 48.584% ± 1.731% | 92.650% ± 0.171% |
| PubMed 200k RCT (10%) | 88.880% ± 9.018% | 99.434% ± 0.157% |
| 20 Newsgroups (5%) | 92.476% ± 1.746% | 96.252% ± 0.122% |
| 20 Newsgroups (10%) | 95.224% ± 0.215% | 99.696% ± 0.065% |
The solver itself is able to solve the training problem: as training proceeds, the mismatch rate decreases and the proportion of samples with residuals below increases in all eight settings, except for the PubMed 200k RCT (5%) where the training did not fully converged. We attribute this to the training configuration shared across all datasets (the hyperparameter search space, the training budget, and the penalty-update schedule), which is not optimal for this dataset; we kept it identical across datasets for a fair comparison.
While we can attain a feasible solution in all but one case on the training set, the trained model and threshold is not guaranteed to be feasible on the unlabeled data, because of generalization gap. Therefore, following the TBAL protocol, for all methods, the checkpoint is selected by its coverage on under the re-estimated deployment threshold to control the error on the unlabeled data.
Moreover, unlike a sigmoid surrogate, whose approximation gap is built into the objective, the reformulated problem itself is exact over the non-singular feasible set; hence, the solver directly optimizes the hard-threshold coverage and error.
Weakness 2 / Question 2 — Matched Comparison with AL-Surrogate+EP
Thank you for pointing this out!
First, both AL-Surrogate+EP and our AL-ERO try to solve the canonical joint classification-selection formulation in Eq. (3.1). Our AL-ERO involves exact reformulation of indicator functions and regularization that attempts to avoid the singular case in the reformulation. By contrast, AL-Surrogate+EP smooths indicator functions directly. The regularization term in our AL-ERO is unique to the reformulation idea, and there is no direct match in AL-Surrogate+EP.
Second, as suggested, we match the other settings of AL-ERO and AL-Surrogate+EP for a fairer comparison during the rebuttal. Below, we use the original experiment settings unless declared otherwise.
- batch strategy: full-batch, deterministic optimization; in each TBAL round, after the shared CE warm-up, AL-Surrogate+EP runs the same number of outer and inner full-batch steps on as AL-ERO;
- checkpoint rule: as in AL-ERO, the checkpoint is selected by the deployment-threshold coverage on , and the deployment threshold is estimated on ;
- grid: in the original experiments, we fixed and swept the learning rate, initial error weight, and CE weight, so that both methods had a similar search budget. We now additionally search , selected on by the same rule as for all other methods (highest mean coverage among values whose mean error is within the target).
As shown in the table below (mean ± SD over five random seeds), under the matched setup, AL-ERO achieves equal or higher mean coverage than AL-Surrogate+EP in seven of eight settings, with all mean errors below the target error.
| Dataset (Target Error) | Selected | AL-Surrogate+EP (Coverage / Error) | AL-ERO (Coverage / Error) |
|---|---|---|---|
| OrganACSMNIST (5%) | 5 | 53.84 ± 0.39 / 3.76 ± 0.07 | 53.59 ± 0.18 / 3.62 ± 0.05 |
| OrganACSMNIST (10%) | 5 | 68.04 ± 0.11 / 8.51 ± 0.04 | 68.15 ± 0.34 / 8.52 ± 0.12 |
| PathMNIST (5%) | 1 | 73.05 ± 0.17 / 3.79 ± 0.04 | 73.45 ± 0.09 / 3.94 ± 0.04 |
| PathMNIST (10%) | 0.1 | 83.19 ± 0.00 / 8.25 ± 0.02 | 83.19 ± 0.01 / 8.31 ± 0.05 |
| PubMed 200k RCT (5%) | 1 | 35.63 ± 0.42 / 4.28 ± 0.05 | 37.22 ± 2.16 / 4.39 ± 0.25 |
| PubMed 200k RCT (10%) | 1 | 62.53 ± 0.38 / 8.54 ± 0.05 | 62.95 ± 0.71 / 8.48 ± 0.08 |
| 20 Newsgroups (5%) | 1 | 23.12 ± 0.02 / 4.62 ± 0.01 | 23.13 ± 0.42 / 4.89 ± 0.20 |
| 20 Newsgroups (10%) | 5 | 30.59 ± 0.03 / 9.39 ± 0.02 | 31.11 ± 0.09 / 9.72 ± 0.10 |
In our revision, we will update the ablation study in Appendix D.1 (Table 7) with this more matched comparison.
Weakness 3 / Question 3 — Reconciling Algorithm 1 and Eq. (3.10) with the Released Code, and the Meaning of
Thank you for pinpointing the discrepancy! All our reported experimental results are generated using the released implementation. We purposefully make our mathematical presentations of these points simplified versions of the actual implementation to minimize distraction by low-level details while keeping the mathematical essence. We will make this clear in our revised version.
Definition of . In all tables, denotes the sample standard deviation.
Weakness 4 / Question 4 — Fixing
Thanks for raising this excellent point! Empirically, fixing and leaving learnable lead to very close results, as shown in the table below.
| Dataset (Target Error) | Learned (Coverage / Error) | Fixed (Coverage / Error, Table 1) |
|---|---|---|
| OrganACSMNIST (5%) | 53.30 ± 0.27 / 3.62 ± 0.04 | 53.59 ± 0.18 / 3.62 ± 0.05 |
| OrganACSMNIST (10%) | 68.18 ± 0.10 / 8.54 ± 0.04 | 68.15 ± 0.34 / 8.52 ± 0.12 |
| PathMNIST (5%) | 73.98 ± 0.20 / 4.16 ± 0.06 | 73.45 ± 0.09 / 3.94 ± 0.04 |
| PathMNIST (10%) | 83.20 ± 0.00 / 8.29 ± 0.01 | 83.19 ± 0.01 / 8.31 ± 0.05 |
| PubMed 200k RCT (5%) | 36.29 ± 1.63 / 4.31 ± 0.16 | 37.22 ± 2.16 / 4.39 ± 0.25 |
| PubMed 200k RCT (10%) | 63.52 ± 1.19 / 8.52 ± 0.10 | 62.95 ± 0.71 / 8.48 ± 0.08 |
| 20 Newsgroups (5%) | 22.66 ± 0.31 / 4.66 ± 0.20 | 23.13 ± 0.42 / 4.89 ± 0.20 |
| 20 Newsgroups (10%) | 31.10 ± 0.06 / 9.84 ± 0.05 | 31.11 ± 0.09 / 9.72 ± 0.10 |
So, in our original version, we focused on the version for simplicity, alongside the handwavy equivalence argument. After careful checking, we agree with the reviewer that there are bugs in the argument. In our revised version, we will include both versions of results in all experiments.
Thank you for pointing out this! Yes, global optimality is intended to be taken over the non-singular feasible domain. We intend to establish the equivalence over the non-singular feasible sets---for both the original problem in Eq. (3.1) and the reformulated problem in Eq. (3.6), as explicitly motivated by Lemma 3.1 and Definition 3.2. However, when we stated the results in the original submission, there were technical glitches. In fact, in our proof of Theorem 3.3, we have implicitly assumed the Eq. (3.6) is over the non-singular feasible set—exactly as suggested by the reviewer; for Eq. (3.1), it is easy to show that there always exist non-singular global solutions—which we also took for granted. These hidden facts/assumptions have caused the current glitches.
In our revision, we will
Theorem 3.3 (Global equivalence over the non-singular feasible sets). Any (θ∗,t∗,s∗) is a global solution to Eq. (3.6) if and only if (θ∗,t∗) is a non-singular global solution to Eq. (3.1).
With singular points excluded from both problems, the proof in Appendix A.3 holds without modification.
One of the goals of the regularization is to promote non-singularity. We agree that it adds a term to the objective and does not strictly enforce non-singularity; it is difficult to enforce the non-equality constraint, i.e., the non-singularity constraints ci(θ)=t,∀i∈I, because they induce an open set that can cause various conceptual and numerical difficulties—typical numerical optimization methods deal with closed and bounded constraint sets only.
To check the singularity numerically in practice, as suggested, we obtain statistics of the proportion of training samples that become almost singular, i.e., with ∣ci−0.5∣<10−6, at both the selected and the last checkpoints, in the following table:
The extremely low fraction of (numerically) singular points suggest that the regularizer is effective in preventing almost all the training samples from singular points, although not all. Moreover, we observe a substantial decrease of the singularity rate from the checking point to the final point in general.