acceptodds
Under review as a conference paper at ICLR 2027

Direct Optimization of Selective Classifier for Auto-Labeling via Exact Reformulation

Abstract

With the increasing demand for large-scale labeled data, auto-labeling has emerged as an effective technique for annotating unlabeled data with minimal human intervention. The goal of auto-labeling is to achieve a coverage as high as possible on the unlabeled data while keeping the error rate below a user-specified value. Threshold-based auto-labeling (TBAL) is a multi-round framework that leverages a classifier to selectively label samples based on whether the confidence scores exceed an estimated threshold. However, standard classifier training does not take into account the error-coverage tradeoff in auto-labeling and may yield confidence scores that poorly separate correct from incorrect predictions. Furthermore, directly optimizing this objective is challenging because both coverage and error involve indicator functions with zero gradients almost everywhere, while replacing them with smooth surrogates introduces an approximation gap that undermines optimality. In this paper, we propose AL-ERO (Auto-Labeling via Exact Reformulation and Optimization), which trains a selective classifier by jointly optimizing classification and selection. AL-ERO uses an exact reformulation of the selection indicator, thereby eliminating the approximation gap introduced by smooth surrogates. Experiments on four datasets covering image and text modalities show that AL-ERO achieves improved auto-labeling performance.

Primary area:
datasets and benchmarks
Keywords:
threshold-based auto-labeling, confidence optimization, exact reformulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sfMQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AL-ERO for training a classifier for threshold-based auto-labeling: maximize empirical coverage at a target error rate using frozen foundation-model representations, with an auxiliary-variable reformulation of the selection indicator solved by an exact penalty method plus a confidence–selection alignment regularizer. Experiments on two image and two text datasets report higher mean coverage than the evaluated baselines at 5% and 10% error targets.

Critical strengths
  1. Standard classifier training does not directly target high coverage at a specified error level, and optimizing prediction and confidence on the same labeled set is a reasonable approach when labels are limited. The auxiliary-variable formulation also avoids choosing the sharpness of a sigmoid surrogate.
  2. The results are encouraging: AL-ERO achieves the highest mean coverage in all eight settings in Table 1 with mean error below target, e.g. coverage on PubMed at 5% error rising from 31.61% (AURC) to 37.22%.
Critical weaknesses
  1. My main concern is the gap between the paper's exactness claim and the objective that is actually optimized. Definition 3.2 and Theorem 3.3 refer to non-singular points, but Eq. (3.6) does not restrict its optimization domain accordingly; at ci=tc_i=t the equality admits selections the original threshold rule cannot express. Appendix A.3's equivalence of non-singular feasible sets does not establish equivalence of global optima if singular points remain admissible. The alignment regularizer discourages singularity but changes the objective and does not rule it out, and deployment checkpoints are selected by coverage on DthD_{\mathrm{th}} without evidence that they satisfy the original hard training constraint. Without this, the theoretical distinction from surrogate optimization is not supported in practice.
  2. The experiments do not isolate the benefit of exact reformulation. The surrogate+EP ablation in Table 7 differs from AL-ERO not only in the indicator treatment but also in the alignment regularizer and the optimizer's batch size (full-batch for AL-ERO vs mini-batch for surrogate+EP in the released code), and all released surrogate+EP configurations use α=1\alpha=1, which keeps the soft selection weights within roughly [0.378, 0.622] over the nominal score range. With Table 7 coverage gains of only 0.14 and 0.09 percentage points on two datasets, it is unclear how much of the gain comes from the reformulation rather than confidence shaping or optimization conditions.
  3. Some implementation details differ from the paper's description: the code uses a ratio-based error penalty rather than Algorithm 1's stated form, adaptively increases the alignment weight rather than treating γ\gamma as a fixed input, and decreases some consistency weights. These may be reasonable choices, but the paper and code should agree on which procedure produced the results.
  4. Fixing the threshold at 0.5 relies on a linear bias absorbing the threshold shift, but that argument does not transfer to maximum softmax probability in a multiclass classifier: a common logit shift leaves probabilities unchanged, while class-specific shifts can change predictions.
Questions for authors
  1. Theorem 3.3 explicitly restricts the candidate solutions to non-singular points; is global optimality also taken over a non-singular feasible domain? Please make this explicit and report, for the deployed checkpoints, the hard-constraint diagnostics the code already computes, including coverage, error, residuals, and selection discrepancies.
  2. Could you compare AL-ERO with a matched full-batch surrogate+EP baseline, a wider sharpness search, and a separate ablation of the alignment regularizer?
  3. Please reconcile Algorithm 1 and Eq. (3.10) with the released code, and state what the ±\pm values in the tables represent.
  4. How does the fixed 0.5 threshold interact with maximum softmax probability in the multiclass setting, and does a learned-threshold version perform comparably?
5 comments

Weakness 1 / Question 1 — Feasible Domain in Theorem 3.3

Thank you for pointing out this! Yes, global optimality is intended to be taken over the non-singular feasible domain. We intend to establish the equivalence over the non-singular feasible sets---for both the original problem in Eq. (3.1) and the reformulated problem in Eq. (3.6), as explicitly motivated by Lemma 3.1 and Definition 3.2. However, when we stated the results in the original submission, there were technical glitches. In fact, in our proof of Theorem 3.3, we have implicitly assumed the Eq. (3.6) is over the non-singular feasible set—exactly as suggested by the reviewer; for Eq. (3.1), it is easy to show that there always exist non-singular global solutions—which we also took for granted. These hidden facts/assumptions have caused the current glitches.

In our revision, we will

  • Write down the detailed argument that there exist non-singular global solutions for Eq. (3.1);
  • Add explicit constraints (ci(θ)≠t,∀i∈I)(c_i(\boldsymbol\theta)\neq t,\forall i\in\mathcal I) to Eq (3.6) to rule out singular global solutions;
  • Correct the statement of Theorem 3.3 as follows:

Theorem 3.3 (Global equivalence over the non-singular feasible sets). Any (θ∗,t∗,s∗)({\boldsymbol\theta }^{\ast},{t}^{\ast},{s}^{\ast}) is a global solution to Eq. (3.6) if and only if (θ∗,t∗)({\boldsymbol\theta }^{\ast},{t}^{\ast}) is a non-singular global solution to Eq. (3.1).

With singular points excluded from both problems, the proof in Appendix A.3 holds without modification.

Weakness 1 — Regularizer Does Not Rule Out Singularity

One of the goals of the regularization is to promote non-singularity. We agree that it adds a term to the objective and does not strictly enforce non-singularity; it is difficult to enforce the non-equality constraint, i.e., the non-singularity constraints ci(θ)≠t,∀i∈I{c}_{i}(\boldsymbol\theta )\ne t , \forall i\in\mathcal I, because they induce an open set that can cause various conceptual and numerical difficulties—typical numerical optimization methods deal with closed and bounded constraint sets only.

To check the singularity numerically in practice, as suggested, we obtain statistics of the proportion of training samples that become almost singular, i.e., with ∣ci−0.5∣<10−6|c_i-0.5|<10^{-6}, at both the selected and the last checkpoints, in the following table:

Dataset (Target Error)∥ci−0.5∥<10−6\|c_i-0.5\|<10^{-6}, Selected (Mean ± SD)∥ci−0.5∥<10−6\|c_i-0.5\|<10^{-6}, Last (Mean ± SD)
OrganACSMNIST (5%)0.854% ± 0.199%0.014% ± 0.013%
OrganACSMNIST (10%)0.566% ± 0.491%0.008% ± 0.008%
PathMNIST (5%)0.684% ± 0.203%0.004% ± 0.009%
PathMNIST (10%)0.356% ± 0.226%0.000% ± 0.000%
PubMed 200k RCT (5%)1.808% ± 0.206%0.324% ± 0.034%
PubMed 200k RCT (10%)0.226% ± 0.080%0.022% ± 0.019%
20 Newsgroups (5%)0.188% ± 0.059%0.136% ± 0.061%
20 Newsgroups (10%)1.068% ± 0.518%0.016% ± 0.017%

The extremely low fraction of (numerically) singular points suggest that the regularizer is effective in preventing almost all the training samples from singular points, although not all. Moreover, we observe a substantial decrease of the singularity rate from the checking point to the final point in general.

Weakness 1 / Question 1 — Hard-Constraint Diagnostics

As requested, we report the hard-constraint diagnostics on the training set of the first TBAL round, at both the deployed (i.e., selected) checkpoint and, for reference, the last checkpoint (the final outer iteration):

  • coverage and error of the hard-threshold selection on training set at the training threshold t=0.5t=0.5, from which the residual of the error constraint can be read directly;
  • selection discrepancy, i.e., the proportion of training samples whose auxiliary selection (si>0.5{s} _{i}>0.5) differs from the hard-threshold selection (ci(θ)>0.5{c} _{i}(\boldsymbol\theta)>0.5), which reflects the violation of the indicator constraints;
  • residual of the indicator constraints, reported as the proportion of training samples whose residual ∣si+[si+ci−t−1]+−[si+ci−t]+∣\left|s_i + [s_i + c_i - t - 1] _+ - [s_i + c_i - t] _+\right| is strictly smaller than 10−610^{-6}.

Each entry is mean ± SD over five seeds; coverage and error are in %.

Dataset (Target Error)Coverage / Error (Selected Checkpoint)Coverage / Error (Last Checkpoint)
OrganACSMNIST (5%)94.73 ± 0.16 / 6.16 ± 0.1593.94 ± 0.09 / 4.31 ± 0.03
OrganACSMNIST (10%)98.75 ± 0.95 / 6.93 ± 0.5299.75 ± 0.03 / 7.44 ± 0.10
PathMNIST (5%)98.19 ± 0.32 / 3.95 ± 0.1399.17 ± 0.03 / 4.02 ± 0.01
PathMNIST (10%)98.77 ± 0.85 / 3.01 ± 0.34100.00 ± 0.01 / 3.41 ± 0.03
PubMed 200k RCT (5%)92.31 ± 0.06 / 9.90 ± 0.1092.80 ± 0.08 / 8.90 ± 0.12
PubMed 200k RCT (10%)91.79 ± 2.97 / 10.93 ± 1.6094.92 ± 0.09 / 9.09 ± 0.09
20 Newsgroups (5%)74.44 ± 0.23 / 5.59 ± 0.2873.85 ± 0.11 / 4.77 ± 0.08
20 Newsgroups (10%)83.78 ± 0.34 / 9.49 ± 0.1783.30 ± 0.06 / 8.82 ± 0.04
Dataset (Target Error)Mismatch Rate (Selected Checkpoint)Mismatch Rate (Last Checkpoint)
OrganACSMNIST (5%)4.022% ± 0.389%0.274% ± 0.027%
OrganACSMNIST (10%)1.010% ± 0.910%0.016% ± 0.005%
PathMNIST (5%)1.368% ± 0.290%0.004% ± 0.009%
PathMNIST (10%)1.252% ± 0.861%0.000% ± 0.000%
PubMed 200k RCT (5%)9.388% ± 0.112%5.460% ± 0.138%
PubMed 200k RCT (10%)5.992% ± 4.809%0.456% ± 0.103%
20 Newsgroups (5%)1.572% ± 0.829%0.180% ± 0.020%
20 Newsgroups (10%)1.540% ± 0.152%0.204% ± 0.036%
Dataset (Target Error)Residual <10−6<10^{-6} (Selected Checkpoint)Residual <10−6<10^{-6} (Last Checkpoint)
OrganACSMNIST (5%)92.766% ± 0.505%99.680% ± 0.027%
OrganACSMNIST (10%)95.304% ± 4.889%99.798% ± 0.008%
PathMNIST (5%)96.204% ± 0.543%99.982% ± 0.019%
PathMNIST (10%)82.890% ± 13.218%99.984% ± 0.005%
PubMed 200k RCT (5%)48.584% ± 1.731%92.650% ± 0.171%
PubMed 200k RCT (10%)88.880% ± 9.018%99.434% ± 0.157%
20 Newsgroups (5%)92.476% ± 1.746%96.252% ± 0.122%
20 Newsgroups (10%)95.224% ± 0.215%99.696% ± 0.065%

The solver itself is able to solve the training problem: as training proceeds, the mismatch rate decreases and the proportion of samples with residuals below 10−610^{-6} increases in all eight settings, except for the PubMed 200k RCT (5%) where the training did not fully converged. We attribute this to the training configuration shared across all datasets (the hyperparameter search space, the training budget, and the penalty-update schedule), which is not optimal for this dataset; we kept it identical across datasets for a fair comparison.

While we can attain a feasible solution in all but one case on the training set, the trained model and threshold is not guaranteed to be feasible on the unlabeled data, because of generalization gap. Therefore, following the TBAL protocol, for all methods, the checkpoint is selected by its coverage on DthD_{\text{th}} under the re-estimated deployment threshold to control the error on the unlabeled data.

Moreover, unlike a sigmoid surrogate, whose approximation gap is built into the objective, the reformulated problem itself is exact over the non-singular feasible set; hence, the solver directly optimizes the hard-threshold coverage and error.

Weakness 2 / Question 2 — Matched Comparison with AL-Surrogate+EP

Thank you for pointing this out!

First, both AL-Surrogate+EP and our AL-ERO try to solve the canonical joint classification-selection formulation in Eq. (3.1). Our AL-ERO involves exact reformulation of indicator functions and regularization that attempts to avoid the singular case in the reformulation. By contrast, AL-Surrogate+EP smooths indicator functions directly. The regularization term in our AL-ERO is unique to the reformulation idea, and there is no direct match in AL-Surrogate+EP.

Second, as suggested, we match the other settings of AL-ERO and AL-Surrogate+EP for a fairer comparison during the rebuttal. Below, we use the original experiment settings unless declared otherwise.

  • batch strategy: full-batch, deterministic optimization; in each TBAL round, after the shared CE warm-up, AL-Surrogate+EP runs the same number of outer and inner full-batch steps on DtrainD_{\text{train}} as AL-ERO;
  • checkpoint rule: as in AL-ERO, the checkpoint is selected by the deployment-threshold coverage on DthD_{\text{th}}, and the deployment threshold is estimated on DthD_{\text{th}};
  • α\alpha grid: in the original experiments, we fixed α=1\alpha=1 and swept the learning rate, initial error weight, and CE weight, so that both methods had a similar search budget. We now additionally search α∈{0.1,1,5,10}\alpha \in \{0.1, 1, 5, 10\}, selected on DhypD_{\text{hyp}} by the same rule as for all other methods (highest mean coverage among values whose mean error is within the target).

As shown in the table below (mean ± SD over five random seeds), under the matched setup, AL-ERO achieves equal or higher mean coverage than AL-Surrogate+EP in seven of eight settings, with all mean errors below the target error.

Dataset (Target Error)Selected α\alphaAL-Surrogate+EP (Coverage / Error)AL-ERO (Coverage / Error)
OrganACSMNIST (5%)553.84 ± 0.39 / 3.76 ± 0.0753.59 ± 0.18 / 3.62 ± 0.05
OrganACSMNIST (10%)568.04 ± 0.11 / 8.51 ± 0.0468.15 ± 0.34 / 8.52 ± 0.12
PathMNIST (5%)173.05 ± 0.17 / 3.79 ± 0.0473.45 ± 0.09 / 3.94 ± 0.04
PathMNIST (10%)0.183.19 ± 0.00 / 8.25 ± 0.0283.19 ± 0.01 / 8.31 ± 0.05
PubMed 200k RCT (5%)135.63 ± 0.42 / 4.28 ± 0.0537.22 ± 2.16 / 4.39 ± 0.25
PubMed 200k RCT (10%)162.53 ± 0.38 / 8.54 ± 0.0562.95 ± 0.71 / 8.48 ± 0.08
20 Newsgroups (5%)123.12 ± 0.02 / 4.62 ± 0.0123.13 ± 0.42 / 4.89 ± 0.20
20 Newsgroups (10%)530.59 ± 0.03 / 9.39 ± 0.0231.11 ± 0.09 / 9.72 ± 0.10

In our revision, we will update the ablation study in Appendix D.1 (Table 7) with this more matched comparison.

Weakness 3 / Question 3 — Reconciling Algorithm 1 and Eq. (3.10) with the Released Code, and the Meaning of ±\pm

Thank you for pinpointing the discrepancy! All our reported experimental results are generated using the released implementation. We purposefully make our mathematical presentations of these points simplified versions of the actual implementation to minimize distraction by low-level details while keeping the mathematical essence. We will make this clear in our revised version.

Definition of ±\pm. In all tables, ±\pm denotes the sample standard deviation.

Weakness 4 / Question 4 — Fixing tt

Thanks for raising this excellent point! Empirically, fixing t=0.5t =0.5 and leaving tt learnable lead to very close results, as shown in the table below.

Dataset (Target Error)Learned tt (Coverage / Error)Fixed tt (Coverage / Error, Table 1)
OrganACSMNIST (5%)53.30 ± 0.27 / 3.62 ± 0.0453.59 ± 0.18 / 3.62 ± 0.05
OrganACSMNIST (10%)68.18 ± 0.10 / 8.54 ± 0.0468.15 ± 0.34 / 8.52 ± 0.12
PathMNIST (5%)73.98 ± 0.20 / 4.16 ± 0.0673.45 ± 0.09 / 3.94 ± 0.04
PathMNIST (10%)83.20 ± 0.00 / 8.29 ± 0.0183.19 ± 0.01 / 8.31 ± 0.05
PubMed 200k RCT (5%)36.29 ± 1.63 / 4.31 ± 0.1637.22 ± 2.16 / 4.39 ± 0.25
PubMed 200k RCT (10%)63.52 ± 1.19 / 8.52 ± 0.1062.95 ± 0.71 / 8.48 ± 0.08
20 Newsgroups (5%)22.66 ± 0.31 / 4.66 ± 0.2023.13 ± 0.42 / 4.89 ± 0.20
20 Newsgroups (10%)31.10 ± 0.06 / 9.84 ± 0.0531.11 ± 0.09 / 9.72 ± 0.10

So, in our original version, we focused on the t=0.5t=0.5 version for simplicity, alongside the handwavy equivalence argument. After careful checking, we agree with the reviewer that there are bugs in the argument. In our revised version, we will include both versions of results in all experiments.

Reviewer_TA7LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

The paper studies threshold-based auto-labeling (TBAL): maximizing automatically labeled coverage under an accepted-label error constraint and a human-label budget. AL-ERO trains a linear head on frozen CLIP or Qwen3 embeddings using a coverage objective and cross-entropy supervision. Following ERO (Peng et al., 2025), it represents each selection indicator through an auxiliary variable and a piecewise-linear equality constraint, and presents a global-equivalence result under a non-singularity condition. Training combines BCE alignment regularization with an exact-penalty scheme, fixes the training threshold at 0.5, and subsequently estimates a deployment threshold on a labeled set DthD_{\mathrm{th}}. Across four image/text datasets and target errors of 5% and 10%, AL-ERO reports the highest mean coverage among the evaluated baselines, with mean errors below target. Ablations examine the CE term, a surrogate-plus-exact-penalty comparator and alternative encoders.

Critical strengths

1. A well-motivated objective for an operational problem. The method directly targets coverage under an error constraint, rather than relying on classification accuracy or calibration alone. Training the classifier for this operating point without reserving training labels for a separate selector is a sensible way to use a limited annotation budget.

2. Consistent empirical gains against a relevant baseline set. Table 1 reports the highest mean coverage for AL-ERO in all eight dataset/target settings, with mean errors below target; for example, coverage on PubMed at 5% is 37.22% versus 31.61% for AURC. The CE ablation (Table 2), post-hoc comparisons (Table 3) and alternative medical encoders (Table 8) provide useful supporting evidence. Shared CE initialization for the direct-training methods, common human-label quotas and released code also facilitate comparison.

Critical weaknesses

1. The global-equivalence claim in Theorem 3.3 is false over the displayed feasible domain.

  • Eq. (3.6) does not exclude singular points: when ci=tc_i = t, the equality holds for every si∈[0,1]s_i \in [0, 1], allowing label-dependent choices among tied examples that no threshold can realize. Consider binary classification with ε=0.1\varepsilon = 0.1. Group A contains ten copies of x=log⁡9x = \log 9 (nine labels 0, one label 1), and group B contains three copies of x=log⁡2x = \log 2 (two labels 0, one label 1). A linear-softmax head with logits (x,0)(x, 0) attains the global CE minimum at confidences 0.9 and 23\frac{2}{3}. Any hard threshold must select whole groups; accepting B alone or both groups exceeds the error cap. This head with t=0.8t = 0.8 therefore achieves both maximal feasible coverage, 1013\frac{10}{13}, and minimal CE, making it a non-singular global optimum of Eq. (3.1) for any β>0\beta > 0. In Eq. (3.6), the same head with t=23t = \frac{2}{3} can accept all of A and only B's two correct points: all constraints hold, coverage is 1213\frac{12}{13}, error is 112\frac{1}{12} and CE is unchanged. This contradicts the implication from an original global optimum to a reformulated global optimum. Appendix A.3, L788–790, establishes equality only of the non-singular feasible subsets.
  • Explicitly restricting the feasible domain could repair the equivalence statement, but would not by itself justify the implemented solver. The ψ\psi regularizer does not enforce non-singularity. For sufficiently small δ>0\delta > 0, setting t=23+δt = \frac{2}{3} + \delta in the construction above gives residual δ\delta at each of B's two auxiliary-selected points, while auxiliary coverage still exceeds hard coverage by 213\frac{2}{13}. Thus, small penalty residuals alone do not establish agreement with hard-threshold selection. Training also fixes t=0.5t = 0.5 using a bias argument (L373–375) that is not generally valid for maximum-softmax confidence. Moreover, the supplied code penalizes a normalized error violation and conditionally updates penalty weights, unlike Eq. (3.10) and Algorithm 1 (classifier_ero.py, L527–540 and L1191–1212). The abstract's claim of "eliminating the approximation gap" is therefore not substantiated for the implemented training procedure.

2. The experiments do not isolate gains from the exact reformulation.

  • Against the closest comparator, AL-Surrogate+EP (Table 7), AL-ERO's mean coverage advantage is 0.09–0.89 percentage points in six of eight settings. The larger gains are on OrganACSMNIST at 10% (+1.71 points) and PubMed at 5% (+6.19 points; 37.22±2.1637.22 \pm 2.16 versus 31.03±0.5131.03 \pm 0.51). These results do not isolate indicator treatment as claimed in Appendix D.1: the supplied implementation uses the full training set at every AL-ERO inner step and an adaptive BCE alignment term, whereas Surrogate+EP uses 256-example minibatches, no alignment term and a sigmoid α\alpha fixed to 1 (run_full_loop.py, L104–105).
  • The reliability of these improvements also needs clarification. The meaning of "±\pm" is unspecified, all released configurations fix the data seed while varying model seeds, and DthD_{\mathrm{th}} is used for both threshold estimation and checkpoint selection (Appendix C.2.2). On 20 Newsgroups at the 5% target, the reported mean error is 4.89±0.204.89 \pm 0.20. Without a definition of the uncertainty and per-run results, this does not establish how often the target is exceeded.
Questions for authors
  1. Is non-singularity intended to restrict the entire feasible set of Eq. (3.6)? Please address the counterexample, state and prove a corrected result, and clarify whether an exactness guarantee applies to the implemented finite-penalty solver. Separately, report hard/auxiliary selection mismatches and minimum ∣ci−0.5∣\lvert c_i - 0.5\rvert margins at the selected checkpoints, evaluated at the training threshold t=0.5t = 0.5.
  2. Which objective and penalty-update schedule generated the tables? On a representative setting, does AL-ERO retain its advantage over Surrogate+EP after matching the batch regime, processed-example budget, alignment regularization and checkpoint rule, with a reasonable α\alpha grid for the surrogate?
  3. Please define "±\pm" and provide paired per-seed coverage/error values and the number of runs exceeding ε\varepsilon. Distinguish variation across model seeds from variation across data splits; a small set of independent data-split repeats would help assess the latter.
8 comments

Weakness 1/ Question 1 — 1st bullet point: technical bug in Theorem 3.3

Thank you for pointing out this! Your construction of the counterexample makes perfect sense, for our current theorem statement.

We intend to establish the equivalence over the non-singular feasible sets---for both the original problem in Eq. (3.1) and the reformulated problem in Eq. (3.6), as explicitly motivated by Lemma 3.1 and Definition 3.2. However, when we stated the results in the original submission, there were technical glitches. In fact, in our proof of Theorem 3.3, we have implicitly assumed the Eq. (3.6) is over the non-singular feasible set—exactly as suggested by the reviewer; for Eq. (3.1), it is easy to show that there always exist non-singular global solutions—which we also took for granted. These hidden facts/assumptions have caused the current glitches.

In our revision, we will Write down the detailed argument that there exist non-singular global solutions for Eq. (3.1); Add explicit constraints (ci(θ)≠t,∀i∈I)(c_i(\boldsymbol\theta)\neq t,\forall i\in\mathcal I) to Eq (3.6) to rule out singular global solutions; Correct the statement of Theorem 3.3 as follows:

Theorem 3.3 (Global equivalence over the non-singular feasible sets). Any (θ∗,t∗,s∗)(\boldsymbol\theta^{\ast}, t^{\ast}, s^{\ast}) is a global solution to Eq. (3.6) if and only if (θ∗,t∗)(\boldsymbol\theta^{\ast}, t^{\ast}) is a non-singular global solution to Eq. (3.1).

Weakness 1 / Question 1 — 2nd bullet point: Regularizer doesn’t enforce non-singularity

One of the goals of the regularization is to promote non-singularity. We agree it does not strictly enforce it; it is difficult to enforce the non-equality constraint, i.e., the non-singularity constraints ci(θ)≠t,∀i∈I{c}_{i}(\boldsymbol\theta )\ne t ,\forall i\in\mathcal I, because they induce an open set that can cause various conceptual and numerical difficulties—typical numerical optimization methods deal with closed and bounded constraint sets only.

To check the singularity numerically in practice, as suggested, we obtain statistics of the proportion of training samples that become almost singular, i.e., with ∣ci−0.5∣<10−6|c_i-0.5|<10^{-6}, at both the selected and the last checkpoints, in the following table:

Dataset (Target Error)∥ci−0.5∥<10−6\|c_i-0.5\|<10^{-6}, Selected (Mean ± SD)∥ci−0.5∥<10−6\|c_i-0.5\|<10^{-6}, Last (Mean ± SD)
OrganACSMNIST (5%)0.854% ± 0.199%0.014% ± 0.013%
OrganACSMNIST (10%)0.566% ± 0.491%0.008% ± 0.008%
PathMNIST (5%)0.684% ± 0.203%0.004% ± 0.009%
PathMNIST (10%)0.356% ± 0.226%0.000% ± 0.000%
PubMed 200k RCT (5%)1.808% ± 0.206%0.324% ± 0.034%
PubMed 200k RCT (10%)0.226% ± 0.080%0.022% ± 0.019%
20 Newsgroups (5%)0.188% ± 0.059%0.136% ± 0.061%
20 Newsgroups (10%)1.068% ± 0.518%0.016% ± 0.017%

The extremely low fraction of (numerically) singular points suggest that the regularizer is effective in preventing almost all the training samples from singular points, although not all. Moreover, we observe a substantial decrease of the singularity rate from the checking point to the final point in general.

Weakness 1 / Question 1 — 2nd bullet point: small penalty residuals alone do not establish agreement with hard-threshold selection

It is true that our theoretical result ∣si+[si+ci−t−1]+−[si+ci−t]+∣=0|s _{i} + [s _{i} + c _{i} - t - 1] _{+} - [s _{i} + c _{i} - t] _{+}| = 0 implies that ∣si−1(ci>t)∣=0\left|s _{i} - \mathbf{1}(c _{i} > t)\right| = 0 in Lemma 3.1 requires equality. However, when the penalty residual is small but non-zero, i.e., ∣si+[si+ci−t−1]+−[si+ci−t]+∣=ε\left|s _{i} + [s _{i} + c _{i} - t - 1] _{+} - [s _{i} + c _{i} - t] _{+}\right| = \varepsilon for an ε>0\varepsilon > 0, it is unclear, a priori, its implication on ∣si−1(ci>t)∣\left|s _{i} - \mathbf{1}(c _{i} > t)\right|.

Through mathematical analysis, we can show that when ε≤∣ci−t∣\varepsilon \le |c _i-t|, ∣si+[si+ci−t−1]+−[si+ci−t]+∣=ε\left|s _i + [s _i + c _i - t - 1] _+ - [s _i + c _i - t] _+\right| = \varepsilon implies that ∣si−1(ci>t)∣=ε\left|s _i - \mathbf{1}(c _i > t)\right| = \varepsilon. The proof is as follows. Write x≐∣si+[si+ci−t−1]+−[si+ci−t]+∣x \doteq \left|s _i + [s _i + c _i - t - 1] _+ - [s _i + c _i - t] _+\right| and y≐∣si−1(ci>t)∣y \doteq \left|s _i - \mathbf{1}(c _i > t)\right|. We have

Lemma. For all si,ci,t∈[0,1],x=min⁡(y, ∣ci−t∣)s_i, c_i, t \in [0,1], x = \min\left(y,\ |c_i - t|\right).

Proof. Let d=ci−td = c_i - t. Since [u−1]+−[u]+=−clip(u,0,1)[u-1] _+ - [u] _+ = -\mathrm{clip}(u, 0, 1), we have x=∣si−clip(si+d,0,1)∣x = \left|s_i - \mathrm{clip}(s_i + d, 0, 1)\right|. If ci>tc_i > t, then y=1−siy = 1 - s_i. Since si+d>0,x=∣si−min⁡(si+d,1)∣=min⁡(d,1−si)=min⁡(∣d∣,y)s_i + d > 0, x = \left|s_i - \min(s_i + d, 1)\right| = \min(d, 1 - s_i) = \min(|d|, y). If ci≤tc_i \le t, then y=siy = s_i. Since si+d≤1,x=∣si−max⁡(si+d,0)∣=min⁡(−d,si)=min⁡(∣d∣,y).■s_i + d \le 1, x = \left|s_i - \max(s_i + d, 0)\right| = \min(-d, s_i) = \min(|d|, y). \blacksquare

Theorem. Let ε>0\varepsilon > 0 and ∣ci−t∣>ε|c_i - t| > \varepsilon. Then x≤εx \le \varepsilon if and only if y≤εy \le \varepsilon, and in this case x=yx = y.

Proof. By the lemma, x=min⁡(y,∣ci−t∣)x = \min(y, |c_i - t|) with ∣ci−t∣>ε|c_i - t| > \varepsilon. If x≤εx \le \varepsilon, then x≠∣ci−t∣x \ne |c_i - t|, so x=yx = y; in particular y≤εy \le \varepsilon. If y≤εy \le \varepsilon, then y<∣ci−t∣y < |c_i - t|, so x=min⁡(y,∣ci−t∣)=yx = \min(y, |c_i - t|) = y; in particular x≤ε.■x \le \varepsilon. \blacksquare

We also confirm our results empirically: we observe that for any sample with a reformulation residual below their corresponding margin, the selector residual equals the reformulation residual. Moreover, the vast majority of samples fall in this regime, as shown in the table below.

Dataset (Target Error)Residual < Margin at Selected Checkpoint (Mean ± SD)Residual < Margin at Last Checkpoint (Mean ± SD)
OrganACSMNIST (5%)92.272% ± 0.496%99.718% ± 0.035%
OrganACSMNIST (10%)96.516% ± 3.451%99.884% ± 0.026%
PathMNIST (5%)95.720% ± 0.489%99.980% ± 0.019%
PathMNIST (10%)93.536% ± 4.879%100.000% ± 0.000%
PubMed 200k RCT (5%)71.148% ± 0.701%92.808% ± 0.180%
PubMed 200k RCT (10%)89.248% ± 8.625%99.494% ± 0.118%
20 Newsgroups (5%)93.500% ± 1.679%97.724% ± 0.113%
20 Newsgroups (10%)94.300% ± 0.712%99.700% ± 0.040%

For the small fraction of samples that fall outside, indeed, their small reformation residuals may or may not lead to small selector residuals. But since their proportion is sufficiently small, their impact on the risk and coverage—the key quantities in the constrained optimization—is limited.

Weakness 1 / Question 2 — 2nd bullet point: misalignment of implementation and mathematics in Eq. (3.10) in algorithm 1

Thank you for pinpointing the discrepancy! All our reported experimental results are generated using the released implementation. We purposefully make our mathematical presentations of these points simplified versions of the actual implementation to minimize distraction by low-level details while keeping the mathematical essence. We will make this clear in our revised version.

Weakness 1 — 2nd bullet point: Fixing t=0.5t=0.5

Thanks for raising this excellent point! Empirically, fixing t=0.5t =0.5 and leaving tt learnable lead to very close results, as shown in the table below.

Dataset (Target Error)Learned tt (Coverage / Error)Fixed tt (Coverage / Error, Table 1)
OrganACSMNIST (5%)53.30 ± 0.27 / 3.62 ± 0.0453.59 ± 0.18 / 3.62 ± 0.05
OrganACSMNIST (10%)68.18 ± 0.10 / 8.54 ± 0.0468.15 ± 0.34 / 8.52 ± 0.12
PathMNIST (5%)73.98 ± 0.20 / 4.16 ± 0.0673.45 ± 0.09 / 3.94 ± 0.04
PathMNIST (10%)83.20 ± 0.00 / 8.29 ± 0.0183.19 ± 0.01 / 8.31 ± 0.05
PubMed 200k RCT (5%)36.29 ± 1.63 / 4.31 ± 0.1637.22 ± 2.16 / 4.39 ± 0.25
PubMed 200k RCT (10%)63.52 ± 1.19 / 8.52 ± 0.1062.95 ± 0.71 / 8.48 ± 0.08
20 Newsgroups (5%)22.66 ± 0.31 / 4.66 ± 0.2023.13 ± 0.42 / 4.89 ± 0.20
20 Newsgroups (10%)31.10 ± 0.06 / 9.84 ± 0.0531.11 ± 0.09 / 9.72 ± 0.10

So, in our original version, we focused on the t=0.5t=0.5 version for simplicity, alongside the handwavy equivalence argument. After careful checking, we agree with the reviewer that there are bugs in the argument. In our revised version, we will include both versions of results in all experiments.

Summary of Weakness 1: Thanks to the reviewer’s very detailed comments! After preparing the above rebuttals, we realize it is inappropriate to say that our method "eliminates the approximation gap", although it tries to minimize the approximation gap in many senses. In our revision, we will tune down this claim accordingly.

Reviewer_9ephOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes AL-ERO, a method for threshold-based auto-labeling that directly trains a selective classifier to maximize auto-labeling coverage while keeping the error rate below a user-specified threshold. AL-ERO jointly optimizes classification and confidence-based selection. A key challenge is that coverage and auto-labeling error depend on hard indicator functions, which have zero gradients almost everywhere. AL-ERO addresses this using an exact reformulation of the selection indicator, following the ERO framework, and optimizes the resulting constrained problem using an exact penalty method.

Critical strengths

The paper addresses a clear and practically relevant problem. For auto-labeling, classifier accuracy alone is not sufficient; confidence scores should separate correct and incorrect predictions well enough to maximize coverage while maintaining a desired error rate.

Critical weaknesses

The methodological novelty appears relatively limited. The main mathematical technique appears to be directly adapted from the ERO framework. Furthermore, while the authors present the lack of joint classification–selection optimization as a limitation of Colander that this paper aims to address, the broader idea of jointly optimizing classification and selection does not appear to be new in the literature. The more specific contribution appears to be adapting this principle to the TBAL objective of maximizing auto-labeling coverage under an explicit error constraint.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Clarification of Novelty and Contributions

We thank the reviewer for the positive assessment and the feedback on novelty.

  • We agree that our main mathematical ideas for indicator reformulation and exact-penalty-based optimization are borrowed from ERO (Peng et al., 2025)---as clearly stated in the paper. But their focus problems, recall at a fixed precision (FPOR), precision at a fixed recall (FROP), and the F1F_1 score (OFOS), are totally different from ours.
  • We also agree that joint classifier-selector optimization has been studied in the selective classification (SC) literature; they are often called in-training SC methods. However, formulating the selector part as risk-controlled coverage maximization is very rare—we only find one existing paper: Gangrade et al., 2021; the well-known SelectiveNet (Geifman & El-Yaniv, 2019) uses a coverage-controlled risk minimization formulation, which is different in motivation. But, Gangrade et al. take the surrogate approach to deal with the indicator function. We hope to compare with Gangrade et al., but we cannot find a faithful implementation of their algorithm, despite their GitHub release with an incomplete version.
  • Our goal of the paper is to develop an effective autolabeling framework. We achieve this by carefully integrating three lines of ideas: classifier-selector joint optimization, risk-constrained coverage maximization, and exact indicator formulation, which advances the state-of-the-art represented by Colander (Vishwakarma et al., 2024a).

We will revise the paper to make all these clear.

References

  • Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statistics, pp. 2179–2187. PMLR, 2021.
  • Yonatan Geifman and Ran El-Yaniv. SelectiveNet: A deep neural network with an integrated reject option. In International Conference on Machine Learning, pp. 2151–2159. PMLR, 2019.
  • Le Peng, Yash Travadi, Chuan He, Ying Cui, and Ju Sun. Exact reformulation and optimization for direct metric optimization in binary imbalanced classification. arXiv preprint arXiv:2507.15240, 2025.
  • Harit Vishwakarma, Yi Chen, Sui Jiet Tay, Satya Sai Srinath Namburi, Frederic Sala, and Ramya Korlakai Vinayak. Pearls from pebbles: Improved confidence functions for auto-labeling. Advances in Neural Information Processing Systems, 37:15983–16015, 2024a.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.