acceptodds
Under review as a conference paper at ICLR 2027

Group-Local Anchoring for Subgroup Intervals in Programmatic Weak Supervision

Abstract

Programmatic weak supervision uses approximate labeling functions to construct training labels. Evaluating these labels requires intervals for class proportions within subgroups of an unlabeled pool. An existing approach derives such intervals from pool-wide statistics estimated using a small labeled sample. These global constraints can provide simultaneous coverage while leaving small-group intervals equal to the full probability range, because changes within a small group have little effect on pool-wide statistics. We study how labels reaching a subgroup determine interval precision. For binary labels, we derive a width lower bound from the pool-to-group size ratio and the room for variation allowed by the global constraints. This identifies a computable group-size threshold below which intervals must span the full probability range. Motivated by this bound, group-local anchoring adds constraints estimated from labeled observations within each group. Joint calibration preserves simultaneous coverage when group selection is independent of the estimation labels. For groups learned from labels, we use disjoint samples for selection and estimation. With 400 labeled draws from the pool, median global-to-anchored width ratios range from 1.21 to 1.69 across five text corpora, with gains concentrated in groups containing at least 32 pool instances. Experiments on text and image corpora further show that reusing selection labels for estimation produces coverage failures concentrated in selectors that memorize labels for individual labeling-function patterns. In the image experiments, independent estimation reduces this undercoverage. Median split-to-reference width ratios are 1.42–1.98, where the reference uses all acquired labels on the same groups. These results link subgroup precision to local labeling information and support independent estimation for learned groups.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
programmatic weak supervision, labeling functions, partial identification, subgroup uncertainty quantification, simultaneous confidence intervals, sample splitting, selective inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ijhpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies simultaneous confidence intervals for subgroup label proportions in a fixed finite pool under programmatic weak supervision. It establishes a lower bound on the width induced by a global moment uncertainty set, using the pool-to-group size ratio and a computable center slack. It then adds group-local Clopper-Pearson anchors, allocates confidence jointly, and uses independent selection and estimation samples when labels determine the anchor groups.

Critical strengths
  1. The paper presents a clear research focus.
  2. The paper provides some essential experiments.
Critical weaknesses
  1. The background and objectives of programmatic weak supervision are severely lacking in the introduction.
  2. The descriptions in the paper are not sufficiently rigorous.
  3. The issues identified in this field are not supported by any references, making it difficult to verify their validity.
  4. The paper contains numerous typographical errors, such as those in the symbolic expressions in lines 145 and 146.
Questions for authors
  1. Why do the results presented in lines 54–58 motivate the proposal of group-local anchoring?
  2. What is the core idea of the paper?
  3. What is the key motivation behind the paper?
  4. Why is the group-locally anchored uncertainty set defined in this way in Definition 5?
4 comments

Thank you for your review. We clarify the argument from problem to construction: valid pool-wide statistics can leave a subgroup's proportion undetermined, so group-local anchoring adds a direct confidence constraint for that group. Intersecting it with the global set retains the available evidence while preserving simultaneous coverage.

What problem does programmatic weak supervision solve?

Programmatic weak supervision uses labeling functions, such as keyword rules, to assign training labels when manual annotation is limited. Because a rule can be reliable on one subgroup and unreliable on another, its overall performance does not determine its precision on the subgroup where it will be used. Our objective is to estimate these subgroup proportions with simultaneous confidence intervals. For a rule predicting class y on group I, the target proportion is exactly that rule's precision on I.

The background comes from established weak-supervision research. Data programming and Snorkel (Ratner et al., 2016, 2017) motivate combining imperfect labeling functions. More directly, Álvarez et al. (2025) construct confidence intervals from global labeling-function statistics. We study the subgroup resolution of that construction. Our theorem then establishes when this global construction leaves a subgroup's proportion completely unresolved.

Why do lines 54–58 motivate local anchoring?

Lines 54–58 motivate local anchoring because global constraints can leave a small group's proportion completely unresolved, even when the pool-wide statistics are valid. Writing rho = n/|I| and s* for the largest feasible minimum slack in the global constraints, Theorem 3 gives W_I >= min(1, rho s*). Hence, when |I| <= n s*, the interval necessarily spans [0,1]. Changing labels throughout that group can remain compatible with the global statistics because the group contributes only |I|/n of each pool average.

Local anchoring removes this dilution by estimating the proportion within I itself. Its constraint is normalized by |I|, so an observed label in the group directly informs the requested quantity. This explains the progression from the width theorem to the proposed construction: first identify why global information cannot resolve the group, then supply a statistically calibrated measurement of that group's proportion.

Why is Definition 5 an intersection of constraints?

Definition 5 uses an intersection to add subgroup information while preserving both global evidence and simultaneous coverage. For binary labels, let p_i be a candidate positive-label probability and let [L_I,H_I] be the group's confidence interval. The local row is

L_I <= (1/|I|) sum_{i in I} p_i <= H_I.

Intersecting these rows with the original global set preserves the information already available. Moreover, if every row covers its true moment, the true assignment p_i = 1{y_i = 1} satisfies the entire intersection. Optimizing any subgroup proportion over that set therefore covers its truth. Allocating failure probabilities with delta_global + sum_I delta_I <= delta proves simultaneous coverage, including for overlapping groups. No independence between the different rows is required.

This construction also explains the observed precision improvement. With 400 labeled draws, the original five-corpus experiment gives median global-to-anchored width ratios of 1.77–2.41 for groups of at least 32 instances. Thus the added local measurements provide information that the pool averages alone cannot supply.

Rigorous notation and readable presentation

We clarify the notation around lines 145–146 by separating the fixed-pool target, candidate probabilities, and sampling randomness. The target is the fixed-pool proportion p*I(y) = |I|^{-1} sum{i in I} 1{y_i = y}, where 1{.} denotes an indicator. A consistent class convention is Y = {0,...,K-1}, with Y = {0,1} in the binary analysis. The empirical estimate uses independent uniform draws with replacement, and the label budget counts draws, including repeated positions. This separates the quantity being estimated from how labels are sampled.

The explanation is organized around one question at each step: what subgroup quantity is needed, why global constraints cannot resolve it, what local measurement supplies, and why joint calibration preserves coverage. This order makes the core idea accessible before introducing the optimization notation and keeps each formula attached to the problem it solves.

There are still many issues that have not been addressed in your responses. If you can provide more responses to address my concerns, I will raise my score.

Thank you for your follow-up. We show when global statistics leave subgroup reliability undetermined and how local label counts constrain it with simultaneous coverage.

W1, Q2–Q3: PWS background, motivation and core idea

Programmatic weak supervision uses labeling functions, such as keyword rules, to supply training labels for an unlabeled pool. Our input is the labeling-function output matrix and a small gold-label sample. Our output is a confidence interval for each queried group's class proportion. For a rule predicting the positive class whenever a keyword appears, this proportion is its precision, indicating whether its weak labels meet a desired reliability level.

The motivation is that valid pool-wide statistics can leave this precision anywhere in [0,1][0,1]. Our core idea is to retain group-specific counts from the same gold sample and require candidate label assignments to agree with both global and local evidence. Thus anchoring adds an estimated group constraint, rather than rescaling a global constraint.

W3: Which claims are supported by prior work?

Data Programming (Ratner et al., 2016) and Snorkel (Ratner et al., 2017) establish the PWS workflow. Álvarez et al. (2025) supply the global confidence-constraint construction we analyze. The sufficient condition for its subgroup interval to span [0,1][0,1] is our Theorem 3.

Q1: Why do lines 54–58 motivate local anchoring?

Local anchoring directly constrains proportions that global averages can leave unresolved. For example, let the pool size be n=1000n=1000, the group size ∣I∣=20|I|=20, and the only global constraint a positive-label mean in [0.48,0.52][0.48,0.52]. With outside-group probabilities fixed at 0.50.5, assigning the group entirely to either class gives pool means 0.490.49 and 0.510.51. Both satisfy the constraint, so the group's proportion can still span [0,1][0,1].

Writing WIW_I for interval width, Theorem 3 applies to every nonempty group under binary labels, a nonempty feasible set and features in [0,1][0,1]: WI≥min⁡1,(n/∣I∣)s⋆W_I\ge\min\\{1,(n/|I|)s^\star\\}, where s⋆s^\star is the largest feasible minimum slack to the global boundaries. Table A14 confirms full-range global intervals in all 1,814 group–repetition pairs below their own original-global (G0) threshold.

Local evidence bounds this unresolved proportion. At fixed global tolerances, Proposition 7 gives WIaug≤min⁡WI,2λIW_I^{\rm aug}\le\min\\{W_I,2\lambda_I\\} for an anchored group with local radius λI\lambda_I and a nonempty augmented set. Empirically, across five text corpora with fixed groups and 400 draws, global intervals including pool prevalence (G1) are a median 1.77–2.41 times as wide as anchored intervals for groups of at least 32 instances (Table A12).

W2, Q4: Why Definition 5 has this form and is valid

Definition 5 adds calibrated subgroup measurements while retaining global evidence. For a group fixed from labeling-function outputs, take independent uniform gold-label draws from the pool with replacement. Let ℓI>0\ell_I>0 count draws in the group and τ^I\hat\tau_I be their positive-label fraction. Conditional on ℓI\ell_I, the positive count is binomial with parameter pI⋆(1)p_I^\star(1), the group's true positive proportion. This gives a Clopper–Pearson (CP) interval [LI,HI][L_I,H_I] at level 1−δI1-\delta_I.

For text, λI=max⁡τ^I−LI,HI−τ^I\lambda_I=\max\\{\hat\tau_I-L_I,H_I-\hat\tau_I\\}. Definition 5 adds

∣1∣I∣∑i∈Ipi(1)−τ^I∣≤λI.\left|\frac{1}{|I|}\sum_{i\in I}p_i(1)-\hat\tau_I\right|\le\lambda_I.

Here pi(1)p_i(1) is a candidate positive-label probability. It equals EpiΦloc\mathbb E_{p_i}\Phi^{\rm loc} for the positive-class indicator feature. Normalization by ∣I∣|I| matches the subgroup target. The text anchor contains the raw CP interval, whereas the image construction uses its endpoints directly. Zero sampled hits give a full-range local constraint.

Intersection preserves simultaneous coverage: whenever every confidence row covers its true moment, the true label assignment belongs to the intersection. With global/local failure budgets δ0+∑IδI≤δ\delta_0+\sum_I\delta_I\le\delta, the union bound gives joint coverage, including overlapping groups and rows sharing labels. Optimizing each group proportion over this set covers its truth.

For label-learned groups, we fix the groups and local budgets before reading independent estimation draws. This preserves the conditional binomial relation (Wasserman and Roeder, 2009).

W4: The expressions at lines 145–146

The indicator glyph at these lines comes from a font mismatch. The corrected expression is

pI⋆(y)=1∣I∣∑i∈I1yi=y,p_I^\star(y)=\frac{1}{|I|}\sum_{i\in I}\mathbf{1}\\{y_i=y\\},

where the indicator equals one when yi=yy_i=y and zero otherwise. TT denotes the number of classes. The binary analysis relabels the two classes as 00 and 11.

We will add this workflow, example and citations to the introduction and standardize indicator notation throughout.

The motivation presented in both the paper and the response is still unclear, and the descriptions focus more on the method. Meanwhile, I still cannot clearly identify the core innovation and main idea of the paper. The paper also lacks further investigation of the field, and no substantial improvements have been made to the paper. I believe the above issues are key factors in evaluating the quality of the paper. If you cannot provide convincing explanations, I will maintain my current score.

Reviewer_h8qPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies confidence intervals for subgroup label proportions in programmatic weak supervision. It shows why global constraints can leave small-group intervals uninformative, and proposes adding confidence constraints estimated from labels within each group. For groups learned from labels, the paper separates group selection from estimation to preserve simultaneous coverage. The analysis is useful and the experiments support the main claims. The main concern is that the additional benefit of the joint optimization over direct group-level confidence intervals appears limited. Overall, I lean toward acceptance.

Critical strengths
  • The paper identifies a concrete limitation of global confidence constraints. Theorem 3 gives a computable condition under which a subgroup interval must span [0,1], which makes the motivation for local information clear.
  • The proposed method is simple, and the coverage argument is easy to follow. The distinction between selecting queries from an already valid uncertainty set and using labels to learn new constraints is also useful.
  • The authors have done a good job validating the main claims. Local anchoring improves median interval width across all five text corpora under the same label and confidence budgets. The matched same-sample and split-sample experiments also demonstrate the coverage problem for the pattern selector.
Critical weaknesses
  • The methodological contribution is somewhat incremental. Local binomial intervals, Bonferroni calibration, and sample splitting are standard tools. The subgroup-width analysis seems to be the main contribution, while the practical benefit of the joint LP is less clear. In Table A13, the local-only constraint already determines the final width in 80%-90% of pairs. Table A5 also reports a median joint/local width ratio of 1. A clearer analysis of when the global constraints add substantial information would strengthen the paper.
  • The smallest groups remain difficult to estimate. Table A12 shows tightening factors of only 1.03-1.12 for groups with fewer than 32 instances. This is consistent with the theory, but it limits the practical improvement for the small groups that motivate the paper.
  • The evaluation mainly measures interval width and coverage. It would be useful to show whether the narrower intervals improve a downstream decision, such as labeling-function selection or weakly supervised model training.
Questions for authors
  • For which group structures or labeling-function configurations does the joint LP substantially outperform local-only intervals? What is the additional computational cost?
  • Could allocating more labels to small groups improve the results in Table A12 while retaining simultaneous coverage and the same annotation budget?
  • Does using the anchored intervals to select labeling functions improve downstream performance?
2 comments

Thank you for your helpful comments and suggestions. To test the joint LP's practical value, we compare weak-rule selection under joint and local intervals. Joint constraints can certify higher-support rules by excluding incompatible local endpoints, increasing labeling coverage over matched local intervals. Comparison with the full-budget local baseline shows where direct estimation already suffices.

What does the joint LP add beyond local confidence intervals?

The joint LP adds information precisely when a target group's local extreme is incompatible with the shared global moments or other anchors. Groups sharing labeling-function patterns are coupled through the same candidate labels. Consequently, a local endpoint that is feasible for a group alone may be excluded by the joint set. If the endpoint remains jointly feasible, local-only and joint endpoints coincide. This explains the frequent equality in Tables A5 and A13 without attributing every gain to global moments alone.

Our methodological contribution is the subgroup-width characterization and the resulting local-information requirement. Clopper–Pearson (CP), joint confidence allocation, and independent estimation provide the statistical tools for implementing that requirement. The contribution therefore has a clear interpretation even when the local row determines the final width.

Do the intervals improve weak-rule selection?

Joint intervals increase certified weak-label coverage by 10.9 points over matched local intervals, with selected labels' error below the 20% threshold. Across five corpora, four original selectors and 80 repetitions, each of the 1,600 settings uses 400 draws: 200 for selection and 200 for independent local estimation. We select the largest-support rule whose error upper bound is at most 20%, and otherwise abstain. Fixed global moments use the same 400 draws.

The table reports the mean fraction of the pool receiving weak labels and cases where a selected rule's true error exceeds 20%. Total failure budget is 0.10. Joint divides it equally between global and local constraints. Matched local retains that local allocation, whereas strong local assigns the full 0.10 to its exact CP intervals.

MethodPool labeled (%)Incorrect certifications
Matched local24.90
Joint35.80
Strong local34.61

As the table shows, joint certification raises weak-label coverage by 10.9 percentage points over matched local. The effect is explained by higher-support certified rules, rather than summing overlapping groups. Compared with strong local, the aggregate gain is 1.2 points. The domain results identify where joint information matters: SMS increases from 76.2% to 91.4%, whereas TREC favors strong local, 30.7% versus 21.3%. Thus the evidence supports a conditional benefit determined by constraint compatibility and confidence allocation.

The selected joint rules have 10.5% label-weighted error, below the fixed 20% threshold. This directly tests the proposed weak-rule selection decision, with utility measured by certified labeling coverage and actual label error.

What is the additional computational cost?

The joint solves take 18–129 ms per setting in the five timed corpus configurations, compared with 0.45–1.00 ms for direct CP endpoints. These measurements use the original pattern selector, exclude common data loading, and solve 10–26 endpoints for 17–179 LF patterns. The joint computation is therefore more expensive, while remaining below 0.13 seconds in these configurations. Its purpose is to exploit cross-group compatibility when that information affects the decision.

Can a fixed budget better serve the smallest groups?

Prespecified within-group sampling can improve small-group precision at the same 400-draw budget while retaining simultaneous coverage. Reserve b_0 uniform pool draws for global moments and assign b_I independent uniform draws within each selected group, with b_0 + sum_I b_I = 400. Choose groups and allocation before observing these estimation labels. Conditional on the allocation, each class count is binomial, so CP bounds and the joint failure-budget allocation remain valid. Overlapping groups do not invalidate the union bound.

This change targets the reason for the 1.03–1.12 factors in Table A12: uniform pool sampling rarely reaches the smallest groups. Their precision is governed by the local count b_I, with an inverse-square-root leading term, so reallocating draws directly increases their information. Global moments must still use the reserved uniform pool sample, rather than an unweighted mixture of group-targeted draws. This gives a coverage-preserving allocation strategy at the same annotation budget without changing the subgroup estimand.

I thank the authors for the clarification. Some of my concerns are addressed with the response and I will keep my current positive rating.

Reviewer_pdRvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies intervals of class proportions within subgroups of an unlabeled pool in programmatic weak supervision. The authors studied how labels reaching a subgroup determine interval precision, proposed a method to achieve a lower bound on subgroup interval width, introduced group-local anchoring with joint coverage control, and explain how local sample size governs attainable precision. The experiments performed on text and image corpora demonstrates the effectiveness of the proposed approach. The approach is mainly practical for binary labels, and has the potential to be extended into multiclass scenarios.

Critical strengths
  1. The paper studies an important problem in programmatic weak supervision by establishing a lower bound on subgroup interval width under global constraints and deriving a computable sufficient condition for full-range intervals.

  2. The experimental evaluation is relatively comprehensive, covering both text and image datasets.

Critical weaknesses
  1. The main text relies heavily on supplementary tables, and many sentences are overloaded with qualifiers, making the writing difficult to follow and obscuring the key findings.

  2. The proposed approach is primarily applicable to binary classification, while its practical coverage and benefits in terms of interval width for multiclass settings require further validation.

  3. The paper lacks direct experimental comparisons with closely related methods targeting subgroup-conditional coverage, multicalibration, or conditional conformal prediction, limiting the assessment of its relative effectiveness.

Questions for authors
  1. How does the proposed method perform compared with existing approaches that provide group-conditional guarantees? Could the authors include direct experimental comparisons with closely related methods discussed in the Related Work section? Such comparisons would help better demonstrate the effectiveness and advantages of the proposed approach.

  2. The manuscript should be carefully revised to improve its clarity and readability, particularly by simplifying overly complex sentences and highlighting the key findings more clearly.

  3. How does the proposed approach perform in multiclass settings? Could the authors provide quantitative results on empirical coverage and interval width to demonstrate the practical effectiveness and potential benefits of extending the method beyond binary classification?

1 comment

Thank you for your careful review and suggestions. We added native six-class TREC and four-class AGNews experiments: local anchoring reduces mean interval width by 48% and 66% relative to global constraints, with all-group, all-class empirical coverage of 79/80 in both datasets. We also compare exact local intervals under the same total confidence and annotation budgets, making the source of the gain explicit.

Do the guarantee and width improvement extend to multiclass labels?

Both simultaneous coverage and the observed width reduction extend to native multiclass labels. The guarantee uses vector-valued anchors: for K classes, each pattern has K nonnegative label masses summing to its size. Every group/class proportion receives a Clopper–Pearson (CP) constraint in the same LP. If all constraints cover their true moments, the true multiclass assignment is feasible and every queried proportion is covered. Allocating 0.05 to global coordinates and 0.05 across group/class coordinates gives a simultaneous 90% guarantee by the union bound, without independence between class counts.

Our new experiment retains the native class labels, using a fixed 1,600-instance pool and the ten highest-coverage labeling functions per dataset. Majority-vote groups are defined without labels. Each of 80 repetitions uses 400 uniform draws with replacement, shared by all methods. Global-only and strong local-only each receive the full failure budget of 0.10. The table reports mean coordinate width and repetitions covering every group/class proportion.

DatasetMethodMean widthSimultaneous coverage
TREC, 6 classesGlobal0.62280/80
Joint0.32179/80
Exact local0.29879/80
AGNews, 4 classesGlobal0.64580/80
Joint0.21779/80
Exact local0.18577/80

As the table shows, anchoring substantially improves resolution beyond binary labels. Exact local intervals are already informative on these fixed majority-vote groups, confirming that direct local information drives the width reduction. The observed coverage describes these 80 repetitions, while the finite-sample guarantee follows from the confidence allocation above.

How do the comparisons relate to group-conditional methods?

Our same-target comparisons use exact local CP, global-moment and prediction-assisted intervals. Group-conditional calibration and conformal methods cover different quantities. Hebert-Johnson et al. (2018) calibrate predictions across groups, while Gibbs et al. (2023) provide conditional prediction-set coverage. Multiaccuracy (Kim et al., 2019) controls subgroup average prediction errors, and multivalid prediction (Jung et al., 2023) protects prediction-set coverage across groups. Our target is a fixed pool's group/class proportion, equal to a class-predicting rule's precision on its firing group. Accordingly, prediction-set size and proportion-interval width measure different outputs.

On the proportion target, the table above compares exact group-local CP with the entire confidence budget and the global-moment construction of Álvarez et al. (2025). It shows both the improvement over global-only information and how much local information alone provides. In the original binary experiments, same-budget finite-pool PPI and StratPPI adaptations add prediction-assisted comparisons. With the prespecified variance floor c=1, their simultaneous coverage ranges are 33–40/40 and 33–39/40 across five corpora. This sensitivity shows why confidence calibration and local label counts must accompany width comparisons.

Joint constraints also have a decision-level benefit when groups share information. In the separate five-corpus rule-selection evaluation, joint intervals label 35.8% of the pool under a 20% certified-error threshold, versus 24.9% for local intervals with matched local confidence allocation and 34.6% for exact local using the full budget. All methods use the same 400 draws. Thus this comparison measures weak-label generation on the same estimand, without comparing it to an incompatible prediction-set metric.

How is the presentation made easier to follow?

We make the main message explicit by explaining the need for local information before introducing the optimization. Global averages can leave a subgroup unresolved, so local labels constrain its proportion. When groups are learned from labels, independent estimation then preserves confidence. The definitions precede the formulas, and the key comparisons above include their setup, result, and interpretation in the response itself. This structure lets the reader follow the conclusion without reconstructing it from supplementary tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.