- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies simultaneous confidence intervals for subgroup label proportions in a fixed finite pool under programmatic weak supervision. It establishes a lower bound on the width induced by a global moment uncertainty set, using the pool-to-group size ratio and a computable center slack. It then adds group-local Clopper-Pearson anchors, allocates confidence jointly, and uses independent selection and estimation samples when labels determine the anchor groups.
Critical strengths
- The paper presents a clear research focus.
- The paper provides some essential experiments.
Critical weaknesses
- The background and objectives of programmatic weak supervision are severely lacking in the introduction.
- The descriptions in the paper are not sufficiently rigorous.
- The issues identified in this field are not supported by any references, making it difficult to verify their validity.
- The paper contains numerous typographical errors, such as those in the symbolic expressions in lines 145 and 146.
Questions for authors
- Why do the results presented in lines 54–58 motivate the proposal of group-local anchoring?
- What is the core idea of the paper?
- What is the key motivation behind the paper?
- Why is the group-locally anchored uncertainty set defined in this way in Definition 5?
4 comments
There are still many issues that have not been addressed in your responses. If you can provide more responses to address my concerns, I will raise my score.
Thank you for your follow-up. We show when global statistics leave subgroup reliability undetermined and how local label counts constrain it with simultaneous coverage.
W1, Q2–Q3: PWS background, motivation and core idea
Programmatic weak supervision uses labeling functions, such as keyword rules, to supply training labels for an unlabeled pool. Our input is the labeling-function output matrix and a small gold-label sample. Our output is a confidence interval for each queried group's class proportion. For a rule predicting the positive class whenever a keyword appears, this proportion is its precision, indicating whether its weak labels meet a desired reliability level.
The motivation is that valid pool-wide statistics can leave this precision anywhere in . Our core idea is to retain group-specific counts from the same gold sample and require candidate label assignments to agree with both global and local evidence. Thus anchoring adds an estimated group constraint, rather than rescaling a global constraint.
W3: Which claims are supported by prior work?
Data Programming (Ratner et al., 2016) and Snorkel (Ratner et al., 2017) establish the PWS workflow. Álvarez et al. (2025) supply the global confidence-constraint construction we analyze. The sufficient condition for its subgroup interval to span is our Theorem 3.
Q1: Why do lines 54–58 motivate local anchoring?
Local anchoring directly constrains proportions that global averages can leave unresolved. For example, let the pool size be , the group size , and the only global constraint a positive-label mean in . With outside-group probabilities fixed at , assigning the group entirely to either class gives pool means and . Both satisfy the constraint, so the group's proportion can still span .
Writing for interval width, Theorem 3 applies to every nonempty group under binary labels, a nonempty feasible set and features in : , where is the largest feasible minimum slack to the global boundaries. Table A14 confirms full-range global intervals in all 1,814 group–repetition pairs below their own original-global (G0) threshold.
Local evidence bounds this unresolved proportion. At fixed global tolerances, Proposition 7 gives for an anchored group with local radius and a nonempty augmented set. Empirically, across five text corpora with fixed groups and 400 draws, global intervals including pool prevalence (G1) are a median 1.77–2.41 times as wide as anchored intervals for groups of at least 32 instances (Table A12).
W2, Q4: Why Definition 5 has this form and is valid
Definition 5 adds calibrated subgroup measurements while retaining global evidence. For a group fixed from labeling-function outputs, take independent uniform gold-label draws from the pool with replacement. Let count draws in the group and be their positive-label fraction. Conditional on , the positive count is binomial with parameter , the group's true positive proportion. This gives a Clopper–Pearson (CP) interval at level .
For text, . Definition 5 adds
Here is a candidate positive-label probability. It equals for the positive-class indicator feature. Normalization by matches the subgroup target. The text anchor contains the raw CP interval, whereas the image construction uses its endpoints directly. Zero sampled hits give a full-range local constraint.
Intersection preserves simultaneous coverage: whenever every confidence row covers its true moment, the true label assignment belongs to the intersection. With global/local failure budgets , the union bound gives joint coverage, including overlapping groups and rows sharing labels. Optimizing each group proportion over this set covers its truth.
For label-learned groups, we fix the groups and local budgets before reading independent estimation draws. This preserves the conditional binomial relation (Wasserman and Roeder, 2009).
W4: The expressions at lines 145–146
The indicator glyph at these lines comes from a font mismatch. The corrected expression is
where the indicator equals one when and zero otherwise. denotes the number of classes. The binary analysis relabels the two classes as and .
We will add this workflow, example and citations to the introduction and standardize indicator notation throughout.
The motivation presented in both the paper and the response is still unclear, and the descriptions focus more on the method. Meanwhile, I still cannot clearly identify the core innovation and main idea of the paper. The paper also lacks further investigation of the field, and no substantial improvements have been made to the paper. I believe the above issues are key factors in evaluating the quality of the paper. If you cannot provide convincing explanations, I will maintain my current score.
Thank you for your review. We clarify the argument from problem to construction: valid pool-wide statistics can leave a subgroup's proportion undetermined, so group-local anchoring adds a direct confidence constraint for that group. Intersecting it with the global set retains the available evidence while preserving simultaneous coverage.
What problem does programmatic weak supervision solve?
Programmatic weak supervision uses labeling functions, such as keyword rules, to assign training labels when manual annotation is limited. Because a rule can be reliable on one subgroup and unreliable on another, its overall performance does not determine its precision on the subgroup where it will be used. Our objective is to estimate these subgroup proportions with simultaneous confidence intervals. For a rule predicting class y on group I, the target proportion is exactly that rule's precision on I.
The background comes from established weak-supervision research. Data programming and Snorkel (Ratner et al., 2016, 2017) motivate combining imperfect labeling functions. More directly, Álvarez et al. (2025) construct confidence intervals from global labeling-function statistics. We study the subgroup resolution of that construction. Our theorem then establishes when this global construction leaves a subgroup's proportion completely unresolved.
Why do lines 54–58 motivate local anchoring?
Lines 54–58 motivate local anchoring because global constraints can leave a small group's proportion completely unresolved, even when the pool-wide statistics are valid. Writing rho = n/|I| and s* for the largest feasible minimum slack in the global constraints, Theorem 3 gives W_I >= min(1, rho s*). Hence, when |I| <= n s*, the interval necessarily spans [0,1]. Changing labels throughout that group can remain compatible with the global statistics because the group contributes only |I|/n of each pool average.
Local anchoring removes this dilution by estimating the proportion within I itself. Its constraint is normalized by |I|, so an observed label in the group directly informs the requested quantity. This explains the progression from the width theorem to the proposed construction: first identify why global information cannot resolve the group, then supply a statistically calibrated measurement of that group's proportion.
Why is Definition 5 an intersection of constraints?
Definition 5 uses an intersection to add subgroup information while preserving both global evidence and simultaneous coverage. For binary labels, let p_i be a candidate positive-label probability and let [L_I,H_I] be the group's confidence interval. The local row is
L_I <= (1/|I|) sum_{i in I} p_i <= H_I.
Intersecting these rows with the original global set preserves the information already available. Moreover, if every row covers its true moment, the true assignment p_i = 1{y_i = 1} satisfies the entire intersection. Optimizing any subgroup proportion over that set therefore covers its truth. Allocating failure probabilities with delta_global + sum_I delta_I <= delta proves simultaneous coverage, including for overlapping groups. No independence between the different rows is required.
This construction also explains the observed precision improvement. With 400 labeled draws, the original five-corpus experiment gives median global-to-anchored width ratios of 1.77–2.41 for groups of at least 32 instances. Thus the added local measurements provide information that the pool averages alone cannot supply.
Rigorous notation and readable presentation
We clarify the notation around lines 145–146 by separating the fixed-pool target, candidate probabilities, and sampling randomness. The target is the fixed-pool proportion p*I(y) = |I|^{-1} sum{i in I} 1{y_i = y}, where 1{.} denotes an indicator. A consistent class convention is Y = {0,...,K-1}, with Y = {0,1} in the binary analysis. The empirical estimate uses independent uniform draws with replacement, and the label budget counts draws, including repeated positions. This separates the quantity being estimated from how labels are sampled.
The explanation is organized around one question at each step: what subgroup quantity is needed, why global constraints cannot resolve it, what local measurement supplies, and why joint calibration preserves coverage. This order makes the core idea accessible before introducing the optimization notation and keeps each formula attached to the problem it solves.