acceptodds
Under review as a conference paper at ICLR 2027

Harm Thresholds and Certified Masks for Consistency Regularization

Abstract

Consistency regularization encourages matching predictions for an input and its augmented version. In multi-label and structured prediction, however, an augmentation can preserve some labels while removing the evidence for others. Model confidence measures prediction certainty, whereas selecting outputs for this penalty requires assessing whether augmentation preserves their labels. We derive harm thresholds by distinguishing outputs according to changes in their conditional label probabilities. In a controlled linear model, these thresholds identify, to leading order, the training sample size beyond which a fixed consistency penalty increases prediction risk relative to unregularized training. The crossover arises because the variance benefit decreases with more data while augmentation-induced bias persists. Guided by this analysis, we calibrate output-selection masks on independent validity labels that indicate whether augmentation preserves each output's target. Under i.i.d. calibration sampling and independently constructed validator scores, the certificates bound, at a chosen confidence level, the fraction of selected outputs whose conditional label probabilities change beyond a specified tolerance. Controlled simulations recover the predicted crossover with a median absolute relative deviation of 3.13%. Experiments on document-level relation extraction and named-entity recognition show that penalties applied to every output encourage unchanged predictions after supporting evidence is removed. At comparable selection rates, validated masks keep the mean frequency of this behavior at or below the unregularized level, whereas random masks increase it. These results support selecting where consistency penalties apply according to label preservation at each output.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
consistency regularization, data augmentation, label-preserving augmentation, multi-label learning, structured prediction, finite-sample certificates, relation extraction, named entity recognition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zUEFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when consistency regularization becomes harmful in multi-label and structured prediction, particularly when data augmentation changes the targets of some output coordinates while preserving others. The authors introduce coordinate-level validity and derive a leading-order harm threshold under a controlled linear model. They further analyze how mask precision and selection rate affect the optimized risk and propose a finite-sample calibration procedure to certify the precision of selected outputs. Experiments on synthetic data, DocRED, and CoNLL-2003 investigate the theoretical predictions and the effects of selective consistency regularization. The paper addresses a meaningful problem and provides a carefully structured theoretical analysis. However, several concerns remain regarding the connection between the theoretical guarantees and practical implementations, the validity of the empirical selection criteria, and the limited improvements in downstream task performance.

Critical strengths

1 The paper identifies an important distinction between model confidence and augmentation validity. The observation that different output coordinates can respond differently to the same augmentation is well motivated and relevant to structured prediction.

2 The paper provides a coherent analysis connecting augmentation-induced bias, variance reduction, sample-size-dependent harm, and the precision–coverage tradeoff of selective regularization. The formal treatment is more substantial than a purely heuristic masking approach.

  1. 1 The synthetic experiments investigate the predicted harm threshold and the reliability of the calibration procedure. The inclusion of informative, boundary-tight, and uninformative validator settings helps illustrate both the benefits and limitations of the certificate.
Critical weaknesses

1 The harm threshold is derived under restrictive assumptions, including marginal-preserving augmentation, a well-specified linear model, simultaneous diagonalizability, and a simplified two-block structure. However, the real-task experiments involve nonlinear neural networks and text deletion operations that generally do not preserve the input distribution. Furthermore, the data-size experiments measure prediction near-invariance rather than the squared prediction risk considered in the theorem. Therefore, the empirical results do not directly validate the proposed harm threshold in realistic settings. 2 The finite-sample certificate requires independently constructed validator scores and i.i.d. calibration coordinates from the target population. However, the real-task evaluations contain dependent coordinates within documents or sentences, and the development data are also used for model selection. Although the paper acknowledges these limitations, the resulting empirical precision estimates are descriptive rather than formally certified population guarantees. 3 The theoretical definition of coordinate validity depends on changes in conditional label probabilities. In contrast, the DocRED and CoNLL experiments operationalize validity through annotated supporting sentences and entity spans. These annotation-based criteria do not necessarily establish whether the underlying conditional label probability actually changes after augmentation. Incomplete evidence annotations or redundant supporting information may introduce discrepancies between the theoretical and empirical validity definitions.

Questions for authors
  1. Under what realistic conditions can the theoretical harm threshold provide useful guidance for nonlinear models trained with non-marginal-preserving augmentations?
  2. Can the authors demonstrate a complete end-to-end certification procedure on DocRED or CoNLL that satisfies the independence assumptions of Theorem 6?
  3. How reliable are evidence annotations as proxies for changes in conditional label probabilities, and how sensitive is VCC to incorrect validity labels?
  4. Given that VCC does not outperform ERM in task F1, what practical applications benefit directly from the observed reduction in collateral near-invariance?
1 comment

Thank you for your detailed review. We derive a nonlinear conflict bound and implement document-level sampling on DocRED. The bound explains why target preservation matters before enforcing agreement, while the calculation meets the precision target for the specified evidence labels.

Q1. Guidance for nonlinear models and distribution-changing edits

For nonlinear models, the practical criterion is target preservation before enforcing agreement. Our conflict bound supports this criterion for distribution-changing edits, while the numerical crossover concerns the controlled linear model.

For predictor f and edit A, let eta denote the target mean. Define D = E[(eta(X)-eta(AX))^2] as target change, C = E[(f(X)-f(AX))^2] as prediction disagreement, and R_pair as average squared prediction error on the two views. The L2 triangle inequality gives

R_pair >= (max(sqrt(D)-sqrt(C),0))^2 / 4.

Thus enforcing agreement when targets change creates an error floor without linearity or marginal preservation. Under exact invariance and an edited-to-original density ratio bounded by kappa, original-view error is at least D/[2(1+kappa)]. This condition transfers the conflict to original-input performance.

The experiments connect this mechanism to measured risk and task performance. In the linear model, squared-risk measurements recover the sample-size crossover with 3.1% median relative deviation. In neural models, the size scan shows all-output consistency trailing ERM by 0.24–0.41 F1 points on DocRED and 2.9–7.5 on CoNLL.

Q2. Certification with dependent coordinates and independent sampling

Our DocRED calculation meets the 95% evidence-label precision target on fixed train and development frames. Each independent document draw contributes one edit and eligible coordinate, so the score/validity pairs are iid across draws. Fixing the validator and threshold order independently of calibration makes Theorem 6 applicable to this document-weighted population.

Each draw samples a document with replacement, deletes one random sentence and selects one positive relation. Eligible documents have at least four sentences. With one fixed rule and threshold, alpha=delta=0.05, the evidence-label results are:

FrameDrawsSelectedEvidence altered among selected95% lower precision
Train3,0001,9772398.3%
Dev1,0006551197.2%

As the table shows, both finite frames meet the 95% evidence-label target. Lower bounds are rounded downward. For semantic certification on new documents, the construction requires verified preservation labels and document sampling independent of validator/model selection. The original development split retains its model-selection role.

Q3. Reliability of evidence proxies and sensitivity to errors

Evidence labels track annotated support, while our sensitivity bounds quantify how semantic errors affect precision and CVR. We prepared 400 blinded views from 200 documents for separate judgments of original and edited texts. We will complete human verification of target preservation before the paper's final acceptance.

On DocRED, the rule's collateral violation rate (CVR) is 27.4%, versus 27.7% for the oracle and 46.8% for random masks, using evidence labels. Invalid-selection rates are 0%/2.3%/25.8% for oracle/rule/random, with the random rate in expectation. This comparison isolates selection quality beyond mask size.

Our sensitivity analysis separates incomplete and redundant evidence. Incomplete annotations can mark a target-changing coordinate as preserved. If epsilon_plus is this fraction among selected outputs, true precision satisfies p >= p_tilde-epsilon_plus, where p_tilde is evidence-label precision. In the homogeneous risk model, the corresponding optimized-risk increase is at most sBepsilon_plus, for fixed selection rate s and violation cost B.

Conversely, redundant evidence can preserve the target after annotated evidence is removed. If such cases comprise fraction q of the evidence-altered pool, a CVR gap Delta implies a gap of at least (Delta-q)/(1-q) on its truly altered subset. Random-minus-rule gaps remain positive for q below 19.5% on DocRED and 33.3% on CoNLL. These are sensitivity thresholds, not measured semantic error rates.

Q4. Practical value of selective consistency training

The practical use is selective consistency training in evidence-grounded relation extraction and entity recognition. At the evaluated weights, VCC improves both F1 and evidence responsiveness over all-output consistency. It reduces CVR from 47.0% to 27.4% on DocRED and from 71.8% to 38.2% on CoNLL, with F1 higher by 0.24/0.98 points. ERM F1 is 61.01/93.93 versus VCC's 60.90/91.97. CVR measures near-invariance after evidence edits. Edited-input correctness is a separate endpoint requiring comparison with the edited targets.

Reviewer_Vb2yOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the potential risks of consistency regularization when data augmentation preserves only a subset of output labels in multi-label and structured prediction tasks. The authors introduce coordinate-level validity to distinguish outputs whose conditional label probabilities remain unchanged after augmentation from those whose targets are altered. Based on a controlled linear model, the paper derives harm thresholds characterizing when augmentation-induced bias outweighs the variance reduction provided by consistency regularization. It further proposes Validated-Coordinate Consistency (VCC), which selectively applies consistency penalties using validity-aware masks and introduces a finite-sample calibration procedure to control the proportion of invalid outputs among selected coordinates.

Critical strengths
  1. The authors recognize that augmentation-induced invariance may be appropriate for some output coordinates but harmful for others, particularly in structured prediction tasks where different outputs depend on different evidence. This coordinate-level perspective provides a meaningful extension beyond conventional confidence-based consistency selection.

  2. The proposed harm threshold characterizes how persistent augmentation-induced bias can eventually outweigh the diminishing variance benefit as training data increases. Furthermore, the analysis of mask precision and selection rate provides an interpretable explanation of why selectively applying consistency penalties can be preferable to indiscriminate regularization.

  3. The authors introduce a finite-sample procedure that controls the probability of selecting a mask whose population precision falls below a specified threshold, under explicit sampling and independence assumptions. This provides a useful statistical perspective on output-level consistency selection.

Critical weaknesses
  1. Theorem 6 requires independently sampled calibration coordinates with verified validity labels, along with a validator and threshold grid constructed independently of the calibration sample. However, the DocRED and CoNLL-2003 experiments employ rule-based validators and report descriptive precision estimates on correlated coordinates, rather than demonstrating the proposed independent calibration procedure. Since certified masking is a central contribution, the authors should evaluate the complete calibration pipeline on real tasks using independent document-level calibration and test splits, explicitly addressing within-document coordinate dependencies.

  2. Definition 1 characterizes coordinate validity through changes in conditional label probabilities, whereas the real-task experiments approximate validity using deleted evidence sentences in DocRED or masked entity mentions in CoNLL-2003. These proxies do not necessarily imply that the underlying target has changed. For example, deleting an annotated supporting sentence may leave sufficient relational evidence elsewhere in the document, while masking an entity mention may still permit its label to be inferred from context. This discrepancy also affects the interpretation of the collateral violation rate, since a prediction change after evidence removal is not necessarily a correct response. I suggest conducting a targeted human-annotated evaluation of augmented examples to assess actual target preservation and measuring prediction correctness or calibration on the edited inputs, rather than relying exclusively on evidence-overlap heuristics and prediction differences.

  3. Table 2 shows that VCC achieves an F1 score of 60.90 on DocRED, compared with 61.01 for unregularized training, and 91.97 on CoNLL-2003, compared with 93.93 for unregularized training. Although VCC substantially reduces undesirable near-invariance, this improvement does not translate into better task accuracy. Furthermore, all consistency-based approaches underperform the unregularized baseline on CoNLL-2003. The experiments use fixed consistency weights within each domain, without systematically examining the trade-off between task performance and collateral invariance. It may be useful to evaluate a wider range of regularization strengths for all methods and reporting F1–CVR trade-off curves.

Minor comment:

  1. It is suggested that the authors include more recent references from the past three years to better reflect the latest developments in this field.
Questions for authors

Please carefully checking the weaknesses.

1 comment

Thank you for your constructive feedback. We address calibration with document-level sampling, whose DocRED implementation meets the evidence-label precision target. The fixed-weight comparison then isolates penalty placement: selected coordinates improve F1 and evidence responsiveness over all-output consistency.

W1. Independent calibration with dependent output coordinates

The DocRED calculation certifies evidence-label precision under independent draws from fixed frames, with both precision bounds exceeding the 95% target. Sampling one edit and eligible coordinate per document draw handles output dependence, making the pairs iid across draws. With the validator and threshold order fixed independently of calibration, Theorem 6 gives the document-weighted guarantee.

We use one fixed rule and threshold, with alpha=delta=0.05. Each draw uniformly samples a document with replacement, deletes one sentence and selects one positive relation. Documents have at least four sentences. The evidence-label bounds below are rounded downward.

FrameDrawsSelectedEvidence altered among selected95% lower precision
Train3,0001,9772398.3%
Dev1,0006551197.2%

As the table shows, both frames pass the 95% evidence-label target. Semantic certification on new documents additionally requires verified preservation labels and independent document-level calibration and test samples, separate from validator/model selection. The original development split retains its model-selection role.

W2. Human target verification and correctness after editing

To test semantic preservation directly, we have prepared 400 blinded views from 200 documents for separate judgments of original and edited texts. We will complete human verification of target preservation before the paper's final acceptance.

Your examples motivate separating redundant evidence from incomplete annotations. Remaining relational evidence or contextual entity cues can preserve a target after annotated evidence is removed. Conversely, incomplete annotations can mark a coordinate preserved when its target changes. If epsilon_plus is the latter fraction among selected outputs, true precision satisfies p >= p_tilde-epsilon_plus, where p_tilde is evidence-label precision.

Target preservation after evidence removal can dilute the CVR gap. If fraction q of the evidence-altered pool actually retains its target, a measured gap Delta implies a gap of at least (Delta-q)/(1-q) on the truly altered subset. Thus the random-versus-rule gap remains positive for q below 19.5% on DocRED and 33.3% on CoNLL. These thresholds describe sensitivity to semantic errors.

The current measurements use dataset evidence labels: CVR measures near-invariance on evidence-altered coordinates. Edited correctness and calibration require comparison with edited targets. Agreement on evidence-preserved DocRED coordinates remains 97.0–98.1% across objectives. This supports selective responsiveness to evidence removal.

W3. F1, ERM, and the requested regularization-strength comparison

Rule masks improve both F1 and evidence responsiveness over all-output masking at the evaluated weights. The comparison fixes lambda=0.1 for DocRED and lambda=1 for CoNLL to isolate penalty placement.

MethodDocRED F1 / CVR (%)CoNLL F1 / CVR (%)
ERM61.01 / 27.693.93 / 48.7
All-output60.66 / 47.090.99 / 71.8
Random mask60.89 / 46.890.65 / 71.4
VCC60.90 / 27.491.97 / 38.2

As the table shows, VCC keeps DocRED CVR at the ERM level while all-output and random masking raise it to about 47%. On CoNLL, VCC lowers CVR relative to ERM and incurs the smallest F1 reduction among these consistency objectives. Its F1 gains over all-output consistency are 0.24/0.98 points.

The existing DocRED size scan also tests lambda proportional to 1/n. Relative to fixed lambda, F1 changes by +0.30/-0.46/-0.22 points at 10%/25%/50% data. This varies a schedule across data sizes, whereas an all-method F1–CVR curve varies weights at fixed size. Thus the reported operating points establish the benefit of penalty placement.

Minor comment. Recent work and our specific contribution

Three recent works clarify the distinction between label quality, edit validity and risk certification. SemiReward (ICLR 2024) learns pseudo-label quality, and CovEReD (EMNLP 2024) studies relation consistency under entity replacement. VCC selects outputs according to preservation under the specified edit. Conformal Risk Control (ICLR 2024)controls expected monotone loss, whereas our certificate controls the probability of returning a mask below its precision target.

Reviewer_M8nWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when consistency regularization can harm predictive performance by enforcing invariance on outputs whose labels change under augmentation. The authors derive theoretical thresholds characterizing when the bias introduced by consistency regularization outweighs its variance reduction benefits. They also propose a framework for selecting valid output coordinates using calibrated masks with finite sample precision guarantees. Experiments on synthetic data, document level relation extraction, and named entity recognition support the theoretical analysis and show that selective regularization reduces undesirable prediction invariance.

Critical strengths

The paper addresses an interesting and relevant problem. The distinction between prediction confidence and whether an augmentation actually preserves the target label is important. The theoretical analysis provides useful insights into when consistency regularization becomes harmful as training data increases.

The proposed calibration framework is well motivated, and the finite sample precision guarantee is a useful contribution. The controlled experiments support the theoretical predictions, while the comparisons with random masks provide evidence that selecting the right outputs matters more than simply reducing the number of regularized outputs.

Critical weaknesses

The theoretical results rely on fairly restrictive assumptions, particularly the linear regression model, simultaneous diagonalizability, and homogeneous coordinate mixture. The connection between these results and the deep learning experiments remains indirect. The experiments on real tasks measure prediction invariance rather than the excess risk characterized by the theory, making it difficult to assess whether the predicted harm thresholds are practically meaningful.

The practical improvements are somewhat limited. In Table 2, the proposed method reduces undesirable prediction invariance, but it does not improve F1 over standard supervised training on either DocRED or CoNLL. Furthermore, the finite sample certification guarantee assumes independent validity labels, whereas the real task experiments use dependent coordinates and evidence annotations as proxies for label preservation. Consequently, the main statistical guarantee is not directly established for the practical applications.

Questions for authors

Can the authors provide additional evidence connecting the theoretical harm threshold to actual prediction risk or task performance in deep neural networks, rather than only the proposed invariance metric?

How would the certification guarantee extend to dependent output coordinates within the same document or sentence? Is there a practical way to obtain valid guarantees in the evaluated settings?

Given that the proposed method does not improve F1 over supervised training, what is the intended practical benefit of reducing prediction invariance? Have the authors considered evaluating robustness or accuracy specifically on examples where supporting evidence changes?

1 comment

Thank you for your thoughtful comments. We connect the risk analysis to measured task F1 and derive a nonlinear error bound, explaining why agreement can be harmful when an edit changes the target. To address dependence, we implement document-level sampling on DocRED, where both evidence-label precision bounds exceed the 95% target.

Q2. Valid guarantees with dependent outputs in one document

We obtain independent calibration units by sampling one coordinate per independently drawn document. Within each document, sample an edit and one eligible coordinate. The resulting score/validity pairs are iid across draws, so Theorem 6 applies when the validator and threshold order are fixed independently of calibration. Precision refers to a random document, edit and coordinate. For a coordinate-pooled target, retaining each document/edit with probability n_d/n_max before sampling a coordinate gives the pooled law, where n_d is its coordinate count and n_max a known upper bound.

The uniform-document construction meets the evidence-label precision target on fixed DocRED frames. Each draw samples with replacement, deletes one random sentence and selects one positive relation. Eligible documents have at least four sentences. With one fixed rule and threshold, alpha=delta=0.05 and prespecified seeds, the results are:

FrameDrawsSelectedEvidence altered among selected95% lower precision
Train3,0001,9772398.3%
Dev1,0006551197.2%

As the table shows, both finite frames meet the 95% evidence-label target. Bounds are rounded downward. Applying the construction to semantic preservation on new documents requires verified target labels and document sampling independent of validator/model selection. Thus the independent sampling unit is the document, even when its outputs are dependent.

Q1. Connecting the harm mechanism to neural prediction risk and F1

The neural training-size scan provides actual task-performance evidence: all-output consistency trails ERM by 0.24–0.41 F1 points on DocRED and 2.9–7.5 on CoNLL. The controlled linear experiment separately recovers the predicted squared-risk crossover with 3.1% median relative deviation.

To explain the conflict beyond the linear model, our added bound relates target change to error for arbitrary predictors and edits. For predictor f and edit A, let eta denote the target mean. Write D = E[(eta(X)-eta(AX))^2] for target change, C = E[(f(X)-f(AX))^2] for prediction disagreement, and R_pair for average squared prediction error across the two views. The L2 triangle inequality gives

R_pair >= (max(sqrt(D)-sqrt(C),0))^2 / 4.

Consequently, reducing disagreement toward zero under target conflict forces an error floor. Under exact invariance and an edited-to-original density ratio bounded by kappa, original-view error is at least D/[2(1+kappa)]. The paired-view result applies without linearity or equal marginals.

Thus the F1 scan establishes a task-performance cost alongside the general conflict mechanism. The numerical crossover and homogeneous-mixture optimum retain their stated model assumptions, while CVR measures responsiveness to evidence removal.

Q3. Practical benefit and accuracy when supporting evidence changes

For evidence-grounded extraction, VCC improves F1 and responsiveness to evidence changes over all-output consistency. Rule masks reduce CVR from 47.0% to 27.4% on DocRED and from 71.8% to 38.2% on CoNLL, while increasing mean F1 by 0.24/0.98 points. ERM F1 is 61.01/93.93 versus VCC's 60.90/91.97.

For edited-input evaluation, CVR measures near-invariance on evidence-altered outputs, while correctness requires comparison with edited targets. We have prepared 400 blinded views from 200 documents for separate judgments of original and edited texts. We will complete human verification of target preservation before the paper's final acceptance. This will assess the correspondence between evidence labels and human judgments.

The error analysis explains why responsiveness matters when the target changes. If target probability changes by g, original prediction error is at most e and prediction disagreement is at most c, edited probability error is at least max(g-e-c,0). Therefore, an accurate original prediction cannot remain arbitrarily close to its old value and still track a sufficiently changed target.

Our sensitivity calculation further accounts for redundant evidence. If fraction q of the evidence-altered pool retains its true target, a CVR gap Delta implies a gap of at least (Delta-q)/(1-q) on its truly altered subset. The random-minus-rule gaps remain positive for q below 19.5% on DocRED and 33.3% on CoNLL. These are sensitivity thresholds, not measured semantic error rates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.