Harm Thresholds and Certified Masks for Consistency Regularization

Consistency regularization encourages matching predictions for an input and its augmented version. In multi-label and structured prediction, however, an augmentation can preserve some labels while removing the evidence for others. Model confidence measures prediction certainty, whereas selecting outputs for this penalty requires assessing whether augmentation preserves their labels. We derive harm thresholds by distinguishing outputs according to changes in their conditional label probabilities. In a controlled linear model, these thresholds identify, to leading order, the training sample size beyond which a fixed consistency penalty increases prediction risk relative to unregularized training. The crossover arises because the variance benefit decreases with more data while augmentation-induced bias persists. Guided by this analysis, we calibrate output-selection masks on independent validity labels that indicate whether augmentation preserves each output's target. Under i.i.d. calibration sampling and independently constructed validator scores, the certificates bound, at a chosen confidence level, the fraction of selected outputs whose conditional label probabilities change beyond a specified tolerance. Controlled simulations recover the predicted crossover with a median absolute relative deviation of 3.13%. Experiments on document-level relation extraction and named-entity recognition show that penalties applied to every output encourage unchanged predictions after supporting evidence is removed. At comparable selection rates, validated masks keep the mean frequency of this behavior at or below the unregularized level, whereas random masks increase it. These results support selecting where consistency penalties apply according to label preservation at each output.