- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when consistency regularization becomes harmful in multi-label and structured prediction, particularly when data augmentation changes the targets of some output coordinates while preserving others. The authors introduce coordinate-level validity and derive a leading-order harm threshold under a controlled linear model. They further analyze how mask precision and selection rate affect the optimized risk and propose a finite-sample calibration procedure to certify the precision of selected outputs. Experiments on synthetic data, DocRED, and CoNLL-2003 investigate the theoretical predictions and the effects of selective consistency regularization. The paper addresses a meaningful problem and provides a carefully structured theoretical analysis. However, several concerns remain regarding the connection between the theoretical guarantees and practical implementations, the validity of the empirical selection criteria, and the limited improvements in downstream task performance.
Critical strengths
1 The paper identifies an important distinction between model confidence and augmentation validity. The observation that different output coordinates can respond differently to the same augmentation is well motivated and relevant to structured prediction.
2 The paper provides a coherent analysis connecting augmentation-induced bias, variance reduction, sample-size-dependent harm, and the precision–coverage tradeoff of selective regularization. The formal treatment is more substantial than a purely heuristic masking approach.
- 1 The synthetic experiments investigate the predicted harm threshold and the reliability of the calibration procedure. The inclusion of informative, boundary-tight, and uninformative validator settings helps illustrate both the benefits and limitations of the certificate.
Critical weaknesses
1 The harm threshold is derived under restrictive assumptions, including marginal-preserving augmentation, a well-specified linear model, simultaneous diagonalizability, and a simplified two-block structure. However, the real-task experiments involve nonlinear neural networks and text deletion operations that generally do not preserve the input distribution. Furthermore, the data-size experiments measure prediction near-invariance rather than the squared prediction risk considered in the theorem. Therefore, the empirical results do not directly validate the proposed harm threshold in realistic settings. 2 The finite-sample certificate requires independently constructed validator scores and i.i.d. calibration coordinates from the target population. However, the real-task evaluations contain dependent coordinates within documents or sentences, and the development data are also used for model selection. Although the paper acknowledges these limitations, the resulting empirical precision estimates are descriptive rather than formally certified population guarantees. 3 The theoretical definition of coordinate validity depends on changes in conditional label probabilities. In contrast, the DocRED and CoNLL experiments operationalize validity through annotated supporting sentences and entity spans. These annotation-based criteria do not necessarily establish whether the underlying conditional label probability actually changes after augmentation. Incomplete evidence annotations or redundant supporting information may introduce discrepancies between the theoretical and empirical validity definitions.
Questions for authors
- Under what realistic conditions can the theoretical harm threshold provide useful guidance for nonlinear models trained with non-marginal-preserving augmentations?
- Can the authors demonstrate a complete end-to-end certification procedure on DocRED or CoNLL that satisfies the independence assumptions of Theorem 6?
- How reliable are evidence annotations as proxies for changes in conditional label probabilities, and how sensitive is VCC to incorrect validity labels?
- Given that VCC does not outperform ERM in task F1, what practical applications benefit directly from the observed reduction in collateral near-invariance?
Thank you for your detailed review. We derive a nonlinear conflict bound and implement document-level sampling on DocRED. The bound explains why target preservation matters before enforcing agreement, while the calculation meets the precision target for the specified evidence labels.
Q1. Guidance for nonlinear models and distribution-changing edits
For nonlinear models, the practical criterion is target preservation before enforcing agreement. Our conflict bound supports this criterion for distribution-changing edits, while the numerical crossover concerns the controlled linear model.
For predictor f and edit A, let eta denote the target mean. Define D = E[(eta(X)-eta(AX))^2] as target change, C = E[(f(X)-f(AX))^2] as prediction disagreement, and R_pair as average squared prediction error on the two views. The L2 triangle inequality gives
R_pair >= (max(sqrt(D)-sqrt(C),0))^2 / 4.
Thus enforcing agreement when targets change creates an error floor without linearity or marginal preservation. Under exact invariance and an edited-to-original density ratio bounded by kappa, original-view error is at least D/[2(1+kappa)]. This condition transfers the conflict to original-input performance.
The experiments connect this mechanism to measured risk and task performance. In the linear model, squared-risk measurements recover the sample-size crossover with 3.1% median relative deviation. In neural models, the size scan shows all-output consistency trailing ERM by 0.24–0.41 F1 points on DocRED and 2.9–7.5 on CoNLL.
Q2. Certification with dependent coordinates and independent sampling
Our DocRED calculation meets the 95% evidence-label precision target on fixed train and development frames. Each independent document draw contributes one edit and eligible coordinate, so the score/validity pairs are iid across draws. Fixing the validator and threshold order independently of calibration makes Theorem 6 applicable to this document-weighted population.
Each draw samples a document with replacement, deletes one random sentence and selects one positive relation. Eligible documents have at least four sentences. With one fixed rule and threshold, alpha=delta=0.05, the evidence-label results are:
As the table shows, both finite frames meet the 95% evidence-label target. Lower bounds are rounded downward. For semantic certification on new documents, the construction requires verified preservation labels and document sampling independent of validator/model selection. The original development split retains its model-selection role.
Q3. Reliability of evidence proxies and sensitivity to errors
Evidence labels track annotated support, while our sensitivity bounds quantify how semantic errors affect precision and CVR. We prepared 400 blinded views from 200 documents for separate judgments of original and edited texts. We will complete human verification of target preservation before the paper's final acceptance.
On DocRED, the rule's collateral violation rate (CVR) is 27.4%, versus 27.7% for the oracle and 46.8% for random masks, using evidence labels. Invalid-selection rates are 0%/2.3%/25.8% for oracle/rule/random, with the random rate in expectation. This comparison isolates selection quality beyond mask size.
Our sensitivity analysis separates incomplete and redundant evidence. Incomplete annotations can mark a target-changing coordinate as preserved. If epsilon_plus is this fraction among selected outputs, true precision satisfies p >= p_tilde-epsilon_plus, where p_tilde is evidence-label precision. In the homogeneous risk model, the corresponding optimized-risk increase is at most sBepsilon_plus, for fixed selection rate s and violation cost B.
Conversely, redundant evidence can preserve the target after annotated evidence is removed. If such cases comprise fraction q of the evidence-altered pool, a CVR gap Delta implies a gap of at least (Delta-q)/(1-q) on its truly altered subset. Random-minus-rule gaps remain positive for q below 19.5% on DocRED and 33.3% on CoNLL. These are sensitivity thresholds, not measured semantic error rates.
Q4. Practical value of selective consistency training
The practical use is selective consistency training in evidence-grounded relation extraction and entity recognition. At the evaluated weights, VCC improves both F1 and evidence responsiveness over all-output consistency. It reduces CVR from 47.0% to 27.4% on DocRED and from 71.8% to 38.2% on CoNLL, with F1 higher by 0.24/0.98 points. ERM F1 is 61.01/93.93 versus VCC's 60.90/91.97. CVR measures near-invariance after evidence edits. Edited-input correctness is a separate endpoint requiring comparison with the edited targets.