acceptodds
Under review as a conference paper at ICLR 2027

Hardness Is Not Enough: What Contrastive Error Audits Identify

Abstract

Contrastive error auditors rank the items of a data pool by their resemblance to synthetic corruptions, and harder, more plausible corruptions are generally preferred. We prove that no criterion computed from population testing difficulty can certify how well such an auditor ranks real errors. With the pool, error prevalence, representation and real error process held fixed, two corruptions can share the complete binary testing region, and hence the optimal ROC curve, every binary Bayes risk and every -divergence, while their optimal scores rank the real errors optimally and in exact reverse. This holds on every atomless pool with a nonconstant error posterior, at every continuous difficulty level, and an exact criterion decides it on finite pools. Fidelity, in contrast, is sufficient: audit regret is bounded by the total-variation distance between the corruption and the error distribution plus the contrastive AUC shortfall. Crossed detector interventions on knowledge graphs, including a prespecified replication in an independent architecture, show that the preferred training corruption depends on the errors evaluated, and changing only those errors reverses the order of two published detectors on FB15k-237, by 0.008 in top-5% precision. The dependence persists in image audits at matched measured difficulty and in human-annotated summaries. Synthetic corruptions should therefore be justified against the target error process and evaluated on held-out target errors alongside controls fitted on the same labels.

Primary area:
learning theory
Keywords:
contrastive learning, anomaly detection, out-of-distribution detection, negative sampling, learning theory, density-ratio estimation, benchmark artifacts

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PN1BOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether contrastive hardness of synthetic corruptions is sufficient to determine the quality of an error auditor trained to distinguish those corruptions from an observed data pool. The central theoretical result shows that it is not. With the observed pool, error prevalence, representation, and latent real-error process held fixed, the authors construct two corruption distributions with identical complete binary testing regions, and therefore identical optimal ROC curves, Bayes risks, and f-divergences, while their population-optimal contrastive scores rank the real errors in opposite directions. The paper further provides a finite-pool characterization of when such reversals are possible.

The second main theoretical contribution gives a positive result: audit regret can be bounded by the discrepancy between the synthetic corruption distribution and the real error distribution, together with the contrastive AUC shortfall. This motivates fidelity to the target error process, rather than hardness alone, as the relevant property of synthetic corruptions.

The theoretical claims are complemented by a broad empirical study across knowledge-graph anomaly detection, image auditing, summarization, information extraction, and question answering. Crossed interventions show that changing the training corruption can change which errors are detected well, and changing only the evaluated error construction can even reverse the ranking of two published detectors on FB15k-237. Additional experiments test matched measured difficulty and human-annotated error processes. The paper also proposes a benchmark diagnostic based on the maximum AUC obtainable from coarse benchmark partitions and shows that simple priors can outperform published detectors in some settings.

The paper makes a strong conceptual point: difficulty of distinguishing synthetic negatives from the data is not, by itself, evidence that the resulting auditor will rank real errors correctly.

Critical strengths
  1. Strong and unusually sharp theoretical identification result.

The main theorem is substantially stronger than showing that a particular hardness metric is insufficient. The authors hold the latent error process, observed pool, prevalence, and representation fixed, while constructing corruptions with the same complete binary testing region but opposite optimal audit rankings. Since the testing region determines the optimal ROC curve, binary Bayes risks, and f-divergences, this rules out a broad class of hardness-only arguments rather than attacking one particular metric. I view this as the most important contribution of the paper.

  1. Careful empirical methodology and unusually transparent treatment of negative results.

The empirical section includes crossed corruption/error interventions, a prespecified replication in an independent detector architecture, an image experiment with matched measured contrastive difficulty, and evaluations involving human-identified errors. Importantly, the authors also report when prespecified strong criteria fail, when effects become unresolved after refitting, when transfer fails, and when simpler same-label controls outperform the proposed matching approach. This makes the empirical evidence more credible and supports the broader conclusion that corruption choice can materially affect audit conclusions.

Critical weaknesses
  1. The positive theoretical result has a substantial operational gap.

Theorem 2 identifies fidelity to the target error distribution as the relevant quantity, but the discrepancy term depends on the unknown real-error distribution. Consequently, the result does not directly provide a label-free procedure for selecting a good synthetic corruption. In practice, determining whether an auxiliary distribution is sufficiently faithful appears to require target-error information, which is precisely what motivates synthetic corruption-based auditing in the first place. Thus, the negative result is strong and actionable as a warning, while the positive prescription is considerably less operational.

  1. The empirical evidence is broad but heterogeneous and only partially supports a general practical effect.

The experiments illustrate corruption dependence, but they do not directly instantiate the population theorem. Several empirical results are also weaker than the headline theoretical result: the image study matches a measured discrimination proxy rather than the complete testing region; the image and summary studies miss their prespecified strong criteria; the FRANK interaction is largely driven by one error type; CoNLL++ is unresolved; the QA generic comparison becomes unresolved when fitting uncertainty is included; and the unchanged QA heads fail to transfer to Data2txt. These results do not invalidate the central theorem, but they make the empirical evidence for a broadly recurring practical phenomenon less decisive than the theoretical contribution.

Questions for authors
  1. Theorem 2 depends on a discrepancy between the auxiliary distribution and the unknown real-error distribution. Is there any practically estimable, label-free or low-label surrogate that the authors believe can be used to select among candidate corruptions? If not, could the authors clarify what actionable corruption-selection procedure they recommend when target-error labels are scarce?

This would materially affect my assessment of the practical significance of the positive result.

  1. Theorem 1 allows unconstrained corruption distributions and the paper explicitly notes that it does not impose locality or semantic-edit constraints. Are there important commonly used corruption families for which the reversal construction can be ruled out, or for which hardness does become informative about audit quality?

A characterization, even for a restricted family, would help determine how directly the impossibility result applies to practical negative-sampling procedures.

  1. Could the authors clarify how they believe the empirical evidence should be weighted relative to the theoretical result? Several studies produce positive interactions, but several prespecified strong criteria fail, some contrasts become unresolved after accounting for refitting, and the Data2txt transfer experiment fails. Is the intended empirical claim only that corruption dependence can occur in realistic fitted systems, rather than that matching corruption semantics generally improves auditing?
Reviewer_oTByOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores whether the difficulty of distinguishing synthetic curruptions from a data pool predicts how well an auditor ranks real errors. Theoretical analysis gives a negative answer by constructing corruptions that are equally hard but with opposite rankings of a fixed error distribution. They then bounds the audit regret using the corruption's distance from real errors plus the contrastive ranking error. Experiments on knowledge graphs, images, and human-annotated summaries show its practical consequences.

Critical strengths
  1. The theoretical contribution is interesting. Theorem 1 presents a strong conclusion separating hardness from fidelity. Theorem 2 further provides a bound that connects audit quality to meaningful quantities, namely the corruption's distance from the real errors, and the contrastive fitting gap. I read some of the proofs, and they look correct to me. This is the major strength that make me lean acceptance.
Critical weaknesses
  1. The connection between Theorem 1 and the experiments is not clear. Only the image experiment matches the two corruptions in measured difficulty. In the knowledge graph experiments, detectors perform better on injected errors when their training negatives are generated the same way as those error, which is expected. Since the two kinds of negatives were not matched in difficulty, the result does not say much about Theorem 1.
Questions for authors

Clarification: On Re-DocRED, type-preserving replacements produce an AUC of 0.436, which is below chance, and lower than the 0.522 for uniform ones (Section 4.4). Are they also harder by the pool-vs-auxiliary AUC used to match difficulty in Section 4.2? If so, this would be a real-error case where the harder corruption gives the worse audit, connecting the experiments more closely to Theorem 1.

Reviewer_YKHjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies contrastive error auditing, where the items are ranked by how much they are similar to synthetic corruptions, and aims to revisit a common assumption that harder and more plausible corruptions lead to better detection of real errors. The paper provides theoretical results, including that fidelity (how closely the corruption captures the real errors) with good contrastive fitting is sufficient for a small audit regret. Various experiments are performed to support the main idea of the paper, with different benchmarks and corruption settings.

Critical strengths

The paper has various positive aspects:

  • The paper aims to address an important and practical question: whether harder synthetic corruptions really lead to better error auditing, which is a common setting/assumption of many detectors and benchmarks.
  • The theoretical notions and definitions are well-formalized.
  • The experiments are reported carefully, and several findings are practically useful.
Critical weaknesses

The paper also contains some points that could be improved or answered:

  • The scope of what the experiments show might need to be clarified. For example, the experiments for the knowledge-graph comparisons are based on synthetic corruptions, this the results might be viewed as somewhat expected. Also, the effect sizes also vary largely (CAGED 0.647 vs CCA 0.033), and the experiments with real errors seem less effective or mixed. Thus it would be helpful to clearly separate which conclusions are supported by real errors and which by synthetic errors.
  • Also, the paper does not contain enough practical guidance on how to choose or construct a good corruption. Theorem 1 is an existence result that uses the latent error probability, and the terms in Theorem 2 cannot be measured without labeled real errors. In the experiments, resemblance is mostly created by design (e.g., using the same construction, semantic similarity) rather than measured, so it is unclear how a practitioner could check whether a corruption is close enough to the target errors.
  • The effect of the error ratio is not studied systematically. The theory suggests that the results can depend on the error ratio. However, the error ratio is fixed within each comparison, and while it differs across datasets, it is not varied in a controlled way. It would be helpful to show how the corruption-choice effects and the detector ranking change when the error ratio varies.
Questions for authors

The questions and suggestions are included in the above sections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.