acceptodds
Under review as a conference paper at ICLR 2027

WARD: Certifying Failures of LLM Judgment Verification

Abstract

Large language models (LLMs) are widely used as judges to assess system responses. Often, a second LLM is deployed to check that judgment. This checker, or Verifier, should support correct judgments and withhold support from wrong ones. In this paper, we investigate whether we can provably certify Verifier failures even when ground-truth judgments are not feasible to collect. To illustrate, consider a product review labeled positive by a classifier and reverse its sentiment while keeping the predicted label fixed. The Judge assesses whether this label is correct for each version. Since the two versions have opposite sentiments, the Judge should accept one classification and reject the other. If the Judge accepts both and the Verifier supports both verdicts, at least one verification decision must be wrong, even without knowing which version is positive. To address this, we propose World and Role Diagnostics (WARD), which links each Judge verdict to the Verifier’s support decision across such paired cases. We prove that combining these case relationships with the judgment-support matches can establish failures that neither kind of evidence can establish alone. Our theory determines all possible error counts consistent with the evidence, allowing a specified number of case relationships to be wrong. Across five LLMs on six tasks spanning sentiment, stereotypes, helpfulness, safety, and bias-focused question answering in English and Spanish, WARD proves 3.70–6.77 additional errors per 100 decisions for four of the five Verifiers beyond the stronger separate check. This gain repeats on both question-answering tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM-as-a-judge, judgment verification, model auditing, partial identification, error certification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pXXSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WARD, a framework for auditing LLM-based verifiers without requiring exact ground-truth labels. WARD uses relational constraints between paired cases to reason about hidden ground-truth judgments. This paper shows that combining these relations with Judge–Verifier links can certify errors that neither source of information can reveal alone. Experiments across multiple tasks and LLMs demonstrate that WARD can identify substantial unavoidable verification errors.

Critical strengths
  1. This paper studies an important and timely problem in LLM-based verification.

  2. The authors provide a clear insight into combining relational constraints with Judge–Verifier links.

  3. The theoretical analysis provides rigorous and sharp error bounds.

  4. The experiments validate the framework across multiple tasks and model families.

Critical weaknesses
  1. The methodological novelty relative to metamorphic testing and partial-identification frameworks could be more clear.

  2. The certificates critically depend on valid OW/AW relations, which may be expensive or difficult to guarantee in realistic settings.

  3. The two-stage Verifier interface is somewhat stylized, limiting conclusions about other commonly used verification pipelines.

  4. The proposed WARD method certifies aggregate failures but does not identify or repair individual errors.

  5. The paper writing quality can be improved. For example, the font sizes in some figures are too small.

Questions for authors

Please refer to the weakness part.

1 comment
AuthorsOpenReview ↗

Response: Why Linked Evidence Makes Verification Failure Provable

We thank the reviewer for recognizing WARD's insight and sharp bounds. The central scientific claim is: WARD proves when Verifier errors must exist even though the Auditor lacks individual correct verdicts. Theorem 1 constructs four decisions where RM and L0 each certify zero errors, yet combined evidence (RL) certifies two.

Our new evidence addresses the main concerns:

New evidencePurposeFinding
Actual vs. certified errorsIs the floor informative?241,828/335,712 (72.03%)
Reliability-threshold auditDoes the conclusion change?17 RL-only FAIL comparisons
Label-free SST-2 / BoolQCan relations exist without labels?RL 8.20% / 31.53%
Direct verification / simple matchingIs Stage A/B necessary?Certificates under both

W1. What is new beyond metamorphic testing and partial identification?

We have explicitly addressed this comparison in Table 1 and Tables A1–A3 (Appendix B). Metamorphic testing detects violations of known relationships between cases. It does not, by itself, establish how many Judge–Verifier support decisions must be wrong when the correct verdicts are hidden. Partial identification provides general tools for bounding unknown quantities, but does not specify which Judge–Verifier links and cross-case relationships must be combined, or establish what additional failures this combination can certify. WARD makes this connection explicit and derives sharp error guarantees. Theorem 1 proves the distinction. Each restricted evidence view permits zero errors, while their combination necessarily certifies two errors among the same four decisions. Our new audit-decision analysis also identifies 17 comparisons where this additional information changes the failure conclusion.

Key takeaway: WARD's contribution lies in the specific information-composition result and its sharp verification-failure certificates. We do not claim metamorphic relations or partial identification as new.


W2. Can the required relations be obtained reliably without costly labels?

WARD explicitly states its relation assumptions and allows some relations to be released when unreliable. We added task-defined relations.

For 500 SST-2 examples, the same review is paired with proposed labels POSITIVE and NEGATIVE.

For 500 BoolQ examples, the same passage and question are paired with YES and NO. Exactly one proposed answer is correct under each binary task's reference definition. We know the verdicts are opposite without knowing which answer is right. No gold labels were used to build pairs or compute certificates. For BoolQ, allowing any 10 of 500 relations to be invalid still gives RL 30.53% versus RM 29.87%. Only RL establishes failure at a 30% threshold.

Key takeaway: Relations can be available without individual labels; imperfect relations are covered by an explicit robustness budget.


W3. Does the certificate depend on the two-stage Verifier?

No. It requires a Judge verdict, the Verifier's linked support decision, and the relation between cases. Stage A is not a theorem assumption.

With direct one-stage verification on SST-2 and BoolQ, RL certifies 8.20% and 31.53% errors. We also audited a deterministic rule that supports the Judge exactly when its verdict matches frozen Stage A. On the same 766,345 decisions, that rule made 292,297 errors versus 335,712 for Stage B; WARD still certified failures. The direct prompts did not include Judge rationales.

Key takeaway: The certificate applies beyond the original Stage-A/B implementation.


W4. Why is aggregate certification useful without locating or repairing errors?

An audit can establish that a safeguard fails without locating every mistake. On PKU-Helpfulness with Mistral as Verifier, suppose both error types must be at most 40%:

EvidenceCertified total error40% requirement
RM27.31%Failure not established
L03.92%Failure not established
RL45.76%FAIL certified

The decisions are identical; only the Auditor's information differs. Across the original tasks, this occurs in 17 threshold comparisons (16 distinct task–Verifier combinations).

Key takeaway: WARD turns complementary evidence into a justified reliability decision, without needing individual error locations.


W5. Regarding presentation clarity.

We will lead with the four-decision example, distinguish actual errors from certified minima, enlarge figure labels, and point readers to the existing task-level and ordered-pair analyses.


WARD's contribution is to establish what an Auditor can prove from incomplete evidence. The theorem provides the separation. The new analyses show that its certificates cover 72.03% of realized errors and sometimes change a failure decision.

Reviewer_LtmVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to audit an LLM Verifier that decides whether to support another LLM's (the Judge's) verdict, when ground truth is unavailable. The idea is to use case pairs whose correct verdicts are known to be opposite (an Original World and an Alternate World), without knowing which verdict is correct in either.

The authors organize audit information into regimes, as outlined in Table 1. Theorem 1 shows a four-decision construction where relations-plus-totals (RM) and links alone (L0) each allow zero error, while relations-plus-links (RL) certifies two errors. Theorem 2 gives the exact set of confusion vectors consistent with the evidence, using four counts per case.

Critical strengths

The problem statement is clear. Asking what an audit can prove from what it records is a useful lens. The theoretical contribution seems correct and modestly useful.

The paper presents a way to handle imperfect relations. The release budget gives a worst-case guarantee without requiring the auditor to know which pairs are invalid.

Critical weaknesses

The main weakness is the gap between the motivation and the evaluation. The paper motivates a label-scarce, relation-rich setting, but evaluates only in a label-rich setting with weak models, a nonstandard verification interface, and no comparison against simple alternatives.

  1. The method is valuable only when verdict-flipping relations can be obtained more cheaply than labels. In all six tasks, the Alternate World was created by annotators or templates that already determine both labels. In general, constructing an "opposite" relation requires the same semantic understanding that would yield the label. A crowdworker who rewrites a positive CAD review to be negative knows both labels, and so does the author of a less-stereotypical CrowS-Pairs sentence. PKU pairs come from existing annotations, and BBQ answers are fixed by template. The experiments therefore discard labels they possess and reconstruct a weaker relation from them.

  2. Report each Verifier's true FP+FN rate next to its certified floor. Without this, a floor of 27% cannot be interpreted; it means very different things if the true error is 30% or 60%.

  3. The verification setup is artificially restrictive:

  • Hiding the Judge's reasoning departs from how verification is done in practice, where checking the argument is the point.
  • The Verifier forms its own verdict, then only checks whether the Judge's scores agree with it. Verification reduces to "re-judge independently, then compare," and Verifier correctness is driven almost entirely by Stage-A accuracy, as the paper's own decomposition in Appendix D.8 implies. One Stage-A verdict is reused across all five Judges.
  1. The five models are 8B–32B open-weight checkpoints, mostly from 2024. Appendix Table A33 reports average binary Stage-A accuracy of only 51.1–65.6% across tasks, near chance for DeepSeek and Mistral. Model quality also matters for the paper's value, not just its validity. Proving that a near-chance Verifier fails is easy, and almost any audit would catch it. The important case is a strong Verifier with rare errors, where certified floors may be close to zero. The experiments never test whether WARD remains informative there.
Questions for authors
  1. Can you show a task where verdict-flipping relations are substantially cheaper to obtain than labels? In all six tasks, the pair creators knew both labels.
  2. On the labeled datasets, what is each Verifier's true FP+FN rate, and how close is the certified RL floor to it?
  3. What is each model's accuracy on each dataset under its standard protocol (e.g., plain zero-shot sentiment classification on the CAD items, standard BBQ accuracy)? How do these compare with published results and with your Stage-A accuracy?
  4. How do the results change with a single-stage Verifier that sees the case and the Judge's judgment, including its reasoning, together?
  5. What does WARD report for a strong, recent Verifier whose true error is low? Does the certificate remain informative in that case?
4 comments
AuthorsOpenReview ↗

Response: From Label-Free Relations to Certified Verification Failure

We thank the reviewer for recognizing WARD's theoretical contribution. The key distinction is between errors a Verifier makes and errors an Auditor can prove without knowing each correct verdict. Theorem 1 shows that two restricted evidence views each allow zero errors, yet combining them proves two errors on the same four decisions.

We conducted targeted additional analyses and experiments:

EvidenceQuestion addressedResult
Actual errors vs. RL floorIs the certificate informative?72.03% of actual errors certified
Label-free SST-2 and BoolQMust relation construction use gold labels?RL 8.20% and 31.53%
Direct verification; Stage-A matchingIs Stage B required?Certificates under both interfaces
Reliability-threshold analysisCan an audit decision change?17 RL-only threshold comparisons

Q1 [W1]. Can opposite-verdict relations be obtained without knowing the labels?

Yes, when a task has exactly two valid answers. For SST-2, we hold the review fixed and propose POSITIVE in one world, NEGATIVE in the other. For BoolQ, we hold the passage and question fixed and propose YES versus NO. Exactly one proposed answer is correct, although we do not know which. We constructed 500 pairs per task and calculated the certificates without consulting gold labels.

With direct verification, RL certifies 246/3,000 errors (8.20%) on SST-2 and 946/3,000 (31.53%) on BoolQ. This establishes a concrete setting where the relation follows from the task, without a semantic rewrite or individual reference labels.

Key takeaway: We can know that two verdicts are opposite without knowing either verdict.


Q2 [W2]. What are the actual FP+FN errors, and how close is RL?

We compared certificates with gold labels only after computing them, on the identical decided records. Across 766,345 decisions, there are 127,085 false-support and 208,627 missed-support errors: 335,712 total. Without using these labels, RL certifies 241,828, or 72.03% of the actual count. It is also informative on lower-error cohorts:

Dataset, VerifierActual errorRL floorTightness
CAD, Qwen1412.01%11.40%94.88%
CAD, Qwen3211.62%10.65%91.67%

Key takeaway: WARD proves a substantial share of the errors that actually occurred.


Q3 [W4]. What is each model's standard task accuracy, compared with Stage A?

Task accuracy measures how well a model answers a question. WARD measures what an Auditor can prove about the Verifier from incomplete evidence. These are different measurements. Appendix Table A33 already reports Stage-A accuracy under its protocol. We have not conducted a separate standard zero-shot benchmark comparison. The new FP+FN analysis instead measures actual Verifier errors alongside certified floors on the same cases. Standard benchmark accuracy does not test the certificate's correctness.

Key takeaway: Task accuracy, verification accuracy, and certifiable error are distinct.


Q4 [W3]. What happens with a single-stage Verifier or a simpler checker?

WARD's certificate does not require Stage A/B. On SST-2 and BoolQ, a Verifier directly assessed each Judge verdict without Stage A, and WARD still certified errors.

We also tested SUPPORT iff the frozen Stage-A verdict matches the Judge's verdict. On the same 766,345 decisions, this deterministic checker made 292,297 errors, compared with 335,712 for Stage B. WARD still certified failures under the simpler, lower-error checker. Direct prompts did not include Judge rationales; that condition was not tested.

Key takeaway: WARD certifies errors from recorded support decisions, independently of Stage B.


Q5 [W4]. Must we use larger, stronger Verifiers with rare errors?

Model size is not part of the certificate's assumptions. An accurate Verifier may make fewer errors. WARD cannot certify errors not forced by the evidence. The CAD results above already show informative floors around 12% actual error. This does not imply that all models make similar errors. Errors that occur and errors that can be proved are different quantities.

Key takeaway: The guarantee is model-agnostic; informativeness depends on the evidence.


Our findings have a practical consequence. For PKU-Helpfulness/Mistral at a 40% reliability limit, RM certifies only 27.31% errors and L0 only 3.92%, whereas RL certifies 45.76%: failure is established. Across the original tasks, this occurs in 17 task–Verifier–threshold comparisons. We request that WARD be assessed for its actual contribution: provable verification failure from incomplete evidence.

AuthorsOpenReview ↗

Gentle reminder for feedback

Dear Reviewer LtmV,

Thank you again for your thoughtful review and for recognizing the value of WARD's theoretical contribution. As the discussion period approaches its end, we wanted to gently follow up on our response and revised manuscript. We have added results addressing several of your main concerns:

  • Label availability: New SST-2 and BoolQ experiments construct opposite-verdict pairs without using reference labels.
  • Certificate informativeness: WARD certifies 72.03% of the observed verification errors. Our response also reports certificate tightness of 94.88% and 91.67% in two lower-error CAD settings.
  • Verification protocol: We evaluated direct verification without Stage A and compared Stage B against a simple deterministic matching baseline.
  • Practical audit decisions: We identified 17 comparisons where combining the evidence certifies a reliability failure that neither restricted view can establish.

All additions and revisions are highlighted in blue in the revised manuscript. We would greatly appreciate if you can provide your feedback on whether these new results address your main concerns, particularly regarding the soundness and informativeness of the contribution. We would be happy to clarify any remaining concerns before the discussion closes. Thank you for your time and consideration.

Thanks for your response. My concern with the paper comes down to one major issue; I believe the paper claims practical relevance based on a narrow experimental setup, one that does not test the situation in which WARD would actually be needed.

WARD is meant for the case where we cannot get labels but can still know that two verdicts must be opposite. The main experiments do not test this case. In all six tasks, the opposite-verdict pairs come from labeled datasets: we know the verdicts are opposite because we already know the labels. The authors’ response to Q1 shows that pairs can be built without labels by swapping the proposed answer, but only for tasks with exactly two valid answers, and only in the new SST-2 and BoolQ experiments. On all six tasks in the paper, the relations were derived from labels the authors already had.

The method is also limited in what it can conclude. If a Verifier’s error rate is within the tolerance, WARD can never certify it as passing (this follows from Proposition 3 together with the FAIL criterion). Therefore, WARD can only tell us that a Verifier is bad. To make an argument for WARDs practicality, the paper would need to show it exposing the failures of a decent Verifier.

The experiments do not show this, and the Verifiers tested are actually very weak. They are wrong on 43.8% of their decisions (335,712 out of 766,345), barely better than a coin flip for a yes/no decision. A simple fixed rule, “support the Judge whenever it agrees with the Stage-A verdict,” makes fewer mistakes (38.1%) than the LLM does in Stage B. Proving that such a Verifier fails is easy, and almost any audit would catch it. This is why I asked for stronger models (Q5). I agree that the certificate itself does not depend on model size, but my point was that a strong Verifier is the case that decides whether WARD matters in practice, and the response does not address it.

I hope my elaboration on my concern is more clear now. I am going to keep my score, but I have assigned a low confidence to my score because this line of work is not my area of expertise. The AC may choose to accept the paper.

AuthorsOpenReview ↗

Clarifying WARD's Practical Significance and Certification Limits

Thank you for explaining your remaining concern. We truly appreciate your careful engagement. We believe three distinctions are important, particularly because our existing results already include a case relevant to your central objection.


1. Are the required case relationships available without reference labels?

The six original tasks use labeled datasets to validate the case relationships and measure actual verification errors. Please note that those labels are not used‾\underline{\textit{not used}} to compute WARD's certificates. These experiments establish whether the certificates are correct and informative.

Our additional SST-2 and BoolQ experiments address the separate question of obtaining relationships without labels. We keep the input unchanged and switch between the two possible answers. Exactly one answer must be correct, although we do not know which. No reference labels are needed to construct these pairs or compute their certificates.

Note that we do not claim that every task permits this construction. The experiments establish concrete settings in which the required relationships are available without knowing individual correct answers.


2. Is the inability to certify PASS a weakness?

We would like to clarify that Proposition 3 establishes a fundamental limit on what any‾\underline{any} sound audit can conclude from the specified incomplete evidence.

Without a reference label fixing the correct orientation, two opposite assignments of the hidden correct verdicts can remain possible. One may make the Verifier appear reliable and the other unreliable. No sound Auditor using only that evidence can certify two-sided PASS under the theorem's conditions.

WARD makes this limitation explicit rather than reporting an unjustified PASS.

It can establish FAIL when every compatible assignment violates the requirement. Otherwise, the audit remains UNRESOLVED. With additional reference information, PASS may also become certifiable. Theorem 7 provides a principled way to select reference-label queries that reduce uncertainty under its stated assumptions. Note that WARD is an audit-and-escalation framework, not merely an error detector. It tells us what the evidence proves and when further checks are needed.


3. Does WARD only establish failures for weak Verifiers?

We believe an important result in Table A22 directly addresses this concern. On CAD, Qwen32 makes 2,437 errors among 20,970 decisions (11.62%). Without using reference labels, WARD certifies at least 2,234 errors (10.65%), capturing 91.67% of the actual errors.

Consider a reliability limit of 10%:

EvidenceGuaranteed errorsFailure at 10%?
Relations and totals (RM)8.59%Not established
Judge–Verifier links (L0)4.37%Not established
Combined evidence (RL)10.65%Certified FAIL

Note above that the same Verifier decisions are used throughout. Only WARD's combined evidence proves failure. This is a lower-error setting where the additional audit information makes a decisive difference. On CAD, Qwen14 provides another informative result: 12.01% actual errors, 11.40% certified, and 94.88% tightness. These results establish that WARD's practical value is not limited to identifying failures in the highest-error settings.

The overall 43.81% error rate pools decisions across all‾\underline{all} tasks and Verifiers. It does not describe every evaluated setting. We also deliberately reported the unfavorable deterministic matching baseline. WARD remains applicable under that simpler verification rule as well.


Key takeaway: WARD does not need a Verifier to be weak to produce an informative audit. Its contribution is to establish what can be proved from incomplete evidence. Our results include a lower-error setting where WARD alone certifies failure, and label-free settings where the required case relationships are obtained without knowing the answers. We hope these clarifications address the distinction between the paper's theoretical contribution and the additional empirical settings that could be studied. Thank you again for your thoughtful assessment.

Reviewer_VFYYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

WARD proposes a framework for certifying errors in LLM-based verification without requiring ground-truth labels for every case. It combines two sources of partial information: relationships between paired cases (e.g., cases expected to receive identical or opposite judgments) and correspondence between a Judge’s verdict and a Verifier’s decision. Theoretically, it derives sharp bounds on the number of verification errors. Experiments across six tasks and five open-weight models demonstrate that combining these sources can reveal additional errors.

Critical strengths
  • Combining cross-case relationships with Judge Verifier correspondence provides a principled way to derive error bounds without complete ground truth.
  • Relatively broad evaluation: Experiments cover six tasks, 18,039 pairs, five models, and both English and Spanish datasets
Critical weaknesses
  • The paper does not convincingly establish how common or necessary the separate Judge–Verifier setup is in real-world LLM evaluation. Moreover, the manually constructed incorrect judgments may introduce artificial artifacts. In practice, alignment between LLM judgments and human judgments is often a more direct concern.

  • WARD provides aggregate error bounds but cannot identify which individual verification decisions are incorrect or help correct them. The practical value of such certificates therefore remains unclear.

  • The evaluation uses only five open-weight models and one prompt family, with no human-grounded validation. It is unclear whether the findings generalize to realistic evaluation settings or stronger models.

  • The evaluation relies exclusively on five relatively small (8B–32B) open-weight models, whose limited capabilities may not adequately represent the behavior of stronger contemporary LLMs. The empirical evidence is also relatively thin, with most quantitative results concentrated in one summary table and one multi-panel figure. Consequently, the reported findings may not generalize to stronger or proprietary models, making the paper's broader claims insufficiently supported.

Questions for authors
  • What concrete applications would benefit from WARD? How sensitive are the guarantees to incorrect paired-case relationships in realistic settings

  • How does WARD relate to the more practical problem of measuring whether LLM judges agree with human judgments? Could the framework provide useful guarantees about human-level evaluation reliability?

  • Could you please provide some experiments on more intelligent models with larger sizes and include some close-source models?

5 comments
AuthorsOpenReview ↗

Clarifying WARD’s Scientific Claim (Part-I): When Incomplete Evidence Is Enough to Prove Verification Failure

We thank the reviewer for recognizing WARD's core insight and broad evaluation. We clarify the central question: When correct verdicts are unavailable, can an Auditor still prove that a Verifier made mistakes? Theorem 1 establishes that two sources of evidence, each insufficient alone, can certify failures when combined. We have conducted the following additional experiments/analyses that strengthen the core contribution:

New evidenceQuestion addressedMain finding
Actual errors vs. certified errorsHow informative is the certificate?Certifies 72.03% of actual errors
Audit-decision analysisDoes combining evidence change a practical conclusion?17 additional certified failure comparisons
Label-free SST-2 & BoolQ, with direct verificationCan relations exist without labels? Is Stage A necessary?RL floors of 8.20% and 31.53%
Relation-error sensitivityWhat if some OW/AW relations are incorrect?BoolQ retains a 30.53% RL floor with 10 possibly invalid pairs

Q1 [W1, W2]. What applications benefit from WARD, and what if paired-case relations are incorrect?

LLM-based verification is already studied in the literature. Eg., Wu et al. and Ma et al. (EMNLP 2025) use meta-judge to evaluate a Judge's judgments, although their verification mechanisms differ from ours. Note that WARD does not need a separate model for verification. We evaluate both self- and cross-verification.

WARD tests whether an Auditor can establish that such verification is unreliable without knowing every correct verdict. Consider an organization requiring both verification error rates to remain at or below 40%. It cannot afford to obtain the correct verdict for every decision, but it needs to know whether its Verifier violates this requirement. Our new results provide a concrete example using PKU-Helpfulness with Mistral as Verifier:

Evidence availableCertified errorsFailure at 40%?
Relation + totals (RM)27.31%Not established
Judge–Verifier links (L0)3.92%Not established
Both combined (RL)45.76%Certified FAIL

The actual error rate, measured separately, is 49.74%. Across six tasks, we find 17 threshold comparisons covering 16 distinct task–Verifier combinations where RL certifies failure but neither restricted view does. The recorded decisions remain identical. Only the information available to the Auditor changes.

Regarding relation validity, WARD already supports a declared budget of incorrect relationships. In our new BoolQ experiment, allowing any 10 of 500 relations to be invalid still leaves RL at 30.53%, compared with RM at 29.87%. Only RL establishes failure at the 30% threshold.

Key takeaway: Combining evidence changes what an Auditor can prove, including when some relationships may be unreliable.


Q2 [W1, W3]. How does WARD relate to agreement with human judgments?

Human-agreement evaluation asks whether model judgments match human references. WARD asks what can be proved when those reference judgments are unavailable for individual cases. These are different, complementary questions.

Note that our benchmarks already contain reference judgments. We deliberately withhold them during certification, then use them to evaluate the certified lower bounds. Across 766,345 decided verification records, we observe 335,712 actual errors. WARD independently certifies at least 241,828, capturing 72.03% of the actual error count without consulting the references during certification. This evaluates how informative the certificate is. It is not presented as a separate human-agreement study.

Key takeaway: WARD establishes what remains provable when complete reference-based evaluation is unavailable.


Q3 [W3, W4]. Why not evaluate larger or closed-source models?

WARD is an auditing framework, not a model-capability benchmark. Its certificates depend on recorded Judge verdicts, linked Verifier support decisions, and valid case relations. Model size and architecture are not assumptions of the guarantee. A model may make many mistakes without the Auditor having enough evidence to prove them. Another may make fewer mistakes but permit a tighter certificate. Hence, errors that occur and errors that can be proved are fundamentally different quantities.

Our controlled comparison keeps the model decisions fixed and changes only the evidence retained by the Auditor. Across five model families, six tasks and 25 ordered Judge–Verifier combinations, RL establishes 3.70–6.77 additional errors per 100 decisions for four of five Verifiers beyond the stronger restricted view. We do not claim identical error rates across models. Our claim concerns what the available information can certify.

Key takeaway: WARD tests the certifying power of audit evidence, not how errors scale with model size.


Please refer to Part II below for additional clarifications.

AuthorsOpenReview ↗

Clarifying WARD’s Scientific Claim (Part-II): When Incomplete Evidence Is Enough to Prove Verification Failure

Continued ...


Additional concerns [W1–W4]. Constructed errors, verification protocol, error repair and evidence breadth

We clarify four points.

First, the datasets provide controlled case relationships. They do not manually manufacture incorrect Judge verdicts. Those verdicts are generated by the evaluated models.

Second, Stage A/B is an experimental interface, not a requirement of the certification theorem. Our new SST-2 and BoolQ experiments use direct verification without Stage A and still produce nonzero certificates. The alternative worlds are constructed by switching between binary candidate answers without consulting the correct labels.

Third, WARD certifies aggregate failure rather than identifying or repairing individual decisions. An Auditor can establish that a reliability requirement has failed without locating each error. The Q1 results demonstrate why this distinction matters.

Fourth, the empirical evidence extends beyond the main summary table. The appendices contain task-level, ordered-pair and relation-budget analyses. We will improve the navigation and enlarge figure labels to make these results easier to follow.

Key takeaway: Neither manually fabricated errors, a particular Verifier implementation, nor individual error repair is necessary for WARD's stated contribution.


WARD distinguishes mistakes a Verifier makes from mistakes an Auditor can prove using limited evidence. Theorem 1 establishes this distinction mathematically. Our additional results demonstrate that the resulting certificates are informative, can change reliability-audit decisions, and can use relations constructed without knowing individual correct answers. We respectfully ask that WARD's soundness be assessed against these claims. If a specific mathematical concern remains, we would welcome identification of the assumption, theorem, or calculation in question.

Thank you for the informative reply and additional experiments. While the new results strengthen the paper, two concerns remain:

  1. Larger models: The theoretical guarantees may be model-independent, but empirical behavior may differ across model capabilities. No larger or closed-source models are evaluated.
  2. Real-world applicability: The proposed threshold example illustrates potential utility but does not demonstrate applicability in realistic deployments.

Overall, the additional evidence is useful, but the concerns regarding empirical generalization, and practical relevance remain insufficiently addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript

We have uploaded a revised WARD manuscript incorporating the additional analyses and methodological clarifications discussed in our responses. The central research question and theoretical guarantees remain unchanged.

The revision includes an analysis of certificate tightness against observed verification errors, 17 task–Verifier–threshold comparisons where only the combined evidence certifies reliability failure, label-free direct-verification experiments on SST-2 and BoolQ, and a deterministic verification baseline. We have also clarified the experimental protocols, evaluation populations, and limitations of these comparisons, with supporting details in the appendices.

All newly added or revised content is highlighted in blue to make the changes easy to identify.

We welcome the AC's and reviewers' assessment of these additions and would be happy to clarify any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.