- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WARD, a framework for auditing LLM-based verifiers without requiring exact ground-truth labels. WARD uses relational constraints between paired cases to reason about hidden ground-truth judgments. This paper shows that combining these relations with Judge–Verifier links can certify errors that neither source of information can reveal alone. Experiments across multiple tasks and LLMs demonstrate that WARD can identify substantial unavoidable verification errors.
Critical strengths
-
This paper studies an important and timely problem in LLM-based verification.
-
The authors provide a clear insight into combining relational constraints with Judge–Verifier links.
-
The theoretical analysis provides rigorous and sharp error bounds.
-
The experiments validate the framework across multiple tasks and model families.
Critical weaknesses
-
The methodological novelty relative to metamorphic testing and partial-identification frameworks could be more clear.
-
The certificates critically depend on valid OW/AW relations, which may be expensive or difficult to guarantee in realistic settings.
-
The two-stage Verifier interface is somewhat stylized, limiting conclusions about other commonly used verification pipelines.
-
The proposed WARD method certifies aggregate failures but does not identify or repair individual errors.
-
The paper writing quality can be improved. For example, the font sizes in some figures are too small.
Questions for authors
Please refer to the weakness part.
Response: Why Linked Evidence Makes Verification Failure Provable
We thank the reviewer for recognizing WARD's insight and sharp bounds. The central scientific claim is: WARD proves when Verifier errors must exist even though the Auditor lacks individual correct verdicts. Theorem 1 constructs four decisions where RM and L0 each certify zero errors, yet combined evidence (RL) certifies two.
Our new evidence addresses the main concerns:
W1. What is new beyond metamorphic testing and partial identification?
We have explicitly addressed this comparison in Table 1 and Tables A1–A3 (Appendix B). Metamorphic testing detects violations of known relationships between cases. It does not, by itself, establish how many Judge–Verifier support decisions must be wrong when the correct verdicts are hidden. Partial identification provides general tools for bounding unknown quantities, but does not specify which Judge–Verifier links and cross-case relationships must be combined, or establish what additional failures this combination can certify. WARD makes this connection explicit and derives sharp error guarantees. Theorem 1 proves the distinction. Each restricted evidence view permits zero errors, while their combination necessarily certifies two errors among the same four decisions. Our new audit-decision analysis also identifies 17 comparisons where this additional information changes the failure conclusion.
Key takeaway: WARD's contribution lies in the specific information-composition result and its sharp verification-failure certificates. We do not claim metamorphic relations or partial identification as new.
W2. Can the required relations be obtained reliably without costly labels?
WARD explicitly states its relation assumptions and allows some relations to be released when unreliable. We added task-defined relations.
For 500 SST-2 examples, the same review is paired with proposed labels POSITIVE and NEGATIVE.
For 500 BoolQ examples, the same passage and question are paired with YES and NO. Exactly one proposed answer is correct under each binary task's reference definition. We know the verdicts are opposite without knowing which answer is right. No gold labels were used to build pairs or compute certificates. For BoolQ, allowing any 10 of 500 relations to be invalid still gives RL 30.53% versus RM 29.87%. Only RL establishes failure at a 30% threshold.
Key takeaway: Relations can be available without individual labels; imperfect relations are covered by an explicit robustness budget.
W3. Does the certificate depend on the two-stage Verifier?
No. It requires a Judge verdict, the Verifier's linked support decision, and the relation between cases. Stage A is not a theorem assumption.
With direct one-stage verification on SST-2 and BoolQ, RL certifies 8.20% and 31.53% errors. We also audited a deterministic rule that supports the Judge exactly when its verdict matches frozen Stage A. On the same 766,345 decisions, that rule made 292,297 errors versus 335,712 for Stage B; WARD still certified failures. The direct prompts did not include Judge rationales.
Key takeaway: The certificate applies beyond the original Stage-A/B implementation.
W4. Why is aggregate certification useful without locating or repairing errors?
An audit can establish that a safeguard fails without locating every mistake. On PKU-Helpfulness with Mistral as Verifier, suppose both error types must be at most 40%:
The decisions are identical; only the Auditor's information differs. Across the original tasks, this occurs in 17 threshold comparisons (16 distinct task–Verifier combinations).
Key takeaway: WARD turns complementary evidence into a justified reliability decision, without needing individual error locations.
W5. Regarding presentation clarity.
We will lead with the four-decision example, distinguish actual errors from certified minima, enlarge figure labels, and point readers to the existing task-level and ordered-pair analyses.
WARD's contribution is to establish what an Auditor can prove from incomplete evidence. The theorem provides the separation. The new analyses show that its certificates cover 72.03% of realized errors and sometimes change a failure decision.