- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
BiMetaJudge audits black-box LLM judges for stable but wrong judgments by testing whether each verdict stays fixed under quality-preserving edits and moves as expected under verified degradations and repairs, then uses these responses for risk-based selective acceptance with a finite-sample certificate. On a self-built benchmark of four domains and five judges, it improves error detection over the strongest baselines, with AUROC 0.824 versus 0.808 and stable-error recall 0.795 versus 0.280.
Critical strengths
The paper targets a real blind spot of consistency-based confidence: judgments that stay stable under repeated sampling and position swaps yet are still wrong. It also proves that consistency alone cannot tell correct judgments from wrong ones.
Critical weaknesses
- The core ideas largely repeat prior work such as CheckList, FBI and Trust or Escalate, so the novelty is limited.
- All results come from a single self-built benchmark, so their credibility is hard to judge.
- The baselines are weak and the closest related methods are not compared.
- The writing is dense and poorly structured, and reads as obvious AI slop.
Questions for authors
See Weaknesses.