- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies why evidence can reveal which claims share the same truth value without identifying which claims are true. It derives the number of trusted anchors needed to resolve this ambiguity, and proposes a theorem about the transfer between verification mechanisms. Experiments on MMLU-Pro, HumanEvalPlus, and studies of frozen model representations, demonstrate when the trusted anchors help, and when their orientation does not transfer.
Critical strengths
- The paper proposes a principled criterion for when agreement can and cannot certify correctness. Perfect relational evidence can leave individual claims uncertifiable, and the value of a trusted label depends on how its source relates to the target. This is a meaningful insight, so I lean toward acceptance.
Critical weaknesses
- The transfer comparisons in Section 4.2 share the same source and target mechanisms, so their successes are not independent. The main text does not specify the null hypothesis or what was randomized in the significance test. Without these information, it is hard to gauge the statistical evidence.
- (Updated November 18) Reading Reviewer Frnq's critical weakness Item 1 raises my severe concern about the novelty and originality of the work. While the authors have partially addressed this in their revised paper, they did not distinguish their contribution from prior work clearly in their initial submission, which they should have done, and the revisions making explicit the much thinner contribution are significant enough to alter a reviewer's judgment.
Questions for authors
For the significance test in Section 4.2, what was the null hypothesis, and what was randomized? How were dependencies between transfer comparisons preserved?