- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks a valid question: When we can spot and fix wrong answers from a black-box language model by looking at its responses to several versions of the same question. These versions are connected by known rules: two answers might need to agree, or one might need to be three times another. Trusted outside checks, called anchors, help tie the answers to something more reliable than the model's own agreement. The authors introduce a number, , that captures how well these rules and checks expose errors in up to responses. Under the stated assumptions, a positive means those errors can be uniquely identified. With noise bounded by , the repair error is at most , and no method can uniformly beat the scale. Tests on synthetic data and four open models answering generated arithmetic and coding questions illustrate these findings. The useful insight is that a model can be consistently wrong; what matters is whether the available relations and checks give us enough independent evidence to correct it.
Critical strengths
Maybe summarized in three important overarching bullet points: 1. Generally clear problem statement and an ambitious endeavour. The authors argue that a set of answers can be fully consistent yet still completely wrong. 2. The tasks are generated with exact labels, which makes the experiments easy to audit and be tractable. 3. The paper carefully distinguishes between whether the correct answer can be uniquely determined in principle and whether a particular algorithm can actually recover it.
Critical weaknesses
Paper's main contribution is the idea of treating repair of related LLM answers as a sparse inverse problem, with regarded as a measure of how hard that repair is. The authors present the theorems as applications of known compressed sensing results. The paper should therefore be judged on what this framing adds and the implementation achieves rather than on complete novelty. To assess, authors should have compared their work with earlier methods that already fix model answers using relations between related questions, such as ConCoRD (Mitchell et al., 2022), Maieutic prompting (Jung et al., 2022) and BeliefBank (Kassner et al., 2021). Additionally, Fluri et al. who already showed that consistency does not imply correctness—as a side, to my observation, none of these works are properly cited.
A new framing of the problem these methods already address should therefore be providing results that the older methods, in the first place, could not have. What would have been opportune real data experiments collapse to majority vote when no anchor is used and to the answer key when they are, which clouds any claims of improvement.
Furthermore, I noiticed the main experiment cannot fail anyway. The replay test has one wrong answer and no added noise at all. Under such conditions, the theorem guarantees perfect recovery whenever is positive, as Table 6 clearly indicates, so the experiment remains nothing more than a cross-check for the theorem that provides no useful information about LLM errors. Some other predictions also follow from the definitions. Duplicate saturation, the anchor transition and consistency blindness hold by construction. Testing them numerically again adds no evidence whatsoever.
It should also be noted that the main text reports four models, but most appendices cover two (tables). Some numbers disagree between tables as well. Some replay counts are larger than what the dataset allows. For example, more replay fields than code items, several promised results are missing, and "SPARSE-RRF" is never even defined.
While the appendix size is not usually a main concern and could plausibly be kept long for a paper centered on theory, it actually runs for this MS over 50 pages, of which the main text refers to less than half. The only test on unmodified model outputs and the only quantitative test of the results appear only in the last appendix, which should be in the main text instead.
Questions for authors
- Can there be more than one replay field per item? Where do the correct answers come from for items for which the model produced no correct answer?
- Is there any result on real data where a larger positive yields better recovery than a smaller positive ?
- How does the math anchor differ from the ground-truth label?
- Does Theorem 5 still hold when noise affects only the anchor rows?
- What regularization was used in the logistic model? Does the coefficient remain finite without it?
- Can you show one real task with partial anchors or non-identity relations where combining relations and anchors outperforms using either one alone?
Response to Author's Rebuttal
Thanks for the detailed response and for some of the changes to the current manuscript. The replay accounting, the analysis of anchor-only noise in Proposition 6, and the regularization analysis mostly addressed my earlier technical questions. The expanded discussion of prior work also clarified the contribution of the decoder-independent recoverability certificate. Yet, I still have a few more remaining concerns.
Of the 400 naturally generated fields in the error classes reported in Table 19, 90 contain exactly one incorrect response and remaining 310 contain shared or multiple errors. The 100% recovery observed at positive margins seems to be therefore conditional on an error class identified using the ground-truth answers. So here is a question: Can the corruption budget (k), or the decision to abstain, be determined from the information available at the inference time? Evaluating a rule specified before the evaluation labels are examined would establish how often the certificate can guide repair in practice.
Secondly, each field contains an original claim, two paraphrases, and a negation. Reversing the sign of the response to the negated claim reduces the relations in this experiment to equality constraints. The results demonstrate that partial anchors can improve recovery in this specific domain. A relation that cannot be reduced to agreement by recording individual responses, such as the relation between a total and its components, would provide a more direct test of the framework’s broader applicability in my opinion. For the present experiment though, would you please specify how the retrieval anchors were verified and whether the coverage was varied across fields or within each field, and if so, what would the impact be?
Even after calibration, adding the cross-checkpoint anchor decreases accuracy relative to RRF-D for three of the four target models in Table 26. For Qwen2.5-7B, accuracy decreases further from 82.8% to 81.2%. Reporting paired gains and losses would assist the reader characterize this effect. It would also be informative to examine whether anchor reliability and the extent to which anchor and target errors coincide would guide the decision to use an anchor or maybe not.
The code experiments show that fewer labeled tests can suffice on this benchmark. To assess the practical savings, authors are advised to report the number and cost of unlabeled probe executions and include a baseline with access to the same labeled tests and probe outputs. Although the authors’ revisions have considerably strengthened the paper, the limited significance of the contribution keeps my assessment at Weak Accept (3).