- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes RESOLVE, an anytime-valid procedure for self-consistency in LLMs that audits whether the most probable individual answer category is labeled PASS or FAIL by an external verifier. The novelty of the proposal is that it does not estimate the total probability of PASS answers, returns the mode, or improve the model’s answers like prior works. Its central distinction is that PASS responses can become increasingly likely to appear in a sample while a FAIL answer remains the plurality winner. RESOLVE uses a fixed evaluator, discovers answer categories online, and reports PASS-favoring, FAIL-favoring, or unresolved. The proposed methodology is backed up by a thorough theoretical analysis based on e-processes and is evaluated across several LLMs and benchmarks.
Critical strengths
- The paper studies the theoretical guarantees for identifying the label of the population model which is different from prior work on self-consistency that mainly study it as a mode estimation problem.
- The proposed theoretical analysis controls false directional reports under continuous inspection, while allowing answer categories to be discovered online. The paper also analyzes how separating the label from exact answer identity can reduce evidence requirements.
- Experiments include three models on the evidence tasks and probe several relevant dimensions: evaluator acceptance versus plurality, repeated-testing error, audit difficulty, and same-label competitors.
Critical weaknesses
- The paper is difficult to parse. The distinction among PASS availability, total PASS mass, plurality label, exact mode, and “report” is central, but the dense notation and layered comparisons make it easy to lose track of the actual target. A running example and a more direct algorithm overview would help.
- RESOLVE depends on a fixed, deterministic, task-specific evaluator and category map. That need not mean an extra LLM judge; an answer key or annotation rule can suffice. But the method does not address evaluator errors or ambiguous criteria, and its usefulness is limited where reliable evaluation is unavailable. I think it is preferable to frame this as a key assumption and scope limitation.
- The formal guarantee is per query under fixed-policy, i.i.d. sampling assumptions. The paper notes that error levels can be allocated across a collection, but it does not provide a dataset-level guarantee by default. The guarantees also concern false reports, not how often the method resolves, how quickly it resolves, or the correctness of reports conditional on returning.
- The empirical support is rather limited.
Questions for authors
-
Can the paper state its core novelty more explicitly relative to CITE? In particular, which contribution is new: the label-level target, online discovery, the evidence construction, or their combination? Could the authors provide a full-method comparison where feasible, in addition to the Fixed-label/Fixed-mode mechanism check?
-
What is required of the evaluator in practice? How sensitive are the guarantee and conclusions to evaluator mistakes, ambiguous labels, or changes in the evaluator? Could the authors include an example showing how a real task's outputs are mapped to categories and labels?
-
Can the authors clarify how to extend the per-query error guarantee to a collection of queries and distinguish the guarantee for the capped monitor from the idealized response-cost bound? Could they also revise the exposition and figures/tables to make the target, algorithm, assumptions, and empirical evidence easier to follow?
7 comments
Response to Reviewer rCJs (2/4)
Answer for Q1 and W4. Novelty relative to CITE, full-method comparisons and new experiments
Which part is new. The combination of the label-level target with a test that stays valid under online discovery. The evidence construction is not new. The betting score is CITE's, and Section 1 now says so. CITE certifies that a supplied answer is the unique mode. Chen et al. (2024) and CALVER state when voting converges to a passing answer but do not test this for a given question. RESOLVE tests the label of the most likely answer, with no candidate and no unique winner.
Full-method comparisons. Mean responses until a report, counting the full budget when there is none.
| Comparison | RESOLVE | Other method |
|---|---|---|
| Full CITE, synthetic, true mode supplied (Table 12(b)), closest rival has the other label / the same label | 243.9 / 236.4 | 365.9 / 1,519.0 |
| Full CITE, synthetic S01–S03 with unknown answers (App. E.1) | 277.7 / 268.5 / 220.4, always reports | 351.5 / 865.3 / 1,022.1, unresolved in 0.05% / 59.6% / 99.6% of runs |
| Full CITE, real GPQA / MMLU-Pro (App. B.4) | 34.54 / 23.05 | 43.97 / 27.61, plus the 64 reference responses it needs to choose its target |
| CS audits of the label, categories known in advance (Table 14), GPQA / MMLU-Pro | 34.54 / 23.05 | best CS 36.47 / 21.29 |
| CS audits of the label, answers learned online (new, Section 5.4) | cheaper by 258 to 319 responses on spread-out synthetic streams | cheaper by 6.04 [5.38, 6.69] responses on 240 real evidence blocks |
The comparison with CS audits is mixed. CS is cheaper when one answer dominates, and RESOLVE when responses are spread over many answers (Figure 4(b), 4(c)). The gain over CITE comes from the label target, not from RESOLVE's particular construction.
New experiments in the revision (W4). A new cohort F1110 (518 questions, 804 response blocks) generated after the reviews, CS audits replayed on all 698 archived blocks, ten synthetic settings with unknown answers (S01–S10), an evaluator replay on 240 blocks, two LLM judges on 5,444 answers and two negative tests of practical use (App. E–H).
Response to Reviewer rCJs (3/4)
Answer for W2 and Q2. The evaluator
We agree, and we now state this as a key assumption and a limit of scope (Sections 2.1 and 6). All results are relative to the evaluator and do not correct its errors.
Requirement (Section 2.1). The evaluator is fixed before the audit and deterministic, so an answer always gets the same label. Each category keeps everything its label depends on, such as the cited evidence. Unreadable outputs follow a fixed rule and stay in the stream.
Mapping example from a real task (Table 23).
| Response | Evidence-based evaluator E0 | Answer-only evaluator E1 |
|---|---|---|
| Gold entity whose support passage is retrieved at rank 1 | PASS | PASS |
| Gold entity whose support passage is at rank 37, outside the retained top 10 | FAIL | PASS |
| Correct verdict with an incomplete evidence set | FAIL | PASS |
So under E0 a FAIL can mean missing support, not a factual error.
Sensitivity. The error guarantee holds for whichever fixed evaluator is used. The conclusions change with it.
- With
E0replaced byE1on the same 240 blocks, 76 reports change (31.7%), all toward favorable (Table 24). - With two frozen LLM judges, 33–50% of reports point in the opposite direction (App. H).
- Another MMLU-Pro parser changes 75 of 200 reports, 21 of them reversed (App. B.5.2).
We do not claim robustness to evaluator errors. A different evaluator asks a different question. Measuring an evaluator's error would need independent reference labels, which we do not have.
Response to Reviewer rCJs (1/4)
Answer for W1 and Q3 (exposition). Readability
Thank you for the concrete suggestions. We rewrote the main text in plain language and added what you asked for.
Label,Update,Decide), applied to the example before the formulas (Section 3).Theorem statements, proofs and Figures 1–3 are unchanged. Blue marks the main new content, and reworded sentences are black.