acceptodds
Under review as a conference paper at ICLR 2027

Can More Samples Fix the Vote? Anytime-Valid Audits of LLM Self-Consistency

Abstract

Self-consistency selects the most frequent answer among repeated language model responses without a verifier. More samples stabilize voting but leave the response distribution unchanged, so the winner need not be acceptable. We propose Response Evaluation with Sequential Outcome-Level Verification via E-processes (RESOLVE), which audits whether population plurality favors PASS or FAIL under a fixed evaluator, leaving generation and selection unchanged. For a fixed policy, deterministic evaluator and i.i.d. responses, false reports remain controlled during continuous inspection. We bound response cost for an uncapped idealization and show when identifying a label can require less evidence than identifying an exact answer. Experiments distinguish answer availability, evaluator acceptance and plurality selection. An unfavorable diagnosis can motivate policy revision, which requires a fresh audit.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
large language models, self-consistency, test-time scaling, LLM evaluation, anytime-valid inference, sequential inference, uncertainty quantification, adaptive sampling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rCJsOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RESOLVE, an anytime-valid procedure for self-consistency in LLMs that audits whether the most probable individual answer category is labeled PASS or FAIL by an external verifier. The novelty of the proposal is that it does not estimate the total probability of PASS answers, returns the mode, or improve the model’s answers like prior works. Its central distinction is that PASS responses can become increasingly likely to appear in a sample while a FAIL answer remains the plurality winner. RESOLVE uses a fixed evaluator, discovers answer categories online, and reports PASS-favoring, FAIL-favoring, or unresolved. The proposed methodology is backed up by a thorough theoretical analysis based on e-processes and is evaluated across several LLMs and benchmarks.

Critical strengths
  1. The paper studies the theoretical guarantees for identifying the label of the population model which is different from prior work on self-consistency that mainly study it as a mode estimation problem.
  2. The proposed theoretical analysis controls false directional reports under continuous inspection, while allowing answer categories to be discovered online. The paper also analyzes how separating the label from exact answer identity can reduce evidence requirements.
  3. Experiments include three models on the evidence tasks and probe several relevant dimensions: evaluator acceptance versus plurality, repeated-testing error, audit difficulty, and same-label competitors.
Critical weaknesses
  1. The paper is difficult to parse. The distinction among PASS availability, total PASS mass, plurality label, exact mode, and “report” is central, but the dense notation and layered comparisons make it easy to lose track of the actual target. A running example and a more direct algorithm overview would help.
  2. RESOLVE depends on a fixed, deterministic, task-specific evaluator and category map. That need not mean an extra LLM judge; an answer key or annotation rule can suffice. But the method does not address evaluator errors or ambiguous criteria, and its usefulness is limited where reliable evaluation is unavailable. I think it is preferable to frame this as a key assumption and scope limitation.
  3. The formal guarantee is per query under fixed-policy, i.i.d. sampling assumptions. The paper notes that error levels can be allocated across a collection, but it does not provide a dataset-level guarantee by default. The guarantees also concern false reports, not how often the method resolves, how quickly it resolves, or the correctness of reports conditional on returning.
  4. The empirical support is rather limited.
Questions for authors
  1. Can the paper state its core novelty more explicitly relative to CITE? In particular, which contribution is new: the label-level target, online discovery, the evidence construction, or their combination? Could the authors provide a full-method comparison where feasible, in addition to the Fixed-label/Fixed-mode mechanism check?

  2. What is required of the evaluator in practice? How sensitive are the guarantee and conclusions to evaluator mistakes, ambiguous labels, or changes in the evaluator? Could the authors include an example showing how a real task's outputs are mapped to categories and labels?

  3. Can the authors clarify how to extend the per-query error guarantee to a collection of queries and distinguish the guarantee for the capped monitor from the idealized response-cost bound? Could they also revise the exposition and figures/tables to make the target, algorithm, assumptions, and empirical evidence easier to follow?

7 comments
AuthorsOpenReview ↗

Response to Reviewer rCJs (1/4)

Answer for W1 and Q3 (exposition). Readability

Thank you for the concrete suggestions. We rewrote the main text in plain language and added what you asked for.

  • A running example. One example is introduced in Section 1 and followed through Sections 2–4, with a real-data counterpart. For one FEVEROUS claim, all 160 responses give the correct verdict and 93 pass, but a failing answer leads the best passing one by 67 to 52, so the vote fails (Section 2.2).
  • The quantities that are easy to confuse. Table 1 separates PASS availability, PASS probability, the plurality label, the mode and the RESOLVE report.
  • A direct algorithm overview. Three steps (Label, Update, Decide), applied to the example before the formulas (Section 3).
  • Experiments. Section 5 opens by tying each experiment to the result it tests, and each subsection ends with its conclusion.

Theorem statements, proofs and Figures 1–3 are unchanged. Blue marks the main new content, and reworded sentences are black.

AuthorsOpenReview ↗

Response to Reviewer rCJs (2/4)

Answer for Q1 and W4. Novelty relative to CITE, full-method comparisons and new experiments

Which part is new. The combination of the label-level target with a test that stays valid under online discovery. The evidence construction is not new. The betting score is CITE's, and Section 1 now says so. CITE certifies that a supplied answer is the unique mode. Chen et al. (2024) and CALVER state when voting converges to a passing answer but do not test this for a given question. RESOLVE tests the label of the most likely answer, with no candidate and no unique winner.

Full-method comparisons. Mean responses until a report, counting the full budget when there is none.

ComparisonRESOLVEOther method
Full CITE, synthetic, true mode supplied (Table 12(b)), closest rival has the other label / the same label243.9 / 236.4365.9 / 1,519.0
Full CITE, synthetic S01–S03 with unknown answers (App. E.1)277.7 / 268.5 / 220.4, always reports351.5 / 865.3 / 1,022.1, unresolved in 0.05% / 59.6% / 99.6% of runs
Full CITE, real GPQA / MMLU-Pro (App. B.4)34.54 / 23.0543.97 / 27.61, plus the 64 reference responses it needs to choose its target
CS audits of the label, categories known in advance (Table 14), GPQA / MMLU-Pro34.54 / 23.05best CS 36.47 / 21.29
CS audits of the label, answers learned online (new, Section 5.4)cheaper by 258 to 319 responses on spread-out synthetic streamscheaper by 6.04 [5.38, 6.69] responses on 240 real evidence blocks

The comparison with CS audits is mixed. CS is cheaper when one answer dominates, and RESOLVE when responses are spread over many answers (Figure 4(b), 4(c)). The gain over CITE comes from the label target, not from RESOLVE's particular construction.

New experiments in the revision (W4). A new cohort F1110 (518 questions, 804 response blocks) generated after the reviews, CS audits replayed on all 698 archived blocks, ten synthetic settings with unknown answers (S01–S10), an evaluator replay on 240 blocks, two LLM judges on 5,444 answers and two negative tests of practical use (App. E–H).

AuthorsOpenReview ↗

Response to Reviewer rCJs (3/4)

Answer for W2 and Q2. The evaluator

We agree, and we now state this as a key assumption and a limit of scope (Sections 2.1 and 6). All results are relative to the evaluator hh and do not correct its errors.

Requirement (Section 2.1). The evaluator is fixed before the audit and deterministic, so an answer always gets the same label. Each category keeps everything its label depends on, such as the cited evidence. Unreadable outputs follow a fixed rule and stay in the stream.

Mapping example from a real task (Table 23).

ResponseEvidence-based evaluator E0Answer-only evaluator E1
Gold entity whose support passage is retrieved at rank 1PASSPASS
Gold entity whose support passage is at rank 37, outside the retained top 10FAILPASS
Correct verdict with an incomplete evidence setFAILPASS

So under E0 a FAIL can mean missing support, not a factual error.

Sensitivity. The error guarantee holds for whichever fixed evaluator is used. The conclusions change with it.

  • With E0 replaced by E1 on the same 240 blocks, 76 reports change (31.7%), all toward favorable (Table 24).
  • With two frozen LLM judges, 33–50% of reports point in the opposite direction (App. H).
  • Another MMLU-Pro parser changes 75 of 200 reports, 21 of them reversed (App. B.5.2).

We do not claim robustness to evaluator errors. A different evaluator asks a different question. Measuring an evaluator's error would need independent reference labels, which we do not have.

Reviewer_1ExqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces RESOLVE, which is a method that audits LLM majority voting to check that continued voting will converge to an answer a fixed evaluator accepts as a PASS answer. This is done by checking whether the probability of the most likely PASS answer will be larger than the probability of the most likely FAIL answer. RESOLVE does this by discovering answers as they appear and betting via e-processes that each new one beats the strongest opposite-label answer.

Critical strengths
  1. The paper does a good job of motivating the problem by showing that what matters for self consistency is whether the most frequent answer is acceptable to the fixed evaluator, and that it holds up in real-world settings.
  2. The paper has a strong anytime-valid guarantee theorem, which is backed up by experiments. This is beneficial because it allows for the guarantees to hold for any number of LLM forward passes, which allows for users to maintain guarantees regardless of their desired maj@k value.
Critical weaknesses
  1. There is limited novelty compared to the prior work. Most of the structure and machinery of the proofs come from the CITE paper (Ota et al. 2026). For single-answer tasks, the target itself is essentially the Chen et al. 2024 papers distinction between queries where the correct answer beats every incorrect one and queries where it does not.
  2. Its not clear what the practical utlity of this method is. Because RESOLVE requires a deterministic evaluator built on reference answers or annotations, which rules out LLM judges, and in any setting where the evaluator is available, it could simply be used to select a passing answer. So the method is closer to being an offline diagnostic than a method used at inference time. I acknowledge that the paper concedes this, but there has been no experiment done in such a way that shows that this method can meaningfully repair the LLM's output. The experiment in Table 17 shows that 24 unfavorable responses remained unfavorable after the method was applied.
Questions for authors

Can you show a concrete situation where RESOLVE’s diagnosis changes a practical decision, such as whether to use voting or a verifier, and leads to a better outcome? Corollary 4.4 shows RESOLVE is faster than CITE. But doesn't this compare RESOLVE's upper bound with CITE's lower bound?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 1Exq (1/4)

Answer for Q1 (second part). Corollary 4.4 compares an upper bound with a lower bound

Thank you for the careful reading. Yes, and this is intended.

Let pp be the mass of the winner, qq the largest mass with the other label, and ss the largest mass of any other answer.

  • RESOLVE (upper bound). Its cost rate is at most 1/I(p,q)1/\mathcal{I}(p,q).
  • Any certifier of the exact answer, CITE included (lower bound, Cor. C.11). Its cost rate is at least 1/I(p,s)1/\mathcal{I}(p,s).

When a same-label rival is closer than every opposite-label rival (p>s>qp > s > q), we have I(p,q)>I(p,s)\mathcal{I}(p,q) > \mathcal{I}(p,s). RESOLVE's guaranteed rate is then strictly better than the best rate any certifier of the exact answer can reach. An upper bound for one target below a lower bound for the other is what separates the two targets, rather than two implementations.

Limits. The statement is asymptotic (α→0\alpha \to 0), it gives no gap when q=sq = s, and it is not a finite-budget claim. A paragraph after Corollary 4.4 now says this.

Finite-budget check (Table 2, α=0.05\alpha = 0.05, 5,000 paired streams). The two audits differ only in the rivals they bet against.

Closest rival of the winnerLabel auditExact-answer auditRatio
has the other label (q=sq = s)188.9244.61.3
has the same label (q<sq < s)187.4946.85.1

With the full procedures (Table 12(b)), RESOLVE costs 243.9 and 236.4 responses in the two cases, and CITE with the true mode supplied costs 365.9 and 1,519.0.

AuthorsOpenReview ↗

Response to Reviewer 1Exq (2/4)

Answer for W1. Novelty relative to CITE and Chen et al. (2024)

We agree on the facts, and Section 1 now states them. The proof tools and the betting score are CITE's. The condition "the most likely answer passes" is that of Chen et al. (2024), extended to several passing answers by CALVER (Thm. 7).

Those works state the condition. They do not test which side of it a given question and pipeline is on. That test is our contribution. It has three parts.

  1. Target. The label of the most likely answer, that is, the sign of Γ=m1−m0\Gamma = m_1 - m_0, the gap between the largest passing mass and the largest failing mass. It needs no candidate answer and no unique winner. For a real FEVEROUS claim, 93 of 160 responses pass, yet a failing answer leads the best passing one by 67 to 52, so the vote fails (Section 2.2).
  2. Validity under discovery. The answer set is unknown and may be infinite, and the error bound holds while answers are discovered and under stopping at any time (Thm. 4.1). At an exact tie inspected after every response, RESOLVE reports falsely in 1.82% of runs and a repeated fixed-sample test in 34.0% (Section 5.2).
  3. Cost. Rivals with the winner's label cost nothing for the label. When the closest rival shares the label, the exact answer needs 28.2 [20.0, 37.0] more responses on archived queries (Table 13) and about 53 more on F1110 (Figure 4(a)).

We do not claim a new betting technique.

AuthorsOpenReview ↗

Response to Reviewer 1Exq (3/4)

Answer for W2 and Q1 (first part). Where a diagnosis changes a practical decision

We agree that with a verifier at inference time, one should use the verifier. We tested whether a diagnosis could still save cost. It could not (App. E.3).

  • Routing each query by its diagnosis did not beat always using verifier search (−0.4 pp PASS [−1.3, 0.0]) and used 53.4 more responses per query.
  • Stopping verifier search after an unfavorable report lost a PASS on 15 of 318 queries (−4.73 pp).

RESOLVE does not speed up inference. Sections 1 and 5.5 now say this.

Where a diagnosis does change a decision. Labels exist only offline, on annotated development data, and the deployed system votes without a verifier.

(a) A report predicts the later vote (Section 5.5, Table 3(a)). On an independent follow-up block of 64 responses, voting passed in 97 of 98 blocks with a favorable report and in 0 of 135 blocks with an unfavorable report. In 26 of those 135 blocks a passing response appeared, and voting still failed.

(b) Choice between two pipelines (Table 3(b)). 274 annotated F1110 claims, with one 0.05 error budget over both pipelines and all claims. Audit rows give a 95% range for the share of claims whose long-run vote passes.

RuleQwen2.5-7Bgpt-oss-20bResponses used
RESOLVE[23.7%, 26.6%][49.6%, 70.4%]36,322
Dynamic CS[22.6%, 27.7%][47.1%, 75.2%]33,607
64-response vote, no error bound68/274 pass160/274 pass35,072

The ranges do not overlap, so the rule chooses gpt-oss-20b. In separate validation responses its vote passes on 160 of the 274 claims and that of Qwen2.5-7B on 69.

Limits. A plain 64-response vote makes the same choice at similar cost. The audit adds the error bound and the option to withhold a choice when the ranges overlap. It does not add accuracy here, and the analysis is retrospective.

Reviewer_5GTSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether the most probable response category passes a fixed evaluator. The proposed method RESOLVE discovers categories as sampling proceeds and accumulates evidence against competitors with the opposite label. It reports a favorable, unfavorable, or unresolved outcome while controlling the probability of false reports under continuous monitoring.

The paper establishes validity for the monitor and an asymptotic response-cost bound for an uncapped idealization. It also compares certifying the winner’s label with identifying the exact winning answer. The experiments cover controlled distributions, prospective audits, and fresh auditing after a prompt revision.

Critical strengths
  1. The theoretical construction directly addresses the label hypothesis. Thm 4.1 accommodates countably many categories discovered during sampling without reusing discovery observations as evidence. Thm 4.3 accounts for discovery time and rank penalties, while Corollary 4.4 establishes a strict asymptotic separation when the closest rival shares the winner’s label.

  2. The matched comparisons isolate a meaningful source of the cost difference. Fixed-label and Fixed-mode use the same candidate, observations, bets, and thresholds. Only the opponent set changes. Table 1 shows that identity certification costs approximately five times as much in the setting where the closer rival shares the winner’s label. Table 12 finds a corresponding concentration of additional cost in the archived queries. These comparisons provide direct evidence for the label-versus-identity distinction, while keeping that evidence separate from comparisons between the complete methods.

Critical weaknesses
  1. Comparative evidence for online discovery in richer answer spaces remains limited. Table 2 evaluates option tasks with only 5 or 11 known categories. In the primary evidence cohorts, only 2 QAMPARI queries and 4 FEVEROUS claims produce multiple PASS categories (Appendix B.1). The evidence-task CS comparison instead gives the comparator access to future realized support and includes only 6 mixed-label blocks per task (Table 15). Thus, the experiments provide stronger evidence for the benefit of changing the certification target than for RESOLVE’s particular construction in richer, unknown answer spaces. This limits the practical significance demonstrated by the experiments without invalidating the theory.

  2. Important empirical findings are difficult to reproduce from the supplied materials. Appendices A-B describe the protocol, but I could not locate the exact frozen prompts, parser and evaluator implementation, query manifests, seeds, or an artifact link. The reproducibility statement says these records have been retained. Access to these materials matters because the results are sensitive to implementation choices. Changing the archived MMLU-Pro parser changes 75 of 200 diagnoses, including 21 directional reversals (Appendix B.5.2). Given that sensitivity, executable specifications are important for checking the reported findings.

Questions for authors
  1. Could the authors provide an accessible artifact containing the frozen extraction and evaluation rules, prompts, cohort identifiers, configurations, and any response-level or derived records that can be shared?

  2. Can the comparison with label-targeted confidence-sequence methods be extended to the unknown-support, multiple-PASS-category setting? Table 2 supplies a known category set, whereas online discovery is an important part of RESOLVE’s intended setting. A comparison using the same response streams and error budget, with explicit treatment of unseen categories, would clarify whether RESOLVE offers an efficiency advantage over alternative methods answering the same label question.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5GTS (1/2)

Answer for W1 and Q2. Label audits when answers are not known in advance

Thank you for this request. We ran the comparison you asked for. The result is mixed, and we report it that way.

What we ran. We built two CS audits of the same label that learn answers online. Dynamic CS keeps a confidence sequence for each answer from the time it is first seen, plus an upper bound on the mass of any answer not yet seen. RI-CS also intersects each interval over time. All methods read the same ordered responses with the same evaluator at level α=0.05\alpha = 0.05 per query, and none is given future answers (Section 5.4, App. E.1).

Results. Δ\Delta is CS cost minus RESOLVE cost in responses, with 95% intervals. Δ>0\Delta > 0 means RESOLVE stops sooner.

DataRESOLVEDynamic CSΔ\Delta
Synthetic S01, second PASS answer of mass 0.16277.7585.3+307.6 [301.0, 314.2]
Synthetic S05, 64 PASS answers264.0578.8+314.8 [308.2, 321.5]
240 real evidence blocks23.817.8−6.04 [−6.69, −5.38]
F1110, gpt-oss-20b (≥ 2 PASS answers in 209/274 blocks)55.758.3+2.54 [0.19, 5.04]
F1110, Qwen2.5-7B (≥ 2 PASS answers in 22/274 blocks)26.320.9−5.38 [−6.58, −4.07]
  • Synthetic streams with many unknown answers (Table 20). Over the seven settings S01–S07, Δ\Delta is +296 to +319 for Dynamic CS and +258 to +288 for RI-CS. No method made a wrong report.
  • Real archived blocks (App. E.2). The CS audits are cheaper by about 6 responses, because real responses are concentrated. Of the 698 archived blocks, 342 contain a single answer.
  • New multi-PASS cohort F1110 (Table 25). After the reviews we generated 274 FEVEROUS claims whose annotations allow several complete evidence sets. With many passing answers RESOLVE is not more expensive. With few, the CS audits are 5 to 6 responses cheaper.

What decides the outcome is how concentrated the responses are. On real blocks RESOLVE is cheaper when the top answer's share is below 0.5, and CS is cheaper above 0.7 (Figure 4(b), 4(c)).

Sections 1 and 5.4 now state that neither label audit is always cheaper, and that their common gain over CITE comes from the label target. This agrees with your point in W1.

Edited on 16 Nov 2026 to fill in the RESOLVE and Dynamic CS costs on the row for the 240 real evidence blocks, which were blank. Nothing else changed.

AuthorsOpenReview ↗

Response to Reviewer 5GTS (2/2)

Answer for W2 and Q1. Code and data

Code and data are now in an anonymous GitHub repository, which is also linked in Section 5 and in the Reproducibility Statement. It contains the following.

  • Extraction and evaluation rules. The frozen FEVEROUS and QAMPARI parsers and evaluators.
  • Prompts and configurations. Prompt builders, decoding settings, seeds, pinned model revisions and experiment settings.
  • Response-level records. The ordered category streams with their labels.
  • Code. The RESOLVE monitor and the two CS audits, Dynamic CS and RI-CS.

How to use it. To check a reported audit, run the same code on the provided streams with the frozen settings and compare the reports and costs. To audit a new pipeline, generate responses with the provided prompts and settings, map them to categories with the frozen parser and evaluator, and run the same audit.

Raw responses and retrieved contexts are not included, because of the source licenses, so the provided records start from parsed categories.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer,

We have carefully addressed your concerns and incorporated the corresponding revisions into our manuscript. As the discussion period is approaching its end, we would be happy to provide any further clarifications if needed.

Best, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

We thank the three reviewers (5GTS, 1Exq, rCJs) for their careful and constructive feedback. This global response summarizes the strengths noted in the reviews, the main concerns, and how the revision and the new experiments address them. Details are in the individual responses.

Strengths highlighted by reviewers

  • [5GTS, rCJs] The paper studies the label of the most likely answer, a target different from exact-mode estimation, and the theory addresses it directly.
  • [5GTS, 1Exq, rCJs] The anytime-valid guarantee (Theorem 4.1) holds under continuous inspection while answers are discovered online, and experiments back it up.
  • [5GTS, rCJs] The matched comparison isolates why the label needs less evidence than the exact answer when a same-label rival is close (Corollary 4.4).
  • [1Exq] The problem is well motivated, and it holds up in real data.
  • [rCJs] The experiments cover three models and probe several relevant dimensions.

Concerns and how we addressed them

  1. Label audits when answers are not known in advance ([5GTS, rCJs]). We built two confidence-sequence (CS) audits that learn answers online and ran them on the same responses at the same level. Neither audit is always cheaper. RESOLVE needs 258 to 319 fewer responses on spread-out synthetic streams. The CS audits need about 6 fewer on 240 real evidence blocks, where one answer usually dominates (Section 5.4, Figure 4).
  2. Novelty relative to CITE and Chen et al. (2024) ([1Exq, rCJs]). The betting score is CITE's and the convergence condition is that of Chen et al. Section 1 now says both. Our contribution is a valid test of which side of that condition a given question is on, with no candidate answer and no unique winner.
  3. Practical use ([1Exq]). RESOLVE is a diagnostic to run before deployment. It does not speed up inference, and we report two attempts that failed (App. E.3). What it supports is a choice made in advance. A later vote passed in 97 of 98 blocks reported favorable and in 0 of 135 reported unfavorable (Section 5.5, Table 3).
  4. Evaluator errors and LLM judges ([1Exq, rCJs]). Reports are relative to a fixed evaluator and do not correct its errors. We now state this as a key assumption (Section 2.1). An answer-only evaluator changes 76 of 240 reports, and two frozen LLM judges reverse 33–50% of reports (App. E.4, H).
  5. What the guarantee covers ([rCJs]). Splitting the error budget over NN audits gives a guarantee for a whole benchmark (Section 4.1). We also state what the guarantee does not cover.
  6. Readability ([rCJs]). We rewrote the main text in plain language and added one example followed through the paper, Table 1 and a three-step overview of the algorithm.
  7. Reproducibility ([5GTS]). Code and data are in an anonymous GitHub repository, also linked in Section 5.

Revised manuscript

  • Text in blue marks the main new content. Reworded sentences are black and do not change the content.
  • The problem, the method, every theorem statement, every proof and Figures 1–3 are the same as in the submission. The main text is still 10 pages.
  • New in the revision are Section 5.5, Figure 4, Tables 1 and 3, Appendices E–H and cohort F1110 (518 questions, 804 response blocks), generated after the reviews.
  • Old Table 2 is now Table 14. Old Tables 15 and 17 are now Tables 17 and 19. The data of old Figure 4 are in Figure 10(b).
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your thoughtful reviews. We would like to follow up on our revised manuscript and individual replies. Section 5.4 compares RESOLVE with two confidence-sequence audits of the same target when answer categories are discovered online. Sections 2.1 and 4.1 clarify the role of the fixed evaluator and the scope of the error guarantee. Section 5.5 explains what the retrospective policy-selection example shows and where its practical conclusions are limited.

As the discussion period draws to a close, we would appreciate hearing whether any concerns remain or whether a specific point would benefit from further clarification. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.