- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes Verify Until Decided (VUD), an executor that reduces computation while preserving the result of a specified finite weighted vote. The authors introduce "ScoreCert", which stops acquiring verifier scores once bounded score intervals establish an invariant winner; and "FutureCert" which extends this reasoning to candidates not yet generated. The certificates account for tie priority and signed scores, and a deferred controller combines generation stopping with terminal score acquisition. Experiments report identical decisions on 2,304 fixed candidate pools, with 52.4% fewer ThinkPRM calls beyond a shared zero-call gate. Additional experiments measure verifier-only and generation-plus-scoring wall time. The guarantee preserves the dense selector’s decision under common score realizations and numerical semantics; it does not guarantee answer correctness.
Critical strengths
- The preserved target and stopping guarantee are precise. The fixed-pool certificate is necessary and sufficient under the declared interval information. The future-candidate extension handles unseen answer groups and negative scores, while explicitly distinguishing safe termination from optimal query selection. This provides a checkable execution contract rather than relying only on observed agreement.
Critical weaknesses
-
The presentation makes the core contribution unnecessarily difficult to understand. The introduction moves into VUD, ScoreCert, and FutureCert before giving a sufficiently concrete explanation of the problem and proposed solution. CISC is used without expanding the acronym. I had to reconstruct the basic algorithm from definitions, theorems, and scattered experimental descriptions. A worked example before the formalism would help distinguish the central idea from its implementation details and extensions.
-
The novelty and practical significance are not sufficiently established. The central principle - stop once the remaining unknown outcomes cannot change the winner - is intuitive and closely related to existing bound-based stopping methods. The paper does not clearly identify the nontrivial technical advance beyond applying this principle to LLM weighted voting. The experiments demonstrate savings against complete execution, but provide limited evidence about the value of the proposed query policy relative to simpler policies. Furthermore, execution details and cost scopes are scattered across the manuscript, making it difficult to determine when the reported savings would translate into a meaningful practical benefit.
Questions for authors
-
What is the principal technical novelty? Please explain, using a concrete example and plain langauge, what your method adds beyond maintaining bounds on partial totals and stopping when the winner is fixed. How does this differ technically from existing exact aggregation and partial-evaluation methods?
-
What does the proposed query policy contribute? With the same stopping certificate, how does the largest-upper-bound policy compare with simple fixed or random query orders in verifier calls and wall time?
-
How should a practitioner interpret the timing results? Please provide a consolidated description of the execution setup for each timing experiment: scorer interface, batching and concurrency, caching, included costs, and whether candidate generation is independently executed or shared between paths. Under which configurations does the method deliver substantial end-to-end savings?
-
What explains the gap between frozen-score and online CISC results? Table 2 reports 81.96% versus 42.44% call savings. Appendix G mentions fresh score acquisition, but were the score transformations, bounds, and batch-check schedules otherwise identical? Which result best represents the intended deployment setting?
-
What is the setup of the experiment? It is very hard to interpret from the paper about the actual experimental setup.
3 comments
Response — follow-up
Thank you for the follow-up. We have clarified the role of the signed extension and added matched execution experiments.
Q1
Support for negative weights is an extension of VUD's guarantee, not its principal contribution. Your analysis of the original deferred controller is correct: with mixed-sign bounds and no observed scores, it cannot stop generation early. In a nontrivial nonpositive interval, the new-group condition also fails with two or more slots remaining, even with observed scores. We have removed “some nonpositive-score regimes” and clarified the contribution in the Introduction.
The signed condition is useful when executing a rule that already uses negative contributions. It preserves that rule while accounting for a future candidate lowering the leader's total. Simply shifting every weight upward can change the winner, because larger answer groups receive a larger bonus.
We also checked this use case on the same 2,304 centered-CISC score vectors as Table 26, keeping the weights 2p−1, bounds [−1,1], and N=8. The added interleaved schedule queries scores once two answer groups are visible and a certificate could become attainable. Table 29 reports counts on these score vectors:
| Centered-CISC execution | Generated candidates | Scoring calls |
|---|---|---|
| Dense | 18,432 | 7,279 |
| Original deferred VUD | 18,432 | 4,192 |
| Interleaved VUD | 17,408 | 5,194 |
All 2,304 selected answers match dense. Interleaving saves 5.56% of candidates and 28.64% of scoring calls relative to dense. It uses more calls than deferred, in exchange for fewer generations. Seven of the early-stopped pools include an observed negative score. Thus the observed-score case occurs in the evaluated data; the zero-generation-saving result applies to the score-free controller. This comparison measures counts, not online time.
This does not change the comparison between voting rules in Table 26: exponential CISC at T=1 gives the same winners with fewer queries, so these data do not favor choosing centered CISC. The signed extension preserves an existing signed rule; it is not a reason to introduce negative weights.
The main contribution is the stopping characterization: ScoreCert handles unknown scores in a fixed pool, and FutureCert also handles candidates that may join existing groups or introduce unseen answers. Both retain the original tie rule and give necessary and sufficient conditions under the stated score bounds. The nonnegative-score experiments below test their practical value without relying on the signed extension.
Response — follow-up
Q2
GLM provides the requested case with the added interleaved schedule, at the original development-selected CISC temperature T=0.05. The original deferred schedule still saves no generation at this temperature. Scoring during generation lets FutureCert stop with the same temperature and voting rule.
Using Table 22's original split and grading convention, CISC at T=0.05 answers 153/192 development questions and 443/576 test questions correctly, compared with SC's 151/192 and 440/576. GLM's T=0.02 and T=0.05 tie on development accuracy; we retain T=0.05 from the original tie rule. For Qwen, SC and all ten CISC temperatures tie at 165/192; for Granite, SC scores 139/192 versus CISC's 138/192. Extending selection to include SC, with lower execution cost resolving accuracy ties, selects SC for Qwen and Granite and retains CISC T=0.05 for GLM. These accuracy differences belong to the voting rules. VUD's contribution is to preserve the selected rule's answers while reducing its execution cost.
For GLM/CISC, we check FutureCert after each candidate. Once two answer groups are visible, we query scores in max-upper order if they could establish the certificate; otherwise, we continue generating. Weights remain in [exp(−20), 1], and the budget remains N=8.
| GLM/CISC, T=0.05, N=8; 576 test questions | Generated candidates | Scoring calls |
|---|---|---|
| Dense | 4,608 | 2,051 |
| Original deferred VUD | 4,608 | 1,121 |
| Interleaved VUD | 4,417 | 1,540 |
All three return the same answer on every test question. Relative to dense, interleaving saves 4.14% of candidates and 24.91% of scoring calls. Dense already skips scoring single-answer pools. Waiting for two groups retains this optimization: every queried score is needed by dense. Deferred saves more calls here; interleaving also avoids 191 generations.
For example, after six candidates on one online GSM8K question, the leading answer's observed total is 2.996 and the rival's is 0.556. The two remaining candidates can raise the rival to at most 2.556, or give a new answer at most 2. Neither can overtake the leader, so both generations are skipped.
We ran the same controller online on the existing 48-question, six-task GLM roster. Both paths independently generated candidates and acquired scores. Total time fell from 53.90 to 52.74 seconds per question (2.14%), with all 48 answers unchanged. Appendix O, Tables 27–28 reports the full-test counts and paired timing.
The released 1.5B verifier gives a larger generation and total-time benefit, using its original probabilities and bounds [0,1]. Scoring with one visible answer group can establish a lead earlier, but may require scores that dense would skip. Across all 2,304 pools, candidates fall from 18,432 to 14,514 (21.26% fewer), while calls rise from 7,279 to 12,189. Every dense answer is preserved.
Table 30 compares both paths on the six-task roster, with 48 questions per generator:
| Generator, released 1.5B, N=8 | Candidates: dense / VUD | Calls: dense / VUD | Seconds/question: dense / VUD | Time saved |
|---|---|---|---|---|
| Qwen | 384/293 | 88/261 | 41.72/33.89 | 18.76% |
| Granite | 384/311 | 186/241 | 50.74/42.73 | 15.78% |
| GLM | 384/301 | 158/242 | 54.30/46.57 | 14.23% |
Both new online experiments use one paired pass, matched per-candidate seeds, singleton BF16 requests, and no prefix cache. Each dense/VUD pair runs independently on the same H800, with execution order alternated across questions. Time includes generation, parsing, all scoring used by the controller, and certification; loading, warm-up, and post-run checks are excluded. All 144 released-verifier pairs return identical answers, with matching common candidate prefixes and independently acquired scores. The earlier approximately 1.5% total-time saving concerned fixed-pool ScoreCert, which leaves generation unchanged. Each new timing comparison uses matched dense settings and includes generation.
For query order, the certificate supplies most of the savings; max-upper adds a further reduction. We fix the certificate, scores, and candidate order across all 2,304 pools. Further reduction is (input-order calls − max-upper calls) / input-order calls:
| Scorer | Input-order calls | Max-upper calls | Further reduction |
|---|---|---|---|
| ThinkPRM | 4,130 | 3,514 | 14.92% |
| CISC, T=1 | 1,825 | 1,313 | 28.05% |
| Released 1.5B | 4,445 | 3,974 | 10.60% |
For ThinkPRM, max-upper uses fewer calls on 384 of the 954 multi-answer pools, the same number on 547, and more on 23; each generator's total falls. This full-matrix result does not depend on critique length. In the released-verifier timing control, mean scoring time over the 128-pool roster falls from 4.497 to 4.186 seconds across five repeats (Table 19).
Two follow-up questions
Thank you for your thorough answer and the added tables. I appreciate your effort to polish the article and pull out extra information to clarify your novelty and technical challenges.
First, I acknowledge that the presentation has slightly improved. Adding worked examples in the Introduction, walking through Figure 2, and consolidating serving/batching setups in Appendix M (Table 23) make the paper clearer, though there is still room for writing improvement.
However, because the revised exposition and new ablations (Tables 4, 12, 19, 20, 22, 26) lay out the mechanics and numbers transparently, they also expose fundamental contradictions between the paper's theoretical selling points and empirical results. I am maintaining my score for now, and look forward to the authors' response to convince me to increase it:
Q1. Contradiction in the Signed-Weight Certificate (
FutureCertwith ℓ<0)In Question 1, you emphasize that a key contribution beyond standard interval bound-checking is Theorem 2's handling of negative weights during generation (e.g., totals A=0.8,B=0.1 with weights in [−1,0.2], where a future candidate joining A at −1 changes the winner).
Yet across all real scorers evaluated, the only negative-weight rule is centered CISC wi=2pi−1∈[−1,1] (Tables 2B, 26). For your deployed controller (Deferred VUD, Sec. 3.2, making no verifier calls before Nmax), whenever ℓ<0<u, the score-free bound is Lg=∣Cg∣ℓ<0 and c=min{0,ℓ,−u}<0. Thus, the new-group condition in Theorem 2 (Lg−u+(r−1)c≥0) is strictly negative for all r≥1 and never holds -- so score-free
FutureCertsaves 0% of generation whenever ℓ<0<u.In Sec. 3.2 (lines 248-253), you note Theorem 2 could still hold with observed scores (as in your A=0.8,B=0.1 example) or "in some nonpositive-score regimes." However:
Meanwhile, for fixed-pool
ScoreCert(Table 26), switching from exp(pi−1)∈[1/e,1] to 2pi−1∈[−1,1] yields the exact same dense winners on all 2,304 pools while cutting call savings nearly in half (81.96% to 42.41%).Given that (i) deployed score-free
FutureCertnever stops early when ℓ<0<u, (ii) the nonpositive regime (ℓ<u≤0) cannot stop early for r≥2 even with observed scores, and (iii) your only real-data signed rule (2pi−1) yields identical winners to exp(pi−1) on all 2,304 pools while halvingScoreCertsavings, in what realistic LLM test-time scaling scenario is the signed-weight formulation actually useful?Q2. Contradiction in Positive-Floor Stopping (Tables 12 & 22) and Query Order (Tables 4 & 19)
When weights are non-negative (ℓ≥0), certificates reduce to simple interval checks (Lg>Uh). For
ScoreCert(Table 4), naive Input Order captures 84.2% of call savings on ThinkPRM (7,279→4,130 vs. 3,514 for Max-upper) and 91.4% on CISC (7,279→1,825 vs. 1,313) across 2,304 pools (in Table 19, 1.5B verifier Input Order takes 4.497s vs. 4.186s, and the 48-pool ThinkPRM gap rests on just 11 non-gated questions -- 31 vs. 44 calls -- amplified by 8,172-token outliers). End-to-end savings with the 1.5B verifier are ~1.5% (Table 20).For
FutureCert, score-free stopping requires a high positive floor ℓ=exp(−1/T), tested via CISC. Comparing full accuracy (Table 12) with held-out T tuning (Table 22) reveals a clear dilemma:FutureCertsaves 0.00% of candidates and -0.17% of time.Across the entire paper, is there even a single generator where (a) CISC at its accuracy-selected temperature beats unweighted SC (or T=10), and (b)
FutureCertdelivers non-zero generation savings at that temperature?