- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a controller that stops heterogeneous LLM ensemble early while preserving its fixed-rule decision. It maps token prefix of running call to a nested split-conformal set of possible outputs, stops once all compatible combinations give the same decision and cancels calls that can no longer change it. The authors provide a statistical guarantee of decision preservation and report substantial token savings with nearly perfect fidelity on multiple choice benchmarks.
Critical strengths
- The formulation provides a simple stopping and cancellation criterion for any fixed deterministic aggregation rule and separates exact decision logic from conformal coverage.
- The evaluation is detailed and transparent, covering complete test splits and itemizing every reference disagreement.
Critical weaknesses
- The evaluation is limited to multiple-choice tasks with generation capped at 128 tokens. It is unclear whether the gains carry over to long-reasoning models where the answer appears late.
- The method is not compared end-to-end with existing early-stopping methods so it is unclear whether it offers any practical advantage over these alternatives.
Questions for authors
- Could the authors evaluate the method on long-reasoning models where the answer appears late?
- Please clarify what finite decision representation the approach would use for open-ended generation.
- Could the authors provide end-to-end comparison with existing early-stopping methods, such as MARS or Parallel-Probe, under a matched generation budget?
2 comments
Reviewer response
Thank you for detailed experiments and thorough response that address most of the concerns. I will keep my original score.
This is a short update on your three questions. The revised paper is uploaded. Since our first response we measured wall time directly and compared the stopping rules on identical observations. Changes made for your review are marked in green.
Long reasoning with late answers (Weakness 1, Question 1)
We raised the cap from 128 to 4,096 tokens and used 128 fresh GSM8K questions. Answers do come late: the first answer that can be parsed from a prefix appears at a median of 1,280 tokens, and 92 of 640 calls reach the cap. Five-sample self-consistency still reaches its decision 28.2% sooner in wall time (95% interval 23.4 to 33.1), and none of the 128 decisions changes. The token saving is smaller, 13.2%, because the decision forms near the end of the reasoning. The time saving is larger because the complete panel must wait for its slowest call and the stopped panel does not. On BFCL tool calls with three models the time saving is 31.9%, with 0 of 104 decisions changed. Both are measured with one query at a time on one GPU, running the complete and the stopped execution of each question back to back (Table 3, Appendix S).
Representation for open-ended outputs (Question 2)
A fixed parser maps each response to an action: a number, a complete tool call with its arguments, or abstention. Over an unbounded vocabulary, the outputs still possible for a call form a finite set or the complement of one, and the stopping test stays exact without listing strings (Appendix P). For free-form dialogue the application must first declare the action. Agreement of surface text is not an action.
MARS and Parallel-Probe under a matched budget (Weakness 2, Question 3)
All methods use the same prompts, seeds, five calls and 4,096-token caps. To compare the stopping rules alone, we gave every rule the same observations: the answers visible in the same prefixes at the same reviews of the same 128 questions. The complete panel answers 113 correctly.
Our rule is the only one that changes no decision, and it saves more than Parallel-Probe at its released settings. The rules that save more stop on the current leader before the unfinished calls have shown an answer, and they change two or three decisions.
Run live with their own forced probes, Parallel-Probe saves 36.4% of tokens and changes 7 of 128 decisions. MARS saves 68.6% and changes 6, but its 16 warm-up traces per question raise its total token cost to about six times that of the complete panel when only five calls are used. On the Qwen3-8B traces released by the Parallel-Probe authors for AIME24, AIME25 and HMMT25, their policies save 20% to 29% and change 8% to 19% of decisions. Our exact rule changes none and saves 6.0% to 12.4% (Appendix T).
In practice the three rules differ in what they promise. Parallel-Probe targets accuracy and cost and does not bound how many answers change. The guarantee of MARS needs warm-up traces for every question. Our rule kept every answer in these tests and still cut latency by about a quarter, with no probes and no per-question preparation.
Do these results answer your three questions? If so, we would be grateful if you would update your score. If something is still open, please tell us what.