- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
I am unable to summarize the paper as I have found it impossible to understand.
Critical strengths
None
Critical weaknesses
The paper reads like an LLM slop. It is impossible to understand what is going on, what is the problem the authors want to solve and more importantly WHY they want to solve it. I can imagine there is some kind of contribution there but at the current state it is not-readable and should be rejected.
Questions for authors
none
2 comments
Response to Reviewer vXbD (2/2): Contributions and revisions
Second difficulty: the failure rates must be estimated. The probability that a given retained state leads to a wrong final answer is generally unknown. It can only be estimated by restoring the saved reasoning state, letting the model continue, and checking the result. Each such continuation, including PRM scoring, took about 10 GPU-seconds on A100 GPUs in our runs, so the number of reruns is the natural measure of cost. The states that determine the worst case are typically rare and lie deep in the search. Each carries little probability, but there are many of them, and together they can change the risk.
Contributions.
- For a fixed selection rule, we characterize the exact range of failure rates consistent with the released routing statistics. Every value in this range is attained by a single valid router, not by combining incompatible ones.
- We determine how many paid reruns are needed to estimate the upper end of this range to accuracy ε. The answer depends on the balance between how many candidates are retained and how much routing information is released. Under pairwise statistics, retaining one candidate requires order 1/ε² reruns, retaining two adds logarithmic factors, and retaining three requires order 1/ε³ once the candidate pool is large.
- The main technical step, also noted by Reviewers D5j6 and i7bA, is to show that a single router can make rare states at many depths of the search important simultaneously. This is why the lower bound applies to an actual router rather than to a mixture of hypothetical ones. For pairwise and three-wise statistics, this router is a simple linear hash.
- We identify when the cost decreases: if the losses are close to the component already determined by the routing statistics, far fewer reruns are required.
Relevance to the ICLR community. Verifier-guided selection is now a standard way to allocate test-time compute in reasoning. Reliability figures for such systems are usually computed by parties with access to the whole pipeline. Our results show that the routing statistics actually released may not determine the risk, and that the number of costly reruns needed to resolve it can be computed in advance.
Revisions.
- The abstract and introduction now begin with this LLM setting and the 25%-versus-0% example, before any notation is introduced.
- Related work opens with a paragraph on verifiers and PRMs in LLM reasoning.
- The introduction ends with the four contributions above.
- To stay within nine pages, two technical passages in Sections 4 and 5 are now given as short summaries; their full statements already appear in Appendices D.12 and E.1.
We would be grateful to know whether the revised introduction and Section 2 make the problem clear, and which parts remain difficult to follow.
Response to Reviewer vXbD (1/2): The problem we study and its relevance to LLM reasoning
We thank the reviewer for the candid assessment. We agree that the submitted version did not state clearly what problem we address or why it matters, and we take responsibility for this. Below we restate the problem through a minimal example; the revised introduction now follows the same structure.
Setting. Many LLM reasoning systems do not return the model's first answer. They sample several candidate solutions or reasoning steps, score them with a verifier (typically a reward model, often a process reward model, PRM), and keep only the highest-scoring few. Best-of-N with a trained verifier (Cobbe et al., 2021), PRM-based step selection (Lightman et al., ICLR 2024) and PRM-guided beam search for test-time compute (Snell et al., ICLR 2025) all follow this pattern. Our experiments use this pipeline: Qwen2.5-Math-1.5B-Instruct generates the steps, and Qwen2.5-Math-PRM-7B or RLHFlow Llama3.1-8B PRM scores them.
Deployed systems often have more than one verifier available, for example a cheap and an expensive one, or two versions under an A/B test. A rule, which we call the router, decides which verifier scores each candidate. Because the verifiers assign different scores, the router affects which candidates survive and, in turn, whether the final answer is correct.
Question. Consider an evaluator outside the system, such as an internal safety team, a third-party evaluator or a regulator. Such an evaluator usually receives summary statistics of the routing rather than the full log, for example: "each candidate is sent to verifier A half of the time, and the choices for any two candidates are independent." We ask two questions. What is the largest failure rate consistent with this information? And if the evaluator can pay to rerun the model, how many reruns are needed to estimate it reliably?
A minimal example (Section 2). Three candidates compete for two slots. For each candidate, one verifier assigns a higher score and the other a lower one. Consider two routers:
Under both routers, every candidate is sent to each verifier exactly half of the time, and any two assignments are independent. The released statistics therefore cannot distinguish them; this is the standard example of three bits that are pairwise but not jointly independent. The routers differ, however, in how often all three candidates receive their low score: one quarter of the time under Router 1, and never under Router 2. If that configuration is the one that leads the search to a wrong answer, the failure rate is 25% under Router 1 and 0% under Router 2. The released information supports only the interval [0%, 25%]; pairwise statistics do not determine the risk.
The same effect appears with real verifiers. On 512 MATH/GSM8K questions with two PRMs and three routing decisions per question, the gap between the lowest and highest compatible risk averages 5.3 percentage points per question when only per-candidate routing rates are released, and 1.5 points when pairwise statistics are added (Table 1). Under pairwise statistics the gap is zero on 482 of the 512 questions and is concentrated on the remaining 30, where the routing changes the selected answer.
(continued in 2/2)