- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how retaining paired error information affects the certification of verification rules built on fixed process reward models (PRMs). Each rule specifies how to encode the history and apply calibrated thresholds to PRM scores to judge the current reasoning step. The authors define finite-roster requirements on a rule's balanced error, disagreement between its decisions under two input encodings, and excess error relative to three canonical rules using the same PRM. They compare paired and marginal risk bounds while holding the models, candidate rules, samples, targets, and error budgets fixed. Experiments use two PRM checkpoints and a filtered ProcessBench population, including a previously unscored remainder of 235 problems. The authors report that pairing increases valid certificate returns from zero to approximately 24% in the primary comparison. Marginal-preserving shuffles largely remove this advantage, while forecasts retaining joint errors better predict subsequent certification outcomes. The paper concludes that joint errors help explain and forecast whether a verification rule can be certified under the specified requirements.
Critical strengths
- The paper addresses a worthwhile problem: how to select a good verification rule for a fixed PRM.
Critical weaknesses
-
The central contribution rests on an unvalidated relative-risk requirement.
Section 3.2 requires a candidate rule's balanced error rate to be no more than three percentage points above the best of CAN F, CAN E, and CAN L using the same fixed PRM, in addition to the absolute-error and disagreement constraints. The paper does not establish this relative requirement as an existing criterion for evaluating verification rules, justify why these three rules define a meaningful reference set, or demonstrate why satisfying this tolerance should determine whether a rule merits certification. The missing foundation concerns the purpose of the relative requirement itself, not simply the numerical choice of three percentage points.
The main experiments then measure how often statistical procedures can certify this author-defined requirement. In Table 4, pairing improves the relative comparisons while leaving absolute-risk passage counts unchanged. The coupling and forecasting experiments further explain or predict outcomes under the same chosen requirement; they do not validate its usefulness for assessing or selecting verification rules under a fixed PRM. Fixing the target in advance and varying its thresholds also do not supply that missing justification.
Paired comparison has a classical statistical basis, which Section 2 acknowledges. Consequently, showing that it produces more certificates for this chosen target does not by itself establish a contribution to evaluating verification rules. The paper first needs to establish why the relative criterion is worth certifying. Without that evidence, its central argument depends on an unsupported premise: that improved certification of this criterion constitutes a meaningful advance in evaluating or selecting verification rules.
-
The paper has serious writing problems that make it difficult to read and understand.
The manuscript does not yet meet the standard of clarity required for publication and needs substantial rewriting and reorganization. The introduction fails to articulate the main contributions. The exposition provides insufficient background, introduces key concepts before explaining them, and prioritizes procedural details over a clear account of the research. Here are a few examples:
-
Insufficient background and delayed definitions. Section 1 discusses shared errors between verification rules before explaining how a rule relates to a fixed PRM. The compressed definition in the third paragraph comes after two paragraphs that already rely on this distinction, leaving the reader without a clear account of what is being compared and why.
-
An unclear object of certification. The abstract says that multi-candidate testing "certifies one checkpoint," whereas Section 5.4 reports certification of the CAN E rule under M1. This shorthand obscures the central distinction between certifying a particular rule and certifying the PRM itself.
-
The audit components lack a clear organizing explanation. Section 3.3 introduces four bound constructions and four candidate-selection programs shortly after the Abs/Near targets. Their separate definitions do not provide a clear overview of how these components combine, so labels such as "Fixed-Near, ATOM-PAIR" require the reader to reconstruct which rule is tested, which conditions must pass, and how the evidence is calculated.
-
The roles of the result analyses need clearer explanation. Section 5.4 moves from repeated-sample component counts to the single prospective audit, candidate selection, threshold sensitivity, and historical certificate checks. Although these analyses are individually identified, their roles in the central argument are not brought together. The final reference to "every previously issued claim" further requires the reader to recover the relevant historical targets from other sections.
-
Questions for authors
-
Basis of the relative requirement. Please justify why a verification rule should be required to have a balanced error rate no more than three percentage points above the best of CAN F, CAN E, and CAN L using the same fixed PRM. Provide supporting prior work or evidence that this requirement is useful for evaluating or selecting verification rules.
-
Clarity and organization. Please revise the exposition to explain how verification rules relate to the PRM and how the audit components fit together. Please clarify what is certified and how the different result analyses support the paper's central argument.
-
Rule design and reference set. Please explain why CAN F, CAN E, and CAN L are constructed using these history encodings and canonicalization procedures, including the one-third and two-thirds split points, and provide supporting theory, empirical evidence, or prior work. Please clarify why these three rules form an appropriate reference set for evaluating other rules under the same PRM, and why the remaining four candidates are excluded.
-
Practical purpose and contribution. Please clarify what practical decision the combined absolute-error, disagreement, and relative-risk requirements are intended to support, and justify their joint use as certification criteria. Please provide evidence showing how the higher certificate-return rates obtained from paired confidence bounds contribute to evaluating or selecting verification rules in practice.