acceptodds
Under review as a conference paper at ICLR 2027

Annotate or Filter? The Redundancy–Rescue Mechanism and Phase Boundaries in LLM Fusion Pipelines

Abstract

Multi-component pipelines such as Archon place a verifier in front of a fuser, assuming that discarding incorrect candidates can only help synthesis. We show that this selection-before-synthesis prior is mechanistically inverted. On MATH-500, five 8B process reward models all degrade fusion: verifier-filtered pipelines reach 0.818–0.866 accuracy versus 0.888 for unfiltered critique-and-fuse, a loss of 2.2–7.0 points. The cause is not verifier incompetence but a redundancy–rescue mechanism. Generator outputs are strongly polarized—most problems are either entirely correct or entirely wrong—so a verifier can act only in the thin mixed tier, where false positives pass errors through while false negatives are absorbed by redundancy (80.2% of problems have at least two correct candidates). The fuser, conversely, is an error corrector: it rescues 41.0% of majority-vote failures and corrupts 1.4%, synthesizing correct answers even from all-wrong sets and thereby breaking the generator ceiling (0.834 0.888). Filtering therefore starves fusion of the incorrect candidates that constitute its corrective signal. We derive a three-tier accuracy decomposition that predicts both the FP/FN asymmetry and a filtering/fusion phase boundary from two observable statistics—candidate redundancy and fuser rescue rate. Strong fusors should receive annotated but unfiltered candidates; weak fusors still benefit from gating. The crossover lies near FP 0.25, well below the operating regime of current 8B PRMs (0.41–0.64). Dual-path cross-validation against a 60-point FP/FN grid and a weak-fuser replication confirm both. The implication is architectural: verifiers should annotate, not filter.

Primary area:
foundation or frontier models, including LLMs
Keywords:
inference-time compute, test-time scaling, process reward models, mathematical reasoning, multi-LLM fusion, error correction, phase boundaries

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QuBzOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • Setting: "fusion pipelines" where a generator samples K=5 answers, a verifier (process reward model, PRM) scores them, and a fuser LLM writes one final answer from the candidates.
  • Claim: the usual design "filter bad candidates with a verifier, then fuse" is wrong for strong fusers. Instead, pass ALL candidates plus per-candidate critiques to the fuser ("Critique-as-Annotation", CaA).
  • Evidence (MATH-500, Qwen-Plus as generator and fuser, five open 8B PRMs as verifiers): majority vote 0.780; perfect-verifier selection ceiling 0.834; CaA 0.888; PRM-filtered fusion ("Gate") 0.818–0.866 (Table 1). All five Gate variants are below CaA by 2.2–7.0 points.
  • Proposed explanation ("redundancy–rescue"): candidate sets are polarized (70.2% all-correct, 16.6% all-wrong, 13.2% mixed), so verifiers can only matter in the thin mixed tier; false negatives are harmless because 80.2% of problems have at least two correct candidates; the fuser "rescues" 34 of 83 all-wrong problems (41.0%) and corrupts only 6 of 390 correct ones.
  • Also: an accounting-style "analytical framework" with a claimed filter/annotate crossover at verifier false-positive rate FP ≈ 0.25; a 60-point simulated FP/FN grid (Table 5, selection only); a weak-fuser ablation with Qwen-Turbo (CaA 0.736 vs oracle-selection 0.768); a step-level overlap analysis (70.6% of all-wrong rescues reuse an equation that appears in a rejected candidate); and "CaA-Protocol", a rule for choosing annotate vs filter from offline calibration.
Critical strengths
  • A clean, practically relevant finding: on MATH-500 with Qwen-Plus, unfiltered critique-and-fuse (0.888) beats every PRM-filtered pipeline (0.818–0.866) and the perfect-selection ceiling (0.834), and 41% of all-wrong candidate sets are still solved by the fuser. If the confound below were removed, this would be a useful data point against the "select then synthesize" default in frameworks like Archon, which is why I would want to see it done properly.
Critical weaknesses
  • The headline comparison does not isolate filtering. CaA = all 5 candidates + 5 Qwen-Plus critiques + fuser; Gate = PRM-filtered candidates, no critiques, + fuser (Sec. 2.2). Two things change at once: filtering AND the presence of critiques (written by the strong model, i.e. 5 extra large-model calls that Gate does not get, Sec. F.4). The missing arms G5+F (no filter, no critiques) and G5+V+C+F (filter + critiques) are exactly what the title question "annotate or filter?" needs. The claim in Sec. 5.3 that "any performance gap is attributable to filtering" is therefore not supported; the gain could come entirely from the critiques. This undermines C1, C2 and C3.
  • The numbers and the "theory" do not hold together. (a) Table 2/4 say 57 rescues (34 all-wrong + 23 mixed) and 6 corruptions, which gives 390+57−6 = 441 = 0.882, but the headline is 444 = 0.888; Eq. 3 silently uses 26 mixed rescues instead of 23; Sec. 3.3 also calls 66 the number of mixed problems "where majority vote is wrong", while Table 2 implies 27. (b) The FP ≈ 0.25 "phase boundary" (Fig. 1b, Sec. 3.5) is a dashed line drawn by hand: the simulated grid only runs selection G5+V (Table 5), never filtered fusion, so no Gate accuracy at any FP < 0.41 was ever measured. The free parameters γ, βFuser, ρmin that define the boundary are never estimated. (c) The Qwen-Turbo "second regime" (Sec. 5.4, App. C) never runs a Gate pipeline at all; it argues that oracle selection G5+V* = 0.768 upper-bounds any Gate, but App. E.2 argues the opposite (Gate exceeds the selection ceiling because it contains a fuser). The two arguments cannot both be used. (d) App. E.1: 0.892 vs 0.888 is 0.4 points, not "4.1pp". (e) Single run at T=0.7, no seeds, no paired test (e.g. McNemar): a 2.2-point gap on 500 problems is 11 items and within ordinary sampling noise; the 95% CI on 0.888 alone is about ±2.8 points. (f) Gate thresholds were tuned for G5+V, not for G5+V+F (App. F.1), biasing the comparison. Together these make the main claims unverifiable as written.
Questions for authors
  1. Please report G5+F (all candidates, no critiques, no filter) and G5+V+C+F (filtered candidates with critiques). Which of the two changes—dropping the filter or adding critiques—produces the gain over Gate?
  2. Please reconcile 0.888 (444/500) with Table 2/4 (384 + 57 = 441 correct, i.e. 0.882), and 26 vs 23 mixed-tier rescues in Eq. 3 vs Table 2. Which is right?
  3. How exactly is θ* = 0.25 obtained? Sec. 3.4/3.5 give no formula, and no Gate run exists at FP < 0.41. Can you run simulated-verifier Gate pipelines (with the fuser) across the grid to show an actual crossover?
  4. For Qwen-Turbo, please run real Gate pipelines (G5+V+F with the 5 PRMs). Given Turbo's 4.4% corruption rate, does Gate even beat G5 = 0.704?
  5. Please add repeated runs (3+ seeds) and a paired significance test for CaA vs each Gate variant, and report compute matched by tokens (critiques are 5 large-model calls; PRM scoring is not).
  6. Sec. 4.5: if the FP-like rate is 60.6% in the all-wrong tier and 1.3% elsewhere, how do the per-verifier FP values of 0.41–0.64 arise? Please define FP precisely.
  7. Step-level overlap (App. E.3): what is the base rate? An independent correct sample would also share setup equations with wrong candidates. Without that control, 70.6% does not show recycling.
3 comments
AuthorsOpenReview ↗

2×2 factorial, McNemar significance, phase-boundary retraction, and corrected FP/overlap

Thank you for the careful review. The shared scorer-correction table is in our general official comment. Our responses to your questions follow.

Q1 — report G5+F and G5+V+C+F. Done — see the 2×2 in our general comment and in the response to eMXu (G5+F = 0.776; G5+V+C+F = Gate = 0.778). The answer to your "which change produces the gain?" question is adding critiques, not dropping the filter: critiques give +15.4 pp (corruption 75 → 0), while filtering is neutral without critiques (−0.8 pp) and harmful with them (−15.2 pp).

Q2 — reconcile 0.888 vs 0.882 and 26 vs 23. Both trace to the substring scorer (see the general comment). The corrected single value is CaA = 0.930 (465/500); the paper's 441/0.882 and the 26-vs-23 mixed-rescue mismatch disappear under the corrected ledger.

Q3 — how is θ*=0.25 obtained? It was not directly measured, and we retract it. Rather than re-running the full simulated grid with the fuser, we measured the endpoints that bracket the entire operating range: perfect-verifier filtered fusion G5+V*+F = 0.648 at FP=0 (below real-verifier Gate 0.71–0.78 and far below CaA 0.930), and the five real-verifier Gates (0.712–0.778). Since CaA (0.930) already beats filtered fusion at FP=0, and filtered fusion is bounded by the selection ceiling 0.828 while CaA breaks it by rescuing all-wrong problems, there is no crossover for this (generator, fuser) pair; the 0.25 boundary was an artifact of simulating selection without the fuser.

Q4 — run real Gate pipelines for Turbo. We withdraw the weak-fuser claim rather than defend it. Two points. First, your question's premise cites the pre-correction numbers: the corrected Turbo corruption is 0.3% (1/360), not 4.4%, and the corrected Turbo CaA (0.794) already exceeds Turbo G5 (0.72) by +7.4 pp, so the "weak fuser needs gating" claim is even less supported under the corrected scorer. Second, the §5.4/Appendix C argument was logically invalid (below an upper bound does not imply below every bounded method), and no real-PRM Turbo Gate exists; we could not run it now (qwen-turbo unavailable on our endpoint). This does not affect the paper's central claim.

Q5 — repeated runs and a paired significance test. We ran a paired McNemar test on the per-problem outputs (N=500), reporting two-sided exact-binomial p-values. CaA significantly outperforms every alternative: vs G5, p = 4.2e-22 (discordant 72 vs 0); vs each of the five Gate variants and the ensemble, all p ≤ 1.2e-16 (Skywork 1.2e-16, EurusPRM-S2 5.7e-28, ensemble 9.5e-26); vs G5+F (the critique ablation), p = 2.5e-15. Conversely, filtering without critiques (G5+F vs G5+V+F) is not significant (p = 0.70, discordant 32 vs 28), confirming filtering alone is neutral. The reported CaA–Gate gaps are therefore not sampling noise. We will add 2–3 seeds where budget permits.

Q6 — define FP precisely. See the eMXu Q3 answer: FP = P(pass | candidate wrong) at the candidate level, with pass = min-max-normalized score ≥ τ. The "60.6% vs 1.3%" in §4.5 mixed P(pass | wrong) with P(wrong | pass); the tier-stratified P(pass | wrong) is 0.53–0.73 (all-wrong) and 0.32–0.92 (mixed), which is how the per-verifier FP values of 0.48–0.75 arise.

Q7 — step-overlap base rate. We added the control. The overlap script carried the same substring bug; with the corrected scorer, the all-wrong rescue reuse rate is 96.2% (51/53), while the base rate on the mixed tier (where recycling is not forced) is 70.9% (39/55). The all-wrong rate exceeds the base rate by ~25 pp, providing partial (not strong) support for recycling; we have softened the micro-evidence claim and no longer present 70.6% as decisive.

Two further points from your review: (i) the App. E.1 "4.1 pp" discrepancy is 0.4 pp (0.892 − 0.888), now corrected; and (ii) on threshold tuning (your W2f), the corrected Gate–CaA gap is 15–22 pp, and even the perfect verifier — which has no threshold to tune — yields filtered fusion of only 0.648, far below CaA 0.930, so no threshold choice can close a gap of this size.

We hope this addresses your concerns.

Thank you for the corrections, even if they are on short notice just before the end of the rebuttal period.

They've addressed many of my concerns and I'm switching the rating accordingly.

AuthorsOpenReview ↗

Thanks for the re-evaluation

Thank you for re-evaluating our submission and for updating your assessment—we genuinely appreciate it, especially given the proximity to the deadline. We are glad that the corrected scorer, the completed 2×2 factorial experiment, and the retraction of unsupported claims have resolved your central concerns.

Thank you again for your careful and technically precise engagement throughout the discussion. It has materially strengthened our paper.

Reviewer_w3K3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the role of the verifier in multicomponent pipelines. It investigates whether or not verifiers inadvertently hurt performance of the models downstream of verificaiton. They show that analytically

Critical strengths

This is a well crafted paper that is both easy to follow and well grounded in both analysis and empirical results. The findings are counterintuitive and potentially impactful. I particularly liked how they broke the problem down into subsequent steps, isolated a mechanism they wanted to study and found that altering this mechanism challenged some fundamental assumptions in this literature.

Critical weaknesses

While the results were well grounded analytically the empirical evidence is limited to just a single benchmark MATH-500. I would expect to see broader evidence that this mechanism functions as claimed before fully trusting these results.

Questions for authors
  1. Is there a way to show that this result generalizes across more dataset? It appears that this mechanism doesn't improve performance on GSMK due to the redundancy in that dataset, are there other benchmarks that could help make this argument stronger?

  2. In the all-wrong setting does the verifier block all solutions from going to the fuser, if so what kind of response is the fuser even capable of? What happens if you select just one of the answers in the all-wrong case.

1 comment
AuthorsOpenReview ↗

Generalization to AIME 2022–2025 and the all-wrong rescue mechanism

Thank you for the review. The shared scorer-correction table is in our general official comment. Our responses to your questions follow.

Q1 — does this generalize beyond MATH-500? Yes — we have now closed this gap empirically. We ran the identical CaA pipeline (Qwen-Plus generator K=5 T=0.7; critic; fuser) on AIME 2022–2025 (120 problems, integer answers, so scoring has no LaTeX/extraction ambiguity). AIME is exactly the regime our mechanism predicts matters: only 13/120 (10.8%) problems are all-correct and 89/120 (74.2%) are all-wrong.

MetricMATH-500AIME 2022–2025
G5 (majority)0.7860.183
CaA0.9300.417
CaA − G5+14.4 pp+23.3 pp
selection ceiling0.8280.258
CaA − ceiling+10.2 pp+15.8 pp
all-wrong share17.2%74.2%
all-wrong rescue53/86 (61.6%)22/89 (24.7%)
corruption01/120

CaA more than doubles G5 on AIME (0.183 → 0.417) and exceeds the perfect-filter ceiling by +15.8 pp — a filter cannot reach 0.26 because 74% of problems have no correct candidate to select. The fuser synthesizes a correct answer from entirely wrong candidates on 22/89 (24.7%) all-wrong problems while corrupting only 1/120. Your observation about GSM8K is exactly right and is now explained: GSM8K saturates (49/50 all-correct) because it has almost no all-wrong tier. The mechanism is mechanism-dependent, not MATH-specific, and this is now demonstrated rather than asserted.

Q2 — what happens in the all-wrong setting? When every candidate is wrong, the verifier (as gate) rejects all of them and Gate falls back to top-1 — so the fuser sees a single wrong candidate (or an empty/forced fallback) and has nothing useful to synthesize from. Selecting just one answer in the all-wrong case inherits that single candidate's error. In contrast, CaA passes all wrong candidates with critiques to the fuser, which recombines fragments of their reasoning to synthesize a correct answer in 22/89 (AIME) and 53/86 (MATH-500) all-wrong problems. This is precisely the capability gating destroys.

We hope this addresses your concerns.

Reviewer_eMXuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether verification should filter candidate solutions before LLM fusion or provide feedback while retaining the candidates. It reports that unfiltered critique-and-fuse outperforms five PRM-filtered fusion pipelines on MATH-500 and proposes a redundancy–rescue explanation: candidates with incorrect final answers may still contain useful intermediate reasoning. The paper further introduces an accuracy decomposition and a calibration protocol intended to predict when annotation or filtering is preferable. I find the architectural question and the reported empirical trend useful, but the current manuscript contains substantial inconsistencies in its numerical accounting and does not yet establish the claimed causal mechanism or predictive phase boundary.

Critical strengths

S1. A meaningful architectural question for inference-time reasoning systems.

The paper challenges the assumption that removing incorrect candidates necessarily improves downstream synthesis. The observation that an incorrect final answer may still contain useful intermediate reasoning is important for designing multi-component inference pipelines. The distinction between selecting an existing answer and synthesizing a new one is particularly relevant to inference-time scaling. The proposed rescue/corruption analysis offers a potentially useful framework for understanding these interactions.

S2. Consistent empirical trends across multiple verifiers, with promising evidence of fusion beyond selection.

The reported experiments show that CaA outperforms all five individual PRM-filtered fusion configurations on MATH-500. The all-wrong rescue analysis is especially interesting: the fuser reportedly solves 34 of 83 problems even though none of the original candidates has a correct final answer.

Importantly, even if the main CaA accuracy is corrected from 0.888 to 0.882 according to Tables 2 and 4, it would still exceed the best reported Gate result of 0.866. Thus, the central empirical observation appears potentially meaningful despite the numerical inconsistencies discussed below.

These findings motivate further investigation of how candidate diversity, partial reasoning correctness, and fusion capability interact.

Critical weaknesses

W1. The experimental comparisons do not isolate the effect of filtering, and the claimed weak-fuser reversal is not directly demonstrated.

The comparison between CaA and Gate changes more than whether candidates are filtered. According to Section 2.2 and Algorithm 1, CaA receives all candidates together with generated critiques, whereas Gate receives PRM-filtered candidates without corresponding critiques. Therefore, the statement in Section 5.3 that "the only difference is pre-filtering" is not supported by the specified experimental setup.

The observed improvement may reflect a combination of candidate retention, additional critique information, and increased reasoning computation. A controlled comparison with and without filtering, while holding critique availability fixed, is necessary to establish the proposed mechanism.

Similarly, the step-level equation-overlap analysis in Section 4.6 and Appendix E provides suggestive correlational evidence, but does not establish that the fuser causally depends on the identified fragments from rejected candidates. The fuser could independently derive similar equations or benefit from the additional critique computation.

The weak-fuser argument in Section 5.4 and Appendix C also contains a logical error. The authors conclude that Turbo-CaA (0.736) must underperform real gated fusion because it is below the oracle-selection accuracy (0.768). However, being below an upper bound does not imply being below every method bounded by it. Moreover, the oracle-selection ceiling does not generally upper-bound a pipeline containing a generative fuser, as the paper itself correctly explains elsewhere.

Appendix C explicitly states that no real-PRM Turbo gated-fusion results are available. Consequently, the central claim that weak fusors benefit from gating remains unverified.

W2. Several core numerical and theoretical claims are internally inconsistent, and the proposed phase boundary lacks direct validation.

First, Section 3.3 reports:

390 + 34 + 26 - 6 = 444,

yielding CaA accuracy 0.888. However, Tables 2 and 4 report 384 retained correct answers and 57 rescues, implying:

(384 + 57) / 500 = 0.882.

Figure 1a and Appendix E.3 also report 23 mixed-tier rescues rather than 26. These discrepancies directly affect the headline accuracy and the proposed rescue/corruption decomposition.

Second, the theoretical formulation uses inconsistent conditioning events for the rescue probability. The reported value 34/83 = 0.410 measures rescue conditional on all candidates being wrong, whereas Tables 2 and 4 imply a rescue probability of 57/110 = 0.518 conditional on majority-vote failure. These quantities are not interchangeable.

Substituting the paper's stated alpha_R = 0.410 into the accuracy formulation in Section 3.4 gives:

Acc(CaA) <= 0.780 + 0.410 * 0.220 = 0.8702,

assuming a nonnegative distraction penalty. Thus, the proposed equation cannot reproduce either the reported accuracy of 0.888 or the table-implied accuracy of 0.882 under its stated definitions.

Third, the phase-boundary evidence is incomplete. Table 5 and Figure 1b evaluate simulated G5+V selection, rather than the actual filtered-fusion architecture G5+V+F. Consequently, the reported 60-point FP/FN grid does not directly establish the claimed CaA-versus-Gate crossover near FP = 0.25. Algorithm 1 also does not provide an explicit, operational expression for computing the phase-boundary function or enough information to reproduce all parameter estimates.

Fourth, Appendix F.2 reverses the correct-to-incorrect and incorrect-to-correct label-flipping definitions relative to the main text's interpretation of FP and FN. The paper also alternates between top-scoring selection and first-passing selection, which have different error propagation properties.

Finally, some secondary numerical claims require correction, including the incorrect GPM outlier attribution in Section 2.3 and the reported 4.1 percentage-point discrepancy between 0.892 and 0.888 in Appendix E.1.

I consider these issues potentially addressable, but they must be resolved before the analytical framework and configuration protocol can be regarded as adequately validated.

Questions for authors

Q1. Can the authors reconcile the main accuracy and rescue/corruption statistics?

Section 3.3 reports 444 correct CaA answers, whereas Tables 2 and 4 imply 441. Is the correct CaA accuracy 0.888 or 0.882? Are there 26 or 23 mixed-tier rescues?

Please provide a consistent per-problem result ledger and complete rescue/corruption transition tables for both Plus and Turbo. Please also distinguish the rescue probability conditional on all-wrong candidates from that conditional on majority-vote failure.

This clarification is important because the discrepancies affect both the main empirical result and the theoretical formulation.

Q2. Can the authors isolate the effect of candidate filtering from the effect of critique generation?

The current CaA and Gate configurations differ in both candidate retention and critique availability.

Could the authors report a controlled 2x2 comparison using the same cached candidates and fuser:

  • All candidates, without critiques.
  • All candidates, with critiques.
  • Filtered candidates, without critiques.
  • Filtered candidates, with the same pre-generated critiques.

This experiment would materially strengthen the proposed explanation by separating the contribution of candidate retention from that of additional critique information.

If feasible, targeted removal or replacement of the identified useful fragments from rejected candidates would further clarify whether the observed equation overlap reflects a causal contribution.

Q3. Can the authors clarify the FP/FN definitions, selection procedure, and theoretical probability terms?

Appendix F.2 describes correct-to-incorrect flips as FP and incorrect-to-correct flips as FN, apparently reversing the definitions used in the main text.

Please specify the positive class, denominator, statistical unit, and actual simulation implementation. Is G5+V based on selecting the highest-scoring candidate or the first candidate passing a threshold?

Additionally, can the authors derive a corrected accuracy equation using consistently conditioned rescue and corruption probabilities? In particular, how does the current formulation account for the discrepancy between 34/83 and 57/110, and for possible corruption in both CaA and Gate?

Q4. What direct evidence supports the proposed filtering/fusion phase boundary near FP = 0.25?

Table 5 and Figure 1b report simulated G5+V rather than G5+V+F. Therefore, the presented grid does not directly demonstrate a crossover between CaA and actual filtered fusion.

Could the authors provide filtered-fusion measurements around the proposed threshold, together with the explicit derivation and numerical estimates used to obtain the boundary?

It would also be valuable to demonstrate that the calibration protocol predicts the preferable architecture on held-out examples rather than merely explaining the observed results retrospectively.

Q5. Can the authors directly validate the claim that weak fusors benefit from gating?

The current Turbo experiment compares CaA with an oracle-selection result, not with a real PRM-filtered fusion pipeline. The upper-bound argument in Section 5.4 does not establish the claimed ranking.

Could the authors provide Turbo-CaA versus Turbo-Gate comparisons using actual PRMs?

Ideally, generator outputs and candidate sets should be held fixed when changing fuser capability, so that the claimed regime transition can be attributed to fusion strength rather than simultaneous changes in the generator, critiques, and fuser.

A convincing response to these questions, particularly the numerical reconciliation and controlled architectural comparisons, could materially improve my assessment.

1 comment
AuthorsOpenReview ↗

Numerical reconciliation, 2×2 factorial, FP/FN definitions, and retracted claims

Thank you for the review. The shared scorer-correction table is in our general official comment.

Q1 — reconcile the main accuracy and rescue/corruption statistics. The 444-vs-441 and 26-vs-23 discrepancies all trace to the substring scorer (see the general comment). With the corrected scorer there is a single consistent ledger: CaA = 465/500 = 0.930, decomposed as 393 G5-correct retained − 0 corrupted + 72 rescued = 465. We distinguish the two conditioning events you flagged: rescue conditional on all-wrong candidates is 53/86 (61.6%); rescue conditional on majority-vote failure (107 problems) is 72/107 (67.3%). Tier ledger: all-correct 357→357, mixed 57→55, all-wrong 86→53 (465 correct, 35 wrong). The full per-problem ledger is in the revised manuscript.

Q2 — isolate filtering from critique generation. We ran the requested 2×2 on the same cached candidates and fuser (all K=5, Qwen-Plus fuser, MATH-500):

no critiqueswith critiques
all candidatesG5+F = 0.776CaA = 0.930
filtered (Skywork)G5+V+F = 0.768Gate = 0.778

Fuser corruption (G5-right problems the fuser then got wrong): CaA 0/393, G5+F 75/393, Gate 70/393, G5+V+F 79/393. The critiques — not candidate retention — are the operative factor. Adding critiques eliminates corruption (75 → 0) and lifts accuracy 0.776 → 0.930 (+15.4 pp); filtering is neutral without critiques (−0.8 pp) and harmful with them (−15.2 pp). The title question is answered sharply: annotate, and do not filter. (Temperature-independent: G5+F is identical at T=0.0 and T=0.7.)

Q3 — FP/FN definitions and selection procedure. We define FP and FN at the candidate level: FP = P(pass | candidate wrong), FN = P(reject | candidate correct). The positive class is "candidate correct", the denominator is all candidates, the statistical unit is the individual candidate (5 per problem). "Pass" means the min-max-normalized score ≥ τ (τ = 0.5 for GPM/QwenPRM/EurusPRM-S2, 0.1 for Skywork/EurusPRM-S1). G5+V selects the highest-scoring candidate (argmax); Gate filters by score ≥ τ and fuses survivors (fallback top-1). This single convention is stated explicitly and used consistently. Recomputed with corrected labels:

verifierFPFNG5+V (argmax)
GPM0.6450.4780.718
Skywork0.6480.2510.826
EurusPRM-S20.4810.4230.806
EurusPRM-S10.7490.3160.758
QwenPRM0.5220.3480.800

The §4.5 "60.6% vs 1.3%" figures mixed two different quantities; we replace them with tier-stratified FP = P(pass | wrong): all-wrong 0.53–0.73, mixed 0.32–0.92. Real 8B PRMs operate at FP ≈ 0.48–0.75, above the filtering-harmful threshold.

Q4 — direct evidence for the phase boundary. We accept that Table 5/Figure 1b evaluated simulated selection (G5+V), not filtered fusion. We measured the two endpoints that bracket the boundary. The perfect-verifier filtered fusion G5+V*+F (FP=0) achieves only 0.648 — below the real-verifier Gate (0.71–0.78) and far below CaA (0.930). Analytically, G5+V+F is bounded by the selection ceiling (1 − p_none = 0.828) regardless of verifier quality, because a filter only selects among existing correct candidates and can never rescue an all-wrong set; CaA breaks this ceiling by rescuing 53/86 all-wrong problems. Hence there is no crossover — CaA beats Gate at every FP — and θ*≈0.25 was an artifact of analyzing selection without the fuser. The revision retracts θ*≈0.25 and documents the no-crossover result.

Q5 — weak fusers benefit from gating. We agree the a fortiori argument in §5.4 was logically invalid and Appendix C admits no real-PRM Turbo Gate was run. We withdraw the weak-fuser claim (§5.4 and Appendix C) and restate it as an untested hypothesis. The corrected Turbo numbers further undermine it: corruption 0.3% (1/360), and CaA (0.794) already exceeds G5 (0.72) by +7.4 pp. We could not re-run it now (qwen-turbo unavailable on our endpoint). The central claim (strong fusers should annotate, not filter) is unaffected.

Two further corrections you flagged: the GPM "outlier exceeds the grid" attribution in §2.3 is wrong — GPM's FP (0.645) lies within the simulated grid (which extends to 0.70), not beyond it; and the App. E.1 figure is a 0.4 pp discrepancy (0.892 − 0.888), not "4.1 pp." Both are corrected.

We hope this addresses your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Scorer correction and revised headline numbers (CaA 0.930, corruption 0)

We thank the reviewers for careful and technically precise feedback. The shared concern — that our numerical accounting was inconsistent, and that the CaA-vs-Gate comparison conflated filtering with critique availability — is correct. We have fixed both at the root and added the requested experiments. The corrected result is stronger than originally reported.

A common correction (addressed by R1-Q1, R3-Q2, and both reviewers' W2)

Our §2.4 stated that answers were compared with "sympy equivalence," but the released scorer used naive substring matching (gold in candidate). This single bug produced every numerical inconsistency. We re-scored all outputs with the documented protocol (3-tier extractor → LaTeX normalization → base-notation tolerance → sympy):

QuantityReportedCorrected
G10.7720.780 (390/500)
G50.7800.786 (393/500)
G5+V* (selection ceiling)0.8340.828 (414/500)
CaA0.8880.930 (465/500)
corruption60
rescue5772 (72/107)
all-wrong rescue34/8353/86 (61.6%)
mixed rescue23/2719/21 (90.5%)

The correction strengthens the claim: CaA rises 0.888 → 0.930, corruption drops 6 → 0, and CaA now breaks the selection ceiling by +10.2 pp (0.930 − 0.828) rather than the reported +5.4 pp.


Changes in the revised manuscript

The revised PDF (uploaded alongside this response) makes the following changes, highlighted in blue:

  1. All Tables 1–7 recomputed with the corrected scorer (CaA 0.930, corruption 0, Gate 0.712–0.778).
  2. New 2×2 factorial table isolating critiques from filtering (§4.2).
  3. Retracted the weak-fuser claim (§5, Appendix C) and the phase-boundary crossover claim (§3.4).
  4. New AIME generalization and paired McNemar tests (Appendix G).
  5. Corrected FP/FN definitions and the step-overlap base rate (§4, Appendix E).
  6. Added an AI-use statement (per ICLR policy).

We believe these corrections and new experiments resolve the reviewers' central concerns and substantially strengthen the paper. We are grateful for the detailed and technically precise reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.