- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
- Setting: "fusion pipelines" where a generator samples K=5 answers, a verifier (process reward model, PRM) scores them, and a fuser LLM writes one final answer from the candidates.
- Claim: the usual design "filter bad candidates with a verifier, then fuse" is wrong for strong fusers. Instead, pass ALL candidates plus per-candidate critiques to the fuser ("Critique-as-Annotation", CaA).
- Evidence (MATH-500, Qwen-Plus as generator and fuser, five open 8B PRMs as verifiers): majority vote 0.780; perfect-verifier selection ceiling 0.834; CaA 0.888; PRM-filtered fusion ("Gate") 0.818–0.866 (Table 1). All five Gate variants are below CaA by 2.2–7.0 points.
- Proposed explanation ("redundancy–rescue"): candidate sets are polarized (70.2% all-correct, 16.6% all-wrong, 13.2% mixed), so verifiers can only matter in the thin mixed tier; false negatives are harmless because 80.2% of problems have at least two correct candidates; the fuser "rescues" 34 of 83 all-wrong problems (41.0%) and corrupts only 6 of 390 correct ones.
- Also: an accounting-style "analytical framework" with a claimed filter/annotate crossover at verifier false-positive rate FP ≈ 0.25; a 60-point simulated FP/FN grid (Table 5, selection only); a weak-fuser ablation with Qwen-Turbo (CaA 0.736 vs oracle-selection 0.768); a step-level overlap analysis (70.6% of all-wrong rescues reuse an equation that appears in a rejected candidate); and "CaA-Protocol", a rule for choosing annotate vs filter from offline calibration.
Critical strengths
- A clean, practically relevant finding: on MATH-500 with Qwen-Plus, unfiltered critique-and-fuse (0.888) beats every PRM-filtered pipeline (0.818–0.866) and the perfect-selection ceiling (0.834), and 41% of all-wrong candidate sets are still solved by the fuser. If the confound below were removed, this would be a useful data point against the "select then synthesize" default in frameworks like Archon, which is why I would want to see it done properly.
Critical weaknesses
- The headline comparison does not isolate filtering. CaA = all 5 candidates + 5 Qwen-Plus critiques + fuser; Gate = PRM-filtered candidates, no critiques, + fuser (Sec. 2.2). Two things change at once: filtering AND the presence of critiques (written by the strong model, i.e. 5 extra large-model calls that Gate does not get, Sec. F.4). The missing arms G5+F (no filter, no critiques) and G5+V+C+F (filter + critiques) are exactly what the title question "annotate or filter?" needs. The claim in Sec. 5.3 that "any performance gap is attributable to filtering" is therefore not supported; the gain could come entirely from the critiques. This undermines C1, C2 and C3.
- The numbers and the "theory" do not hold together. (a) Table 2/4 say 57 rescues (34 all-wrong + 23 mixed) and 6 corruptions, which gives 390+57−6 = 441 = 0.882, but the headline is 444 = 0.888; Eq. 3 silently uses 26 mixed rescues instead of 23; Sec. 3.3 also calls 66 the number of mixed problems "where majority vote is wrong", while Table 2 implies 27. (b) The FP ≈ 0.25 "phase boundary" (Fig. 1b, Sec. 3.5) is a dashed line drawn by hand: the simulated grid only runs selection G5+V (Table 5), never filtered fusion, so no Gate accuracy at any FP < 0.41 was ever measured. The free parameters γ, βFuser, ρmin that define the boundary are never estimated. (c) The Qwen-Turbo "second regime" (Sec. 5.4, App. C) never runs a Gate pipeline at all; it argues that oracle selection G5+V* = 0.768 upper-bounds any Gate, but App. E.2 argues the opposite (Gate exceeds the selection ceiling because it contains a fuser). The two arguments cannot both be used. (d) App. E.1: 0.892 vs 0.888 is 0.4 points, not "4.1pp". (e) Single run at T=0.7, no seeds, no paired test (e.g. McNemar): a 2.2-point gap on 500 problems is 11 items and within ordinary sampling noise; the 95% CI on 0.888 alone is about ±2.8 points. (f) Gate thresholds were tuned for G5+V, not for G5+V+F (App. F.1), biasing the comparison. Together these make the main claims unverifiable as written.
Questions for authors
- Please report G5+F (all candidates, no critiques, no filter) and G5+V+C+F (filtered candidates with critiques). Which of the two changes—dropping the filter or adding critiques—produces the gain over Gate?
- Please reconcile 0.888 (444/500) with Table 2/4 (384 + 57 = 441 correct, i.e. 0.882), and 26 vs 23 mixed-tier rescues in Eq. 3 vs Table 2. Which is right?
- How exactly is θ* = 0.25 obtained? Sec. 3.4/3.5 give no formula, and no Gate run exists at FP < 0.41. Can you run simulated-verifier Gate pipelines (with the fuser) across the grid to show an actual crossover?
- For Qwen-Turbo, please run real Gate pipelines (G5+V+F with the 5 PRMs). Given Turbo's 4.4% corruption rate, does Gate even beat G5 = 0.704?
- Please add repeated runs (3+ seeds) and a paired significance test for CaA vs each Gate variant, and report compute matched by tokens (critiques are 5 large-model calls; PRM scoring is not).
- Sec. 4.5: if the FP-like rate is 60.6% in the all-wrong tier and 1.3% elsewhere, how do the per-verifier FP values of 0.41–0.64 arise? Please define FP precisely.
- Step-level overlap (App. E.3): what is the base rate? An independent correct sample would also share setup equations with wrong candidates. Without that control, 70.6% does not show recycling.
3 comments
Thank you for the corrections, even if they are on short notice just before the end of the rebuttal period.
They've addressed many of my concerns and I'm switching the rating accordingly.
Thanks for the re-evaluation
Thank you for re-evaluating our submission and for updating your assessment—we genuinely appreciate it, especially given the proximity to the deadline. We are glad that the corrected scorer, the completed 2×2 factorial experiment, and the retraction of unsupported claims have resolved your central concerns.
Thank you again for your careful and technically precise engagement throughout the discussion. It has materially strengthened our paper.
2×2 factorial, McNemar significance, phase-boundary retraction, and corrected FP/overlap
Thank you for the careful review. The shared scorer-correction table is in our general official comment. Our responses to your questions follow.
Q1 — report G5+F and G5+V+C+F. Done — see the 2×2 in our general comment and in the response to eMXu (G5+F = 0.776; G5+V+C+F = Gate = 0.778). The answer to your "which change produces the gain?" question is adding critiques, not dropping the filter: critiques give +15.4 pp (corruption 75 → 0), while filtering is neutral without critiques (−0.8 pp) and harmful with them (−15.2 pp).
Q2 — reconcile 0.888 vs 0.882 and 26 vs 23. Both trace to the substring scorer (see the general comment). The corrected single value is CaA = 0.930 (465/500); the paper's 441/0.882 and the 26-vs-23 mixed-rescue mismatch disappear under the corrected ledger.
Q3 — how is θ*=0.25 obtained? It was not directly measured, and we retract it. Rather than re-running the full simulated grid with the fuser, we measured the endpoints that bracket the entire operating range: perfect-verifier filtered fusion G5+V*+F = 0.648 at FP=0 (below real-verifier Gate 0.71–0.78 and far below CaA 0.930), and the five real-verifier Gates (0.712–0.778). Since CaA (0.930) already beats filtered fusion at FP=0, and filtered fusion is bounded by the selection ceiling 0.828 while CaA breaks it by rescuing all-wrong problems, there is no crossover for this (generator, fuser) pair; the 0.25 boundary was an artifact of simulating selection without the fuser.
Q4 — run real Gate pipelines for Turbo. We withdraw the weak-fuser claim rather than defend it. Two points. First, your question's premise cites the pre-correction numbers: the corrected Turbo corruption is 0.3% (1/360), not 4.4%, and the corrected Turbo CaA (0.794) already exceeds Turbo G5 (0.72) by +7.4 pp, so the "weak fuser needs gating" claim is even less supported under the corrected scorer. Second, the §5.4/Appendix C argument was logically invalid (below an upper bound does not imply below every bounded method), and no real-PRM Turbo Gate exists; we could not run it now (qwen-turbo unavailable on our endpoint). This does not affect the paper's central claim.
Q5 — repeated runs and a paired significance test. We ran a paired McNemar test on the per-problem outputs (N=500), reporting two-sided exact-binomial p-values. CaA significantly outperforms every alternative: vs G5, p = 4.2e-22 (discordant 72 vs 0); vs each of the five Gate variants and the ensemble, all p ≤ 1.2e-16 (Skywork 1.2e-16, EurusPRM-S2 5.7e-28, ensemble 9.5e-26); vs G5+F (the critique ablation), p = 2.5e-15. Conversely, filtering without critiques (G5+F vs G5+V+F) is not significant (p = 0.70, discordant 32 vs 28), confirming filtering alone is neutral. The reported CaA–Gate gaps are therefore not sampling noise. We will add 2–3 seeds where budget permits.
Q6 — define FP precisely. See the eMXu Q3 answer: FP = P(pass | candidate wrong) at the candidate level, with pass = min-max-normalized score ≥ τ. The "60.6% vs 1.3%" in §4.5 mixed P(pass | wrong) with P(wrong | pass); the tier-stratified P(pass | wrong) is 0.53–0.73 (all-wrong) and 0.32–0.92 (mixed), which is how the per-verifier FP values of 0.48–0.75 arise.
Q7 — step-overlap base rate. We added the control. The overlap script carried the same substring bug; with the corrected scorer, the all-wrong rescue reuse rate is 96.2% (51/53), while the base rate on the mixed tier (where recycling is not forced) is 70.9% (39/55). The all-wrong rate exceeds the base rate by ~25 pp, providing partial (not strong) support for recycling; we have softened the micro-evidence claim and no longer present 70.6% as decisive.
Two further points from your review: (i) the App. E.1 "4.1 pp" discrepancy is 0.4 pp (0.892 − 0.888), now corrected; and (ii) on threshold tuning (your W2f), the corrected Gate–CaA gap is 15–22 pp, and even the perfect verifier — which has no threshold to tune — yields filtered fusion of only 0.648, far below CaA 0.930, so no threshold choice can close a gap of this size.
We hope this addresses your concerns.