acceptodds
Under review as a conference paper at ICLR 2027

Diagnose, Then Select: What Makes Critique-Based Refinement Effective?

Abstract

Large language models increasingly rely on critique to refine their reasoning, yet existing approaches treat critique as an undifferentiated block of feedback and provide more information than may be necessary. We argue that critique-based refinement has two separate bottlenecks: the feedback must carry the causal information needed to repair an error, and the resulting revision must be reliably accepted or rejected. First, we investigate which parts of a critique make an error repairable by decomposing one structured record into isolated correctness-verdict, error-location, and error-diagnosis views, and compare them with the full critique. Across three refinement protocols, two model families, and two mathematical-reasoning benchmarks, diagnosis-only feedback recovers of the wrong-to-correct rate of a full critique while using of its feedback tokens; verdicts and locations recover only and . Correction alone, however, is an incomplete objective: critique also destabilizes answers that were already correct. Under solver–critic refinement on MATH-500, diagnosis corrects of the weaker Llama solver's wrong answers but corrupts of its correct ones, a net loss of points, while the same feedback gains points for the stronger Qwen solver. To address the second bottleneck, we propose **DiSel**-Consensus, a sampling-based revision selector that accepts a diagnosis-conditioned revision only when one independently sampled auxiliary revision supports it. At a common calibration risk budget, Consensus reduces correct-to-wrong transitions from under unconditional revision to , retains wrong-to-correct transitions, and gains accuracy points with 923 marginal tokens per trial. Compared with **DiSel**-Pointwise selection baseline, **DiSel**-Consensus achieves a significantly lower corruption rate, while the difference in net accuracy is not statistically significant.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Critique-Based Refinement, Self-Correction, Confidence-Based Selection, Multi-Agent Debate, Token Efficiency, Test-Time Inference, Resource-Efficient AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_254ROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that using critique to refine model reasoning has two bottlenecks: (1) the feedback must have enough causal information to fix an error, and (2) the model should know whether to accept or reject the revision. They then propose DISEL and four variants, and evaluate them across three protocols, two model families, and two mathematical benchmarks. Results show that their method recovers 98.8% of the wrong-to-correct rate of a full critique while using only 30.9% of the tokens.

Critical strengths

This paper shows that targeted feedback, which is the diagnosis of incorrect steps, can be effective and can save token usage.

Critical weaknesses
  1. The overall pipeline feels rule-based and hence includes a lot of design choices. For example, Eq (1) has a selector-specific threshold, which needs tuning, and the auxiliary answer also has specific rules to map to its actual vote. In lines 288-290, the authors also mention, "When repeat j is treated as the primary revision, we use the next repeat cyclically, j+ = 1 + (j mod 16)", which feels arbitrary. All these choices should be backed up either by a reference or an empirical result.

  2. Revision selectors have four variations. There's no clear winner, so it is also not clear which one to use under which scenario.

  3. The major weakness is the lack of baselines. At the very least, it should be compared with self-consistency [1], Best-of-N [2], simple test-time scaling (s1 [3]). Closely related, Magicore [4] is also an external baseline that should be compared with, since it also addresses the error identification issue with finer-grained refinement.

  4. The evaluation benchmark only includes math with two datasets, which is too limited in both number and domain. To show better generalization, the author should consider including GPQA and MMLU-Pro.

References:

Questions for authors
  1. What does it mean in Figure 1, the verdict shows "OBJECT" and the caption says "The verdict only objects to the answer"? I didn't get this. Shouldn't the verdict be correct or incorrect?

  2. It makes sense that diagnosis is effective, but in practice the challenge is how to get that diagnosis. It's not clear in the paper how this is obtained. I only found in lines 193 to 195, "Each record is generated once, by the paired critic, the solver itself, or a same-size peer depending on the protocol, and all four views are rendered from it" but this wasn't clear. How are the four views rendered?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 254R

Thank you for identifying the need to justify design choices, give deployment guidance, add test-time-scaling baselines, and broaden evaluation. We address each point in turn below.

W1. Calibration and pairing. Eq. (1)’s threshold is not tuned on test accuracy. For each selector/protocol it maximizes training-fold W2C subject to C2W≤ρ, using leave-one-year-out AIME or leave-one-subject-out MATH-500 calibration. The held-out group never selects its threshold. Cyclic pairing makes every one of 16 frozen revisions a primary once and an auxiliary once, without self-pairing or labels. Recomputing all nonzero offsets 1–15 at ρ=1% gives:

Table R1. Sensitivity of DiSeL-Consensus to the auxiliary offset (six-slice macro, ρ = 1%).

Macro metricRangeSubmitted +1
Selection8.02–11.13%10.85%
W2C10.06–10.43%10.41%
C2W.28–.60%0.34%
ΔAccuracy+2.51–2.86 pp+2.80 pp

Thus +1 is balanced and label-independent, not uniquely favorable or theoretically optimal.

W2. Which selector to use. We agree that no selector dominates, and we will add the following deployment guidance:

Table R2. Selector guidance (six-slice macro, ρ = 1%).

Use caseSelectorW2C/C2WTokens
Cost-conscious verifierPointwise9.63/1.42%431
Lowest observed corruption, black-boxConsensus10.41/0.34%923
Comparative judge despite costPairwise8.36/1.13%1,544
Logprobs availableGenLP8.27/2.21%0

Consensus may abstain completely under strict calibration. Pointwise is the default when tokens matter.

W3. Test-time-scaling baselines. We added self-consistency and Best-of-N on Qwen3-32B / MATH-500. Tokens count all generation and judging per problem.

Table R3. Self-consistency and Best-of-N (Qwen3-32B, MATH-500).

MethodNAccuracyTokens/problem
Self-consistency1 / 2 /487.60/88.00/89.80%902/1,807/3,651
Self-consistency8 / 1691.40/92.80%7,226/14,474
Best-of-N (Qwen3-235B judge)1 / 2 /487.60/89.60/89.60%1,752/3,502/7,052
Best-of-N (Qwen3-235B judge)8 / 1690.60 / 90.40%14,119 / 28,347
DiSeL-Consensus–91.17%4,590

At a comparable budget, DiSeL-Consensus (4,590 tokens) is more accurate than self-consistency with N = 4 (89.80% at 3,651 tokens) and every Best-of-N point up to N = 4. Matching the best Best-of-N point (N = 8) would require about 3× more tokens, and self-consistency needs N = 8 (7,226 tokens) to exceed it. Unlike both, DiSeL preserves the initial answer when evidence is weak and explicitly controls the W2C–C2W trade-off. The two approaches are also complementary, since DiSeL can be applied to a self-consistency output.

Regarding s1, its budget-forcing technique can be applied at test time without additional training. However, due to the limited rebuttal period, our empirical comparisons are limited to self-consistency and Best-of-N. We will discuss s1 as an important test-time-scaling baseline and identify its evaluation as future work. MAgICoRe also targets over-correction and error localization, but relies on external process/outcome reward models and iterative multi-agent refinement, whereas we study which content of a single critique is useful. We will discuss MAgICoRe in related work rather than claim an empirical comparison.

W4. Additional domain. We added MMLU-Pro Computer Science:

Table R4. Diagnosis-conditioned selection on MMLU-Pro Computer Science.

Solver/criticPointwiseConsensus
Llama-70B/Qwen-235B11.79 / 1.32%0 / 0%
Qwen-32B/Qwen-235B7.57 / 1.02%4.55 / 0.42%

Pointwise retains corrections at about 1% corruption outside mathematics, while Consensus remains conservative and abstains entirely for the Llama-70B pair. We have not yet evaluated GPQA and will note this as a limitation.

Q1. Meaning of OBJECT in Figure 1. OBJECT means that the critic rejects the response as incorrect. We will relabel Figure 1 to say exactly this.

Q2. How diagnosis and the four views are obtained. For each frozen initial solution, the critic (or the solver itself, or a same-size peer, depending on the protocol) is called once to produce one tagged record containing a verdict, an error location, and a diagnosis. The verdict-, location-, and diagnosis-field views are deterministic projections of that record, and the full critique renders all of its fields. All field comparisons therefore reuse the same underlying feedback event rather than regenerating different critiques. The exact prompts are in Appendix B, and we will move a short description of this procedure into the main text.

Thank you again for your thoughtful review. We wanted to briefly follow up on our rebuttal above, where we addressed your questions and concerns. If you have a chance to take a look, we’d be happy to clarify anything that’s still unclear. Thank you for your time.

Thank the authors for their detailed responses. While they have partially addressed my comments, I still think the fact that there's no clear winner to choose at test time is weakening the contribution. Therefore, I decided to keep my score.

Reviewer_RD4kOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper discuss LLM self-correction and critique-based refinement in mathematical reasoning, using model-generated feedback and sampling-based revision selection rather than formal verification. The authors examine four critique views: verdict, error location, diagnosis, and full critique. Diagnosis-only feedback retains 98.8% of the wrong-to-correct performance of full critique while using 30.9% of its feedback tokens. The authors proposed DiSEL-Consensus, which uses agreement with an independently sampled auxiliary revision to select revisions under a calibrated corruption-risk budget. The method substantially reduces correct-to-wrong transitions, but its net-accuracy advantage over the strongest competing selector is not statistically significant. The finding shows this particular feedback-and-revision pipeline; their dependence on prompting, critic reliability, and solver capability remains an important consideration.

Critical strengths

1. The paper distinguishes two important concepts: error repair and revision acceptance. The paper separates the ability to correct an incorrect answer from the decision of whether to accept the revised answer.

2. The method reduces harmful revisions. DiSEL-Consensus performs well in the experiments, reducing answer corruption while retaining useful corrections.

Critical weaknesses
  1. Broader coverage of related work is needed. The discussion would benefit from a broader search of related work in formal mathematics and code generation/repair, where feedback-driven correction and verification have received considerable attention. There be relevant connections that are not currently discussed, and I would encourage the authors to examine these before drawing broader conclusions about critique-based refinement.

  2. The interpretation of the critique ablation could be more cautious. Diagnosis may implicitly contain error-location information and corrective hints, so the comparison does not fully separate these contributions. The results support diagnosis-only feedback as a useful option under the tested settings, but its broader advantages may depend on prompt design and the capabilities of the solver and critic.

  3. The practical advantage over competing selectors remains somewhat unclear. Although Consensus reduces corruption, its overall accuracy advantage over the strongest competing selector is not statistically established, and it requires additional generation. A clearer discussion of the trade-off between accuracy, conservativeness, and computational cost would strengthen the practical case. Given the focus on mathematical reasoning, I would encourage the authors to consider a tool-assisted verification baseline, for example using an SMT solver where the reasoning can be appropriately formalized. Such a comparison would help clarify when LLM-generated critiques and consensus-based selection offer benefits over externally checked feedback.

Questions for authors
  1. Could the authors clarify how this work relates to feedback-guided refinement and verification in formal mathematics, program synthesis, and code repair, and what distinguishes its main contribution from these existing research directions?
  2. To what extent should the main findings be understood as general properties of critique-based refinement, rather than outcomes specific to the chosen models, prompts, and tasks?
  3. Could the authors discuss how feedback quality, solver capability, and revision selection contribute to the observed successes and failures, and what the current evidence allows us to conclude about their respective roles?
  4. Could the authors include stronger LLM-as-a-judge baselines, such as a GPT-based judge, to provide a broader comparison with the proposed selection method? And extended more additional state-of-the-art models, including stronger solver–critic combinations, to examine whether the main findings hold across different model capabilities?
  5. The author used a temperature other than 0. Could the author provide more temperatures and conduct multiple tests instead of relying on a single result?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RD4k (1/2)

Thank you for highlighting the distinction between repair and acceptance. Below, we respond to each of the weaknesses and questions you raised.

W1/Q1. Related work and external verification. We will add formal/process verification (Let’s Verify Step by Step/MATH-VF/APOLLO), execution-guided program repair (RLEF/FeedbackEval), and iterative refinement (MAgICoRe).

W2. Interpretation of the critique ablation. We agree that the diagnosis field is not a pure causal explanation: it may implicitly carry error-location information and corrective cues. To quantify this, we ran a conservative lexical audit of all 1,263 wrong-cohort diagnosis fields. Only 4.0% contain an explicit wrong verdict, and 14.9% an explicit structural location. Diagnosis nonetheless retains 98.8% of full-critique W2C (95% CI [91.4, 111.5], which includes 100%) at 30.9% of the feedback tokens. We therefore do not claim that diagnosis is semantically independent of location or hints. We claim only that, under our prompts and solver–critic pairs, the diagnosis field is a compact repair signal. We will rename “isolated diagnosis” to “diagnosis-field view” and state explicitly that its advantage may depend on prompt design and model capability.

Q2. Generality of the findings. We view our findings as robust within the tested scope rather than as general properties of critique-based refinement. The diagnosis result holds across three protocols, two model families, and two benchmarks: diagnosis retains 98.8% of full-critique W2C at 30.9% of the feedback tokens in the nine-cell aggregate. The net effect of refinement, by contrast, is clearly model-dependent, because the same feedback helps the Qwen-32B solver but corrupts the Llama-8B solver on MATH-500 (Table 1). To probe task dependence, we added MMLU-Pro Computer Science (protocol-macro W2C/C2W):

Solver/critic configurationPointwiseConsensus
Llama-70B/Qwen-235B11.79/1.32%0/0%
Qwen-32B/Qwen-235B7.57/1.02%4.55/.42%

Pointwise retains corrections at about 1% corruption outside mathematics, while Consensus remains conservative and abstains entirely for the Llama-70B pair. Our evidence is still limited to one prompt design, open-weight models, and math plus multiple-choice CS. We will state these limits explicitly and avoid claims beyond them.

W3. Practical advantage, trade-offs, and external verification. We agree that Consensus is not a uniformly better selector, and we will frame the selectors as a trade-off among accuracy, conservativeness, and cost. At the common ρ=1% budget, the six-slice macro results are (Figure 5; paired tests in Table 3):

Table R4. Accuracy–corruption–cost trade-off (six-slice macro, ρ=1%).

SelectorW2CC2WΔAccTokens/trial
GenLP8.27%2.21%+1.42 pp0
Pairwise8.36%1.13%+2.17 pp1,544
Pointwise9.63%1.42%+2.46 pp431
Consensus10.41%0.34%+2.80 pp923

Against the strongest competitor, Pointwise, Consensus significantly reduces corruption (−1.08 pp C2W, 95% CI [−2.14, −0.32], p<.0002 (Table 3 in manuscript)). Its net-accuracy advantage, however, is not statistically resolved (+0.35 pp, [−0.23, 1.00], p=.27), and it requires one additional generation (923 vs. 431 tokens/trial). We will therefore recommend Pointwise for cost-sensitive use, and Consensus when log probabilities are unavailable and low corruption is the priority. On external verification: SMT, CAS, or unit-test checking is preferable whenever a trustworthy formal specification exists. Our natural-language MATH/AIME trajectories lack one, so these methods are complementary to, rather than interchangeable with, critique-based selection. We will add this discussion and the corresponding related work.

AuthorsOpenReview ↗

Response to Reviewer RD4k (2/2)

Q3. Roles of feedback content, solver capability, and selection. Our design isolates three factors. For feedback content, the initial answer and critique record are fixed, so differences between views reflect content alone: diagnosis recovers 98.8% of full-critique W2C, whereas verdict and location recover only 42.6% and 62.3%. For solver capability, the same diagnosis feedback has opposite net effects across solvers: under Always revise on MATH-500, the Llama-8B solver is corrupted on 18.62% of its initially correct answers (−4.02 pp), while the Qwen-32B solver gains +1.77 pp (Table 1). For selection, the initial answer, diagnosis, and candidate revision are fixed, so differences between selectors reflect acceptance alone: Consensus reduces macro C2W from 13.60% to 0.34% while retaining about two thirds of Always revise’s corrections. One limitation remains. Varying the protocol (solver–critic, self-refinement, peer debate) changes the feedback source but does not identify a causal effect of feedback quality, since we have no oracle measure of critique quality; we will state this explicitly.

Q4. Stronger judges and models. Our Pointwise and order-controlled bidirectional Pairwise selectors are already LLM-as-a-judge baselines, evaluated on identical frozen candidate pairs. A proprietary GPT-based judge would be a useful addition, but it requires new inference, which we could not complete within the rebuttal period. Our current coverage spans Llama-8B and Qwen-32B solvers with Llama-70B and Qwen-235B critics, including cross-family pairs, so the findings already hold across a range of solver capabilities and critic strengths.

Q5. Sampling and repeated runs. Our results are not based on a single revision sample. For each frozen initial solution and critique record, we generate 16 conditionally independent revisions at temperature 0.6 and top-p 0.95. We aggregate all revisions and compute uncertainty by resampling entire problem clusters, preserving the dependence induced by the shared initial answer and critique. Thus, our estimates incorporate downstream revision-generation variability rather than relying on one stochastic outcome. We acknowledge that the initial solution and critique are each sampled once and that sensitivity across decoding temperatures is not evaluated.

Thank you for the authors' response and additional clarifications. However, the evaluation still relies on a single sampling temperature and does not include stronger LLM judges. Unlike random seeds, sampling temperature directly affects the model's output distribution and reasoning behavior, so repeated sampling at a fixed temperature does not address sensitivity to this important experimental setting. Moreover, the claim that the experiments isolate three factors is not fully supported by the experimental design. These unresolved issues continue to raise concerns about the paper's contribution and the validity of its conclusions. Since the rebuttal does not adequately address my main concerns, I will maintain my original rating.

Reviewer_FhcHOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies critique-based refinement by (i) ablating which fields of a fixed structured critique (verdict, location, diagnosis, full) drive wrong→correct vs correct→wrong flips, and (ii) proposing DiSEL-Consensus, which accepts a diagnosis-conditioned revision only when an independently sampled auxiliary revision agrees.

Experiments are on MATH-500 and AIME with Qwen/Llama solver–critic, self-refinement, and debate setups. Diagnosis-only is reported to recover most of full-critique w→c at lower token cost; Consensus is evaluated mainly as a safer acceptance policy under a small out-of-group corruption budget.

Critical strengths

Holding one critique record fixed while rendering field views is a reasonable ablation design, and jointly reporting w→c and c→w is the right two-sided framing for refinement risk.

Critical weaknesses
  1. Presentation / organization. The paper packs many protocols (solver–critic, self-refine, debate; Pointwise vs Consensus; ρ budgets; parser-bypass rules) into dense figures and appendix tables without a crisp takeaway. Core claims jump between “diagnosis is the efficient content unit” and “Consensus is a better selector,” but the writing never cleanly separates content ablation from selection policy, which makes the contribution feel scattered.
  2. Systems claim is weak and overstated. DiSEL-Consensus is essentially an agreement vote. At ρ=1% its net-ACC edge over Pointwise is not significant, it costs more tokens (~923 vs ~430), and under the same budget it abstains from all revisions in multiple slices (AIME cross-family; MATH-500 Llama-8B/70B in Table 1). Calling this an effective “Diagnose, Then Select” system overclaims a safer operating point.
Questions for authors
  1. For the nine-cell diagnosis-vs-full comparison, can you report a paired bootstrap CI on the token-normalized correction gap, and the fraction of cases where diagnosis text alone already contains an explicit location or “wrong” verdict?

  2. At ρ=1%, Consensus abstains entirely on some slices—what is the selection rate per slice, and does a mildly relaxed ρ (e.g. 2–5%) recover useful w→c there without exceeding Pointwise’s c→w?

  3. The presentation conflates content ablation and selection; can you restructure the main claims so each section has one testable hypothesis and one primary table?

  4. Does diagnosis remain the efficient unit for code or open-ended generation where “causal explanation” is less crisply graded?

3 comments
AuthorsOpenReview ↗

Response to Reviewer FhcH

Thank you for recognizing the value of the fixed-record ablation and two-sided W2C/C2W evaluation. Below, we respond to each of the weaknesses and questions you raised.

W1/Q3. Presentation and organization. We agree that the paper conflated two questions. We will reorganize it around H1 (Q1 in our manuscript): which field-restricted view of one fixed critique supports repair, and H1 (Q2 in our manuscript): which selector should accept a diagnosis-conditioned revision.

Q1. Diagnosis versus full critique (bootstrap CI) and explicit location/verdict rate.

Diagnosis versus full critique (bootstrap CI). We added a paired, problem-clustered bootstrap across the nine non-degenerate benchmark × solver × protocol cells:

Table R1a. Repair effectiveness.

QuantityEstimate95% CI
Diagnosis W2C10.30%[8.86, 11.80]
Full W2C10.99%[9.42, 12.64]
Absolute gap−0.69 pp[−1.58, +0.15]
W2C ratio98.81%[91.35, 111.53]

Table R1b. Token efficiency.

QuantityRatio95% CI
Feedback tokens (diagnosis/full)30.94%[30.33, 31.52]
Corrections/token (diagnosis/full)3.32×[3.06, 3.75]

Explicit location/verdict rate. In a conservative lexical audit of 1,263 wrong-cohort diagnoses, explicit verdict, location, or either appeared in 4.04%, 14.89%, and 18.21% (among 940 non-NONE: 5.43%, 20.00%, and 24.47%). These are lexical lower bounds. Thus, the result supports efficiency rather than semantic isolation: diagnosis often supports repair without restating verdict/location fields. We will use diagnosis-field view instead of isolated diagnosis.

Q2. Risk-budget sensitivity. Entries are selection/W2C/C2W percentages. T is marginal tokens/trial and is constant across ρ because the auxiliary is generated before the thresholded decision.

Table R2. DiSeL-Consensus by slice as ρ is relaxed. Cells: selection / W2C / C2W (%). Slices are benchmark × solver/critic: L70/Q235 = Llama-70B solver + Qwen-235B critic; L8/L70 = Llama-8B solver + Llama-70B critic; Q32/Q235 = Qwen-32B solver + Qwen-235B critic. PW C2W = Pointwise C2W at ρ=1%. Tok. = marginal Consensus tokens per trial, independent of ρ.

Sliceρ=1%ρ=2%ρ=5%PW C2WTok.
AIME L70/Q2350/0/00/0/02.52/.62/5.561.041,724
AIME L8/L7017.21/1.74/017.21/1.74/017.21/1.74/03.12936
AIME Q32/Q23530.06/8.28/.6633.32/8.58/1.6438.53/9.86/1.971.481,657
MATH L70/Q23511.24/27.17/.4317.41/29.21/1.5821.71/32.12/2.721.08468
MATH L8/L700/0/00/0/08.04/8.07/2.731.01551
MATH Q32/Q2356.61/25.24/.956.64/25.36/.966.64/25.36/.96.78201

ρ=2% does not revive either zero-selection slice. ρ=5% does, but C2W exceeds Pointwise in both (AIME: 5.56% vs 1.04% / MATH: 2.73% vs 1.01%), so no tested relaxation recovers corrections there while preserving Pointwise’s safety.

W2. Scope of the Consensus claim. These are equal-weight macros over the six submitted solver–critic slices (AIME and MATH-500 × three solver/critic configurations), recomputed from the frozen candidate pairs using the submitted out-of-group ρ=1% calibration. At ρ=1%, Pointwise obtains 9.63% W2C/1.42% C2W/+2.46 pp/431 tokens and Consensus 10.41%/.34%/+2.80 pp/923 tokens. Their net-accuracy difference is not statistically resolved. A parser-matched Always-revise control has +3.25 pp but 8.21% C2W. We will state that Pointwise is the cost-conscious option and Consensus the lower-corruption black-box option that may abstain completely.

Q3. One hypothesis, one table per section. Finally, each main section will have one hypothesis and one primary table. Protocol/parser details move to the appendix.

Q4. Beyond math. MMLU-Pro Computer Science and Physics extends diagnosis-only selection beyond math. Each table reports equal-weight macro averages over solver–critic, self-refinement, and peer debate at the predeclared ρ=1% risk budget. ΔAcc is relative to the frozen initial accuracy, and Tok. is marginal selector tokens per trial.

Table R3a. MMLU-Pro Computer Science.

Model pairSelectorW2CC2WΔAccTok.
Llama-70B/Qwen-235BAlways revise30.27%18.70%−1.11 pp0
DiSeL-GenLP1.33%1.14%−0.25 pp0
DiSeL-Pairwise4.90%1.14%+1.03 pp735
DiSeL-Pointwise11.79%1.32%+3.39 pp251
DiSeL-Consensus0.00%0.00%0.00 pp1,285
Llama-8B/Llama-70BAlways revise15.29%27.26%−7.96 pp0
DiSeL-GenLP1.12%1.21%−0.15 pp0
DiSeL-Pairwise0.77%0.95%−0.17 pp827
DiSeL-Pointwise1.47%0.90%+0.18 pp287
DiSeL-Consensus0.00%0.00%0.00 pp1,407
Qwen-32B/Qwen-235BAlways revise15.38%5.08%−0.70 pp0
DiSeL-GenLP5.19%1.18%+0.18 pp0
DiSeL-Pairwise6.72%1.08%+0.59 pp215
DiSeL-Pointwise7.57%1.02%+0.82 pp68
DiSeL-Consensus4.55%0.42%+0.64 pp1,221

Table R3b. MMLU-Pro Physics.

Model pairSelectorW2CC2WΔAccTok.
Llama-70B/Qwen-235BAlways revise27.02%18.58%−1.84 pp0
DiSeL-GenLP2.90%0.96%+0.46 pp0
DiSeL-Pairwise7.91%1.03%+2.25 pp812
DiSeL-Pointwise8.96%1.01%+2.65 pp277
DiSeL-Consensus0.00%0.00%0.00 pp1,605
Llama-8B/Llama-70BAlways revise12.76%22.72%−5.59 pp0
DiSeL-GenLP2.37%1.07%+0.59 pp0
DiSeL-Pairwise1.80%0.98%+0.36 pp670
DiSeL-Pointwise2.48%1.06%+0.65 pp238
DiSeL-Consensus0.00%0.00%0.00 pp1,893
Qwen-32B/Qwen-235BAlways revise18.54%2.19%+1.80 pp0
DiSeL-GenLP11.57%0.75%+1.62 pp0
DiSeL-Pairwise15.02%0.74%+2.30 pp227
DiSeL-Pointwise16.70%0.72%+2.63 pp65
DiSeL-Consensus5.12%0.14%+0.87 pp1,455

Across the two domains, Pointwise produces a positive accuracy gain in all six model-pair configurations. Strict Consensus is substantially more conservative as it abstains completely for both Llama configurations in both domains and produces positive, lower-corruption gains only for the Qwen configurations. The zero entries therefore indicate abstention under strict calibration, not evidence of zero population risk. These results reinforce our revised framing. Selector choice depends on solver capability, corruption tolerance, and inference budget.

Thank you again for your thoughtful review. We wanted to briefly follow up on our rebuttal above, where we addressed your questions and concerns. If you have a chance to take a look, we’d be happy to clarify anything that’s still unclear. Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.