- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors argue that using critique to refine model reasoning has two bottlenecks: (1) the feedback must have enough causal information to fix an error, and (2) the model should know whether to accept or reject the revision. They then propose DISEL and four variants, and evaluate them across three protocols, two model families, and two mathematical benchmarks. Results show that their method recovers 98.8% of the wrong-to-correct rate of a full critique while using only 30.9% of the tokens.
Critical strengths
This paper shows that targeted feedback, which is the diagnosis of incorrect steps, can be effective and can save token usage.
Critical weaknesses
-
The overall pipeline feels rule-based and hence includes a lot of design choices. For example, Eq (1) has a selector-specific threshold, which needs tuning, and the auxiliary answer also has specific rules to map to its actual vote. In lines 288-290, the authors also mention, "When repeat j is treated as the primary revision, we use the next repeat cyclically, j+ = 1 + (j mod 16)", which feels arbitrary. All these choices should be backed up either by a reference or an empirical result.
-
Revision selectors have four variations. There's no clear winner, so it is also not clear which one to use under which scenario.
-
The major weakness is the lack of baselines. At the very least, it should be compared with self-consistency [1], Best-of-N [2], simple test-time scaling (s1 [3]). Closely related, Magicore [4] is also an external baseline that should be compared with, since it also addresses the error identification issue with finer-grained refinement.
-
The evaluation benchmark only includes math with two datasets, which is too limited in both number and domain. To show better generalization, the author should consider including GPQA and MMLU-Pro.
References:
Questions for authors
-
What does it mean in Figure 1, the verdict shows "OBJECT" and the caption says "The verdict only objects to the answer"? I didn't get this. Shouldn't the verdict be correct or incorrect?
-
It makes sense that diagnosis is effective, but in practice the challenge is how to get that diagnosis. It's not clear in the paper how this is obtained. I only found in lines 193 to 195, "Each record is generated once, by the paired critic, the solver itself, or a same-size peer depending on the protocol, and all four views are rendered from it" but this wasn't clear. How are the four views rendered?
3 comments
Thank you again for your thoughtful review. We wanted to briefly follow up on our rebuttal above, where we addressed your questions and concerns. If you have a chance to take a look, we’d be happy to clarify anything that’s still unclear. Thank you for your time.
Thank the authors for their detailed responses. While they have partially addressed my comments, I still think the fact that there's no clear winner to choose at test time is weakening the contribution. Therefore, I decided to keep my score.
Response to Reviewer 254R
Thank you for identifying the need to justify design choices, give deployment guidance, add test-time-scaling baselines, and broaden evaluation. We address each point in turn below.
W1. Calibration and pairing. Eq. (1)’s threshold is not tuned on test accuracy. For each selector/protocol it maximizes training-fold W2C subject to C2W≤ρ, using leave-one-year-out AIME or leave-one-subject-out MATH-500 calibration. The held-out group never selects its threshold. Cyclic pairing makes every one of 16 frozen revisions a primary once and an auxiliary once, without self-pairing or labels. Recomputing all nonzero offsets 1–15 at ρ=1% gives:
Table R1. Sensitivity of DiSeL-Consensus to the auxiliary offset (six-slice macro, ρ = 1%).
Thus +1 is balanced and label-independent, not uniquely favorable or theoretically optimal.
W2. Which selector to use. We agree that no selector dominates, and we will add the following deployment guidance:
Table R2. Selector guidance (six-slice macro, ρ = 1%).
Consensus may abstain completely under strict calibration. Pointwise is the default when tokens matter.
W3. Test-time-scaling baselines. We added self-consistency and Best-of-N on Qwen3-32B / MATH-500. Tokens count all generation and judging per problem.
Table R3. Self-consistency and Best-of-N (Qwen3-32B, MATH-500).
At a comparable budget, DiSeL-Consensus (4,590 tokens) is more accurate than self-consistency with N = 4 (89.80% at 3,651 tokens) and every Best-of-N point up to N = 4. Matching the best Best-of-N point (N = 8) would require about 3× more tokens, and self-consistency needs N = 8 (7,226 tokens) to exceed it. Unlike both, DiSeL preserves the initial answer when evidence is weak and explicitly controls the W2C–C2W trade-off. The two approaches are also complementary, since DiSeL can be applied to a self-consistency output.
Regarding s1, its budget-forcing technique can be applied at test time without additional training. However, due to the limited rebuttal period, our empirical comparisons are limited to self-consistency and Best-of-N. We will discuss s1 as an important test-time-scaling baseline and identify its evaluation as future work. MAgICoRe also targets over-correction and error localization, but relies on external process/outcome reward models and iterative multi-agent refinement, whereas we study which content of a single critique is useful. We will discuss MAgICoRe in related work rather than claim an empirical comparison.
W4. Additional domain. We added MMLU-Pro Computer Science:
Table R4. Diagnosis-conditioned selection on MMLU-Pro Computer Science.
Pointwise retains corrections at about 1% corruption outside mathematics, while Consensus remains conservative and abstains entirely for the Llama-70B pair. We have not yet evaluated GPQA and will note this as a limitation.
Q1. Meaning of
OBJECTin Figure 1.OBJECTmeans that the critic rejects the response as incorrect. We will relabel Figure 1 to say exactly this.Q2. How diagnosis and the four views are obtained. For each frozen initial solution, the critic (or the solver itself, or a same-size peer, depending on the protocol) is called once to produce one tagged record containing a verdict, an error location, and a diagnosis. The verdict-, location-, and diagnosis-field views are deterministic projections of that record, and the full critique renders all of its fields. All field comparisons therefore reuse the same underlying feedback event rather than regenerating different critiques. The exact prompts are in Appendix B, and we will move a short description of this procedure into the main text.