- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes One Model, Many Minds(OMMM) where test-time compute is spent on different reasoning strategies instead of repeatedly sampling the same reasoning mode. Basically the idea is ordinary self-consistency produces highly correlated errors while forcing different strategies can reduce this correlation, improve coverage and provide a useful agreement-based confidence signal. Overall I like the main idea of the paper but the paper still needs stronger theoretical justification for connecting error decorrelation to ensemble performance and clearer evidence that the gains translate to final prediction accuracy rather than mainly coverage.
Critical strengths
-
The main idea is simple and the paper focuses on error correlation and tries to explain why repeated sampling can saturate instead of only reporting accuracy gains. I think this gives the work a clearer and more useful motivation.
-
The experimental analysis is fairly thorough. The appendix includes useful checks on strategy compliance, alternative strategy sets, model-size effects, cross-model comparisons etc. I actually liked their cross-strategy agreement idea. The paper does not only use diversity to improve accuracy but also shows that agreement between different strategies can give a meaningful signal of confidence.
Critical weaknesses
-
The way the paper connects pairwise error correlation to coverage (lines 193–204) is not convincing to me. The paper reduces the whole error structure to one number, the average pairwise correlation and then uses that to estimate an effective ensemble size. But Coverage@K depends on how all strategies fail together not just on pairwise relationships, right? IMO two ensembles could have the same average accuracy and the same but very different patterns of joint failure and therefore different coverage.
-
OMMM often finds the correct answer with at least one strategy but the vote does not always recover it. For example, on Sonnet AIME-24, Coverage@5 is 96.7% while majority-vote accuracy is only 86.7% and on GPQA it is 47.0% vs 35.9% (Table 2). So the strategies contain substantially more correct solutions than the final vote captures. It seems like the strategies are finding useful additional solutions but somehow the current aggregation step is not able to make full use of them. Also Table 2 does not report Coverage@5. Could the authors clarify where the 96.7% and 47.0% values come from?
-
D-FUSEr [1] already studies the problem of concentrated errors in test-time aggregation and they also argue that diverse failures can improve majority voting. Even though OMMM uses a different mechanism forcing different reasoning strategies at inference time but the broader motivation is somewhat similar to me. I think the paper should be clearer that the main novelty is the strategy-forcing mechanism rather than the general idea that error diversity matters for aggregation.
-
formula is heuristic rather than theoretically justified for coverage. If I understand correctly the expression comes from variance reduction for equicorrelated variables. But the paper then plugs this effective sample size into the independent-coverage formula . I dont see a derivation showing that this should hold for binary success/failure events with correlated strategies. Appendix J shows that the approximation fits the experiments well but I think it should be presented as an empirical approximation rather than a general theoretical result.
Inconsistencies:
-
The main text (296–297) says strategy forcing “decorrelates significantly more than prompt variation” but Appendix D shows this is not true for every reasoner. For Qwen3-235B, difference is not significant . The wording should be more qualified.
-
Table 3 caption says the high-agreement AIME slices are small but this does not match all rows. For Sonnet, 60% of 30 AIME problems is about 18 problems and for DeepSeek, 50% is about 15. Also Table 8 caption says that "the ensemble benefit is larger for more capable models" but the GPQA results do not really follow that pattern. OMMM improves over self-consistency by +5.6 points for Qwen3-32B and +2.5 for Qwen3-Next-80B but drops by 3.5 points for Qwen3-235B. The paragraph below the table gives a more nuanced explanation so I think the caption should be revised to avoid suggesting a general trend that is not consistent across benchmarks.
-
Table 7 text says agreement is better calibrated on every hard-benchmark cell but Sonnet AIME-24 is clearly worse for agreement 0.230 vs 0.113 and DeepSeek AIME-24 is a tie 0.107 vs 0.107.
Questions for authors
- Authors need to clarify the theoretical motivation behind . In particular why should the usual effective-sample-size expression based on pairwise correlation be expected to predict Coverage@K for correlated binary outcomes?
I am not asking the authors to run more experiments, I think the paper already includes a fairly substantial empirical evaluation. My hesitation is mainly on the theoretical side+some inconsistencies i mentioned. Particularly I am not fully convinced by the way the paper connects pairwise error correlation, effective ensemble size, coverage and final majority-vote accuracy. So I am currently leaning slightly negative.
5 comments
I went through the revised paper and rebuttal. Paper seems much better now but I still have a few reservations, so I am not really able to increase my rating.
Again on novelty I understand the distinction pretty much now. But the AM-GM argument still feels quite general and not specific to reasoning strategies. So while I do see novelty in the strategy forcing mechanism itself.. I think the broader conceptual or theoretical framing is overstated. But tbh I am not opposed to acceptance on this point alone.
My remaining reservation is more about what is left after your correlation clarification. For pairwise correlation is mainly an empirical diagnostic rather than a theoretical guarantee for coverage or majority vote acc. And since plain OMMM is mostly comparable to self-consistency while the clearer gain appears with adjudication, I see the practical contribution more as producing a better candidate pool for a stronger aggregator to exploit. Its useful but it is a narrower story than your original decorrelation/ensemble framing.
One other thing I am still unsure about is that a lot of the new evidence is now at the 30K tok cap while the main experiments were at 4096 tok. The correlation ordering also changes for some models at the larger cap. So I am not fully sure how robust the main story is to the reasoning budget or how much of the new result depends on this much larger inference regime.
Again I really appreciate the authors effort and dedication in addressing the comments. I hope my rating does not discourage you in any way. I still have the concerns mentioned above but if the other reviewers are convinced and support acceptance, I have no objection from my side. Wishing you all the best with the submission.
We thank the reviewer for reading the revision carefully and for the encouraging and constructive feedback. We agree that the paper should make a narrower claim than our original ensemble framing, and the revision does so; we address the three remaining points below.
Generality of the coverage bound. We agree with the reviewer that the AM–GM bound is general, and we do not view the bound itself as OMMM's conceptual novelty. It is a general condition, not an explanation of why reasoning strategies produce complementary candidates: for a given problem, candidates with different success probabilities reach higher coverage than i.i.d. candidates with the same mean success probability. Whether a particular diversity source produces such profiles is an empirical question, and our contribution is to study reasoning strategies as a controllable source of this diversity within one model. The answer depends on the model and the source. At the 30K cap, strategy forcing lowers Sonnet's mean pairwise error correlation from 0.62 (temperature sweep) to 0.51, against 0.46 for the five-model reference, and gives the highest Coverage@5 among Sonnet's single-model sources (95.9%, vs. 94.6% for the temperature sweep), whereas prompt rephrasing, another way to condition generations differently, raises the correlation to 0.68. The bound therefore gives a general perspective on coverage, while our experiments test whether strategy forcing produces complementary candidates; §3.2 presents the bound in this role, and the revised appendix makes explicit that the effect of a diversity source must be established empirically.
What remains after the correlation clarification. We agree with the reviewer that, for K > 2, pairwise correlation is an empirical diagnostic rather than a guarantee of coverage or majority-vote accuracy, and we have narrowed the practical claim to the one the reviewer describes: the revised conclusion presents OMMM as a way to build a complementary candidate pool from one model, and our experiments show that two different aggregators can exploit this pool to improve accuracy. With the same adjudication applied to both methods, OMMM's candidates gain 3.1–6.2 points from adjudication, against 0–2.1 for self-consistency at comparable coverage, a pooled advantage of +2.6 points (95% CI [+0.7, +4.7]) on 290 competition-math problems over three models, at 1.03–1.28× the tokens. The benefit is not limited to adjudication. A second training-free aggregator, cross-model agreement selection, uses three jurors from model families that never generate in OMMM (Kimi-K2.5, gpt-oss-120b and GLM-5.3-Flash, one answer each) and returns the pool answer most jurors agree with, breaking ties by pool frequency and falling back to the most frequent pool answer when no juror's answer is in the pool. Applied identically to both pools at the original 4,096-token cap, it gives OMMM +4.9 points (95% CI [+2.7, +7.0]) over self-consistency on AIME-24, AIME-25 and OlympiadBench.
Robustness to the reasoning budget. We agree with the reviewer that the 30K results should not be treated as interchangeable with the original 4,096-token experiments. The two settings differ mainly in how often generations reach the cap: on the same 290 competition-math problems, the total (input plus output) token count reaches the 4,096-token cap for 11% (Sonnet), 30% (DeepSeek) and 62% (Qwen) of strategy generations, against at most 1.2% at 30K; these are upper bounds on output-cap hits rather than confirmed truncations. The revised paper reports the two settings separately and labels each result with its cap. OMMM helps in both regimes, through different measurements. At 4K, its Coverage@5 exceeds self-consistency's by +7.0 points (95% CI [+4.7, +9.2]) pooled over the three models (+16.2 for Qwen), and the cross-model aggregator turns this into the +4.9-point accuracy gain above; at 30K, where both pools reach about 88% coverage, the gain appears through adjudication (+2.6). The correlation result depends on the model and the budget, and we report it that way: strategy forcing has the lowest mean pairwise error correlation (point estimate) among single-model sources for all four reasoners at 4K and for Sonnet and Qwen at 30K, six of the seven model–budget combinations, but not for DeepSeek at 30K; the five-model reference stays lower throughout, except for Kimi-K2-Thinking at 4K, where the two are statistically indistinguishable. The revised §5.1 states this ordering per model and cap rather than as a general claim.
We appreciate the reviewer's point that these results support a more focused contribution than our original framing. The revision presents OMMM as a strategy-conditioned method for generating candidates, evaluates their coverage and error dependence across budgets, and measures how stronger aggregators exploit the pool. We are grateful for the reviewer's engagement throughout the discussion.
Gentle reminder: author–reviewer discussion period ending in a few days
Dear Reviewer LvWd,
We wanted to send a brief reminder that the author–reviewer discussion period concludes in a few days.
In our responses, we measured coverage, majority voting and adjudication on the same candidates (with the same adjudication, OMMM improves over self-consistency by 2.6 points [+0.7, +4.7]); presented K_eff as a descriptive summary only; and, in the follow-up, described D-FUSEr in full, explained how OMMM targets a different quantity (which problems each candidate solves, rather than which wrong answers a sampler produces), and replaced the "strictly improves the ensemble" statement in §3.2 with a precise coverage guarantee and the additional condition majority voting requires.
If these updates address your concerns, we would be glad to hear it, and we would be grateful if you would consider reassessing your rating accordingly. If there are any remaining questions or concerns, we would be happy to address them during the remaining discussion period.
Thank you again for the time and effort you put into reviewing our work.
Best regards,
Authors