- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Equilibrium of Thoughts, which iterates between a critic's concrete objection and a reasoner's revision, stopping on critic concession or a round cap. Experiments on eight benchmarks vary critics and initializers, reporting gains from cross-family critique and compatibility with Tree-of-Thoughts.
Critical strengths
- Experiments cover eight benchmarks, multiple reasoners, and several critic and initializer choices, with useful ablations and paired significance tests.
- The fixed-reasoner critic sweeps suggest that external feedback can substantially help difficult reasoning tasks. The prompt-only method also composes with existing search strategies.
Critical weaknesses
- The proposed method exhibits limited novelty, and the attribution of its contributions remains incomplete. External critique followed by revision has clear precedents, including critique-model supervision at test time (Xi et al., 2024, arXiv:2411.16579). Model-family difference is used as a proxy for decorrelation, but error correlations are not measured, and critic capability is not cleanly controlled.
- Critic concession does not certify correctness. The claimed Nash-equilibrium, preferred-extension, and Tarski fixed-point equivalences lack the necessary game, argumentation, and order-theoretic conditions. Moreover, Appendix D uses a marginal objection-soundness probability as though it were conditioned on an incorrect answer in the accuracy-drift derivation.
- The compute comparison is not fully fair. Despite the claim of matched budgets, Table 10 reports about 251K tokens for EoT (ToT, Opus) versus 60K for debate. The pooled AIME improvement over ToT alone is not statistically significant (Table 14, p=0.293). Cost, latency, and critic strength should be controlled before claiming superiority over debate.
- Algorithm 1 presents a single model for both attack and revision and returns the current answer on oscillation, while the text specifies cross-model attack and majority-vote cycle resolution. Table 3's nominal token budgets also differ markedly from measured usage in Table 10.
Questions for authors
- Can the authors provide valid conditions for the three equilibrium interpretations and correct the probability conditioning in Appendix D? How is a critic's 'no objection' distinguished from a missed error?
- Can the authors measure reasoner–critic error overlap and compare capability-matched same-family and cross-family critics, as well as a generic external-critique baseline?
- Can the authors compare methods at matched measured tokens, monetary cost, and latency, preferably with the same strong critic, and reconcile Algorithm 1 and the budget tables with the implementation?
I am willing to raise my score if you solve the above issues.
Gentle reminder: author–reviewer discussion period ending in a few days
Dear Reviewer GzLc,
We wanted to send a brief reminder that the author–reviewer discussion period concludes in a few days.
In our response, we re-ran the comparisons at a 30K per-call cap on 290 competition-math problems and reported measured tokens, list-price cost and sequential depth for every method; measured each critic's standalone accuracy and error overlap with the reasoner, including a capability-matched pair of critics; reported the ToT result as a composition with its cost; rewrote Algorithm 1 to match the implementation exactly; scoped the game-theoretic, argumentation and fixed-point interpretations; made the conditioning in Proposition 1 explicit; separated critic concession from the other stopping conditions; and added Xi et al. (2024) to the related work.
If these updates address your concerns, we would be glad to hear it, and we would be grateful if you would consider reassessing your rating accordingly. If there are any remaining questions or concerns, we would be happy to address them during the remaining discussion period.
Thank you again for the time and effort you put into reviewing our work.
Best regards,
Authors