- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces a new method for evaluating the reasoning ability of models. The authors argue that simply evaluating whether the final answer is correct cannot fully reflect a model's reasoning ability. To address this limitation, the paper proposes an evaluation framework involving two agents, A and B. Specifically, the framework first asks the model to generate a correct reasoning process, referred to as the "Thesis." Then, Agent B uses a predefined prompt to challenge the answer in the Thesis and generate an "Antithesis." Finally, the model is asked to combine the Thesis and Antithesis to determine whether it can still produce the correct answer. Based on this framework, the authors find that although different models may achieve similar accuracy on the original tasks, their performance differences become more pronounced under the proposed evaluation method.
Critical strengths
-
The problem addressed in this paper is indeed important. Evaluating only whether the final answer is correct cannot fully reflect a model's reasoning ability. Therefore, developing new methods to assess reasoning ability is a meaningful research direction.
-
The paper is generally well-written and clear. The authors provide clear descriptions of the proposed evaluation methodology and evaluation procedure, as well as a clear analysis of the experimental findings.
Critical weaknesses
-
I have some concerns about the evaluation setup in this paper. I do not think the scores obtained using this evaluation method accurately reflect a model's reasoning ability. In the prompt given to Agent B, the authors instruct the model to always provide a counterargument, even when the original answer is correct. I do not quite understand the purpose of this design choice. In particular, in the o3 model's Antithesis response shown in Figure 3, the model goes beyond the scope of the mathematical problem and instead questions the assumptions in the problem statement. As a result, it produces an answer that differs from the expected answer. However, I do not think this should be considered an "incorrect" response, since the model is simply following Agent B's prompt. Asking a model to combine such a response with the Thesis response and still arrive at the correct answer seems to test its ability to resist distracting information rather than its reasoning ability. The key challenge is whether the model should answer the mathematical question as posed or step outside the problem and question its assumptions. This is different from what is usually understood as "reasoning" ability.
-
I think several of the “key advantages” claimed by the authors in the Introduction are somewhat overstated. In point (1), the authors claim that “SIEV requires no prompt engineering.” However, they actually use specific prompts for Agents A and B, so I do not understand why this should be considered an advantage. Similarly, regarding point (4), “Natural compatibility with multi-agent systems,” I think this is a fairly common property. It is unclear to me what makes this particularly challenging or advantageous compared with other benchmarks.
-
The range of models evaluated by the authors is also quite limited. The evaluation does not include any models from the Claude or Gemini families, and the GPT models tested are not the latest versions (even considering the submission date). In addition, the ordering of the models in Table 1 seems somewhat hard to read.
Questions for authors
Please focus on addressing the concerns raised in the Weaknesses section, particularly clarifying what abilities the proposed evaluation method actually measures and what its specific advantages are.
3 comments
Thank you very much for the authors' response and for providing extensive additional experiments.
Regarding W1, I appreciate the authors' clarification, but I believe my main concern remains unresolved. The authors state that "Such an objection is still an argument, and a model that cannot tell a right assumption from a wrong one has not shown genuine reasoning." However, in my view, Agent B's response cannot really be considered an "argument." Instead, it changes the original problem to some extent. Therefore, I believe a more rigorous approach would be to explicitly state in the prompt that "You should never question this part of the problem statement," which would be more consistent with the authors' explanation. However, the authors did not include such an instruction, so I still believe that the proposed method lacks sufficient rigor.
Furthermore, the authors' response regarding the advantages has not fully convinced me either. I do not believe that these two points constitute advantages significant enough to warrant such strong emphasis.
Thank you for engaging with our response, and for proposing a concrete way to make the protocol more rigorous.
W1. Rigor: an explicit instruction in Agent B's prompt not to question the problem statement. Following your suggestion, we ran SIEV again with this instruction added to Agent B's prompt. We applied it to the whole problem statement and made it explicit. The sentence we added is: "You should never question the problem statement: take its premises, its numbers and what it asks as given, and argue only within the problem as posed."
Everything else is exactly as in the paper:
- each model's thesis (its first answer) is the same as in the original runs;
- Agent B is still asked to oppose the thesis.
We ran this for 16 models, three runs each:
- eight recent models: Claude-Haiku-4.5, Claude-Sonnet-5, Claude-Sonnet-5.5, Gemini-3.8-Flash, Gemini-4-Argon, GPT-5.3-Codex, GPT-5.6-Luna and MAI-Code-1.1-Flash;
- eight open-weight models: DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, Ministral-8B, Phi-4, Qwen2.5-72B, Qwen3-8B and Qwen3-32B.
All 16 models answered the first 300 questions of the GSM8K test set and 300 MMLU questions that cover all 57 subjects.
The table gives the Synthesis Score (accuracy after the dialectical process) with the original prompt and with your instruction. For brevity, it shows 8 of the 16 models; Table 9 in the new Appendix L gives all 16. Each value is the mean over the three runs ± the standard deviation. Diff = (with your instruction) − (with the original prompt).
| Model | GSM8K: original prompt | GSM8K: with your instruction | Diff | MMLU: original prompt | MMLU: with your instruction | Diff |
|---|---|---|---|---|---|---|
| Claude-Haiku-4.5 | 84.8 ± 0.3 | 85.1 ± 0.8 | +0.3 | 60.5 ± 2.5 | 55.6 ± 1.5 | −4.9 |
| Claude-Sonnet-5 | 96.9 ± 0.3 | 97.4 ± 0.2 | +0.5 | 90.8 ± 1.2 | 90.0 ± 0.7 | −0.8 |
| Claude-Sonnet-5.5 | 97.3 ± 0.0 | 97.5 ± 0.3 | +0.2 | 92.4 ± 1.0 | 92.6 ± 1.1 | +0.2 |
| Gemini-3.8-Flash | 98.7 ± 0.3 | 98.3 ± 0.0 | −0.4 | 95.5 ± 0.4 | 96.0 ± 0.3 | +0.5 |
| Gemini-4-Argon | 99.7 ± 0.3 | 99.7 ± 0.3 | 0.0 | 95.9 ± 0.4 | 95.7 ± 0.3 | −0.2 |
| GPT-5.6-Luna | 88.1 ± 0.3 | 90.4 ± 1.0 | +2.3 | 83.2 ± 0.8 | 84.6 ± 0.9 | +1.4 |
| DeepSeek-R1 | 89.3 ± 1.7 | 88.3 ± 1.5 | −1.0 | 74.9 ± 1.1 | 74.4 ± 1.7 | −0.5 |
| DeepSeek-V3 | 85.6 ± 1.6 | 85.0 ± 0.8 | −0.6 | 74.8 ± 1.2 | 75.9 ± 0.6 | +1.1 |
On average over all 32 model–benchmark pairs (16 models, two benchmarks), differs by only 1.3 points between the two prompts. The Dialectic Score (DS) also changes little: by 1.3 points on average. Appendix L (Table 9) gives all metrics.
With your instruction, SIEV's results stay very close to the results with the original prompt, and the two versions of the protocol lead to the same conclusions. The revised paper reports this evaluation in Appendix L, and §4.2 refers to it.
So SIEV's results hold when Agent B is told never to question the problem statement.
W2. The advantages listed in the Introduction. Your comment is about two items in the list of key advantages in the Introduction (§1): advantage (1), "Benchmark- and model-agnostic", and advantage (4), "Natural compatibility with multi-agent systems".
- Advantage (1). We agree that its wording put the emphasis on the prompts, which is not its point. Its point is that SIEV is benchmark- and model-agnostic: it can be applied to any existing benchmark and to any model as they are, without changing the benchmark's questions. This is what distinguishes SIEV from benchmark-dependent evaluations such as GSM-Symbolic and GSM-Plus, which build new, modified versions of the GSM8K questions. SIEV uses each benchmark's original questions, with the same fixed prompts for every benchmark and model (Appendix B). In the revised paper, advantage (1) states this point directly: "SIEV applies to any existing benchmark and any model as they are, including benchmarks thought saturated: no benchmark is rewritten, no model is changed, and nothing is tuned per benchmark or model (Appendix B). On GSM8K and MMLU, it turns even saturated benchmarks into diagnostics of reasoning behavior."
- Advantage (4), "Natural compatibility with multi-agent systems", is removed from the list. The multi-agent setting remains the motivation for the cross-model experiments (Appendix E).
Your follow-up comments helped us improve the paper further: the revision now includes the evaluation with your instruction (Appendix L, Table 9, referenced in §4.2), and its list of advantages in the Introduction is more precise, so that it does not overstate them (advantage (1) rewritten, advantage (4) removed).
With your remaining points now addressed, we would appreciate it if you would raise your rating score. Thank you for your time.
Thank you for your review. We share your starting point that final-answer correctness cannot fully reflect a model's reasoning ability; SIEV is built on it.
The protocol, stated precisely (§3, Appendix B).
W1 and your question: what SIEV measures. You write: "This is different from what is usually understood as 'reasoning' ability." That is SIEV's point: the usual view, which judges reasoning by a correct answer, is not the complete picture. A student who has memorized a solution scores the same as one who understands the problem; challenged by a teacher with a contradicting solution, the one who memorized may give up the right answer, while the one who understands can say why it is invalid. SIEV shows the same split in models: on GSM8K, GPT-5-chat's thesis accuracy is 96.4 and its synthesis accuracy 56.2, while Gemini-4-Argon's are both 98.4 (Table 1). So the claim is directional, as the paper states: a low score is evidence of fragile reasoning behind a correct answer; a high score is not necessarily a certificate of reasoning ability.
On the key challenge you describe: SIEV does not ask a model to step outside the problem. Agent B is asked to oppose even a correct thesis, so that every correct answer meets a substantive alternative (the controlled condition); how it opposes is up to the model, and it can simply solve the same problem another way, which is why the antithesis is sometimes right where the thesis was wrong (Figure 4). Figure 3 shows one case, in which O3 chose to question the problem's assumptions. You are right that this is not a failure: its × marks only that its answer differs from the benchmark's, which makes it an opposing antithesis (i.e., a good antithesis); §3 now says both. Such an objection is still an argument, and a model that cannot tell a right assumption from a wrong one has not shown genuine reasoning; O3's synthesis keeps the correct answer, noting that the objections step outside the problem as posed. Two facts in the paper show that the synthesis drop is a reasoning failure, not a response to distraction:
W2. Two of the claimed advantages. First, as the introduction now states: SIEV is not a new benchmark but a structured way to evaluate models on the benchmarks that already exist, with no new data and no rewriting.
W3. Model range and Table 1. Following your comment, we extended the model set in Table 1 to 34 models, adding Claude-Haiku-4.5, Claude-Sonnet-5, Claude-Sonnet-5.5, Gemini-3.8-Flash, Gemini-4-Argon, GPT-5.3-Codex, GPT-5.4, GPT-5.4-mini, GPT-5.5, GPT-5.6-Luna and MAI-Code-1.1-Flash, all under the same protocol. The pattern holds: on GSM8K their thesis accuracy lies between 96.4 and 98.4, while their synthesis accuracy ranges from 84.0 to 98.4 (56.5 to 95.3 on MMLU). Table 1 is now one list sorted by DS on GSM8K.
Thank you for your comments, which helped us improve the paper; the revision now includes, in §3, why the antithesis opposes even a correct thesis and what ✓ and × mark; §1's revised advantages list; and Table 1 and §4.1 with recent Claude, Gemini and GPT models.
With all of your points now addressed in the revision, we kindly ask you to raise your rating to 4, and your soundness and informativeness scores.