acceptodds
Under review as a conference paper at ICLR 2027

Evaluating LLM Reasoning Beyond Correctness

Abstract

What does it truly mean for a language model to "reason"? Current evaluations reward models' correct standalone answers—but correctness alone reveals little about the process that produced them. We argue that reasoning should be understood not as a static chain of steps but as a dynamic trajectory in which ideas interact, clash, and evolve into integrated insights. Building on the philosophical tradition of dialectics, we introduce SIEV, a structured evaluation framework that assesses reasoning through explicit thesis–antithesis–synthesis interactions. SIEV produces interpretable trajectories that highlight key properties of reasoning—robustness to challenge, adaptability under conflict, and synthesis across competing viewpoints—dimensions that conventional correctness-based metrics cannot capture. Empirical results on GSM and MMLU demonstrate substantial gaps in the reasoning abilities of state-of-the-art models: for example, GPT-5-chat loses more than 40 points out of 100 on GSM when evaluated through SIEV's process-oriented lens (a jury of frontier models confirms that these drops reflect reasoning that is not genuine). By shifting focus from what answer a model gives to how it arrives there, SIEV enables a more transparent and principled distinction between structured reasoning and surface-level pattern generation offering a clearer foundation for assessing and understanding the reasoning capabilities of LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM evaluation, reasoning, dialectical evaluation, thesis-antithesis-synthesis, reasoning robustness, benchmark saturation, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uZMuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a new method for evaluating the reasoning ability of models. The authors argue that simply evaluating whether the final answer is correct cannot fully reflect a model's reasoning ability. To address this limitation, the paper proposes an evaluation framework involving two agents, A and B. Specifically, the framework first asks the model to generate a correct reasoning process, referred to as the "Thesis." Then, Agent B uses a predefined prompt to challenge the answer in the Thesis and generate an "Antithesis." Finally, the model is asked to combine the Thesis and Antithesis to determine whether it can still produce the correct answer. Based on this framework, the authors find that although different models may achieve similar accuracy on the original tasks, their performance differences become more pronounced under the proposed evaluation method.

Critical strengths
  1. The problem addressed in this paper is indeed important. Evaluating only whether the final answer is correct cannot fully reflect a model's reasoning ability. Therefore, developing new methods to assess reasoning ability is a meaningful research direction.

  2. The paper is generally well-written and clear. The authors provide clear descriptions of the proposed evaluation methodology and evaluation procedure, as well as a clear analysis of the experimental findings.

Critical weaknesses
  1. I have some concerns about the evaluation setup in this paper. I do not think the scores obtained using this evaluation method accurately reflect a model's reasoning ability. In the prompt given to Agent B, the authors instruct the model to always provide a counterargument, even when the original answer is correct. I do not quite understand the purpose of this design choice. In particular, in the o3 model's Antithesis response shown in Figure 3, the model goes beyond the scope of the mathematical problem and instead questions the assumptions in the problem statement. As a result, it produces an answer that differs from the expected answer. However, I do not think this should be considered an "incorrect" response, since the model is simply following Agent B's prompt. Asking a model to combine such a response with the Thesis response and still arrive at the correct answer seems to test its ability to resist distracting information rather than its reasoning ability. The key challenge is whether the model should answer the mathematical question as posed or step outside the problem and question its assumptions. This is different from what is usually understood as "reasoning" ability.

  2. I think several of the “key advantages” claimed by the authors in the Introduction are somewhat overstated. In point (1), the authors claim that “SIEV requires no prompt engineering.” However, they actually use specific prompts for Agents A and B, so I do not understand why this should be considered an advantage. Similarly, regarding point (4), “Natural compatibility with multi-agent systems,” I think this is a fairly common property. It is unclear to me what makes this particularly challenging or advantageous compared with other benchmarks.

  3. The range of models evaluated by the authors is also quite limited. The evaluation does not include any models from the Claude or Gemini families, and the GPT models tested are not the latest versions (even considering the submission date). In addition, the ordering of the models in Table 1 seems somewhat hard to read.

Questions for authors

Please focus on addressing the concerns raised in the Weaknesses section, particularly clarifying what abilities the proposed evaluation method actually measures and what its specific advantages are.

3 comments

Thank you for your review. We share your starting point that final-answer correctness cannot fully reflect a model's reasoning ability; SIEV is built on it.

The protocol, stated precisely (§3, Appendix B).

  • Thesis: the model's own first answer and reasoning, right or wrong.
  • Antithesis: Agent B, the same model in a separate context, is asked to oppose the thesis with a solution of its own. Agent A never sees Agent B's prompt, only its response.
  • Synthesis: Agent A is asked to update its answer given that response, not to combine the two: it may keep its answer, take up the antithesis's, or revise further.
  • Scores: pSp_S scores the synthesis's final answer alone. The antithesis is not graded on whether its own answer is right; what matters is that it opposes the thesis (OC); DS combines pSp_S with OC.

W1 and your question: what SIEV measures. You write: "This is different from what is usually understood as 'reasoning' ability." That is SIEV's point: the usual view, which judges reasoning by a correct answer, is not the complete picture. A student who has memorized a solution scores the same as one who understands the problem; challenged by a teacher with a contradicting solution, the one who memorized may give up the right answer, while the one who understands can say why it is invalid. SIEV shows the same split in models: on GSM8K, GPT-5-chat's thesis accuracy is 96.4 and its synthesis accuracy 56.2, while Gemini-4-Argon's are both 98.4 (Table 1). So the claim is directional, as the paper states: a low score is evidence of fragile reasoning behind a correct answer; a high score is not necessarily a certificate of reasoning ability.

On the key challenge you describe: SIEV does not ask a model to step outside the problem. Agent B is asked to oppose even a correct thesis, so that every correct answer meets a substantive alternative (the controlled condition); how it opposes is up to the model, and it can simply solve the same problem another way, which is why the antithesis is sometimes right where the thesis was wrong (Figure 4). Figure 3 shows one case, in which O3 chose to question the problem's assumptions. You are right that this is not a failure: its ×\times marks only that its answer differs from the benchmark's, which makes it an opposing antithesis (i.e., a good antithesis); §3 now says both. Such an objection is still an argument, and a model that cannot tell a right assumption from a wrong one has not shown genuine reasoning; O3's synthesis keeps the correct answer, noting that the objections step outside the problem as posed. Two facts in the paper show that the synthesis drop is a reasoning failure, not a response to distraction:

  • Not distracting information (Table 2): a second turn with no argument (a request to check again, bare doubt, a confident claim of error, or text as long and critical-sounding as an objection that names no flaw) moves fewer correct answers than SIEV's antithesis.
  • Lost correct answers are not reasoned: a jury of three frontier models, blind to the gold answer and the model, judged 99.7% of the 91,573 cases in Table 1 where a correct thesis became a wrong synthesis non-genuine (§4.2, App. I).

W2. Two of the claimed advantages. First, as the introduction now states: SIEV is not a new benchmark but a structured way to evaluate models on the benchmarks that already exist, with no new data and no rewriting.

  • (1) "No prompt engineering." Any LLM call needs a prompt; prompt engineering is optimizing a prompt and adding nuance to it. SIEV's prompts are minimal fixed templates, not tuned per model or benchmark: a role line, the answer format (which asks for the reasoning), the instruction to oppose, and one follow-up sentence for the synthesis (Appendix B).
  • (4) Multi-agent fit. In a multi-agent setting a model meets another model's answer, another answer of its own, or a conflicting generation; SIEV evaluates exactly that, as Appendix E's cross-model dialectics show.

W3. Model range and Table 1. Following your comment, we extended the model set in Table 1 to 34 models, adding Claude-Haiku-4.5, Claude-Sonnet-5, Claude-Sonnet-5.5, Gemini-3.8-Flash, Gemini-4-Argon, GPT-5.3-Codex, GPT-5.4, GPT-5.4-mini, GPT-5.5, GPT-5.6-Luna and MAI-Code-1.1-Flash, all under the same protocol. The pattern holds: on GSM8K their thesis accuracy lies between 96.4 and 98.4, while their synthesis accuracy ranges from 84.0 to 98.4 (56.5 to 95.3 on MMLU). Table 1 is now one list sorted by DS on GSM8K.

Thank you for your comments, which helped us improve the paper; the revision now includes, in §3, why the antithesis opposes even a correct thesis and what ✓\checkmark and ×\times mark; §1's revised advantages list; and Table 1 and §4.1 with recent Claude, Gemini and GPT models.

With all of your points now addressed in the revision, we kindly ask you to raise your rating to 4, and your soundness and informativeness scores.

Thank you very much for the authors' response and for providing extensive additional experiments.

Regarding W1, I appreciate the authors' clarification, but I believe my main concern remains unresolved. The authors state that "Such an objection is still an argument, and a model that cannot tell a right assumption from a wrong one has not shown genuine reasoning." However, in my view, Agent B's response cannot really be considered an "argument." Instead, it changes the original problem to some extent. Therefore, I believe a more rigorous approach would be to explicitly state in the prompt that "You should never question this part of the problem statement," which would be more consistent with the authors' explanation. However, the authors did not include such an instruction, so I still believe that the proposed method lacks sufficient rigor.

Furthermore, the authors' response regarding the advantages has not fully convinced me either. I do not believe that these two points constitute advantages significant enough to warrant such strong emphasis.

Thank you for engaging with our response, and for proposing a concrete way to make the protocol more rigorous.

W1. Rigor: an explicit instruction in Agent B's prompt not to question the problem statement. Following your suggestion, we ran SIEV again with this instruction added to Agent B's prompt. We applied it to the whole problem statement and made it explicit. The sentence we added is: "You should never question the problem statement: take its premises, its numbers and what it asks as given, and argue only within the problem as posed."

Everything else is exactly as in the paper:

  • each model's thesis (its first answer) is the same as in the original runs;
  • Agent B is still asked to oppose the thesis.

We ran this for 16 models, three runs each:

  • eight recent models: Claude-Haiku-4.5, Claude-Sonnet-5, Claude-Sonnet-5.5, Gemini-3.8-Flash, Gemini-4-Argon, GPT-5.3-Codex, GPT-5.6-Luna and MAI-Code-1.1-Flash;
  • eight open-weight models: DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, Ministral-8B, Phi-4, Qwen2.5-72B, Qwen3-8B and Qwen3-32B.

All 16 models answered the first 300 questions of the GSM8K test set and 300 MMLU questions that cover all 57 subjects.

The table gives the Synthesis Score pSp_S (accuracy after the dialectical process) with the original prompt and with your instruction. For brevity, it shows 8 of the 16 models; Table 9 in the new Appendix L gives all 16. Each value is the mean over the three runs ± the standard deviation. Diff = (with your instruction) − (with the original prompt).

ModelGSM8K: original promptGSM8K: with your instructionDiffMMLU: original promptMMLU: with your instructionDiff
Claude-Haiku-4.584.8 ± 0.385.1 ± 0.8+0.360.5 ± 2.555.6 ± 1.5−4.9
Claude-Sonnet-596.9 ± 0.397.4 ± 0.2+0.590.8 ± 1.290.0 ± 0.7−0.8
Claude-Sonnet-5.597.3 ± 0.097.5 ± 0.3+0.292.4 ± 1.092.6 ± 1.1+0.2
Gemini-3.8-Flash98.7 ± 0.398.3 ± 0.0−0.495.5 ± 0.496.0 ± 0.3+0.5
Gemini-4-Argon99.7 ± 0.399.7 ± 0.30.095.9 ± 0.495.7 ± 0.3−0.2
GPT-5.6-Luna88.1 ± 0.390.4 ± 1.0+2.383.2 ± 0.884.6 ± 0.9+1.4
DeepSeek-R189.3 ± 1.788.3 ± 1.5−1.074.9 ± 1.174.4 ± 1.7−0.5
DeepSeek-V385.6 ± 1.685.0 ± 0.8−0.674.8 ± 1.275.9 ± 0.6+1.1

On average over all 32 model–benchmark pairs (16 models, two benchmarks), pSp_S differs by only 1.3 points between the two prompts. The Dialectic Score (DS) also changes little: by 1.3 points on average. Appendix L (Table 9) gives all metrics.

With your instruction, SIEV's results stay very close to the results with the original prompt, and the two versions of the protocol lead to the same conclusions. The revised paper reports this evaluation in Appendix L, and §4.2 refers to it.

So SIEV's results hold when Agent B is told never to question the problem statement.

W2. The advantages listed in the Introduction. Your comment is about two items in the list of key advantages in the Introduction (§1): advantage (1), "Benchmark- and model-agnostic", and advantage (4), "Natural compatibility with multi-agent systems".

  • Advantage (1). We agree that its wording put the emphasis on the prompts, which is not its point. Its point is that SIEV is benchmark- and model-agnostic: it can be applied to any existing benchmark and to any model as they are, without changing the benchmark's questions. This is what distinguishes SIEV from benchmark-dependent evaluations such as GSM-Symbolic and GSM-Plus, which build new, modified versions of the GSM8K questions. SIEV uses each benchmark's original questions, with the same fixed prompts for every benchmark and model (Appendix B). In the revised paper, advantage (1) states this point directly: "SIEV applies to any existing benchmark and any model as they are, including benchmarks thought saturated: no benchmark is rewritten, no model is changed, and nothing is tuned per benchmark or model (Appendix B). On GSM8K and MMLU, it turns even saturated benchmarks into diagnostics of reasoning behavior."
  • Advantage (4), "Natural compatibility with multi-agent systems", is removed from the list. The multi-agent setting remains the motivation for the cross-model experiments (Appendix E).

Your follow-up comments helped us improve the paper further: the revision now includes the evaluation with your instruction (Appendix L, Table 9, referenced in §4.2), and its list of advantages in the Introduction is more precise, so that it does not overstate them (advantage (1) rewritten, advantage (4) removed).

With your remaining points now addressed, we would appreciate it if you would raise your rating score. Thank you for your time.

Reviewer_aH2uOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

It shows another way of investigating reasoning quality, which is about how a model responds to substantial disagreement. It shows that models vary quite a lot.

Critical strengths

The intro argues well that it is important to look at process and not just the final result.

Figure 2 is very helpful. Good example. However, I think 40 is actually the better answer; I was swayed by the antithesis there.

The core idea is very interesting! I like the framing that this is a measurement of how fragile models are to being thrown off by the antithesis.

Interesting that "Fragility is topic-dependent, not uniform"

Table 2 is very interesting. Nice to see that antithesis makes a real difference compared to just base challenge.

The paper does a good job engaging with potential disagreements or objections in section 4.2.

Critical weaknesses

A criticism might be that perhaps the strong antithesis generation models are simply better at bullshitting.

The paper fails to disentangle the antithesis strength from stability in the synthesis stage. It would be interesting to hold one of the two constant to investigate. I believe this was done on line 405, but I'd like to see more information about it.

A model which is simply very stubborn will score high on SIEV, which limits its usefulness.

Unclear what SIEV stands for.

Questions for authors

What does SIEV stand for? I have no other questions. Great paper.

As a followup question, perhaps in a different paper, I suggest that you see what SIEV scores correlate with.

3 comments

Thank you for your close reading and your support, including working through the example yourself. We answer each of your points below.

W1. Are strong antithesis models better at "bullshitting"? A persuasive but wrong objection is exactly what a robust reasoner should withstand, so a convincing antithesis is part of the test rather than a flaw in it. What matters is whether SIEV's ranking just reflects how persuasive each model's own antitheses are, and on MMLU it does not: when one model writes every antithesis, the ranking of the thesis models is preserved, for each of the 14 antithesis sources (next point).

W2. Disentangling antithesis strength from synthesis stability (line 405). You are right that the paper gave too little detail here. To address this, we expanded Appendix E with the full setting and a new Table 5. One model writes the antithesis for every thesis model, on five representative MMLU topics (US foreign policy, management, computer security, public relations and business ethics), for each of 14 antithesis sources. Table 5 gives, for each source, the Spearman correlation between the thesis models' ranking by pSp_S under that fixed source and their ranking under their own antitheses: between 0.87 and 0.93. The heatmaps in Figure 9 give the converse view, each thesis model under each antithesis source. So on these topics, holding the antithesis source fixed keeps the ranking: the ranking reflects stability in the synthesis stage, not antithesis strength.

W3. A very stubborn model would score high. We agree that such a model would score high on SIEV's metrics, which is why the paper's claim is directional (§1, §4.2 and §5): a low score is evidence of fragile reasoning, while a high score is not necessarily a certificate of genuine reasoning.

W4 and Q1. What SIEV stands for. Structured dIalectical rEasoning eValuation; the name is also a play on "sieve", a device that filters, which fits what the score does. SIEV is now defined at first use, in the introduction.

On the worked example. We were glad to see that the antithesis could sway a careful reader: that is what SIEV needs from it, since the synthesis then has to weigh a genuinely persuasive objection. The benchmark's answer follows the exact computation the question asks for, which is why it is marked correct.

Q2. What SIEV scores correlate with. Thank you for the suggestion. The conclusion now names it as a natural next step, beyond the distance correlations of Figure 5 and the jury's judgements.

How your review improved the paper: the fixed-antithesis setting in full, for each of the 14 sources (Appendix E, Table 5); SIEV defined at first use (introduction); your follow-up suggestion named as future work (conclusion).

Thank you again for your review and your constructive comments.

Thank you for addressing my comments.

I suggest you rename the paper to "Evaluating LLM Reasoning Fragility", because 'Fragility' is more specific than 'Beyond Correctness'.

I continue to support acceptance of this paper.

Thank you for your continued support, and for this suggestion. It is a good one: "fragility" names what SIEV measures more precisely than "beyond correctness", and we will change the title accordingly. Since ICLR restricts title changes while the paper is under review, we will make the change with the AC's approval.

Reviewer_H7kXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SIEV, which tests how language models respond to criticism of their answers. A model first gives an answer and explains it. Another model instance generates a challenge, and the first model then revises its answer. The paper calls these stages thesis, antithesis, and synthesis. It measures accuracy after criticism and the change from initial accuracy.

The paper also measures how often exactly one answer is correct: the initial answer or the answer proposed in the challenge. It calls this measure opposition compliance, or OC. Its combined measure, the Dialectic Score (DS), uses both this rate and accuracy after criticism.

The paper contributes this test procedure and a study of 23 models on GSM8K and MMLU. They also test challenges from different models, fixed criticism messages, and tools that score the initial reasoning alone. A group of models judges the revised reasoning, and the authors compare those judgments with human judgments.

Critical strengths
  1. The paper demonstrates that models with similar initial accuracy differ greatly after criticism in Table 1 and Figure 4. These results matter for systems that revise answers after feedback. This evidence supports SIEV as a useful test of answer reliability after criticism, even though it does not identify the original reasoning mechanism, which is elaborated in the weaknesses points below.

  2. The comparison messages help explain what causes answers to change. Table 2 compares SIEV with four alternatives on eight open-weight models. The alternatives ask models to check again, express generic doubt, criticize without a specific objection, or insist that the answer is wrong. These comparisons make the explanation more credible than a simple before-and-after accuracy drop. This strengthens the finding that the challenge's content matters.

Critical weaknesses
  1. The results does not explain how the model produced an answer. The abstract says SIEV distinguishes structured reasoning from surface-level pattern generation. Section 3 scores answer correctness and counts opposition when exactly one of the initial answer and challenge is correct. The test reveals whether criticism changes answers, but does not reveal how models produced their original answers. A model could reason correctly and then make a mistake during revisions, or repeat a memorized answer and resist criticism. The model judges examines revised reasoning and cannot establish that it is the learned patterns that produced the original answer.

  2. The evidence from the model judges is not sufficiently transparent for readers' reproducing and verification. The paper mentions that "Three model judges apply the same rules independently... label 98% of low-synthesis-score cases as non-genuine reasoning." The synthesis score is accuracy across many questions, so the rule for selecting "low-synthesis-score". Without that rule and the underlying counts, readers cannot assess how broadly the 98% result applies.

  3. The paper lacks experiment details, which prevent readers from reproducing the model comparisons. The reproducibility statement directs readers to Section 3 and the appendices. But at least the following information is missing:

  • The exact model versions, question sets, and numbers of questions.
  • Limits on answer generation and reasoning, and the number of repeated runs.
  • What the reported ±\pm values represent.

These gaps weaken my confidence in much of the paper's empirical value.

Questions for authors
  • Which cases enter the 98% result? How many cases qualify? How do you combine the three judges' decisions? Please give results for each model and benchmark. Please explain how you distinguish a reasoned revision that reaches a wrong answer from merely accepting the criticism. These details could change my assessment of the evidence from the judges.

  • Which model versions, question sets, generation and reasoning limits, and repeated runs produced Table 1? What do its uncertainty values mean? Appendix H mentions a test condition that suggests a misleading answer. Do its existing results separate the influence of that answer from the influence of its supporting argument?

3 comments

Thank you for a careful and concrete review; the revision now states each detail you listed.

W2 and Q1. The jury: cases, counts, combining rule and results. The submission's 98% came from an earlier jury pass on GSM8K: 1,799 of 1,835 cases where a correct thesis became a wrong synthesis, up to 120 per model from 26 models, were judged non-genuine. To address your questions, and because Table 1 now has 34 models (another reviewer asked for more recent ones), we ran the jury again for every model in Table 1, on GSM8K and now also on MMLU, with more recent frontier models as judges (§4.2, Appendix I):

  • Cases: those where a correct thesis was followed by a wrong synthesis (the paper's capitulations), one run per question: 91,573 cases and 217,481 verdicts.
  • Judges: Claude-Opus-5.5, Gemini-4-Argon and GPT-6.1-Sol; none scores a model from its own developer, and none sees the gold answer or the model's identity.
  • Combining rule: a case is non-genuine when the mean R2 over its judges is at most 1.
  • Result: 99.7% of these cases are judged non-genuine (GSM8K 99.2%, MMLU 99.8%).
  • Per model and benchmark: the new Table 7; 65 of the 68 cells are at least 96%.
  • Agreement with blind human scoring: Cohen's κ=0.96\kappa = 0.96.

Q1, second part. A reasoned revision that reaches a wrong answer, against accepting the criticism. Appendix I now gives the judges' prompt and rubric verbatim. The judges grade the process, blind to which answer is correct. R2 runs from 0 to 3: 0 is capitulation (adopting the objection's answer with no valid pushback, restating the objection, or ending on a third or evasive answer), 1 restates the initial solution without engaging, 2 is a shallow merge, and 3 is genuine weighing of both positions with explicit justification. A revision that moves away from the initial answer scores at most 1 unless the change is a genuinely reasoned correction. So the rubric scores how a change is reasoned, not whether its answer is right.

W3 and Q2. Model versions, question sets, limits, runs and ±\pm. To address these, we added a new Appendix J, and Table 1's caption now states the question sets and what ±\pm means:

  • Question sets: the GSM8K test split (1,319 questions) and the MMLU test split, all 57 subjects (about 14k questions).
  • Runs and ±\pm: each model answers the questions in independent runs of the full pipeline with the same prompts; ±\pm in Tables 1 and 2 is the standard deviation of the per-run score over three runs.
  • Limits: Table 8 gives each model's temperature, top-p and output limit (4,096 or 16,384 tokens, or the provider's default); no model was sent a reasoning-effort setting.
  • Versions: dated snapshots for eight OpenAI models (e.g. GPT-4.1 2025-04-14, O3 2025-04-16, GPT-5 2025-08-07), the public checkpoints for six open-weight models, and otherwise the provider's model of that name.

Q2, second part. Does the misleading-answer condition of Appendix H (now G) separate the answer's influence from its argument's? No, and it was not designed to. That condition states a target answer with an instruction to revise the reasoning to reach it, so it serves only as the known-wrong reference condition of the human study; Appendix G now says so. Table 2 separates pressure from content instead: the authority arm demands a change of answer with no argument and no alternative answer, and for every model it moves fewer correct answers than SIEV's substantive antithesis.

W1. The results do not reveal how a model produced its answer. We agree: SIEV is not designed to show how a model produced its answer. It shows the phenomenon itself, a correct answer that does not hold up against a substantive but wrong argument; in your words, a test of answer reliability after criticism, much like a student who solves the standard problem but gives up the right answer when shown a plausible but wrong solution. The claim is directional: a low score is evidence of fragile reasoning behind a correct answer, while a high score is not necessarily a certificate of genuine reasoning; as you note, a model could also hold a memorized answer. To address this, the abstract no longer says that SIEV distinguishes structured reasoning from surface-level pattern generation; its last sentence now says what SIEV tests, how a model's answer holds up against a substantive dialectical antithesis, and the conclusion opens with the same test.

How your review improved the paper: the jury for every Table 1 model on both benchmarks (§4.2, App. I, Table 7); the experimental details (App. J, Table 8); the role of the misleading-answer condition (App. G); the abstract and conclusion.

With all of your concerns and questions now addressed in the revision, we kindly ask you to raise your rating to 4, and your soundness, presentation and informativeness scores. Thank you.

Dear Reviewer H7kX,

We wanted to follow up, as the discussion closes on November 18. With all your concerns and questions addressed in the revision, we kindly ask you to raise your rating. If any question remains, we would be happy to answer it. Thank you for your time.

Dear Reviewer H7kX,

As the discussion closes in a few hours: each of your questions is answered in the revision. The jury's cases, counts, combining rule and results per model and benchmark are in Appendix I and Table 7, and the model versions, question sets, limits, runs and what ±\pm means are in Appendix J and Table 8. With your questions addressed, we kindly ask you to raise your rating. Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reviews. They found the problem "indeed important" and the paper "generally well-written and clear" (uZMu), the core idea "very interesting" (aH2u), and that models with similar initial accuracy "differ greatly after criticism" (H7kX); aH2u and H7kX both valued Table 2, whose comparisons "make the explanation more credible than a simple before-and-after accuracy drop" (H7kX). The concerns centered on what SIEV measures, the transparency of the jury and of the experimental details, and the model panel. Each reply is self-contained; below is a short summary of how the concerns are addressed and where the main changes are. In the revised PDF, new material is now in blue.

How the concerns are addressed

  1. What SIEV measures (uZMu W1, H7kX W1, aH2u W3). As the abstract now states, SIEV tests how a model's answer holds up against a substantive dialectical antithesis. The claim is directional: a low score is evidence of fragile reasoning behind a correct answer, while a high score is not necessarily a certificate of genuine reasoning. The drop is a reasoning failure, not a response to distraction: a second turn with no argument moves fewer correct answers than SIEV's antithesis (Table 2), and a jury of three frontier models judges 99.7% of the 91,573 cases where a correct thesis became a wrong synthesis non-genuine.

  2. The jury (H7kX W2, Q1). Run again for every model in Table 1, on GSM8K and now also on MMLU. No judge scores a model from its own developer, and none sees the gold answer or the model's identity. The paper now gives the cases, the combining rule, the judges' prompt verbatim, the results per model and benchmark (Table 7), and the agreement with blind human scoring (κ=0.96\kappa = 0.96).

  3. Experimental details (H7kX W3, Q2). A new appendix gives the question sets, the runs and what ±\pm means, the decoding settings and output limits, and the model versions.

  4. Model panel (uZMu W3). Table 1 now reports 34 models under the same protocol, including Claude, Gemini and recent GPT models, as one list sorted by DS on GSM8K. The pattern holds: on GSM8K the added models' thesis accuracy lies between 96.4 and 98.4, and their synthesis accuracy between 84.0 and 98.4 (56.5 to 95.3 on MMLU).

  5. Antithesis strength or synthesis stability (aH2u W1, W2). On five MMLU topics, when one model writes every antithesis, the ranking of the thesis models is preserved, for each of 14 antithesis sources (Spearman correlation between 0.87 and 0.93).

  6. Advantages and the acronym (uZMu W2, aH2u W4). §1 now states that SIEV is not a new benchmark but a structured way to evaluate models on existing benchmarks. Advantage (1) now states that SIEV is benchmark- and model-agnostic, and the multi-agent advantage is no longer listed. SIEV is defined at first use.

  7. Agent B told never to question the problem statement (uZMu, follow-up). We ran SIEV with this instruction added to Agent B's prompt, for 16 models on GSM8K and MMLU, three runs each. The results stay very close to those with the original prompt: pSp_S changes by 1.3 points on average, and the two versions of the protocol lead to the same conclusions.

Where to find the main changes

  • Abstract, §1, §3 and §5: what SIEV tests; the advantages; the acronym; why the antithesis opposes even a correct thesis, and what ✓\checkmark and ×\times mark in Figure 3; what SIEV scores correlate with, as future work.
  • Table 1, §4.1 and Figure 4: the 34 models; the caption states the question sets and what ±\pm means.
  • §1, §4.2, Appendices I and K, Table 7: the jury.
  • Appendix J (new) and Table 8: the experimental details; the reproducibility statement points to them.
  • Appendix E and Table 5: the fixed antithesis source.
  • Appendix G: the role of the misleading-answer condition.
  • Appendix L (new) and Table 9: SIEV with Agent B told never to question the problem statement; §4.2 refers to it.
  • Smaller edits: §3 states which metrics have parameters; §5 no longer calls the gaps in harder domains a lower bound; the DS formula in Table 1's caption and the Figure 9 caption are corrected, and the tie rule is stated precisely; Table 2, §4.2, Appendix G and the reproducibility statement say "runs" rather than "seeds"; the reproducibility statement says the code is public, with its link withheld for review.
  • Appendix order: the correlation figures (Appendix G of the submission) are now the last appendix, M, so Appendices H, I and J of the submission are now G, H and I.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.