acceptodds
Under review as a conference paper at ICLR 2027

One Edit, Three Worlds: Evaluating LLM Reasoning Beyond Answer Accuracy

13 views

Abstract

A reliable reasoner must be invariant to changes that do not matter and sensitive to the one change that does. Benchmarks measure the first property well and the second rarely, because every item is scored in isolation. Few prior studies, if any, systematically analyze whether the answer follows the premise that decides it. We propose an evaluation principle that tests both properties together. It builds sibling problems that differ in exactly one premise, and it counts an answer as correct only if the answer changes with that premise and stays the same when the problem is asked again together with its siblings. We call the resulting score verified accuracy. We instantiate the principle as One Edit, Three Worlds (OETW), a benchmark of 1,200 triads, each a base problem with three siblings whose one edited premise leaves the target with no valid answer, exactly one, or two, built on three constraint problems and certified by a solver rather than an LLM judge. The edit is minimal for the model: siblings are byte-identical outside the edited premise, surface-feature classifiers stay near chance, and where probabilities can be read the siblings receive nearly identical ones before reasoning begins. As problems grow, the raw accuracy of seven frontier and open-weight models falls to 33% and their verified accuracy to 2%, where the first is the level of random guessing, and the second means almost every isolated correct answer fails to survive its siblings. What remains is not random guessing but a low-entropy, model-dependent fallback, and neither abstention nor clarification repairs it.

Primary area:
datasets and benchmarks
Keywords:
large language models, reasoning evaluation, solver-certified benchmarks, LLM reliability, systematic bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wSUQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes One Edit, Three Worlds (OETW), an evaluation framework intended to assess LLM reasoning beyond isolated answer accuracy. The main idea is to construct sibling problems that are identical except for one decisive premise. The edited premise changes the target answer space among three states: NONE, UNIQUE, and MULTIPLE.

The paper introduces "verified accuracy", which requires a correct answer to remain correct across multiple views of the same sibling, including separate, joint, and pairwise presentations. The goal is to jointly evaluate sensitivity to a decisive premise and invariance to irrelevant presentation changes.

OETW is instantiated on three synthetic constraint-problem families: CrossChoice, rectangular Sudoku, and a bipartite-matching problem. The benchmark contains 1,200 solver-certified triads. The authors evaluate seven models and report that, as problem size increases, raw accuracy approaches approximately 33%, while verified accuracy drops much further, to around 2% in the post-frontier regime. The paper further analyzes reasoning trajectories for open-weight models, fallback behavior, abstention, and clarification.

Critical strengths

1. The benchmark construction is careful and provides strong control over shortcut explanations.

The use of solver-certified sibling problems is a substantial strength. The three siblings differ only in a single pivot premise, and the authors additionally perform byte-level identity checks, surface-feature classifier audits, and, for open-weight models, probability readouts before reasoning. These controls make the central comparison much cleaner than many perturbation-based reasoning benchmarks.

2. The paper highlights a useful distinction between isolated correctness and reliable reasoning across related instances.

The gap between raw and verified accuracy is informative. In particular, the observation that models can remain near 33% raw accuracy while verified accuracy drops close to zero clearly illustrates that an isolated correct answer may provide little evidence that the model actually tracked the decisive premise. I find this diagnostic perspective useful even though I have concerns about the novelty and generality of the proposed framework.

Critical weaknesses

1. I find the methodological novelty limited and largely compositional.

The central framework combines several ideas that already have close precedents: minimal-pair and contrast-set evaluation, directional/sensitivity testing, invariance and consistency evaluation, solver-certified synthetic reasoning tasks, and repeated evaluation of the same underlying example under different views. The paper itself discusses many of these lines of work, including CheckList, contrast sets, BLiMP, GSM-Symbolic, counterfactual reasoning, and consistency-based evaluation.

The main new element appears to be the particular combination of these ideas into a three-sibling construction together with the verified-accuracy criterion. I agree that this combination is carefully designed, but I am not convinced that it constitutes a sufficiently strong new evaluation principle rather than a useful integration of existing evaluation techniques.

The later analyses of reasoning trajectories, fallback behavior, abstention, clarification, relabeling, and recoding make the paper empirically richer, but in my view they broaden the study more than they deepen the central methodological contribution.

2. The claimed generality is not empirically demonstrated beyond three synthetic constraint families.

The paper frames OETW as a general evaluation principle for reasoning and discusses how it could be instantiated on mathematical word problems, code, text-to-SQL, planning, formal mathematics, legal reasoning, and reading comprehension. However, the actual benchmark and all primary experiments are restricted to three synthetic constraint families.

The authors themselves acknowledge that applicability beyond these tasks is currently a design claim. This is a substantial limitation because many of the properties that make OETW clean—exact answer-space enumeration, solver certification, precisely controlled single-premise edits, and NONE/UNIQUE/MULTIPLE states—are particularly natural for synthetic constraint problems.

I would find the generality claim considerably more convincing if the authors instantiated the same principle on at least one qualitatively different, existing reasoning benchmark rather than only describing how this could be done.

Questions for authors

1. What empirical evidence demonstrates that OETW provides substantially different information from existing minimal-pair, contrast-set, or invariance/consistency evaluations?

The paper clearly explains conceptual differences from prior work, but I would like to understand whether the proposed verified-accuracy protocol changes model rankings or exposes failures that would not already be captured by simpler minimal-pair sensitivity or consistency tests. Such a comparison would materially affect my assessment of the novelty of the evaluation framework.

2. Can the authors instantiate the OETW principle on at least one non-synthetic reasoning benchmark?

Table 3 describes how the framework could be applied to mathematical word problems, code, text-to-SQL, planning, formal mathematics, and other tasks, but these remain proposed extensions. A concrete experiment on one qualitatively different benchmark would substantially strengthen the claim that OETW is a general reasoning-evaluation principle rather than a benchmark design specialized to solver-friendly constraint problems.

3. Can the authors better separate problem complexity from prompt-length effects?

Problem size and prompt length increase together in the current construction, and the paper acknowledges that post-frontier failure may partially reflect long-context limitations. Can the authors provide a control in which logical difficulty changes while input length is approximately matched, or input length changes while the underlying reasoning difficulty is held fixed? This would clarify what the observed "frontier" is actually measuring.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer wSUQ

We thank the reviewer for the review and for the positive remarks on the controls of the construction and on what the gap between raw and verified accuracy shows. Response 1 gives the comparison with existing tests that the reviewer asked for.

Response 1 (W1, Q1). Verified accuracy catches failures that existing tests miss, and it changes model rankings. The reviewer asked whether verified accuracy changes rankings or exposes failures that simpler tests would not. It does both. We computed three existing tests on the same responses (Section 5.2, Table 3). The “invariance” column asks whether an answer survives renaming the states, and the “minimal-pair sensitivity” column asks whether two siblings get different answers; both follow the metrics used in CheckList (Ribeiro et al., 2020). “Contrast consistency” (Gardner et al., 2020) asks whether all three siblings are right when each is asked alone. Our verified triads apply the same all-three rule to verified answers. Medians in %:

Invariance (Ribeiro et al., 2020)Minimal-pair sensitivity (Ribeiro et al., 2020)Contrast consistency (Gardner et al., 2020)Verified triads (ours)Verified accuracy (ours)
Mostly solved (10 cells)91.796.790.07087.2
Collapsed, mostly one answer (15 cells)1000002.0
Collapsed, scattered answers (4 cells)50.553.71.001.2
Random answers33.366.73.70.01.2

The table shows that verified accuracy is high only where the models solve the problems, and each existing test gets at least one row wrong. Invariance gives a perfect score to a model that always gives the same answer. Minimal-pair sensitivity rewards noise, so random answers score 67%. Contrast consistency catches both, but it overstates when models still solve the problems (90% against 70% verified triads), because it never asks a sibling again.

Verified accuracy is a more accurate test, so it can change the ranking of models and show more clearly which models are better. On 4x4 Sudoku, contrast consistency ranks Qwen and Gemma above Luna and Terra (95 and 93 against 90 and 82). Verified accuracy ranks them below Luna and Terra (76 and 58 against 90 and 84). So the more accurate test shows that Luna and Terra are the better models on this problem.

Verified accuracy reads all of these cases correctly, which is why we see the combination as a contribution. OETW also has two elements the cited methods lack. Its edit decides whether an answer exists and whether it is unique, and the model's own probabilities do not distinguish the siblings before it reasons.

Response 2 (W2, Q2). OETW now runs on two existing benchmarks built by other authors. We converted Knights and Knaves and the NATURAL PLAN calendar task, both stated in natural language (see General Response 1.2 for more details). The properties the reviewer lists hold on both. An exact solver finds every answer, each edit changes one statement or one busy time, and the three siblings have no answer, one, or two. As in our own triads, the siblings cannot be told apart without reasoning (General Response 1.3). Verified accuracy also changes rankings here, as asked in Q1. On Knights and Knaves with 16 to 20 people, raw accuracy ranks Gemma above Qwen and Luna, and verified accuracy ranks it below both. So OETW is not limited to our three constraint problems.

Response 3 (Q3). Yes. Longer prompts at fixed difficulty do not cause the collapse, and harder problems at nearly fixed length do. The reviewer asked us to separate problem difficulty from prompt length. With difficulty fixed, we padded 4x4 Sudoku to the prompt lengths of collapsed problems, and Gemma and Qwen still solve it (General Response 2.1). With length nearly fixed, Sudoku prompts grow only from about 390 to 710 tokens, yet six of the seven models collapse on 9x9 and 12x12 Sudoku (General Response 2.2). So the collapse comes from harder problems, not from longer prompts.

Reviewer_cbfVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OETW, a benchmark for testing whether a model’s answer responds to the premise that determines it. Each item consists of three related problems that differ in one premise. The edit makes the target have no feasible answer, exactly one answer, or two possible answers. The benchmark contains 1,200 triads across three constraint-problem families, with answer sets certified by exact solvers. Models answer each problem separately and in joint and pairwise presentations. The proposed verified accuracy counts a sibling as correct only when its classification is correct across four views. Across seven models, the paper finds that increasing problem size leads to low verified accuracy and increasingly repetitive answers.

Critical strengths

The main strength of the paper is that it makes a failure hidden by ordinary accuracy directly observable: a model can answer some instances correctly while failing to distinguish problems whose answers should change. The solver-certified sibling construction provides a reliable way to test this without an LLM judge. The analysis is also more informative than simply reporting lower scores on harder problems. By examining differentiation rates and answer distributions, the paper shows that failures often take the form of a consistent, model-dependent fallback. The label and surface-feature controls strengthen the benchmark, while the abstention and clarification experiments examine whether models can recognise or recover from these failures.

Critical weaknesses
  • Verified accuracy combines several different failure modes. A sibling must be classified correctly in isolation, in a joint request, and in two pairwise requests. These views differ in context and instructions, including information about whether the siblings have different states. A failure can therefore reflect insufficient sensitivity to the edited premise, difficulty processing several alternatives, or instability under a changed elicitation format. Section 6 acknowledges that the metric combines these behaviours, but the headline raw-to-verified gap is still easy to interpret as a direct measure of premise tracking. I would like the paper to give greater prominence to the separate contributions.

  • The claim that the edit cannot be detected before reasoning is too strong. Byte identity outside the pivot establishes where the text changes. Near-chance surface classifiers establish that the tested features do not reliably predict the state. Similar state probabilities at the initial readout establish that this particular elicitation does not distinguish the siblings. None of these establishes that the model cannot represent the difference or that its consequence is generally inaccessible before an explicit reasoning trace. The checks are useful, but I would describe them as evidence against the tested shortcuts rather than an operational demonstration of an “invisible” edit. The distinction drawn from previous perturbation benchmarks depends too heavily on this interpretation.

  • Section 4 defines a cell as post-frontier when the confidence interval for raw accuracy contains one-third. Failure to distinguish accuracy from one-third is not the same as demonstrating that the two are equivalent. The balanced construction also allows a constant policy to achieve exactly one-third accuracy, which the paper itself illustrates. The answer-distribution and differentiation analyses provide stronger evidence of fallback than the grouping criterion alone. The conclusions about this regime should rely on those behavioural measurements rather than treating inclusion of chance in an interval as evidence that the model can no longer derive answers.

  • Problem size is entangled with prompt length and inference budget. Larger instances require more computation but also produce longer inputs. The paper acknowledges that these effects are not separated, and the open-weight models receive smaller reasoning budgets than the API models. This does not invalidate the benchmark results, but it limits what the observed collapse establishes about reasoning sensitivity itself. The findings describe behaviour under the chosen resource settings.

  • All three families are synthetic constraint problems presented in formal notation. Table 3 gives plausible extensions to other tasks, but these are proposed applications rather than evaluated ones. Several tasks also require more than an execution oracle to certify that an answer is impossible or non-unique; a test suite, for example, does not generally certify that no program can satisfy a specification. The benchmark is useful on its own, but the paper should distinguish its demonstrated contribution from the additional work needed to establish the same guarantees elsewhere.

Questions for authors
  1. Among the raw-correct answers that fail verification, how many fail the joint view, a pairwise view, or both?
  2. After excluding truncated and malformed responses, do the post-frontier cells still show near-zero differentiation and the same dominant fallback states?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer cbfV (1/2)

We thank the reviewer for the review and for the positive remarks on the sibling construction and on the analysis of the fallback. Response 1 gives the per-view breakdown the reviewer asked for.

Response 1 (W1, Q1). No single view causes the loss, and the models already fail to track the premise on separate requests. The reviewer asked how the raw-correct siblings that fail verification split across views. Section 5.2 now reports this, with the full table in Appendix N.

Not verifiedFail joint onlyFail pairwise onlyFail both
Pre-frontier (mostly solved)11.7%24.4%64.9%10.7%
Transition (partly solved)70.5%17.4%42.3%40.3%
Post-frontier (collapsed)93.5%10.5%31.0%58.4%
Random answers in each view7.7%30.8%61.5%

"Not verified" is the share of raw-correct siblings that fail verification. The other three columns split these failures and sum to 100%. "Fail joint only" means the sibling fails the joint request, which asks all three siblings in one prompt, but passes both pairwise requests, which ask it with one other sibling. "Fail pairwise only" means it passes the joint request but fails at least one pairwise request. "Fail both" means it fails the joint request and at least one pairwise request. The last row is the split that independent random answers in each request would give.

To read the table, compare each row with the last row. If one request format caused the failures, for example asking three siblings at once, the failures would pile up in its column. In the collapsed row they do not. The split (10.5%, 31.0%, 58.4%) is close to the random split (7.7%, 30.8%, 61.5%). A hard format would also hurt models that can solve the problems, yet in the mostly solved row only 11.7% of correct answers fail, and most of these fail a single request. So the extra requests do not cause the loss.

Premise tracking can also be judged without any joint or pairwise request. On separate requests alone, the collapsed models give all three siblings the same answer in 79% of triads at the median, against 11% for random answers (Section 5.2). So they fail to track the edited premise before any other format is involved. Verified accuracy adds one more requirement, that a correct answer stays correct when it is asked again, because an answer that changes when asked again cannot be relied on. Where the models collapse, the gap between raw and verified accuracy therefore comes from answers that do not depend on the edited premise, not from a hard request format.

Response 2 (Q2). Yes, without truncated or malformed responses, three different answers stay rare and wrong answers still name one state. The four API models and GPT-OSS produce no truncated or malformed responses, so their results on the problems where they collapse need no exclusion. On these problems, they give three different answers to at most 15% of triads, against 22% for random answers, and most wrong answers (95% to 97% at the median) name the same state. Gemma and Qwen often reach their 8k thinking budget, so we reran them with a 128k budget (General Response 2.3). They still give three different answers to under 7% of triads. So truncated responses and the token budget are not the main reason for the collapse.

AuthorsOpenReview ↗

Reply to Reviewer cbfV (2/2)

Response 3 (W2). Before reasoning, every check we can run finds no difference between the siblings. We thank the reviewer for this point, and Section 5.1 now states the claim in terms of behavior. The siblings differ only in the edited premise, and simple classifiers cannot predict the state from it. Before reasoning, the three open-weight models give the three siblings almost the same state probabilities, and in every cell the correct state gets 0.33 to 0.34, what a guess gets. The same readout does show a difference when the model has one. On 4x4 Sudoku, as the model reasons, the readout separates the siblings, and the probability of the correct state rises above 0.8. So the readout is able to show a difference, yet before reasoning it shows none.

The reviewer also notes that our contrast with perturbation benchmarks depends on this claim. It does not, because two differences hold without it. First, our edit decides whether an answer exists at all and whether it is unique, and a solver certifies this, whereas a typical perturbation only changes the value of the answer. Second, verified accuracy checks two things at once. The answer must change when the edited premise changes, and it must stay the same when the sibling is asked again. Both differences hold whatever the model encodes before reasoning.

Response 4 (W3). The post-frontier label only groups cells for reporting; our conclusions rest on how the models answer. Section 4 labels a cell (one model on one problem at one size) post-frontier when the 95% confidence interval of its raw accuracy contains one third, and uses this label only to report medians. As the reviewer suggests, our conclusions about post-frontier cells come from how the models answer. In every post-frontier cell, at most 15% of triads get three different answers, against 22% for random answers (Response 2), and the answer is almost unrelated to the correct state (Cramér's V, a 0-to-1 measure of association, is at most 0.12). The rule also does not inflate the post-frontier group. Cells near one third with a narrow interval that excludes it, such as Gemma on 12x12 Sudoku (30.7%), are not labeled post-frontier. So our conclusions about post-frontier cells come from these measurements, not from the label.

Response 5 (W4). Prompt length and budget do not explain the collapse. The reviewer notes that problem size is entangled with prompt length and with the reasoning budget. We tested each factor separately. First, we padded an easy problem, 4x4 Sudoku, to the prompt lengths of collapsed problems, and Gemma and Qwen still solve it (General Response 2.1). Second, Sudoku prompts grow only from about 390 to 710 tokens as the grid grows, yet six of the seven models collapse on 9x9 and 12x12 Sudoku (General Response 2.2). Third, we reran Gemma and Qwen with a 15 times larger budget, and they still collapse (General Response 2.3). Finally, on problems where the API models collapse, at least 93.5% of their responses end within the 8,192-token open-weight budget (General Response 2.4). So the collapse follows the difficulty of the problems, and it appears under every budget we tested.

Response 6 (W5). Two rows of the transfer table are now tested, and the code row is corrected. Table 5 (Table 3 in the submission) now marks with a dagger the two rows we tested, NATURAL PLAN and Knights and Knaves (General Response 1.2). We thank the reviewer for the suggestion, and we have corrected the code row in the paper. A test suite cannot show that no program meets a specification, so the row now asks for a proof in that case.

Reviewer_UkjrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes One Edit, Three Worlds (OETW), an evaluation principle for reasoning problems with decidable answers. OETW pairs a shared base with a single pivot premise. Changing the pivot’s value determines whether the target answer space is NONE, UNIQUE, or MULTIPLE. The authors operationalize “minimal edit” with three checks: byte identity outside the pivot, chance-level surface classifiers, and near-identical model probabilities before reasoning. They instantiate OETW across three constraint families: CrossChoice, Rectangular Sudoku, and Three-zone DM. They release 1,200 solver-certified triads and evaluate seven models using a paired protocol with separate, joint, pairwise, relabel, recode, abstention, and clarification requests.

Critical strengths

The benchmark’s construct validity is supported by its use of constraint problems with exact solvers, independent checkers, byte-identity audits, and surface-feature classifier controls.

Requiring correct answers across separate, joint, and pairwise views tests for constant-answer policies, fallback behavior, and invariance failures that raw accuracy alone can miss. The finding that raw accuracy can reach 33% while verified accuracy is near zero is a substantive contribution to reasoning evaluation.

Critical weaknesses

The evidence is limited to three synthetic constraint families with formal statements. The paper argues that the principle extends to natural language, code, SQL, planning, and legal reasoning, but does not test any of these settings.

Larger instances produce longer prompts, and open-weight models have much smaller reasoning budgets than API models. The trajectory evidence also covers only open-weight models. The reported finding that the worlds are never separated could therefore reflect context-length or budget limits, rather than failed derivation alone. The paper acknowledges this possibility but does not test it separately.

Questions for authors
  1. Can the authors provide at least one natural-language or agent-style OETW instantiation with an exact oracle, even on a small subset, to test whether the principle extends beyond formal constraint notation?

  2. How do the authors distinguish long-context effects from reasoning failures?

  3. To address the compute confound, can the authors report verified accuracy for API and open-weight models under matched reasoning budgets, or at least provide results for API models with capped reasoning budgets?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer Ukjr

We thank the reviewer for the review and for the positive remarks on the construction and its controls and on the gap between 33% raw and near-zero verified accuracy. Response 1 and Response 2 answer the reviewer's three questions with new results, and Response 3 answers the concern about trajectories.

Response 1 (W1, Q1). OETW now runs in natural language and on two existing benchmarks. The reviewer asked whether the principle extends beyond formal constraint notation. We restated two of our problems in natural language, as staffing and party-planning problems (see General Response 1.1 for more details). We also converted two existing benchmarks, Knights and Knaves and NATURAL PLAN, each checked by an exact solver (General Response 1.2). In natural language the collapse still appears in all five models we ran. This experiment confirms that the collapse does not come from formal notation.

Response 2 (W2, Q2, Q3). Prompt length and budget do not explain the collapse. On long contexts (Q2), we padded an easy problem to the prompt lengths of collapsed problems, and the models still solve it. Harder problems with short prompts still collapse (see General Response 2.1 and 2.2). On budgets (Q3), we reran the open-weight models with a 15 times larger budget, and they still collapse (see General Response 2.3). On problems where the API models collapse, at least 93.5% of their responses already end within 8,192 tokens, the open-weight budget. A cap at this budget would therefore affect at most 6.5% of their responses, so the token budget is not the main reason that the API models fail (General Response 2.4).

Response 3 (W2). The API models have no trajectories, but their answers show the same failure. We cannot draw trajectories for the API models, because they do not expose token probabilities. Their final answers, however, show the same failure as the open-weight models. The three siblings of a triad always have three different correct answers. Yet on problems where the API models collapse, they give all three siblings the same answer in most triads (86% at the median). Their answers do not change when the edited premise changes.

Reviewer_UkjrOpenReview ↗

Official Comment by Reviewer Ukjr

I appreciate the author’s detailed rebuttal. They provided more results that addressed my concerns, so I have decided to lift up my score.

AuthorsOpenReview ↗

Reply to Reviewer Ukjr

We thank the reviewer for reading the response and the additional results.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response 1: OETW beyond formal constraint problems

We thank the three reviewers for their careful reading, and for their positive remarks on the sibling construction and on the gap between raw and verified accuracy. General Response 1 and General Response 2 answer the concerns the three reviewers share, and each reviewer's reply then answers every remaining weakness and question.

OETW builds triads, three versions of one problem (siblings) that differ only in one edited premise. The edit leaves the target with no valid answer, exactly one, or two, which we call the sibling's state. Raw accuracy scores each sibling asked alone, and verified accuracy counts a correct answer only if it also holds when the sibling is asked together with its siblings. As problems grow, the raw accuracy of seven models falls to 33%, the level of random guessing, and their verified accuracy to 2%. We call this the collapse.

All three reviewers note that the evidence covers only three synthetic constraint problems stated in formal notation, and that Table 3 of the submission proposes other tasks without testing them. We ran OETW in natural language (1.1) and on two existing benchmarks by other authors (1.2), and checked that, as before, the new siblings cannot be told apart without reasoning (1.3).

1.1 Natural language does not remove the collapse. The first question is whether the collapse depends on formal notation. To test this, we restated two of our three problems in plain English as staffing and party-planning problems, with the same instances, answers and evaluation protocol (Section 5.5, Appendix P). In both problems the three siblings differ in one sentence, a qualification or a rule, and N is the problem size.

  • Three-zone DM -> staffing problem. We formulate Three-zone DM, a bipartite matching problem, as a manager assigning N employees to N projects, one employee per project, and each employee only to a project they are qualified for. The premises are sentences such as "Fiona is qualified for Project Oak". The question asks which project a named employee works on.

  • CrossChoice -> party-planning problem. We formulate the CrossChoice problem as N friends each deciding whether to come, under rules such as "at least one of these is true: Douglas does not come, Hannah comes, or Martha does not come". The question asks whether one named friend comes.

The collapse still appears in all five models we ran. Gemma and Qwen score 0.3-7.7% verified accuracy at every size, against 0.0-4.3% in formal notation. At the sizes where the API models collapse in formal notation (Terra and Luna at DM 48 and CrossChoice 20, Sol at DM 72 and CrossChoice 33), they score 32.0-37.3% raw and 0.7-10.0% verified in natural language. The collapse therefore comes from the problems themselves, not from how they are written.

1.2 On existing benchmarks, verified accuracy shows what raw accuracy misses. The second question is whether the principle works on benchmarks that other authors built, so we converted two of them (Section 5.5, Appendix P). In Knights and Knaves (Xie et al., 2024), each person is either a knight, who always tells the truth, or a knave, who always lies. The puzzle asks who is which. We used the published puzzles with 3 to 8 people and new ones with 10 to 20 people from the authors' generator. In the NATURAL PLAN calendar task (Zheng et al., 2024), the question asks when a meeting can start, given each participant's busy times. Each original puzzle or calendar task becomes one sibling. The other two siblings change one statement of one person, or move one busy time of one participant. An exact solver finds all valid answers of every sibling, and a second, independent implementation agrees on all of them.

With 16 to 20 people, raw and verified accuracy rank the models differently. Gemma answers more siblings correctly than Qwen and Luna (83% raw, against 79% and 74%), yet its verified accuracy is lower than both (36%, against 56% and 44%). Raw accuracy ranks Gemma above these two models, and verified accuracy ranks it below them. Verified accuracy therefore adds information even on benchmarks we did not build.

1.3 As in the paper, the new siblings cannot be told apart without reasoning. We checked this in the same three ways as the paper, on all 1,720 new triads (Appendix P). The three siblings are identical except for the edited premise. Simple classifiers that see only the edited premise predict the state no better than chance. Before reasoning, Gemma and Qwen give the three siblings almost the same probabilities for the three states, and the correct state gets about one third, no better than a guess. So any difference in a model's answers to the three siblings comes from its reasoning, as in the original triads.

AuthorsOpenReview ↗

General Response 2: Prompt length and reasoning budget

All three reviewers note that larger problems also have longer prompts. Two of them add that the open-weight models reason under a smaller budget (8,192 thinking tokens) than the API models. The collapse could therefore come from prompt length or budget rather than from difficulty. We tested each factor separately (Section 5.4, Appendix O): length with difficulty fixed (2.1), difficulty with length nearly fixed (2.2), a 15 times larger budget for the open-weight models (2.3), and the budget the API models actually use (2.4).

2.1 Long prompts alone do not cause the collapse. The first test keeps difficulty fixed and changes only length. We took 4x4 Sudoku, which Gemma and Qwen solve, and padded each prompt with unrelated prose to 5.5k, 13k and 22k tokens. These match the prompt lengths of three problems on which Gemma has collapsed (Three-zone DM 48 and 96, CrossChoice 46). Both models still solve the padded problems. Even at 22k tokens, raw accuracy is 82% for Gemma and 98% for Qwen, far above chance (33%). On the Three-zone DM and CrossChoice problems of the same length, Gemma is at chance. Length alone therefore does not cause the collapse.

2.2 Hard problems cause the collapse even with short prompts. Now we keep the length almost fixed and make the problem harder. In Sudoku, a 12x12 prompt has about 710 tokens, not much more than the 390 tokens of a 4x4 prompt. Still, six of the seven models collapse on 9x9 and 12x12 Sudoku. In contrast, Sol and Terra still solve Three-zone DM 24, although its prompts are about three times longer. So the collapse comes from harder problems, not from longer prompts.

2.3 A 15 times larger budget does not remove the collapse. Next we gave the open-weight models more room to think. We reran Gemma and Qwen with a 128k thinking budget instead of 8k, with everything else unchanged. On problems they partly solved before, raw accuracy rises by 19 to 32 points. On problems where they had collapsed, the extra budget does not help, even though Gemma now thinks about ten times longer. Their answers still do not change with the edited premise. Each triad has three different correct answers, yet Gemma and Qwen give three different answers to fewer than 7% of these triads, less often than random answers would (22%). A short budget therefore does not explain the collapse.

2.4 The API models collapse while using less than the open-weight budget. A fair comparison needs the API models under the same 8,192-token budget as the open-weight models. On problems where the API models collapse, they already stay within the 8,192-token budget. At least 93.5% of their responses end on their own within 8,192 tokens. They even think less there than on problems they partly solve, for example about 1.1k tokens instead of 3.4k for Terra. In the other direction, the open-weight models still collapse with a 128k budget (2.3). So the difference in budget does not explain the collapse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.