- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes One Edit, Three Worlds (OETW), an evaluation framework intended to assess LLM reasoning beyond isolated answer accuracy. The main idea is to construct sibling problems that are identical except for one decisive premise. The edited premise changes the target answer space among three states: NONE, UNIQUE, and MULTIPLE.
The paper introduces "verified accuracy", which requires a correct answer to remain correct across multiple views of the same sibling, including separate, joint, and pairwise presentations. The goal is to jointly evaluate sensitivity to a decisive premise and invariance to irrelevant presentation changes.
OETW is instantiated on three synthetic constraint-problem families: CrossChoice, rectangular Sudoku, and a bipartite-matching problem. The benchmark contains 1,200 solver-certified triads. The authors evaluate seven models and report that, as problem size increases, raw accuracy approaches approximately 33%, while verified accuracy drops much further, to around 2% in the post-frontier regime. The paper further analyzes reasoning trajectories for open-weight models, fallback behavior, abstention, and clarification.
Critical strengths
1. The benchmark construction is careful and provides strong control over shortcut explanations.
The use of solver-certified sibling problems is a substantial strength. The three siblings differ only in a single pivot premise, and the authors additionally perform byte-level identity checks, surface-feature classifier audits, and, for open-weight models, probability readouts before reasoning. These controls make the central comparison much cleaner than many perturbation-based reasoning benchmarks.
2. The paper highlights a useful distinction between isolated correctness and reliable reasoning across related instances.
The gap between raw and verified accuracy is informative. In particular, the observation that models can remain near 33% raw accuracy while verified accuracy drops close to zero clearly illustrates that an isolated correct answer may provide little evidence that the model actually tracked the decisive premise. I find this diagnostic perspective useful even though I have concerns about the novelty and generality of the proposed framework.
Critical weaknesses
1. I find the methodological novelty limited and largely compositional.
The central framework combines several ideas that already have close precedents: minimal-pair and contrast-set evaluation, directional/sensitivity testing, invariance and consistency evaluation, solver-certified synthetic reasoning tasks, and repeated evaluation of the same underlying example under different views. The paper itself discusses many of these lines of work, including CheckList, contrast sets, BLiMP, GSM-Symbolic, counterfactual reasoning, and consistency-based evaluation.
The main new element appears to be the particular combination of these ideas into a three-sibling construction together with the verified-accuracy criterion. I agree that this combination is carefully designed, but I am not convinced that it constitutes a sufficiently strong new evaluation principle rather than a useful integration of existing evaluation techniques.
The later analyses of reasoning trajectories, fallback behavior, abstention, clarification, relabeling, and recoding make the paper empirically richer, but in my view they broaden the study more than they deepen the central methodological contribution.
2. The claimed generality is not empirically demonstrated beyond three synthetic constraint families.
The paper frames OETW as a general evaluation principle for reasoning and discusses how it could be instantiated on mathematical word problems, code, text-to-SQL, planning, formal mathematics, legal reasoning, and reading comprehension. However, the actual benchmark and all primary experiments are restricted to three synthetic constraint families.
The authors themselves acknowledge that applicability beyond these tasks is currently a design claim. This is a substantial limitation because many of the properties that make OETW clean—exact answer-space enumeration, solver certification, precisely controlled single-premise edits, and NONE/UNIQUE/MULTIPLE states—are particularly natural for synthetic constraint problems.
I would find the generality claim considerably more convincing if the authors instantiated the same principle on at least one qualitatively different, existing reasoning benchmark rather than only describing how this could be done.
Questions for authors
1. What empirical evidence demonstrates that OETW provides substantially different information from existing minimal-pair, contrast-set, or invariance/consistency evaluations?
The paper clearly explains conceptual differences from prior work, but I would like to understand whether the proposed verified-accuracy protocol changes model rankings or exposes failures that would not already be captured by simpler minimal-pair sensitivity or consistency tests. Such a comparison would materially affect my assessment of the novelty of the evaluation framework.
2. Can the authors instantiate the OETW principle on at least one non-synthetic reasoning benchmark?
Table 3 describes how the framework could be applied to mathematical word problems, code, text-to-SQL, planning, formal mathematics, and other tasks, but these remain proposed extensions. A concrete experiment on one qualitatively different benchmark would substantially strengthen the claim that OETW is a general reasoning-evaluation principle rather than a benchmark design specialized to solver-friendly constraint problems.
3. Can the authors better separate problem complexity from prompt-length effects?
Problem size and prompt length increase together in the current construction, and the paper acknowledges that post-frontier failure may partially reflect long-context limitations. Can the authors provide a control in which logical difficulty changes while input length is approximately matched, or input length changes while the underlying reasoning difficulty is held fixed? This would clarify what the observed "frontier" is actually measuring.
Reply to Reviewer wSUQ
We thank the reviewer for the review and for the positive remarks on the controls of the construction and on what the gap between raw and verified accuracy shows. Response 1 gives the comparison with existing tests that the reviewer asked for.
Response 1 (W1, Q1). Verified accuracy catches failures that existing tests miss, and it changes model rankings. The reviewer asked whether verified accuracy changes rankings or exposes failures that simpler tests would not. It does both. We computed three existing tests on the same responses (Section 5.2, Table 3). The “invariance” column asks whether an answer survives renaming the states, and the “minimal-pair sensitivity” column asks whether two siblings get different answers; both follow the metrics used in CheckList (Ribeiro et al., 2020). “Contrast consistency” (Gardner et al., 2020) asks whether all three siblings are right when each is asked alone. Our verified triads apply the same all-three rule to verified answers. Medians in %:
The table shows that verified accuracy is high only where the models solve the problems, and each existing test gets at least one row wrong. Invariance gives a perfect score to a model that always gives the same answer. Minimal-pair sensitivity rewards noise, so random answers score 67%. Contrast consistency catches both, but it overstates when models still solve the problems (90% against 70% verified triads), because it never asks a sibling again.
Verified accuracy is a more accurate test, so it can change the ranking of models and show more clearly which models are better. On 4x4 Sudoku, contrast consistency ranks Qwen and Gemma above Luna and Terra (95 and 93 against 90 and 82). Verified accuracy ranks them below Luna and Terra (76 and 58 against 90 and 84). So the more accurate test shows that Luna and Terra are the better models on this problem.
Verified accuracy reads all of these cases correctly, which is why we see the combination as a contribution. OETW also has two elements the cited methods lack. Its edit decides whether an answer exists and whether it is unique, and the model's own probabilities do not distinguish the siblings before it reasons.
Response 2 (W2, Q2). OETW now runs on two existing benchmarks built by other authors. We converted Knights and Knaves and the NATURAL PLAN calendar task, both stated in natural language (see General Response 1.2 for more details). The properties the reviewer lists hold on both. An exact solver finds every answer, each edit changes one statement or one busy time, and the three siblings have no answer, one, or two. As in our own triads, the siblings cannot be told apart without reasoning (General Response 1.3). Verified accuracy also changes rankings here, as asked in Q1. On Knights and Knaves with 16 to 20 people, raw accuracy ranks Gemma above Qwen and Luna, and verified accuracy ranks it below both. So OETW is not limited to our three constraint problems.
Response 3 (Q3). Yes. Longer prompts at fixed difficulty do not cause the collapse, and harder problems at nearly fixed length do. The reviewer asked us to separate problem difficulty from prompt length. With difficulty fixed, we padded 4x4 Sudoku to the prompt lengths of collapsed problems, and Gemma and Qwen still solve it (General Response 2.1). With length nearly fixed, Sudoku prompts grow only from about 390 to 710 tokens, yet six of the seven models collapse on 9x9 and 12x12 Sudoku (General Response 2.2). So the collapse comes from harder problems, not from longer prompts.