acceptodds
Under review as a conference paper at ICLR 2027

HyperLogic: A Hard, Forward-Authored Chinese Logical Reasoning Benchmark with Execution-Derived Answers

Abstract

Existing logic benchmarks primarily measure models' ability to answer reasoning questions directly. Scalable benchmarks often generate text from formal structures, which makes answers easy to compute but fixes the formalization before the problem is written. Forward construction preserves the challenge of finding a faithful formalization, yet makes difficulty and answer reliability harder to control. We introduce HyperLogic, a forward-construction pipeline that separates problem authoring from answer generation. A multi-agent workflow hardens undergraduate-authored Chinese seeds without solving them; two agents from different model families independently translate each finished item into executable finite-domain models; their encodings and solver-derived answers undergo layered, agent-assisted adjudication under human-expert oversight. HyperLogic-Base contains items and sub-questions and separates seven frontier models by percentage points in strict item accuracy (–%). HyperLogic-Hard contains items with larger, coupled search spaces, on which no model exceeds % accuracy in direct answering. We also use Hard to evaluate agents' ability to formalize and solve problems with tools, comparing a code sandbox alone with one that includes our logic modeling library. The sandbox improves every model by – points; adding the library helps five models and hurts two. These results highlight the difficulty of faithful formalization even with tool access.

Primary area:
datasets and benchmarks
Keywords:
Logical reasoning, LLM evaluation, Benchmark construction, Natural language formalization, Tool-assisted agentic reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_MXMKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a forward-construction pipeline, termed HyperLogic, that separates problem authoring and answer generation. This pipeline incorporates a four-role agent loop to formulate strategy library with expert seeds, an answer generation to formulate problems and answers, and an evaluation process the validate the answers. Then, two datasets, HyperLogic-Base and HyperLogic-Hard, are established.

Critical strengths

This paper established two Chinese reasoning benchmarks.

Critical weaknesses
  1. This paper is poorly organized and is not well motivated. Which characteristics makes chinese logical reasoning challenge than the reasoning in English or other language? The claim that 'scalable benchmarks often generate text from formal structures, which makes answers easy to compute but fixes the formalization before problem is written' requires clear explanation with evidences.

  2. The coverage of the proposed benchmarks is to be clearly demonstrated and explained.

  3. In the answer generation phase, two models are dopted to independently formalize each item, but the agreement between thheir formalizations does does not guarantee that both represent the oroginal text consistently. The author should clarify the details and how would the agents do if disagreement or unallignment occurs.

  4. The paper provides only part of the benchmarks. How to assess the quality of the benchmark data and its applicability? For the HyperLogic, the authors did not quantify whether and how the answering errors come from incorrect reasoning, or incorrect formalization, or the tools?

  5. The novelty of both the pipeline and the benchmark is to be clearly demonstrated by clear statements and experiments.

Questions for authors

/

5 comments
AuthorsOpenReview ↗

Response to Reviewer MXMK (1/5)

We sincerely thank the reviewer for the careful review and detailed comments, which have helped us improve the paper. We have addressed each concern below and uploaded a revised paper, with all changes highlighted in blue. We hope these responses help the reviewer better understand our paper and are useful in reassessing it.

W1: Organization and motivation; why Chinese; evidence for the reverse-construction claim

Organization and motivation. We have reorganized the paper around two questions, one per tier (Introduction; Sections 4.1 and 4.2), and the contributions now lead with the two benchmarks (the construction pipeline is the means of building them):

  1. Where does the logical reasoning of frontier models still fall short? To answer this, a benchmark must remain hard for frontier models and separate them; otherwise no failures are left to study. Existing logic benchmarks no longer do so: on five of the seven in Table 1, three frontier models are within three points of each other. HyperLogic-Base remains hard and spreads seven frontier models over 33.0 points, which leaves room to study the failures themselves: controlled ablations show that the gap between models grows with the size of the case space, and error and trace analyses show how the models fail (W5).
  2. When a problem exceeds what a model can work through in context, can an agent formalize the natural-language rules faithfully and solve the problem with tools? The scale of a logical problem has no upper bound, and real rule systems such as regulations, contracts or system configurations quickly outgrow what a model can enumerate in context. Past that point, the natural approach is to write the rules as a formal model and hand the search to a solver. A solver, however, only guarantees that its answer follows from the formal model, not from the original text; whether the model is a faithful translation of the text is left to the agent. Existing benchmarks do not focus on this ability. We therefore built HyperLogic-Hard, on which no model exceeds 16% accuracy when answering directly, to measure it; even the best agent reaches only 57.0%.

Why Chinese. We do not claim that Chinese is intrinsically harder than English or other languages. We chose Chinese because the seeds were originally written by native speakers, free of translationese, and because Chinese logical-reasoning benchmarks remain relatively scarce (ChLogic, LogiQA). The pipeline is language-agnostic; the difficulty comes from the size of the state space, the coupling of constraints and faithful formalization. We thank the reviewer for raising this point; to make the benchmark accessible beyond Chinese, we will release a verified English version of the dataset (Appendix A.1).

AuthorsOpenReview ↗

Response to Reviewer MXMK (2/5)

W1: Organization and motivation; why Chinese; evidence for the reverse-construction claim (continued)

The reverse-construction claim. Reverse-constructed benchmarks such as RuleTaker, ProofWriter, LogicBench, ZebraLogic and SATBench generate the text from a formal structure, so every item comes with a predefined formalization from which its answer is computed. A forward-constructed problem has no such canonical form: the same problem text can be formalized in different ways. In constructing HyperLogic, two agents from different model families formalized each item independently, and their encodings were almost never structurally identical (only one pair in Base and none in Hard), even when they gave the same answers. On one item, for example, the background describes ten commands sent to a side door, a guide and a safety lock, each of which may succeed, fail because of a synchronization fault, or not run at all because its preconditions are not met. One encoding gives each command exactly these three outcomes, so a command that could not have changed anything simply did not run; the other gives each command a yes/no fault flag, so it still distinguishes whether such a command suffered a fault. Both encodings answer every question the same way, yet the first distinguishes 38 possible histories and the second 240: they disagree on what counts as a distinct situation.

This matters because modeling is itself a central step of logical reasoning, and a main source of difficulty in HyperLogic. To test this, we rendered 50 items (30 Base, 20 Hard) back into reverse-constructed versions from their gold encodings (numbered variables, one constraint per line, no narrative), keeping logic, state space and answers unchanged. The results show that all three models score markedly lower on the original items than on their reverse-constructed versions, by 14–18 points on Base and 32–47 points on Hard. This shows that forward construction itself adds substantial difficulty.

Item accuracy (%)Base originalBase reverseHard originalHard reverse
GPT-5.6 Sol78.993.335.070.0
DeepSeek V4 Pro48.966.723.355.0
Doubao Seed 2.1 Pro42.260.018.365.0

Base: direct answering. Hard: sandbox agents, same harness and time limit as the paper.

W2: Coverage

HyperLogic consists of deductive problems over finite domains, set in everyday life, institutional rules, technical systems and fictional puzzles. It covers the following types of logic (Section 3.1 and Appendix A.2 of the revision):

  • Propositional logic: conditionals, biconditionals and exclusive-or.
  • First-order logic over finite domains: counting quantifiers ("exactly / at least / at most k"), assignments and orderings (all-different, ranking), temporal order and state changes, definite descriptions ("the unique x such that") and arithmetic (sums, comparisons).
  • Modal reasoning: questions about what is possible (true in some feasible world), necessary (true in every feasible world), unique or incompatible, and about enumerating and counting the feasible worlds.
  • Counterfactual reasoning: hypothetical questions in which premises are added, retracted or replaced before reasoning again.
  • Reasoning about sets of premises: finding minimal sets of premises, such as one that explains a contradiction, and testing whether a set of conditions is jointly consistent.
AuthorsOpenReview ↗

Response to Reviewer MXMK (3/5)

W3: Agreement does not guarantee faithfulness

We agree that agreement alone does not guarantee faithfulness, and agreement is not our only check. Two independent encodings make errors much easier to find than one: wherever they disagree, at least one of them must be wrong, which points the expert directly to the rule at issue. Each item is therefore formalized independently by two agents from different model families, and both disagreements and agreements are checked (Section 2.4; Appendices C.1 and C.2):

  • When the encodings disagree, a logic expert, assisted by an agent, traces the disagreement to the source rules, repairs the faulty encoding and reruns it; items whose text is genuinely ambiguous are clarified or excluded.
  • When the encodings agree, they may still share a misreading. Automated fidelity checks therefore test each encoding against the text, looking for situations it admits that the text forbids (a missing rule) and situations it excludes that the text allows (an invented rule), and the expert also checks agreeing answers for shared misreadings.

Both cases occur in practice. Among the sub-questions that went to adjudication, the disagreement between the two encodings exposed an encoding error in 61.8%. More directly related to your concern, in another 8.8% the two encodings agreed but shared the same misreading. These shared misreadings were caught by our checking pipeline and expert review and repaired before release.

Two encodingsShare of adjudicated sub-questions
Agree, both correct29.4%
Agree, both wrong8.8%
Disagree, one correct58.2%
Disagree, both wrong3.6%

W4: Partial release; data quality; sources of answering errors

Partial release. We released a uniformly sampled 30% of each tier (69 Seed, 59 Base and 30 Hard items) with gold answers, so the released subset is representative of each tier. We hold back the rest to limit contamination. Test items that are published in full tend to enter the training data of later models, after which scores no longer measure the ability the benchmark targets (Sainz et al., 2023; Golchin & Surdeanu, 2024); models have been shown to score lower on fresh problems matched to a public benchmark than on the benchmark itself (Zhang et al., 2024). Not publishing test data in plain text is therefore a recommended mitigation (Jacovi et al., 2023). If the paper is accepted, we will consider releasing a larger share of the data.

Data quality. We randomly sampled 30 Base items (137 sub-questions) and 20 Hard items from the finalized benchmark and had them audited blind by native Chinese-speaking experts with backgrounds in logic, using coding agents as tools; the audit found no gold errors in the sample. The released data has also been checked independently: Reviewer 37As, a native Chinese speaker, audited all public Seed items and found only one gold error, which we have corrected. After this audit, the reviewer concluded: "… The benchmark creation and auditing process appear rigorous from the description given in the paper. From the supplement, incorrect ground truth rate appears low, and ambiguous question rate also appears low. …"

References

  • Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., and Agirre, E. (2023). NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark. Findings of EMNLP 2023.
  • Golchin, S. and Surdeanu, M. (2024). Time Travel in LLMs: Tracing Data Contamination in Large Language Models. ICLR 2024.
  • Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., Lyu, Q., Hendryx, S., Kaplan, R., Lunati, M., and Yue, S. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. NeurIPS 2024.
  • Jacovi, A., Caciularu, A., Goldman, O., and Goldberg, Y. (2023). Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks. EMNLP 2023.
Reviewer_nEoHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces hyperlogic, which is a benchmark of Chinese logic reasoning. It uses the "forward construction" process to separate the question writing from the answer generation. The process uses an agent workflow composed of four roles and two sets of strategy libraries for different difficulty levels to strengthen the manually written initial logic related topics. Then, two independent formal agents transform each question into a finite field theoretical model, and deduce the answer with the help of a logic solver. The benchmark has also been reviewed by multiple experts to ensure its coverage, the sufficiency and necessity of conditions, and the consistency of answers.

Critical strengths
  1. This is a truly auditable answer generation process. By separating the formal construction from the answer generation, and adopting two independent formal schemes, the logic solver is executed and the decision is made under the supervision of experts. I think this engineering scheme can effectively deal with the common annotation quality problems in logical reasoning benchmark. The facts obtained through the analysis and experiment strongly prove that it is not enough to judge the accuracy only by the consistency of the answers. I think this insight is crucial to the motivation of the article, because it directly solves the problem of lack of credibility in the existing benchmark.

  2. Hyperlogic is a benchmark with good discrimination. Now that the performance of large models is approaching the upper limit, its ability in model comparison logic reasoning has clear practical value.

Critical weaknesses
  1. The core methodology proposed in the article has not been validated. The author proposed "forward construction", "four role agent workflow", and "two independent standardized forms" as design choices, but did not conduct ablation experiments or compare them with simpler solutions. And in my opinion, it's actually about arranging instructions and assigning tasks for different LLM driven agents, and improving data transmission. This is more like a carefully designed multi-agent structure for Novelty, and the actual performance effect is unknown.

  2. I think it is not acceptable for the author to use the current benchmark as a motivation due to its lack of clear differentiation in evaluating model capabilities. I acknowledge that the author has invested considerable resources and time in the construction process of this benchmark. But I don't think it's enough to make me believe that this is a work with research value. If we are only building a new benchmark to continue ranking the model, I think its scientific significance is limited, and a more difficult benchmark does not necessarily mean more meaningful. The more important question that the author needs to answer is: what important new problems have we discovered by increasing the complexity of these combinatorial logic constraints and other issues? Regarding this, I may have discovered a potential motivation that the author wants to express: existing agents, even if able to call professional solvers, may fail due to their inability to correctly understand and formalize natural language rules. If this is the starting point, I think it may be more easily accepted, but the author's subsequent experimental analysis may involve similar content, but it has not formed a particularly complete argument.

  3. Following the second point. I think this paper has some useful empirical results, but it remains at the level of "reporting phenomena" and has not risen to the level of "explanatory mechanisms". Perhaps I would prefer to see these explanations.

Questions for authors

I mixed Weakness and Question together. Author can clarify the issues I mentioned in Weakness. I am looking forward to seeing mechanism explanations that truly support insights rather than boring basic experiments.

7 comments
AuthorsOpenReview ↗

Response to Reviewer nEoH (1/6)

We thank the reviewer for the careful review and constructive comments. We have responded to each comment below and uploaded a revised paper, with all changes highlighted in blue. Following your suggestion, we have added a range of mechanistic analyses of why models fail (W2 and W3). We hope they offer useful insights and help the reviewer better understand and reassess our paper.

W1 (1/3): "Forward construction"

Existing reverse-constructed logic benchmarks (RuleTaker, ProofWriter, LogicBench, ZebraLogic, SATBench) generate each problem from a formal structure, so every natural-language problem comes with a determinate, canonical formalization. A forward-constructed problem has no such canonical form: the same problem text can be formalized in different ways. In constructing HyperLogic, two agents from different model families formalized each item independently, and their encodings were almost never structurally identical (only one pair in Base and none in Hard), even when they gave the same answers. On one item, for example, the background describes ten commands sent to a side door, a guide and a safety lock, each of which may succeed, fail because of a synchronization fault, or not run at all because its preconditions are not met. One encoding gives each command exactly these three outcomes, so a command that could not have changed anything simply did not run; the other gives each command a yes/no fault flag, so it still distinguishes whether such a command suffered a fault. Both encodings answer every question the same way, yet the first distinguishes 38 possible histories and the second 240: they disagree on what counts as a distinct situation. This shows how diverse the modeling of forward-constructed problems is.

This matters because modeling is itself a central step of logical reasoning, and a main source of difficulty in HyperLogic. To test this, we reverse-constructed 50 items (30 Base, 20 Hard) from their gold encodings, keeping logic, state space and answers unchanged. The results show that all three models score markedly lower on the original items than on their reverse-constructed versions, by 14–18 points on Base and 32–47 points on Hard. This shows that forward construction itself adds substantial difficulty.

Item accuracy (%)Base originalBase reverseHard originalHard reverse
GPT-5.6 Sol78.993.335.070.0
DeepSeek V4 Pro48.966.723.355.0
Doubao Seed 2.1 Pro42.260.018.365.0

Base: direct answering. Hard: sandbox agents, same harness and time limit as the paper.

AuthorsOpenReview ↗

Response to Reviewer nEoH (2/6)

W1 (2/3): "Four role agent workflow"

Our hardening workflow has three components: a strategy library, four agent roles that draw on it, and a reviewer loop that sends drafts back for revision. All three are designed to make the problems harder. First, the generator needs to know how to make a problem harder. The strategy library supplies this: it encodes expert knowledge of what makes a logic problem hard and where frontier models fail. Without it, the generator only knows to "make the problem harder", not how. Second, the authoring work should be broken into steps small enough to do well. The four roles assign analyzing the seed and planning the hardening, rewriting the long background, and writing matching questions to separate agents, so each agent concentrates on one step. Third, a single pass rarely produces items that are hard for state-of-the-art models, so difficulty is built up over several rounds: the reviewer checks each candidate, points out concrete ways to make it harder, and sends it back for the next round.

In the ablation (table below), each component makes the items substantially harder. Differences are in item accuracy on the three-model mean:

  • Strategy library: −33.2 points. Without the library, items are almost always solved (96.5%; even the weakest model, Doubao, solves 93.6%). With it, the first draft falls to 63.3%.
  • Four roles vs. a single prompt: −24.7 points. With the same strategy library, single-prompt items are solved 88.0% of the time, against 63.3% for the four-role first draft.
  • Reviewer loop: −20.0 points over two rounds. Accuracy falls at every round for all three models (Doubao 38 → 32 → 12%, DeepSeek 64 → 56 → 46%, GPT 88 → 86 → 72%), while median feasible worlds grow from 21.5 to 36 to 55.5. Feasible worlds are a main source of item difficulty (W3 below).
Item accuracy (%)GPT-5.6 SolDeepSeek V4 ProDoubao Seed 2.1 ProMeanFeasible worlds (median)
Single prompt96.092.076.088.0—
Four roles, no library100.095.793.696.5—
Four roles, first draft88.064.038.063.321.5
+ review round 186.056.032.058.036
+ review round 272.046.012.043.355.5

All arms harden the same 50 seeds with the same generator (GPT-5.6 Sol, maximum reasoning). Single prompt: one call containing the strategy library. Four roles, no library: the four-role workflow without the strategy library. Four roles: the full workflow, scored at the first draft and after each of two review rounds.

W1 (3/3): "Two independent standardized forms"

The two independent formalizations serve to keep errors out of the gold answers. A solver returns exactly what an encoding implies, so if a single encoding misreads the background, the misreading becomes the gold answer, and an expert reviewing that one encoding has nothing pointing to the error. We therefore have two agents from different model families formalize each item independently: a disagreement exposes one side's error, which is then adjudicated and repaired, while misreadings shared by both encodings are caught by the automated fidelity checks and expert review (Section 2.4; Appendices C.1 and C.2).

Among the sub-questions that went to adjudication, 61.8% had an encoding error that the disagreement between the two encodings exposed; with a single formalization, no disagreement would have flagged these errors, leaving them entirely to expert review.

Two encodingsShare of adjudicated sub-questions
Agree, both correct29.4%
Agree, both wrong8.8%
Disagree, one correct58.2%
Disagree, both wrong3.6%
AuthorsOpenReview ↗

Response to Reviewer nEoH (3/6)

W2: Motivation beyond a harder leaderboard

We agree that a harder leaderboard is not, by itself, a scientific contribution. Separating frontier models is one purpose of HyperLogic-Base and a precondition for measuring progress, but not the only one: because Base remains hard for frontier models, it also shows where their logical reasoning falls short. We locate these shortcomings with controlled ablations on Base and finer-grained analyses of how models fail (W3 below).

Second, the scale of a logical problem has no upper bound, so problems eventually exceed what a model can work through in context. Past that point, a problem is not just harder; the way it must be solved changes. The model can no longer enumerate in context and must write the rules as a formal model and hand the search to tools. Hard is built past this point (Section 2.2), so it measures a different ability from Base, the one you identified: whether an agent that can call solvers can understand and formalize natural-language rules faithfully.

The Results and Findings section of the submission (Findings 4–6) had already shown that a sandbox lifts every model yet the best agent reaches only 57.0%, that our modeling library does not consistently help, and that most agents write plain Python. To better understand where these agents fall short, and to provide mechanistic explanations that truly support insights, we analyzed the agents' final scripts and trajectories in depth. Our findings follow; they are now Findings 6–9 of the revision (Sections 4.2 and 4.3, with details in Appendix E.4).

New finding 1 (Finding 6 in the revision): Agents fail at reading the rules, not at search. We used GPT-6.1 Sol to analyze why each wrong agent attempt that left a final script went wrong. The results show that for 85.5% of wrong attempts, the main error is a misread rule: the program runs as written, but its encoding of some background rule does not match the text (a rule omitted, a constraint added, or a condition, scope or bound changed). Another 4.3% misread the question, and only 2.4% are computation errors, where the encoding is right but the program fails to compute its answer. Thus, writing code is not the hard part; turning natural-language rules into a program that runs correctly is. This agrees with the reverse-construction experiment in W1: when the same Hard items are given as formal specifications, so that no rules need to be read from natural language, sandbox accuracy rises by 32–47 points.

Changing the tool does not change this. On the same model and item, writing plain Python, importing a solver or using our modeling library gives about the same accuracy, and the errors are distributed almost identically on every route (table below). This also explains why our library does not consistently help (Finding 5): it changes how the search runs, but the main problem is misread rules.

Main error of wrong attemptsPlain PythonImported solverOur library
Misread a rule85.0%85.6%88.0%
Misread the question4.3%4.2%4.3%
Computation error2.9%1.9%0.5%
Other or unclear7.9%8.3%7.2%
Reviewer_zdvVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces HyperLogic, a Chinese logical reasoning benchmark with two difficulty levels: HyperLogic-Base for distinguishing models using direct natural-language reasoning, and HyperLogic-Hard that has more complex problems and evaluates reasoning with code execution and symbolic tools. The authors also propose a pipeline in which human-written logic problems are progressively hardened by LLM agents, translated to formal models, and finally solved using symbolic solvers. The results are then checked through an expert-supervised adjudication process.

Critical strengths
  • Considers a timely and relevant benchmark-construction problem by preserving the challenge of formalizing natural-language statements instead of generating problems exclusively from predefined symbolic structures.
  • Separating authoring, formalization, solver execution, and expert adjudication is a well-motivated methodological design.
  • The proposed benchmark demonstrates substantial difficulty and differentiation among frontier models, with meaningful room for future improvement.
  • The agent experiments provide good evidence about the limitations of tool-augmented logical reasoning.
Critical weaknesses
  • The paper does not isolate the contribution of the four-role workflow, reviewer feedback loop, or hardening strategy libraries relative to simpler prompting or single-agent baselines.
  • More controlled analyses are needed to determine whether performance differences primarily reflect logical reasoning complexity or other factors such as long-context comprehension, enumeration, and output reliability.
  • The authors do not provide a blinded, independent assessment of residual errors in the finalized benchmark.
Questions for authors

See weaknesses section

3 comments
AuthorsOpenReview ↗

Response to Reviewer zdvV (1/3)

We thank the reviewer for the careful review and constructive comments. We have responded to each comment below and uploaded a revised paper, with all changes highlighted in blue. We hope these responses help the reviewer better understand our paper and are useful in reassessing it.

W1: Contribution of the four-role workflow, reviewer loop and strategy library

All three are designed to make the problems harder. First, the generator needs to know how to make a problem harder. The strategy library supplies this: it encodes expert knowledge of what makes a logic problem hard and where frontier models fail. Without it, the generator only knows to "make the problem harder", not how. Second, the authoring work should be broken into steps small enough to do well. The four roles assign analyzing the seed and planning the hardening, rewriting the long background, and writing matching questions to separate agents, so each agent concentrates on one step. Third, a single pass rarely produces items that are hard for state-of-the-art models, so difficulty is built up over several rounds: the reviewer checks each candidate, points out concrete ways to make it harder, and sends it back for the next round.

In the ablation (Table R1), each component makes the items substantially harder. Differences are in item accuracy on the three-model mean:

  • Strategy library: −33.2 points. Without the library, items are almost always solved (96.5%; even the weakest model, Doubao, solves 93.6%). With it, the first draft falls to 63.3%.
  • Four roles vs. a single prompt: −24.7 points. With the same strategy library, single-prompt items are solved 88.0% of the time, against 63.3% for the four-role first draft.
  • Reviewer loop: −20.0 points over two rounds. Accuracy falls at every round for all three models (Doubao 38 → 32 → 12%, DeepSeek 64 → 56 → 46%, GPT 88 → 86 → 72%), while median feasible worlds grow from 21.5 to 36 to 55.5. Feasible worlds are a main source of item difficulty; details are in W2 below and in Finding 2 of the revision.

Table R1 — Workflow ablation (50 seeds; item accuracy, %; one rollout at maximum thinking)

ArmGPT-5.6 SolDeepSeek V4 ProDoubao Seed 2.1 ProMeanFeasible worlds (median)
Single prompt96.092.076.088.0—
Four roles, no library100.095.793.696.5—
Four roles, first draft88.064.038.063.321.5
+ review round 186.056.032.058.036
+ review round 272.046.012.043.355.5

All arms harden the same 50 seeds with the same generator (GPT-5.6 Sol, maximum reasoning). Single prompt: one call containing the strategy library. Four roles, no library: the four-role workflow without the strategy library. Four roles: the full workflow, scored at the first draft and after each of two review rounds.

AuthorsOpenReview ↗

Response to Reviewer zdvV (2/3)

W2: Why are the items hard?

Thank you for this suggestion. Following it, we ran controlled analyses that separate logical complexity from long-context reading, enumeration and output reliability. In short, the size of the case space, i.e. the number of feasible worlds and the number of cases an answer must cover, drives both item difficulty and the gaps between models. On small case spaces all seven models are close; as the space grows, GPT-5.6 Sol and Claude Opus 5 hold steady while the other five fall. Background length and failures to return an answer are not the main causes.

We examine three factors: feasible worlds (logical complexity), cases covered (enumeration, in both list and count form) and background length (long-context reading). The main evidence is controlled ablations on fixed Base items (Table R2), each changing one factor while keeping the others fixed, with the gold answers recomputed by the solver.

  • Feasible worlds (100 items): adding verified facts that fix or restrict 1–2 variables shrinks the worlds to 0.25–0.6 (mid) and about 0.2 or less (small) of the original, at near-constant length.
  • Cases covered (100 questions from 50 items): we take one enumeration sub-question from each item and ask it on its own, once as a list and once as a count; removing output fields shrinks the cases an answer must cover to 0.4–0.75 (mid) and 0.15–0.45 (small) of the original, with the worlds unchanged.
  • Background length (100 items): neutral, scene-consistent narrative expands the background, which averages ~3,300 Chinese characters (about 4k tokens), to ~1.5× and ~2×, with the answers unchanged.

As supporting evidence, we split the released Base benchmark (195 items) into tertiles of each factor (Table R3). Throughout, the spread is the gap between the best and the worst of the seven models.

Taking the factors in turn:

  • Feasible worlds are the main driver. Shrinking the worlds on fixed items makes them easier for five of the seven models, and the weaker the model, the more it gains: Kimi K3 and Grok 4.6 gain 11 and 15 points, and DeepSeek V4 Pro, GLM-5.3 and Doubao 26–34 points, while GPT-5.6 Sol and Claude Opus 5 barely change. In other words, more feasible worlds widen the gap between models: the spread grows from 15.6 points at the small level to 49.6 at the original. The released benchmark shows the same pattern: the models are within 7.0 points of each other in the lowest tertile of feasible worlds and 52.5 points apart in the highest, where GPT-5.6 Sol and Claude Opus 5 hold steady and GLM-5.3 and Doubao fall the most.
  • Cases covered are a second driver. Beyond the number of feasible worlds, difficulty also depends on how many cases an answer must cover, i.e. how much the model has to enumerate. When we hold the worlds fixed and reduce only the cases to cover, GPT-5.6 Sol and Claude Opus 5 barely change, Kimi K3 and Grok 4.6 improve by 5 and 7 points, and DeepSeek V4 Pro, Doubao and GLM-5.3 improve by 11–24 points. The models that struggle with many feasible worlds are thus the same ones that struggle to enumerate many cases, and the gap between models grows with the number of cases: the spread rises from 9.7 points at the small level to 28.7 at the original. The released benchmark shows a similar trend, with the spread growing from 13.7 points in the lowest tertile of cases covered to 59.1 in the highest.
  • Background length has no significant effect. Doubling the background with neutral narrative changes no model's accuracy significantly and leaves the ranking essentially unchanged. The length drops that GLM-5.3 and Doubao show on the released benchmark mostly reflect a confound: in Base, longer backgrounds also have more feasible worlds.
  • Output failures do not change these conclusions. On Base, apart from Claude's refusals (excluded as in the paper), almost all unanswered attempts come from GLM-5.3 hitting the output limit: it tends to keep reasoning and re-checking until its output budget runs out. The other models rarely fail to return an answer.
AuthorsOpenReview ↗

Response to Reviewer zdvV (3/3)

W2: Why are the items hard? (continued)

Table R2 — Controlled ablations on fixed Base items (item accuracy, %; one rollout at maximum thinking; refusals excluded as in the paper; each factor uses its own set of items, so the original columns differ)

ModelFeasible worlds (small / mid / orig)Cases covered (small / mid / orig)Background length (orig / 1.5× / 2×)
GPT-5.6 Sol80.6 / 82.7 / 80.691.0 / 91.0 / 91.071.7 / 72.7 / 74.7
Claude Opus 575.3 / 76.3 / 72.096.7 / 97.8 / 95.777.7 / 79.8 / 75.5
Kimi K369.0 / 58.0 / 58.087.0 / 85.0 / 82.065.7 / 58.6 / 56.6
Grok 4.669.0 / 66.0 / 54.091.0 / 87.0 / 84.053.5 / 55.6 / 53.5
DeepSeek V4 Pro70.0 / 58.0 / 44.088.0 / 86.0 / 77.038.0 / 48.0 / 48.0
GLM-5.370.7 / 48.5 / 42.491.0 / 74.0 / 67.048.5 / 51.5 / 48.5
Doubao Seed 2.1 Pro65.0 / 47.0 / 31.087.0 / 79.0 / 72.042.4 / 51.5 / 42.4

Table R3 — Released Base benchmark split into tertiles of each factor (195 items; sub-question accuracy, %)

ModelFeasible worlds (low / mid / high)Cases covered (low / mid / high)Background length (low / mid / high)
GPT-5.6 Sol87.5 / 94.8 / 95.890.3 / 92.8 / 96.289.2 / 95.8 / 92.9
Claude Opus 588.3 / 91.2 / 89.287.0 / 88.7 / 90.589.4 / 91.5 / 85.7
Kimi K385.8 / 82.9 / 77.384.6 / 80.5 / 72.785.6 / 81.8 / 77.1
Grok 4.684.0 / 84.6 / 80.982.7 / 85.6 / 75.084.4 / 82.9 / 81.1
DeepSeek V4 Pro81.3 / 83.2 / 79.080.6 / 81.4 / 76.383.3 / 85.5 / 74.6
GLM-5.386.9 / 71.3 / 43.376.6 / 65.0 / 37.183.6 / 61.1 / 56.0
Doubao Seed 2.1 Pro82.9 / 76.6 / 60.880.4 / 73.1 / 50.183.4 / 70.0 / 65.6

Tertile cut-offs: 11 and 56 feasible worlds; 4 and 12.7 listed cases (4.7 and 28.3 for count); 2,276 and 3,902 characters.

The controlled ablations and the tertile analysis are now in the revision (Finding 2 and Figure 2; Appendix E.1).

Beyond these factors, the revision adds new findings on why models fail in both tiers (Section 4). On Base, the leading models mostly misread individual rules, while the others also lose track of cases (Finding 3). On Hard, agents fail mainly at reading the rules rather than at search (Finding 6). In both tiers, misreadings concentrate on sentences that demand precision, the models' own checks rarely revise their first reading, and only the leading models solve items consistently (Findings 7–9). These findings further explain the performance differences the reviewer asked about.

W3: Blinded, independent assessment of residual errors

We randomly sampled 30 Base items (137 sub-questions) and 20 Hard items from the finalized benchmark and had them audited blind by native Chinese-speaking experts with backgrounds in logic, using coding agents as tools. The audit found no gold errors in the sample. In addition, Reviewer 37As, a native Chinese speaker, independently audited all public Seed items and found only one gold error, which we have corrected. In the reviewer's words: "… The benchmark creation and auditing process appear rigorous from the description given in the paper. From the supplement, incorrect ground truth rate appears low, and ambiguous question rate also appears low. …"

Reviewer_37AsOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper presents a difficult Chinese logical reasoning benchmark that current frontier models struggle with.

Critical strengths
  1. The paper presents a logical reasoning benchmark that’s more suitable for current frontier models than previous benchmarks, and is able to discriminate them.
  2. The benchmark creation and auditing process appear rigorous from the description given in the paper. From the supplement, incorrect ground truth rate appears low, and ambiguous question rate also appears low. I also downloaded the supplementary material and asked Claude Code to audit potentially incorrect or ambiguous ground truths. Among seed questions (69 in the supplement), Claude Code flagged 1 as incorrect ground truth (“人和人相处”), which I manually confirmed, and 3 as likely being ambiguous. However, I manually checked those 3 and found them to be unambiguous and the ground truths to be correct. In fact, I think this makes the benchmark more meaningful, as the variety of natural language logic constructs makes formalization an important aspect that the benchmark tests. (Note that I am a native Chinese speaker.)
Critical weaknesses
  1. The grading uses an LLM as a judge. Some answers are quite long (over a thousand characters) – mainly for questions that require enumerating all possibilities. There’s a potential concern that the LLM’s ability to judge semantic equivalence degrades for such answers. Is there evidence that this is actually not the case?
  2. (minor) The “人和人相处” seed has incorrect ground truth. The first two premises give “相互理解” -> “好好相处”, which contradicts the consequent of the third premise, hence its antecedent is false, so the answer to the second subquestion should be “一定”.

Update: Both points have been adequately addressed by the authors, so I raised my soundness score from 2 to 3 and kept my overall score at 4 (accept).

Questions for authors

See Weakness 1.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 37As

We thank the reviewer for the positive assessment and for checking the public seed items by hand. We address both points below; the revised paper highlights all changes in blue.

W1: Is the LLM judge reliable for long answers?

Yes; we checked this directly. We drew 300 graded sub-questions stratified by tier, answer length and verdict, so that long answers are heavily over-represented (125 of the 300 have an answer of at least 1,000 characters, against about 12% overall). Two experts labelled each one with LLM assistance (code-based expansion and element-wise comparison of long enumerations), blind to the judges' verdicts and to which model produced the answer; they agreed on 99.0% (κ = 0.98), and the rest were adjudicated.

Agreement with the expert labelsAll 300Long answers (≥ 1,000 chars, n = 125)
Judge used in the paper (DeepSeek V4.1 Flash; DeepSeek V4 Pro on Hard)95.8%94.0%
GPT-6.1 Sol98.7%100%

Current frontier models are reliable judges even for long enumeration answers. The paper's judge errs rarely: weighted back to the full benchmark, its errors change sub-question accuracy by at most 0.5 points on Seed, Base or Hard. A stronger judge, GPT-6.1 Sol, agrees with the experts on all 125 long answers. This analysis is now in Appendix D.3 of the revision.

W2: The "人和人相处" seed item

Thank you for the careful check — you are right. Premises 1–2 entail "相互理解 → 好好相处", which contradicts the consequent of premise 3, so its antecedent must be false and the answer to the second part should be "一定". We use several layers of checking (model-based screening, expert review and solver validation) to keep such errors rare, but this one slipped through all of them. We have corrected the item and re-scored it with the paper's judge. The correction changes each model's Seed accuracy by at most 0.44 points; the only change in order is that GLM-5.3 and Grok 4.6 become tied. The revision reports the corrected Seed results in Table 2.

Great -- both my comments have been addressed. I've raised my soundness score to 3 and kept my overall score at 4 (accept).

Thank you for reading our response and raising your soundness score; we really appreciate your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.