- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed a forward-construction pipeline, termed HyperLogic, that separates problem authoring and answer generation. This pipeline incorporates a four-role agent loop to formulate strategy library with expert seeds, an answer generation to formulate problems and answers, and an evaluation process the validate the answers. Then, two datasets, HyperLogic-Base and HyperLogic-Hard, are established.
Critical strengths
This paper established two Chinese reasoning benchmarks.
Critical weaknesses
-
This paper is poorly organized and is not well motivated. Which characteristics makes chinese logical reasoning challenge than the reasoning in English or other language? The claim that 'scalable benchmarks often generate text from formal structures, which makes answers easy to compute but fixes the formalization before problem is written' requires clear explanation with evidences.
-
The coverage of the proposed benchmarks is to be clearly demonstrated and explained.
-
In the answer generation phase, two models are dopted to independently formalize each item, but the agreement between thheir formalizations does does not guarantee that both represent the oroginal text consistently. The author should clarify the details and how would the agents do if disagreement or unallignment occurs.
-
The paper provides only part of the benchmarks. How to assess the quality of the benchmark data and its applicability? For the HyperLogic, the authors did not quantify whether and how the answering errors come from incorrect reasoning, or incorrect formalization, or the tools?
-
The novelty of both the pipeline and the benchmark is to be clearly demonstrated by clear statements and experiments.
Questions for authors
/
5 comments
Response to Reviewer MXMK (2/5)
W1: Organization and motivation; why Chinese; evidence for the reverse-construction claim (continued)
The reverse-construction claim. Reverse-constructed benchmarks such as RuleTaker, ProofWriter, LogicBench, ZebraLogic and SATBench generate the text from a formal structure, so every item comes with a predefined formalization from which its answer is computed. A forward-constructed problem has no such canonical form: the same problem text can be formalized in different ways. In constructing HyperLogic, two agents from different model families formalized each item independently, and their encodings were almost never structurally identical (only one pair in Base and none in Hard), even when they gave the same answers. On one item, for example, the background describes ten commands sent to a side door, a guide and a safety lock, each of which may succeed, fail because of a synchronization fault, or not run at all because its preconditions are not met. One encoding gives each command exactly these three outcomes, so a command that could not have changed anything simply did not run; the other gives each command a yes/no fault flag, so it still distinguishes whether such a command suffered a fault. Both encodings answer every question the same way, yet the first distinguishes 38 possible histories and the second 240: they disagree on what counts as a distinct situation.
This matters because modeling is itself a central step of logical reasoning, and a main source of difficulty in HyperLogic. To test this, we rendered 50 items (30 Base, 20 Hard) back into reverse-constructed versions from their gold encodings (numbered variables, one constraint per line, no narrative), keeping logic, state space and answers unchanged. The results show that all three models score markedly lower on the original items than on their reverse-constructed versions, by 14–18 points on Base and 32–47 points on Hard. This shows that forward construction itself adds substantial difficulty.
| Item accuracy (%) | Base original | Base reverse | Hard original | Hard reverse |
|---|---|---|---|---|
| GPT-5.6 Sol | 78.9 | 93.3 | 35.0 | 70.0 |
| DeepSeek V4 Pro | 48.9 | 66.7 | 23.3 | 55.0 |
| Doubao Seed 2.1 Pro | 42.2 | 60.0 | 18.3 | 65.0 |
Base: direct answering. Hard: sandbox agents, same harness and time limit as the paper.
W2: Coverage
HyperLogic consists of deductive problems over finite domains, set in everyday life, institutional rules, technical systems and fictional puzzles. It covers the following types of logic (Section 3.1 and Appendix A.2 of the revision):
- Propositional logic: conditionals, biconditionals and exclusive-or.
- First-order logic over finite domains: counting quantifiers ("exactly / at least / at most k"), assignments and orderings (all-different, ranking), temporal order and state changes, definite descriptions ("the unique x such that") and arithmetic (sums, comparisons).
- Modal reasoning: questions about what is possible (true in some feasible world), necessary (true in every feasible world), unique or incompatible, and about enumerating and counting the feasible worlds.
- Counterfactual reasoning: hypothetical questions in which premises are added, retracted or replaced before reasoning again.
- Reasoning about sets of premises: finding minimal sets of premises, such as one that explains a contradiction, and testing whether a set of conditions is jointly consistent.
Response to Reviewer MXMK (3/5)
W3: Agreement does not guarantee faithfulness
We agree that agreement alone does not guarantee faithfulness, and agreement is not our only check. Two independent encodings make errors much easier to find than one: wherever they disagree, at least one of them must be wrong, which points the expert directly to the rule at issue. Each item is therefore formalized independently by two agents from different model families, and both disagreements and agreements are checked (Section 2.4; Appendices C.1 and C.2):
- When the encodings disagree, a logic expert, assisted by an agent, traces the disagreement to the source rules, repairs the faulty encoding and reruns it; items whose text is genuinely ambiguous are clarified or excluded.
- When the encodings agree, they may still share a misreading. Automated fidelity checks therefore test each encoding against the text, looking for situations it admits that the text forbids (a missing rule) and situations it excludes that the text allows (an invented rule), and the expert also checks agreeing answers for shared misreadings.
Both cases occur in practice. Among the sub-questions that went to adjudication, the disagreement between the two encodings exposed an encoding error in 61.8%. More directly related to your concern, in another 8.8% the two encodings agreed but shared the same misreading. These shared misreadings were caught by our checking pipeline and expert review and repaired before release.
| Two encodings | Share of adjudicated sub-questions |
|---|---|
| Agree, both correct | 29.4% |
| Agree, both wrong | 8.8% |
| Disagree, one correct | 58.2% |
| Disagree, both wrong | 3.6% |
W4: Partial release; data quality; sources of answering errors
Partial release. We released a uniformly sampled 30% of each tier (69 Seed, 59 Base and 30 Hard items) with gold answers, so the released subset is representative of each tier. We hold back the rest to limit contamination. Test items that are published in full tend to enter the training data of later models, after which scores no longer measure the ability the benchmark targets (Sainz et al., 2023; Golchin & Surdeanu, 2024); models have been shown to score lower on fresh problems matched to a public benchmark than on the benchmark itself (Zhang et al., 2024). Not publishing test data in plain text is therefore a recommended mitigation (Jacovi et al., 2023). If the paper is accepted, we will consider releasing a larger share of the data.
Data quality. We randomly sampled 30 Base items (137 sub-questions) and 20 Hard items from the finalized benchmark and had them audited blind by native Chinese-speaking experts with backgrounds in logic, using coding agents as tools; the audit found no gold errors in the sample. The released data has also been checked independently: Reviewer 37As, a native Chinese speaker, audited all public Seed items and found only one gold error, which we have corrected. After this audit, the reviewer concluded: "… The benchmark creation and auditing process appear rigorous from the description given in the paper. From the supplement, incorrect ground truth rate appears low, and ambiguous question rate also appears low. …"
References
- Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., and Agirre, E. (2023). NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark. Findings of EMNLP 2023.
- Golchin, S. and Surdeanu, M. (2024). Time Travel in LLMs: Tracing Data Contamination in Large Language Models. ICLR 2024.
- Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., Lyu, Q., Hendryx, S., Kaplan, R., Lunati, M., and Yue, S. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. NeurIPS 2024.
- Jacovi, A., Caciularu, A., Goldman, O., and Goldberg, Y. (2023). Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks. EMNLP 2023.
Response to Reviewer MXMK (1/5)
We sincerely thank the reviewer for the careful review and detailed comments, which have helped us improve the paper. We have addressed each concern below and uploaded a revised paper, with all changes highlighted in blue. We hope these responses help the reviewer better understand our paper and are useful in reassessing it.
W1: Organization and motivation; why Chinese; evidence for the reverse-construction claim
Organization and motivation. We have reorganized the paper around two questions, one per tier (Introduction; Sections 4.1 and 4.2), and the contributions now lead with the two benchmarks (the construction pipeline is the means of building them):
Why Chinese. We do not claim that Chinese is intrinsically harder than English or other languages. We chose Chinese because the seeds were originally written by native speakers, free of translationese, and because Chinese logical-reasoning benchmarks remain relatively scarce (ChLogic, LogiQA). The pipeline is language-agnostic; the difficulty comes from the size of the state space, the coupling of constraints and faithful formalization. We thank the reviewer for raising this point; to make the benchmark accessible beyond Chinese, we will release a verified English version of the dataset (Appendix A.1).