acceptodds
Under review as a conference paper at ICLR 2027

LIUYAO-Bench: A Hierarchical Benchmark for Expert-Domain Verification in Classical Chinese Texts

Abstract

Liuyao is a rule-governed expert-text tradition in classical Chinese where judgments depend on hexagram configurations, calendrical context, line relations, and transformations. Evaluating large language models (LLMs) in this domain requires examining both their knowledge of individual rules and their ability to verify interpretations of specific cases. We introduce LIUYAO-BENCH, a hierarchical benchmark containing 3,060 binary verification instances drawn from four classical Liuyao texts. The benchmark organizes evaluation into three layers: (1) basic knowledge, covering concepts, relations, and rule statements; (2) intermediate-state interpretation, covering yongshen selection, line strength, and dynamic line relations; and (3) full-case judgment verification, checking candidate final judgments against structured case records. All three layers ask models to assess a supplied candidate, allowing comparison across different scopes of verification. Across ten contemporary LLMs, the best model reaches 79.77% overall accuracy, while dynamic relations and complete cases remain difficult. On a balanced 300-item subset, two practitioners achieve 88.67% mean accuracy, compared with 79.00% for the best model on the same items. Both practitioners outperform all evaluated models on Dynamics and Full-Case. Additional experiments with Qwen3-8B show that chain-of-thought prompting improves overall benchmark accuracy, while fine-tuning improves overall accuracy on a held-out source book; full-case performance remains limited in both settings. LIUYAO-BENCH provides a structured setting for examining the gap between local rule knowledge and case-level verification in a classical Chinese expert domain.

Primary area:
datasets and benchmarks
Keywords:
large language models, benchmarking, expert-domain reasoning, classical Chinese, hierarchical evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q26UOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LIUYAO-Bench contains 3,060 binary verification items drawn from classical Chinese liuyao texts, covering basic knowledge, intermediate-state judgments, and complete-case judgments. The paper evaluates ten LLMs and compares them with practitioners on a balanced subset of 300 items. It also studies prompting and fine-tuning with Qwen3-8B, including evaluation on a held-out source book. Models perform better on local knowledge than on complete cases, show substantial differences between positive and negative items, and benefit unevenly from adaptation.

Critical strengths
  1. The benchmark covers an underrepresented textual domain and clearly defines its target as agreement with historical interpretive conventions, without claiming real-world predictive validity.

  2. The intermediate tasks offer useful detail about domain-specific judgments.

  3. The paper also recognizes that correct binary answers alone do not establish faithful reasoning.

Critical weaknesses
  1. The Full-Case task combines judgments about rule compatibility with recovery of events recorded in the source texts.

  2. Disagreements with the benchmark labels need closer examination against the source texts. Both practitioners give the same answer that differs from the benchmark on 20 of 300 items, including seven of 50 Full-Case items.

  3. It remains unclear how much of the benchmark requires the supplied case context.

  4. The evaluation is not documented fully enough to reproduce the main model comparison.

Questions for authors

Please refer to the Weakness Section.

Reviewer_3gX5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies source-based binary verification in classical Chinese liuyao texts. It introduces LIUYAO-Bench, containing 3,060 items across basic knowledge, intermediate-state interpretation, and full-case judgment. Experiments evaluate ten LLMs, compare practitioners on 300 matched items, and test Qwen3-8B prompting and fine-tuning. Results reveal substantially weaker case-level performance than local knowledge verification and uneven gains from adaptation.

Critical strengths
  • Source-based verification in classical Chinese liuyao is a meaningful evaluation target that broad language benchmarks do not directly cover.
  • The paper provides a structured benchmark of 3,060 verification items drawn from four books, with task categories, serialized case fields, examples, and a terminology glossary.
  • Ten LLMs are evaluated, practitioners answer the same balanced 300-item subset, and source-grouped fine-tuning tests.
Critical weaknesses
  • Some Full-Case labels follow recorded outcomes, but the question, date, and hexagram fields are not shown to determine those outcomes under explicit textual rules. The task may therefore mix rule verification with recall of omitted narratives, weakening the interpretation of low accuracy as reasoning failure.

  • No candidate-only, shuffled-context, or lexical baseline tests whether scores require the intended case evidence, leaving shortcut-based explanations unresolved, especially given the large positive-negative performance gaps.

  • The 444 matched cases do not establish that intermediate statements are necessary steps for final judgments (Appendix A.5). Correct intermediate answers followed by wrong final answers therefore cannot isolate compositional reasoning failure.

Questions for authors
  • For Full-Case items, what evidence shows that each label can be determined from the released fields without remembering the original narrative?
  • What accuracies do candidate-only, shuffled-context, and simple lexical baselines achieve by subtask and polarity? Do paired positive/negative edits and paraphrases preserve the ranking and the Full-Case gap?
  • Can the three intermediate items be checked for actual relevance to the final judgment before interpreting their joint success or failure?
Reviewer_eUu6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LIUYAO-Bench, a benchmark of 3,060 binary verification questions derived from four classical Chinese texts on Liuyao, a traditional rule-based divination system. The benchmark organizes evaluation into three layers: basic knowledge of concepts and rules, intermediate-state interpretation (yongshen selection, line strength, and dynamic relations), and full-case judgment verification. Each task requires models to determine whether a given statement is consistent with source-derived rules or recorded interpretations.

The authors evaluate ten LLMs and compare their performance with two human practitioners on a shared subset of 300 questions. They also examine the effects of chain-of-thought prompting and QLoRA fine-tuning using Qwen3-8B.

The results show that models generally perform better on basic knowledge questions than on full-case judgments. The highest overall model accuracy is 79.77%, while the two practitioners achieve an average accuracy of 88.67% on the shared subset, compared with 79.00% for the best model. Chain-of-thought prompting improves overall accuracy but provides limited benefits for some intermediate tasks, while fine-tuning improves performance on both in-distribution and source-held-out evaluations.

Critical strengths

The paper introduces 3,060 verification instances derived from four classical Chinese texts, covering domain knowledge, intermediate-state judgments, and complete cases. The structured evaluation tasks and source-linked cases provide a useful resource for studying how LLMs handle specialized rules and context-dependent judgments. The inclusion of human practitioner evaluations further strengthens the dataset's potential value as a research resource.

The experiments reveal substantial variation in performance across the subtasks, particularly between basic knowledge and full-case verification. The large asymmetry between positive and negative accuracy for several models is especially informative, as it shows that overall accuracy can obscure important differences in verification behavior. The prompting and fine-tuning experiments also demonstrate that performance improvements are not uniform across tasks.

Critical weaknesses

The benchmark defines correctness through agreement with interpretations or outcomes recorded in the source texts, but it does not establish that these labels can be uniquely derived from the provided case information and domain rules. This is particularly concerning for Full-case judgments, where multiple interpretations may be defensible. In the practitioner evaluation, both experts agreed with each other but disagreed with the benchmark labels on 20 of 300 items, while their agreement on Full-case questions was only 80%. These results do not necessarily indicate incorrect labels, but they raise important questions about their reliability and the treatment of alternative interpretations. Without stronger independent validation of the labels and their supporting evidence, it remains unclear whether incorrect model predictions reflect failures in domain reasoning or disagreement with a particular source interpretation. This uncertainty directly affects the validity of the benchmark's central evaluation target.

The paper motivates LIUYAO-Bench by arguing that final-answer accuracy cannot reveal where models struggle with domain knowledge and rule application. However, the proposed hierarchy evaluates independently presented binary judgments rather than connected steps in a reasoning process. Consequently, lower accuracy on Full-case questions does not establish that models understand individual rules but fail to apply or combine them. Although the authors analyze 444 cases linked across intermediate and final tasks, they do not establish that the evaluated intermediate judgments are necessary for the corresponding final decisions. Moreover, substantial differences between positive and negative accuracy suggest that response tendencies may also influence the observed performance gaps. The experiments demonstrate variation across tasks and models, but do not sufficiently establish what underlying capabilities or failure mechanisms these differences represent. This substantially weakens the benchmark's claimed diagnostic value and its contribution beyond existing fine-grained evaluation approaches.

Questions for authors

In Sections 1 and 2.2, the paper motivates LIUYAO-Bench as a way to distinguish domain knowledge from rule application. How does this evaluation differ substantively from existing benchmarks such as LAiW and RuleArena, which already assess different stages of expert reasoning and rule application? What specific capability can LIUYAO-Bench measure that these benchmarks cannot?

In Sections 1 and 3.1, what makes Liuyao particularly suitable for evaluating rule-based reasoning? Given that other domains, such as legal reasoning, taxation, and formal rule systems, also require the application of interacting rules, what evaluation advantage does Liuyao provide that justifies its selection?

In Sections 3.1 and 3.2, the benchmark defines correctness through agreement with source-derived interpretations. How do the authors establish that a given judgment is uniquely determined by the domain rules rather than being one of several defensible interpretations? How are alternative but internally consistent interpretations handled when assigning binary labels?

In Section 4.2 and Appendix A.3 (Table 11), the two practitioners disagree on 10 of the 50 Full-case items, and both practitioners agree with each other but disagree with the benchmark labels on 20 of the 300 evaluated items. Have these disagreements been independently examined to distinguish annotation errors, differences in interpretive conventions, and genuine ambiguity? What evidence supports retaining the original labels in these cases?

In Section 4.2, what qualifications, experience, and interpretive traditions do the practitioners represent? Which source materials and annotation guidelines were available to them, and how was consistency in their use of domain rules established? Could the authors provide the annotation instructions and the criteria used by the third practitioner to resolve disagreements?

In Section 3.2 (Instance Construction and Quality Control), could the authors clarify the complete annotation pipeline, including how positive statements were extracted, how negative candidates were generated, and what proportion of items underwent independent manual verification? What procedures ensured that changing a candidate judgment produced an unambiguously incorrect statement rather than another plausible interpretation?

In Section 3.2 (Negative Candidate Construction), could the authors evaluate whether a model can predict the labels using only the candidate statements without the case context? Since negative examples are created by modifying source-supported statements, how do the authors rule out linguistic shortcuts or systematic differences between positive and negative candidates?

In Sections 3.1 and 5.2, the hierarchy evaluates separate binary judgments rather than successive steps in a reasoning process. What additional diagnostic information does this design provide beyond reporting accuracy on different subtasks? In particular, how can the benchmark identify which rule or intermediate judgment is responsible for an incorrect final decision?

In Section 4.1, Appendix B, and Table 2, what exact model versions, decoding parameters, thinking configurations, and output-token budgets were used for each API model? Given that each model contributes a single finalized prediction set, how stable are the reported accuracies and rankings across repeated inference runs and reasonable prompt variations?

In Section 5.1 and Appendix A.4 (Tables 14 and 15), the leading model changes depending on whether micro accuracy, three-layer macro accuracy, or balanced accuracy is used. Which aggregation metric best represents the intended evaluation objective, and why? Can the authors provide systematic paired comparisons between models to establish which reported performance differences are statistically distinguishable?

In Section 5.5 (Table 4) and Appendix E.1, CoT improves overall Qwen3-8B accuracy but produces almost no improvement on Strength and Dynamics. What explains this difference across subtasks? Since CoT uses a larger output budget and retries some initially unparseable responses with an increased budget, can the authors isolate the effect of reasoning prompts from output-budget and retry effects?

In Section 5.5 and Appendix E.4 (Table 21), CoT improves Qwen3-8B accuracy on the IID subset from 56.86% to 65.20%, but reduces accuracy on the source-held-out OOD subset from 55.47% to 51.46%. Is this reversal statistically reliable, and what differences between the evaluation subsets account for it? Does the result indicate a systematic limitation of CoT or sensitivity to the particular source distribution?

In Section 4.3 and Appendix E.3–E.4, QLoRA fine-tuning produces substantial improvements using answer-only binary supervision. What evidence shows that this improvement reflects better domain-rule application rather than adaptation to the benchmark's label distribution, candidate construction patterns, or other dataset-specific characteristics? Would the improvement persist on newly constructed cases with different candidate formulations?

Finally, in Sections 5 and 6, what concrete conclusions about LLM reasoning can be drawn from the benchmark beyond differences in classification accuracy? In particular, which experimental findings demonstrate that models possess relevant rule knowledge but fail specifically to apply or integrate it, rather than being affected by response bias, source familiarity, or task difficulty? What evidence supports interpreting LIUYAO-Bench as a diagnostic benchmark for rule application rather than a collection of domain-specific verification tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.