In Sections 1 and 2.2, the paper motivates LIUYAO-Bench as a way to distinguish domain knowledge from rule application. How does this evaluation differ substantively from existing benchmarks such as LAiW and RuleArena, which already assess different stages of expert reasoning and rule application? What specific capability can LIUYAO-Bench measure that these benchmarks cannot?
In Sections 1 and 3.1, what makes Liuyao particularly suitable for evaluating rule-based reasoning? Given that other domains, such as legal reasoning, taxation, and formal rule systems, also require the application of interacting rules, what evaluation advantage does Liuyao provide that justifies its selection?
In Sections 3.1 and 3.2, the benchmark defines correctness through agreement with source-derived interpretations. How do the authors establish that a given judgment is uniquely determined by the domain rules rather than being one of several defensible interpretations? How are alternative but internally consistent interpretations handled when assigning binary labels?
In Section 4.2 and Appendix A.3 (Table 11), the two practitioners disagree on 10 of the 50 Full-case items, and both practitioners agree with each other but disagree with the benchmark labels on 20 of the 300 evaluated items. Have these disagreements been independently examined to distinguish annotation errors, differences in interpretive conventions, and genuine ambiguity? What evidence supports retaining the original labels in these cases?
In Section 4.2, what qualifications, experience, and interpretive traditions do the practitioners represent? Which source materials and annotation guidelines were available to them, and how was consistency in their use of domain rules established? Could the authors provide the annotation instructions and the criteria used by the third practitioner to resolve disagreements?
In Section 3.2 (Instance Construction and Quality Control), could the authors clarify the complete annotation pipeline, including how positive statements were extracted, how negative candidates were generated, and what proportion of items underwent independent manual verification? What procedures ensured that changing a candidate judgment produced an unambiguously incorrect statement rather than another plausible interpretation?
In Section 3.2 (Negative Candidate Construction), could the authors evaluate whether a model can predict the labels using only the candidate statements without the case context? Since negative examples are created by modifying source-supported statements, how do the authors rule out linguistic shortcuts or systematic differences between positive and negative candidates?
In Sections 3.1 and 5.2, the hierarchy evaluates separate binary judgments rather than successive steps in a reasoning process. What additional diagnostic information does this design provide beyond reporting accuracy on different subtasks? In particular, how can the benchmark identify which rule or intermediate judgment is responsible for an incorrect final decision?
In Section 4.1, Appendix B, and Table 2, what exact model versions, decoding parameters, thinking configurations, and output-token budgets were used for each API model? Given that each model contributes a single finalized prediction set, how stable are the reported accuracies and rankings across repeated inference runs and reasonable prompt variations?
In Section 5.1 and Appendix A.4 (Tables 14 and 15), the leading model changes depending on whether micro accuracy, three-layer macro accuracy, or balanced accuracy is used. Which aggregation metric best represents the intended evaluation objective, and why? Can the authors provide systematic paired comparisons between models to establish which reported performance differences are statistically distinguishable?
In Section 5.5 (Table 4) and Appendix E.1, CoT improves overall Qwen3-8B accuracy but produces almost no improvement on Strength and Dynamics. What explains this difference across subtasks? Since CoT uses a larger output budget and retries some initially unparseable responses with an increased budget, can the authors isolate the effect of reasoning prompts from output-budget and retry effects?
In Section 5.5 and Appendix E.4 (Table 21), CoT improves Qwen3-8B accuracy on the IID subset from 56.86% to 65.20%, but reduces accuracy on the source-held-out OOD subset from 55.47% to 51.46%. Is this reversal statistically reliable, and what differences between the evaluation subsets account for it? Does the result indicate a systematic limitation of CoT or sensitivity to the particular source distribution?
In Section 4.3 and Appendix E.3–E.4, QLoRA fine-tuning produces substantial improvements using answer-only binary supervision. What evidence shows that this improvement reflects better domain-rule application rather than adaptation to the benchmark's label distribution, candidate construction patterns, or other dataset-specific characteristics? Would the improvement persist on newly constructed cases with different candidate formulations?
Finally, in Sections 5 and 6, what concrete conclusions about LLM reasoning can be drawn from the benchmark beyond differences in classification accuracy? In particular, which experimental findings demonstrate that models possess relevant rule knowledge but fail specifically to apply or integrate it, rather than being affected by response bias, source familiarity, or task difficulty? What evidence supports interpreting LIUYAO-Bench as a diagnostic benchmark for rule application rather than a collection of domain-specific verification tasks?