acceptodds
Withdrawn from ICLR 2027

OmniEduBench: A Chinese Dataset for Evaluating Subject Knowledge and Pedagogical Judgment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating large language models (LLMs) for educational use requires distinguishing subject knowledge from judgment about how to respond to learners. Knowledge-oriented assessments provide limited evidence about such pedagogical judgment, motivating complementary evaluation in Chinese educational contexts. We introduce OmniEduBench, a Chinese dataset for evaluating subject-knowledge problem solving and scenario-based pedagogical judgment. It comprises approximately 24.6K question–answer pairs spanning 41 academic subjects and 20 pedagogical competencies. The knowledge component covers 11 question formats across multiple educational stages, while the pedagogical component primarily uses multiple-choice scenarios involving instructional guidance, feedback, and learner support. The dataset combines public educational resources, private institutional materials, and model-generated scenarios, followed by cleaning, model-based selection, and human review. We evaluate 15 LLMs and analyze the sensitivity of open-ended answer scoring to the choice of LLM judge. Results show substantial variation across subjects and pedagogical categories, while evaluator choice materially affects absolute scores. Findings highlight the need for scoring criteria and evaluator validation when interpreting educational benchmarks. OmniEduBench provides a resource for diagnosing subject-knowledge and pedagogical-judgment limitations, with pedagogical scores interpreted as scenario-based judgments rather than direct measures of teaching effectiveness.

Primary area:
datasets and benchmarks
Keywords:
Chinese educational benchmark, Subject knowledge, Pedagogical judgment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8DksOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents OmniEduBench, a Chinese dataset made up of two parts: a set of questions that test knowledge and reasoning, and a set of questions that test pedagogical judgement. The knowledge and reasoning questions are inspired by datasets like GSM8k, MMLU, HumanEval, but in Chinese. The motivation for this is that English benchmarks, and Chinese benchmarks that contain English questions translated to Chinese, do not capture linguistic and cultural nuances, and fail to evaluate Chinese-specific knowledge. Evaluations are comprehensive, showing results on a variety of open-weights and proprietary models, as well as three different models used as LLM-as-a-judge.

Critical strengths

The paper is easy to follow, with clear writing and nice figures. A lot of work was done in terms of filtering questions using humans and models. Evaluation is comprehensive, featuring a diverse set of 15 language models, comprised of English and Chinese, proprietary and open-weight models. I think the problem of understanding whether language models exhibit biases, and uncovering overlooked domains that language models may be weak on is of great importance.

Critical weaknesses

I think the motivation isn't well enough explained: the authors discuss potential gaps between translated English benchmarks and Chinese nuances and linguistic properties, but I don't see any concrete examples of this. I would have liked to see some examples of problems that make sense in English but not in Chinese, or examples where models miss some nuance.

More generally, I do not see any examples of actual questions from the dataset presented in the paper or appendix. The paper states that the data comes from private and public sources, but is vague on more precise details. I also have some concerns regarding the LLM-generated data. Initially, 44% of the questions are LLM-generated, while after LLM filtering and further filtering by 50 experts, they make up only 20.8% of the final benchmark. It would be helpful to understand the reason for this large decrease, and whether it suggests differences in quality between generated and collected questions. Moreover, many questions are graded using LLM-as-a-judge, and while the different judges produce similar model rankings on average, the absolute scores vary considerably. I am also not sure I understand lines 464-465: how is GPT-4o determined to be a worse scoring model when human ground-truth labels are not available? Given that LLMs are used both to generate some of the questions and to judge some of the answers, I am concerned about how much noise this may introduce into the benchmark.

Lastly, I think the evaluation would benefit from including more recent models. Several of the strongest models evaluated, including GPT-4o, Gemini 2.5 Pro, Claude Sonnet 4, and Qwen3, are already relatively old. Given the fairly high scores reported, with Gemini 2.5 Pro already scoring above 65% on average on both components, it would be useful to evaluate newer models to better understand how much headroom remains in the benchmark.

Minor:

The reported numbers are inconsistent: in the main text: 18,121 + 6,481 = 24,602, whereas in Table 1: 18,179 + 6,387 = 24,566

The main results table, Table 4, is split into averages for each part of the benchmark, it would've been nice to see an average across the entire benchmark for every model.

Questions for authors
  1. Would it be possible to see concrete examples of Chinese linguistic or cultural nuances that translated English benchmarks fail to capture, as well as representative questions from the dataset?
  2. Why does the fraction of LLM-generated questions decrease from 44% to 20.8% after filtering? Given the variation across LLM judges, how much noise might LLM generation and judging introduce? How is GPT-4o determined to be a worse judge without ground truth labels?
  3. Given that Gemini 2.5 Pro already scores above 65%, how much headroom remains for current frontier models, i.e. how would more recent frontier models do on the benchmarks?
Reviewer_qbgAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce OmniEduBench, a Chinese dataset for evaluating subject-knowledge problem solving and scenario-based pedagogical judgment.

It assesses two aspects of LLM performance: educational knowledge understanding and teaching-related capabilities. The knowledge component covers 11 question types across multiple educational stages, while the pedagogical component mainly uses multiple-choice scenarios involving instructional guidance, feedback, and learner support. The authors evaluate 15 LLMs and observe varied performance on OmniEduBench, suggesting its usefulness in distinguishing model capabilities.

Critical strengths
  1. Simple, intuitive and easy-to-understand but reasonable research logic.
  2. Fluent and natural writing.
Critical weaknesses
  1. Limited distinction from prior work and insufficient support for some contribution claims

1.1. Existing benchmarks already cover diverse subjects, question formats, and educational capabilities**

The authors claim that existing education benchmarks often cover a single subject or question type and overlook cultivation. However, Edu-Values [1] evaluates subject knowledge, educational knowledge and skills, professional ethics, and educational philosophy across five question formats. MathDial [2] provides teacher–simulated-student dialogues annotated with pedagogical strategies. MRBench [3] assesses tutoring responses through mistake identification, mistake location, guidance, and actionability, among other dimensions.

[1] Edu-Values

[2] MathDial

[3] MRBench

1.2. The cultivation assessment does not fully support the claimed capabilities**

The cultivation component mainly uses multiple-choice questions. These cannot directly establish whether models can provide appropriate feedback, sustain instructional guidance, or improve student learning. Yet the main paper makes broader claims about assessing models’ ability to support holistic educational objectives. These claims exceed what the assessment can demonstrate.

  1. Insufficient validation of scoring and data selection raises reliability and fairness concerns

2.1. Lower scores do not establish that a scoring model is less accurate**

In the final paragraph of Section 4.3, the authors report that GPT-4o assigns lower overall scores to all five models in Table 6 and conclude that it performs poorly as a scoring model. However, scoring quality should be assessed against a trustworthy reference, rather than by score levels. Although the appendix mentions human verification, it provides insufficient quantitative evidence that GPT-4o is less accurate than the other judges. This leaves the score differences unexplained and weakens confidence in the results.

2.2. Model-based filtering may introduce selection bias**

The authors retain questions answered incorrectly by QwQ and Qwen3, potentially biasing the dataset toward these models’ weaknesses. Although acknowledged in the limitations, the impact on the data distribution and model rankings is not adequately assessed.

Questions for authors
  1. How were the six dimensions within knowledge and cultivation determined, and what theory supports them? The paper mentions the Ministry of Education’s undergraduate program catalogue, expert discussions, and reference materials, but their connection to the final categories and the assignment of subjects or competencies remains unclear.

  2. Were the 50 master’s students and 5 experts trained before verification? How did you ensure that they understood and consistently applied the five criteria? For the experts’ review of the 15% sample from each data source, what rubric and rating scale were used? Which inter-rater agreement metric is reported in Table 2, and how was it calculated?

  3. Please provide the exact model/API versions, and reasoning modes you used. Some evaluated models are older versions of their families; newer versions may perform better on OmniEduBench.

  4. Why was HARD constructed using 11 of the 15 evaluated models, excluding the other four? How were their results combined to determine sample difficulty?

  5. Please clarify the inconsistent dataset counts. In Table 1, 18,179 knowledge samples plus 6,387 cultivation samples total 24,566, not 24,602. The main text instead reports 18,121 + 6,481 = 24,602, and both sets of counts appear in Table 1.

  6. Could you provide more detailed case studies and error analyses? Examining the types of errors different models make, beyond reporting accuracy, would help demonstrate what the benchmark reveals about their capabilities.

Reviewer_iswROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OmniEduBench, a Chinese educational benchmark designed to evaluateLLMs across two dimensions: subject-matter knowledge and pedagogical judgment. The benchmark includes 24,602 question-answer pairs, covering 41 subject areas, 20 teaching competencies, and 11 question formats. The dataset was constructed from publicly available educational resources, internal institutional materials, and teaching scenarios generated by a large language model, and underwent data cleaning, dual-model screening, and manual verification. The authors evaluated 15 large language models under different prompt settings, constructing a more challenging subset, “OmniEduBench HARD,” and explored the sensitivity of the evaluation results to different large model judges.

Critical strengths

The benchmark covers a wide range of academic subjects, educational levels, and question formats, while incorporating pedagogical judgment alongside traditional knowledge assessment. This multidimensional design provides a potentially useful resource for analyzing different aspects of LLM capabilities in Chinese educational contexts.

Critical weaknesses

The novelty of this paper compared to existing educational benchmarks (such as EduBench, published at ACL 2026) have not yet been fully demonstrated. More importantly, Section 3.2 states that the dual-model filtering process only retains questions for which both QwQ-32B and Qwen3-235B answered incorrectly. However, Table 4 shows that these models achieved surprisingly high accuracy on the benchmark set constructed from them (QwQ: 53.87% for knowledge class and 70.27% for training class. Qwen3-235B: 40.82% for knowledge class and 63.74% for training class). Without explaining potential differences in prompts, scoring processes, or subsequent modifications after filtering, these results are unlikely to align with the construction process claimed in the paper. Furthermore, filtering problems only based on the errors of two specific models may introduce model dependency selection bias rather than truly reflecting the inherent difficulty of the task itself. There are also unexplained inconsistencies in the dataset statistics: Table 1 shows that there are 18121 knowledge-related questions and 6481 training-related questions, but the actual total number of questions in each category is 18179 and 6387 respectively. It is noteworthy that the average accuracy reported in Table 4 matches the weighted average calculated using the number of inconsistent categories, which makes me question the actual sample size being evaluated. Furthermore, the expert verification process lacks sufficient detail, failing to explain the qualifications of the annotators, the method for calculating consistency among raters, and the verification process for the gold answer. The scoring process assisted by the LLM also lacks independent human calibration, although Table 6 shows significant differences in scoring results among different evaluators. Finally, although the project has a public GitHub repository, it only contains a placeholder for "Code come soon" and does not provide benchmark data or evaluation code, which further limits the reproducibility of the research.

Questions for authors
  1. Section 3.2 states that only questions incorrectly answered by both QwQ-32B and Qwen3-235B were retained. Why do these same models still achieve relatively high accuracies in Table 4? Were different prompts, scoring methods, or model configurations used?
  2. Why do the category counts in Table 1 not match the reported dataset totals? Which sample counts were actually used to calculate the results in Table 4?
  3. How were the 50 master students and 5 senior experts selected and assigned to different subjects? How was inter-rater agreement calculated, and how were disagreements resolved?
  4. How were the reference answers for LLM-generated pedagogical scenarios independently validated, particularly when multiple teaching strategies could be considered appropriate?
  5. How was the accuracy of the LLM-assisted scoring method validated against human judgments, especially given the substantial score differences across judges in Table 6?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.