- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents OmniEduBench, a Chinese dataset made up of two parts: a set of questions that test knowledge and reasoning, and a set of questions that test pedagogical judgement. The knowledge and reasoning questions are inspired by datasets like GSM8k, MMLU, HumanEval, but in Chinese. The motivation for this is that English benchmarks, and Chinese benchmarks that contain English questions translated to Chinese, do not capture linguistic and cultural nuances, and fail to evaluate Chinese-specific knowledge. Evaluations are comprehensive, showing results on a variety of open-weights and proprietary models, as well as three different models used as LLM-as-a-judge.
Critical strengths
The paper is easy to follow, with clear writing and nice figures. A lot of work was done in terms of filtering questions using humans and models. Evaluation is comprehensive, featuring a diverse set of 15 language models, comprised of English and Chinese, proprietary and open-weight models. I think the problem of understanding whether language models exhibit biases, and uncovering overlooked domains that language models may be weak on is of great importance.
Critical weaknesses
I think the motivation isn't well enough explained: the authors discuss potential gaps between translated English benchmarks and Chinese nuances and linguistic properties, but I don't see any concrete examples of this. I would have liked to see some examples of problems that make sense in English but not in Chinese, or examples where models miss some nuance.
More generally, I do not see any examples of actual questions from the dataset presented in the paper or appendix. The paper states that the data comes from private and public sources, but is vague on more precise details. I also have some concerns regarding the LLM-generated data. Initially, 44% of the questions are LLM-generated, while after LLM filtering and further filtering by 50 experts, they make up only 20.8% of the final benchmark. It would be helpful to understand the reason for this large decrease, and whether it suggests differences in quality between generated and collected questions. Moreover, many questions are graded using LLM-as-a-judge, and while the different judges produce similar model rankings on average, the absolute scores vary considerably. I am also not sure I understand lines 464-465: how is GPT-4o determined to be a worse scoring model when human ground-truth labels are not available? Given that LLMs are used both to generate some of the questions and to judge some of the answers, I am concerned about how much noise this may introduce into the benchmark.
Lastly, I think the evaluation would benefit from including more recent models. Several of the strongest models evaluated, including GPT-4o, Gemini 2.5 Pro, Claude Sonnet 4, and Qwen3, are already relatively old. Given the fairly high scores reported, with Gemini 2.5 Pro already scoring above 65% on average on both components, it would be useful to evaluate newer models to better understand how much headroom remains in the benchmark.
Minor:
The reported numbers are inconsistent: in the main text: 18,121 + 6,481 = 24,602, whereas in Table 1: 18,179 + 6,387 = 24,566
The main results table, Table 4, is split into averages for each part of the benchmark, it would've been nice to see an average across the entire benchmark for every model.
Questions for authors
- Would it be possible to see concrete examples of Chinese linguistic or cultural nuances that translated English benchmarks fail to capture, as well as representative questions from the dataset?
- Why does the fraction of LLM-generated questions decrease from 44% to 20.8% after filtering? Given the variation across LLM judges, how much noise might LLM generation and judging introduce? How is GPT-4o determined to be a worse judge without ground truth labels?
- Given that Gemini 2.5 Pro already scores above 65%, how much headroom remains for current frontier models, i.e. how would more recent frontier models do on the benchmarks?