acceptodds
Under review as a conference paper at ICLR 2027

RabbitBench: Evaluating Large Language Models across Diverse Occupations through Occupational Qualification Examinations

Abstract

As large language models (LLMs) increasingly participate in occupational work, there is a growing need to comprehensively evaluate their knowledge and capabilities across diverse occupational domains. However, existing benchmarks still lack comprehensive and robust coverage across a broad range of occupations. Occupational qualification examinations support comprehensive and robust LLM evaluation through expert-designed questions that systematically cover occupational tasks and the knowledge and capabilities they require. Following official Chinese guidelines, we collect and organize accessible and evaluable examinations from 26 types of occupational qualifications to construct RabbitBench. The benchmark spans 8 mid-level and 23 minor-level occupational groups in the Chinese occupational classification system, providing broader coverage than existing benchmarks. It includes single-choice, multiple-choice, true-or-false, and subjective questions, together with high-quality reference answers, enabling the evaluation of multiple aspects of model performance. We evaluate a diverse set of LLMs on RabbitBench. GPT-5.6-Sol and Gemini-3.1-Pro achieve the strongest performance but at substantially higher API costs, while Qwen3.5 shows a favorable cost–performance trade-off. Across occupations, general capability dominates overall performance, but similarly capable models exhibit distinct occupational strengths. Within engineering qualifications, models consistently perform better on knowledge-oriented than application-oriented subjects. Together, these results show that RabbitBench enables broad and fine-grained analysis of occupational capabilities beyond aggregate model scores. An anonymous project page is available at https://rabbitbench.github.io/.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Occupational Evaluation, Qualification Examinations, Benchmarking, Vertical Domain Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rQF5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RabbitBench contains 26,723 questions from 26 Chinese occupational qualification examinations, covering eight mid-level and 23 minor occupational groups. The authors evaluate eight models on all objective questions and 507 subjective questions. They analyze occupational performance, knowledge- versus application-oriented subjects, consistency across three rubric-and-judge systems, and API costs.

Critical strengths
  1. The mapping from qualifications to occupational categories and examination subjects makes the benchmark useful for identifying weaknesses in specific professional areas. This organization gives the dataset practical value beyond its size.
  2. The analysis goes beyond aggregate rankings. Across seven engineering qualifications, all eight models score higher on knowledge-oriented subjects. The three scoring systems also produce highly consistent overall model rankings, supporting the stability of the relative comparisons.
Critical weaknesses
  1. Data validation is difficult to assess quantitatively. Appendix B describes manual review by three team members, but gives no sample size or measured error rates. It also lacks a detailed source and examination-year inventory. Potential overlap between public past papers and model training data is not assessed in the reported experiments. These reporting gaps leave uncertainty about data reliability and generalization; they do not establish that contamination occurred.
  2. The subjective evaluation needs clearer sampling and validation. The selection rule for the 507 questions is unspecified, and the subset is concentrated in engineering and law. Agreement across three scoring systems supports ranking stability, but provides no direct assessment of rubric validity or agreement with domain-expert judgments. A targeted expert check would help assess the professional validity of these scores.
Questions for authors
  1. Can you provide a source and examination-year inventory and a quantitative summary of the existing manual checks? Is there any available evidence concerning training overlap?
  2. How were the 507 subjective questions selected, and how do they compare with the full pool of 1,204? Has any sample of rubrics and scoring decisions been checked by domain experts?
  3. How were the comparison benchmarks in Table 1 mapped to occupational categories? Please provide the mappings and common counting rules so that the coverage comparison can be checked.
  4. Does "Occupational Fundamentals" in Table 9 include the 931 shared public-foundation questions? Are these questions included in the qualification-level scores in Figure 4? Please clarify the inclusion rules.
Reviewer_hzd8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RabbitBench, an exam-based benchmark for evaluating LLMs across diverse occupational domains. It contains 26,723 questions from 26 Chinese professional qualifications, covering 8 mid-level and 23 minor-level occupational groups. The dataset includes both objective and subjective questions to assess occupational knowledge and capabilities.

The authors evaluate eight LLMs and analyze their performance across occupations, question types, and API costs. The authors also find that models with similar overall scores show different occupational strengths and generally perform better on knowledge-oriented than application-oriented subjects. For subjective questions, the authors use rubric-based scoring and validate its consistency through three LLM-based evaluation systems.

Critical strengths
  1. Broad Occupational Coverage
    RabbitBench contains 26,723 questions from 26 professional qualifications, covering 8 mid-level and 23 minor-level occupational groups. This provides broader occupational coverage than existing exam-based benchmarks.

  2. Comprehensive Evaluation
    The benchmark includes both objective and subjective questions. The authors evaluate eight LLMs and analyze their performance across different occupations, subjects, and API costs, providing a detailed comparison of model capabilities.

  3. Useful Occupational Insights
    The analysis shows that models with similar overall performance can have different strengths across occupations. This highlights the value of occupation-level evaluation beyond overall benchmark scores.

Critical weaknesses
  1. Limited Novelty
    RabbitBench expands the occupational coverage and dataset size compared to existing benchmarks like QualBench. However, its main contribution is largely an expansion of existing datasets, without introducing new evaluation methods or assessment dimensions.

  2. Potential Data Contamination
    RabbitBench uses publicly available exam materials and online question banks, which may overlap with LLM pre-training data. The authors should discuss this risk and conduct experiments to examine its impact on model performance and rankings.

  3. Limited Knowledge–Application Analysis
    The authors find that models perform better on knowledge-oriented subjects than on application-oriented ones. However, application-oriented questions often involve more complex calculations and reasoning, making them inherently harder. Therefore, the lower accuracy may simply reflect differences in question difficulty rather than a specific weakness in applying knowledge. Without controlling for difficulty, this conclusion is not well supported. Moreover, the finding is mainly discussed in Section 4.4, with limited further analysis, making it less convincing as a major contribution.

  4. Lack of Human Validation
    The authors report high agreement among three LLM-based scoring systems for subjective questions. However, agreement does not necessarily ensure scoring accuracy. The authors could have domain experts independently grade a sample of questions (e.g., 100) and compare their scores with those of the LLM judges to further validate the evaluation.

Questions for authors
  1. Beyond broader occupational coverage and a larger dataset, what new insights or evaluation capabilities does RabbitBench provide compared to existing benchmarks such as QualBench?

  2. Have the authors investigated potential overlap between the collected exam questions and LLM pre-training data? How might such contamination affect model performance and rankings?

  3. How do the authors distinguish the effects of question difficulty from differences in knowledge application ability? Could the lower accuracy on application-oriented subjects simply reflect their greater difficulty?

Reviewer_GSDrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

RabbitBench is an exam-based benchmark constructed from Chinese occupational qualification examinations. It contains 26,723 questions spanning 26 qualification types, including objective questions and constructed-response questions. The authors evaluate eight LLMs, use exact-match scoring for objective questions and LLM-generated rubrics for subjective responses, and analyze occupational performance profiles, knowledge-versus-application differences, and API cost–performance trade-offs. The stated motivation is to provide broader and more fine-grained coverage of occupational knowledge and capabilities than existing benchmarks.

Critical strengths

The benchmark represents a substantial data-collection and organization effort. Its coverage of multiple qualification types and inclusion of different response formats could make it a useful resource for evaluating Chinese occupational-examination knowledge.

The authors also provide a concrete comparison of occupational coverage with existing exam-based benchmarks. This helps articulate the intended resource contribution, even though broader coverage does not by itself establish that the benchmark measures a new or practically important capability.

The subjective evaluation includes a cross-system consistency check rather than relying solely on one rubric-and-judge combination. The reported agreement provides evidence that aggregate scores and rankings are not entirely dependent on a single scoring system.

Critical weaknesses

The purpose and contribution of the benchmark are not yet clear.

The main contribution is the expansion and organization of occupational examination coverage. However, the paper does not convincingly demonstrate what important capability distinction becomes measurable through this expansion that existing exam-based evaluations fail to capture.

The manuscript’s own comparison identifies closely related resources, including AGIEval, QualBench, and KMMLU-Pro/Redux. Relative to these benchmarks, the evidence primarily establishes broader coverage under the selected occupational taxonomy. Trusting a new benchmark is costly, however, and I could not identify when one should adopt this one.

Some errors are not reproducible, and the evaluation protocol may be at fault.

Several examples raise questions. Take the item asking who is liable when a police officer kills someone off duty: I put this to the web version of GPT and it gave the correct answer without searching. On the “police” item, common sense does not require that an institution always be liable for a killing, so the model is not wrong either. At least in the web-GPT environment it does not answer incorrectly; the difference may come from the system prompt.

It is possible that the protocol does not make clear that this is a professional examination, or is actively misleading. The paper also does not give the content of the prompt template; I suggest adding it.

The police-officer compensation example is analyzed as a failure to distinguish official conduct from a personal act. However, for a diagnostic claim, the authors should explain how the presented wording uniquely fixes the intended relationship between the assault and the exercise of official authority. The question is not simply whether a model matches the examination key, but whether a plausible alternative interpretation has been excluded. This observation does not establish that the key is wrong; it identifies a missing validation step before drawing a strong conclusion about reasoning capability.

The tour-guide example similarly depends on a particular occupational meaning of “one post, two responsibilities.” The manuscript itself distinguishes that meaning from another interpretation used by the models. This is evidence of a source-specific knowledge discrepancy, but it does not establish inability to reason correctly when the relevant definition is supplied.

The benchmark would benefit from a clearer separation among missing knowledge, missing contextual specification, ambiguity, and genuine reasoning errors. The current illustrative cases do not establish how frequently these explanations occur across the dataset.

Forbidding search weakens the practical relevance.

Today’s large models would normally search the web for questions of this kind unless the answer is obviously self-contained, so a search-free protocol weakens the practical relevance of the setting. Separately, the authors use one response per question. This is a defensible choice for estimating single-attempt performance over a large dataset, but it is insufficient for interpreting selected failures as stable model weaknesses. The error-analysis examples would be more informative if evaluated under repeated sampling and controlled clarification of the relevant assumptions.

Presentation.

For Table 2, I suggest a two-tier leaderboard bar chart. For Figure 2, I suggest annotating the numbers; at present no clear pattern is apparent. The writing overall is average, but the authors may not be native English speakers, and the main contribution of a benchmark paper is the dataset rather than the prose, so this is a minor issue.

Questions for authors
  1. (Important) Please provide the prompt template.

  2. What concrete capability distinction or practically useful model-selection conclusion does RabbitBench establish beyond the closest existing occupational examination benchmarks? Is there evidence beyond the number of covered categories and questions?

  3. As noted under weaknesses, some of the errors are not reproducible: the web version of ChatGPT, for example, does not make them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.