- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RabbitBench contains 26,723 questions from 26 Chinese occupational qualification examinations, covering eight mid-level and 23 minor occupational groups. The authors evaluate eight models on all objective questions and 507 subjective questions. They analyze occupational performance, knowledge- versus application-oriented subjects, consistency across three rubric-and-judge systems, and API costs.
Critical strengths
- The mapping from qualifications to occupational categories and examination subjects makes the benchmark useful for identifying weaknesses in specific professional areas. This organization gives the dataset practical value beyond its size.
- The analysis goes beyond aggregate rankings. Across seven engineering qualifications, all eight models score higher on knowledge-oriented subjects. The three scoring systems also produce highly consistent overall model rankings, supporting the stability of the relative comparisons.
Critical weaknesses
- Data validation is difficult to assess quantitatively. Appendix B describes manual review by three team members, but gives no sample size or measured error rates. It also lacks a detailed source and examination-year inventory. Potential overlap between public past papers and model training data is not assessed in the reported experiments. These reporting gaps leave uncertainty about data reliability and generalization; they do not establish that contamination occurred.
- The subjective evaluation needs clearer sampling and validation. The selection rule for the 507 questions is unspecified, and the subset is concentrated in engineering and law. Agreement across three scoring systems supports ranking stability, but provides no direct assessment of rubric validity or agreement with domain-expert judgments. A targeted expert check would help assess the professional validity of these scores.
Questions for authors
- Can you provide a source and examination-year inventory and a quantitative summary of the existing manual checks? Is there any available evidence concerning training overlap?
- How were the 507 subjective questions selected, and how do they compare with the full pool of 1,204? Has any sample of rubrics and scoring decisions been checked by domain experts?
- How were the comparison benchmarks in Table 1 mapped to occupational categories? Please provide the mappings and common counting rules so that the coverage comparison can be checked.
- Does "Occupational Fundamentals" in Table 9 include the 931 shared public-foundation questions? Are these questions included in the qualification-level scores in Figure 4? Please clarify the inclusion rules.