- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EHR-Bench, a Chinese recruitment benchmark comprising 8,336 instances across 12 tasks, six capability families, and five recruitment scenarios. The benchmark is constructed using public resources, synthetic augmentation, and expert-reviewed annotations, with task specifications informed by a private operational dataset. The authors evaluate 12 LLMs, analyze task-specific performance differences, and compare five models against a private operational reference. The paper also releases evaluation protocols, scoring tools, and per-sample results.
Critical strengths
- Comprehensive coverage of recruitment tasks.
The benchmark brings together 12 tasks spanning generation, extraction, classification, similarity, retrieval, and ranking. Evaluating these tasks within a common framework is practically useful for comparing LLMs across recruitment operations.
- Extensive evaluation and documentation.
The authors evaluate 12 models and provide detailed task-level analyses, source-specific comparisons, and additional baselines. The expert annotation process, quality checks, and released evaluation artifacts also support reproducibility and future benchmarking efforts.
Critical weaknesses
- Limited novelty and insufficient motivation for the unified framework.
Many of the individual capabilities evaluated in EHR-Bench have already been extensively studied, including through multilingual recruitment benchmarks and related resources such as ResumeBench, PJB, and WorkRB. In particular, WorkRB already provides a framework for evaluating multiple work-related tasks.
While combining these capabilities into a single Chinese recruitment benchmark has practical value, the paper does not sufficiently explain what new research questions this integration enables or what limitations of existing benchmarks it fundamentally addresses. The main distinction appears to be broader task coverage within one evaluation suite rather than a substantially new evaluation methodology.
The proposed framework's utility beyond the current benchmark also remains unclear. The released tools allow researchers to evaluate additional models on the same tasks, but the paper does not demonstrate how the framework can be extended to new recruitment operations, datasets, or organizational contexts. More importantly, it does not establish what additional insights the unified evaluation provides beyond separately evaluating the individual capabilities.
- Limited generalizable findings and unclear implications of the experimental results.
Despite evaluating 12 LLMs across a wide range of tasks, the primary findings remain largely descriptive. For example, the paper reports that models with similar Overall scores exhibit different task-specific strengths and that performance varies across data sources and candidate pools. While useful, these observations are not particularly surprising for a multi-task benchmark.
The paper would be stronger if it identified more substantive findings about recruitment AI capabilities or explained how the results should change model evaluation and selection practices. At present, the discussion largely reiterates performance differences without developing a sufficiently clear takeaway.
Some results could support more meaningful conclusions but are not fully investigated. For instance, PJF balanced accuracy ranges from 52.01–56.64% on behavior-derived examples versus 83.64–95.22% on constructed examples. This raises an important question about whether constructed benchmarks adequately capture real-world recruitment judgments. However, differences in label definitions and data construction make it difficult to interpret the underlying cause.
Similarly, the public and private benchmarks produce identical Overall rankings for five shared models, but only 65% of task-level pairwise comparisons agree in direction. This limits how strongly the aggregate agreement can support claims about practical model selection.
Overall, the paper presents extensive experimental results without sufficiently establishing which findings constitute new knowledge or why the unified benchmark is necessary to obtain them.
Questions for authors
-
What specific research questions or evaluation capabilities does EHR-Bench enable that cannot be adequately addressed using existing resources such as WorkRB, PJB, and other task-specific recruitment benchmarks? Beyond broader task coverage, what is the central methodological or scientific contribution of the proposed framework?
-
What are the main generalizable findings from evaluating these 12 tasks together? Can the authors demonstrate a concrete model-selection or diagnostic insight that would be missed if the tasks were evaluated independently?
-
How reusable is the framework beyond evaluating additional models on the current dataset? Can the authors demonstrate or provide a concrete account of how it would accommodate new recruitment tasks, organizational settings, or evaluation datasets?
-
How do the main findings change when behavior-derived and constructed instances are analyzed separately? Given the substantial performance differences between these sources, what evidence supports interpreting their combined scores as measuring comparable recruitment capabilities?
3 comments
Response to Reviewer tm73 (2/2)
Q3. Reusability for new tasks, organizations, and datasets
Each task is a directory with data.jsonl, label.json, meta.json, and an optional rubric.md; tasks and capability families are registered in eval/scripts/common.py, prompt builders in prompts.py, and scorers in compute_metrics.py. A new task needs a task directory, a prompt builder, and a scorer entry that can reuse the existing span-F1, PathScore, macro-F1, balanced-accuracy, nDCG, Recall/MRR, and rubric-judge scorers; registering it under a family includes it in that family and Overall. The paper already shows two forms of reuse: the private reference (§6) applies the same specifications, formats, and scorers to an enterprise's private operational data—the route by which an organization evaluates models on its own data—and the public collection was itself extended with new sources (441 JDG instances, 486 PJF pairs) merged through source tags. As a new-task example, visual resume parsing adds a PDF/image input adapter and a structured-field schema, with a scorer adapting the existing precision/recall/F1 utilities to field matching, registered under Extraction. Appendix I describes these steps.
Q4. Behavior-derived vs. constructed instances analyzed separately
The main findings are unchanged. Recomputing Overall with PJF scored on one stratum at a time, the behavior-only variant reproduces the reported order exactly (ρ = 1.00) and the constructed-only variant gives ρ = 0.98, with the same top two models (Appendix F.1, Table 36).
The two strata are a deliberate design. In the constructed stratum, fit follows from explicit evidence in the two texts, testing whether a model reads and compares it correctly; the behavior stratum keeps evaluations and behavior from real recruitment, whose labels carry learnable predictive signal that zero-shot LLMs do not yet exploit—closer to real use and harder. Both draw inputs from the same public source with the same instruction and format; the key difference is pair construction and label evidence.
The combined score is interpretable in the same way as Overall: a fixed-composition summary, with diagnosis read from the strata. That rankings on the two strata are nearly uncorrelated shows they measure two complementary facets of fit judgment (F3). The revision reports both strata for all twelve models (Table 5) and states in §5.3 that current LLMs cannot yet reproduce behavior-derived judgments—an open problem EHR-Bench makes measurable.
Identical public and private Overall rankings, but 65% task-level agreement
We quantified the task-level correspondence; it supports private model selection more strongly than the pooled 65% suggests (§6.2, Figure 5; Appendix G.4, Tables 50–51).
(1) Clear public gaps transfer. The 65% counts all pairs equally, including many near-ties. Among the 118 comparisons not tied publicly, agreement rises with the public gap: 42% below one point, 59% at 1–2, 67% at 2–3, and 88% (37/42) at three points or more. Disagreement concentrates on pairs the public benchmark itself separates only weakly.
(2) The public benchmark narrows the candidate set. Validating the public top two models of each task on the private data and keeping the better one comes within 0.43 points of the private best on average (maximum 1.42), versus 1.10 for two randomly chosen models under the same rule; the shortlist contains the private best on 6 of 12 tasks.
(3) Shared models. The internal data involve candidate privacy and are evaluated only in a compliant environment; these five are all models available there that were also evaluated publicly, and the comparison will extend as more are deployed. Their Overall order is identical (10/10 pairs), and omitting any capability family keeps the public–private rank correlation at 0.90–1.00 (Appendix G.3).
The revision therefore recommends a screen–shortlist–validate protocol: screen with the public Overall, shortlist with task-level scores, and confirm on target operational data (§6.2, §7).
Thank you for the detailed rebuttal and additional analyses. I appreciate the clearer evidence for the benchmark's practical utility, particularly in task-specific model selection.
However, my main concern remains. The findings largely quantify already recognized limitations of multi-task evaluation rather than establish substantially new, generalizable insights about recruitment AI. The constructed–behavior gap is interesting but difficult to interpret given the different label definitions, while the public–private results are limited to five models from one organization.
Although the rebuttal strengthens the paper, I still find its scientific contribution beyond broader benchmark coverage limited. I will therefore maintain my score of 2.
Response to Reviewer tm73 (1/2)
Thank you for recognizing the coverage, experiments, and documentation, and for asking what new understanding the unified evaluation provides. Your review already points to two results that could support stronger conclusions—the behavior-derived vs. constructed gap and the public–private correspondence. We agree, analyzed both further, and base the answers below on these new results. Revised passages are shown in blue in the updated PDF.
Q1. What can EHR-Bench answer that WorkRB, PJB, and other resources cannot?
The difference lies in what is evaluated and in the type of evidence, not only in task count (Table 1). ResumeBench evaluates resume parsing and PJB resume retrieval; WorkRB is an excellent framework for ranking and classification interfaces, mainly for embedding and ranking models. None evaluates LLM generation and judgment outputs, combines behavior-derived with constructed labels, or provides an operational-data reference. EHR-Bench adds JD generation, fit analysis with explanation, query parsing, and resume deduplication, with per-instance provenance and a public–private comparison. It can therefore answer three questions existing resources cannot:
Our methodological contribution is a benchmark construction and validation process constrained by operational data: internal data define task specifications, taxonomy, sampling targets, and scoring; public and controlled-augmentation materials implement them with per-instance provenance; and a public–private comparison on shared models tests validity. To our knowledge, this is the first such process among public recruitment evaluation resources. The introduction and contributions are rewritten around these questions (§1).
Q2. Generalizable findings that separate evaluation would miss
The revision organizes the results into four findings (§5.1–§5.3, §6.2), with implications in §7.
F1: Recruitment capability is multi-dimensional; single-task results cannot substitute for one another. Model rankings correlate weakly across tasks (mean Spearman 0.28; 13 of 66 pairs negative; Appendix E.2). Claude-opus-5, first Overall, ranks 10th on talent similarity, while the SimT leader, DeepSeek v4-Pro, ranks 11th Overall: selecting by Overall would pick a poor model for similar-talent discovery, and a SimT-only benchmark would rank a near-last model first. Omitting Retrieval or Ranking changes the leader (Appendix E.1).
F2: Explanation quality, verdict accuracy, and output validity are distinct dimensions. GLM-5.3 has a higher PJFA composite than Claude-opus-5 (65.75 vs. 64.25) but lower verdict accuracy (79.82% vs. 80.96%), leads JDG yet returns invalid permutations on 43 of 361 T2J groups, and only 34% of MiniMax-M3's PJFA outputs parse (Table 5).
F3: High scores on constructed pairs do not transfer to real recruitment behavior. The twelve LLMs reach 84–95% balanced accuracy on constructed pairs but 52–57% on behavior-derived pairs, and rankings on the two strata are nearly uncorrelated (ρ = −0.19). The behavior-derived labels carry learnable predictive signal that zero-shot LLMs do not yet exploit: a supervised job-side TF-IDF probe (62.7–63.3) and the enterprise's specialized model both exceed every LLM. A fit benchmark built only from constructed pairs would miss this gap, and recruitment-domain supervision is a clear direction.
F4: Public rankings guide private model selection (see 2/2).
(continued in 2/2)