acceptodds
Under review as a conference paper at ICLR 2027

EHR-Bench: A Chinese Multi-Task Benchmark for Recruitment AI

Abstract

Recruitment applications span diverse operations, from drafting job descriptions and interpreting queries to organizing profiles, assessing fit, and recommending candidates and jobs. Systematic LLM evaluation therefore requires task coverage that reflects this range of business needs. We introduce EHR-Bench, an open Chinese benchmark comprising twelve tasks across six capability families and five recruitment scenarios. The tasks cover generation, extraction, classification, retrieval, semantic similarity, and reciprocal ranking within one evaluation framework. An internal recruitment dataset guides task definitions, the occupation taxonomy and sampling targets, output requirements, and scoring protocols. Following these specifications, we construct 8,336 task instances from public materials and controlled augmentation, with source-aware expert review and documented provenance. We evaluate twelve LLMs through task-level, capability-level, and aggregate comparisons. Closely ranked models exhibit distinct task strengths, and performance varies across input sources, reference evidence, and candidate pools. Five shared models retain the same observed Overall ordering on a private operational reference, with variation in capability and task correspondence. EHR-Bench provides a reproducible foundation for systematic model assessment, task-specific model selection, and further research on LLMs for recruitment. Data, evaluation code, and per-sample scoring records are available at https://anonymous.4open.science/r/ehr-bench-review.

Primary area:
datasets and benchmarks
Keywords:
recruitment AI, multi-task benchmark, Chinese language, large language model evaluation, person-job matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tm73OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EHR-Bench, a Chinese recruitment benchmark comprising 8,336 instances across 12 tasks, six capability families, and five recruitment scenarios. The benchmark is constructed using public resources, synthetic augmentation, and expert-reviewed annotations, with task specifications informed by a private operational dataset. The authors evaluate 12 LLMs, analyze task-specific performance differences, and compare five models against a private operational reference. The paper also releases evaluation protocols, scoring tools, and per-sample results.

Critical strengths
  1. Comprehensive coverage of recruitment tasks.

The benchmark brings together 12 tasks spanning generation, extraction, classification, similarity, retrieval, and ranking. Evaluating these tasks within a common framework is practically useful for comparing LLMs across recruitment operations.

  1. Extensive evaluation and documentation.

The authors evaluate 12 models and provide detailed task-level analyses, source-specific comparisons, and additional baselines. The expert annotation process, quality checks, and released evaluation artifacts also support reproducibility and future benchmarking efforts.

Critical weaknesses
  1. Limited novelty and insufficient motivation for the unified framework.

Many of the individual capabilities evaluated in EHR-Bench have already been extensively studied, including through multilingual recruitment benchmarks and related resources such as ResumeBench, PJB, and WorkRB. In particular, WorkRB already provides a framework for evaluating multiple work-related tasks.

While combining these capabilities into a single Chinese recruitment benchmark has practical value, the paper does not sufficiently explain what new research questions this integration enables or what limitations of existing benchmarks it fundamentally addresses. The main distinction appears to be broader task coverage within one evaluation suite rather than a substantially new evaluation methodology.

The proposed framework's utility beyond the current benchmark also remains unclear. The released tools allow researchers to evaluate additional models on the same tasks, but the paper does not demonstrate how the framework can be extended to new recruitment operations, datasets, or organizational contexts. More importantly, it does not establish what additional insights the unified evaluation provides beyond separately evaluating the individual capabilities.

  1. Limited generalizable findings and unclear implications of the experimental results.

Despite evaluating 12 LLMs across a wide range of tasks, the primary findings remain largely descriptive. For example, the paper reports that models with similar Overall scores exhibit different task-specific strengths and that performance varies across data sources and candidate pools. While useful, these observations are not particularly surprising for a multi-task benchmark.

The paper would be stronger if it identified more substantive findings about recruitment AI capabilities or explained how the results should change model evaluation and selection practices. At present, the discussion largely reiterates performance differences without developing a sufficiently clear takeaway.

Some results could support more meaningful conclusions but are not fully investigated. For instance, PJF balanced accuracy ranges from 52.01–56.64% on behavior-derived examples versus 83.64–95.22% on constructed examples. This raises an important question about whether constructed benchmarks adequately capture real-world recruitment judgments. However, differences in label definitions and data construction make it difficult to interpret the underlying cause.

Similarly, the public and private benchmarks produce identical Overall rankings for five shared models, but only 65% of task-level pairwise comparisons agree in direction. This limits how strongly the aggregate agreement can support claims about practical model selection.

Overall, the paper presents extensive experimental results without sufficiently establishing which findings constitute new knowledge or why the unified benchmark is necessary to obtain them.

Questions for authors
  1. What specific research questions or evaluation capabilities does EHR-Bench enable that cannot be adequately addressed using existing resources such as WorkRB, PJB, and other task-specific recruitment benchmarks? Beyond broader task coverage, what is the central methodological or scientific contribution of the proposed framework?

  2. What are the main generalizable findings from evaluating these 12 tasks together? Can the authors demonstrate a concrete model-selection or diagnostic insight that would be missed if the tasks were evaluated independently?

  3. How reusable is the framework beyond evaluating additional models on the current dataset? Can the authors demonstrate or provide a concrete account of how it would accommodate new recruitment tasks, organizational settings, or evaluation datasets?

  4. How do the main findings change when behavior-derived and constructed instances are analyzed separately? Given the substantial performance differences between these sources, what evidence supports interpreting their combined scores as measuring comparable recruitment capabilities?

3 comments
AuthorsOpenReview ↗

Response to Reviewer tm73 (1/2)

Thank you for recognizing the coverage, experiments, and documentation, and for asking what new understanding the unified evaluation provides. Your review already points to two results that could support stronger conclusions—the behavior-derived vs. constructed gap and the public–private correspondence. We agree, analyzed both further, and base the answers below on these new results. Revised passages are shown in blue in the updated PDF.

Q1. What can EHR-Bench answer that WorkRB, PJB, and other resources cannot?

The difference lies in what is evaluated and in the type of evidence, not only in task count (Table 1). ResumeBench evaluates resume parsing and PJB resume retrieval; WorkRB is an excellent framework for ranking and classification interfaces, mainly for embedding and ranking models. None evaluates LLM generation and judgment outputs, combines behavior-derived with constructed labels, or provides an operational-data reference. EHR-Bench adds JD generation, fit analysis with explanation, query parsing, and resume deduplication, with per-instance provenance and a public–private comparison. It can therefore answer three questions existing resources cannot:

  • RQ1: Does strength on one recruitment operation carry over to others? This needs the same models evaluated across operations.
  • RQ2: Do high scores on constructed samples reflect real recruitment judgments? This needs constructed and behavior-derived labels within one task.
  • RQ3: Do public rankings transfer to an enterprise's private data? This needs a public–private comparison under shared specifications.

Our methodological contribution is a benchmark construction and validation process constrained by operational data: internal data define task specifications, taxonomy, sampling targets, and scoring; public and controlled-augmentation materials implement them with per-instance provenance; and a public–private comparison on shared models tests validity. To our knowledge, this is the first such process among public recruitment evaluation resources. The introduction and contributions are rewritten around these questions (§1).

Q2. Generalizable findings that separate evaluation would miss

The revision organizes the results into four findings (§5.1–§5.3, §6.2), with implications in §7.

F1: Recruitment capability is multi-dimensional; single-task results cannot substitute for one another. Model rankings correlate weakly across tasks (mean Spearman 0.28; 13 of 66 pairs negative; Appendix E.2). Claude-opus-5, first Overall, ranks 10th on talent similarity, while the SimT leader, DeepSeek v4-Pro, ranks 11th Overall: selecting by Overall would pick a poor model for similar-talent discovery, and a SimT-only benchmark would rank a near-last model first. Omitting Retrieval or Ranking changes the leader (Appendix E.1).

F2: Explanation quality, verdict accuracy, and output validity are distinct dimensions. GLM-5.3 has a higher PJFA composite than Claude-opus-5 (65.75 vs. 64.25) but lower verdict accuracy (79.82% vs. 80.96%), leads JDG yet returns invalid permutations on 43 of 361 T2J groups, and only 34% of MiniMax-M3's PJFA outputs parse (Table 5).

F3: High scores on constructed pairs do not transfer to real recruitment behavior. The twelve LLMs reach 84–95% balanced accuracy on constructed pairs but 52–57% on behavior-derived pairs, and rankings on the two strata are nearly uncorrelated (ρ = −0.19). The behavior-derived labels carry learnable predictive signal that zero-shot LLMs do not yet exploit: a supervised job-side TF-IDF probe (62.7–63.3) and the enterprise's specialized model both exceed every LLM. A fit benchmark built only from constructed pairs would miss this gap, and recruitment-domain supervision is a clear direction.

F4: Public rankings guide private model selection (see 2/2).

(continued in 2/2)

AuthorsOpenReview ↗

Response to Reviewer tm73 (2/2)

Q3. Reusability for new tasks, organizations, and datasets

Each task is a directory with data.jsonl, label.json, meta.json, and an optional rubric.md; tasks and capability families are registered in eval/scripts/common.py, prompt builders in prompts.py, and scorers in compute_metrics.py. A new task needs a task directory, a prompt builder, and a scorer entry that can reuse the existing span-F1, PathScore, macro-F1, balanced-accuracy, nDCG, Recall/MRR, and rubric-judge scorers; registering it under a family includes it in that family and Overall. The paper already shows two forms of reuse: the private reference (§6) applies the same specifications, formats, and scorers to an enterprise's private operational data—the route by which an organization evaluates models on its own data—and the public collection was itself extended with new sources (441 JDG instances, 486 PJF pairs) merged through source tags. As a new-task example, visual resume parsing adds a PDF/image input adapter and a structured-field schema, with a scorer adapting the existing precision/recall/F1 utilities to field matching, registered under Extraction. Appendix I describes these steps.

Q4. Behavior-derived vs. constructed instances analyzed separately

The main findings are unchanged. Recomputing Overall with PJF scored on one stratum at a time, the behavior-only variant reproduces the reported order exactly (ρ = 1.00) and the constructed-only variant gives ρ = 0.98, with the same top two models (Appendix F.1, Table 36).

The two strata are a deliberate design. In the constructed stratum, fit follows from explicit evidence in the two texts, testing whether a model reads and compares it correctly; the behavior stratum keeps evaluations and behavior from real recruitment, whose labels carry learnable predictive signal that zero-shot LLMs do not yet exploit—closer to real use and harder. Both draw inputs from the same public source with the same instruction and format; the key difference is pair construction and label evidence.

The combined score is interpretable in the same way as Overall: a fixed-composition summary, with diagnosis read from the strata. That rankings on the two strata are nearly uncorrelated shows they measure two complementary facets of fit judgment (F3). The revision reports both strata for all twelve models (Table 5) and states in §5.3 that current LLMs cannot yet reproduce behavior-derived judgments—an open problem EHR-Bench makes measurable.

Identical public and private Overall rankings, but 65% task-level agreement

We quantified the task-level correspondence; it supports private model selection more strongly than the pooled 65% suggests (§6.2, Figure 5; Appendix G.4, Tables 50–51).

(1) Clear public gaps transfer. The 65% counts all pairs equally, including many near-ties. Among the 118 comparisons not tied publicly, agreement rises with the public gap: 42% below one point, 59% at 1–2, 67% at 2–3, and 88% (37/42) at three points or more. Disagreement concentrates on pairs the public benchmark itself separates only weakly.

(2) The public benchmark narrows the candidate set. Validating the public top two models of each task on the private data and keeping the better one comes within 0.43 points of the private best on average (maximum 1.42), versus 1.10 for two randomly chosen models under the same rule; the shortlist contains the private best on 6 of 12 tasks.

(3) Shared models. The internal data involve candidate privacy and are evaluated only in a compliant environment; these five are all models available there that were also evaluated publicly, and the comparison will extend as more are deployed. Their Overall order is identical (10/10 pairs), and omitting any capability family keeps the public–private rank correlation at 0.90–1.00 (Appendix G.3).

The revision therefore recommends a screen–shortlist–validate protocol: screen with the public Overall, shortlist with task-level scores, and confirm on target operational data (§6.2, §7).

Thank you for the detailed rebuttal and additional analyses. I appreciate the clearer evidence for the benchmark's practical utility, particularly in task-specific model selection.

However, my main concern remains. The findings largely quantify already recognized limitations of multi-task evaluation rather than establish substantially new, generalizable insights about recruitment AI. The constructed–behavior gap is interesting but difficult to interpret given the different label definitions, while the public–private results are limited to five models from one organization.

Although the rebuttal strengthens the paper, I still find its scientific contribution beyond broader benchmark coverage limited. I will therefore maintain my score of 2.

Reviewer_CbXnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EHR-Bench, a Chinese benchmark designed for the field of AI in recruitment. The benchmark encompasses twelve tasks across six capability categories and spans five distinct recruitment scenarios. It comprises 8,336 task instances constructed from publicly available resources and has been used to evaluate 12 large language models (LLMs).

Critical strengths

The benchmark provides extensive task coverage, detailed evaluation interfaces, documented provenance, and extensive quality control efforts. The authors report a model comparison of public and private data, providing a real-world business reference with potential value.

Critical weaknesses

While significant effort has been invested in EHR-Bench regarding task coverage and dataset construction, the benchmark falls short in terms of research contribution and the validity of its evaluation. The paper’s primary findings focus on performance disparities across various models; however, it fails to clearly articulate how this benchmark differs from existing ones—notably lacking a systematic assessment of capabilities specific to the recruitment domain. Furthermore, the evaluation metric for generation tasks—a manually weighted combination of ROUGE-L scores and LLM-as-a-Judge scores—lacks verification of alignment with independent human expert assessments. Additionally, some ranking tasks exhibit limited discriminative power; simple lexical matching methods (such as TF-IDF) achieve near-saturated performance, suggesting that the current tasks may be too simplistic to effectively evaluate complex recruitment-related objectives.

Questions for authors
  1. Why were these specific six capabilities chosen to categorize the 12 tasks? Generation, Extraction, Classification, Retrieval, STS, and Ranking are traditional NLP task types. They don't necessarily correspond to the capability structure specific to the recruitment domain.

  2. In the context of Person–Job Fit tasks, where a mix of historical hiring labels and expert-judgment labels is used, what exactly does the benchmark measure: the prediction of hiring decisions, or a candidate’s actual competence for the role?

  3. The outputs for the two generation tasks, JD Generation and Person–Job Fit Analysis, are scored, and these scores are then combined with ROUGE-L metrics and accuracy rates using a weighted approach. However, the study lacks a validation step to assess the alignment between these automated scores and evaluations provided by independent experts in the recruitment field.

5 comments
AuthorsOpenReview ↗

Response to Reviewer CbXn (1/2)

Thank you for recognizing the task coverage, evaluation interfaces, documented provenance, and quality control, and the value of the public–private comparison. Revised passages are shown in blue in the updated PDF.

Q1. Why these six capabilities, and how they reflect the recruitment domain

The design proceeds from business operation to task to capability family. The twelve tasks implement operations that run on an enterprise recruitment platform (Appendix A.2, Table 9): drafting JDs, parsing recruitment queries, filing jobs and candidates by occupation, discovering similar candidates and jobs, deduplicating resumes, judging and explaining fit, and recommending in both directions. The families group these operations under common names so that results map onto the modules of a recruitment system. The revision states this design order in §3.1 and adds each task's recruitment operation to Table 2.

The domain-specific requirements lie in each task: three-level occupation paths over 18 first-level categories, ranking in both directions, same-person identity across resume versions, fit verdicts with evidence-based analyses, and labels grounded in recruitment behavior records and expert judgments.

The results show clear domain-specific weaknesses of general-purpose LLMs that neither Overall nor any single task reveals: model rankings correlate only weakly across tasks (mean Spearman 0.28; 13 of 66 task pairs negative); balanced accuracy is 52–57% on behavior-derived fit pairs versus 84–95% on constructed ones; accuracy drops by 15–20 points below the first occupation level; 77–95% of dissimilar same-occupation jobs are judged similar; and on the private reference the enterprise's specialized solution exceeds every LLM on person–job fit and talent-to-job ranking (74.30 vs. 65.76; 65.98 vs. 63.34). These gaps are where recruitment knowledge resides and where EHR-Bench points future work.

Q2. Does PJF measure the prediction of hiring decisions or actual competence?

PJF evaluates fit judgment at the screening stage: whether a model can predict the recruitment fit outcome of a resume–job pair from the given information. This is the operation recruitment models are deployed to perform. Two kinds of reference evidence support it (Appendix A.1): behavior-derived pairs from the real recruitment process—478 that received a satisfied evaluation from the hiring side, 215 applied without satisfaction, and 207 browsed without application—and 486 added pairs judged by domain experts from the same kind of text. Both strata use the same instruction and format and are reported separately, showing performance under real-interaction and explicit pair-judgment conditions.

The two goals are aligned: fit judgments in recruitment are grounded in whether the candidate can do the job, and the screening-stage fit outcome is the observable, evaluable expression of that competence—the standard setting in person–job matching research. Directly put: the task evaluates the observable fit outcome at the screening stage, and competence for the role is the goal this judgment serves. Both strata are reported for all twelve models (Figure 4a, Table 5), double-annotation agreement is κ = 0.86, and scoring PJF on either stratum alone leaves the Overall order stable (ρ = 1.00 and 0.98; Appendix F.1, Table 36). The definition is now stated explicitly in §3.2.

(continued in 2/2)

AuthorsOpenReview ↗

Response to Reviewer CbXn (2/2)

Q3. Alignment of automatic generation scores with independent experts

We completed this validation during the rebuttal (§4.1, Table 3; Appendix B.3, Table 18). Four models from different leaderboard positions produced outputs for 50 source-stratified inputs per task (400 outputs). Two recruitment experts not involved in dataset construction rated them independently on a 1–5 scale along the rubric dimensions (for PJFA, including whether the verdict is reasonable), with model identities concealed and output order shuffled. Intervals are two-sided 95% bootstrap intervals from 10,000 resamples with inputs as clusters.

  • Inter-expert agreement: quadratic-weighted κ = 0.74 [0.66, 0.80] for JDG and 0.72 [0.64, 0.79] for PJFA; ratings differ by at most one level on about 97% of items.
  • Rubric vs. experts: Spearman 0.43 [0.31, 0.54] and 0.39 [0.26, 0.50].
  • Task composite vs. experts: 0.33 [0.20, 0.45] and 0.37 [0.24, 0.48].

The experts judge output quality consistently, and both the rubric and the composite correlate significantly with their ratings. The rubric agrees best: it assesses requirement alignment, evidential support, and actionability, and carries the quality discrimination. ROUGE-L alone aligns weakly with experts (0.12 / 0.15), as expected for a reproducible reference-overlap component that complements rather than replaces the rubric. The conclusions also hold under weightings closer to expert judgment: raising the rubric's share of the text score from 0.2 to 1.0 (no ROUGE-L at all) keeps the Overall ranking correlated at ≥ 0.97 with the reported one and the same top model (Table 3, right; Appendix B.4). Table 5 lists the rubric scores of both generation tasks for all twelve models.

The scoring is supported in three further ways: the rubric dimensions come from the operational standards used to assess JDs and fit analyses (Appendices A.2, B.2); the judge, Qwen-3.6-Flash, is not an evaluated system, and Qwen3.8-max from the same family leads neither rubric (Appendix F.3); and half of the PJFA score is verdict agreement with the expert label, independent of the judge.

On "TF-IDF achieves near-saturated performance on some ranking tasks"

We would like to clarify that TF-IDF is a lexical baseline we provide deliberately to calibrate the lexical difficulty of each candidate pool; the results show that the tasks are not saturated and clearly separate models (Table 6, Appendix F.4).

First, only T2J of the five candidate settings has a high lexical baseline. On public-interaction TR pools the best LLM reaches only 45.55; on J2T, TF-IDF is 14 points above uniform and the best LLM adds a further 17 (56.97 → 74.16).

Second, T2J also separates models. The best LLM narrows TF-IDF's gap to a perfect score from about 11 points to 2.8, the twelve models span 84.92–97.19, and the task also tests whether a model returns a valid complete permutation.

Third, the two ranking settings differ in lexical solvability, which is itself a useful finding. In T2J a content-rich resume is matched against occupationally distinct jobs, so keywords give strong cues; in J2T all candidates match the job's keywords and must be told apart by depth of experience. TF-IDF reaches 89 on T2J but 57 on J2T, while LLMs reach 97 and 74. EHR-Bench thus separates lexically solvable conditions from those where LLMs gain most over lexical matching.

The revision adds a "gain over TF-IDF" column to Table 6 and states these points in §5.3.

Thank you for carefully addressing my comments and providing additional explanations and experiments. The revisions have improved the paper to some extent. However, I believe three major concerns remain insufficiently addressed.

  • The authors further explain the design process from recruitment operations to tasks and then to capability categories. However, these explanations mainly show that EHR-Bench covers more recruitment operations, but do not sufficiently justify the domain specificity of its capability taxonomy. Generation, Extraction, Classification, Retrieval, STS, and Ranking are still traditional NLP task types. The authors have not clearly explained why these six categories were selected instead of other recruitment-specific capabilities, or justified the coverage and rationale of this taxonomy. More importantly, adding new business scenarios and tasks does not necessarily constitute a substantial benchmark contribution. The authors introduce research questions about cross-task capability transfer, differences between behavior-derived and constructed labels, and transfer between public and private data. However, the experiments mainly demonstrate these phenomena within EHR-Bench. They do not sufficiently show that these research questions cannot be studied using existing resources, or that the proposed data construction process provides unique evaluation capabilities. Therefore, the paper still appears to be an extension of traditional NLP tasks to the recruitment domain, and its independent research contribution as a benchmark remains insufficiently established.

  • The authors clearly define PJF as person–job fit judgment at the recruitment screening stage and report results separately for real recruitment behavior and expert judgments. This helps clarify the evaluation objective. However, I still have concerns about the validity of the labels. Recruitment behaviors, such as browsing without applying or applying without receiving a satisfactory evaluation, may be influenced by candidate preferences, recruiter preferences, and other factors. They do not necessarily reflect a candidate's actual competence for the job. Furthermore, stable model rankings across the two data sources do not prove that they measure the same construct. Therefore, although the operational definition of PJF is now clearer, there is still insufficient evidence that the labels reliably represent person–job fit at the screening stage.

  • I appreciate the additional evaluation by independent recruitment experts, which directly addresses the previous concern about the lack of human validation for automatic scoring. However, the Spearman correlations between the rubric and expert ratings are only 0.39–0.43, while those between the composite scores and expert ratings are 0.33–0.37. These results provide only limited evidence of evaluation validity.

Overall, I appreciate the authors' efforts to address the reviewers' comments and acknowledge that the additional experiments and analyses have improved the paper. However, the revisions mainly improve the task explanations and experimental validation, while the fundamental concerns about the research contribution, capability taxonomy, and evaluation validity remain insufficiently addressed.

I therefore do not believe the current evidence is sufficient to change my overall assessment of the paper, and I maintain my original rating.

Reviewer_P7EDOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces EHR-Bench, a Chinese benchmark covering twelve recruitment tasks across six capability areas. It combines public and synthetic data with expert review. The authors evaluate twelve LLMs, identify their strengths across tasks, and compare benchmark results. However, some data construction details and evaluation choices still need clarification.

Critical strengths

The benchmark covers a broad range of recruitment tasks and could be useful for evaluating LLMs in Chinese recruitment applications. The experimental evaluation is well documented. Beyond overall scores, the authors report task-level and component results. These details help readers understand the models' different strengths and limitations. The release of evaluation code and per-sample results also makes the reported scores easier to inspect and verify.

Critical weaknesses
  1. Some task-specific data construction and preprocessing steps remain unclear. For JDG and PJFA (Table 2), the choice of weights and the suitability of the combined scores need stronger justification.
  2. After checking the released data and code, I also have concerns about whether some reference answers and inputs support reliable evaluation (see questions below). These issues make it difficult to judge whether the scores reflect the intended abilities. Some samples need to be checked carefully (for example, task 'post_classification', id: post_class::001449).
  3. The related work review is not thorough enough, especially the discussion of existing recruitment benchmarks and pretrained language models in recruitment domain.
Questions for authors
  1. Could the authors justify the evaluation metrics for JDG and PJFA? How were the component weights chosen? (e.g. the 0.6/0.4 split between ROUGE-L and the LLM judge for JDG) In the released data, JDG sample jd_gene::000004 includes detailed salary arrangements, while jd_gene::000080 includes a company introduction. These details are not provided in the input constraints, so models cannot be expected to reproduce them. This makes me question whether ROUGE-L is appropriate here. Have the authors scored the reference answers using the rubric to check for any conflicts between the two?

  2. Could the authors clarify how the resume data was processed? In some samples (e.g. PJFA, PJF, TR, SimT, and Dedup tasks), parts of the resume are keyword lists separated by “|”. Were they produced during preprocessing? How did the authors check that they can represent resume properly?

  3. Appendix A.5 describes the general resume synthesis pipeline, but could the authors provide more details on how the synthetic materials are turned into examples and reference answers? The task-specific construction and labeling steps are still unclear to me, making it difficult to assess the reliability.

  4. Could the authors expand the related work review, provide more detailed comparisons with existing recruitment benchmarks, and relevant domain-specific pretrained models in recruitment domain?

  5. Could the authors provide more detailed statistics on the benchmark data, such as the distribution of education levels? This would help assess potential imbalances and biases in the data.

3 comments
AuthorsOpenReview ↗

Response to Reviewer P7ED (1/2)

Thank you for inspecting the released data and code and for pointing to concrete samples. We traced each to its source record and assessed its impact on the full dataset; in every case the Overall ranking of the twelve models is unchanged. Revised passages are shown in blue in the updated PDF.

Q1. JDG/PJFA metrics and weights; salary and company text in references; rubric check of references

(1) Metric design. The two components answer different questions. ROUGE-L measures how close a generated JD is to a real, published job description; it is deterministic and independent of any judge. The rubric evaluates the JD itself on seven operational dimensions, including alignment with job requirements, actionability, and consistency (Table 17). Combining reference overlap with an LLM judge is common practice (Lin, 2004; Zheng et al., 2023; Liu et al., 2023). Because ROUGE-L spans 16.5–20.7 and the rubric 79.3–91.1 across models, 0.6/0.4 gives both components comparable influence on between-model differences. PJFA gives half of its score to the verdict, the primary output of that operation, and splits the text half in the same ratio (0.3R + 0.2J).

(2) Conclusions do not depend on the weights. Recomputing all weight settings from the released per-sample components, an equal 0.5/0.5 split and 0.7/0.3 give exactly the same Overall order of the twelve models; across a ROUGE-L share of 0–0.8 the top model is unchanged and the rank correlation with the reported order is at least 0.97. GLM-5.3 leads JDG under every weight, as it has both the highest ROUGE-L and the highest rubric score. The sweep is summarized in Table 3 (right) and given in full in Appendix B.4 (Table 19); Table 5 reports the rubric scores of both generation tasks for all twelve models, and Table 20 gives rubric-only generation scores.

(3) jd_gene::000004 and jd_gene::000080. Both come from the 500 native-Chinese public postings, whose references are the original job descriptions; real postings often include compensation or company paragraphs, and the two input titles carry the corresponding cues. Models are not expected to reproduce employer-specific facts, and we verified that this content does not affect model comparison. A deliberately permissive rule flags every reference mentioning compensation or benefits, a company introduction, an address, or contact details: 305 of 941, mostly routine compensation statements that are a regular part of Chinese postings, and 10 with a company introduction. Removing all 305 and recomputing on the remaining 636 gives a JDG ranking correlated at 0.97 with the original, and the Overall order of all twelve models is unchanged (Appendix B.4). The paper already removes company introductions and contact details from the 441 company-board postings (§3.2, Appendix A.4); the next data release extends this cleaning to the 500 Chinese postings, keeping the current version for reproducibility.

(4) Rubric scores of the references. As you suggested, we scored all 941 JDG and 872 PJFA references with the official judge model and prompt (Appendix B.4, Table 21). Their mean rubric scores (66.71 / 59.54) are below those of model outputs (79.3–91.1 / 87.1–94.6). The two components play different roles: the reference anchors outputs to real recruitment content, while the rubric applies uniform operational requirements that published postings were not written to meet. On model outputs the two components are positively associated (within-model Spearman 0.15 for JDG, 0.41 for PJFA) while capturing different aspects. The blinded expert evaluation further supports the rubric's ability to discriminate output quality (see our response to Reviewer CbXn, Q3), and, as shown in (2), the conclusions also hold when the rubric receives more weight.

(continued in 2/2)

AuthorsOpenReview ↗

Response to Reviewer P7ED (2/2)

Q2. "|"-separated keyword lists in resumes

These are the native format of the public source data, not a product of our preprocessing. The records come from the public Tianchi–Zhaopin person–job matching dataset (Appendix A.3), whose provider releases the work-experience field as "|"-separated keywords and fields such as city as anonymized codes, for privacy. We keep the field content and only concatenate fields with their names; synthetic resumes do not use this format. Three points support its use: (i) the behavior-derived satisfied labels were given by the hiring side on the same information (Appendix A.1); (ii) annotators who saw only these inputs, without source labels, agree closely (Cohen's κ 0.86 on PJF, 0.80 on SimT, 0.92 on Dedup; Table 8); and (iii) short, tag-like profiles are a real talent-pool condition, reported separately from full-length synthetic resumes (Appendix F). The revision adds this explanation to §3.2 and a "source field → evaluation input" example in Appendix A.3.

Q3. How synthetic materials become instances and references

For synthetic materials, models generate the resume text, while domain experts determine the final labels from the relevant task materials (Appendices A.1, A.5). The revision consolidates the construction chain of all twelve tasks into one table (Table 7, Appendix A): source materials, how inputs and references are formed (including the behavior-derived positives and two kinds of negatives for PJF, and the positive/negative construction for SimT, SimJ, Dedup, and the candidate tasks), and the human steps with their agreement. All tasks also pass uniform quality control: stratified spot checks on 1,250 records, independent double annotation on 760, and a final audit confirming 395/400 records (98.75%; §3.3, Appendix A.1).

post_class::001449 (Weakness 2)

This is an English-language posting (a German automotive company recruiting a logistics manager) from the public source; city and experience are the source's anonymized codes. Its label, "Logistics › Warehousing and distribution planning", matches the content (transportation and warehousing) and the source's own sub-type ("Logistics manager/supervisor"), so the label is correct. It is the only predominantly English input among the 1,064 job-classification samples, and its effect on Overall is at most 0.006 points; the next release removes it. Together with the analyses in Q1 and the 98.75% audit accuracy, this shows that the scores reflect the intended abilities and that the model comparisons are robust.

Q4. Related work

Table 1 and Appendix H already compared ResumeBench, JobResQA, PJB, and WorkRB. The revision (i) adds to Table 1 (and Table 52) whether each resource evaluates LLM generation and judgment outputs, its label evidence, and whether it provides an operational-data reference; EHR-Bench is the only Chinese resource that covers generation, extraction, classification, similarity, retrieval, and ranking together with a private operational reference; and (ii) discusses domain-specific encoders in §2 and Appendix H—JobBERT (Decorte et al., 2021), ESCOXLM-R (Zhang et al., 2023), and ConFit / ConFit v2 (Yu et al., 2024; 2025). These can be evaluated directly on our classification, similarity, and candidate interfaces, while EHR-Bench also covers generation and explanation tasks, which is why general-purpose LLMs are our main evaluation targets.

Q5. Education and other statistics

In addition to the 18-occupation distribution and its comparison with the operational data (Appendix G.1), the revision reports education and experience distributions (Appendix A.7, Tables 14–15). Candidates' highest education is mainly bachelor's (39.8%) and associate (27.5%), with 4.6% master's or above; jobs' minimum requirements are mainly bachelor's (33.8%), associate (26.2%), or no requirement (26.7%). Experience spans new entrants to more than ten years. Missing values form a separate category.

AuthorsOpenReview ↗

Brief follow-up on our response

Thank you again for your thoughtful review. We would be grateful if you could consider our point-by-point responses to your questions and the supporting analyses when revisiting your overall assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank the three reviewers for their careful and constructive reviews. We appreciate the reviewers' recognition of the benchmark's broad task coverage, its documented and reproducible evaluation, and its potential practical value.

Core contribution. EHR-Bench jointly evaluates content generation, fit judgment, and output validity under a common set of recruitment task specifications. By stratifying behavior-derived and constructed sources and by comparing public data with a private operational reference, it turns task scores into interpretable capability diagnoses and model-selection guidance. The analyses added during the rebuttal yield three concrete conclusions: high-quality explanations do not substitute for accurate fit judgments and valid structured outputs; high scores on constructed pairs do not transfer directly to real recruitment behavior; and the public leaderboard effectively narrows the set of models that must be validated on a target business.

In response to the reviewers' questions, we completed the following new analyses and experiments. Except for the expert evaluation and the judge scoring of reference texts, every result can be reproduced directly from the released data and per-sample results.

  1. Generation scores are now compared against an independent, blinded expert evaluation. Two recruitment experts who were not involved in dataset construction independently rated 400 model outputs with model identities concealed. Inter-expert weighted κ\kappa is 0.74 / 0.72 (JDG / PJFA). The rubric judge score correlates significantly and positively with expert ratings (Spearman 0.43 / 0.39), as does the task composite (0.33 / 0.37).
  2. Conclusions do not depend on the generation-metric weights. Changing the ROUGE-L / rubric weights from 0.6/0.4 to 0.5/0.5 or 0.7/0.3 leaves the Overall order of all twelve models unchanged. Even when ROUGE-L is dropped entirely, the Overall ranking correlates at 0.97 with the original and the top system is unchanged.
  3. The named samples were checked individually and followed by a full screening. After removing every JDG reference that mentions compensation, company introductions, or similar content, the Overall order of the twelve models is unchanged. The named job-classification sample is the only one of its kind among 1,064, and its label is correct.
  4. Joint evaluation yields conclusions that single-task evaluation cannot. The mean rank correlation of models between pairs of the twelve tasks is only 0.28: strength on one task barely predicts strength on another. The top Overall system ranks 7th–10th on three tasks.
  5. High scores on constructed samples do not imply agreement with real recruitment judgments. The twelve LLMs reach 84–95% on expert-constructed person–job pairs but only 52–57% on behavior-derived pairs, and model rankings on the two strata are almost unrelated (ρ=−0.19\rho=-0.19). The paper's main conclusions are unchanged when the strata are analyzed separately.
  6. The public leaderboard gives quantifiable support for private model selection. If the public leaderboard is used to shortlist two candidates per task, the better of the two is on average only 0.43 points below the best model on the private operational set, versus 1.10 points for two candidates chosen at random. Among model pairs separated by at least 3 points on the public leaderboard, 88% retain the same ordering on the private set.

We have uploaded a revised paper; revised passages are shown in blue. The introduction and findings are rewritten around three research questions (§1, §5–§7), and we add the expert evaluation (§4.1 and Table 3; Appendix B.3), weight sensitivity and reference scoring (Table 3; Appendix B.4), generation components and PJF strata for all twelve systems (§5.2 and Table 5; Appendix F.1), the public–private selection analysis (§6.2 and Figure 5; Appendix G.4), a per-task construction table (Table 7), cross-task rank correlations (Appendix E.2), education and experience statistics (Appendix A.7), guidance for extending the benchmark (Appendix I), and an expanded related-work comparison (§2 and Table 1; Appendix H). Point-by-point responses follow.

AuthorsOpenReview ↗

General Update on the Revised EHR-Bench Manuscript

We thank the reviewers for their thoughtful questions. EHR-Bench turns enterprise hiring workflows into twelve public tasks with documented sources, expert-reviewed references, and reproducible scoring. By evaluating shared models on the same tasks in an enterprise setting, we can test whether differences on the public benchmark matter when choosing a model for a particular hiring operation.

The revision makes that connection explicit. Section 5 shows how model strengths vary across tasks, and §6.2 tests the resulting selection decision. Among the five shared models, shortlisting by the target task yielded better enterprise scores on average across the twelve tasks than shortlisting by overall rank, with the same two validations per task.

We hope the reviewers and AC will weigh this contribution and the additional evidence when revisiting their assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.