acceptodds
Under review as a conference paper at ICLR 2027

ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge

Abstract

We present ORQA, a method for testing occupation-level knowledge in large language models. Prior methods either map abstract LLM skills to occupations via task definitions or utilize expert knowledge which is difficult to obtain at scale and expensive. ORQA complements both of these methods by connecting O*NET occupations to trusted occupation-specific websites (such as regulatory agencies, licensing bodies, professional organizations, and government publications) and converting these into source-traceable question-answer pairs. A combination of an automated pipeline and human review produces a set of high quality questions about occupations. The question set created via our method covers 116 occupations from all 21 major groups in the SOC, with 480 questions sourced from 187 different websites. Each question is designed to probe a real-world skill question that is relevant to the occupation in question. We test 15 state-of-the-art frontier and open-weight models via this method. Claude Opus 4.6, GPT-5.4 and Claude Sonnet 4.6 all perform the best at approximately 58-62% while smaller open-weight models achieve approximately 33-41% performance. Performance varies significantly across occupations. Healthcare-related occupations achieve the highest performance (78%) while Office and Administrative Support achieve approximately 40%. Performance on individual occupations (e.g. Sheet Metal Workers and Fish and Game Wardens) is essentially zero. We also find that open-ended questions and weighting by wage bill do not significantly affect the ranking of models on this benchmark.

Primary area:
datasets and benchmarks
Keywords:
benchmarks, evaluation, large language models, occupational knowledge, professional knowledge, question answering, O*NET, source-traceable data

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_FxLQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ORQA, a framework for constructing source-traceable occupational knowledge questions from authoritative documents linked to O*NET occupations. Its pipeline combines evidence extraction, question generation, automated verification, difficulty filtering, and human review. The resulting benchmark contains 480 questions spanning 116 occupations and 187 source hosts. Fifteen models are evaluated using closed-book multiple-choice and open-ended questions, supplemented by retrieval experiments, economic weighting, and comparisons with external benchmarks. An audit involving 64 practitioners provides additional evidence about answer agreement, source support, and occupational relevance. The main contribution is a framework and evaluation resource for source-grounded professional knowledge across a broad range of occupations, including those underserved by benchmarks focused on digital work.

Critical strengths
  1. The framework provides an auditable approach to occupational knowledge evaluation. Linking questions to occupations, source documents, and supporting excerpts makes the intended ground truth inspectable and enables coverage beyond common digital tasks. This combination of occupational breadth and source traceability constitutes a useful contribution when interpreted as evaluating documented professional knowledge.

  2. The paper validates the resource from several complementary perspectives. The practitioner audit provides evidence beyond automated verification, while open-ended evaluation, retrieval conditions, and clustered uncertainty estimates help characterize what the benchmark measures. The released responses also support independent checking: I reproduced the principal closed-book accuracies and approximately reproduced the paired confidence interval comparing Claude Opus 4.6 with GPT-5.4. Together, these analyses provide meaningful support for the resource and its aggregate results.

Critical weaknesses
  1. Some questions omit conditions necessary to determine which professional rule applies. ORQA-005 and ORQA-006 ask broadly about actuarial continuing education, but their source excerpts concern General Qualification Standards and Specific Qualification Standards, respectively. The stems do not preserve this distinction. Similarly, ORQA-000 asks about an actor generally, although the cited requirement specifically concerns an Equity actor or stage manager. These examples raise a concrete concern about preserving applicability conditions during question generation; they do not establish that the underlying requirements contradict each other. The practitioner audit provides useful positive evidence, but its restriction to occupations recruitable through an online participant platform limits generalization to the rest of the bank. The prevalence and impact of this particular question-specification problem remain unclear. High retrieval accuracy alone cannot resolve it, because retrieved passages may supply context missing from the closed-book question.

  2. The occupation-specific and economic interpretations are less well supported than the aggregate benchmark results. In the released bank, 58 of 116 occupations have at most three questions, and all three Sheet Metal Workers questions come from one document. The paper appropriately acknowledges small-sample limitations, but suggestions about using these results for occupation-specific model selection should remain explicitly exploratory. Separately, the supplied main_figure_iclr.py weights individual questions by occupational mean annual wage, without incorporating employment. This differs from the stated occupation-level wage-bill weighting and also makes total occupational weight depend on question count. The discrepancy requires clarification and correction before the economic robustness claim can be assessed, although it does not affect the unweighted closed-book scores.

Questions for authors
  1. Question-specification clarification: What checks ensure that the question preserves necessary jurisdiction, agreement, qualification category, and effective-date conditions? Please address ORQA-000, ORQA-005, and ORQA-006 specifically. Do the existing human reviews or practitioner audit provide an estimate of how frequently such conditions are omitted? Any available analysis of model performance after correcting or excluding affected questions would materially inform my assessment of benchmark validity.

  2. Implementation clarification: Which weighting implementation generated Figure 2? The supplied script assigns mean annual wage to individual questions, whereas the manuscript describes occupation-level weights based on employment multiplied by mean wage. Please reconcile the discrepancy and clarify whether the reported robustness conclusion holds when occupational mean accuracies are weighted according to the stated wage-bill definition.

  3. Please clarify how Figure 1 was produced and whether generative AI tools were used. If so, please specify their role and update the AI Use Statement accordingly.

3 comments

We thank the reviewer for their thorough review, and in particular for taking the time to reproduce our results from the released responses and to read our figure script. We agree with all three points and have addressed each of them below.

Omitted applicability conditions (W1, Q1): We agree that ORQA-000 keys a rule for Equity actors, and that ORQA-005 and ORQA-006 key the General and Specific Qualification Standards without saying which. We have corrected the question text to name the condition and released it as version 2 of the bank with a corrections file. The original release is kept alongside it, so the reported results remain reproducible. Re-running the eleven queryable models on the corrected questions confirms the diagnosis: with the original question text every model answered all three questions wrong on every run (0 of 99 answers across 11 models and 3 runs). With the condition stated, 78 of 99 are correct, and the ranking is unchanged. The verifier's ambiguity check rewrites the question text when the source carries a limitation such as a jurisdiction, but it did not catch a qualification category or an agreement type, and we have described this gap in the paper. On prevalence, we measured it directly. Two models from different providers, GPT-4.1 and Gemini 2.5 Flash, each labelled every question for a missing condition of the kind the reviewer describes, such as a jurisdiction or a qualification category. The two models agreed on 70% of questions (kappa 0.42). Where they disagreed, a third model, o3, decided. Both flag 163 of 480 questions (34%), and the majority estimate is 211 (44%). Flagged questions are harder (frontier mean 49.1% on them, 64.7% without), but the closed-book ranking of all fifteen models is identical with and without them. We have released the labels. The reviewer is also right that the audit covers only platform-recruitable occupations. The bank-wide labels let us check how representative those occupations are: questions in the 20 audited occupations are flagged slightly less often than the rest (28% against 35% on the conservative estimate), so the audit is a mildly favourable sample of the bank, and the bank-wide estimate is the one we rely on. Two human annotators then independently adjudicated a blind sample of 90 questions. Of the 50 flagged by both models, both annotators confirmed a missing condition on 32% and at least one annotator on 68%. Of 25 unflagged questions, both confirmed 8% and at least one 36%. Applied to the whole bank, about 18% of questions have a missing condition that two independent readers both confirm, against the 34% model estimate, and about 52% have one that at least one reader confirms, so the model estimate sits between the strict and lenient human readings. The two annotators agreed with each other on 62% of questions (kappa 0.28), which shows that whether a condition is needed is itself a judgment call.

Depth and wage weighting (W2, Q2): The reviewer is right on both counts: 58 occupations have at most three questions, and all three Sheet Metal Workers questions cite one document. We have added both facts to Section 4, added the number of distinct source documents per occupation to the occupation roster, and rewritten the model-selection suggestion as a direction for a larger bank. Dropping those occupations leaves the ordering unchanged except o3 and GPT-5.2, 0.3 points apart. Going further, we re-ran the pipeline with no changes to its quality checks, making further passes over the existing sources and searching for new ones. This added 226 questions for a bank of 706 and raised 23 further occupations to ten or more questions (e.g., Sheet Metal Workers from 3 to 10), and the ranking is unchanged. On weighting, the reviewer is right and we are grateful for the catch: the script weighted questions by mean wage without employment. We reproduced the submitted panel from that formula, then regenerated it as a weighted mean of occupation-level accuracies with OEWS employment times mean wage. The three leading models keep their order (Claude Opus 4.6 69.4%, GPT-5.4 67.7%, Claude Sonnet 4.6 66.3%). Below them GPT-5.3 moves above Gemini 3.1 Pro and Claude Haiku 4.5 below GPT-4o, so the claim is narrowed to the frontier ordering.

Figure 1 (Q3): Figure 1 was produced with OpenAI's Images 2.6 from a written description of the stages. The conceptual design and all text in the figure were provided by the authors, who reviewed it for accuracy against Section 3. No other figure was made this way, and we have added this disclosure to the AI Use Statement.

We are glad the reviewer found the framework auditable and the validation meaningful. None of the corrections changes the aggregate ranking. We would be grateful if the reviewer would consider these additions in reassessing the submission, and we are glad to answer anything further during the discussion period.

Thank you for the detailed response and additional analyses. The corrected wage weighting, expanded coverage, and clarified claims address several of my concerns. However, the new audit suggests that omitted applicability conditions extend beyond the three corrected examples, and the rebuttal does not establish that this issue has been systematically resolved. The unchanged model ranking is reassuring but does not fully address question validity. I therefore maintain my rating.

We thank the reviewer for the follow-up. We agree that our previous response measured how often questions might omit a condition but did not show that the problem had been resolved. We have since extended the review to the whole bank, with human readers deciding each case, and describe below what was found and what was changed.

Identification of candidate questions: Two models from different providers checked whether each question states enough for a reader to know which rule applies, and a third resolved disagreements. Of the original 480 questions, 211 were flagged for review, 163 by both models. A flag does not mean the keyed answer is wrong, since in every case the answer is supported by its source; it means a condition from that source, such as the jurisdiction or the agreement in force, may need to be stated in the question, which is the issue the reviewer identified in ORQA-000, ORQA-005 and ORQA-006.

Human verification of flagged questions: Two human annotators independently read all 211 flagged questions against their sources, blind to the model labels. Both found a missing condition in 53 questions and at least one did in 95. A third reader then adjudicated the 42 disagreements and confirmed 17 more. That leaves 70 questions, 14.6% of the original bank, with a confirmed missing condition.

Review of unflagged questions: We also tested whether the models were missing cases. In the sample of 25 unflagged questions reported in our first response, both annotators found a gap in 8% and at least one in 36%, so automated screening alone was not enough. The human review was therefore extended to all 422 questions that passed the automated screen across the original and the added questions. Each was read against its source to check that it states what is needed to determine a single correct answer. Questions found to omit a condition were set aside for correction, so every question that passed screening has now had an individual human reading.

Correction of confirmed questions: All 70 confirmed questions from the original bank were rewritten to state the missing condition. A second reader checked every rewrite against its source to confirm that the question now says when the rule applies and does not give the answer away. 68 passed on the first check and two after a further revision. None had to be dropped. The same procedure was applied to the 226 questions added during the rebuttal: human review confirmed 33 that needed a condition stated, and these were revised with the same checks. The original and corrected wording of every revised question is documented, and questions flagged by the models but not confirmed by the readers keep their screening label in the released records.

Change to the construction pipeline: The reviewer's examples exposed a gap in our checks. A question could pass because its answer was entailed by the source even when the question did not say which rule applied. We have added a separate check that the question must contain the information needed to identify the governing rule before it enters the bank. Questions that fail it are corrected and re-checked before they are used.

Addressing question validity: These changes address the reviewer's concern about whether individual questions are valid. We now check not only whether the source supports the keyed answer but also whether the question itself gives a reader enough information to identify that answer, and every confirmed case of a missing condition has been corrected and checked again before it can be used. The revised process fixes problems with the questions themselves, independently of the finding that the overall results are unchanged.

the human readers have checked every flagged question, every confirmed case corrected and verified, and every question that passed screening read individually. All changes to the manuscript are marked in blue in the revision.

We appreciate the reviewer pressing on this. The examples pointed to a weakness our original pipeline did not handle, and the added check now closes it. We hope the individual checking and correction of the questions gives stronger evidence that they are sufficiently specified, and we would be grateful if the reviewer would take these changes into account when reconsidering their assessment. We are glad to answer anything further during the discussion period.

Reviewer_Q4ntOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on testing LLM knowledge in various occupations. It argues that existing benchmarks mainly focus on general capabilities or domain tasks with limited occupational coverage that require extensive expert labor. Meanwhile, current economic studies estimate the overall impact of AI systems on the labor market rather than evaluating a system for specific jobs.
To achieve broader occupational coverage with higher automation and specific evaluation, the paper takes an economic perspective to systematically evaluate LLM knowledge across 116 occupations in the real labor market. Specifically, the authors propose an automated framework to construct a source-traceable benchmark. They select official occupations from the U.S. Bureau of Labor Statistics weighted by national wage bills. For each job, they extract source-grounded evidence cards from authoritative regulatory documents and automatically generate question-answer pairs. These pairs undergo a seven-step filtering pipeline alongside human verification to ensure quality, culminating in a closed-book evaluation using both multiple-choice and open-ended questions. The final benchmark consists of 480 items spanning 116 jobs.
Overall, the paper demonstrates a clear effort to automate the construction of an economically weighted, source-traceable benchmark. Its well-structured filtering pipeline and alignment with real-world labor market distributions represent notable aspects of the work. The authors claim that experimental results demonstrate consistent performance trends across models, aligning with external benchmark metrics. However, I have substantial concerns regarding the incomplete representation of knowledge scope across different occupations, the extremely sparse sampling per job, and the limited practical utility of relying on static closed-book evaluations.

Critical strengths

Principled Data Filtering Pipeline: The authors design a refined multi-step automated filtering framework alongside human verification, ensuring high-quality item generation from official evidence cards and producing a final answer-pair distribution that closely aligns with the real-world labor market.

Critical weaknesses
  1. The benchmark risks oversimplifying professional knowledge, making it insufficient to evaluate model knowledge in specific occupations and far less capable of reflecting real-world model usefulness.

a) From a source perspective, lines 112 to 114 state that the benchmark scope is explicitly restricted to source-traceable occupational knowledge derived from official documents, such as reporting responsibilities, professional recommendations, and regulatory guidelines, rather than overall occupational capability. This raises questions about certain professions. For instance, as noted around line 029, models exhibit near-zero performance in individual occupations like Sheet Metal Workers and Fish and Game Wardens. In these roles, true competence hinges on practical, hands-on skills that text-based benchmarks fail to capture.

b) From a scale perspective, the benchmark scale remains relatively small compared to established benchmarks. With only 480 items distributed across 116 jobs, the benchmark averages merely 4 questions per occupation, whereas benchmarks like GDPval provide approximately 30 tasks per occupation. Such sparse sampling for a given job may lead to high evaluation variance and limited statistical reliability.

  1. The evaluation relies on a static closed-book setup, which diverges from how LLMs are deployed in real-world professional workflows. Regulatory and procedural knowledge is strictly factual, low-tolerance, and publicly accessible online, making it a primary use case for Retrieval-Augmented Generation (RAG) rather than pure parametric memorization. Table 14 in appendix indicates that web search boosts performance above 80%, while RAG achieves near-saturated performance close to 100%. If models can easily solve these items when dynamically retrieving web information, testing closed-book memorization yields limited practical insight for realistic model usage.
Questions for authors

Given that the experimental results in the Appendix show near-saturated performance when models are augmented with web search or RAG, what unique value does testing pure parametric memory provide to the community?

1 comment

We thank the reviewer for their thoughtful comments. We have incorporated the feedback and address the mentioned issues in the following points.

Scope (W1a): We agree that ORQA does not measure hands-on competence, and we have updated Section 1 to say so where the scope is defined. ORQA measures what the documents governing an occupation say a practitioner must know or do. The near-zero scores for Sheet Metal Workers and Fish and Game Wardens mean that models do not state those codified rules from memory, not that the occupations are unmeasurable. Numeric-threshold questions are the hardest kind (46% for frontier models against 60 to 68% for procedures), consistent with the reviewer's reading.

Scale (W1b): We accept the comparison with GDPval's roughly 30 tasks per occupation, since ORQA trades depth for breadth. We have stated the average of about four questions per occupation, labelled every occupation-level result exploratory, and removed any claim to occupation-level rankings. We have also done what the concern implies and added questions. We re-ran the pipeline with no changes to its quality checks. It made further passes over the existing source documents and searched for new ones, which added 226 questions for a bank of 706. Occupations with at most three questions fell from 58 to 25 and those with ten or more rose from 2 to 25. The ranking of the eleven models that could still be queried is identical to the original, both on the full 706-question bank (e.g., Claude Opus 4.6 63.4%, GPT-5.4 59.9%, Claude Sonnet 4.6 58.2%, Gemini 3.1 Pro 56.8%) and on the 25 occupations that now have ten or more questions (251 questions). The added questions are about as hard as the original bank (e.g., Claude Opus 4.6: 65.6% on them against 62.4%). Dropping the thin occupations instead of adding to them gives the same answer: on the 58 occupations with four or more questions the ordering is unchanged except that o3 and GPT-5.2, 0.3 points apart, swap.

Value of closed-book evaluation (W2, question): Retrieval does raise scores substantially, and we read that as a finding. With the source passage retrievable, frontier models score 93 to 99%. The questions are therefore answerable, and the models apply a rule once they see it. The closed-book gap therefore measures whether a model knows or can find the governing rule, not whether it can process it. The fixed retriever is an upper bound, not a deployment: its index holds 96% of the source documents. Provider web search over the open web is the realistic setting. It still leaves frontier models at 78 to 80%, and those 20 points are errors made with search available. Closed-book is what remains when no retrieval happens (e.g., the technician without connectivity, or the worker who does not know a rule exists and so does not search for it). It is also the condition under which practitioners answered unaided (71.5% against 57.6% for frontier models). And it is the quantity that tracks GDPval (rank correlations 0.90 and 0.92). We have kept closed-book as the primary condition and report the two retrieval conditions beside it. We have also added a paragraph stating which deployment setting each condition corresponds to.

We are glad the reviewer found the filtering pipeline principled and the labor-market alignment notable. The scope is now stated in the reviewer's terms, the bank is 47% larger with an unchanged ranking, and the retrieval results locate the deficit rather than remove the reason to measure it. We would be grateful if the reviewer would consider these additions in reassessing the submission, and we are glad to answer anything further during the discussion period.

Reviewer_mgb2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ORQA, a question-answering benchmark for evaluating the occupational knowledge of large language models. Using O*NET occupational classifications and authoritative sources, the authors develop an automated pipeline to generate source-grounded questions, resulting in a dataset of 480 questions across 116 occupations. They evaluate 15 LLMs and further validate the benchmark through open-ended questions, wage-bill weighting, comparisons with existing benchmarks, and an audit by practicing professionals. The results show substantial variation in model performance across occupations, while retrieval augmentation significantly improves accuracy.

Critical strengths
  • Evaluating LLMs in the context of specific occupations is a worthwhile direction. ORQA uses documents from regulatory agencies, professional associations, and industry standards to construct source-traceable questions, providing a practical approach to generating occupational knowledge benchmarks at scale.

  • The authors evaluate 15 LLMs across different occupations, response formats, and retrieval settings. They also conduct an audit with practicing professionals, providing additional evidence for the correctness and occupational relevance of the questions. These experiments help characterize the benchmark and its potential applications.

Critical weaknesses

W1. Insufficient definition of the research problem and evaluation metrics, leaving the benchmark's validity unclear.

The paper aims to evaluate LLMs' occupational knowledge but does not clearly define this concept or justify the need for a new benchmark beyond existing knowledge and occupational task evaluations.

  • In §1 and §3, the authors link questions to occupations through O*NET classifications and authoritative documents. However, knowledge extracted from these sources does not necessarily represent the core knowledge required for an occupation. Although the authors acknowledge this limitation in §1 and §5, the paper lacks a clear definition of the evaluation target, knowledge coverage, sampling strategy, and occupation-level metrics. This concern extends to the experimental results. With only 480 questions across 116 occupations, many occupations have just 3–4 questions. Their accuracy may depend heavily on question selection and difficulty, while repeated trials do not address the limited sample size. Thus, the occupation-level conclusions in §4 lack sufficient statistical support.

  • §2 does not clearly establish ORQA's contribution beyond GDPval and prior work on automatically generated occupational exams. While ORQA emphasizes authoritative sources and answer traceability, the authors do not demonstrate their benefits in evaluation quality and occupational representativeness.

W2. Inconsistencies in dataset statistics and validation procedures undermine the reliability of the benchmark.

  • Table 4 (p. 17) and Table 8 (p. 31) report different question counts for the same SOC groups, despite both describing the final 480-item dataset. For example, Office and Administrative Support has 23 vs. 14 questions, Installation and Repair has 13 vs. 9, and Transportation and Moving has 25 vs. 38. These discrepancies raise concerns about the dataset composition and the reliability of occupation-level results.

  • Appendix A.14 (p. 34) states that a maximum of three raters per occupation was automatically enforced. However, Table 13 (p. 36) reports six raters for Software Developers and four for several other occupations. Furthermore, §1 describes approximately 67% agreement between automated screening and human judgments, whereas Appendix A.2 shows that only 34% of the 185 retained items were rated Good, 34% Okay, and 32% Bad. This reflects the human acceptance rate among retained items rather than overall agreement between automated and human decisions. These inconsistencies call into question the reported validation procedures and quality control results.

Questions for authors

Q1. Motivation. Why is the gap between existing benchmarks and real-world occupational needs important enough to warrant a separate benchmark? What new insights or practical value would ORQA provide?

Q2. Evaluation target and metrics. Could the authors clarify what ORQA measures and how the sampled questions support representative and comparable occupation-level evaluations?

Q3. Contribution beyond existing benchmarks. Could the authors add a ✓/✗ comparison table with GDPval and prior automated occupational exams, highlighting evaluation targets, occupational coverage, source traceability, and quality control?

Q4. Reliability of occupation-level results. How robust are the findings with only 3–4 questions per occupation? How do the authors explain the discrepancies with GDPval in Appendix A.5?

Q5. Data and validation inconsistencies. Please clarify the conflicting question counts in Tables 4 and 8, the mismatch between the stated three-rater cap and Table 13, and the calculation of the reported 67% agreement.

4 comments

We thank the reviewer for their detailed and thoughtful comments, and in particular for catching the inconsistencies in W2, all of which we have corrected. We have incorporated the feedback and address the mentioned issues in the following points.

Definition, sampling and metrics (W1, Q1, Q2): We have added a definition at the start of Section 3: ORQA measures agreement with documented professional guidance. That is what the bodies governing an occupation, such as regulators and licensing boards, state in writing that practitioners must know or follow. It does not measure hands-on skill or knowledge outside those documents. The sampling rule is stated in one place: occupations are allocated across the 21 SOC major groups in proportion to each group's share of the national wage bill, taken in order of wage bill within each group, with a floor of three questions where sources allowed. The metric is closed-book accuracy over three seeds, with bootstrap intervals that resample occupations and then questions. On motivation: capability benchmarks are not indexed by occupation, usage studies do not compare models, and expert-built evaluations such as GDPval cover tens of occupations. ORQA's practical value is a model-by-occupation table that can extend to occupations no expert panel will cover. Because every question carries its passage, practitioners could check it: 84% of audit ratings agreed with the correct answer. On representativeness, the same practitioners rated 75% of the questions 4 or 5 out of 5 for relevance to their own work. The closest prior work is self-built occupational exams (del Rio-Chanona and Einsiedler, 2025). There a model writes both the questions and the answer key from O*NET task statements, so the same question recurs for every occupation listing that task, and nothing is grounded in a source. Each ORQA question comes from that occupation's own documents.

Comparison beyond existing benchmarks (Q3): We have added the table the reviewer asked for to Section 2.

ORQAGDPvalSelf-built exams (del Rio-Chanona and Einsiedler, 2025)
Evaluation targetDocumented professional guidanceLong-form deliverablesTask knowledge as the model understands it
Ground truthVerbatim passage in an authoritative documentExpert pairwise preferenceModel-written answer key
Occupations116, all 21 SOC groups44, 9 sectorsScales with O*NET
Source traceabilityYes, per questionNoNo
Quality controlVerifier, difficulty probe, rubric gate, author review, practitioner auditExpert reviewModel self-grading

Reliability of occupation-level results (Q4): We agree that with three or four questions per occupation the per-occupation scores are not estimates, and we have removed language that treats them as such. We have also added questions: we re-ran the pipeline with no changes to its quality checks, making further passes over the existing sources and searching for new ones. This added 226 questions for a bank of 706 and raised the number of occupations with ten or more questions from 2 to 25. The ranking of every queryable model is identical on the full bank and on those 25 occupations. On the GDPval discrepancies: the two instruments measure different things (recall of a governing provision versus expert preference on a deliverable), and we have moved that caveat to the first sentence of Appendix A.5.

Inconsistencies (W2, Q5): Tables 4 and 8: the funnel table was built from run logs whose slate placed Freight Forwarders (9 questions) and Riggers (4 questions) in Transportation. Those 13 questions account for every difference listed, so Table 4, which follows SOC codes, is correct, and we have regenerated the funnel table to match. Rater cap: the cap of three was a survey quota on completed sessions, and participants already in the survey were allowed to finish, which is how six raters occurred (stated in Appendix A.14). The 67% figure: it is indeed a human acceptance rate for gate-retained questions, not agreement. We have replaced it with a stronger measurement: the production gate run over all 400 human-annotated questions, not only the retained ones, with a released script. It retains 57, all rated good or okay by the human annotators (93% good), against 58% acceptable in the ungated pool. Its three-way label matches the human annotators' on 70.8% of questions, and its errors run toward rejection.

We appreciate the reviewer's acknowledgment that evaluating LLMs in the context of specific occupations is a worthwhile direction. None of the corrections changes an aggregate result, and the bank is now 47% larger with the same ranking. We hope these additions address the concerns, and we would be grateful if the reviewer would consider updating their assessment. We would be happy to clarify anything further during the discussion period.

Reviewer_mgb2OpenReview ↗

Follow-up Questions on the Author Response

Thank you for your response. However, I still have a few concerns:

  1. Abount motivation. Author clarifies what existing benchmarks do not cover, but why is filling this gap important? What research questions or practical needs does ORQA address that existing evaluations cannot?
  2. The number of retained items drops from 185 to 57 out of 400. Were the same Gate settings used? Were these 400 items involved in developing or tuning the Gate? If so, can the reported 70.8% agreement be considered independent validation?
  3. My original concern was whether scores based on only 3–4 questions per occupation are reliable. The unchanged overall model ranking across different occupation subsets does not establish occupation-level stability.

P.S. It would be helpful if you could highlight the changes in the revised manuscript (e.g., using a different color), so reviewers can follow the revisions more easily.

We thank the reviewer for the follow-up and address the three concerns and the request in turn.

Motivation and practical need: The gap is that no existing evaluation measures whether a model knows the written rules that govern an occupation, across the economy. GDPval (Patwardhan et al., 2025) measures expert preference on deliverables for 44 occupations, self-built occupational exams (del Rio-Chanona and Einsiedler, 2025) have the model write both question and answer key with no source behind them, and exposure and usage studies (Eloundou et al., 2023; the Anthropic Economic Index) measure where AI could be used or is used without comparing models. ORQA is the first to measure that knowledge occupation by occupation, grounded in the occupation's own documents, and to compare models on it. Three needs depend on such a measure:

  • Exposure and usage estimates need a capability measure for the same occupations. ORQA supplies one indexed by occupation, so it can be combined with them and with labor statistics (Appendix A.5).
  • Model developers have little evidence on non-digital occupations, where ORQA shows the largest gaps.
  • Every question cites the passage it is keyed to, so the bodies that write the rules can check whether a model states them.

Gate settings and the 70.8% agreement: The two counts come from different gate settings, and the 70.8% was measured on the development set. The 185 came from an earlier, looser version of the gate used during development, whose outputs were not kept, which is why we withdrew that figure. The 57 come from the production gate that built the bank, re-run over the same 400 questions. The 400 questions were used to develop the gate: the rubric it applies was written from the two annotators' tags on those questions, so agreement on those 400 shows only that the rubric fits the labels it was written from. The paper now says this.

Out-of-sample validation of the gate: The independent check is in Appendix A.19. 118 questions that were not among the 400 (58 released questions and 60 the pipeline had rejected) were graded blind by the same two human annotators and by the gate. On these questions the gate's accept or reject decision agrees with the annotators on 87% and 85% of questions, and the two annotators agree with each other on 82%, so the gate matches each annotator about as well as they match each other. Of the 60 questions the gate retained, 95% and 93% were acceptable to the annotators. Of the 18 it labelled bad, 89% and 94% were bad to them, and all 18 to at least one. On questions it had no part in developing, the gate therefore keeps what the annotators would keep and rejects what they would reject, which is the independent validation the 70.8% could not provide.

Reliability of occupation-level scores: We agree, and we have measured it directly from the stored responses. For the 37 occupations that have at least three original and three added questions, a model's score on the original questions and on the added ones correlate at only 0.17. The 95% interval for a single occupation's score spans about 90 points with three questions and about 40 with ten. Estimating how much of the variation between occupations is noise gives a reliability of 0.29 for a score based on four questions, 0.51 for ten and 0.8 for about 40.

The paper now reports this analysis in Section 4 and the appendix. Occupation-level numbers are presented as a measure of reliability and of how many questions an occupation needs, the low-scoring occupations named in the paper are given as small-sample examples, comparisons between SOC groups are limited to those whose intervals do not overlap (for example 70%, interval 63 to 77, on Healthcare against 48%, interval 38 to 59, on Installation, Maintenance and Repair), and the contribution claimed from the results is the aggregate one, the fifteen-model comparison with uncertainty estimates.

Revised manuscript with changes highlighted: We have uploaded a revision with every change since submission marked in blue.

We appreciate the reviewer pressing on these points. We hope the out-of-sample check and the reliability analysis resolve the remaining concerns, and we would be grateful if the reviewer would take them into account when revisiting their assessment. We are happy to clarify anything further during the discussion period.

Reviewer_mgb2OpenReview ↗

Response to the Author

Thank you for the further clarification and additional analyses. However, my main concern regarding the motivation remains.

You still have not fully understood my point. The fact that existing benchmarks do not evaluate something does not justify the necessity of the proposed benchmark. These are two separate issues. Proposing a new benchmark certainly requires a clear justification of its significance. For example, a particular capability of a model may be important enough to deserve our attention, while existing benchmarks fail to cover it. However, statements in the current manuscript, such as “As language models become general-purpose work tools, the evaluation question shifts from whether a model has an abstract capability to its practical impact across the economy: whether it can help a worker in a particular occupation answer the questions that arise in practice, and whether one model is more useful than another for that occupation,” are insufficient to establish this point.

In fact, providing such guidance is no longer my responsibility as a reviewer. Moreover, the current manuscript appears to have undergone substantial revisions compared with the original submission. Therefore, I will maintain my original assessment and rating.

Reviewer_Pp4POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ORQA, a framework for constructing occupation-indexed professional knowledge questions from authoritative sources. The pipeline links O*NET occupations to trusted publishers, extracts evidence passages, generates multiple-choice questions, and applies automated checks, difficulty filtering, balancing, and author review. The resulting benchmark contains 480 questions across 116 occupations and 21 SOC major groups. The authors evaluate 15 models using closed-book multiple-choice and open-ended questions, alongside wage-weighted aggregation and retrieval experiments. Leading models achieve approximately 59–62% closed-book multiple-choice accuracy, with substantial variation across occupations. A practitioner audit supports the correctness and relevance of many sampled items, while retrieval substantially improves model performance.

Critical strengths

Auditable occupational coverage. Linking each answer to an identifiable source passage makes the benchmark inspectable, and the inclusion of trades and other non-digital occupations broadens coverage beyond many existing agent benchmarks. This is a useful resource contribution even if the underlying construction techniques are familiar.

Useful validation beyond a model leaderboard. The audit by 64 practitioners on 81 items provides evidence about answer correctness and occupational relevance. The fixed-retrieval experiments further help distinguish failures to recall professional facts from failures to answer with relevant evidence. These analyses strengthen the interpretation of the benchmark within its stated scope.

Critical weaknesses

Limited novelty and insufficient evidence for the added value of the proposed pipeline. The main methodological distinction appears to be the combination of occupation-specific source selection and traceability with established retrieval, evidence extraction, QA generation, and model-based filtering. Section 2 already discusses occupation-oriented evaluation and automatically generated occupational exams. Source grounding is a useful distinction, but the paper does not establish how much the full pipeline improves over a simpler source-grounded QA baseline using the same occupations and source pool. For example, a matched-budget comparison could measure human acceptance, occupational relevance, coverage, and cost per accepted item. The filtering diagnostics in Appendix A.11 describe substantial processing and rejection, but they do not isolate the benefit of the individual design choices. As a dataset contribution, the breadth is valuable; however, 480 items spread across 116 occupations provide limited depth for the occupation-level comparisons that motivate the paper. I therefore find the demonstrated contribution incremental relative to the scope of the proposed framework.

The open-ended evaluation needs clearer, validated rubrics for professional answer quality. The paper already uses a human-calibrated rubric gate for item construction; my concern is specifically about scoring model responses. Appendix A.4 describes three judges returning binary yes/no decisions, followed by majority voting. This protocol does not establish how consistently the judges handle partially correct procedures, omitted conditions, jurisdictional applicability, or an otherwise correct answer containing an unsupported recommendation. For procedure and regulation questions, explicit per-item rubrics grounded in the evidence could specify required elements, acceptable alternatives, and critical errors. Binary correctness can be appropriate for an atomic numerical fact, but its adequacy for broader professional responses needs validation. The practitioner audit evaluates the questions, reference answers, and sources; it does not directly validate the judges' scoring of model-generated free responses. I would find the open-ended results more convincing with practitioner–judge agreement on a sample of model answers and an analysis of whether rubric-based scoring changes the conclusions. The reported 19% of relevance ratings at 1 or 2 also makes clearer validation against occupational requirements consequential.

Questions for authors

Evidence for the contribution: What evidence shows that the full construction pipeline yields better occupational questions than a simpler source-grounded QA pipeline with the same source pool and budget? Are there existing comparisons or ablations reporting human acceptance, occupational relevance, and cost per accepted item?

Clarification and validation of answer scoring: What exact criteria do the open-ended judges apply to omissions, partially correct procedures, acceptable alternatives, and unsupported additions? Are explicit per-item scoring rubrics available, and has their application been checked against practitioners' judgments of model-generated responses? Such evidence could materially change my assessment of the open-ended evaluation.

1 comment

We thank the reviewer for their thoughtful comments. We have produced the evidence both weaknesses asked for and address the mentioned issues in the following points.

Novelty and value of the pipeline (W1, Q1): The reviewer is right that the techniques in our pipeline are established and that source-grounded benchmark construction exists in other domains. What ORQA does first is apply them to occupational knowledge with every question traceable to a verbatim passage from a body that governs the occupation. In self-built occupational exams (del Rio-Chanona and Einsiedler, 2025) a model writes both the questions and the answer key from O*NET task statements. Our answer key is a passage from the source, not a model's belief. There the same task statement produces the same question for every occupation that lists it. We have updated Section 1 to state the contribution in these terms and added a comparison table to Section 2.

On isolating the pipeline's benefit: the previous version of this bank (1,021 questions, built the same way but without the human-calibrated gate) was graded by two human annotators on 400 questions as good, okay or bad against a written rubric. They rated 42% bad. The gate built from those labels retains 57 of the 400, every one rated good or okay (93% good). We also built the baseline the reviewer describes. The generator alone was run over the same 1,885 source passages, with none of the pipeline's checks. It produced 1,373 questions, of which the full pipeline kept 42. Two human annotators then graded a blind sheet of 60 generator-only questions the pipeline had rejected and 60 released questions from the same occupations, with the same rubric as before. They rated 38% and 50% of the generator-only questions bad, against 12% and 7% of the released ones, and both found 47% of generator-only and 82% of released questions acceptable. The commonest generator-only failures were a research paper as the source, a suggestion rather than a requirement, and a source outside the occupation. The calibrated gate, run over a larger blind sheet of 371 questions, gives the same picture (34.5% against 0.6% bad). On cost, the full pipeline's measured cost in the expansion run was about 4.60 USD per accepted question. The generator with the gate as its only filter would cost about 1 USD per accepted question at the prices we paid, so the full pipeline costs roughly four to five times more per accepted question. The difference buys the entailment, distractor and difficulty checks that the gate does not apply.

Scoring of open-ended answers (W2, Q2): As the reviewer notes, we did not state the judges' criteria. We have added the full judge prompt to Appendix A.4: a match means the answer states the same specific recommendation or fact, and paraphrase is allowed. The prompt does not cover an otherwise correct answer that adds an unsupported recommendation, a limitation we now state. We also ran the rubric scoring the reviewer proposes: Claude Opus 4.7 wrote one rubric per question and the same judges re-scored all 7,184 responses. Rubric scoring is stricter but the order matches the binary rule except one adjacent swap (Spearman 0.996), and the two rules disagree on 24% of answers.

We then ran the practitioner check the reviewer asks for. We recruited practitioners on Prolific from the 20 audited occupations and screened them for at least two years of current practice in the role, with a field knowledge check. Each graded all 15 models' stored answers to a question from their own occupation, blind to model and judges. They graded first unaided and then against the source with the rubric, tagging any error found. Across the 49 sessions completed so far (165 answers with a practitioner majority), practitioners agree with the binary judges on 79.4% (kappa 0.52) and with the rubric judges on 80.6% (kappa 0.57). The binary judges accepted 12 answers that practitioners did not. On 10 of those, at least one practitioner marked the answer as missing a condition the rule depends on, which is the gap the reviewer identified, and the next judge prompt will check for it. The raters also confirmed the questions: 44 of 49 said the source clearly supports the correct answer, and relevance to their own work averaged 4.4 of 5. So both scoring rules track practitioner judgment at about 80%, and the ordering is the same under both. The study is about two-thirds complete (49 of 72 planned sessions), and we will update the numbers when the remaining sessions finish.

We are glad the reviewer found the auditable coverage useful and the practitioner audit a strength. Both tests the reviewer asked for support the ordering we report, while the 24% of answers on which the two rules disagree is why we now report both rules. We would be grateful if the reviewer would consider these additions in reassessing the submission, and we are glad to answer anything further during the discussion period.

Comments on the paper

1 comment

We thank all four reviewers for the careful and constructive feedback. Before responding individually, we summarize the additional experiments and validations we have run during this response period.

Expanding the question bank: We re-ran the pipeline with no changes to its quality checks. It made further passes over the existing source documents and searched for new ones. This added 226 questions that passed every quality check, for a bank of 706. Occupations with at most three questions fell from 58 to 25, and occupations with ten or more rose from 2 to 25. The ranking of every model that could still be queried is identical to the original 480-question ranking, both on the full 706-question bank and on the 25 occupations that now have ten or more questions (251 questions). The added questions are about as hard as the original ones (e.g., Claude Opus 4.6: 65.6% against 62.4%). Dropping the thin occupations instead gives the same answer: on the 58 occupations with four or more questions the ordering is unchanged except that o3 and GPT-5.2, 0.3 points apart, swap. Two human annotators graded a blind sample of 60 of the new questions with the same rubric as the original bank: 82% and 92% were acceptable, against 88% and 93% for released original questions graded in the same session.

Practitioner grading of model answers: We recruited practitioners on Prolific from the 20 occupations in the original audit, screened for at least two years of current practice in the role, with a field knowledge check. Each graded all fifteen models' stored open-ended answers to a question from their own occupation, blind to the model and the judges. Our three LLM judges (GPT-4o, Claude Haiku 4.5 and Gemini 3.1 Pro) each check whether a model's answer states the same fact or procedure as the correct option, with the majority deciding. Across the 49 sessions completed so far (735 ratings, 165 answers with a practitioner majority), practitioners agree with those judges on 79.4% of answers (Cohen's kappa 0.52) and with rubric-based judges on 80.6% (kappa 0.57). The study is about two-thirds complete (49 of 72 planned sessions), and we will update the numbers when the remaining sessions finish.

Rubric-based re-scoring of open-ended answers: Per-item rubrics (required elements, acceptable alternatives, critical errors) were written from each source and all 7,184 stored responses were re-judged. Rubric scoring is stricter (2 to 14 points lower) but the order is the same as under the binary rule except one adjacent swap (Spearman 0.996).

Omitted applicability conditions: We have corrected the three questions ORQA-000, ORQA-005 and ORQA-006 and released them as version 2 of the bank with a corrections file. Two models (GPT-4.1 and Gemini 2.5 Flash) flagged 34% of questions (both agree) or 44% (majority) as silent on a needed condition. The ranking of all fifteen models is identical with and without them. Re-running the models on the corrected questions: 0 of 99 answers (11 models, 3 runs each) correct with the original question text, 78 of 99 with the corrected text.

Generator-only baseline: On the same 1,885 source passages the generator alone produced 1,373 questions, and the full pipeline kept 42. A blind re-grade of 200 generator-only and 171 released questions with the rubric the human annotators wrote found 34.5% of generator-only questions bad and 21.5% able to pass the gate, against 0.6% bad and 80.1% able to pass for released questions.

Corrections to the manuscript: The wage-bill weighting script weighted questions by mean wage alone, without employment, so we regenerated the panel under the stated definition (employment times mean wage per occupation) and the three leading models keep their order. We have regenerated Table 8 so that it follows SOC codes and matches Table 4. We have corrected the rater-cap wording. We have added a definition of what ORQA measures (documented professional guidance) to Section 3, stated that it does not measure hands-on skill, and labelled every occupation-level result exploratory.

We hope these additional experiments help address the reviewers' common concerns and provide a more substantiated picture of the paper's contributions. We would be grateful if the reviewers could take this new evidence into consideration in their overall reassessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.