- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ORQA, a framework for constructing source-traceable occupational knowledge questions from authoritative documents linked to O*NET occupations. Its pipeline combines evidence extraction, question generation, automated verification, difficulty filtering, and human review. The resulting benchmark contains 480 questions spanning 116 occupations and 187 source hosts. Fifteen models are evaluated using closed-book multiple-choice and open-ended questions, supplemented by retrieval experiments, economic weighting, and comparisons with external benchmarks. An audit involving 64 practitioners provides additional evidence about answer agreement, source support, and occupational relevance. The main contribution is a framework and evaluation resource for source-grounded professional knowledge across a broad range of occupations, including those underserved by benchmarks focused on digital work.
Critical strengths
-
The framework provides an auditable approach to occupational knowledge evaluation. Linking questions to occupations, source documents, and supporting excerpts makes the intended ground truth inspectable and enables coverage beyond common digital tasks. This combination of occupational breadth and source traceability constitutes a useful contribution when interpreted as evaluating documented professional knowledge.
-
The paper validates the resource from several complementary perspectives. The practitioner audit provides evidence beyond automated verification, while open-ended evaluation, retrieval conditions, and clustered uncertainty estimates help characterize what the benchmark measures. The released responses also support independent checking: I reproduced the principal closed-book accuracies and approximately reproduced the paired confidence interval comparing Claude Opus 4.6 with GPT-5.4. Together, these analyses provide meaningful support for the resource and its aggregate results.
Critical weaknesses
-
Some questions omit conditions necessary to determine which professional rule applies. ORQA-005 and ORQA-006 ask broadly about actuarial continuing education, but their source excerpts concern General Qualification Standards and Specific Qualification Standards, respectively. The stems do not preserve this distinction. Similarly, ORQA-000 asks about an actor generally, although the cited requirement specifically concerns an Equity actor or stage manager. These examples raise a concrete concern about preserving applicability conditions during question generation; they do not establish that the underlying requirements contradict each other. The practitioner audit provides useful positive evidence, but its restriction to occupations recruitable through an online participant platform limits generalization to the rest of the bank. The prevalence and impact of this particular question-specification problem remain unclear. High retrieval accuracy alone cannot resolve it, because retrieved passages may supply context missing from the closed-book question.
-
The occupation-specific and economic interpretations are less well supported than the aggregate benchmark results. In the released bank, 58 of 116 occupations have at most three questions, and all three Sheet Metal Workers questions come from one document. The paper appropriately acknowledges small-sample limitations, but suggestions about using these results for occupation-specific model selection should remain explicitly exploratory. Separately, the supplied main_figure_iclr.py weights individual questions by occupational mean annual wage, without incorporating employment. This differs from the stated occupation-level wage-bill weighting and also makes total occupational weight depend on question count. The discrepancy requires clarification and correction before the economic robustness claim can be assessed, although it does not affect the unweighted closed-book scores.
Questions for authors
-
Question-specification clarification: What checks ensure that the question preserves necessary jurisdiction, agreement, qualification category, and effective-date conditions? Please address ORQA-000, ORQA-005, and ORQA-006 specifically. Do the existing human reviews or practitioner audit provide an estimate of how frequently such conditions are omitted? Any available analysis of model performance after correcting or excluding affected questions would materially inform my assessment of benchmark validity.
-
Implementation clarification: Which weighting implementation generated Figure 2? The supplied script assigns mean annual wage to individual questions, whereas the manuscript describes occupation-level weights based on employment multiplied by mean wage. Please reconcile the discrepancy and clarify whether the reported robustness conclusion holds when occupational mean accuracies are weighted according to the stated wage-bill definition.
-
Please clarify how Figure 1 was produced and whether generative AI tools were used. If so, please specify their role and update the AI Use Statement accordingly.
3 comments
Thank you for the detailed response and additional analyses. The corrected wage weighting, expanded coverage, and clarified claims address several of my concerns. However, the new audit suggests that omitted applicability conditions extend beyond the three corrected examples, and the rebuttal does not establish that this issue has been systematically resolved. The unchanged model ranking is reassuring but does not fully address question validity. I therefore maintain my rating.
We thank the reviewer for the follow-up. We agree that our previous response measured how often questions might omit a condition but did not show that the problem had been resolved. We have since extended the review to the whole bank, with human readers deciding each case, and describe below what was found and what was changed.
Identification of candidate questions: Two models from different providers checked whether each question states enough for a reader to know which rule applies, and a third resolved disagreements. Of the original 480 questions, 211 were flagged for review, 163 by both models. A flag does not mean the keyed answer is wrong, since in every case the answer is supported by its source; it means a condition from that source, such as the jurisdiction or the agreement in force, may need to be stated in the question, which is the issue the reviewer identified in ORQA-000, ORQA-005 and ORQA-006.
Human verification of flagged questions: Two human annotators independently read all 211 flagged questions against their sources, blind to the model labels. Both found a missing condition in 53 questions and at least one did in 95. A third reader then adjudicated the 42 disagreements and confirmed 17 more. That leaves 70 questions, 14.6% of the original bank, with a confirmed missing condition.
Review of unflagged questions: We also tested whether the models were missing cases. In the sample of 25 unflagged questions reported in our first response, both annotators found a gap in 8% and at least one in 36%, so automated screening alone was not enough. The human review was therefore extended to all 422 questions that passed the automated screen across the original and the added questions. Each was read against its source to check that it states what is needed to determine a single correct answer. Questions found to omit a condition were set aside for correction, so every question that passed screening has now had an individual human reading.
Correction of confirmed questions: All 70 confirmed questions from the original bank were rewritten to state the missing condition. A second reader checked every rewrite against its source to confirm that the question now says when the rule applies and does not give the answer away. 68 passed on the first check and two after a further revision. None had to be dropped. The same procedure was applied to the 226 questions added during the rebuttal: human review confirmed 33 that needed a condition stated, and these were revised with the same checks. The original and corrected wording of every revised question is documented, and questions flagged by the models but not confirmed by the readers keep their screening label in the released records.
Change to the construction pipeline: The reviewer's examples exposed a gap in our checks. A question could pass because its answer was entailed by the source even when the question did not say which rule applied. We have added a separate check that the question must contain the information needed to identify the governing rule before it enters the bank. Questions that fail it are corrected and re-checked before they are used.
Addressing question validity: These changes address the reviewer's concern about whether individual questions are valid. We now check not only whether the source supports the keyed answer but also whether the question itself gives a reader enough information to identify that answer, and every confirmed case of a missing condition has been corrected and checked again before it can be used. The revised process fixes problems with the questions themselves, independently of the finding that the overall results are unchanged.
the human readers have checked every flagged question, every confirmed case corrected and verified, and every question that passed screening read individually. All changes to the manuscript are marked in blue in the revision.
We appreciate the reviewer pressing on this. The examples pointed to a weakness our original pipeline did not handle, and the added check now closes it. We hope the individual checking and correction of the questions gives stronger evidence that they are sufficiently specified, and we would be grateful if the reviewer would take these changes into account when reconsidering their assessment. We are glad to answer anything further during the discussion period.
We thank the reviewer for their thorough review, and in particular for taking the time to reproduce our results from the released responses and to read our figure script. We agree with all three points and have addressed each of them below.
Omitted applicability conditions (W1, Q1): We agree that ORQA-000 keys a rule for Equity actors, and that ORQA-005 and ORQA-006 key the General and Specific Qualification Standards without saying which. We have corrected the question text to name the condition and released it as version 2 of the bank with a corrections file. The original release is kept alongside it, so the reported results remain reproducible. Re-running the eleven queryable models on the corrected questions confirms the diagnosis: with the original question text every model answered all three questions wrong on every run (0 of 99 answers across 11 models and 3 runs). With the condition stated, 78 of 99 are correct, and the ranking is unchanged. The verifier's ambiguity check rewrites the question text when the source carries a limitation such as a jurisdiction, but it did not catch a qualification category or an agreement type, and we have described this gap in the paper. On prevalence, we measured it directly. Two models from different providers, GPT-4.1 and Gemini 2.5 Flash, each labelled every question for a missing condition of the kind the reviewer describes, such as a jurisdiction or a qualification category. The two models agreed on 70% of questions (kappa 0.42). Where they disagreed, a third model, o3, decided. Both flag 163 of 480 questions (34%), and the majority estimate is 211 (44%). Flagged questions are harder (frontier mean 49.1% on them, 64.7% without), but the closed-book ranking of all fifteen models is identical with and without them. We have released the labels. The reviewer is also right that the audit covers only platform-recruitable occupations. The bank-wide labels let us check how representative those occupations are: questions in the 20 audited occupations are flagged slightly less often than the rest (28% against 35% on the conservative estimate), so the audit is a mildly favourable sample of the bank, and the bank-wide estimate is the one we rely on. Two human annotators then independently adjudicated a blind sample of 90 questions. Of the 50 flagged by both models, both annotators confirmed a missing condition on 32% and at least one annotator on 68%. Of 25 unflagged questions, both confirmed 8% and at least one 36%. Applied to the whole bank, about 18% of questions have a missing condition that two independent readers both confirm, against the 34% model estimate, and about 52% have one that at least one reader confirms, so the model estimate sits between the strict and lenient human readings. The two annotators agreed with each other on 62% of questions (kappa 0.28), which shows that whether a condition is needed is itself a judgment call.
Depth and wage weighting (W2, Q2): The reviewer is right on both counts: 58 occupations have at most three questions, and all three Sheet Metal Workers questions cite one document. We have added both facts to Section 4, added the number of distinct source documents per occupation to the occupation roster, and rewritten the model-selection suggestion as a direction for a larger bank. Dropping those occupations leaves the ordering unchanged except o3 and GPT-5.2, 0.3 points apart. Going further, we re-ran the pipeline with no changes to its quality checks, making further passes over the existing sources and searching for new ones. This added 226 questions for a bank of 706 and raised 23 further occupations to ten or more questions (e.g., Sheet Metal Workers from 3 to 10), and the ranking is unchanged. On weighting, the reviewer is right and we are grateful for the catch: the script weighted questions by mean wage without employment. We reproduced the submitted panel from that formula, then regenerated it as a weighted mean of occupation-level accuracies with OEWS employment times mean wage. The three leading models keep their order (Claude Opus 4.6 69.4%, GPT-5.4 67.7%, Claude Sonnet 4.6 66.3%). Below them GPT-5.3 moves above Gemini 3.1 Pro and Claude Haiku 4.5 below GPT-4o, so the claim is narrowed to the frontier ordering.
Figure 1 (Q3): Figure 1 was produced with OpenAI's Images 2.6 from a written description of the stages. The conceptual design and all text in the figure were provided by the authors, who reviewed it for accuracy against Section 3. No other figure was made this way, and we have added this disclosure to the AI Use Statement.
We are glad the reviewer found the framework auditable and the validation meaningful. None of the corrections changes the aggregate ranking. We would be grateful if the reviewer would consider these additions in reassessing the submission, and we are glad to answer anything further during the discussion period.