acceptodds
Under review as a conference paper at ICLR 2027

SMILE: Eliciting Expert Derivational Knowledge for Agent Benchmarks

Abstract

Evaluating AI agents on real-world work requires realistic inputs (prompts and references), deliverables, and expert-defined evaluation criteria (rubrics). Rubrics encode experts' criterial knowledge for assessing deliverables. These components alone do not make explicit the expert knowledge used to derive deliverables from inputs. We represent this derivational knowledge as atomic rules (ARs). Here, we introduce SMILE, a resource-efficient framework for building real-world agent benchmarks comprising inputs, outputs, and the derivation process connecting them. Expert-annotated or document-grounded ARs support evaluation of rule application in agent trajectories and expose relationships between tasks. These links support task maintenance: when a rule changes, its associated tasks can be identified and revised. They also support synthesis by recombining ARs into new tasks. As an instantiation of SMILE, we introduce SMILE, which comprises tasks from nine domains spanning Korean industry and public institutions. Our evaluation of frontier models reveals incomplete task execution and failures to satisfy critical requirements.

Primary area:
datasets and benchmarks
Keywords:
Agent Benchmarks, Derivational Knowledge

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zCCiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SMILE is a framework for constructing agent benchmarks from expert descriptions of how a task should be solved. It organizes this knowledge into workflow subquestions, skills, and atomic rules, and connects these to deliverables and grading criteria. The accompanying KIIA benchmark contains 196 tasks across nine Korean professional domains. The paper evaluates agents on the benchmark, analyzes their trajectories and submitted artifacts, and uses the rule library to generate additional tasks through SMILE-Forge. The synthesis experiments measure rule and skill coverage and textual diversity.

Critical strengths
  1. The annotations explain why a particular decision is required, not just what the final answer should contain. Separating conceptual knowledge, judgment procedures, codified requirements, and practical heuristics helps here. The links between these rules, workflow steps, and grading criteria should make it easier to inspect a failed task and locate the missing judgment.

  2. The task examples are convincing. They require agents to reconcile records, distinguish legal requirements, check evidence, and produce documents or payment decisions that someone could actually use. Several examples expose mistakes that a fluent response would hide. The task collection is the part of the paper I would most like to see released. Its expert review records are helpful, although the depth of review varies across domains.

Critical weaknesses
  1. I am not yet confident in the grading of agent outputs. The Gemini judge prompts are documented, and some checks are deterministic, but I could not find a comparison with independent expert grades. A mistaken gate decision can set an entire task score to zero, so the judge's error rate could materially affect the model comparisons. The trajectory analysis has a related limitation: groundedness is nearly saturated even for agents with much weaker final scores, and the judge sees the final submission. I do not think these scores establish that the agents recalled the right rules and then failed specifically at applying them.

  2. The claims about saving resources, maintaining tasks, and expanding the benchmark need more evidence. Expert hours are reported, but there is no comparison showing that this procedure saves effort. The rule-to-task links could help with maintenance, but no actual update and its cost are shown. The 113 synthesized tasks are assessed mainly through coverage and textual diversity; in the matched comparison, they contain fewer skills and atomic rules than the originals. That is not necessarily a problem, but less repeated wording and new rule identifiers do not tell us whether the tasks are valid or expose useful agent failures. I would either narrow these claims or add direct validation.

Questions for authors
  1. Have domain experts independently graded a sample of agent outputs, especially gate failures and borderline cases? If so, how was the sample chosen, how often did experts disagree with the judge, and did correcting those decisions change the model ordering?

  2. Have you already compared construction effort with another procedure, tested a rule update, or had experts review the synthesized tasks? Please separate results already available from plans for future work. For the synthesized tasks, how often did experts find an incorrect requirement, an unsupported rule, or an ambiguous grading criterion?

1 comment

[W1/Q1. Expert grading of agent outputs]

We thank the reviewer for these questions. We have added a comparison with independent expert grades of a sample of agent outputs (Appendices D and E).

  • In each of seven domains, we drew one task at random from each of the low-, middle- and high-score groups of Opus 5 (21 tasks, including two tasks set to 0 by a gate). Experts marked each rubric item of the Opus 5 deliverables as met or not met (676 labels). Coordinating with the partner organizations took longer than the rebuttal period allows, so we report the seven domains whose expert grading and gate review arrived in time. We will add the other domains' results to the paper when they arrive.
  • Averaged over domains, the original judge (Gemini 3.8 Flash) agrees with the experts on 84.6% of the labels, and κ is 0.40. With GPT-6.1-Sol or Opus 5.5 as the judge, κ is 0.58 or 0.59 (Opus 5.5 refused to judge two tasks, which are excluded for it). For comparison, κ between two experts is 0.57 in the two domains where two experts graded the same items.
  • The original judge tends to be less strict than the experts: it marks as not met 43.6% of the labels that experts mark as not met (Table 18).
  • Of 53 gates in seven domains, two experts both accept 79%, both reject 8% and disagree on 13%.
  • On the 49 expert labels of gate items, the original judge agrees with the experts on 85.7% (κ = 0.15), and GPT-6.1-Sol on 87.8% (κ = 0.56) (Appendix D.1).
  • On these tasks, Opus 5 scores 74.2 under the original judge and 55.8 under the expert labels (Table 19).

[W1. Trajectory score saturation]

We thank the reviewer for this comment. Groundedness saturated because the judge rarely checked whether a step violated a linked AR, and we added a separate query for this check (Appendix C.4, Table 15). The added query also receives the gold deliverable, which it uses only as a reference for the correct application of each rule. In a pilot on 174 steps, the pass rate falls from 97.1% to 77.6% without the gold deliverable and to 72.4% with it, so most of the change comes from checking each AR separately (Appendix C.4). Groundedness is now 73.8–86.4 instead of 94.5–98.7, and its correlation with deliverable scores across models rises from 0.17 to 0.93. It no longer stays high when the final score is low: among the runs whose deliverable score is 0, the share with a groundedness of 100% falls from 63.0% to 7.6%. It also separates models: all seven models pass 82.4% of the steps without the check, and 47.6% with it. We also agree that these scores do not show that agents recalled the right rules and failed only at applying them, because the judge also sees the final submission. The submitted Section 6.3 suggested this, and we removed it.

[W2/Q2. Resource, maintenance and synthesis claims]

We thank the reviewer for this comment. As the reviewer suggests, we separate what is done from what is not, and we narrowed the claim on resources.

  • We made one rule update, whose links identified four skills and three tasks to update (Figure 4). We also generated 700 synthetic tasks at about USD 1 to 4 per task, report the scores of four models on them (Appendix C.2.2–C.2.3), and revised the trajectory scores (Appendix C.4).
  • We have not compared construction effort with another procedure, and we have not measured the cost of the rule update. No expert reviewed the synthetic tasks, so we cannot report how often they contain an incorrect requirement, an unsupported rule or an ambiguous grading criterion.
  • We removed "resource-efficient" from the abstract, introduction and conclusion. The paper reports expert review time for most domains (Appendix B.2) and does not claim a saving.
Reviewer_ETJBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper starts with the motivation that agent evals require expert knowledge which is implicitly captured in inputs, evaluation criteria etc. The paper argues that this knowledge should be explicitly captured in "atomic rules" (ARs). The author present SMILE, a framework to represent such ARs and KIIA, a Korean benchmark of 196 tasks from 9 industries.

The main part of the paper is then describing how the authors create this KIIA benchmark and the dataset using a mix of expert-provided seeds and LLM / coding-agent-based generation. The authors show experiments with several LLMs on this dataset.

The paper, unfortunately, stays very high-level on the AR - they are never formally defined. In the end the ARs seems to be natural language rubrics, similar to grading notes or assertions, leaving it questionable where the novelty and contribution of the paper is.

Critical strengths
  • The paper creates a benchmark for several Korean industries. This can be a meaningful contribution for a resource and dataset track. The details on how the required environment is created and how realistic this is would stll need to be strengthened (e.g. do you have realistic accounting documents, payroll, project records, for the "Review R&D tax credit" task) but the benchmark seems to be the main concrete contribution of the paper.
Critical weaknesses

The paper's core contribution—Atomic Rules (ARs)—lacks clear definition, motivation, and demonstrated novelty. This undermines the entire SMILE framework.

Underspecified Core Concept: While the motivation for ARs appears, interesting, the paper never rigorously defines what ARs are or how they differ from existing evaluation rubrics. The authors present ARs as "atomic rules" capturing expert knowledge, but never formalise the concept, it doesn't even provide a clear motivating example. I assume that the blue boxes in Figure 2 are ARs. Statements like "Verify claimed costs against source records." seem to be rather task specific checks or grading notes which are specific to the task. This does not match the claim that ARs are task independent (l. 160). ARs, thus appear substantively similar grading notes and assertions from prior work (Barres et al., 2025; Chong et al., 2025). Without a clear delineation of what makes ARs distinct, the paper's central claim of methodological novelty is not established.

Insufficient Motivation and Structure: The introduction fails to explain why ARs are necessary beyond existing rubrics, how they are systematically structured, or provide concrete examples of AR design. Readers cannot assess whether ARs can adequately capture the multifaceted aspects required for rigorous agent evaluation. Without this foundation, the subsequent framework feels poorly motivated.

Experiments Don't Support Core Claims: If the paper's thesis is that ARs improve evaluation quality, the experiments should directly demonstrate this through comparative evaluation. Specifically, the paper should (1) compare AR-based evaluation against alternative rubrics on the same tasks, and (2) show that AR-based scores correlate more strongly with human judgment. No such controlled comparison is conducted. The paper instead reports model performance under AR evaluation but provides no evidence that ARs themselves are superior to other evaluation approaches.

Atomicity and Reusability Undemonstrated: The paper claims ARs are "reusable and atomic," yet provides no systematic analysis of how individual ARs generalize across tasks, domains, or model types. No statistics on AR overlap, abstraction level, or compositionality are presented. This makes it impossible to verify whether ARs truly function as claimed minimal, reusable units of knowledge or whether they are task-specific annotations that happen to be written in natural language.

References

  • Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ-bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025.

  • Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis Penny Chong, Harshavardhan Abichandani, Jiyuan Shen, Atin Ghosh, Min Pyae Moe, Yifan Mai, Daniel Dahlmeier

Questions for authors
  • The paper starts with the motivation that expert knowledge should be extracted into ARs but Figure 3 suggests that the tasks are generated based on the ARs, instead of the other way round.
1 comment

[W1. AR definition]

We thank the reviewer for the comment. We refined the definition of ARs, and we clarify how they differ from the grading notes of Chong et al. and the assertions of Barres et al. Section 2 (Related Work) now cites both works.

  • The submitted Section 3.1 defined ARs in one sentence. The revised Section 3.1 gives a definition and an example of each of the four types (C1–C4).
  • The example that the reviewer quotes from Figure 2, "Verify claimed costs against source records", is not specific to that task. It applies to any task that reviews claimed costs.
  • A grading note (Chong et al.) or an assertion (Barres et al.) is written for one task and states what the agent must do or produce in that task. The atomic subtasks of Barres et al. are problems to be resolved, which are composed into tasks. An AR is written once, does not mention a task, and is linked to every task that uses it. Of the 1,438 ARs, 392 are used by two or more tasks (Appendix C.3).

[W2. Why ARs are needed beyond rubrics]

We thank the reviewer for this comment. A rubric states what a correct deliverable contains, whereas an AR states the knowledge used to produce it. ARs therefore support three things that rubrics do not.

  • Evaluation becomes finer. Checking each linked AR at each workflow step shows where a model fails and separates models: all seven models pass 82.4% of the steps without the check, and 47.6% with it (Sections 4 and 6.1, Appendix C.4).
  • New tasks can be synthesized to extend the benchmark. We generated 700 tasks from ARs at about USD 1 to 4 per task, and the tendency of Table 2 is maintained on them (Section 6.2, Appendix C.2.2–C.2.3).
  • Tasks can be maintained. When an expert denied one Retail Banking AR, its links identified the four skills and three tasks to update (Section 6.3, Appendix C.3).

[W3. AR-based vs. rubric-based evaluation]

We thank the reviewer for this comment. We do not claim that ARs replace rubrics. In SMILE, rubrics score the deliverable, and ARs are used to check each step of the trajectory.

  • For the comparison on the same tasks, we compare the trajectory evaluation of the same runs with and without the AR check. Without the check, all seven models pass 82.4% of the steps. With it, they pass 47.6% (Appendix C.4).
  • For the correlation with human judgment, we now report the agreement of the rubric scores with expert grades. The original judge agrees with the experts on 84.6% of the labels (κ = 0.40), and flagship judges reach κ = 0.58 and 0.59 (Appendix D).

[W4. Atomicity and reusability]

We thank the reviewer for this comment. We added reuse statistics (Appendix C.3, Table 14). Of the 1,438 ARs, 392 (27.3%) are used by two or more tasks. For atomicity, ARs are split by two operational criteria (Section 3.1): a non-expert given the rule can solve the task by logical reasoning, and the rule is not tied to one task. Because knowledge can be stated at many levels of abstraction, we define the unit operationally by these two criteria rather than formally, and domain experts reviewed the resulting ARs (Appendix B.1, Table 6).

[Q. Direction of Figure 3]

We thank the reviewer for this question. Figure 3 shows both directions. KIIA is built from expert scenarios and expert review, and its ARs are annotated as part of each task. Forge then generates new tasks (KIIA-Syn) from these ARs.

[S. Realism of references]

We thank the reviewer for this suggestion. We added a paragraph on the input materials to Appendix B. The input records are created in three steps (Section 3.2).

  • First, experts provide seed scenarios and source material. The source material is real wherever possible: the full text of the relevant statutes from the national law database (Tax Advisory, Clinical Research Ethics), official government guidelines (Clinical Research Ethics), official statutory forms (Tax Advisory, Hospital Care), published papers and public database snapshots such as BacDive (Science R&D), the HWPX forms used in practice (Public R&D Administration), and, in some Retail Banking tasks, values from the partner organization's database with pseudonymized columns.
  • Second, researchers draft the case records, such as payrolls or research protocols, with a coding agent. The records follow the structure of the official forms. In Clinical Research Ethics, we analyzed real IRB submission documents with the help of the expert and wrote fictional documents with the same structure.
  • Third, experts review whether the task and its references reflect actual practice, and we revise them.

For instance, the R&D tax credit task that the reviewer mentions (Figure 2, Tax Advisory) gives a payroll, an attendance summary, a staff roster, outsourcing contracts and a project plan, with fictional contents in the formats of real documents.

Reviewer_Pw98OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SMILE represents expert derivational knowledge as reusable atomic rules linked to skills and task workflows. These links support trajectory evaluation, task maintenance, and synthesis. The authors introduce KIIA, containing 196 tasks across nine Korean professional domains, with 1,446 rules and expert involvement in construction. Evaluation combines deliverable rubrics with LLM-based trajectory judgments. The paper reports difficulties in knowledge application and rationale generation, and adds 113 synthetic tasks with expanded aggregate rule coverage and lower lexical repetition.

Critical strengths
  1. Valuable professional task resource. The benchmark includes concrete deliverables and domain-specific requirements in taxation, clinical research ethics, banking, and other Korean institutional settings.
  2. Useful explicit knowledge representation. Linking rules, evidence, and workflows makes task assumptions inspectable. Expert feedback exposes errors that internally consistent generated materials can overlook.
  3. Detailed protocol documentation. The appendices describe annotation arrangements, execution and grading prompts, and limitations of the synthesis results.
Critical weaknesses
  1. The framework's incremental benefit is not established. The paper does not show whether ARs improve diagnosis over rubric-only or workflow-only evaluation, reduce construction cost, or enable effective maintenance after a knowledge change. Its methodological contribution therefore remains largely a plausible integration of existing components.

  2. Scoring reliability is insufficiently validated. Expert review during construction does not establish agreement between experts and the automated judges. Most outcome judges receive no original references and treat rubric facts as true (Appendix H.2.1). With 525 gates capable of zeroing entire tasks, independent validation of rubric correctness and gate judgments is particularly important.

  3. High trajectory scores have an unresolved interpretation. Table 4 approaches ceiling in many domains despite substantially lower outcome scores. The examples support the existence of application failures, but the aggregate pattern could also reflect coarse workflow steps or insensitive judging. The diagnostic value of the process metric needs validation.

  4. Dataset and synthesis validity are unevenly demonstrated. Manufacturing and Defence receives scenario-only expert review. For synthetic tasks, additional rule IDs and lower TF-IDF similarity do not establish professional quality or new reasoning demands. At matched task counts, synthetic tasks use substantially fewer distinct skills and fewer skills per task (Table 8), limiting claims about richer capability coverage.

Questions for authors
  1. Which reported result demonstrates an AR-specific advantage in diagnostic accuracy, construction cost, or task maintenance over a simpler baseline?
  2. What independent expert audits or judge-human agreement measurements exist for final rubric labels, gates, and trajectory states?
3 comments

[W1/Q1. AR-specific benefit over simpler baselines]

We thank the reviewer for the comment. The revised paper shows an AR-specific benefit for maintenance and for diagnosis.

  • For maintenance, an expert denied a Retail Banking AR, and its links identified the four skills and three tasks to update (Section 6.3, Figure 4).
  • For diagnosis, the check of each linked AR adds two things (Appendix C.4, Table 15). First, it shows at which step a model fails: 19.1% of the steps that passed before violate a linked AR, and among the runs whose deliverable score is 0, the share with a groundedness of 100% falls from 63.0% to 7.6%. Second, it separates models: the share of steps that all seven models pass falls from 82.4% to 47.6%, groundedness changes from 94.5–98.7 to 73.8–86.4, and its correlation with deliverable scores across models rises from 0.17 to 0.93.
  • For construction cost, we did not compare construction effort with a procedure without ARs, and we removed "resource-efficient" from the paper. We report only the generation cost of synthetic tasks, about USD 1 to 4 per task.

[W2/Q2. Scoring reliability: judge vs. expert]

We thank the reviewer for this comment. We now report agreement with experts for rubric labels and gates (Appendices D and E).

  • In each of seven domains, we drew one task at random from each of the low-, middle- and high-score groups of Opus 5 (21 tasks). Experts marked each rubric item of the Opus 5 deliverables as met or not met (676 labels). Coordinating with the partner organizations took longer than the rebuttal period allows, so we report the seven domains whose expert grading and gate review arrived in time. We will add the other domains' results to the paper when they arrive.
  • Averaged over domains, the original judge (Gemini 3.8 Flash) agrees with the experts on 84.6% of the labels, and κ is 0.40. With GPT-6.1-Sol or Opus 5.5 as the judge, κ is 0.58 or 0.59 (Opus 5.5 refused to judge two tasks, which are excluded for it). For comparison, κ between two experts is 0.57 in the two domains where two experts graded the same items.
  • The original judge tends to be less strict than the experts: it marks as not met 43.6% of the labels that experts mark as not met (Table 18).
  • Of 53 gates in seven domains, two experts both accept 79%, both reject 8% and disagree on 13% (Appendix E).
  • On the 49 expert labels of gate items, the original judge agrees with the experts on 85.7% (κ = 0.15), and GPT-6.1-Sol on 87.8% (κ = 0.56) (Appendix D.1).

[W3. Why trajectory scores are near ceiling]

We thank the reviewer for this fruitful comment. As the reviewer correctly indicated, insensitive judging caused the near-ceiling scores: the judge rarely checked whether a step violated a linked AR. With a separate query for this check and the same workflow steps, groundedness falls from 94.5–98.7 to 73.8–86.4, and its correlation with deliverable scores (mean over the eight document domains in Table 2) across models rises from 0.17 to 0.93 (Appendix C.4, Table 15).

[W4. Dataset and synthesis validity]

We thank the reviewer for this comment. It raises two points, and we answer them separately.

  • On the synthetic tasks, we agree that new rule IDs and lower TF-IDF similarity do not establish their quality. We therefore generated 700 new synthetic tasks, 100 in each of seven selected domains, at about USD 1 to 4 per task, and evaluated four models on them (Appendix C.2.2–C.2.3). About 100 workers generate tasks in parallel, and each worker completes a task in about one hour. The tendency of Table 2 is maintained, but the synthetic tasks separate GPT-5.6-Sol and GPT-5.6-Terra less clearly than the expert-reviewed original tasks do: their average gap is 2.2 points, against 8.1. The average number of skills per task also decreases, from 7.5 to 4.8. These results suggest that the synthetic tasks, although they separate models less clearly than the expert-reviewed tasks, can extend the benchmark at low cost and complement the expert-reviewed tasks.
  • On Manufacturing and Defence, we agree. Experts reviewed only the scenarios in this domain, as Section 3 states.

Thank you for the maintenance example, the explicit checks for violations of linked atomic rules, and the expert audit. The new analysis addresses my concern about why the original trajectory scores were near ceiling. My remaining question is about the practical effect of gate disagreements: the original judge’s κ against expert gate labels is 0.15, and a single gate can zero an entire task. In the audited cases, how often would using the expert gate labels change whether a task receives a zero?

We thank the reviewer for the follow-up question. With the expert gate labels in place of the gate verdicts of the original judge (Gemini 3.8 Flash), whether a task receives a zero stays the same in 12 of the 16 expert gradings of tasks with gate items, and changes in 4. The table below gives the counts for all four judges.

JudgeCases comparedChangesJudge: non-zero, expert: zeroJudge: zero, expert: non-zero
Gemini 3.8 Flash (original)16431
Sonnet 5.516523
GPT-6.1-Sol16413
Opus 5.514220

Opus 5.5 refused to judge two Science R&D tasks, so it is compared on 14 cases. We will add this table to Appendix D.

Comments on the paper

1 comment

We thank all three reviewers for their constructive feedback. The reviews share four concerns: LM-judge reliability (R1, R3), the definition and role of atomic rules (R1, R2), near-ceiling trajectory scores (R1, R3), and the validity of the synthetic tasks and of our cost and maintenance claims (R1, R3). Changes are marked in red in the revised PDF.

In brief: (1) LM-judge reliability. On 21 sampled tasks graded by experts, the original judge (Gemini 3.8 Flash) agrees with the experts on 84.6% of the labels (κ = 0.40). Flagship judges (GPT-6.1-Sol and Opus 5.5) reach κ = 0.58 and 0.59, and κ between two experts is 0.57 in the two domains graded by two experts. (2) Atomic rules. We refined the definition of atomic rules in Section 3.1, with an example of each of the four types, and we report their reuse across tasks. (3) Trajectory scores. The trajectory scores were near the ceiling because the judge decided each workflow step in a single query and rarely checked whether the agent violated an atomic rule linked to that step. We added a separate query that includes a concrete procedure for this check. The trajectory score (groundedness) then falls from 94.5–98.7 to 73.8–86.4, its correlation with deliverable scores across models rises from 0.17 to 0.93, and the share of steps that all seven models pass falls from 82.4% to 47.6%. (4) Synthetic tasks and cost claims. We generated 700 new synthetic tasks, report their cost and the scores of four models, and removed the untested claim of resource efficiency.

The detailed responses are below, under the following titles.

  • (1) LM-judge reliability
    • R1 [W2/Q2. Scoring reliability: judge vs. expert]
    • R3 [W1/Q1. Expert grading of agent outputs]
  • (2) Atomic rules
    • R1 [W1/Q1. AR-specific benefit over simpler baselines]
    • R2 [W1. AR definition]
    • R2 [W2. Why ARs are needed beyond rubrics]
    • R2 [W4. Atomicity and reusability]
  • (3) Trajectory scores
    • R1 [W3. Why trajectory scores are near ceiling]
    • R3 [W1. Trajectory score saturation]
  • (4) Synthetic tasks and cost claims
    • R1 [W4. Dataset and synthesis validity]
    • R3 [W2/Q2. Resource, maintenance and synthesis claims]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.