- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SMILE is a framework for constructing agent benchmarks from expert descriptions of how a task should be solved. It organizes this knowledge into workflow subquestions, skills, and atomic rules, and connects these to deliverables and grading criteria. The accompanying KIIA benchmark contains 196 tasks across nine Korean professional domains. The paper evaluates agents on the benchmark, analyzes their trajectories and submitted artifacts, and uses the rule library to generate additional tasks through SMILE-Forge. The synthesis experiments measure rule and skill coverage and textual diversity.
Critical strengths
-
The annotations explain why a particular decision is required, not just what the final answer should contain. Separating conceptual knowledge, judgment procedures, codified requirements, and practical heuristics helps here. The links between these rules, workflow steps, and grading criteria should make it easier to inspect a failed task and locate the missing judgment.
-
The task examples are convincing. They require agents to reconcile records, distinguish legal requirements, check evidence, and produce documents or payment decisions that someone could actually use. Several examples expose mistakes that a fluent response would hide. The task collection is the part of the paper I would most like to see released. Its expert review records are helpful, although the depth of review varies across domains.
Critical weaknesses
-
I am not yet confident in the grading of agent outputs. The Gemini judge prompts are documented, and some checks are deterministic, but I could not find a comparison with independent expert grades. A mistaken gate decision can set an entire task score to zero, so the judge's error rate could materially affect the model comparisons. The trajectory analysis has a related limitation: groundedness is nearly saturated even for agents with much weaker final scores, and the judge sees the final submission. I do not think these scores establish that the agents recalled the right rules and then failed specifically at applying them.
-
The claims about saving resources, maintaining tasks, and expanding the benchmark need more evidence. Expert hours are reported, but there is no comparison showing that this procedure saves effort. The rule-to-task links could help with maintenance, but no actual update and its cost are shown. The 113 synthesized tasks are assessed mainly through coverage and textual diversity; in the matched comparison, they contain fewer skills and atomic rules than the originals. That is not necessarily a problem, but less repeated wording and new rule identifiers do not tell us whether the tasks are valid or expose useful agent failures. I would either narrow these claims or add direct validation.
Questions for authors
-
Have domain experts independently graded a sample of agent outputs, especially gate failures and borderline cases? If so, how was the sample chosen, how often did experts disagree with the judge, and did correcting those decisions change the model ordering?
-
Have you already compared construction effort with another procedure, tested a rule update, or had experts review the synthesized tasks? Please separate results already available from plans for future work. For the synthesized tasks, how often did experts find an incorrect requirement, an unsupported rule, or an ambiguous grading criterion?
[W1/Q1. Expert grading of agent outputs]
We thank the reviewer for these questions. We have added a comparison with independent expert grades of a sample of agent outputs (Appendices D and E).
[W1. Trajectory score saturation]
We thank the reviewer for this comment. Groundedness saturated because the judge rarely checked whether a step violated a linked AR, and we added a separate query for this check (Appendix C.4, Table 15). The added query also receives the gold deliverable, which it uses only as a reference for the correct application of each rule. In a pilot on 174 steps, the pass rate falls from 97.1% to 77.6% without the gold deliverable and to 72.4% with it, so most of the change comes from checking each AR separately (Appendix C.4). Groundedness is now 73.8–86.4 instead of 94.5–98.7, and its correlation with deliverable scores across models rises from 0.17 to 0.93. It no longer stays high when the final score is low: among the runs whose deliverable score is 0, the share with a groundedness of 100% falls from 63.0% to 7.6%. It also separates models: all seven models pass 82.4% of the steps without the check, and 47.6% with it. We also agree that these scores do not show that agents recalled the right rules and failed only at applying them, because the judge also sees the final submission. The submitted Section 6.3 suggested this, and we removed it.
[W2/Q2. Resource, maintenance and synthesis claims]
We thank the reviewer for this comment. As the reviewer suggests, we separate what is done from what is not, and we narrowed the claim on resources.