- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WorkCraft-Arena, a benchmark for evaluating multimodal models’ ability to create work artifacts from heterogeneous workspace materials. It consists of 100 tasks covering 10 professional roles and six artifact types, spanning a diverse range of real-world work scenarios. The authors further propose a unified agent execution and evaluation framework that evaluates faithfulness and aesthetics as separate dimensions, while distinguishing professional readiness from relative aesthetic preference. The framework also tracks model performance throughout the workflow, from material understanding and artifact creation to self-revision, enabling a more fine-grained assessment of the overall artifact-generation process.
Critical strengths
-
The benchmark spans 10 professional roles and six artifact types, with a relatively balanced distribution of evaluation cases across categories. This helps prevent the results from being dominated by particular professional settings or artifact types.
-
The framework evaluates faithfulness and aesthetics separately while examining performance across material understanding, content organization, final delivery, and self-revision. This provides a useful basis for analyzing models’ strengths and limitations in work artifact generation.
-
The authors involve professional designers in the evaluation and compare multiple candidate VLM judges against human judgments, providing empirical support for judge selection.
Critical weaknesses
-
The novelty and incremental contributions are not sufficiently clarified. The paper brings together work artifact generation, faithfulness evaluation, professional design standards, and aesthetic preference comparison, but its distinction from prior work is not sufficiently articulated. For example, HCB [1] already distinguishes professional consensus from individual aesthetic preferences, while PresentBench [2] provides fine-grained evaluation of slides generated from background materials. The main additions of WorkCraft-Arena appear to be its broader coverage of artifact types and its unified evaluation framework. However, the paper does not clearly explain what new evaluation capabilities these extensions enable beyond those offered by existing benchmarks.
-
Several experimental results appear inconsistent with the accompanying descriptions, making the results difficult to interpret. For example, Appendix G.3 states that Excel has the highest mean aesthetic score at 0.702, whereas Table 11 reports a higher score of 0.719 for Markdown. In addition, Appendix G.2 states that tasks without a delivered artifact receive a faithfulness score of zero. However, Kimi has a delivery rate of 0.42 and a mean score of 0.399 on delivered tasks, which would imply an overall score of approximately 0.168 under this rule, rather than the 0.312 reported in Table 10. If the overall score instead retains process-level credit for tasks without a delivered artifact, the authors should clarify the aggregation procedure and revise the corresponding description accordingly.
-
Independent human validation of the final benchmark results is lacking. The paper includes human verification during task construction and uses a subset of artifacts for judge calibration, but it does not provide sufficient evidence that the automated evaluation agrees with human judgments at the level of final model scores, system rankings, and key experimental conclusions. Additional independent human evaluation across different professional roles, artifact types, and quality levels would help assess whether the proposed evaluation criteria generalize beyond the calibration set and whether the resulting automated scores are reliable. If feasible, extending such validation to the full benchmark would provide substantially stronger evidence.
-
The definitions of the main evaluation metrics and the relationships among them are unclear. Table 3 reports overall scores, stage-level scores, and fine-grained diagnostic metrics, but Table 6 does not fully define these metrics or clearly establish how the metrics in the two tables are related. For example, the computation of Text rendering and Delivery readiness is not sufficiently explained. In addition, Aesthetics and AestheticPass appear to have different meanings in the text but are presented under a single label in Table 3. The authors should standardize the terminology and clearly specify the definition, evaluation unit, sample set, and aggregation procedure for each metric to improve both interpretability and reproducibility.
-
The manuscript contains several editorial errors that should be corrected. For example, “WorkCraft-Anera” in the contribution list on page 3 should be corrected to “WorkCraft-Arena,” and the caption of Table 11 on page 25 contains the unresolved reference “Table ??”. The authors should also carefully check the manuscript for spelling errors, broken cross-references, and inconsistencies in terminology.
[1] Hopkins, Aspen, et al. "The Human Creativity Benchmark." arXiv preprint arXiv:2606.30561 (2026). [2] Chen, Xin-Sheng, et al. "Presentbench: A fine-grained rubric-based benchmark for slide generation." arXiv preprint arXiv:2603.07244 (2026).
Questions for authors
-
What is the paper’s central contribution beyond the closest existing benchmarks, such as HCB and PresentBench? Does it primarily concern task design, evaluation methodology, or process-level diagnosis? How do the experiments demonstrate its distinct value?
-
What motivates the selection of the metrics in Table 3, their assignment to evaluation dimensions, and their aggregation procedures? How do the authors validate that these metrics measure the intended capabilities? Could the authors provide a complete mapping to Table 6 and clarify how individual metrics are aggregated into stage-level and overall scores?
6 comments
Response to Reviewer Jczr ( Part 2/5)
W2. Inconsistencies between results and descriptions
Several experimental results appear inconsistent with the accompanying descriptions, making the results difficult to interpret. For example, Appendix G.3 states that Excel has the highest mean aesthetic score at 0.702, whereas Table 11 reports a higher score of 0.719 for Markdown. In addition, Appendix G.2 states that tasks without a delivered artifact receive a faithfulness score of zero. However, Kimi has a delivery rate of 0.42 and a mean score of 0.399 on delivered tasks, which would imply an overall score of approximately 0.168 under this rule, rather than the 0.312 reported in Table 10. If the overall score instead retains process-level credit for tasks without a delivered artifact, the authors should clarify the aggregation procedure and revise the corresponding description accordingly.
Response. Thank you for finding both discrepancies. We rechecked the format-level table and the underlying run records.
Table 11. Markdown has the highest mean archived composite Aesthetics score, 0.719 (0.718621 across 76 valid records). XLSX is next at 0.702 (0.702386 across 100). Appendix G.3 incorrectly called XLSX the highest. The updated text gives the correct ordering and limits the interpretation to the evaluated task mix and score.
Table 10. Your estimate of approximately 0.168 follows from the original wording. Kimi's 42 scorable files comprise 24 formal finals and 18 drafts. The remaining 58 runs can retain measurable process credit and average 0.249953. The archived aggregation is , as reported in Table 10. The same decomposition gives 0.416283 for MiniMax.
| System | Formal finals | Draft only | No artifact | Composite with file | Composite without file | All-task composite |
|---|---|---|---|---|---|---|
| GPT-5.6-Sol | 96 | 4 | 0 | 0.673124 | N/A | 0.673124 |
| Claude Opus 5 | 93 | 3 | 4 | 0.686416 | 0.327175 | 0.672046 |
| Qwen 3.7 Plus | 97 | 3 | 0 | 0.558326 | N/A | 0.558326 |
| MiniMax M3 | 46 | 31 | 23 | 0.458847 | 0.273787 | 0.416283 |
| Kimi K3 | 24 | 18 | 58 | 0.398505 | 0.249953 | 0.312345 |
| Gemini 3.1 Pro | 99 | 0 | 1 | 0.394128 | 0.000000 | 0.390187 |
For scorable formal finals, Claude and GPT-5.6-Sol have conditional final-faithfulness means of 0.750 and 0.729. Across all tasks, with absent or unrenderable finals contributing zero, the means are 0.698 and 0.700. This makes explicit how both quality and delivery affect the outcome.
Revision. We clarified the main results and Table 3’s reporting conventions (Page 8, Lines 388–415). The appendix explains the provenance groups and the all-task denominator (Pages 24–25, Lines 1294–1309), reports strict-final and end-to-end quality (Page 25, Lines 1311–1327), and clarifies the delivery figure’s caption (Page 25, Lines 1342–1344). We also corrected the format-level description and Table 12’s caption (Pages 25–26, Lines 1348–1351).
W3. Independent validation of benchmark results
Independent human validation of the final benchmark results is lacking. The paper includes human verification during task construction and uses a subset of artifacts for judge calibration, but it does not provide sufficient evidence that the automated evaluation agrees with human judgments at the level of final model scores, system rankings, and key experimental conclusions. Additional independent human evaluation across different professional roles, artifact types, and quality levels would help assess whether the proposed evaluation criteria generalize beyond the calibration set and whether the resulting automated scores are reliable. If feasible, extending such validation to the full benchmark would provide substantially stronger evidence.
Response. Thank you for recognising our use of professional designers and for emphasising independent validation of the final results. In the original meta-evaluation, three professional designers scored 30 anonymised artifacts from five models across six formats. They assigned professional tiers, compared same-task designs, and repeated comparisons with the presentation order reversed.
| Original calibration measure | Comparison set | Selected judge |
|---|---|---|
| AestheticPass tier agreement | 64 items on which all three designers agreed | 0.855 |
| Design preference agreement | 132 pairs on which all three designers agreed | 0.652 |
| Design preference agreement | 396 matched judge–designer pairs | 0.571 |
| Order-swap consistency | Repeated pairs with order reversed | 0.851 |
These results informed judge selection and describe agreement and order stability on the calibration set. We agree that final scores, rankings, and conclusions warrant a separate test.
Response to Reviewer Jczr ( Part 3/5)
Our independent validation uses benchmark outputs from outside the judge calibration tasks, with coverage across formats, professional roles, and quality levels. Factual reviewers can consult the original materials and source locations. Design reviewers judge anonymised artifacts. We match human and automated judgments by task, artifact version, and evaluation target, and compare the six system rankings on the same validation subset.
| Target | Unit / statistic | Human–human | Automatic–human |
|---|---|---|---|
| Source facts, numbers, and constraints | Item-level Macro-F1 | 0.95 | 0.92 |
| AestheticPass professional readiness | Tier judgment, weighted κ | 0.87 | 0.88 |
| Same-task design preference | Pairs, agreement including ties | 0.82 | 0.86 |
| System-level final faithfulness ranking | Six systems, Kendall τ-b | 0.96 | 0.93 |
| System-level professional readiness ranking | Six systems, Kendall τ-b | 0.92 | 0.89 |
| System-level design preference ranking | Six systems, Kendall τ-b | 0.75 | 0.79 |
On this subset, automatic–human agreement is 0.92 Macro-F1 for facts, numbers, and constraints, 0.88 weighted κ for professional-readiness judgments, and 0.86 for matched design preferences. The corresponding ranking agreements are 0.93 for final faithfulness, 0.89 for professional readiness, and 0.79 for design preference. The differences across targets reinforce the value of reporting each one separately.
Revision. The original judge-calibration comparison is reported in Table 7 (Page 21, Lines 1099–1118). The appendix now explains how the independent human validation was conducted and reports the item-, pair-, and system-level results in Table 21 (Pages 29–30, Lines 1559–1576).
W4. Definitions and terminology in the main results table
The definitions of the main evaluation metrics and the relationships among them are unclear. Table 3 reports overall scores, stage-level scores, and fine-grained diagnostic metrics, but Table 6 does not fully define these metrics or clearly establish how the metrics in the two tables are related. For example, the computation of Text rendering and Delivery readiness is not sufficiently explained. In addition, Aesthetics and AestheticPass appear to have different meanings in the text but are presented under a single label in Table 3. The authors should standardize the terminology and clearly specify the definition, evaluation unit, sample set, and aggregation procedure for each metric to improve both interpretability and reproducibility.
Response. We apologise for the confusion caused by the original labels. Thank you for highlighting where Table 3 was difficult to trace to Table 6. We checked the scorecards and aggregation code and reproduce the three definitions here.
| Original label | Calculation and unit | Revised label |
|---|---|---|
| Text rendering | Mean of valid E2.FactRecall, E2.NumericAcc, E6.FactRecallFinal, and E6.FaithfulnessChecklist leaves per run, followed by a mean over scorable runs. This combines process and artifact evidence. | Textual fidelity composite |
| Delivery readiness | Completeness and BaselinePolish in AestheticPass. Fail, Pass, and Excellent map to 0, 0.6, and 1. We average the valid mapped facet scores per system. GPT-5.6-Sol's 0.527 is reproducible from the raw judgments. | DeliveryReadiness rubric group |
| Aesthetics / AestheticPass | The original line reports composite Aesthetics, the mean of valid E4 leaves, E6.AestheticPass, and applicable E6 visual leaves, first by run and then by system. Pure AestheticPass aggregates judge scores within facets, facets within groups, and then gives valid groups equal weight. | Aesthetics (composite) and AestheticPass as separate rows |
For Claude, composite Aesthetics is 0.690 and pure AestheticPass is 0.606 over 89 scored files. For GPT-5.6-Sol, the figures are 0.661 and 0.601 over 96 scored files. DeliveryReadiness measures the state of the design under the rubric. Formal delivery and renderability have their own indicators. Q2 gives the full mapping.
Revision. Table 3 now reports composite Aesthetics, AestheticPass, and Taste Elo separately and uses the revised textual-fidelity label (Page 8, Lines 409–415). Table 14 also reports the number of valid AestheticPass scores per system (Page 27, Lines 1404–1412). The metric appendix specifies the scoring leaves and averaging units for these results (Page 30, Lines 1600–1608) and the definition of delivery readiness (Page 31, Lines 1622–1623).
Response to Reviewer Jczr ( Part 1/5)
Thank you for recognising the task coverage, process diagnostics, and role of professional designers. We appreciate your detailed checks of the tables and metric definitions, and include concrete calculations and examples below.
W1. Contribution relative to HCB and PresentBench
Response. Thank you for identifying the closest comparisons. HCB distinguishes professional standards from individual taste and evaluates several creative phases. PresentBench provides task-specific criteria for slides grounded in background materials. Both directly informed our approach. HCB · PresentBench
Our main contribution is a reproducible benchmark of material-driven work-artifact tasks with traceable process records and versions. For the same deck, one can check professional quality and design preference, and also recover the task evidence, the formal final, and the self-check versions.
(1) Long-horizon evaluation from process evidence. Our 100 tasks contain 2,041 input files across 19 input types and six output formats. Agents find relevant facts, organise them for a brief, and create, inspect, and revise a file. The benchmark retains source locations, constraints, process files, snapshots, and the provenance of the scored file. A low score can therefore be traced to recorded evidence, final content, presentation, or delivery. At the same time, the evaluation checks professional-readiness criteria on single artifacts and compares design choices within the same brief.
(2) Evaluating models' aesthetic capabilities across formats. The two aesthetic rubrics start with format-appropriate general criteria and derive task-specific checks from the brief and annotations. The faithfulness DAG derives object, attribute, and relation checks from task evidence. We ask (RQ1) how a common protocol can evaluate aesthetics across work artifacts, (RQ2) what evidence and design challenges it reveals for foundation models, and (RQ3) what recording, delivery, and revision capabilities work-assistant harnesses need.
For example, MiniMax matched 1/22 facts in the process record and 21/22 in final HTML text. Claude matched 19/22 in the process record and 2/22 in its final Markdown text, with 17 process matches absent from the final-text matches. These cases give reviewers concrete sources and passages to inspect and illustrate why reporting stage and final results together is useful.
Revision. We stated the three research questions in the Introduction (Page 2, Lines 101–105) and clarified the contributions (Page 3, Lines 111–125). Related Work explains the relationship to PresentBench (Page 3, Lines 133–138) and HCB (Page 3, Lines 158–161). The appendix now includes two concrete cases linking process records to the evidence retained in scored artifacts (Page 27, Lines 1440–1446).