acceptodds
Under review as a conference paper at ICLR 2027

WorkCraft-Arena: How Can Models Turn Heterogeneous Workspaces to Well-Designed Work Artifacts?

Abstract

Multimodal models are increasingly serving as work assistants, turning heterogeneous workspace materials into slides, HTML pages, posters, and documents. Evaluating these artifacts requires more than checking task completion, because long-horizon creation can lose evidence between understanding and delivery. Additionally, existing aesthetic assessments lack an exploration of the professional delivery requirements and diverse design tastes of work artifacts. We introduce WorkCraft-Arena, a benchmark of 100 tasks across 10 professional work roles and six artifact types. We propose a unified design agent harness that records all creation processes from material understanding to self-revision for complete evaluation. Our arena has established the first comprehensive aesthetic evaluation system for work assistant artifacts, which separates faithfulness from aesthetics and further distinguishes AestheticPass, an absolute professional-readiness judgment, from AestheticElo, a pairwise measure of relative design preference. We conduct experiments across six frontier models and observe that faithfulness and aesthetics are almost orthogonal on the Pareto front. AestheticPass and AestheticElo will present different performance in the system. The results also show that factual grounding, design taste, and delivery reliability diagnose distinct capabilities for models. WorkCraft-Arena therefore provides both a credible measurement system for artifact aesthetics and diagnostic evidence for developing multimodal models and agent harnesses that produce reliable, improvable professional work.

Primary area:
datasets and benchmarks
Keywords:
multimodal benchmark, aesthetic evaluation, agent, process-level evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JczrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorkCraft-Arena, a benchmark for evaluating multimodal models’ ability to create work artifacts from heterogeneous workspace materials. It consists of 100 tasks covering 10 professional roles and six artifact types, spanning a diverse range of real-world work scenarios. The authors further propose a unified agent execution and evaluation framework that evaluates faithfulness and aesthetics as separate dimensions, while distinguishing professional readiness from relative aesthetic preference. The framework also tracks model performance throughout the workflow, from material understanding and artifact creation to self-revision, enabling a more fine-grained assessment of the overall artifact-generation process.

Critical strengths
  1. The benchmark spans 10 professional roles and six artifact types, with a relatively balanced distribution of evaluation cases across categories. This helps prevent the results from being dominated by particular professional settings or artifact types.

  2. The framework evaluates faithfulness and aesthetics separately while examining performance across material understanding, content organization, final delivery, and self-revision. This provides a useful basis for analyzing models’ strengths and limitations in work artifact generation.

  3. The authors involve professional designers in the evaluation and compare multiple candidate VLM judges against human judgments, providing empirical support for judge selection.

Critical weaknesses
  1. The novelty and incremental contributions are not sufficiently clarified. The paper brings together work artifact generation, faithfulness evaluation, professional design standards, and aesthetic preference comparison, but its distinction from prior work is not sufficiently articulated. For example, HCB [1] already distinguishes professional consensus from individual aesthetic preferences, while PresentBench [2] provides fine-grained evaluation of slides generated from background materials. The main additions of WorkCraft-Arena appear to be its broader coverage of artifact types and its unified evaluation framework. However, the paper does not clearly explain what new evaluation capabilities these extensions enable beyond those offered by existing benchmarks.

  2. Several experimental results appear inconsistent with the accompanying descriptions, making the results difficult to interpret. For example, Appendix G.3 states that Excel has the highest mean aesthetic score at 0.702, whereas Table 11 reports a higher score of 0.719 for Markdown. In addition, Appendix G.2 states that tasks without a delivered artifact receive a faithfulness score of zero. However, Kimi has a delivery rate of 0.42 and a mean score of 0.399 on delivered tasks, which would imply an overall score of approximately 0.168 under this rule, rather than the 0.312 reported in Table 10. If the overall score instead retains process-level credit for tasks without a delivered artifact, the authors should clarify the aggregation procedure and revise the corresponding description accordingly.

  3. Independent human validation of the final benchmark results is lacking. The paper includes human verification during task construction and uses a subset of artifacts for judge calibration, but it does not provide sufficient evidence that the automated evaluation agrees with human judgments at the level of final model scores, system rankings, and key experimental conclusions. Additional independent human evaluation across different professional roles, artifact types, and quality levels would help assess whether the proposed evaluation criteria generalize beyond the calibration set and whether the resulting automated scores are reliable. If feasible, extending such validation to the full benchmark would provide substantially stronger evidence.

  4. The definitions of the main evaluation metrics and the relationships among them are unclear. Table 3 reports overall scores, stage-level scores, and fine-grained diagnostic metrics, but Table 6 does not fully define these metrics or clearly establish how the metrics in the two tables are related. For example, the computation of Text rendering and Delivery readiness is not sufficiently explained. In addition, Aesthetics and AestheticPass appear to have different meanings in the text but are presented under a single label in Table 3. The authors should standardize the terminology and clearly specify the definition, evaluation unit, sample set, and aggregation procedure for each metric to improve both interpretability and reproducibility.

  5. The manuscript contains several editorial errors that should be corrected. For example, “WorkCraft-Anera” in the contribution list on page 3 should be corrected to “WorkCraft-Arena,” and the caption of Table 11 on page 25 contains the unresolved reference “Table ??”. The authors should also carefully check the manuscript for spelling errors, broken cross-references, and inconsistencies in terminology.

[1] Hopkins, Aspen, et al. "The Human Creativity Benchmark." arXiv preprint arXiv:2606.30561 (2026). [2] Chen, Xin-Sheng, et al. "Presentbench: A fine-grained rubric-based benchmark for slide generation." arXiv preprint arXiv:2603.07244 (2026).

Questions for authors
  1. What is the paper’s central contribution beyond the closest existing benchmarks, such as HCB and PresentBench? Does it primarily concern task design, evaluation methodology, or process-level diagnosis? How do the experiments demonstrate its distinct value?

  2. What motivates the selection of the metrics in Table 3, their assignment to evaluation dimensions, and their aggregation procedures? How do the authors validate that these metrics measure the intended capabilities? Could the authors provide a complete mapping to Table 6 and clarify how individual metrics are aggregated into stage-level and overall scores?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Jczr ( Part 1/5)

Thank you for recognising the task coverage, process diagnostics, and role of professional designers. We appreciate your detailed checks of the tables and metric definitions, and include concrete calculations and examples below.

W1. Contribution relative to HCB and PresentBench

The novelty and incremental contributions are not sufficiently clarified. The paper brings together work artifact generation, faithfulness evaluation, professional design standards, and aesthetic preference comparison, but its distinction from prior work is not sufficiently articulated. For example, HCB [1] already distinguishes professional consensus from individual aesthetic preferences, while PresentBench [2] provides fine-grained evaluation of slides generated from background materials. The main additions of WorkCraft-Arena appear to be its broader coverage of artifact types and its unified evaluation framework. However, the paper does not clearly explain what new evaluation capabilities these extensions enable beyond those offered by existing benchmarks.

Response. Thank you for identifying the closest comparisons. HCB distinguishes professional standards from individual taste and evaluates several creative phases. PresentBench provides task-specific criteria for slides grounded in background materials. Both directly informed our approach. HCB · PresentBench

Our main contribution is a reproducible benchmark of material-driven work-artifact tasks with traceable process records and versions. For the same deck, one can check professional quality and design preference, and also recover the task evidence, the formal final, and the self-check versions.

(1) Long-horizon evaluation from process evidence. Our 100 tasks contain 2,041 input files across 19 input types and six output formats. Agents find relevant facts, organise them for a brief, and create, inspect, and revise a file. The benchmark retains source locations, constraints, process files, snapshots, and the provenance of the scored file. A low score can therefore be traced to recorded evidence, final content, presentation, or delivery. At the same time, the evaluation checks professional-readiness criteria on single artifacts and compares design choices within the same brief.

(2) Evaluating models' aesthetic capabilities across formats. The two aesthetic rubrics start with format-appropriate general criteria and derive task-specific checks from the brief and annotations. The faithfulness DAG derives object, attribute, and relation checks from task evidence. We ask (RQ1) how a common protocol can evaluate aesthetics across work artifacts, (RQ2) what evidence and design challenges it reveals for foundation models, and (RQ3) what recording, delivery, and revision capabilities work-assistant harnesses need.

For example, MiniMax matched 1/22 facts in the process record and 21/22 in final HTML text. Claude matched 19/22 in the process record and 2/22 in its final Markdown text, with 17 process matches absent from the final-text matches. These cases give reviewers concrete sources and passages to inspect and illustrate why reporting stage and final results together is useful.

Revision. We stated the three research questions in the Introduction (Page 2, Lines 101–105) and clarified the contributions (Page 3, Lines 111–125). Related Work explains the relationship to PresentBench (Page 3, Lines 133–138) and HCB (Page 3, Lines 158–161). The appendix now includes two concrete cases linking process records to the evidence retained in scored artifacts (Page 27, Lines 1440–1446).

AuthorsOpenReview ↗

Response to Reviewer Jczr ( Part 2/5)

W2. Inconsistencies between results and descriptions

Several experimental results appear inconsistent with the accompanying descriptions, making the results difficult to interpret. For example, Appendix G.3 states that Excel has the highest mean aesthetic score at 0.702, whereas Table 11 reports a higher score of 0.719 for Markdown. In addition, Appendix G.2 states that tasks without a delivered artifact receive a faithfulness score of zero. However, Kimi has a delivery rate of 0.42 and a mean score of 0.399 on delivered tasks, which would imply an overall score of approximately 0.168 under this rule, rather than the 0.312 reported in Table 10. If the overall score instead retains process-level credit for tasks without a delivered artifact, the authors should clarify the aggregation procedure and revise the corresponding description accordingly.

Response. Thank you for finding both discrepancies. We rechecked the format-level table and the underlying run records.

Table 11. Markdown has the highest mean archived composite Aesthetics score, 0.719 (0.718621 across 76 valid records). XLSX is next at 0.702 (0.702386 across 100). Appendix G.3 incorrectly called XLSX the highest. The updated text gives the correct ordering and limits the interpretation to the evaluated task mix and score.

Table 10. Your estimate of approximately 0.168 follows from the original wording. Kimi's 42 scorable files comprise 24 formal finals and 18 drafts. The remaining 58 runs can retain measurable process credit and average 0.249953. The archived aggregation is 0.42×0.398505+0.58×0.249953≈0.3123450.42\times0.398505+0.58\times0.249953\approx0.312345, as reported in Table 10. The same decomposition gives 0.416283 for MiniMax.

SystemFormal finalsDraft onlyNo artifactComposite with fileComposite without fileAll-task composite
GPT-5.6-Sol96400.673124N/A0.673124
Claude Opus 593340.6864160.3271750.672046
Qwen 3.7 Plus97300.558326N/A0.558326
MiniMax M34631230.4588470.2737870.416283
Kimi K32418580.3985050.2499530.312345
Gemini 3.1 Pro99010.3941280.0000000.390187

For scorable formal finals, Claude and GPT-5.6-Sol have conditional final-faithfulness means of 0.750 and 0.729. Across all tasks, with absent or unrenderable finals contributing zero, the means are 0.698 and 0.700. This makes explicit how both quality and delivery affect the outcome.

Revision. We clarified the main results and Table 3’s reporting conventions (Page 8, Lines 388–415). The appendix explains the provenance groups and the all-task denominator (Pages 24–25, Lines 1294–1309), reports strict-final and end-to-end quality (Page 25, Lines 1311–1327), and clarifies the delivery figure’s caption (Page 25, Lines 1342–1344). We also corrected the format-level description and Table 12’s caption (Pages 25–26, Lines 1348–1351).

W3. Independent validation of benchmark results

Independent human validation of the final benchmark results is lacking. The paper includes human verification during task construction and uses a subset of artifacts for judge calibration, but it does not provide sufficient evidence that the automated evaluation agrees with human judgments at the level of final model scores, system rankings, and key experimental conclusions. Additional independent human evaluation across different professional roles, artifact types, and quality levels would help assess whether the proposed evaluation criteria generalize beyond the calibration set and whether the resulting automated scores are reliable. If feasible, extending such validation to the full benchmark would provide substantially stronger evidence.

Response. Thank you for recognising our use of professional designers and for emphasising independent validation of the final results. In the original meta-evaluation, three professional designers scored 30 anonymised artifacts from five models across six formats. They assigned professional tiers, compared same-task designs, and repeated comparisons with the presentation order reversed.

Original calibration measureComparison setSelected judge
AestheticPass tier agreement64 items on which all three designers agreed0.855
Design preference agreement132 pairs on which all three designers agreed0.652
Design preference agreement396 matched judge–designer pairs0.571
Order-swap consistencyRepeated pairs with order reversed0.851

These results informed judge selection and describe agreement and order stability on the calibration set. We agree that final scores, rankings, and conclusions warrant a separate test.

AuthorsOpenReview ↗

Response to Reviewer Jczr ( Part 3/5)

Our independent validation uses benchmark outputs from outside the judge calibration tasks, with coverage across formats, professional roles, and quality levels. Factual reviewers can consult the original materials and source locations. Design reviewers judge anonymised artifacts. We match human and automated judgments by task, artifact version, and evaluation target, and compare the six system rankings on the same validation subset.

TargetUnit / statisticHuman–humanAutomatic–human
Source facts, numbers, and constraintsItem-level Macro-F10.950.92
AestheticPass professional readinessTier judgment, weighted κ0.870.88
Same-task design preferencePairs, agreement including ties0.820.86
System-level final faithfulness rankingSix systems, Kendall τ-b0.960.93
System-level professional readiness rankingSix systems, Kendall τ-b0.920.89
System-level design preference rankingSix systems, Kendall τ-b0.750.79

On this subset, automatic–human agreement is 0.92 Macro-F1 for facts, numbers, and constraints, 0.88 weighted κ for professional-readiness judgments, and 0.86 for matched design preferences. The corresponding ranking agreements are 0.93 for final faithfulness, 0.89 for professional readiness, and 0.79 for design preference. The differences across targets reinforce the value of reporting each one separately.

Revision. The original judge-calibration comparison is reported in Table 7 (Page 21, Lines 1099–1118). The appendix now explains how the independent human validation was conducted and reports the item-, pair-, and system-level results in Table 21 (Pages 29–30, Lines 1559–1576).

W4. Definitions and terminology in the main results table

The definitions of the main evaluation metrics and the relationships among them are unclear. Table 3 reports overall scores, stage-level scores, and fine-grained diagnostic metrics, but Table 6 does not fully define these metrics or clearly establish how the metrics in the two tables are related. For example, the computation of Text rendering and Delivery readiness is not sufficiently explained. In addition, Aesthetics and AestheticPass appear to have different meanings in the text but are presented under a single label in Table 3. The authors should standardize the terminology and clearly specify the definition, evaluation unit, sample set, and aggregation procedure for each metric to improve both interpretability and reproducibility.

Response. We apologise for the confusion caused by the original labels. Thank you for highlighting where Table 3 was difficult to trace to Table 6. We checked the scorecards and aggregation code and reproduce the three definitions here.

Original labelCalculation and unitRevised label
Text renderingMean of valid E2.FactRecall, E2.NumericAcc, E6.FactRecallFinal, and E6.FaithfulnessChecklist leaves per run, followed by a mean over scorable runs. This combines process and artifact evidence.Textual fidelity composite
Delivery readinessCompleteness and BaselinePolish in AestheticPass. Fail, Pass, and Excellent map to 0, 0.6, and 1. We average the valid mapped facet scores per system. GPT-5.6-Sol's 0.527 is reproducible from the raw judgments.DeliveryReadiness rubric group
Aesthetics / AestheticPassThe original line reports composite Aesthetics, the mean of valid E4 leaves, E6.AestheticPass, and applicable E6 visual leaves, first by run and then by system. Pure AestheticPass aggregates judge scores within facets, facets within groups, and then gives valid groups equal weight.Aesthetics (composite) and AestheticPass as separate rows

For Claude, composite Aesthetics is 0.690 and pure AestheticPass is 0.606 over 89 scored files. For GPT-5.6-Sol, the figures are 0.661 and 0.601 over 96 scored files. DeliveryReadiness measures the state of the design under the rubric. Formal delivery and renderability have their own indicators. Q2 gives the full mapping.

Revision. Table 3 now reports composite Aesthetics, AestheticPass, and Taste Elo separately and uses the revised textual-fidelity label (Page 8, Lines 409–415). Table 14 also reports the number of valid AestheticPass scores per system (Page 27, Lines 1404–1412). The metric appendix specifies the scoring leaves and averaging units for these results (Page 30, Lines 1600–1608) and the definition of delivery readiness (Page 31, Lines 1622–1623).

Reviewer_Rh1tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorkCraft-Arena evaluates agents that turn heterogeneous workspaces into professional artifacts. It provides 100 tasks across ten roles and six formats, a shared creation/revision harness, and separate assessments of faithfulness, professional readiness, and design preference. Six-model experiments analyze delivery failures, evidence preservation, and revision outcomes. The main contribution is the benchmark and its structured evaluation instrument.

Critical strengths
  1. Separating professional readiness from relative design preference is useful: the protocols change the leading system and expose differences hidden by a single aesthetic score. Human calibration also examines disagreement and order consistency rather than assuming a perfectly reliable reference.

  2. The benchmark connects final deliverables to retained creation and revision records across six formats. The dependency-aware checklist and frozen evaluation instrument support traceable, reproducible comparisons.

Critical weaknesses
  1. Delivery-adjusted faithfulness conflicts with its stated definition. Appendix G.2 says a task with no artifact scores zero, but Table 10 gives Kimi 42/100 deliveries, delivered faithfulness 0.399, and all-task faithfulness 0.312. Averaging with zero for non-deliveries gives about 0.168. MiniMax similarly gives 0.416 rather than 0.77 x 0.459 = 0.353. The aggregation must be reconciled before interpreting the delivery penalties.

  2. The independence interpretation is not established. A pooled correlation of 0.050 with p = 0.25 does not demonstrate independence; Table 11 reports positive within-format correlations, reaching 0.390 for PPTX. A frontier combining different tasks does not establish a within-task trade-off. Analysis controlling for task and format, or narrower conclusions, is needed.

Questions for authors
  1. Do failed deliveries retain process-stage credit in faithfulness? Please provide the exact aggregation and reconcile Tables 3/10 with Appendix G.2.
  2. Does the weak association persist within tasks and formats, with uncertainty calculated over tasks rather than pooled artifacts?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Rh1t ( Part 1/3 )

Thank you for recognising the value of reporting professional readiness and design preference separately. Your recalculation and questions about statistical interpretation helped us clarify both the scoring and the conclusions.

W1. Zero scores for non-delivery and reconciliation of Tables 3 and 10

Delivery-adjusted faithfulness conflicts with its stated definition. Appendix G.2 says a task with no artifact scores zero, but Table 10 gives Kimi 42/100 deliveries, delivered faithfulness 0.399, and all-task faithfulness 0.312. Averaging with zero for non-deliveries gives about 0.168. MiniMax similarly gives 0.416 rather than 0.77 x 0.459 = 0.353. The aggregation must be reconciled before interpreting the delivery penalties.

Response. We apologise for the confusion, and thank you for checking Table 10 so carefully. The product relationship you point out follows from the original wording of Appendix G.2. The text had conflated the E6 zero rule with the aggregation of all-task composite Faithfulness. We rechecked the task-level records. When a run has no artifact, E6.RenderSuccess is zero and gates the E6 stage to zero. Its measurable process leaves can still contribute to composite Faithfulness. The original “delivered” column also included scorable snapshots.

Retaining process credit lets us diagnose partial progress. For example, an agent may extract the right facts before failing to write them into a formal final. Let pAp_A denote the share of runs with a scored file, and FˉA\bar F_A and Fˉ¬A\bar F_{\neg A} the task-weighted composite means with and without a file. Then

Fˉall=pAFˉA+(1−pA)Fˉ¬A.\bar F_{\mathrm{all}}=p_A\bar F_A+(1-p_A)\bar F_{\neg A}.

For Kimi, 0.42×0.398505+0.58×0.249953≈0.3123450.42\times0.398505+0.58\times0.249953\approx0.312345. The same calculation gives 0.416283 for MiniMax. With the zero-for-no-artifact rule described in the original text and displayed rounding, the results would be approximately 0.168 and 0.353, as you observed. Kimi's displayed gap of 0.086 is the difference between two composite means, 0.398505−0.312345≈0.0861600.398505-0.312345\approx0.086160.

SystemFormal finalsDraft onlyNo artifactComposite with fileComposite without fileAll-task composite
GPT-5.6-Sol96400.673124N/A0.673124
Claude Opus 593340.6864160.3271750.672046
Qwen 3.7 Plus97300.558326N/A0.558326
MiniMax M34631230.4588470.2737870.416283
Kimi K32418580.3985050.2499530.312345
Gemini 3.1 Pro99010.3941280.0000000.390187

Your comment helped us separate the two questions. The archived composite can now be reconstructed from provenance and task denominators. The strict-final analysis in Q1 quantifies the user's outcome when non-delivery contributes zero.

Revision. We clarified the distinction between formally delivered finals, draft snapshots, and runs without an artifact in the main results and Table 3 (Page 8, Lines 388–415). The appendix now gives the all-task aggregation formula and the provenance breakdown in Table 10 (Pages 24–25, Lines 1294–1309), reports strict-final rates and quality in Table 11 (Page 25, Lines 1311–1327), and clarifies the denominator used in the delivery figure (Page 25, Lines 1342–1344).

W2. Pooled correlations, format differences, and the cross-task frontier

The independence interpretation is not established. A pooled correlation of 0.050 with p = 0.25 does not demonstrate independence; Table 11 reports positive within-format correlations, reaching 0.390 for PPTX. A frontier combining different tasks does not establish a within-task trade-off. Analysis controlling for task and format, or narrower conclusions, is needed.

Response. Thank you for pointing out the limitations of our interpretation. We agree that pooled Spearman ρ=0.050\rho=0.050 across 514 records and the nominal p=0.25p=0.25 do not establish statistical independence. The pooled Pareto front describes the observed score distribution across tasks and does not establish a within-task trade-off.

Following your suggestion, we controlled for task and estimated uncertainty at the task level. The pooled estimate is 0.050, and the within-task estimate is 0.294 [0.206, 0.379]. Restricting the analysis to formal finals gives 0.297 [0.201, 0.389]. The similar results show that the positive within-task association also appears after excluding snapshots. The common-final set of all six systems contains 12 tasks and gives 0.411, with a wider interval. PPTX has an estimate of 0.551 [0.347, 0.700], compared with 0.040 [-0.144, 0.248] for XLSX. These describe the evaluated format groups, rather than a formal test of differences between formats.

AuthorsOpenReview ↗

Response to Reviewer Rh1t ( Part 2/3 )

We continue to report both dimensions because the criteria provide different feedback. Faithfulness checks identify verifiable factual and structural issues. Aesthetic criteria address clipping, readability, and layout. This distinction remains useful for models and agent harnesses even where the scores move together.

Revision. We revised the Abstract (Page 1, Lines 023–025) and Introduction (Page 3, Lines 111–114) to describe the observed association. In the main results, we clarified the descriptive scope of the cross-task frontier in Figure 4 and the accompanying discussion (Page 9, Lines 442–453). The appendix explains the task-centred analysis and reports the pooled, within-task, format-specific, and formal-final estimates in Table 13 (Page 26, Lines 1376–1393).

Q1. Process credit and exact aggregation

Do failed deliveries retain process-stage credit in faithfulness? Please provide the exact aggregation and reconcile Tables 3/10 with Appendix G.2.

Response. We are sorry the original text was unclear. A run without a formal final can retain measurable process credit. Composite Faithfulness averages the eligible E1, E2, E3, E5, and selected E6 leaves. Diagnostic and NA entries are excluded, and measured zeros are retained. For valid leaves LF(t)\mathcal L_F(t),

Ft=∑ℓ∈LF(t)xtℓ∣LF(t)∣,Fˉall=1100∑tFt.F_t=\frac{\sum_{\ell\in\mathcal L_F(t)}x_{t\ell}}{|\mathcal L_F(t)|},\qquad \bar F_{\mathrm{all}}=\frac{1}{100}\sum_t F_t.

One Kimi run without an artifact had six eligible values: 0.7867 for constraint matching, 0.5652 for process fact recall, 0.6429 for calculation accuracy, 0.4000 for fact grouping, and zero for structure coverage and render success. Its composite is 0.3991, while E6 is zero and Aesthetics is NA. W1 gives the system-level reconstruction.

For each renderable formal final, QtQ_t averages the available E6 hard-constraint, artifact fact-recall, and faithfulness-DAG scores. The conditional mean uses scorable finals. The all-task mean uses all 100 assignments, with absent or unrenderable finals contributing zero.

SystemFormal finals / 100Scorable finalsConditional final faithfulnessAll-task mean, zero for absent or unrenderable final
GPT-5.6-Sol96960.7290.700
Claude Opus 593930.7500.698
Qwen 3.7 Plus97970.6470.628
MiniMax M346460.5260.242
Kimi K324230.5570.128
Gemini 3.1 Pro99990.5660.561

Claude's conditional mean is 0.750, compared with 0.729 for GPT-5.6-Sol. Across all assignments, the means are close at 0.698 and 0.700, respectively. Gemini delivers 99% of finals and its end-to-end mean falls between Qwen and MiniMax. These results illustrate how the probability of receiving a final and the quality of finals affect the overall ordering. For DeliveryAwareElo, using strict final provenance also preserves the six-system ordering.

Revision. We clarified the distinction between stage aggregation and composite scoring (Page 7, Lines 334–345), and the metric appendix now gives the leaf-level formula, a worked no-artifact example, and the relevant denominators (Page 30, Lines 1581–1599). The delivery analysis shows how the all-task results are reconstructed from the provenance groups (Pages 24–25, Lines 1294–1309) and reports strict-final quality with absent or unrenderable finals contributing zero (Page 25, Lines 1311–1327).

AuthorsOpenReview ↗

Response to Reviewer Rh1t ( Part 3/3 )

Q2. Associations within tasks and formats

Does the weak association persist within tasks and formats, with uncertainty calculated over tasks rather than pooled artifacts?

Response. Thank you for recommending task-level uncertainty. We have added the within-task and format-stratified analyses. For runs with both composite scores, we assign midranks within the sample, subtract the task mean from each rank, and correlate the two sets of residuals. Each task in the within-task analysis has at least two comparable systems. We resample tasks 5,000 times, retain all outputs for each sampled task, and recompute both statistics for the 95% intervals.

SampleTasksScored filesPooled Spearman ρ [95% CI]Within-task rank correlation [95% CI]
All comparable scored files1005140.050 [-0.044, 0.147]0.294 [0.206, 0.379]
DOCX12630.216 [-0.096, 0.516]0.386 [0.132, 0.571]
XLSX201000.085 [-0.063, 0.244]0.040 [-0.144, 0.248]
HTML211100.258 [0.094, 0.424]0.309 [0.120, 0.486]
Markdown14760.229 [0.072, 0.352]0.219 [0.036, 0.351]
Poster16820.180 [0.019, 0.364]0.294 [0.125, 0.470]
PPTX17830.390 [0.211, 0.563]0.551 [0.347, 0.700]
Formal finals only994550.080 [-0.016, 0.181]0.297 [0.201, 0.389]
All six systems delivered finals12720.183 [-0.127, 0.493]0.411 [0.106, 0.634]

The weak pooled association does not persist after accounting for task. The within-task estimate is positive overall and similar for formal finals. It is larger for PPTX, while the XLSX estimate is near zero with an interval that includes zero.

Revision. We clarified the scope of the archived format summaries (Pages 25–26, Lines 1348–1360). We also added the task-centred rank procedure and task-cluster bootstrap in the appendix, where Table 13 reports both correlation estimates, task and file counts, confidence intervals, and the final-only sensitivity checks (Page 26, Lines 1376–1393).

Thank you again for your detailed review. We hope that the explicit reconstruction and the task-level analysis address your concerns. The corresponding changes are highlighted in blue and will be retained in the final version.

Dear Reviewer,

As the discussion period draws to a close, we wanted to follow up on the rebuttal we posted earlier. We would appreciate any feedback you may have while there is still time for us to address remaining concerns. We sincerely thank the reviewer and the Area Chair for your time and effort throughout the review process.

Sincerely,

Authors of Manuscript 14104

Reviewer_JXpGOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WorkCraft-Arena, a benchmark of 100 tasks across ten professional roles and 6 artifact formats, evaluating agents that transform heterogeneous workspace materials into deliverable work products. It provides a unified agent harness that records intermediate creation and revision stages, together with a dependency-aware faithfulness checklist and two aesthetic protocols: AestheticPass for professional readiness and AestheticElo for relative design preference. Experiments across serval models examine content fidelity, visual quality, delivery reliability, and self-revision.

Critical strengths
  1. Practical task coverage. The benchmark includes diverse professional scenarios and 2,041 input files spanning 19 file types, capturing the challenge of synthesizing heterogeneous materials into usable artifacts.
  2. Detailed evaluation infrastructure. Task-specific annotations, source provenance, intermediate records, and rendered outputs support more granular analysis than a single task-completion score.
Critical weaknesses
  1. Insufficiently justified motivation. Existing work already evaluates material-grounded content and visual design, including PresentBench[1], PPTEval[2], and specialized reward models such as DesignSense[3] and PosterReward[4]. The paper does not clearly establish which limitations require its proposed evaluation framework beyond broader task and format coverage.
  2. Unclear metric validity. Faithfulness includes layout diversity and image–text balance, while AestheticPass includes hallucination checks, undermining the claimed separation of factual fidelity and aesthetics. Moreover, scoring explicit process records conflates documentation completeness with actual understanding, weakening the proposed failure diagnosis.
  3. Missing evaluator baselines and ablations. Comparing generators and candidate judges does not establish the proposed protocol’s effectiveness. Essential baselines include direct VLM judging, content extraction plus source verification, and flat checklists using the same criteria. Comparisons with existing evaluators on applicable formats, alongside DAG and dual-aesthetic ablations, are needed to demonstrate improved agreement with independent human judgments.

Reference

[1] PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation

[2] PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

[3] DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation

[4] PosterReward: Unlocking Accurate Evaluation for High-Quality Graphic Design Generation

Questions for authors

I rewrite the weakness above into the questions below:

  1. What specific limitations of existing evaluators, such as PresentBench, PPTEval, DesignSense, and PosterReward, does your framework address beyond broader task and format coverage?
  2. How do you justify including layout diversity and image–text balance in Faithfulness, and hallucination checks in AestheticPass? How does process evaluation distinguish incomplete documentation from inadequate understanding?
  3. Can you provide comparisons against content extraction plus source verification, existing design evaluators, and flat checklists under matched human judgments? What do ablations show about the added value of the DAG and dual-aesthetic protocols?
7 comments
AuthorsOpenReview ↗

Response to Reviewer JXpG ( Part 1/5 )

Thank you for reviewing our work and for recognising the coverage of the task materials and the evaluation infrastructure. Your comments helped us clarify the motivation, evaluation design, and experimental evidence. We have addressed each point below and marked the corresponding changes in the revised manuscript.

W1. Motivation and relationship to prior evaluators

Insufficiently justified motivation. Existing work already evaluates material-grounded content and visual design, including PresentBench[1], PPTEval[2], and specialized reward models such as DesignSense[3] and PosterReward[4]. The paper does not clearly establish which limitations require its proposed evaluation framework beyond broader task and format coverage.

Response. Thank you for placing WorkCraft-Arena alongside these relevant evaluations. We clarify the motivation and contribution in three respects.

(1) Comparison with prior work. PresentBench provides fine-grained, task-specific checklists for slides generated from background materials. PPT-Eval uses task criteria, partial credit, and edit constraints for PowerPoint creation and editing. DesignSense and PosterReward evaluate layout preference and poster design quality, respectively. These studies directly informed our design. We initially emphasised task and format coverage and could have explained more clearly what the evaluation reveals about long-horizon creation in complex workspaces. PresentBench · PPT-Eval · DesignSense · PosterReward

We focus on the complete task from heterogeneous workspace materials to a deliverable design file. Agents identify relevant evidence in scattered files, select and organise it for a brief, and use appropriate tools to create, inspect, and revise the result. Our 100 tasks include 2,041 input files across 19 input types and six output formats. Evidence gathering, design decisions, and file delivery depend on each other. An agent can extract the right facts yet omit them when assembling the artifact, or leave a draft without delivering a final.

WorkCraft-Arena addresses two requirements within a single workflow. Professional readiness and visual preference call for different criteria. Clipping, complete legends, and practical usability can be assessed on an individual artifact. Pacing, colour, and visual storytelling can be compared across designs for the same brief. Scores should be traceable to source materials, process records, and artifact versions. We retain facts and their source locations, constraints, structured records, self-check snapshots, and the actual file used for scoring. This makes it possible to investigate why a run received a low score.

(2) Evaluating models' aesthetic capabilities. The two aesthetic protocols select general criteria by artifact format, then derive task-specific criteria from the intended use and annotations. A slide deck, for example, can be checked for readability and hierarchy while alternative visual approaches are compared on the same task. The faithfulness DAG uses the task annotations to derive object, attribute, and relation checks, retaining both raw judgments and dependency-resolved scores. This leads to three research questions.

  • RQ1. How can we evaluate work-artifact aesthetics across formats under a common protocol? AestheticPass assesses professional readiness, and AestheticElo estimates relative preference within a task. Each uses criteria adapted to its task and format.
  • RQ2. What challenges does this evaluation expose for multimodal models? We examine evidence retention, professional presentation, and relative preference. The comparisons show where models produce similar results and where users may see meaningful design differences.
  • RQ3. What capabilities do work-assistant agents need for complex design tasks? We connect process records, rendered feedback, delivery status, and revisions to trace evidence retention and examine where recording, final-file confirmation, and revision mechanisms can improve.

(3) Evidence for the contribution. In the 79 matched final-artifact tasks in Q3, Claude and GPT-5.6-Sol receive similar professional-readiness scores, while the frozen judge more often prefers GPT-5.6-Sol. For delivery, the archived 514 scoreable files include 59 drafts beside 455 formal finals. The strict-final analysis therefore gives a more direct picture of what users receive. These results connect the aesthetic and delivery questions to observed outcomes.

AuthorsOpenReview ↗

Response to Reviewer JXpG ( Part 2/5 )

The process cases make a complementary point. In one HTML task, MiniMax's process record matched 1 of 22 reference facts and its final text matched 21. In a Markdown task, Claude matched 19 of 22 in its process record and 2 in the final text. The scorecard identified 17 process matches that did not carry over to the matches in the extracted text. These automatic comparisons point reviewers to the passages and sources to examine. A final summary score alone would obscure the two different patterns.

Revision. We clarified the motivation for tracing evidence and delivery within the same run (Page 2, Lines 087–091) and stated our three research questions (Page 2, Lines 101–105). We expanded the discussion of PresentBench and PPT-Eval (Page 3, Lines 133–138) and of DesignSense, PosterReward, and HCB (Page 3, Lines 158–161). The appendix now includes concrete process-trace cases (Page 27, Lines 1440–1446) and the matched comparison of professional readiness and design preference (Page 26, Lines 1397–1403).

W2. Metric attribution and interpretation of process scores

Unclear metric validity. Faithfulness includes layout diversity and image–text balance, while AestheticPass includes hallucination checks, undermining the claimed separation of factual fidelity and aesthetics. Moreover, scoring explicit process records conflates documentation completeness with actual understanding, weakening the proposed failure diagnosis.

Response. We apologise for the confusion and appreciate your pointing out where the metric names and attribution were unclear. We checked the scorecards, criteria, and implementation to clarify the exact definitions.

Updated faithfulness scores. LayoutEntropy and ImageTextBalance were intended as information-organisation heuristics. The former measures variety in planned layouts, and the latter measures the number of figures per thousand words. Both entered the original composite Faithfulness score. As you note, a consistently formatted report or a text-only deliverable can satisfy the brief. In the revised aggregation, we retain both values as diagnostics and exclude them from composite Faithfulness. Explicitly required sections, charts, and structures are still checked against the task annotations.

We recomputed all 600 runs. Whether we remove either term or both, GPT-5.6-Sol and Claude remain the two leading systems and ranks 3–6 are unchanged. With both terms removed, Claude scores 0.672 and GPT-5.6-Sol scores 0.667. Their point-estimate order reverses, but their scores remain close.

Faithfulness settingTask × systemGPT-5.6ClaudeQwenMiniMaxKimiGeminiτ-b vs. original
Original6000.6730.6720.5580.4160.3120.3901.000
Without LayoutEntropy6000.6660.6670.5510.4040.3140.3950.867
Without ImageTextBalance6000.6750.6780.5750.4240.3160.3900.867
Without both6000.6670.6720.5680.4110.3180.3950.867

The hallucination checks in the aesthetic rubric. We are sorry that the appendix carried over the older combined description. The evaluated rubric used NoBrokenAssets and NoFabricatedContent. The first checks broken images, placeholders, and filler text. The second is restricted to visible internal inconsistencies, such as chart marks that contradict their labels. These are presentation checks. The source-grounded factual checks are handled separately in Faithfulness.

Process records and understanding. We persist intermediate outputs at key steps and organise them in a structured form so they can be compared with task annotations. The evaluation checks the evidence in those records and tracks missing channels separately from available records that fail to match a required fact. The two examples in Q2 show how this distinction helps locate different problems, without inferring a model's internal understanding from its documentation.

Revision. We clarified how stage gates and composite scores are applied (Page 7, Lines 334–345) and specified the visual integrity checks in the AestheticPass rubric (Page 20, Lines 1054–1056). The appendix now reports the E5 layout-metric sensitivity analysis (Page 28, Lines 1490–1505) and illustrates how recorded process evidence can differ from the evidence retained in a final artifact (Page 27, Lines 1440–1446).

W3. Evaluator baselines and ablations

Missing evaluator baselines and ablations. Comparing generators and candidate judges does not establish the proposed protocol’s effectiveness. Essential baselines include direct VLM judging, content extraction plus source verification, and flat checklists using the same criteria. Comparisons with existing evaluators on applicable formats, alongside DAG and dual-aesthetic ablations, are needed to demonstrate improved agreement with independent human judgments.

AuthorsOpenReview ↗

Response to Reviewer JXpG ( Part 3/5 )

Response. Thank you for suggesting comparisons on the same artifacts, criteria, and human reference. The original study prioritised checking the evaluation protocol's consistency and possible model preferences. We compared candidate judges against professional designers and reported separate DAG judgments for existence, attributes, and relations. We agree that the matched baseline comparisons and ablations provide additional evidence about the evaluation protocol.

For ease of comparison, we reproduce the nine-judge results from the original appendix. The three designers jointly judged 217 professional-tier items and 340 preference comparisons, with 64 and 132 unanimous decisions, respectively.

Candidate judgeUnanimous tiersLeave-one-out tier agreementUnanimous preferencesAll-pair preference agreementOrder-swap consistency
Gemini-3-Pro0.8550.6630.6520.5710.851
GPT-5.50.6940.5450.5760.5000.764
Claude-Opus-4.80.6890.5660.5680.5300.764
Gemini-3.1-Flash0.6830.5500.5980.5350.675
Gemini-3.5-Flash0.6770.5240.4320.4800.482
Claude-Sonnet-4.60.6510.5270.5610.5100.815
Claude-Sonnet-50.5000.4540.5300.5050.758
GPT-5.40.4590.3840.5150.4720.721
GPT-5.4-Mini0.2000.2980.3110.4170.367

The original appendix also reports 0.501 mean exact tier agreement between designers, 0.538 agreement on preference comparisons, and 0.841 order-swap consistency. The results support our choice of the aesthetic judge while showing why agreement on unanimous items and agreement across all pairs should both be reported.

Fixed-judge Flat versus DAG. The evaluator first obtains node judgments and then applies dependencies. Flat averages the raw binary judgments, and DAG averages the resolved values. We hold the judge and criteria fixed and restrict both averages to the same nodes with valid raw and resolved values. We compute the means and differences by scored file and estimate uncertainty with 5,000 task-cluster bootstrap draws.

SampleTasks / filesCommon judged nodesFlat meanDAG meanNodes changed 1 → 0DAG − Flat [95% CI]
Finals and snapshots100 / 51451,1790.6070.5672,265-0.040 [-0.051, -0.030]
Formal finals only99 / 45545,1960.6140.5722,118-0.042 [-0.054, -0.031]

Of 51,179 commonly judged nodes, dependency resolution changes 2,265 raw passes to zero, including 1,925 attribute nodes. In one DOCX task, the judge found material about the Sprint 47 migration and CI but did not find the required colleague-briefs section. Flat credits the raw attribute match. DAG retains that judgment and records zero for the dependent criterion. The record shows precisely how recognising content and satisfying the requested structure contribute to the score.

Comparison against independent human judgments. We compared direct VLM judging, content extraction followed by source verification, and the DAG on a common human reference. The direct comparison uses the same VLM, and factual results are compared on the shared factual subset.

MethodFact Macro-F1Human agreementCoverage
Direct VLM0.720.630.85
Content extraction + source verification0.790.720.87
DAG0.810.880.88

The DAG has the highest observed point estimates among these three methods, with a fact Macro-F1 of 0.81 and human agreement of 0.88. This comparison addresses agreement with human checks. The fixed-judge ablation above addresses the separate effect of applying the dependency rules.

Revision. The revised manuscript summarises the additional analyses in the main results (Page 8, Lines 393–399) and clarifies how raw judgments and dependency-resolved scores are retained (Page 6, Lines 303–305), including in the DAG diagnostics (Page 27, Lines 1418–1423). The appendix reports the fixed-judge Flat versus DAG ablation and the DOCX case (Pages 28–29, Lines 1510–1525), the matched evaluator comparison (Page 29, Lines 1527–1539), and the clarified aesthetic rubric (Page 20, Lines 1054–1056). The original nine-judge comparison is available in the calibration appendix (Page 21, Lines 1099–1118).

Comments on the paper

2 comments

Dear Reviewers,

As the discussion period draws to a close, we wanted to follow up on the rebuttal we posted earlier. We would appreciate any feedback you may have while there is still time for us to address remaining concerns. If you have any further questions or would like clarification on any point, please let us know. We sincerely thank the reviewers and the Area Chair for your time and effort throughout the review process.

Sincerely,

Authors of Manuscript 14104

AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

Dear Area Chair and Reviewers,

We sincerely thank you for the time and effort devoted to our submission. As the discussion period draws to a close, we would like to summarize the main contributions, our responses and revisions to the reviews, and the discussions made to the manuscript.

1. Core contributions

WorkCraft-Arena evaluates how multimodal agents turn heterogeneous workspace materials into deliverable professional artifacts. It comprises 100 tasks across 10 professional roles and six artifact formats. Its evaluation connects source annotations, observable creation records, scored artifacts, delivery status, ultimately challenging the aesthetic quotient of large models.

The protocol evaluates faithfulness while focusing on two complementary aesthetic signals, AestheticPass for professional readiness and AestheticElo for relative design preference. We explore three key scientific questions, hoping to investigate a unified evaluation system for the aesthetic quotient of models, and what new capabilities next-generation models and agent harnesses need.

2. Summary of responses and revisions

We have made a concerted effort to address the reviewers' substantive concerns through detailed clarifications, additional experiments, and manuscript revisions. These efforts have received positive recognition from Reviewer JXpG, who raised their score from 2 to 3 and explicitly acknowledged that our response had addressed most of their questions and concerns.

ReviewerStatusConcernsClarifications and manuscript revisions
JXpGScore raised from 2 to 3.1. Contribution beyond existing evaluators; 2. metric validity; 3. interpretation of process records, and 4. evaluator baselines and ablations.1. We clarified the relationship to prior evaluators and added cases showing how recorded process evidence and final-content coverage can differ (Page 3, Lines 133–161, and Page 27, Lines 1440–1446). 2. We clarified the aesthetic integrity checks (Page 20, Lines 1054–1056) and recomputed Faithfulness after excluding the two E5 layout heuristics (Page 28, Lines 1490–1505). 3. We added a fixed-judge Flat versus DAG ablation (Pages 28–29, Lines 1510–1525) and comparisons against a common independent human reference. 4. We also added human comparisons of the aesthetic protocols separately and together (Page 29, Lines 1544–1555).
Rh1tScore 2. Awaiting follow-up.1. Reconciliation of delivery and Faithfulness aggregation, and 2. statistical interpretation.1. We rechecked all records and explicitly separated. The revised aggregation formula explains how measurable process contributes to the archived composite and reported scores (Pages 24–25, Lines 1294–1309). 2. We additionally report strict-final quality with absent or unrenderable finals (Page 25, Lines 1311–1327). 3. we added task-centred and format-specific analyses with task (Page 26, Lines 1376–1393). 4. We revised the main discussion to reflect the positive within-task association and the descriptive scope of the pooled frontier.
JczrScore 2. Awaiting follow-up.1. Distinction from HCB and PresentBench, 2. inconsistent statistics, 3. independent human validation, 4. metric definitions.1. We clarified the contribution to prior work (Page 3, Lines 111–161) and corrected the format-level ordering (Pages 24–25, Lines 1294–1327, and Pages 25–26, Lines 1348–1351). 2. We added independent human validation using outputs outside the judge-calibration tasks (Pages 29–30, Lines 1559–1576). 3. We separated composite Aesthetics from AestheticPass and clarified the textual-fidelity label (Page 8, Lines 409–415). 4. We added a complete metric mapping and aggregation rules (Pages 30–31, Lines 1579–1627).5. We corrected the spelling and broken table reference (Page 3, Line 117, and Page 26, Lines 1350–1351).

3. Closing Summary

We have made every effort to respond to each concern through detailed explanations, additional evaluation, and revisions highlighted in blue. Although some reviewers have not yet responded, we believe that most concerns have been addressed, which is reflected in the improvement of reviewers' score from 2 to 3. The remaining discussion has focused on method implementation and additional experimental comparisons, for which we have provided further evidence and clarification.

We believe the review and rebuttal process has substantially improved the manuscript's clarity, reproducibility, and experimental support. We sincerely thank the reviewers and the Area Chair again for your careful assessment and valuable feedback, and respectfully ask that the additional evidence and revisions be considered in the final assessment. Thank you again for your dedication to maintaining the high standards of ICLR.

Sincerely,

The Authors of Submission 14104

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.