- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces OfficeEditBench, a benchmark for evaluating whether agents can complete requested edits and their required downstream updates to existing Office files while preserving protected content and native editability. It comprises 170 tasks involving XLSX, PPTX, and DOCX files and analyzes 510 task–system outcomes from three systems. The evaluation distinguishes file delivery, target completion, preservation, and full contract satisfaction. Its main contributions are an evaluation design that explicitly specifies task-level edit scope and a diagnostic analysis of maintenance failures.
Critical strengths
- The benchmark evaluates whether agents can complete requested Office edits while preserving protected content and native editability, reflecting an important requirement of real-world workflows.
- Strict acceptance and partial scores distinguish target completion from full task success, while concrete cases illustrate failures involving formulas, dependent updates, and retained conditions..
Critical weaknesses
- The authors conduct construction checks using reference outputs and deliberately flawed variants, but do not evaluate how closely automatic judgments on actual agent outputs agree with independent human assessments. In particular, the paper itself reports a case in which an exact approved value is rejected because it differs from a rounded reference. It therefore remains unclear how reliably automatic failures reflect substantive editing errors. Since the main results and failure analysis rely on these judgments, this gap limits confidence in the benchmark’s central measurement instrument.
- The rationale for replacing the original Easy set with Easy v2, the criteria for designating WorkBuddy Hard as exploratory, and the correction and final-record selection procedures for Doubao Hard are not sufficiently explained, making it difficult to trace how the current tasks and results were finalized. In particular, a paper proposing a reusable benchmark should clearly specify the criteria for dividing tasks into Easy, Medium, and Hard and the capabilities each subset is intended to evaluate. However, the current partitioning is explained in terms of design intent, without sufficient justification for why it is appropriate when task scope, language, checks, and scoring weights vary together. If the subsets represent difficulty levels, evidence supporting their ordering is needed; if they represent task types, clear classification criteria are needed. Without this information, it remains unclear how future studies should use each subset for evaluation and interpret subset-level performance.
Questions for authors
- Can the authors provide independent assessments of a sample of actual agent outputs and quantify the impact of the identified evaluator issues on the aggregate results? The sample should include both automatically accepted and rejected outputs.
- Can the authors explain the changes leading to Easy v2 and the designation, correction, and selection procedures for the WorkBuddy and Doubao Hard records? Were evaluator changes applied consistently to all relevant results?
- What specific criteria are used to assign tasks to Easy, Medium, and Hard, and which capabilities is each subset designed to evaluate? Please explain the evidence supporting the appropriateness of this partitioning and its intended use in future research.
2 comments
read the response
I have read the authors' response. My question regarding evaluations of independent human assessment was the most critical question. Unfortunately, the rebuttal does not directly address my concern.
Response to Reviewer ktMC
Thank you for asking how the benchmark judgments, task versions, and tier definitions should be interpreted. These distinctions are central to using OfficeEditBench.
1. What supports the verifier’s judgments, and how should the rounding case be interpreted? The benchmark provides two kinds of evidence. First, construction checks test whether the verifier distinguishes reference outputs from unchanged inputs and from outputs with targeted defects. For Easy v2, it accepts all 50 references and rejects all 50 variants that make the requested edit but also introduce an unrelated change. Second, the check-level records support inspection of actual outputs, including omissions that can be verified directly against the request. OBM-058, for example, updates a rule without consistently applying it to the affected case. OBM-037 illustrates a different phenomenon: the exact approved value disagrees with a rounded reference. We retain the frozen scores and identify this discrepancy explicitly so that readers can distinguish a reference-sensitive verdict from a substantive editing error. The existing evidence does not quantify human–verifier agreement or the aggregate effect of alternative predicates.
2. Why Easy v2, and how were the Hard records finalized? Easy v2 gives the entry tier a clear scope: a localized edit to one existing artifact. The retired Easy collection included cross-file tasks; the active version uses single-file tasks so that Easy tests local target correctness and preservation without requiring cross-file coordination. Table 2 uses Easy v2 for all three agents and does not pool the two inventories.
For Hard, exploratory is the provenance label retained from the original WorkBuddy run. It does not denote a separate scoring rule or a different formal retry allowance. For Doubao, Table 2 reports the designated final snapshot, without pooling earlier snapshots or taking the best score per task. Comparing the immediately preceding snapshot with the final one, all 50 stored artifact check records, semantic and preservation scores, and strict verdicts are unchanged. Interaction records changed for five tasks; the composite changed for two, moving the mean from 43.35 to 43.52. Thus, the final transition did not change the artifact judgments or the strict-pass conclusion. This comparison concerns the final transition, rather than every earlier evaluator revision.
The formal inclusion rule was shared across all three systems: score the first formal attempt not invalidated by a network interruption, retain failures and low scores, and allow retries only for network interruptions. Debugging records were outside the formal scoring round. The reported results use the frozen criteria for the active suite; this does not imply that every earlier development snapshot was replayed.
3. How should future studies use the three tiers? The tiers are defined by the maintenance scope. Easy tests a localized edit within one artifact, including whether unrelated content survives. Medium tests edits that require contextual evidence or propagation across dependent locations, sometimes across files. Hard tests coordinated maintenance of a workbook, presentation, and document under a shared task contract. These categories let future studies ask where a system breaks down: on local edits, on propagation, or on coordinating a full package. The observed results are consistent with this progression in task demands. Because language, scope, and scoring rules also vary, the tiers should be used as task families rather than as a calibrated numerical difficulty scale.