acceptodds
Under review as a conference paper at ICLR 2027

The Last Mile Is the File: OfficeEditBench for Preservation-Aware Office Editing

Abstract

A small Office edit creates two obligations: propagate every required update and leave protected state untouched. Updating too little leaves dependencies inconsistent; updating too much changes content the user did not authorize. We introduce OfficeEditBench, a 170-task benchmark for change-scoped maintenance of spreadsheets, presentations, and documents. Task contracts specify required updates, protected state, native structures, and applicable interaction requirements. Across 510 archived task–system outcomes from WorkBuddy, Doubao, and Codex, we distinguish file delivery, target completion, and verifier-defined acceptance. Hard package-valid delivery ranges from 92% to 100%, yet no selected output satisfies the complete contract. Case analysis highlights why local correctness is insufficient: an updated value can lose its generating formula, a revised rule can fail to reach related conclusions, and a new deadline can omit a retained prerequisite. These mechanisms connect artifact-level checks to the continued maintainability of Office files. We analyze maintenance failures while distinguishing frozen automatic verdicts from human acceptability. OfficeEditBench provides a testbed for completing required changes while preserving the logic and scope of existing work.

Primary area:
datasets and benchmarks
Keywords:
AI Agents, Office Document Editing, Benchmarking, Preservation-Aware Evaluation, Change-Scoped Maintenance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ktMCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces OfficeEditBench, a benchmark for evaluating whether agents can complete requested edits and their required downstream updates to existing Office files while preserving protected content and native editability. It comprises 170 tasks involving XLSX, PPTX, and DOCX files and analyzes 510 task–system outcomes from three systems. The evaluation distinguishes file delivery, target completion, preservation, and full contract satisfaction. Its main contributions are an evaluation design that explicitly specifies task-level edit scope and a diagnostic analysis of maintenance failures.

Critical strengths
  1. The benchmark evaluates whether agents can complete requested Office edits while preserving protected content and native editability, reflecting an important requirement of real-world workflows.
  2. Strict acceptance and partial scores distinguish target completion from full task success, while concrete cases illustrate failures involving formulas, dependent updates, and retained conditions..
Critical weaknesses
  1. The authors conduct construction checks using reference outputs and deliberately flawed variants, but do not evaluate how closely automatic judgments on actual agent outputs agree with independent human assessments. In particular, the paper itself reports a case in which an exact approved value is rejected because it differs from a rounded reference. It therefore remains unclear how reliably automatic failures reflect substantive editing errors. Since the main results and failure analysis rely on these judgments, this gap limits confidence in the benchmark’s central measurement instrument.
  2. The rationale for replacing the original Easy set with Easy v2, the criteria for designating WorkBuddy Hard as exploratory, and the correction and final-record selection procedures for Doubao Hard are not sufficiently explained, making it difficult to trace how the current tasks and results were finalized. In particular, a paper proposing a reusable benchmark should clearly specify the criteria for dividing tasks into Easy, Medium, and Hard and the capabilities each subset is intended to evaluate. However, the current partitioning is explained in terms of design intent, without sufficient justification for why it is appropriate when task scope, language, checks, and scoring weights vary together. If the subsets represent difficulty levels, evidence supporting their ordering is needed; if they represent task types, clear classification criteria are needed. Without this information, it remains unclear how future studies should use each subset for evaluation and interpret subset-level performance.
Questions for authors
  1. Can the authors provide independent assessments of a sample of actual agent outputs and quantify the impact of the identified evaluator issues on the aggregate results? The sample should include both automatically accepted and rejected outputs.
  2. Can the authors explain the changes leading to Easy v2 and the designation, correction, and selection procedures for the WorkBuddy and Doubao Hard records? Were evaluator changes applied consistently to all relevant results?
  3. What specific criteria are used to assign tasks to Easy, Medium, and Hard, and which capabilities is each subset designed to evaluate? Please explain the evidence supporting the appropriateness of this partitioning and its intended use in future research.
2 comments
AuthorsOpenReview ↗

Response to Reviewer ktMC

Thank you for asking how the benchmark judgments, task versions, and tier definitions should be interpreted. These distinctions are central to using OfficeEditBench.

1. What supports the verifier’s judgments, and how should the rounding case be interpreted? The benchmark provides two kinds of evidence. First, construction checks test whether the verifier distinguishes reference outputs from unchanged inputs and from outputs with targeted defects. For Easy v2, it accepts all 50 references and rejects all 50 variants that make the requested edit but also introduce an unrelated change. Second, the check-level records support inspection of actual outputs, including omissions that can be verified directly against the request. OBM-058, for example, updates a rule without consistently applying it to the affected case. OBM-037 illustrates a different phenomenon: the exact approved value disagrees with a rounded reference. We retain the frozen scores and identify this discrepancy explicitly so that readers can distinguish a reference-sensitive verdict from a substantive editing error. The existing evidence does not quantify human–verifier agreement or the aggregate effect of alternative predicates.

2. Why Easy v2, and how were the Hard records finalized? Easy v2 gives the entry tier a clear scope: a localized edit to one existing artifact. The retired Easy collection included cross-file tasks; the active version uses single-file tasks so that Easy tests local target correctness and preservation without requiring cross-file coordination. Table 2 uses Easy v2 for all three agents and does not pool the two inventories.

For Hard, exploratory is the provenance label retained from the original WorkBuddy run. It does not denote a separate scoring rule or a different formal retry allowance. For Doubao, Table 2 reports the designated final snapshot, without pooling earlier snapshots or taking the best score per task. Comparing the immediately preceding snapshot with the final one, all 50 stored artifact check records, semantic and preservation scores, and strict verdicts are unchanged. Interaction records changed for five tasks; the composite changed for two, moving the mean from 43.35 to 43.52. Thus, the final transition did not change the artifact judgments or the strict-pass conclusion. This comparison concerns the final transition, rather than every earlier evaluator revision.

The formal inclusion rule was shared across all three systems: score the first formal attempt not invalidated by a network interruption, retain failures and low scores, and allow retries only for network interruptions. Debugging records were outside the formal scoring round. The reported results use the frozen criteria for the active suite; this does not imply that every earlier development snapshot was replayed.

3. How should future studies use the three tiers? The tiers are defined by the maintenance scope. Easy tests a localized edit within one artifact, including whether unrelated content survives. Medium tests edits that require contextual evidence or propagation across dependent locations, sometimes across files. Hard tests coordinated maintenance of a workbook, presentation, and document under a shared task contract. These categories let future studies ask where a system breaks down: on local edits, on propagation, or on coordinating a full package. The observed results are consistent with this progression in task demands. Because language, scope, and scoring rules also vary, the tiers should be used as task families rather than as a calibrated numerical difficulty scale.

Reviewer_ktMCOpenReview ↗

read the response

I have read the authors' response. My question regarding evaluations of independent human assessment was the most critical question. Unfortunately, the rebuttal does not directly address my concern.

Reviewer_SmFuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark of 170 tasks evaluating whether AI agents can edit Excel, PowerPoint, and Word files while preserving unrelated content and structure. Experiments with WorkBuddy, Doubao, and Codex show that none fully satisfy the requirements of the Hard tasks. The results highlight limitations in reliable Office editing, although the evaluation relies on automatic verifiers without independent human validation.

Critical strengths
  1. Important and well-defined evaluation problem. This work identifies a practical weakness of AI agents, such as editing an existing artifact requires not only producing the correct new content but also preserving unaffected information and maintaining dependencies.
  2. Interpretable evaluation and good failure analysis; such as strict acceptance from partial completion, allowing readers to distinguish missing files, and incorrect requested edits.
Critical weaknesses
  1. Insufficient benchmark validation. The evaluation relies on automatic verifiers without independent human validation. Appendix A.2 identifies cases where correct outputs were rejected due to rounding or exact wording requirements, raising concerns about the reliability of strict success rates.
  2. Limited experimental control and generalization. The systems were evaluated under different retry and selection policies, limiting fair comparison. Additionally, the 50 Hard tasks are derived from only four scenarios, raising concerns about generalizability.
  3. Limited novelty. Preservation-aware evaluation has already been explored in PPT-Eval, DeckEdit-Bench, and OmegaUse-OfficeVal. The contribution primarily lies in combining existing evaluation concepts into a new benchmark.
Questions for authors
  1. Could the authors provide an independent human evaluation of a representative subset of accepted and rejected outputs, including estimated agreement with the automatic verifier? In particular, how often do valid alternative edits fail exact-value, wording, or structural checks?

  2. How do retries, execution budgets, and selection policies differ across WorkBuddy, Doubao, and Codex? Can the authors identify a comparable subset or otherwise demonstrate that the main conclusions are robust to these differences?

  3. Given that Hard tasks reuse four source scenarios, what evidence supports generalization to unseen workflows? Could the authors report results by scenario family and characterize the diversity of required edit operations and dependencies?

2 comments
AuthorsOpenReview ↗

Response to Reviewer SmFu

Thank you for highlighting the value of separating file delivery, target completion, and preservation. We address your questions in turn.

1. What do the verifier results tell us about actual editing quality? OfficeEditBench makes the reason for a failed verdict inspectable. For example, in OBM-058, the output updated the delay rule to include flights delayed by 15 minutes or more, but still labeled a flight at exactly 15 minutes as on time. The task instruction and the edited artifact suffice to establish the inconsistency. By comparison, OBM-037 contains the approved exact value and was rejected by a check against a rounded reference. The check-level records let us distinguish these two cases rather than treating every automated rejection as the same kind of failure. The submitted study does not estimate independent human agreement or the frequency of valid alternatives rejected by the verifier.

2. Were retries, budgets, and output selection comparable? WorkBuddy, Doubao, and Codex followed the same first-formal-attempt protocol on the same 170 tasks. Each agent had one formal attempt per task, with a retry allowed only if a network interruption invalidated that attempt. Non-delivery, task failure, and low scores were retained. No output was selected on the basis of its score. Gold outputs and the verifier were withheld from all agents. The additional attempts discussed in the appendix include pre-evaluation debugging and network recovery; they were not additional eligible submissions. The appendix did not distinguish these records clearly enough from the formal scoring round. The common attempt budget should be distinguished from realized time or token consumption, which the archive does not measure consistently.

3. What varies across the four Hard scenario families? The families cover public-company reporting, online retail, capital projects and procurement, and environmental compliance. Within these workspaces, tasks vary which source evidence authorizes the edit, which dependent values or statements must change, and which records or conditions must remain intact. These are meaningful variations: updating a value in a workbook and its native slide chart requires a different dependency check from preserving a condition while revising a document or review record. The 50 tasks provide repeated tests of such operations within four workspaces; the current evidence does not estimate performance on unseen workspaces.

4. What is distinctive relative to prior preservation-aware benchmarks? Our contribution is a task contract for change-scoped maintenance across existing Office artifacts. It specifies both where a change must propagate and where it must stop, including native structures that need to remain editable. In OBM-032, the approved value must agree across the workbook, the native PowerPoint chart, and the slide conclusion, while the raw sales records and unrelated entries are protected. Checking any one output in isolation would miss part of that contract. This complements prior work on preservation-aware editing by making cross-artifact dependencies an explicit unit of evaluation.

Thank you for the clarification. The examples help distinguish genuine editing failures from verifier errors, but the absence of independent human validation and an estimate of false-rejection frequency leaves my main validation concern unresolved. The clarification of a common first-formal-attempt protocol addresses my concern about retry and selection policies, although differences in realized execution budgets remain unquantified. The explanation of variation within the four scenario families is useful, but does not establish generalization to unseen workflows. The remaining validation and generalization concerns lead me to maintain my score.

Reviewer_dmd8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents OfficeEditBench, a benchmark of 170 tasks for evaluating whether agents can correctly edit Excel, PowerPoint, and Word artifacts while propagating required changes and preserving non-target content and structure. The paper evaluates three agent systems across Easy, Medium, and Hard tasks, with increasing requirements for contextual reasoning, cross-artifact coordination, and preservation. Results show reasonable performance on Easy and Medium tasks but a substantial performance gap on Hard tasks, indicating that current agents struggle as editing requires more complex, coordinated changes across artifacts.

Critical strengths
  • Important problem formulation: The paper highlights an important limitation of existing editing evaluations: successful editing requires not only making the requested change but also propagating its consequences while preserving unrelated state.
  • Fine-grained evaluation: The benchmark separately measures target/semantic correctness, preservation, and complete task success, enabling more informative diagnosis than a single end-to-end success metric.
  • Useful evaluation abstraction: The decomposition into required changes, dependent consequences, and preservation constraints provides a useful framework that could inform future benchmarks for stateful artifact editing.
Critical weaknesses
  • Limited scope and durability: The benchmark covers a relatively narrow set of Office-editing tasks, with the Hard set concentrated in a small number of source scenarios, and the paper does not demonstrate a scalable process for generating diverse new workspaces with reliable change and preservation checks.
  • Limited empirical evidence: The evaluation only considers a small, heterogeneous set of agent-product snapshots under nonuniform evaluation conditions, making the claimed SOTA gap difficult to interpret.
  • Verifier validity is insufficiently established: Because strict task success requires satisfying a large conjunction of automatically specified semantic and preservation checks, the lack of independent human validation makes it unclear how much of the dramatic Hard-task performance gap reflects genuine editing failures versus limitations of the verifier.
Questions for authors

See comments above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dmd8

Thank you for recognizing the value of making required changes, downstream consequences, and preservation constraints explicit. We clarify what the benchmark and the three-system evaluation establish.

1. What is reusable beyond the four Hard workspaces? The reusable contribution is the task contract: it identifies the authoritative evidence, enumerates the required updates and their dependencies, specifies the parts of the existing artifacts that are protected, and makes the resulting checks inspectable. An evaluator can use this structure to distinguish a missed downstream update from an unauthorized edit, even when both result in a strict failure. The four Hard workspaces demonstrate the approach across different domains, but do not establish that new workspaces can be generated and validated at scale.

2. What does the evaluation establish across systems? All three agents received the same task inventory and were evaluated under the same frozen scoring criteria and first-formal-attempt rule. Only network interruptions allowed retries, and unsuccessful or low-scoring formal outcomes were retained. The central result does not depend on a ranking: within each of the three systems, 92–100% of Hard tasks yielded a package that passed the file checks, yet no selected output passed every task requirement. This shows the diagnostic value of measuring delivery separately from what the delivered files accomplish. Three product snapshots do not establish a general state-of-the-art ranking.

3. Does the strict conjunction obscure substantive failures? The benchmark exposes the component results precisely so that readers can investigate this question. For example, among WorkBuddy’s 46 complete Hard deliveries, none passed every target semantic check. Relaxing preservation checks alone would therefore not change the strict result for those deliveries under the existing semantic criteria. The case analysis also identifies concrete failures that can be read directly from an output, such as a revised rule that was not propagated to an affected conclusion. These observations support the distinction between delivering an editable package and completing the requested maintenance. The exact strict-pass rate remains tied to the frozen verifier; we do not interpret it as a human usability rate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.