- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents the DocOops benchmark, a benchmark to evaluate AI agent capabilities at editing documents such as word docs, pdfs, slides, spreadsheets, etc. The authors collected a total of 210 documents, each associated with a task across 4 difficulty levels (L1 -> L4). They then proposed a reference-based verifier for each task, and used it to evaluate various LLMs within various agentic harnesses, finding that the models make errors that are more prevalent as editing tasks become more complex.
Critical strengths
Studying complex document editing in agentic frameworks is an important topic.
Critical weaknesses
I do not believe that the work, as it is presented, significantly contributes to the community. Throughout the work, there is a severe lack of details, that makes it hard to follow what was done, how it was verified, and whether the results are novel and significant. To be frank, the work feels more like a sales pitch for a framework (Harbor framework) than an actual scientific contribution. See the non-exhaustive list of questions below that would help address the concerns, and make the contribution transparent enough that it is reproducible and disgestible by the community.
Questions for authors
- Where did you collect the documents you use in the DocOps benchmark.
- Are they real or synthetic documents?
- In each environment, how many total documents are there (since L4 requires multi-document operation apparently)? How many total tokens does that represent?
- How did you come up with the tasks within each environment? Are they realistic? How do you sample across L1 -> L4 tasks, this seems like it would take a lot of work to setup correctly, yet we have no details.
- It seems like your evaluation protocol (3.3) is reference-based (i.e., it requires linguistic anchors to be present, and certain structural predicates to be present). How were these obtained? This seems like a nontrivial amount of work. If you used LLMs to build the references, is that sound?
- There are likely many potential correct solutions in certain docops environments (i.e., there isn't a single correct excel formula for a given task). Does your evaluation protocol handle such cases correctly, and how did you test for it?
- The level of description of the verification is inadequate. You point to Appendix D, but it just discusses a manual audit you conducted, and not how the Strucutral Predicates, Linguistic anchors, and preservation predicates actually work. Having worked in the field of document editing, I can tell you I am fairly dubious that simple "anchor matching" is sufficient to evaluate complex editing task completion. Can you provide a compelling explanation and example of how this evaluation works and why it is the best way to perform evaluation for this task?
- "All experiments are executed through Harbor". What is Harbor, why is it needed? It is mentioned throughout the draft, but it is unclear what its purpose is. If it is just an engineering detail, why is it important to mention?
- The AI figures (Figure 1 and Figure 4) are generally not useful, as they are just high-level illustrations. I would remove them and instead use the precious space to give concrete details on what you actually did.
4 comments
Part 2:
Q5: It seems like your evaluation protocol is reference based
DocOps evaluates whether the delivered document satisfies the success conditions of the task, not whether it matches a single reference answer or reference document. As described in Section 3.3 (lines 223 to 230), the user request specifies the required changes, the source files establish facts and their relationships, and the original document together with the editing scope defines the state to preserve. The verifier checks these conditions directly through content, structure, and preservation checks. These conditions constrain the editing outcome without prescribing an editing method or a unique output representation. Different correct implementations pass when they satisfy the same functional and preservation requirements. A formula aggregation task, for instance, checks computational functionality and the recalculated results without matching a particular formula string. Grounding expected facts and states in source files therefore does not require the model to reproduce a particular reference solution.
Q6: there isn't a single correct excel formula for a given task
DocOps verifiers assess whether the final artifact satisfies the task requirements, regardless of how many solutions exist. A task may admit several correct solutions, and when the delivered file satisfies the same success conditions, a different editing method is not a reason for rejection.
We explicitly account for the possibility of multiple correct Excel formulas that you raise. In the budget allocation task the user requests formula-based totals for Maintenance and Operations. The verifier derives the expected totals from the original cost records and the mapping, checks that the designated cells contain formulas, clears cached values in a private copy, recalculates with LibreOffice, and then checks the result for each category. It neither matches a particular formula string nor relies on existing cached values. We tested eight implementations: SUMIF, SUMIFS, arithmetic references to source cells, SUMPRODUCT with the mapping, FILTER-based dynamic aggregation, an XLOOKUP helper, named ranges, and a helper sheet. All eight passed. All four negative controls were rejected, including swapped category formulas that keep the grand total unchanged but assign amounts to the wrong categories. This shows that the verifier separates differences in computational implementation from violations of the requested requirements.
The same principle applies across other file formats and difficulty levels. We check target objects, functionality, relationships, and preserved state, with the aim of covering all valid implementations without prescribing an operation sequence or a unique internal file representation. The possibility of several solutions is therefore built into the design of our evaluation protocol. We also elaborate on this point in Appendix D of the revised manuscript (lines 888 to 900).
Q7: Can you provide a compelling explanation and example of how this evaluation works and why it is the best way to perform evaluation for this task?
The main reason for this choice of evaluation is that, compared with the widely used LLM judge approach, inspecting the final file directly turns operation success into executable, auditable conditions and locates the specific requirements that remain unmet. Deterministic checks assess the requested changes, the preservation of unrelated content, and the native structure, and they return a consistent decision for the same file and the same verifier. Reviewer 1T1k also notes that evaluating artifacts directly makes the results easier to reproduce and failures easier to diagnose. On verification efficiency and cost, these checks run automatically and locally and support parallel batch execution without an additional LLM call for each artifact, which avoids judge model inference fees and service call overhead.
Figure 4 gives a concrete example. The user requests an “Owner” column between “Milestone” and “Due Date” in a native slide table, but the agent only overlays a text box to imitate a new column. An LLM judge that relies on screenshots or extracted text may see the expected header and layout and consider the task complete, without establishing whether the native table really contains the new column. The verifier parses the native table directly and checks its column count, column order, and header correspondence to establish whether “Owner” is an editable table column, and it localizes the missing structural modification. This illustrates the value of deterministic verification. It checks the requested state of the native object directly, it avoids confusing visual completion with functional completion, and it produces specific, auditable evidence of the error.
Part 3:
Q8: What is Harbor, why is it needed?
Harbor, developed by the Terminal-Bench team, is one of the most widely used frameworks for agent evaluation. Representative agent benchmarks that use Harbor for evaluation include Terminal-Bench 2.0 [1], SkillsBench [2], TerminalWorld [3], and BankerToolBench [4]. These applications show the broad influence of Harbor as shared evaluation infrastructure, and its adoption also extends to frontier model evaluation. The Claude Sonnet 4.6 system card reports Terminal-Bench 2.0 evaluation using Harbor with Terminus-2 [5], and the official Kimi K3 technical blog reports evaluation using the official Harbor implementation of PostTrainBench [6]. We chose Harbor for its standardized containerized execution, its compatibility with several agents, and its support for parallel batch evaluation.
In practice, DocOps uses Harbor to manage task environments, agent invocation, resource and time limits, collection of artifacts and trajectories, and verifier execution. This lets different models and harnesses be evaluated under shared task definitions and explicit execution conditions, and it makes the experiments easier to replicate and extend. Harbor handles orchestration and execution, while the DocOps tasks and verifiers define the operation requirements and the success criteria. Reporting Harbor in the paper makes this experimental execution procedure explicit.
[1] Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. ICLR 2026.
[2] SkillsBench: Benchmarking how well agent skills work across diverse tasks. NeurIPS 2026.
[3] TerminalWorld: Benchmarking agents on real-world terminal tasks. NeurIPS 2026.
[4] BankerToolBench: Evaluating AI agents in end-to-end investment banking workflows. NeurIPS 2026.
[5] System card: Claude Sonnet 4.6. Anthropic, 2026.
https://www.anthropic.com/claude-sonnet-4-6-system-card
[6] Kimi K3 tech blog: Open frontier intelligence. Moonshot AI, 2026.
https://www.kimi.ai/blog/kimi-k3
Q9: The AI figures (Figure 1 and Figure 4) are generally not useful, as they are just high level illustrations.
We would like to firmly clarify that the figures in the paper were created and laid out by the authors, rather than generated as AI images. Figures 1 and 4 were composed by hand in PowerPoint using SVG assets, and the data plots were produced from the experimental statistics.
Figures 1 and 4 serve clear purposes in helping readers understand the paper. Figure 1 brings task construction, native artifacts, verification, and the operation and difficulty taxonomy together into one view of the DocOps design. Figure 4 places concrete requests, agent behavior, and violated conditions side by side, showing typical errors in global state maintenance, computational semantics, and native structure modification. The example of a text box imitating a table column, in particular, makes the difference between plausible appearance and a correct operation immediately visible. These figures provide a methodological overview and an explanation of failure mechanisms, and they complement the text and the quantitative results in a way that is common in scientific writing.
As the discussion period is about to conclude, we wanted to briefly follow up on whether our responses have addressed your main concerns. We would be very happy to provide any further clarification if helpful.
In any case, we sincerely appreciate your time and constructive feedback throughout the review process.
Thank you for recognizing the importance of complex document editing. We clarify the construction and evaluation details below and answer each of your questions.
Part1:
As described in Section 3.2 (lines 186 to 206), the native input documents in DocOps are synthesized around tasks inspired by practical office needs. We screened 600 candidate seed sources, comprising 472 community requests, 125 workflow references or cases, and three automation templates, and retained 482.
To further characterize the diversity and the initial condition of these synthetic documents, we complement their design characteristics with document metadata. The design emphasizes information distributed across files, initial issues that have determinate solutions, and requests to modify only selected content. Structurally, of 123 workbooks, 49 contain multiple worksheets and 12 contain hidden sheets. In addition, 42 Word documents contain native tables, 13 presentations contain speaker notes, and 14 PDFs contain bookmarks. Some documents also keep unfinished content outside the requested scope, which requires agents to respect the intended local edit. These characteristics and statistics reflect our attention to content, native structure, and editing boundaries, and they preserve the operational demands and complexity of practical office work inside synthetic documents. We also elaborate on this point in Appendix B of the revised manuscript (lines 787 to 795).
As shown in Figure 6a in Appendix B, the 210 task environments in DocOps contain 463 native input documents in total. We also computed token counts with the Qwen3-8B tokenizer. The table summarizes input document counts, documents per task, total tokens, and mean tokens per task by difficulty.
These statistics also show how document scale relates to our difficulty design. L3 tests consistency across several workflow steps within a single document that contains richer text, while L4 uses 4 to 10 input documents per task to emphasize information alignment and coordinated editing across documents.
As stated in Section 3.1 (lines 165 to 177) and Section 3.2 (lines 198 to 206), we group atomic operations into content, format, and structure, and we design the tasks in each environment around these operations using predefined difficulty levels. Difficulty levels are established during task design, where they guide task formalization and the construction of source documents, rather than assigned afterwards to label and sample tasks that were already designed. Specifically, L1 requires a single atomic edit, L2 combines several operations within one document, L3 requires a document-level workflow that keeps global consistency across multiple steps, and L4 requires information alignment and coordinated editing across documents. The progression therefore reflects growing planning requirements and growing state dependencies within and across documents.
To reflect practical office needs, and as we emphasized in our response to Q1 & Q2, the tasks also focus on three properties. The information needed for an edit is distributed across files, the initial documents contain unresolved issues, and users request changes to only selected content. These properties translate into information dependencies, required operations, and preservation boundaries, which connects our difficulty design to the editing requirements of real office work.