acceptodds
Under review as a conference paper at ICLR 2027

DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations

Abstract

As autonomous agents rapidly evolve, their ability to reliably manipulate ubiquitous digital documents has become critical for enabling general-purpose AI assistants and automating complex workspace workflows. In this paper, we introduce DocOps, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities. Based on DocOps, we systematically evaluate representative closed- and open-source models across various agentic harnesses, revealing that even the most advanced frontier configurations still exhibit profound limitations when handling highly coupled, long-range tasks. Furthermore, a fine-grained analysis of existing agents' manipulation behaviors uncovers 3 key failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata. Ultimately, our work exposes the capability boundaries of agents in maintaining global document consistency, shedding light on the future design of robust, non-destructive agents for complex digital ecosystems.

Primary area:
datasets and benchmarks
Keywords:
Document operations, Autonomous agents, Office documents, Benchmark, Deterministic verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QsK7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents the DocOops benchmark, a benchmark to evaluate AI agent capabilities at editing documents such as word docs, pdfs, slides, spreadsheets, etc. The authors collected a total of 210 documents, each associated with a task across 4 difficulty levels (L1 -> L4). They then proposed a reference-based verifier for each task, and used it to evaluate various LLMs within various agentic harnesses, finding that the models make errors that are more prevalent as editing tasks become more complex.

Critical strengths

Studying complex document editing in agentic frameworks is an important topic.

Critical weaknesses

I do not believe that the work, as it is presented, significantly contributes to the community. Throughout the work, there is a severe lack of details, that makes it hard to follow what was done, how it was verified, and whether the results are novel and significant. To be frank, the work feels more like a sales pitch for a framework (Harbor framework) than an actual scientific contribution. See the non-exhaustive list of questions below that would help address the concerns, and make the contribution transparent enough that it is reproducible and disgestible by the community.

Questions for authors
  1. Where did you collect the documents you use in the DocOps benchmark.
  2. Are they real or synthetic documents?
  3. In each environment, how many total documents are there (since L4 requires multi-document operation apparently)? How many total tokens does that represent?
  4. How did you come up with the tasks within each environment? Are they realistic? How do you sample across L1 -> L4 tasks, this seems like it would take a lot of work to setup correctly, yet we have no details.
  5. It seems like your evaluation protocol (3.3) is reference-based (i.e., it requires linguistic anchors to be present, and certain structural predicates to be present). How were these obtained? This seems like a nontrivial amount of work. If you used LLMs to build the references, is that sound?
  6. There are likely many potential correct solutions in certain docops environments (i.e., there isn't a single correct excel formula for a given task). Does your evaluation protocol handle such cases correctly, and how did you test for it?
  7. The level of description of the verification is inadequate. You point to Appendix D, but it just discusses a manual audit you conducted, and not how the Strucutral Predicates, Linguistic anchors, and preservation predicates actually work. Having worked in the field of document editing, I can tell you I am fairly dubious that simple "anchor matching" is sufficient to evaluate complex editing task completion. Can you provide a compelling explanation and example of how this evaluation works and why it is the best way to perform evaluation for this task?
  8. "All experiments are executed through Harbor". What is Harbor, why is it needed? It is mentioned throughout the draft, but it is unclear what its purpose is. If it is just an engineering detail, why is it important to mention?
  9. The AI figures (Figure 1 and Figure 4) are generally not useful, as they are just high-level illustrations. I would remove them and instead use the precious space to give concrete details on what you actually did.
4 comments

Thank you for recognizing the importance of complex document editing. We clarify the construction and evaluation details below and answer each of your questions.

Part1:

Q1 & Q2: Where did you collect the documents you use in the DocOps benchmark. Are they real or synthetic documents?

As described in Section 3.2 (lines 186 to 206), the native input documents in DocOps are synthesized around tasks inspired by practical office needs. We screened 600 candidate seed sources, comprising 472 community requests, 125 workflow references or cases, and three automation templates, and retained 482.

To further characterize the diversity and the initial condition of these synthetic documents, we complement their design characteristics with document metadata. The design emphasizes information distributed across files, initial issues that have determinate solutions, and requests to modify only selected content. Structurally, of 123 workbooks, 49 contain multiple worksheets and 12 contain hidden sheets. In addition, 42 Word documents contain native tables, 13 presentations contain speaker notes, and 14 PDFs contain bookmarks. Some documents also keep unfinished content outside the requested scope, which requires agents to respect the intended local edit. These characteristics and statistics reflect our attention to content, native structure, and editing boundaries, and they preserve the operational demands and complexity of practical office work inside synthetic documents. We also elaborate on this point in Appendix B of the revised manuscript (lines 787 to 795).

Q3: In each environment, how many total documents are there (since L4 requires multi document operation apparently)? How many total tokens does that represent?

As shown in Figure 6aa in Appendix B, the 210 task environments in DocOps contain 463 native input documents in total. We also computed token counts with the Qwen3-8B tokenizer. The table summarizes input document counts, documents per task, total tokens, and mean tokens per task by difficulty.

DifficultyTasksInput documentsDocuments per taskTotal tokensMean tokens per task
L1505016,744134.88
L24040113,583339.58
L36060127,139452.32
L4603134 to 1030,620510.33

These statistics also show how document scale relates to our difficulty design. L3 tests consistency across several workflow steps within a single document that contains richer text, while L4 uses 4 to 10 input documents per task to emphasize information alignment and coordinated editing across documents.

Q4: How did you come up with the tasks within each environment? Are they realistic?

As stated in Section 3.1 (lines 165 to 177) and Section 3.2 (lines 198 to 206), we group atomic operations into content, format, and structure, and we design the tasks in each environment around these operations using predefined difficulty levels. Difficulty levels are established during task design, where they guide task formalization and the construction of source documents, rather than assigned afterwards to label and sample tasks that were already designed. Specifically, L1 requires a single atomic edit, L2 combines several operations within one document, L3 requires a document-level workflow that keeps global consistency across multiple steps, and L4 requires information alignment and coordinated editing across documents. The progression therefore reflects growing planning requirements and growing state dependencies within and across documents.

To reflect practical office needs, and as we emphasized in our response to Q1 & Q2, the tasks also focus on three properties. The information needed for an edit is distributed across files, the initial documents contain unresolved issues, and users request changes to only selected content. These properties translate into information dependencies, required operations, and preservation boundaries, which connects our difficulty design to the editing requirements of real office work.

Part 2:

Q5: It seems like your evaluation protocol 33˙3\.3 is reference based

DocOps evaluates whether the delivered document satisfies the success conditions of the task, not whether it matches a single reference answer or reference document. As described in Section 3.3 (lines 223 to 230), the user request specifies the required changes, the source files establish facts and their relationships, and the original document together with the editing scope defines the state to preserve. The verifier checks these conditions directly through content, structure, and preservation checks. These conditions constrain the editing outcome without prescribing an editing method or a unique output representation. Different correct implementations pass when they satisfy the same functional and preservation requirements. A formula aggregation task, for instance, checks computational functionality and the recalculated results without matching a particular formula string. Grounding expected facts and states in source files therefore does not require the model to reproduce a particular reference solution.

Q6: there isn't a single correct excel formula for a given task

DocOps verifiers assess whether the final artifact satisfies the task requirements, regardless of how many solutions exist. A task may admit several correct solutions, and when the delivered file satisfies the same success conditions, a different editing method is not a reason for rejection.

We explicitly account for the possibility of multiple correct Excel formulas that you raise. In the budget allocation task the user requests formula-based totals for Maintenance and Operations. The verifier derives the expected totals from the original cost records and the mapping, checks that the designated cells contain formulas, clears cached values in a private copy, recalculates with LibreOffice, and then checks the result for each category. It neither matches a particular formula string nor relies on existing cached values. We tested eight implementations: SUMIF, SUMIFS, arithmetic references to source cells, SUMPRODUCT with the mapping, FILTER-based dynamic aggregation, an XLOOKUP helper, named ranges, and a helper sheet. All eight passed. All four negative controls were rejected, including swapped category formulas that keep the grand total unchanged but assign amounts to the wrong categories. This shows that the verifier separates differences in computational implementation from violations of the requested requirements.

The same principle applies across other file formats and difficulty levels. We check target objects, functionality, relationships, and preserved state, with the aim of covering all valid implementations without prescribing an operation sequence or a unique internal file representation. The possibility of several solutions is therefore built into the design of our evaluation protocol. We also elaborate on this point in Appendix D of the revised manuscript (lines 888 to 900).

Q7: Can you provide a compelling explanation and example of how this evaluation works and why it is the best way to perform evaluation for this task?

The main reason for this choice of evaluation is that, compared with the widely used LLM judge approach, inspecting the final file directly turns operation success into executable, auditable conditions and locates the specific requirements that remain unmet. Deterministic checks assess the requested changes, the preservation of unrelated content, and the native structure, and they return a consistent decision for the same file and the same verifier. Reviewer 1T1k also notes that evaluating artifacts directly makes the results easier to reproduce and failures easier to diagnose. On verification efficiency and cost, these checks run automatically and locally and support parallel batch execution without an additional LLM call for each artifact, which avoids judge model inference fees and service call overhead.

Figure 4cc gives a concrete example. The user requests an “Owner” column between “Milestone” and “Due Date” in a native slide table, but the agent only overlays a text box to imitate a new column. An LLM judge that relies on screenshots or extracted text may see the expected header and layout and consider the task complete, without establishing whether the native table really contains the new column. The verifier parses the native table directly and checks its column count, column order, and header correspondence to establish whether “Owner” is an editable table column, and it localizes the missing structural modification. This illustrates the value of deterministic verification. It checks the requested state of the native object directly, it avoids confusing visual completion with functional completion, and it produces specific, auditable evidence of the error.

Part 3:

Q8: What is Harbor, why is it needed?

Harbor, developed by the Terminal-Bench team, is one of the most widely used frameworks for agent evaluation. Representative agent benchmarks that use Harbor for evaluation include Terminal-Bench 2.0 [1], SkillsBench [2], TerminalWorld [3], and BankerToolBench [4]. These applications show the broad influence of Harbor as shared evaluation infrastructure, and its adoption also extends to frontier model evaluation. The Claude Sonnet 4.6 system card reports Terminal-Bench 2.0 evaluation using Harbor with Terminus-2 [5], and the official Kimi K3 technical blog reports evaluation using the official Harbor implementation of PostTrainBench [6]. We chose Harbor for its standardized containerized execution, its compatibility with several agents, and its support for parallel batch evaluation.

In practice, DocOps uses Harbor to manage task environments, agent invocation, resource and time limits, collection of artifacts and trajectories, and verifier execution. This lets different models and harnesses be evaluated under shared task definitions and explicit execution conditions, and it makes the experiments easier to replicate and extend. Harbor handles orchestration and execution, while the DocOps tasks and verifiers define the operation requirements and the success criteria. Reporting Harbor in the paper makes this experimental execution procedure explicit.

[1] Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. ICLR 2026.

[2] SkillsBench: Benchmarking how well agent skills work across diverse tasks. NeurIPS 2026.

[3] TerminalWorld: Benchmarking agents on real-world terminal tasks. NeurIPS 2026.

[4] BankerToolBench: Evaluating AI agents in end-to-end investment banking workflows. NeurIPS 2026.

[5] System card: Claude Sonnet 4.6. Anthropic, 2026.

https://www.anthropic.com/claude-sonnet-4-6-system-card

[6] Kimi K3 tech blog: Open frontier intelligence. Moonshot AI, 2026.

https://www.kimi.ai/blog/kimi-k3

Q9: The AI figures (Figure 1 and Figure 4) are generally not useful, as they are just high level illustrations.

We would like to firmly clarify that the figures in the paper were created and laid out by the authors, rather than generated as AI images. Figures 1 and 4 were composed by hand in PowerPoint using SVG assets, and the data plots were produced from the experimental statistics.

Figures 1 and 4 serve clear purposes in helping readers understand the paper. Figure 1 brings task construction, native artifacts, verification, and the operation and difficulty taxonomy together into one view of the DocOps design. Figure 4 places concrete requests, agent behavior, and violated conditions side by side, showing typical errors in global state maintenance, computational semantics, and native structure modification. The example of a text box imitating a table column, in particular, makes the difference between plausible appearance and a correct operation immediately visible. These figures provide a methodological overview and an explanation of failure mechanisms, and they complement the text and the quantitative results in a way that is common in scientific writing.

As the discussion period is about to conclude, we wanted to briefly follow up on whether our responses have addressed your main concerns. We would be very happy to provide any further clarification if helpful.

In any case, we sincerely appreciate your time and constructive feedback throughout the review process.

Reviewer_1T1kOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DocOps, a benchmark for evaluating autonomous agents on complex operations involving native-format documents. It contains 210 tasks across Excel, Word, PowerPoint, and PDF, organized into four difficulty levels ranging from atomic edits to cross-document workflows. Each task includes source artifacts, a natural-language instruction, optional document skills, and a deterministic verifier that checks both requested changes and preservation requirements. The authors evaluate multiple open- and closed-source models with several execution harnesses, with and without skills. The strongest reported configuration, GPT-5.5 with Codex and skills, achieves a 67.1% pass rate. Performance declines on longer and cross-document workflows, and the analysis identifies state-tracking errors, shallow semantic verification, and destructive edits to native document structure as recurring failure modes. The authors also report a human audit of 128 verifier judgments and a mutation test of verifier sensitivity.

Critical strengths
  1. The benchmark targets a meaningful evaluation gap. Document operations require agents to make scoped edits while preserving unrelated content and native structure. DocOps evaluates the resulting artifacts directly with deterministic checks, making its results more reproducible and diagnosable than evaluations based only on textual responses or an LLM judge.

  2. The evaluation offers useful breadth and diagnostic evidence. The benchmark spans four formats and multiple workflow complexities, and the study compares models, harnesses, and skill settings. Its findings—that performance drops on complex workflows and that harness and skill effects depend on the model—provide concrete guidance for future document-agent research.

Critical weaknesses
  1. The benchmark’s representativeness is not fully established. The paper describes a controlled, synthetic task-construction pipeline based on collected seeds, standardized prompting, and synthesized source artifacts. It also reports one main human reviewer, but does not establish independent reviewer agreement. This makes it difficult to assess how well the benchmark captures the ambiguity, variety, and document quality of real workplace requests.

  2. The verifier validation leaves some uncertainty about benchmark reliability. The human audit covers 128 judgments, with 122 agreements, and the mutation test covers 180 injected violations across 36 outputs. These checks are useful, but their coverage may not capture subtle semantic damage or unintended edits outside the verifier’s tested predicates. Since the primary score is binary and depends on the verifier, undetected false passes or false failures could affect the measured results.

Questions for authors
  1. How were the 128 verifier judgments selected across formats, difficulty levels, and predicate types? Could you report the false-pass and false-fail rates by these categories, or clarify whether an independent reviewer audit was conducted?

  2. The mutation test applies five injected mutations to each of 36 outputs. How representative are these mutations of realistic unintended edits, particularly semantic changes that preserve document structure? What kinds of errors are not currently covered by the verifier?

  3. The main results include unavailable model–harness combinations due to protocol incompatibility. Could you report matched comparisons on the common subset of tasks and settings, to clarify how much of the observed difference is attributable to the harness versus the model–harness pairing?

3 comments

Thank you for your constructive review and for recognizing the evaluation value and breadth of coverage of DocOps. We address each concern below and provide the additional evidence you requested.

Part1:

W1: how well the benchmark captures the ambiguity, variety, and document quality of real workplace requests.

Thank you for this suggestion. We agree that an office benchmark should reflect the ambiguity and diversity of real workplace requests as well as the initial condition of the documents. We address the three aspects in turn.

  • Request ambiguity. To handle ambiguity in real requests, we use the user request and the source files together to specify the editing goal and the boundaries of preservation, while allowing different editing methods that reach the same goal. During task review, candidate tasks whose completion criteria remain unclear from this information are revised or excluded through human review, so that every task goal is explicit.

  • Diversity of sources and workflows. To quantify diversity, we characterized the seed sources and the input files. As stated in Section 3.2 (lines 190 to 196), the 600 candidate sources comprise 472 community requests, 125 workflow references or cases, and three automation templates, of which 482 were retained after screening. The current 210 tasks contain 463 native input files. The 60 L4 tasks that span multiple formats each involve 4 to 10 input files and cover varied sources and requirements for coordination across documents.

  • Initial document condition. To characterize the initial structures and editing conditions, we compiled document metadata. Of 123 workbooks, 49 contain multiple worksheets and 12 contain hidden sheets. In addition, 42 Word documents contain native tables, 13 presentations contain speaker notes, and 14 PDFs contain bookmarks. These statistics reflect the structural complexity of the documents. Some documents also keep unfinished content outside the region to be edited, which requires models to respect the scope of a local edit while they resolve the specified issue, mirroring the unfinished drafts that appear in workplace settings.

Together, these design choices and statistics show that DocOps covers real request sources, dependencies across files, and varied initial document states while keeping completion criteria explicit. We have also been expanding DocOps with real workplace documents to cover a wider variety of initial document conditions and editing requests. We also elaborate on this point in Appendix B of the revised manuscript (lines 787 to 795).

W2 & Q1: How were the 128 verifier judgments selected across formats, difficulty levels, and predicate types?

Thank you for this suggestion. We have added a breakdown of the 128 reviewed decisions by document format and task difficulty. The audit covers every format at every difficulty level in DocOps, providing a basis for assessing verifier reliability across these task conditions. As the table shows, all 16 combinations of format and difficulty are represented. Each format accounts for 27 to 42 decisions 211˙21\.1% to 32\.8% of the audit, with no single format accounting for a majority. The reviewed outputs also span 51 configurations across 12 models and four harnesses, so the audit is not confined to any one model or execution setting. We also provide further details in Appendix D of the revised manuscript (lines 852 to 858 and Table 4).

Document formatL1L2L3L4Total reviewed
Excel13915542
Word6861030
PowerPoint1348227
PDF4813429

Part2:

Q2: How representative are these mutations of realistic unintended edits, particularly semantic changes that preserve document structure?

Thank you for this suggestion. Two examples clarify how the mutation tests relate to real editing scenarios.

Correct values assigned to the wrong fields. In a presentation task the source data specifies “Open Items = 6” and “Late Items = 2”. We swap the two values while keeping the labels, text boxes, and slide structure intact, which simulates a copy and paste misalignment or an agent assigning values to the wrong metrics. Although the correct numbers are still present, the verifier identifies and rejects the incorrect relationship between labels and values.

A change in representation without a change in meaning. In a document produced for a task we split the same text into several text runs while keeping the visible content and the relationships between labels and values unchanged. The output still passes verification. This simulates the internal representation changes that commonly arise during formatting and shows that the verifier accepts such edits when they do not affect the task result.

DocOps therefore designs its verifiers around task success conditions grounded in the source documents, checking facts, relationships, and preservation requirements while allowing different implementations that satisfy those conditions. Our mutation tests build on this design by introducing different types of erroneous input to assess whether the verifiers make accurate judgments according to the task requirements. The examples above illustrate this process and support both the adequacy of our verification coverage and the reliability of the judgments.

Q3: how much of the observed difference is attributable to the harness versus the pairing of model and harness?

Thank you for this suggestion. We compared harness switches for the same eight open-source models on the common set of 210 tasks without skills, using 32 configurations from Table 1. We find that general-purpose, programmable harnesses achieve higher mean pass rates than agents built on a tool-based interface, but the benefit depends on the pairing of model and harness, consistent with Section 4.2.4 (lines 449 to 455).

The table reports mean pass rate changes, the number of models that improve or deteriorate, and the number of “opposite effect tasks” for six harness switches. The last count refers to tasks on which at least one model moves from fail to pass while another moves from pass to fail. Terminus-2 performs best overall across the eight open-source models, while switching between Codex and Claude Code produces opposite effects across models on 46 tasks, which shows that harness benefits depend on the paired model.

Harness switchMean Δ ppppModels ↑ / ↓Opposite effect tasks / 210
DocTools → Terminus-2+27.448 / 012
DocTools → Codex+16.077 / 120
DocTools → Claude Code+21.858 / 013
Terminus-2 → Codex-11.371 / 743
Terminus-2 → Claude Code-5.600 / 833
Codex → Claude Code+5.777 / 146

For the same models, enabling the same skill bundle raises mean pass rates under Codex and Claude Code by 0.89 and 2.20 percentage points, respectively, yet two and three models deteriorate. This further supports our finding that skill benefits remain inconsistent (lines 462 to 470).

As the discussion period is nearing its conclusion, we wanted to briefly check whether our responses and additional analyses have addressed your main concerns. We would be very happy to provide any further clarification if helpful.

We sincerely appreciate your time and constructive feedback throughout the review process.

Reviewer_eKKGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DocOps, a benchmark of 210 tasks covering Excel, Word, PowerPoint, and PDF operations. The tasks range from local edits to workflows involving multiple input and output files. Each task has a deterministic verifier that checks the requested changes, native document structure, and specified content that should remain unchanged. The authors evaluate different models, harnesses, and skill settings. Results show that agents struggle more with complex workflows, and performance depends strongly on the execution environment.

Critical strengths
  1. The paper makes document operations more standardized across different file formats. For me, I think checking whether the original structure and other content are preserved is useful.
  2. The multi-file input and output tasks are interesting. These tasks require the model to keep information consistent across documents, which is important for practical document work.
Critical weaknesses
  1. I am not fully sure how much is new compared with previous benchmarks. It seems more like bringing existing document operations together and making the evaluation more standardized. This can be useful, but the difference from PPTArena, SpreadsheetBench, and OfficeBench needs to be clearer.
  2. Some analysis is missing. We do not know how much the scores change after adding structure and preservation checks. Also, only three closed-source models are evaluated, and two are GPT models. It is hard to see whether different strong models have different limitations on this benchmark.
Questions for authors
  1. Could you report the scores with and without the structure and preservation checks? It will be great to know how many outputs would otherwise pass but are rejected by these checks.
  2. Could you give some concrete examples that previous benchmarks cannot evaluate but DocOps can? Is the main difference the checks, or the multi-file workflows?
  3. Could you add more closed-source models if the cost is acceptable? I would like to see whether they fail on different kinds of tasks.
3 comments

Thank you for your interest in DocOps and for recognizing its value. We address your concerns and questions in detail below.

Part1:

W1: the difference from PPTArena, SpreadsheetBench, and OfficeBench needs to be clearer.

As stated in Appendix A (Table 2), we have compared DocOps with SpreadsheetBench, PPTArena, and OfficeBench. Here we explain how DocOps differs along three dimensions: task scope, evaluation requirements, and agent system analysis. The comparison makes our central contribution clear: DocOps takes an operation-centered view, treating documents as stateful computational objects and evaluating each edit as a state transition. On this basis we design complete deterministic tasks and verifiers that require the requested target state, validity of the native format, and preservation of all relevant state outside the requested edit.

BenchmarkFormatsNative file editCross document workflowDeterministic target verifierExplicit preservationHarness & skill analysis
SpreadsheetBenchXLSX✓××××
PPTArenaPPTX✓××××
OfficeBenchDOCX, XLSX, PDF + apps✓✓✓××
DocOps (ours)XLSX, DOCX, PPTX, PDF✓✓✓✓✓

Task scope. DocOps covers content, formatting, and structural operations across XLSX, DOCX, PPTX, and PDF, ranging from individual edits to workflows within and across documents. Beyond format coverage or application switching, the tasks test how agents coordinate dependent operations and how they align information from several sources inside the delivered documents.

Evaluation requirements. DocOps checks deterministic success conditions directly in the final documents, where those conditions are defined by the user request and the source files. It verifies the required native document state, the functionality the document must retain, and the preservation constraints together, and it accepts different implementations that satisfy these conditions. OfficeBench also uses deterministic checks, but it assesses workflow completion mainly through exact or fuzzy matching against expected outputs and through task-specific code execution. It does not establish native document state and explicit preservation as joint evaluation targets at the benchmark level.

Agent system analysis. DocOps compares models across execution harnesses on the same tasks and examines skill injection within a fixed model and harness pairing. This shows how model capability, execution interface, and procedural guidance jointly shape performance, and we read the results together with an analysis of failures in document operations.

Taken together, these dimensions make DocOps a unified framework for evaluating and diagnosing reliable document operations, a contribution that goes beyond adding formats or extending task coverage.

Q1 & W2: Could you report the scores with and without the structure and preservation checks?

Thank you for this suggestion. We compared pass rates with and without the structure and preservation checks. The results show that our verifiers reliably identify outputs that break structural functionality or violate preservation requirements, so such harmful edits are not counted as successes, which supports reliable evaluation of document operations. The table reports the aggregate results and two representative combinations across 42 configurations. Each of the seven Gemma and Qwen model variants is evaluated on the full task set under six settings. Disabling both checks raises the overall pass rate from 24.66% to 34.58% and admits 875 additional outputs. These outputs satisfy the remaining checks but fail the structural or preservation requirements. Structural or preservation errors can also occur together with other errors, in which case disabling these checks does not change the task outcome. The number of affected tasks may therefore be larger, and the additional passes give only a lower bound. We also address this point in Appendix D of the revised manuscript (lines 901 to 909 and Table 5).

SettingFull checksWithout structure and preservation checksAdditional passes
Overall: 7 models × 6 settings2,175/8,820 246˙624\.66%3,050/8,820 345˙834\.58%875 +99˙2pp+9\.92 pp
Qwen3.5-35B-A3B · Codex w/o skill33/210 157˙115\.71%48/210 228˙622\.86%15 +71˙4pp+7\.14 pp
Qwen3.5-122B-A10B · Claude Code w/o skill62/210 295˙229\.52%85/210 404˙840\.48%23 +109˙5pp+10\.95 pp

Part 2:

Q2: Could you give some concrete examples that previous benchmarks cannot evaluate but DocOps can? Is the main difference the checks, or the multi file workflows?

To illustrate the kinds of document operations that DocOps can evaluate, we present one task from a single-document workflow and one from a cross-document workflow. The core difference from existing benchmark tasks is that we design each task together with its verifier around deterministic operations. The user request and the source files jointly define which states must change and which must be preserved. Deterministic verifiers then check those conditions directly in the delivered documents without prescribing how the agent performs the edits. Cross-document workflows add further constraints that involve several sources. The two examples follow.

Single-document example: preparing a PDF for a board meeting. This task requires removing an obsolete cover and the draft markers from an existing PDF, arranging the four retained pages in the specified order while keeping their original text, and creating a bookmark for each page. The verifier checks these requirements one by one in the delivered PDF. It compares the text of the retained pages with the original, verifies the page order, and resolves the actual destination of every bookmark. In one tested erroneous output the “Management Summary” bookmark pointed to the wrong page. The verifier therefore rejected the file and identified the incorrect bookmark destination.

Cross-document example: preparing a client-facing invoice packet. This task combines a release register x˙lsx\.xlsx, the original invoices p˙df\.pdf, a redaction policy d˙ocx\.docx, and an approved appendix p˙df\.pdf. We define the editing operations together with their acceptance criteria. Invoices are selected according to the register, each invoice keeps its vendor and amount, tax IDs and bank routing numbers are replaced according to the policy, duplicate and internal pages are removed, and the specified appendix is inserted. Working from the same sources, the verifier checks field preservation and replacement for each invoice number, then checks page composition and order. This turns cross-document editing requirements into explicit, executable acceptance conditions.

This design ties correctness to the operations the user actually requested. Implementation differences that do not affect functionality are accepted, while edits that break functionality or integrity, or that change business meaning, trigger the corresponding checks. We believe that this approach to task design could also inform targeted corrective feedback in future agent training.

Q3: Could you add more closed source models if the cost is acceptable?

Thank you for this suggestion. DocOps already evaluates GPT and Claude, two representative closed-source families that are also among the most widely used in current agent research, including OSWorld [1] and τ-bench [2]. To address your concern further, we have added experiments with Kimi K3. The table reports task pass rates for the completed configurations.

ModelDocToolsTerminus-2Codex w/o skillsCodex w/ skillsClaude Code w/o skillsClaude Code w/ skills
Kimi K3Pending58.10%53.33%53.33%52.86%53.33%

With this addition our evaluation spans seven model families: GPT, Claude, DeepSeek, Gemma, Qwen, GLM, and Kimi. These cover most representative open- and closed-source families. The highest recorded pass rate of Kimi K3 is 58.10%, which further highlights how demanding reliable document operations remain. We would also be glad to include newly released models in future evaluations to broaden our coverage.

[1] OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. NeurIPS 2024.

[2] τ-bench: A benchmark for tool-agent-user interaction in real-world domains. ICLR 2025.

As the discussion period is about to conclude, we wanted to briefly check whether our responses and additional experiments have addressed your main concerns. We would be very happy to provide any further clarification if helpful.

In any case, we sincerely appreciate your time and constructive feedback throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.