acceptodds
Under review as a conference paper at ICLR 2027

Omni-CoWork: From Collaborative Audio-Visual Discussions to Workspace Edits

Abstract

Large language model (LLM) agents can substantially support collaborative work by assisting with coding, document editing, and content creation. However, when teams discuss changes around shared artifacts, users must still consolidate their evolving decisions into explicit instructions before delegating edits to agents, adding effort and delaying execution. This raises a key question: can agents infer and implement a group's final decisions directly from audio-visual discussions into workspace edits? To explore it, we introduce Omni-CoWork, a benchmark for translating collaborative audio-visual discussions into verifiable workspace edits. It contains 124 tasks with human-recorded views and 1,213 synthetic tasks, with executable checks assessing both required changes and preservation constraints. Across seven agents, task success reaches at most 4.0% and 24.3% on the two splits, respectively, whereas substantial gains with gold instructions point to intent understanding as a major bottleneck. Investigating this bottleneck, we find that richer audio-visual descriptions alone remain insufficient: task success improves when agents ground discussion references in artifacts and revisit evidence during execution. These findings highlight the need for grounded collective reasoning: recovering final collective decisions, grounding them in artifacts, and verifying their realization as the workspace evolves. We translate these requirements into an Omni-CoWork skill that yields consistent gains on two agents but leaves most tasks unsolved, underscoring the need for more effective methods to bridge the gap to reliable practical use.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal agents, Collaborative work, Audio-visual understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F2ypOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors release a benchmark for measuring how agents pick up tasks in audio-visual discussions. Using the findings of this benchmark, the authors combine these into a skill used as context into an llm which when used consistently improves performance.

Critical strengths

The outcome of this paper summarized as a skill is definitely useful, and condensing this into something that can be given into context to an existing llm is more useful than finetuning these models. Extensive construction pipeline. A lot of room on the benchmark for new models to perform better on with tasks successes of max 4% on human-recorded tasks, and 24.3% on synthetic tasks.

Critical weaknesses
  • The overall idea is useful, but I believe it fits maybe less in the academic setting of ICLR. The application is definitely interesting as an industrial application or software that can be used for collaborating with agents. It is mainly targeted for usage in workspace setting.
  • A benchmark like this could benefit from having a human evaluation baseline. If the authors would assign these tasks to a human (say a human assistant) how good it would perform (Table 3). Can agents do better inference, acting and checking than humans on workspaces (without gold instructions).
  • I'm missing a bit a conclusion on a solution, the authors merely pose issues that current models have still. How would this best be tackled? The authors write that both richer descriptions does not necessarily help a lot, and adding extra context into the prompt in form of a skill only helps a bit. I would like a bit more focus on what these effective methods potentially could be that would solve the gap between collective intent and workspace execution.
Questions for authors
  • For Quality control (section 3.2) I'm missing a human check on the synthetic data. How do the authors know these are correct tasks and the synthetic tasks in the dataset do not contain any bias or hallucinations, and the outcome is correct?
  • Can the authors explain a bit more why human-recorded task success is so much lower than synthetic task success? Could it be attributed also partly to the difference in the way tasks are defined, where human tasks have 3-6 subtasks with average 27.2 discussion items over an average 17.2 minutes, the synthetic videos do not necessarily have those statistics?
  • Line 377: "Yet most tasks remain unsolved with joint input, motivating further investigation into whether richer audio-visual evidence can improve intent understanding".
Could the authors do this further investigation during the rebuttal phase?
  • One particular question I was wondering is also the refusal rate of the agents. I personally have noticed especially in healthcare and medicine that agents refuse applying actions, because of safety alignment / guardrails. Since the authors especially outline this anti-doping review task as concrete task example (Appendix A.1), I wonder whether the authors have seen if there are any tasks in this benchmark that agents would refuse to do? I believe this is important to mention in the paper.
9 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the thoughtful feedback and constructive questions, which helped us clarify the paper’s academic contribution, strengthen the evaluation with human baselines and quality checks, and further develop Omni-CoWork through a two-agent extension. We address each concern point by point below.

W1: Alignment with ICLR’s Academic Scope

W1: The overall idea is useful, but I believe it fits maybe less in the academic setting of ICLR. The application is definitely interesting as an industrial application or software that can be used for collaborating with agents. It is mainly targeted for usage in workspace setting.

We thank the reviewer for recognizing the usefulness of this work. We would like to clarify that the workspace serves as an experimental environment; the paper’s academic contribution lies in formulating the research problem of recovering and correctly implementing final collective intent from multi-person multimodal discussions, and in evaluating agents’ ability to address it, which aligns with ICLR’s academic scope. Specifically, this contribution is reflected in three aspects: new research question, the evaluation design, and the diagnostic findings.

  1. New research question. Without pre-specified edit instructions, agents must recover the final decisions from multi-person discussions, distinguish accepted, rejected, and withdrawn proposals, and ground spoken and visual references in the actual workspace files.
  2. Verifiable evaluation. Executable checks measure both required edits and preservation of content that should remain unchanged, enabling objective evaluation of whether the final workspace reflects the agreed intent.
  3. Diagnostic findings. With the executor fixed, gold instructions substantially improve success (Table 3); joint perception of discussions and workspaces also outperforms recording-only plans (§5.3). These findings highlight bottlenecks in recovering and grounding intent.

Related ICLR papers provide useful precedents: VideoWebArena (ICLR 2025) similarly evaluates how agents translate video understanding into task execution, while Collaborative Gym (ICLR 2026) shares our focus on connecting human communication with agent execution in shared task environments. These parallels support Omni-CoWork’s relevance to ICLR as a benchmark for studying agent capabilities in practical environments.

[1] Lawrence Keunho Jang, Yinheng Li, Dan Zhao, Charles Ding, Justin Lin, Paul Pu Liang, Rogerio Bonatti, and Kazuhito Koishida. VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.

[2] Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration. In The Fourteenth International Conference on Learning Representations (ICLR), 2026.

AuthorsOpenReview ↗

Response to W2

W2: Human Performance Baseline

W2: A benchmark like this could benefit from having a human evaluation baseline. If the authors would assign these tasks to a human (say a human assistant) how good it would perform (Table 3). Can agents do better inference, acting and checking than humans on workspaces (without gold instructions).

We thank the reviewer for suggesting a human performance baseline. We conducted an additional evaluation with 15 PhD students, covering all 124 human-recorded tasks and 120 sampled synthetic tasks (approximately 10% of the synthetic split), given the rebuttal time constraints.

Protocol (Appendix B.7). Participants received no gold instructions. They first watched the full meeting video without workspace access. They then had 10 minutes to inspect and edit the workspace and could revisit the video as needed during this phase. When the time limit was reached, participants stopped editing, and the workspace state at that point was used to compute task success, E-Rate, and P-Rate with the same executable checks as in Table 3. After the task-performance evaluation, the same participants also assessed task quality in terms of correctness, realism, and hallucination (see our response to Q1).

Results. The human baseline results below are now included in Table 3 of the revised manuscript, with asterisks marking the sampled synthetic scores. All scores are percentages.

SplitTasksTask successE-RateP-Rate
Human-recorded12487.996.899.6
Synthetic (sampled)12092.597.599.5

On the full human-recorded split, humans exceed the gold-instruction reference in task success (87.9% vs. 61.3%) and E-Rate (96.8% vs. 94.0%), while matching its P-Rate (99.6%). The high E-Rate and P-Rate indicate that participants complete most required edits while rarely altering protected content.

This advantage may reflect humans’ ability to continuously integrate visual changes and speaker voices to track evolving intent, and to precisely revisit relevant video segments during editing. Under the same human evaluation protocol, success is lower on human-recorded tasks than on sampled synthetic tasks (87.9% vs. 92.5%), consistent with their greater complexity, more subtasks, and more pronounced shifts in intent.

AuthorsOpenReview ↗

Response to W3 & Q3

W3 & Q3: Effective Methods and Richer Audio-Visual Evidence

W3: I'm missing a bit a conclusion on a solution, the authors merely pose issues that current models have still. How would this best be tackled? The authors write that both richer descriptions does not necessarily help a lot, and adding extra context into the prompt in form of a skill only helps a bit. I would like a bit more focus on what these effective methods potentially could be that would solve the gap between collective intent and workspace execution.

Q3: Line 377: "Yet most tasks remain unsolved with joint input, motivating further investigation into whether richer audio-visual evidence can improve intent understanding".
Could the authors do this further investigation during the rebuttal phase?

We thank the reviewer for requesting clearer methodological directions and further investigation of richer audiovisual evidence. In response, we conducted additional experiments during the rebuttal period to address two questions:

  1. How can richer audiovisual evidence be used concretely to improve intent understanding?
  2. How can the findings in Section 5 and the Omni-CoWork skill be extended into a more effective method for connecting collective intent with workspace execution?

Starting points from Section 5. For the design below, we label three relevant results as follows:

  • Finding 1. Audio and vision are complementary, but joint input remains insufficient (§5.1).
  • Finding 2. Workspace context improves intent recovery and grounding during planning (§5.3).
  • Finding 3. Interleaved perception outperforms upfront perception under the same consultation budget (§5.4).

During rebuttal, we added Omni-CoWork skill component ablations (Appendix C.7, Table 15) and failure-mode attribution (Appendix C.6, Tables 13–14) to the revised manuscript:

  • Component ablations. Removing any component lowers task success for Gemini and Seed on both splits, supporting the importance of intent recovery and edit verification. Removing INTENT causes the largest synthetic-task drop for Gemini.
  • Failure attribution. Perception is the largest overall failure category, and repeated perception calls can also exhaust the execution budget, reinforcing perception as the main bottleneck.

These findings highlight perception as the main bottleneck, with Omni agents showing lower mean perception-error rates than LLM-based agents. Meanwhile, Opus outperforms Gemini with gold instructions on the analysis tasks (§5.2). These complementary strengths motivate a division of labor between an Omni perceiver and an LLM executor.

Extending Omni-CoWork into a two-agent system. We expand INTENT into an Omni agent's iterative perception and work-order construction, while the executor retains GROUND and REALIZE. This extension is reported in §5.6 of the revised manuscript, with implementation details in Appendix C.8:

  1. Agentic perception and work-order construction. The Omni agent integrates visual and spoken evidence while inspecting the meeting video in segments (based on Finding 1), jointly examining the relevant workspace content (based on Finding 2). It then uses targeted replay, high-frame-rate playback, and checking tools to refine its understanding. It organizes the richer audiovisual evidence and recovered decisions into an actionable work order for the execution agent.
  2. Execution with targeted follow-up. The non-Omni execution agent uses the work order to locate targets, apply edits, and verify the results. When questions arise during execution, it can ask the Omni agent for clarification (based on Finding 3). The Omni agent revisits the relevant evidence and supplies targeted perceptual information to support continued execution.

Evaluation of the two-agent extension. We paired Gemini 3.8 Flash as the omni agent with the five LLM executors below. The table reports task success (%) for the Table 3 baselines, original skill, and two-agent extension, with standalone Gemini and Seed for comparison. '\' denotes inapplicable configurations.

AgentHuman-recorded: original setupHuman-recorded: + original skillHuman-recorded: two-agent setupSynthetic: original setupSynthetic: + original skillSynthetic: two-agent setup
Opus 4.80.03.28.119.420.530.8
DeepSeek V4 Pro4.05.78.918.122.028.9
Grok 4.64.04.85.716.823.528.4
Qwen3.7 Max0.00.01.68.212.921.4
Kimi K31.62.44.89.713.928.9
Gemini 3.8 Flash4.08.1\24.327.9\
Seed 2.1 Pro1.63.2\21.323.7\

These results show that the two-agent system further improves task success, pointing to a promising research direction: designing more effective workflows that coordinate perception and execution agents. Given the time constraints, we have conducted only a preliminary exploration and will continue investigating more efficient coordination methods after the rebuttal period.

Reviewer_kzdMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Omni-CoWork, a new benchmark that requires AI agents to infer workspace edits directly from collaborative audio-visual team discussions without explicit textual edit instructions. Evaluations across seven agents reveal low task success rates, identifying recovering collective intent from multimodal conversations as the core bottleneck, even though audio and visual information complement each other.

Critical strengths
  1. The paper addresses an interesting and meaningful problem of translating collaborative audio-visual discussions into actionable workspace edits.

  2. Omni-CoWork covers diverse artifact types and evaluates both required edits and preservation constraints.

Critical weaknesses
  1. The paper lacks a systematic error analysis separating failures in decision extraction, artifact localization, and edit execution. As a result, it remains unclear which stage contributes most to the low task success rates.

  2. The proposed Omni-CoWork skill is evaluated on only two agents, with relatively small performance gains. Without comparisons against simpler prompting strategies or individual skill components, it is unclear whether the improvements come from the proposed grounded collective reasoning framework or general task-specific guidance.

Questions for authors
  1. Could the authors provide a stage-wise failure analysis distinguishing errors in decision extraction, artifact localization, and edit execution, to identify which stage contributes most to task failures?

  2. Could the authors conduct ablation studies on the INTENT, GROUND, and REALIZE components of the Omni-CoWork skill, and compare it with simpler prompting strategies to better understand the sources of its performance gains?

4 comments
AuthorsOpenReview ↗

Response to W1 & Q2

We thank the reviewer for the thoughtful feedback and constructive questions, which helped us strengthen the analysis of agent failure modes and clarify the contribution of each Omni-CoWork skill component. We address each concern point by point below.

W1 & Q1: Stage-Wise Failure Analysis

W1: The paper lacks a systematic error analysis separating failures in decision extraction, artifact localization, and edit execution. As a result, it remains unclear which stage contributes most to the low task success rates.

Q1: Could the authors provide a stage-wise failure analysis distinguishing errors in decision extraction, artifact localization, and edit execution, to identify which stage contributes most to task failures?

We thank the reviewer for requesting a systematic analysis of decision extraction, artifact localization, and edit execution. Our additional failure-mode attribution study, now included in Appendix C.6 of the revised manuscript, identifies the first erroneous step in each failed trajectory and distinguishes perception errors, localization errors, execution errors, and turn-budget exhaustion. Table 13 defines the taxonomy, and Table 14 reports the category and subtype results.

Taxonomy and attribution protocol. We first sampled 50 failed tasks per configuration for manual analysis, covering the seven agents and the Opus 4.8 + Gold Instructions reference. This informed a taxonomy of four categories: perception errors, localization errors, execution errors, and turn-budget exhaustion. The first three contain 11 subtypes, while turn-budget exhaustion is not subdivided, giving 12 assignable labels in total. We then used the following pipeline to analyze failed trajectories:

  1. Locate and explain the first error. GPT-6.1-sol compares the gold instructions with the agent's actual execution trajectory to identify the first erroneous step and explain why it is incorrect.
  2. Assign the failure mode. GPT-6.1-sol receives the trajectory segment surrounding the first erroneous step and the explanation produced in Step 1, together with the gold instructions and the final edited workspace, and assigns a failure mode.
  3. Manually annotate unresolved cases. Tasks that GPT-6.1-sol cannot classify are assigned failure labels by human annotators.

Each failed task receives one primary failure label. All percentages below use the full set of 1,337 tasks (124 human-recorded and 1,213 synthetic) as the denominator. The four failure categories together account for the overall failure rate.

Agent configurationOverall failure rate (%)Perception (%)Localization (%)Execution (%)Turn-budget exhaustion (%)
Claude Opus 4.882.428.622.426.55.0
DeepSeek V4 Pro83.228.920.715.618.0
Grok 4.684.423.613.620.027.1
Qwen3.7 Max92.630.126.035.80.7
Kimi K391.018.813.316.042.9
Gemini 3.8 Flash77.621.623.819.312.9
Seed 2.1 Pro80.623.313.320.723.2
Opus 4.8 + Gold Instructions42.87.26.529.10.0

Findings from the attribution analysis.

  1. Perception is the most prominent bottleneck overall. Across the seven standard agents, perception accounts for 29.6% of failed runs, making it the largest individual failure category, followed by execution (26.0%), localization (22.5%), and turn-budget exhaustion (21.9%). The two Omni agents, Gemini and Seed, have perception error rates of 21.6% and 23.3% of all tasks, respectively. Their mean rate is 22.5%, compared with 26.0% for the five LLM-based agents, consistent with the value of native audiovisual perception for recovering task information.

  2. Failures differ across agents, and perception difficulties can also manifest as budget exhaustion. Localization errors are the largest category for Gemini, execution errors for Qwen, and turn-budget exhaustion for Kimi and Grok. Kimi has the highest budget-exhaustion rate, affecting 42.9% of all tasks. Further inspection shows that 66.9% of Kimi's budget-exhaustion failures involve repeatedly calling the perception model until the turn budget is exhausted. This suggests that difficulties in acquiring and using perceptual evidence can also appear as prolonged perception loops, reinforcing the importance of perception beyond failures directly assigned to that category.

  3. Gold instructions reduce perception and localization errors, while execution remains challenging. For Opus, the perception error rate drops from 28.6% to 7.2%, and the localization error rate from 22.4% to 6.5%, measured over all 1,337 tasks. Execution errors then account for 68.0% of the remaining failures. These results support the benefit of explicit requirements and target locations while highlighting the need for reliable edit execution. Since the analysis assigns each trajectory its first error, the larger share of execution errors describes the remaining failure composition and does not establish that gold instructions impair execution.

AuthorsOpenReview ↗

Response to W2 & Q2

W2 & Q2: Skill Component Ablations

W2: The proposed Omni-CoWork skill is evaluated on only two agents, with relatively small performance gains. Without comparisons against simpler prompting strategies or individual skill components, it is unclear whether the improvements come from the proposed grounded collective reasoning framework or general task-specific guidance.

Q2: Could the authors conduct ablation studies on the INTENT, GROUND, and REALIZE components of the Omni-CoWork skill, and compare it with simpler prompting strategies to better understand the sources of its performance gains?

We thank the reviewer for suggesting component ablations and comparisons with simpler prompting. We would like to clarify that the original manuscript already reports two complementary results:

  1. Simple-prompt baselines across seven agents. The seven agents in Table 3 use the standard task prompts in Appendix D.1, which ask agents to follow the meeting's decisions and preserve unrelated content, without the Omni-CoWork skill. These results serve as our simple-prompt baselines.
  2. Full-skill improvements on Gemini and Seed. Section 5.5 and Figure 6 compare these baselines with the full Omni-CoWork skill. Task success improves by 4.1 and 1.6 percentage points on human-recorded tasks and by 3.6 and 2.4 points on synthetic tasks for Gemini and Seed, respectively.

To further examine the generality and sources of these gains, we added two sets of rebuttal experiments. All scores below are task success percentages.

1. Extending the skill evaluation to all seven agents.

We additionally evaluated the full skill on all five LLM-based agents, extending coverage from two Omni agents to all seven agents in Table 3. The table combines these new results with the original Gemini and Seed results:

AgentHuman-recorded: standardHuman-recorded: + skillSynthetic: standardSynthetic: + skill
Opus 4.80.03.219.420.5
DeepSeek V4 Pro4.05.718.122.0
Grok 4.64.04.816.823.5
Qwen3.7 Max0.00.08.212.9
Kimi K31.62.49.713.9
Gemini 3.8 Flash4.08.124.327.9
Seed 2.1 Pro1.63.221.323.7

The skill improves synthetic-task success for all seven agents and human-recorded success for six, while Qwen remains at 0.0% on human-recorded tasks. These results show that the gains extend beyond the original two Omni agents to LLM-based agents.

2. Ablating INTENT, GROUND, and REALIZE.

Appendix C.7 and Table 15 of the revised manuscript now report the Gemini and Seed component ablations, removing INTENT, GROUND, and REALIZE individually from the full skill. The full-skill results serve as the reference for comparison.

Prompt configurationGemini: Human-recordedGemini: SyntheticSeed: Human-recordedSeed: Synthetic
Full Omni-CoWork skill8.127.93.223.7
Skill without INTENT2.420.81.620.5
Skill without GROUND2.421.42.421.2
Skill without REALIZE3.222.61.620.2

Removing any component lowers task success for both Gemini and Seed on both splits relative to the full skill. For Gemini, removing INTENT causes the largest drop on synthetic tasks (7.1 percentage points) and ties with GROUND for the largest drop on human-recorded tasks (5.7 points). For Seed, removing REALIZE causes the largest drop on synthetic tasks (3.5 points) and ties with INTENT for the largest drop on human-recorded tasks (1.6 points). These model-dependent differences highlight the importance of both intent recovery and edit verification. Together with the gains across four agents, the results support the coordinated use of INTENT, GROUND, and REALIZE.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer kzdM,

Thank you again for your time and thoughtful review. As the ICLR rebuttal period is drawing to a close, we would like to ensure that our responses have adequately addressed your concerns. If you have any further questions or concerns, we would be happy to discuss and address them before the discussion ends.

Thanks for the authors' efforts and response. It addresses most of my concerns, and I will keep my score

Reviewer_QNPaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Omni-CoWork, a benchmark of 124 human-recorded tasks and 1213 synthetic tasks, for evaluating whether agents can infer collective intent from audiovisual discussions and translate it into verifiable workspace edits across a range of artifacts. Experiments show only 4% task success on human-recorded tasks and 24% on synthetic tasks. Subsequent analyses identify grounded reasoning—connecting evolving collective intent to specific workspace artifacts—as a key bottleneck in performance.

Critical strengths
  • Ambitious, important task: The paper focuses on recovering collective intent from multimodal collaboration and translating it into action.
  • Broad coverage: The benchmark contains many artifact and operation types, making the targeted capability more broad than other domain-specific benchmarks.
  • Strong empirical experiments: The experiments are extensive, including controlled ablations with gold instructions, different audiovisual representations and modalities, workspace grounding, and interleaved perception/execution to identify where performance breaks down.
Critical weaknesses
  • Limited ecological validity: Although the benchmark covers diverse artifacts, all scenarios and discussions are synthetically constructed. Thus it is unclear how well the scripted interactions capture the complexity of natural collaborative work.
  • Many capabilities are bundled together: Success requires audiovisual perception, tracking accepted/rejected decisions, intent inference, reference resolution, artifact grounding, planning, editing, and verification. The paper has ablation experiments that help to disentangle these factors, but a low end-to-end score still cannot be attributed to a single capability.
  • Limited scale of human-recorded tasks: The overall dataset is reasonably sized at 1337 tasks, but the more realistic human-recorded split contains only 124 tasks. This means conclusions about real audiovisual interaction are supported by a relatively small test set.
Questions for authors

See comments above.

4 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the thoughtful feedback and constructive questions, which helped us strengthen the evidence for task realism, deepen our failure analysis, and clarify the contribution of the human-recorded split. We address each concern point by point below.

W1: Ecological Validity of Synthetically Constructed Scenarios and Discussions

W1: Limited ecological validity: Although the benchmark covers diverse artifacts, all scenarios and discussions are synthetically constructed. Thus it is unclear how well the scripted interactions capture the complexity of natural collaborative work.

We thank the reviewer for raising this concern. We would like to clarify how our tasks reflect practical collaboration through evolving group decisions, multimodal references, and grounding in editable artifacts. We further support their realism with an additional human evaluation conducted during the rebuttal period.

Collaborative features represented in the benchmark.

  1. Evolving decisions across participants and subtasks. Accepted proposals, rejections, withdrawals, and non-task dialogue are interleaved, including multi-round negotiation and reversals. The on-screen workspace views also switch as the meeting progresses (§3.2.2 and Appendices A.4, B.2, and B.5). Human-recorded meetings contain 3–6 subtasks and 3–6 speaker roles, averaging 27.2 discussion items over 17.2 minutes. These interleaved intentions and changing visual contexts require agents to continuously track evolving decisions and ground them in the workspace, capturing a key challenge of collaborative work.
  2. Implicit intent and multimodal references. Requirements range from direct requests to problem descriptions and comments implying an edit. Expressions such as “this cell” or “this part” require combining spoken requirements with visual pointing or selection; cross-file requirements additionally require consulting the relevant workspace document (Appendices A.4 and B.3).
  3. Grounding in editable artifacts and real applications. Discussions refer to concrete objects in the workspace, including cells, text blocks, slide elements, and media intervals. In the human-recorded split, operators follow the scripts while interacting with real applications through selection, pointing, scrolling, and media playback. Operators can calibrate visual targets before recording, with corrections propagated to the task specification, reference workspace, and verifier (Appendix B.5).

Additional human realism assessment.

During the rebuttal period, we invited 15 PhD students to assess all 124 human-recorded tasks and 120 sampled synthetic tasks (approximately 10% of the synthetic split). This assessment, reported in Appendix B.7 and Table 9 of the revised manuscript, evaluates the naturalness and contextual plausibility of dialogue, participant roles, decision dynamics, and audiovisual coordination, including checks for implausible interactions or stereotyped roles.

Realism scoreHuman-recorded (124 tasks)Synthetic (120 sampled tasks)
2: Realistic98.4%92.5%
1: Broadly realistic, with minor issues1.6%7.5%
0: Clearly unrealistic or substantially distorted0.0%0.0%

These construction choices and human realism assessments support the benchmark's ability to capture key complexities of collaborative work, including evolving decisions, multimodal references, and grounding in editable artifacts.

AuthorsOpenReview ↗

Response to W2

W2: Disentangling Capabilities and Attributing End-to-End Failures

W2: Many capabilities are bundled together: Success requires audiovisual perception, tracking accepted/rejected decisions, intent inference, reference resolution, artifact grounding, planning, editing, and verification. The paper has ablation experiments that help to disentangle these factors, but a low end-to-end score still cannot be attributed to a single capability.

We thank the reviewer for highlighting the need to distinguish the capabilities underlying task failures. Our additional failure-mode attribution analysis, now included in Appendix C.6 of the revised manuscript, identifies where unsuccessful trajectories first go wrong. Table 13 defines the taxonomy, and Table 14 reports the category and subtype results.

Taxonomy and attribution protocol. We first sampled 50 failed tasks per configuration for manual analysis, covering the seven agents and the Opus 4.8 + Gold Instructions reference. This informed a taxonomy of four categories: perception errors, localization errors, execution errors, and turn-budget exhaustion. The first three contain 11 subtypes, while turn-budget exhaustion is not subdivided, giving 12 assignable labels in total. We then used the following pipeline to analyze failed trajectories:

  1. Locate and explain the first error. GPT-6.1-sol compares the gold instructions with the agent's actual execution trajectory to identify the first erroneous step and explain why it is incorrect.
  2. Assign the failure mode. GPT-6.1-sol receives the trajectory segment surrounding the first erroneous step and the explanation produced in Step 1, together with the gold instructions and the final edited workspace, and assigns a failure mode.
  3. Manually annotate unresolved cases. Tasks that GPT-6.1-sol cannot classify are assigned failure labels by human annotators.

Each failed task receives one primary failure label. All percentages below use the full set of 1,337 tasks (124 human-recorded and 1,213 synthetic) as the denominator. The four failure categories together account for the overall failure rate.

Agent configurationOverall failure rate (%)Perception (%)Localization (%)Execution (%)Turn-budget exhaustion (%)
Claude Opus 4.882.428.622.426.55.0
DeepSeek V4 Pro83.228.920.715.618.0
Grok 4.684.423.613.620.027.1
Qwen3.7 Max92.630.126.035.80.7
Kimi K391.018.813.316.042.9
Gemini 3.8 Flash77.621.623.819.312.9
Seed 2.1 Pro80.623.313.320.723.2
Opus 4.8 + Gold Instructions42.87.26.529.10.0

Findings from the attribution analysis.

  1. Perception is the most prominent bottleneck overall. Across the seven standard agents, perception is the largest individual failure category, accounting for 29.6% of failed runs. The two Omni agents, Gemini and Seed, have perception error rates of 21.6% and 23.3% of all tasks, respectively. Their mean rate is 22.5%, compared with 26.0% for the five LLM-based agents, consistent with the value of native audiovisual perception for recovering task information.

  2. Failures differ across agents, and perception difficulties can also manifest as budget exhaustion. Localization errors are the largest category for Gemini, execution errors for Qwen, and turn-budget exhaustion for Kimi and Grok. Kimi has the highest budget-exhaustion rate, affecting 42.9% of all tasks. Further inspection shows that 66.9% of Kimi's budget-exhaustion failures involve repeatedly calling the perception model until the turn budget is exhausted. This suggests that difficulties in acquiring and using perceptual evidence can also appear as prolonged perception loops, reinforcing the importance of perception beyond failures directly assigned to that category.

  3. Gold instructions reduce perception and localization errors, while execution remains challenging. For Opus, the perception error rate drops from 28.6% to 7.2%, and the localization error rate from 22.4% to 6.5%, measured over all 1,337 tasks. Execution errors then account for 68.0% of the remaining failures. These results support the benefit of explicit requirements and target locations while highlighting the need for reliable edit execution. Since the analysis assigns each trajectory its first error, the larger share of execution errors describes the remaining failure composition and does not establish that gold instructions impair execution.

AuthorsOpenReview ↗

Response to W3

W3: Scale of the Human-Recorded Split and Scope of the Conclusions

Limited scale of human-recorded tasks: The overall dataset is reasonably sized at 1337 tasks, but the more realistic human-recorded split contains only 124 tasks. This means conclusions about real audiovisual interaction are supported by a relatively small test set.

We thank the reviewer and acknowledge that human-recorded tasks make up a smaller share of the benchmark. However, compared with synthetic tasks, they contain richer collaborative content and are more complex on average, involving longer meetings, more subtasks, and more decisions to recover and implement. Table 2 and Appendix A.4 highlight these differences:

ComparisonSyntheticHuman-recorded
Number of tasks1,213124
Task composition89.2% are single-focusAll contain 3–6 subtasks (mean 4.0)
Mean recording duration9.4 min17.2 min
Mean discussion items9.927.2
Mean accepted changes5.412.0
Mean items that should not trigger edits4.515.2

The human-recorded tasks thus involve longer discussions and more decisions to recover and distinguish from rejected proposals, withdrawals, and non-task dialogue. In aggregate, the 124 meetings comprise 35.6 hours of video and 3,370 discussion items, including 1,486 accepted changes. They also span 13 professional sectors and 44 fine-grained domains (Appendices A.2–A.3).

These characteristics provide substantial content depth and scenario diversity within the human-recorded split, supporting the study of collective intent recovery and workspace execution across extended meetings recorded in real applications. The markedly lower agent success rates on human-recorded tasks (Table 3) further highlight their complexity and the substantial room for future research.

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewer QNPa,

Thank you again for your time and thoughtful review. As the ICLR rebuttal period is drawing to a close, we would like to ensure that our responses have adequately addressed your concerns. If you have any further questions or concerns, we would be happy to discuss and address them before the discussion ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.