- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Authors release a benchmark for measuring how agents pick up tasks in audio-visual discussions. Using the findings of this benchmark, the authors combine these into a skill used as context into an llm which when used consistently improves performance.
Critical strengths
The outcome of this paper summarized as a skill is definitely useful, and condensing this into something that can be given into context to an existing llm is more useful than finetuning these models. Extensive construction pipeline. A lot of room on the benchmark for new models to perform better on with tasks successes of max 4% on human-recorded tasks, and 24.3% on synthetic tasks.
Critical weaknesses
- The overall idea is useful, but I believe it fits maybe less in the academic setting of ICLR. The application is definitely interesting as an industrial application or software that can be used for collaborating with agents. It is mainly targeted for usage in workspace setting.
- A benchmark like this could benefit from having a human evaluation baseline. If the authors would assign these tasks to a human (say a human assistant) how good it would perform (Table 3). Can agents do better inference, acting and checking than humans on workspaces (without gold instructions).
- I'm missing a bit a conclusion on a solution, the authors merely pose issues that current models have still. How would this best be tackled? The authors write that both richer descriptions does not necessarily help a lot, and adding extra context into the prompt in form of a skill only helps a bit. I would like a bit more focus on what these effective methods potentially could be that would solve the gap between collective intent and workspace execution.
Questions for authors
- For Quality control (section 3.2) I'm missing a human check on the synthetic data. How do the authors know these are correct tasks and the synthetic tasks in the dataset do not contain any bias or hallucinations, and the outcome is correct?
- Can the authors explain a bit more why human-recorded task success is so much lower than synthetic task success? Could it be attributed also partly to the difference in the way tasks are defined, where human tasks have 3-6 subtasks with average 27.2 discussion items over an average 17.2 minutes, the synthetic videos do not necessarily have those statistics?
- Line 377: "Yet most tasks remain unsolved with joint input, motivating further investigation into whether richer audio-visual evidence can improve intent understanding". Could the authors do this further investigation during the rebuttal phase?
- One particular question I was wondering is also the refusal rate of the agents. I personally have noticed especially in healthcare and medicine that agents refuse applying actions, because of safety alignment / guardrails. Since the authors especially outline this anti-doping review task as concrete task example (Appendix A.1), I wonder whether the authors have seen if there are any tasks in this benchmark that agents would refuse to do? I believe this is important to mention in the paper.
9 comments
Response to W2
W2: Human Performance Baseline
W2: A benchmark like this could benefit from having a human evaluation baseline. If the authors would assign these tasks to a human (say a human assistant) how good it would perform (Table 3). Can agents do better inference, acting and checking than humans on workspaces (without gold instructions).
We thank the reviewer for suggesting a human performance baseline. We conducted an additional evaluation with 15 PhD students, covering all 124 human-recorded tasks and 120 sampled synthetic tasks (approximately 10% of the synthetic split), given the rebuttal time constraints.
Protocol (Appendix B.7). Participants received no gold instructions. They first watched the full meeting video without workspace access. They then had 10 minutes to inspect and edit the workspace and could revisit the video as needed during this phase. When the time limit was reached, participants stopped editing, and the workspace state at that point was used to compute task success, E-Rate, and P-Rate with the same executable checks as in Table 3. After the task-performance evaluation, the same participants also assessed task quality in terms of correctness, realism, and hallucination (see our response to Q1).
Results. The human baseline results below are now included in Table 3 of the revised manuscript, with asterisks marking the sampled synthetic scores. All scores are percentages.
| Split | Tasks | Task success | E-Rate | P-Rate |
|---|---|---|---|---|
| Human-recorded | 124 | 87.9 | 96.8 | 99.6 |
| Synthetic (sampled) | 120 | 92.5 | 97.5 | 99.5 |
On the full human-recorded split, humans exceed the gold-instruction reference in task success (87.9% vs. 61.3%) and E-Rate (96.8% vs. 94.0%), while matching its P-Rate (99.6%). The high E-Rate and P-Rate indicate that participants complete most required edits while rarely altering protected content.
This advantage may reflect humans’ ability to continuously integrate visual changes and speaker voices to track evolving intent, and to precisely revisit relevant video segments during editing. Under the same human evaluation protocol, success is lower on human-recorded tasks than on sampled synthetic tasks (87.9% vs. 92.5%), consistent with their greater complexity, more subtasks, and more pronounced shifts in intent.
Response to W3 & Q3
W3 & Q3: Effective Methods and Richer Audio-Visual Evidence
W3: I'm missing a bit a conclusion on a solution, the authors merely pose issues that current models have still. How would this best be tackled? The authors write that both richer descriptions does not necessarily help a lot, and adding extra context into the prompt in form of a skill only helps a bit. I would like a bit more focus on what these effective methods potentially could be that would solve the gap between collective intent and workspace execution.
Q3: Line 377: "Yet most tasks remain unsolved with joint input, motivating further investigation into whether richer audio-visual evidence can improve intent understanding". Could the authors do this further investigation during the rebuttal phase?
We thank the reviewer for requesting clearer methodological directions and further investigation of richer audiovisual evidence. In response, we conducted additional experiments during the rebuttal period to address two questions:
- How can richer audiovisual evidence be used concretely to improve intent understanding?
- How can the findings in Section 5 and the Omni-CoWork skill be extended into a more effective method for connecting collective intent with workspace execution?
Starting points from Section 5. For the design below, we label three relevant results as follows:
- Finding 1. Audio and vision are complementary, but joint input remains insufficient (§5.1).
- Finding 2. Workspace context improves intent recovery and grounding during planning (§5.3).
- Finding 3. Interleaved perception outperforms upfront perception under the same consultation budget (§5.4).
During rebuttal, we added Omni-CoWork skill component ablations (Appendix C.7, Table 15) and failure-mode attribution (Appendix C.6, Tables 13–14) to the revised manuscript:
- Component ablations. Removing any component lowers task success for Gemini and Seed on both splits, supporting the importance of intent recovery and edit verification. Removing INTENT causes the largest synthetic-task drop for Gemini.
- Failure attribution. Perception is the largest overall failure category, and repeated perception calls can also exhaust the execution budget, reinforcing perception as the main bottleneck.
These findings highlight perception as the main bottleneck, with Omni agents showing lower mean perception-error rates than LLM-based agents. Meanwhile, Opus outperforms Gemini with gold instructions on the analysis tasks (§5.2). These complementary strengths motivate a division of labor between an Omni perceiver and an LLM executor.
Extending Omni-CoWork into a two-agent system. We expand INTENT into an Omni agent's iterative perception and work-order construction, while the executor retains GROUND and REALIZE. This extension is reported in §5.6 of the revised manuscript, with implementation details in Appendix C.8:
- Agentic perception and work-order construction. The Omni agent integrates visual and spoken evidence while inspecting the meeting video in segments (based on Finding 1), jointly examining the relevant workspace content (based on Finding 2). It then uses targeted replay, high-frame-rate playback, and checking tools to refine its understanding. It organizes the richer audiovisual evidence and recovered decisions into an actionable work order for the execution agent.
- Execution with targeted follow-up. The non-Omni execution agent uses the work order to locate targets, apply edits, and verify the results. When questions arise during execution, it can ask the Omni agent for clarification (based on Finding 3). The Omni agent revisits the relevant evidence and supplies targeted perceptual information to support continued execution.
Evaluation of the two-agent extension. We paired Gemini 3.8 Flash as the omni agent with the five LLM executors below. The table reports task success (%) for the Table 3 baselines, original skill, and two-agent extension, with standalone Gemini and Seed for comparison. '\' denotes inapplicable configurations.
| Agent | Human-recorded: original setup | Human-recorded: + original skill | Human-recorded: two-agent setup | Synthetic: original setup | Synthetic: + original skill | Synthetic: two-agent setup |
|---|---|---|---|---|---|---|
| Opus 4.8 | 0.0 | 3.2 | 8.1 | 19.4 | 20.5 | 30.8 |
| DeepSeek V4 Pro | 4.0 | 5.7 | 8.9 | 18.1 | 22.0 | 28.9 |
| Grok 4.6 | 4.0 | 4.8 | 5.7 | 16.8 | 23.5 | 28.4 |
| Qwen3.7 Max | 0.0 | 0.0 | 1.6 | 8.2 | 12.9 | 21.4 |
| Kimi K3 | 1.6 | 2.4 | 4.8 | 9.7 | 13.9 | 28.9 |
| Gemini 3.8 Flash | 4.0 | 8.1 | \ | 24.3 | 27.9 | \ |
| Seed 2.1 Pro | 1.6 | 3.2 | \ | 21.3 | 23.7 | \ |
These results show that the two-agent system further improves task success, pointing to a promising research direction: designing more effective workflows that coordinate perception and execution agents. Given the time constraints, we have conducted only a preliminary exploration and will continue investigating more efficient coordination methods after the rebuttal period.
Response to W1
We thank the reviewer for the thoughtful feedback and constructive questions, which helped us clarify the paper’s academic contribution, strengthen the evaluation with human baselines and quality checks, and further develop Omni-CoWork through a two-agent extension. We address each concern point by point below.
W1: Alignment with ICLR’s Academic Scope
We thank the reviewer for recognizing the usefulness of this work. We would like to clarify that the workspace serves as an experimental environment; the paper’s academic contribution lies in formulating the research problem of recovering and correctly implementing final collective intent from multi-person multimodal discussions, and in evaluating agents’ ability to address it, which aligns with ICLR’s academic scope. Specifically, this contribution is reflected in three aspects: new research question, the evaluation design, and the diagnostic findings.
Related ICLR papers provide useful precedents: VideoWebArena (ICLR 2025) similarly evaluates how agents translate video understanding into task execution, while Collaborative Gym (ICLR 2026) shares our focus on connecting human communication with agent execution in shared task environments. These parallels support Omni-CoWork’s relevance to ICLR as a benchmark for studying agent capabilities in practical environments.
[1] Lawrence Keunho Jang, Yinheng Li, Dan Zhao, Charles Ding, Justin Lin, Paul Pu Liang, Rogerio Bonatti, and Kazuhito Koishida. VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[2] Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration. In The Fourteenth International Conference on Learning Representations (ICLR), 2026.