- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a benchmark for the quality of agent deliverables, measuring the agent's ability to complete long-process, high-precision tasks end-to-end, and uses objective standards for scoring.
Critical strengths
The paper moves beyond conventional question-answering benchmarks by requiring agents to produce structured records, quantitative annotations, and editable artifacts from multimodal evidence. The 45 tasks spanning eight families provide a useful setting for evaluating end-to-end task completion rather than isolated perception or reasoning abilities.
Critical weaknesses
The main problem with this paper is that a sufficient logical loop has not yet been formed between the proposal of the benchmark, the construction of the task, and the subsequent experimental verification.
- The benchmark fails to provide a reasonable metric for the problems posed. The introduction emphasizes that existing multimodal benchmarks primarily focus on perception and reasoning abilities, failing to adequately assess an agent's capacity for continuous evidence gathering, cross-modal reasoning, and sophisticated tool usage. However, this paper does not adequately demonstrate whether the constructed task can systematically measure these abilities, nor does it prove that these abilities are the primary factors contributing to the task's difficulty.
- The experiment failed to systematically explain the source of the task difficulty. The experimental design and related analysis in this paper focus on the agent's performance itself, failing to demonstrate the key aspects of the benchmark's assessment of the agent's existing capabilities, nor does it analyze, based on the benchmark performance, "which aspects of the agent's existing capabilities are insufficient and how this leads to the agent's inadequate delivery capabilities that this paper seeks to investigate."
Questions for authors
- Why is the threshold for Ablation set to 0.15, while the target threshold for Development Agent is 0.10? The article uses a Development Agent score below 0.10 as the difficulty calibration target for the task development phase. However, I observed that the threshold for the Ablation part is 0.15. Why does the agent score improve under more lenient conditions after ablating one of the modalities?
- Why are the threshold used during review and validation differs significantly from the performance in subsequent tests? The article mentions using a state-of-the-art strong model when employing the Development Agent. What exactly does this mean? My main question is that you limited its score to no more than 0.1 when building the benchmark, yet in subsequent tests, some models still scored significantly higher than 0.1? Were the models in subsequent tests stronger than those used when building the benchmark?
- What is the relationship between agent behavior analysis and the core research question of the paper? Figures 6 and 7 analyze behaviors such as Operation Grouping, Saved Scripts, and Sub-agent Delegation. However, these experiments primarily demonstrate the differences in execution strategies among different agents. Could the authors further explain the direct significance of these analyses for validating the multimodal evidence integration and professional work delivery capabilities emphasized by the Benchmark?
We thank the reviewer for these questions. They go to the core of the paper: whether the benchmark measures what it says, and why agents fail. We agree the paper did not close this loop, and we try to close it here.
On whether the tasks measure the stated abilities: "This paper does not adequately demonstrate whether the constructed task can systematically measure these abilities, nor does it prove that these abilities are the primary factors contributing to the task's difficulty."
Response: We answer in two steps: which tasks need each ability, and what happens to the score when that ability cannot be used.
When any one of the three is taken away, the score falls from 0.48 to below 0.09. So each ability is needed, and together they account for almost all of the score.
On the source of difficulty: "Which aspects of the agent's existing capabilities are insufficient and how this leads to the agent's inadequate delivery capabilities."
Response: With everything available, the best agents still score only about 0.28. The paper's data show where the score is lost, step by step:
So the weak point of the best agents is precise measurement, not finding and not writing the file. The weaker agents fail earlier: they more often hand in nothing usable or miss most items. The new runs agree: when an agent is asked to answer without the video, it gives the right number of items with the wrong times.
On the thresholds: "Why does the agent score improve under more lenient conditions after ablating one of the modalities?"
Response: It does not, and we are sorry the figure suggested this. Scores with a modality removed are far lower than with the full input (0.084 against 0.478, see the table above). The two numbers in Figure 3 are not pass marks for agents. 0.10 is a recommended target for the full input while building a task; the hard requirement is that the task is long. 0.15 is not a condition for accepting a task; that sentence and figure entry were written in error and are corrected.
On scores above the development target: "Were the models in subsequent tests stronger than those used when building the benchmark?"
Response: Yes. Tasks were built and tested with the agents available at the time, through Claude Code, Codex and Antigravity. The agents in the paper are a newer generation. The target is also a recommendation, not a limit. Even so, 60% of all attempts in the paper still score below 0.10.
On the behaviour analysis: "Could the authors further explain the direct significance of these analyses for validating the multimodal evidence integration and professional work delivery capabilities?"
Response: Each behaviour in Figures 6 and 7 is one of the three abilities in action.
The last row is the clearest link from behaviour to score, and it explains why the two leading agents end up tied.
We hope these answers address the reviewer's concerns. We will reorganise Section 3 around this chain of reasoning, and we welcome further questions.