- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MMTB, it's a benchmark of 105 multimedia tasks performed in terminals. Tasks are derived from practitioner workflows and require agents to inspect media, act through terminal tools, and produce artifacts. The paper also introduces a harness called Terminus-MM. It gives terminal agents native image audio, and video access. Controlled modality ablations show that native media access improves success rate, while command-line conversion is still useful and would lead to different failure modes.
Critical strengths
-
MMTB constructs realistic terminal workflows that are becoming more common in recent agent developments. Grounding tasks in practitioner examples and evaluating the final workspace state are useful in practice, and they resemble how the harnesses are typically evaluated.
-
The controlled harness ablations isolate the value of native audio and video access, by keeping the agent loop and tasks fixed. The comparison with command-line conversion gives a useful comparison. It's interesting to see how native perception and tool use complement each other in harnesses.
Critical weaknesses
- The benchmark contains 105 highly heterogeneous tasks, which is okay, if considering other benchmarks like Terminal Bench. But the reported low success rates, a few tasks might flip and change the overall ranking, so the conclusions may be unstable. It would be helpful to check the trends like if each topic in the taxonomy has a general relative ranking against the others, and provide some explanations.
Questions for authors
-
Can you provide some details on how stable the evaluations are, and whether the main modality access conclusions hold in reach task rather than only in the final aggregated results?
-
How many final task verifiers were independently reviewed, and what fraction of failures were manually checked for false negatives caused? Wondering if the low success rate might be a result of overly strict judging rules.
4 comments
[Q2. Verifier review and false rejections.]
A blind audit covers the decisions of 39 of the 105 verifiers, 13.2% of the audited rejections were false, and relaxing the judging rules raises success without changing the comparison.
Verifier review. Eight judges who did not build the tasks graded 74 real agent outputs without seeing the verifier's decision, the model, or the harness (Table R5; protocol in Appendix B.13 of the revision). The verifier agrees with the judges as much as the judges agree with each other (Table R6).
Failures checked. Of the 848 failed runs in the audited set (seed-1 runs of Terminus-2, Terminus-KIRA, Terminus-IA, Terminus-IV, and Terminus-MM on the two Gemini backbones), 348 produced no output file and are correct rejections verified from the logs. Of the remaining 500, 38 are in the audit, and the verifier had wrongly rejected 5 of them (13.2%). Counting the 348 no-output runs together with the 340 rejected runs whose outputs could be rebuilt would lower the rate to 6.5%.
Table R5. Verifier decisions against blind human judgment on 74 agent outputs from 39 tasks.
| Verifier decision | Judged correct | Judged incorrect | Verifier error |
|---|---|---|---|
| Accepted (36) | 32 | 4 | 4 / 36 false acceptances (11.1%) |
| Rejected (38) | 5 | 33 | 5 / 38 false rejections (13.2%) |
Table R6. Agreement between the verifiers and the human judges, and among the judges, next to the same statistics reported for WebVoyager's automatic evaluator (He et al., 2024).
| Evaluator vs. human judges | Among human judges | |
|---|---|---|
| MMTB verifiers | 87.8% (65 / 74), Cohen's κ = 0.76 | Fleiss' κ = 0.72 |
| WebVoyager evaluator | 85.3%, Cohen's κ = 0.70 | Fleiss' κ = 0.70 |
Strict judging rules. With the pass threshold lowered to τ = 0.5 for every task, where half of the verifier score suffices, Terminus-MM reaches 28.3% binary success on Gemini-2.5-Flash and 47.0% on Gemini-3.1-Pro, against 20.0% and 34.0% at the original settings. At every setting, the four harnesses without native audio or video access stay below Terminus-MM, also in partial success, which uses no pass threshold (row 'no τ'). The four are Terminus-2, Terminus-KIRA, and the new controlled text-only and image-only baselines Terminus-T and Terminus-I (Table R7; single runs; Appendix B.12 of the revision). Table R7 also varies the overlap threshold θ for matching time intervals (9 tasks) and the tolerance δ for matching timestamps (13 tasks).
Table R7. Binary success (%) with one common pass threshold or matching tolerance in place of the task-specific one, one parameter at a time. Three-seed means for Terminus-2, Terminus-KIRA, and Terminus-MM; single runs for Terminus-T and Terminus-I. The row 'no τ' is partial success (%), which uses no pass threshold.
| Setting | Terminus-2 | Terminus-KIRA | Terminus-T | Terminus-I | Terminus-MM |
|---|---|---|---|---|---|
| Gemini-2.5-Flash | |||||
| original settings | 6.0 | 7.3 | 4.8 | 6.7 | 20.0 |
| τ = 0.5 | 12.7 | 12.7 | 7.6 | 13.3 | 28.3 |
| τ = 0.6 | 10.5 | 10.2 | 6.7 | 9.5 | 24.8 |
| τ = 0.7 | 7.3 | 7.0 | 5.7 | 8.6 | 21.9 |
| τ = 0.8 | 5.7 | 6.3 | 4.8 | 6.7 | 19.7 |
| τ = 0.9 | 4.1 | 5.7 | 3.8 | 5.7 | 16.2 |
| τ = 1.0 | 3.5 | 4.4 | 3.8 | 3.8 | 15.2 |
| no τ (partial success) | 13.0 | 12.8 | 9.3 | 11.9 | 27.5 |
| θ = 0.25 | 6.0 | 7.3 | 4.8 | 6.7 | 20.0 |
| θ = 0.5 | 6.0 | 7.3 | 4.8 | 6.7 | 20.0 |
| θ = 0.75 | 6.0 | 7.3 | 4.8 | 6.7 | 20.0 |
| δ = 0.5 s | 6.0 | 7.0 | 4.8 | 6.7 | 18.1 |
| δ = 1 s | 6.0 | 7.0 | 4.8 | 6.7 | 18.7 |
| δ = 2 s | 6.3 | 7.3 | 4.8 | 6.7 | 20.6 |
| Gemini-3.1-Pro | |||||
| original settings | 15.6 | 12.7 | 16.2 | 21.9 | 34.0 |
| τ = 0.5 | 19.0 | 17.1 | 21.9 | 26.7 | 47.0 |
| τ = 0.6 | 17.8 | 15.2 | 20.0 | 25.7 | 41.6 |
| τ = 0.7 | 15.9 | 13.7 | 16.2 | 21.9 | 37.8 |
| τ = 0.8 | 15.2 | 12.7 | 14.3 | 20.0 | 34.0 |
| τ = 0.9 | 14.6 | 9.8 | 12.4 | 17.1 | 29.2 |
| τ = 1.0 | 12.7 | 9.2 | 9.5 | 14.3 | 24.4 |
| no τ (partial success) | 18.8 | 16.2 | 19.8 | 25.9 | 45.8 |
| θ = 0.25 | 15.6 | 12.7 | 16.2 | 21.9 | 34.6 |
| θ = 0.5 | 15.6 | 12.7 | 16.2 | 21.9 | 34.0 |
| θ = 0.75 | 15.2 | 12.7 | 16.2 | 21.9 | 33.7 |
| δ = 0.5 s | 15.6 | 12.7 | 15.2 | 21.0 | 33.0 |
| δ = 1 s | 15.6 | 12.7 | 16.2 | 21.9 | 33.3 |
| δ = 2 s | 15.9 | 12.7 | 16.2 | 21.9 | 34.6 |
Reference: He et al., WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models, ACL 2024.
Thank you for providing further details. It's good to see that the overall trend remains consistent in the evaluation. Since there are reported false rejections "13.2% of the audited rejections were false", would these be corrected in the revision?
Thank you for the follow-up.
Yes, one of the five false rejections was a verifier error and is corrected in the revision: the reference answer did not match the task media, so we corrected it and re-scored the task's outputs. In the other four, the output does not meet a requirement stated in the task instructions, which the human judges did not catch, so the verifier's rejections were correct. A false rejection in the audit means that the human judges graded an output as correct while the verifier rejected it. Human judgments can also be mistaken, so we checked all five individually against the task instructions and media.
The corrected results are in Section 2, Tables 3, 5, 8, 11, 12, 25, and 27, Figures 7 and 8, and Appendices B.13 and B.15 of the revision. Only two agent runs and one human-study result change, so the paper's conclusions on the gain from native access are unchanged.
We thank the reviewer for the thoughtful comments, and we address them below.
[W1 + Q1. Stability across seeds, tasks, and categories.]
The modality-access conclusion is stable: with native audio and video access (Terminus-MM), binary and partial success are above those of Terminus-2 and Terminus-KIRA in every seed and, pooled over the backbones, in every meta-category, and per task more tasks improve than regress.
Seeds. Even the lowest seed with native audio and video access is above the highest seed without it on every backbone (Table R9). Seed-to-seed variation in binary success of these three harnesses is at most 3.3 percentage points (SD; Table 3).
Table R9. Binary / partial success (%) in each seed. Seeds of different harnesses are independent runs.
Categories. The gain in binary success is largest in Enterprise & Compliance and smallest in Performance & Coaching (Table R10). Access to audio and video matters most in Enterprise & Compliance; in Performance & Coaching the limit is most likely the fine-grained judgment the tasks require rather than access (Appendix B.8 of the revision).
Table R10. Binary / partial success (%) by meta-category, pooled over the three backbones (three-seed means).
Tasks. Per task (three-seed means), 39 tasks improve and 9 regress against Terminus-2 on Gemini-3.1-Pro in binary success (70 and 14 in partial success), and improvements outnumber regressions on the other two backbones and against Terminus-KIRA (Appendix B.10 of the revision, Tables 17 to 19).