acceptodds
Under review as a conference paper at ICLR 2027

MMTB: Evaluating Terminal Agents on Multimedia-File Tasks

21 views

Abstract

Terminals provide a powerful interface for AI agents by exposing diverse tools for automating complex workflows, yet existing terminal-agent benchmarks largely focus on tasks grounded in text, code, and structured files. However, many real-world workflows require practitioners to work directly with audio and video files. Working with such multimedia files calls for terminal agents not only to understand multimedia content, but also to convert auditory and visual evidence across related files into appropriate actions. To evaluate terminal agents on multimedia-file tasks, we introduce MultiMedia-TerminalBench (MMTB), a benchmark of 105 tasks across 5 meta-categories where terminal agents directly operate with audio and video files. Alongside MMTB, we propose Terminus-MM, a multimedia harness that extends Terminus-KIRA with audio and video perception for terminal agents. Together, MMTB and Terminus-MM support a controlled study of multimedia terminal agents, revealing how different forms of multimedia access shape task outcomes and determine which evidence agents rely on to construct executable terminal workflows. MMTB media and metadata are released at https://huggingface.co/datasets/mmtb-anonymous/mmtb-media.

Primary area:
datasets and benchmarks
Keywords:
benchmark, terminal agents, multimedia, audio-visual perception, Multi-modal Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DtefOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MMTB, it's a benchmark of 105 multimedia tasks performed in terminals. Tasks are derived from practitioner workflows and require agents to inspect media, act through terminal tools, and produce artifacts. The paper also introduces a harness called Terminus-MM. It gives terminal agents native image audio, and video access. Controlled modality ablations show that native media access improves success rate, while command-line conversion is still useful and would lead to different failure modes.

Critical strengths
  1. MMTB constructs realistic terminal workflows that are becoming more common in recent agent developments. Grounding tasks in practitioner examples and evaluating the final workspace state are useful in practice, and they resemble how the harnesses are typically evaluated.

  2. The controlled harness ablations isolate the value of native audio and video access, by keeping the agent loop and tasks fixed. The comparison with command-line conversion gives a useful comparison. It's interesting to see how native perception and tool use complement each other in harnesses.

Critical weaknesses
  1. The benchmark contains 105 highly heterogeneous tasks, which is okay, if considering other benchmarks like Terminal Bench. But the reported low success rates, a few tasks might flip and change the overall ranking, so the conclusions may be unstable. It would be helpful to check the trends like if each topic in the taxonomy has a general relative ranking against the others, and provide some explanations.
Questions for authors
  1. Can you provide some details on how stable the evaluations are, and whether the main modality access conclusions hold in reach task rather than only in the final aggregated results?

  2. How many final task verifiers were independently reviewed, and what fraction of failures were manually checked for false negatives caused? Wondering if the low success rate might be a result of overly strict judging rules.

4 comments

We thank the reviewer for the thoughtful comments, and we address them below.

[W1 + Q1. Stability across seeds, tasks, and categories.]

The modality-access conclusion is stable: with native audio and video access (Terminus-MM), binary and partial success are above those of Terminus-2 and Terminus-KIRA in every seed and, pooled over the backbones, in every meta-category, and per task more tasks improve than regress.

Seeds. Even the lowest seed with native audio and video access is above the highest seed without it on every backbone (Table R9). Seed-to-seed variation in binary success of these three harnesses is at most 3.3 percentage points (SD; Table 3).

Table R9. Binary / partial success (%) in each seed. Seeds of different harnesses are independent runs.

Backbone, seedTerminus-2Terminus-KIRATerminus-MM
Gemini-3.1-Pro, 112.4 / 16.210.5 / 15.937.1 / 46.9
Gemini-3.1-Pro, 215.2 / 18.316.2 / 18.534.3 / 45.8
Gemini-3.1-Pro, 319.0 / 21.911.4 / 14.030.5 / 44.6
Gemini-2.5-Flash, 16.7 / 13.66.7 / 12.922.9 / 30.5
Gemini-2.5-Flash, 25.7 / 12.96.7 / 12.819.0 / 24.5
Gemini-2.5-Flash, 35.7 / 12.48.6 / 12.718.1 / 27.6
Qwen3.5-Omni-Flash, 12.9 / 6.63.8 / 6.19.5 / 12.2
Qwen3.5-Omni-Flash, 21.9 / 7.83.8 / 3.78.6 / 11.6
Qwen3.5-Omni-Flash, 32.9 / 4.61.9 / 4.68.6 / 13.5

Categories. The gain in binary success is largest in Enterprise & Compliance and smallest in Performance & Coaching (Table R10). Access to audio and video matters most in Enterprise & Compliance; in Performance & Coaching the limit is most likely the fine-grained judgment the tasks require rather than access (Appendix B.8 of the revision).

Table R10. Binary / partial success (%) by meta-category, pooled over the three backbones (three-seed means).

Meta-category (tasks)Terminus-2Terminus-KIRATerminus-MM
Media Production (40)7.8 / 13.86.9 / 11.716.4 / 25.8
Performance & Coaching (23)7.7 / 12.16.3 / 10.412.1 / 17.2
Enterprise & Compliance (23)7.7 / 10.212.1 / 13.933.8 / 40.8
Personal & Education (13)8.5 / 12.97.7 / 9.428.2 / 32.7
Operations & Research (6)11.1 / 16.11.9 / 5.520.4 / 34.6

Tasks. Per task (three-seed means), 39 tasks improve and 9 regress against Terminus-2 on Gemini-3.1-Pro in binary success (70 and 14 in partial success), and improvements outnumber regressions on the other two backbones and against Terminus-KIRA (Appendix B.10 of the revision, Tables 17 to 19).

[Q2. Verifier review and false rejections.]

A blind audit covers the decisions of 39 of the 105 verifiers, 13.2% of the audited rejections were false, and relaxing the judging rules raises success without changing the comparison.

Verifier review. Eight judges who did not build the tasks graded 74 real agent outputs without seeing the verifier's decision, the model, or the harness (Table R5; protocol in Appendix B.13 of the revision). The verifier agrees with the judges as much as the judges agree with each other (Table R6).

Failures checked. Of the 848 failed runs in the audited set (seed-1 runs of Terminus-2, Terminus-KIRA, Terminus-IA, Terminus-IV, and Terminus-MM on the two Gemini backbones), 348 produced no output file and are correct rejections verified from the logs. Of the remaining 500, 38 are in the audit, and the verifier had wrongly rejected 5 of them (13.2%). Counting the 348 no-output runs together with the 340 rejected runs whose outputs could be rebuilt would lower the rate to 6.5%.

Table R5. Verifier decisions against blind human judgment on 74 agent outputs from 39 tasks.

Verifier decisionJudged correctJudged incorrectVerifier error
Accepted (36)3244 / 36 false acceptances (11.1%)
Rejected (38)5335 / 38 false rejections (13.2%)

Table R6. Agreement between the verifiers and the human judges, and among the judges, next to the same statistics reported for WebVoyager's automatic evaluator (He et al., 2024).

Evaluator vs. human judgesAmong human judges
MMTB verifiers87.8% (65 / 74), Cohen's κ = 0.76Fleiss' κ = 0.72
WebVoyager evaluator85.3%, Cohen's κ = 0.70Fleiss' κ = 0.70

Strict judging rules. With the pass threshold lowered to τ = 0.5 for every task, where half of the verifier score suffices, Terminus-MM reaches 28.3% binary success on Gemini-2.5-Flash and 47.0% on Gemini-3.1-Pro, against 20.0% and 34.0% at the original settings. At every setting, the four harnesses without native audio or video access stay below Terminus-MM, also in partial success, which uses no pass threshold (row 'no τ'). The four are Terminus-2, Terminus-KIRA, and the new controlled text-only and image-only baselines Terminus-T and Terminus-I (Table R7; single runs; Appendix B.12 of the revision). Table R7 also varies the overlap threshold θ for matching time intervals (9 tasks) and the tolerance δ for matching timestamps (13 tasks).

Table R7. Binary success (%) with one common pass threshold or matching tolerance in place of the task-specific one, one parameter at a time. Three-seed means for Terminus-2, Terminus-KIRA, and Terminus-MM; single runs for Terminus-T and Terminus-I. The row 'no τ' is partial success (%), which uses no pass threshold.

SettingTerminus-2Terminus-KIRATerminus-TTerminus-ITerminus-MM
Gemini-2.5-Flash
original settings6.07.34.86.720.0
τ = 0.512.712.77.613.328.3
τ = 0.610.510.26.79.524.8
τ = 0.77.37.05.78.621.9
τ = 0.85.76.34.86.719.7
τ = 0.94.15.73.85.716.2
τ = 1.03.54.43.83.815.2
no τ (partial success)13.012.89.311.927.5
θ = 0.256.07.34.86.720.0
θ = 0.56.07.34.86.720.0
θ = 0.756.07.34.86.720.0
δ = 0.5 s6.07.04.86.718.1
δ = 1 s6.07.04.86.718.7
δ = 2 s6.37.34.86.720.6
Gemini-3.1-Pro
original settings15.612.716.221.934.0
τ = 0.519.017.121.926.747.0
τ = 0.617.815.220.025.741.6
τ = 0.715.913.716.221.937.8
τ = 0.815.212.714.320.034.0
τ = 0.914.69.812.417.129.2
τ = 1.012.79.29.514.324.4
no τ (partial success)18.816.219.825.945.8
θ = 0.2515.612.716.221.934.6
θ = 0.515.612.716.221.934.0
θ = 0.7515.212.716.221.933.7
δ = 0.5 s15.612.715.221.033.0
δ = 1 s15.612.716.221.933.3
δ = 2 s15.912.716.221.934.6

Reference: He et al., WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models, ACL 2024.

Thank you for providing further details. It's good to see that the overall trend remains consistent in the evaluation. Since there are reported false rejections "13.2% of the audited rejections were false", would these be corrected in the revision?

Thank you for the follow-up.

Yes, one of the five false rejections was a verifier error and is corrected in the revision: the reference answer did not match the task media, so we corrected it and re-scored the task's outputs. In the other four, the output does not meet a requirement stated in the task instructions, which the human judges did not catch, so the verifier's rejections were correct. A false rejection in the audit means that the human judges graded an output as correct while the verifier rejected it. Human judgments can also be mistaken, so we checked all five individually against the task instructions and media.

The corrected results are in Section 2, Tables 3, 5, 8, 11, 12, 25, and 27, Figures 7 and 8, and Appendices B.13 and B.15 of the revision. Only two agent runs and one human-study result change, so the paper's conclusions on the gain from native access are unchanged.

Reviewer_se1vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MultiMedia-TerminalBench (MMTB), a benchmark of 105 tasks spanning five meta-categories and 16 workflow categories. Agents receive media files in a persistent, containerized terminal workspace and must produce artifacts such as edited media, timestamps, annotations, or structured records. Task-specific deterministic verifiers provide partial credit and binary success. The authors curate workflow-inspired scenarios, substitute licensed or synthetic media, and screen tasks using oracle solutions, dummy failures, agent trajectories, and manual checks. They also introduce Terminus-MM, which adds native image, standalone-audio, and video-file tools to a terminal agent and masks tools according to the initial workspace. Experiments compare tool-access configurations across five backbones and two off-the-shelf CLI agents. On three multimodal backbones, the full-access configuration improves over text-only and image-access baselines; the strongest three-seed result is 34.0% binary success for Gemini-3.1-Pro with Terminus-MM. A human pilot on 40 selected non-expert tasks achieves 70.0%, compared with 30.0% for that agent, under different interface and time conditions. The paper further analyzes conversion overhead, solver overlap, and failure signatures. Its main contribution is the benchmark and the documented gap between multimedia file access and reliable artifact production, rather than a new learning algorithm.

Critical strengths

MMTB provides 105 tasks that require terminal agents to use multimedia evidence and produce verifiable output files. Each task includes a containerized workspace, a verifier, and an oracle solution, with checks for solvability, shortcuts, and grading errors. This makes it a useful benchmark for evaluating multimedia understanding and terminal execution together.

Critical weaknesses

The modality comparisons also change other parts of the harness. Terminus-KIRA uses a different prompt, while Terminus-MM additionally applies tool masking. Table 5 shows that masking itself improves performance, making it unclear how much of the reported gain comes from native multimedia access. The comparisons should use the same prompt structure and masking policy.

Questions for authors
  1. Do the gains from native multimedia access remain when all modality variants use the same prompt structure and tool-masking policy?
  2. How were the verifier thresholds calibrated, and were the failures in the human study due to task difficulty, unclear instructions, or grading errors?
4 comments

We thank the reviewer for the thoughtful comments, and we address them below.

[W1 + Q1. Prompt structure and tool masking.]

The gains from native multimedia access remain when all modality variants use the same prompt structure and tool-masking policy. We added two controlled baselines that share Terminus-MM's harness implementation and prompt template, Terminus-T (no native perception tool) and Terminus-I (native image access only), and turned off Terminus-MM's tool masking (modality masking). Adding native audio and video access to Terminus-I raises binary / partial success from 6.7 / 11.9% to 17.1 / 26.7% on Gemini-2.5-Flash and from 21.9 / 25.9% to 33.3 / 42.9% on Gemini-3.1-Pro (Table R1; single runs, to be repeated with multiple seeds in the final version). Following the reviewer's suggestion, the revision states the harness differences in Section 3.1 and reports both baselines in the main results (Table 3).

Table R1. Modality ablation with one harness implementation and one prompt template (Binary % / Partial %, n = 105 tasks; three-seed rows mean ± SD, other rows single runs). The Gemini-3.1-Pro values of Terminus-IA and Terminus-MM w/o modality masking are updated after the reference correction of one task (Appendix B.13 of the revision).

HarnessPerception toolsRunsGemini-2.5-FlashGemini-3.1-Pro
Terminus-T (new)none14.8 / 9.316.2 / 19.8
Terminus-I (new)view_image16.7 / 11.921.9 / 25.9
Terminus-IAview_image, listen_audio3 seeds12.7 ± 2.0 / 19.6 ± 3.734.0 ± 2.4 / 42.7 ± 4.3
Terminus-IVview_image, watch_video3 seeds19.4 ± 2.7 / 26.5 ± 3.831.4 ± 5.0 / 44.5 ± 1.8
Terminus-MM w/o modality maskingall three117.1 / 26.733.3 / 42.9
Terminus-MMall three, masked per workspace3 seeds20.0 ± 2.5 / 27.5 ± 3.034.0 ± 3.3 / 45.8 ± 1.2

[Q2. Thresholds and the human-study failures.]

The thresholds were set by each task's author when the task was built and were not calibrated on held-out outputs; a blind audit now checks the verifiers against human judgment. Of the 12 human-study failures, 10 were participant errors, one came from an unclear instruction, and one was a grading error that the revision corrects.

Calibration. Each threshold reflects how much error the deliverable tolerates and is stated in the task's verifier. In the blind audit, the false-acceptance rate is 11.1% and the false-rejection rate is 13.2% (Table R5), and the verifier agrees with the human judges as much as the judges agree with each other (Table R6). Eight judges who did not build the tasks graded real agent outputs without seeing the verifier's decision, the model, or the harness (protocol in Appendix B.13 of the revision). The false-rejection rate counts only rejected runs that produced an output.

Table R5. Verifier decisions against blind human judgment on 74 agent outputs from 39 tasks.

Verifier decisionJudged correctJudged incorrectVerifier error
Accepted (36)3244 / 36 false acceptances (11.1%)
Rejected (38)5335 / 38 false rejections (13.2%)

Table R6. Agreement between the verifiers and the human judges, and among the judges, next to the same statistics reported for WebVoyager's automatic evaluator (He et al., 2024).

Evaluator vs. human judgesAmong human judges
MMTB verifiers87.8% (65 / 74), Cohen's κ = 0.76Fleiss' κ = 0.72
WebVoyager evaluator85.3%, Cohen's κ = 0.70Fleiss' κ = 0.70

Thresholds. With common values in place of the task-specific settings, the four harnesses without native audio or video access (Terminus-2, Terminus-KIRA, Terminus-T, Terminus-I) stay below Terminus-MM at every setting on both Gemini backbones (Table R7; Appendix B.12 of the revision). We re-scored every run with one common value in place of the task-specific one, one parameter at a time: the pass threshold τ on the verifier's score, the overlap threshold θ for matching time intervals (9 tasks), or the tolerance δ for matching timestamps (13 tasks). The comparison is the same in partial success, which uses no pass threshold (row 'no τ').

Table R7. Binary success (%) with one common pass threshold or matching tolerance in place of the task-specific one, one parameter at a time. Three-seed means for Terminus-2, Terminus-KIRA, and Terminus-MM; single runs for Terminus-T and Terminus-I. The row 'no τ' is partial success (%), which uses no pass threshold.

SettingTerminus-2Terminus-KIRATerminus-TTerminus-ITerminus-MM
Gemini-2.5-Flash
original settings6.07.34.86.720.0
τ = 0.512.712.77.613.328.3
τ = 0.610.510.26.79.524.8
τ = 0.77.37.05.78.621.9
τ = 0.85.76.34.86.719.7
τ = 0.94.15.73.85.716.2
τ = 1.03.54.43.83.815.2
no τ (partial success)13.012.89.311.927.5
θ = 0.256.07.34.86.720.0
θ = 0.56.07.34.86.720.0
θ = 0.756.07.34.86.720.0
δ = 0.5 s6.07.04.86.718.1
δ = 1 s6.07.04.86.718.7
δ = 2 s6.37.34.86.720.6
Gemini-3.1-Pro
original settings15.612.716.221.934.0
τ = 0.519.017.121.926.747.0
τ = 0.617.815.220.025.741.6
τ = 0.715.913.716.221.937.8
τ = 0.815.212.714.320.034.0
τ = 0.914.69.812.417.129.2
τ = 1.012.79.29.514.324.4
no τ (partial success)18.816.219.825.945.8
θ = 0.2515.612.716.221.934.6
θ = 0.515.612.716.221.934.0
θ = 0.7515.212.716.221.933.7
δ = 0.5 s15.612.715.221.033.0
δ = 1 s15.612.716.221.933.3
δ = 2 s15.912.716.221.934.6

Reference: He et al., WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models, ACL 2024.

[Q2, continued.]

Human-study failures. Table R8 gives the cause of each of the 11 failed task results (Appendix B.15 of the revision, Table 27). One further result had failed because of a grading error: the verifier's reference answer did not match the task media. The revision corrects the reference, so this result now passes and is not in the table; the human pass rate is 72.5% (70.0% before the correction).

Table R8. Causes of the 11 failed task results (of 40) in the human study.

CauseFailed task results
Participant error: wrong content or wrong edit9
Participant error: required output missing or in an invalid format1
Unclear instruction1
Total11

Dear Reviewer se1v,

Thank you again for your review. If any question remains after our response, we would be glad to answer it before the discussion closes.

Reviewer_H2q2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MMTB introduces 105 multimedia-file tasks that evaluate terminal agents through their final outputs. The paper also proposes Terminus-MM, which provides native multimedia access, and studies how different interfaces affect task success, efficiency, and failure modes.

Critical strengths
  1. The benchmark evaluates practically meaningful multimedia workflows. Agents must inspect and manipulate files, then produce usable outputs such as annotations, structured records, or edited media. This connects multimedia understanding with planning, tool execution, and delivery. The 105 tasks across five broad categories provide a useful setting for evaluating these capabilities together.

  2. The evaluation infrastructure and supporting analyses are substantial. Tasks include fixed environments, oracle solutions, and automatic verifiers, with construction-time audits for incorrect judgments and shortcuts. The paper also provides repeated runs for several models, modality-masking ablations, budget experiments, and failure analysis. A human pilot supports solvability on 40 selected tasks, although its GUI access and unrestricted time differ from the agent protocol. These efforts strengthen the benchmark’s reproducibility and usefulness.

Critical weaknesses
  1. The comparisons do not fully isolate the contribution of native perception. The video interface includes audio, and Terminus-KIRA uses a different prompt. Agents relying on command-line conversion also incur additional orchestration costs. A comparison with a strong terminal agent using preconfigured ASR, OCR, and video-analysis tools under the same budget would help distinguish native-perception benefits from interface and tool-configuration effects.

  2. Independent calibration of the verifiers remains limited. The construction-time audits are valuable, but they do not provide an independent estimate of false acceptance and false rejection on new agent outputs. Tasks involving temporal boundaries or other numerical tolerances may also be sensitive to acceptance thresholds. Without further calibration, it remains unclear how much these choices affect reported success rates and model rankings.

Questions for authors
  1. With shared prompts and a silent-video condition, do the modality-ablation conclusions still hold?
  2. Under the same budget, how much advantage remains over a strong terminal agent with preconfigured ASR, OCR, and video-analysis tools?
  3. What are verifier false-accept and false-reject rates on independently reviewed outputs, and are rankings stable under reasonable threshold changes?
6 comments

We thank the reviewer for the thoughtful comments, and we address them below.

[W1 + Q1. Shared prompts and a silent-video condition.]

With shared prompts, the modality-ablation conclusion holds: native audio and video access still raises success over image-only access on both Gemini backbones. The silent-video condition corrects one point: most of the gain the paper attributed to video comes from the soundtrack.

Shared prompts. We added two controlled baselines that share Terminus-MM's harness implementation and prompt template, Terminus-T (no native perception tool) and Terminus-I (native image access only), and turned off Terminus-MM's tool masking (modality masking). Adding native audio and video access to Terminus-I raises binary / partial success from 6.7 / 11.9% to 17.1 / 26.7% on Gemini-2.5-Flash and from 21.9 / 25.9% to 33.3 / 42.9% on Gemini-3.1-Pro (Table R1; single runs, to be repeated with multiple seeds in the final version).

Table R1. Modality ablation with one harness implementation and one prompt template (Binary % / Partial %, n = 105 tasks; three-seed rows mean ± SD, other rows single runs). The Gemini-3.1-Pro values of Terminus-IA and Terminus-MM w/o modality masking are updated after the reference correction of one task (Appendix B.13 of the revision).

HarnessPerception toolsRunsGemini-2.5-FlashGemini-3.1-Pro
Terminus-T (new)none14.8 / 9.316.2 / 19.8
Terminus-I (new)view_image16.7 / 11.921.9 / 25.9
Terminus-IAview_image, listen_audio3 seeds12.7 ± 2.0 / 19.6 ± 3.734.0 ± 2.4 / 42.7 ± 4.3
Terminus-IVview_image, watch_video3 seeds19.4 ± 2.7 / 26.5 ± 3.831.4 ± 5.0 / 44.5 ± 1.8
Terminus-MM w/o modality maskingall three117.1 / 26.733.3 / 42.9
Terminus-MMall three, masked per workspace3 seeds20.0 ± 2.5 / 27.5 ± 3.034.0 ± 3.3 / 45.8 ± 1.2

Silent video. With the soundtrack removed from the video given to watch_video, Terminus-IV falls close to Terminus-I (binary success 9.5% and 20.0%, against 6.7% and 21.9%; Table R2). With the video muted and listen_audio retained, native audio and video access still raises success over image-only access.

Table R2. Controlled ablation with the soundtrack removed from the video given to watch_video (Binary % / Partial %; single runs with a shared prompt template and without modality masking).

HarnessPerception toolsGemini-2.5-FlashGemini-3.1-Pro
Terminus-T (new)none4.8 / 9.316.2 / 19.8
Terminus-I (new)view_image6.7 / 11.921.9 / 25.9
Terminus-IV, muted (new)view_image, watch_video without soundtrack9.5 / 17.020.0 / 25.1
Terminus-MM w/o modality masking, muted (new)view_image, watch_video without soundtrack, listen_audio16.2 / 21.228.6 / 39.8

Cost. The cost comparison of Section 4.2 is also a modality-ablation conclusion, so we extended it to the controlled baselines. On the tasks both solve, where the baseline must recover a missing modality through terminal tools, Terminus-T and Terminus-I incur 6.9× and 7.8× the API cost of Terminus-MM on average (Table R3).

Table R3. The paper's Table 4 with Terminus-T and Terminus-I added: API-cost and turn ratios to Terminus-MM on matched co-success tasks (Gemini-3.1-Pro, seed 1; mean and worst case; T, I, A, V = native text, image, audio, video access).

HarnessNative accessMatched tasksAPI cost ratio, meanworstTurn ratio, meanworst
Terminus-2T74.12×26.48×1.00×2.71×
Terminus-KIRAT+I41.63×3.40×1.39×2.00×
Terminus-T (new)T116.90×29.02×2.65×4.83×
Terminus-I (new)T+I117.77×23.45×3.53×9.83×
Terminus-AT+A144.38×19.37×1.77×3.83×
Terminus-VT+V131.84×11.64×1.15×2.17×
Terminus-IAT+I+A134.20×30.11×2.21×8.10×
Terminus-IVT+I+V147.72×42.49×2.01×5.88×
Terminus-AVT+A+V42.00×6.08×1.28×2.06×

Following the reviewer's points, the revision states the harness differences in Section 3.1, reports both baselines in the main results (Table 3), and corrects the reading of the video gain in Section 4.1, with the silent-video condition in Appendix B.9 (Table 16).

[Q2. A strong terminal agent with preconfigured tools.]

The advantage of native audio and video access remains on Gemini-2.5-Flash when the text-only and image-only baselines get preconfigured ASR, OCR, and video-analysis tools under the same budget. With the toolkit, Terminus-T and Terminus-I both reach 6.7% binary success against 12.4% for Terminus-MM without modality masking, and 13.0% and 16.7% partial success against 21.0% (Table R4; single runs). The toolkit itself changes the baselines' binary success little: Terminus-T rises from 4.8% to 6.7% and Terminus-I is unchanged.

Toolkit. We built it from the paper's own runs. A scan of the 3,150 three-seed Gemini trajectories recorded which programs the agents used or tried to install. We preinstalled the most-used programs in every family that appeared: ffmpeg, ASR, OCR, frame extraction and scene detection, audio analysis, image processing, and PDF tools. We also cached the model weights the agents most often requested and described the toolkit in the prompt (Appendix B.11 of the revision, Table 20; one sentence in Section 4.3).

Table R4. Binary / partial success (%), shell commands, and agent time per task on Gemini-2.5-Flash with and without the preinstalled toolkit. Single runs; the Terminus-MM row is the paper's three-seed result.

HarnessToolkitBinary / PartialCommandsTime (s)
Terminus-Tno4.8 / 9.318.3133
Terminus-Tyes6.7 / 13.041.0151
Terminus-Ino6.7 / 11.924.4189
Terminus-Iyes6.7 / 16.735.0171
Terminus-MM w/o modality maskingno17.1 / 26.713.8230
Terminus-MM w/o modality maskingyes12.4 / 21.019.4143
Terminus-MMno20.0 ± 2.5 / 27.5 ± 3.019.4175

[W2 + Q3. Independent calibration and threshold sensitivity.]

On independently reviewed outputs, the verifiers' false-acceptance rate is 11.1% and their false-rejection rate is 13.2%, and the comparison between harnesses with and without native audio or video access does not change under threshold changes.

Error rates. Eight judges who did not build the tasks graded real agent outputs without seeing the verifier's decision, the model, or the harness, and we compared their grades with the verifier's decisions (Table R5; protocol and 95% intervals in Appendix B.13 of the revision). The verifier agrees with the judges as much as the judges agree with each other, which is the comparison by which WebVoyager validates its automatic evaluator (Table R6; He et al., 2024). The false-rejection rate counts only rejected runs that produced an output; including the rejected runs that produced none lowers it to 6.5%.

Table R5. Verifier decisions against blind human judgment on 74 agent outputs from 39 tasks.

Verifier decisionJudged correctJudged incorrectVerifier error
Accepted (36)3244 / 36 false acceptances (11.1%)
Rejected (38)5335 / 38 false rejections (13.2%)

Table R6. Agreement between the verifiers and the human judges, and among the judges, next to the same statistics reported for WebVoyager's automatic evaluator (He et al., 2024).

Evaluator vs. human judgesAmong human judges
MMTB verifiers87.8% (65 / 74), Cohen's κ = 0.76Fleiss' κ = 0.72
WebVoyager evaluator85.3%, Cohen's κ = 0.70Fleiss' κ = 0.70

Thresholds. At every setting, on both Gemini backbones, the four harnesses without native audio or video access (Terminus-2, Terminus-KIRA, Terminus-T, Terminus-I) stay below Terminus-MM (Table R7; Appendix B.12 of the revision). We re-scored every run with one common value in place of the task-specific one, one parameter at a time: the pass threshold τ on the verifier's score, the overlap threshold θ for matching time intervals (9 tasks), or the tolerance δ for matching timestamps (13 tasks). The comparison is the same in partial success, which uses no pass threshold (row 'no τ').

Table R7. Binary success (%) with one common pass threshold or matching tolerance in place of the task-specific one, one parameter at a time. Three-seed means for Terminus-2, Terminus-KIRA, and Terminus-MM; single runs for Terminus-T and Terminus-I. The row 'no τ' is partial success (%), which uses no pass threshold.

SettingTerminus-2Terminus-KIRATerminus-TTerminus-ITerminus-MM
Gemini-2.5-Flash
original settings6.07.34.86.720.0
τ = 0.512.712.77.613.328.3
τ = 0.610.510.26.79.524.8
τ = 0.77.37.05.78.621.9
τ = 0.85.76.34.86.719.7
τ = 0.94.15.73.85.716.2
τ = 1.03.54.43.83.815.2
no τ (partial success)13.012.89.311.927.5
θ = 0.256.07.34.86.720.0
θ = 0.56.07.34.86.720.0
θ = 0.756.07.34.86.720.0
δ = 0.5 s6.07.04.86.718.1
δ = 1 s6.07.04.86.718.7
δ = 2 s6.37.34.86.720.6
Gemini-3.1-Pro
original settings15.612.716.221.934.0
τ = 0.519.017.121.926.747.0
τ = 0.617.815.220.025.741.6
τ = 0.715.913.716.221.937.8
τ = 0.815.212.714.320.034.0
τ = 0.914.69.812.417.129.2
τ = 1.012.79.29.514.324.4
no τ (partial success)18.816.219.825.945.8
θ = 0.2515.612.716.221.934.6
θ = 0.515.612.716.221.934.0
θ = 0.7515.212.716.221.933.7
δ = 0.5 s15.612.715.221.033.0
δ = 1 s15.612.716.221.933.3
δ = 2 s15.912.716.221.934.6

Reference: He et al., WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models, ACL 2024.

Comments on the paper

1 comment

General response.

We thank the reviewers. Their questions led to four additions. Each is reported in the reply to the reviewer who asked and in the revision; section, appendix, table, and figure numbers without an R refer to the revision.

  1. Controlled ablation (H2q2, se1v). With the harness, prompt template, and tool masking held fixed, native audio and video access still raises success over image-only access on both Gemini backbones. Most of the gain the paper attributed to video comes from the soundtrack (Section 4.1; Appendix B.9).
  2. Tool-equipped baselines (H2q2). Giving the new controlled baselines Terminus-T (text only) and Terminus-I (image only) a preinstalled toolkit does not close their gap to native access on Gemini-2.5-Flash (Section 4.3; Appendix B.11).
  3. Verifier calibration (H2q2, se1v, Dtef). A blind audit of real agent outputs finds 11.1% false acceptances and 13.2% false rejections, and the verifier agrees with the human judges as much as the judges agree with each other. The comparison between harnesses with and without native audio or video access is the same at every threshold setting (Appendices B.12 and B.13).
  4. Stability (Dtef). With native audio and video access (Terminus-MM), binary and partial success are above those of Terminus-2 and Terminus-KIRA in every seed on all three backbones and, pooled across backbones, in every meta-category (Appendix B.10).

The runs added for these analyses (Terminus-T, Terminus-I, and the silent-video and toolkit conditions) are single runs; we will repeat them with multiple seeds and report the means in the final version.

Changes are marked in blue in the revision and listed below.

Summary of changes in the revision.

  • Section 3.1 and Table 2 state the harness differences: Terminus-T, -I, -IA, -IV, and -MM share one harness implementation and prompt template and differ in their native perception tools, Terminus-MM additionally masks tools per task, and Terminus-2 and Terminus-KIRA are external baselines.
  • Terminus-T and Terminus-I (single runs on the two Gemini backbones) are added to Tables 3, 4, 6, 8, 11, 12, 15, and 32 and Figures 7, 8, and 10.
  • Section 3.3 points to the threshold sensitivity and the blind audit. Sections 4.1 to 4.3 include the controlled baselines in the baseline comparison, correct the reading of the video gain with the silent-video result, update the cost and turn ratios, and add one sentence on the toolkit.
  • New appendix sections B.9 to B.13: silent video (Table 16), stability across seeds and meta-categories (Tables 17 to 19), preinstalled toolkit (Table 20), threshold sensitivity (Table 21), and blind audit (Tables 22 and 23).
  • Appendix B.15 adds the causes of the 11 failed human-study results (Table 27).
  • Appendices A.1 to A.4, B.6, B.8, and B.17 update counts and numbers for the added baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.