acceptodds
Under review as a conference paper at ICLR 2027

GUI-HITO: Benchmarking History-Induced Tool Overuse in Multimodal Agents

Abstract

Knowing the answer does not guarantee that a computer-use agent will answer directly. We show that prior tool use can bias agents toward further action even when the current request requires no interaction—a failure we call history-induced tool overuse. We introduce GUI-HITO, a paired benchmark that probes this answer–act boundary with English and Chinese questions across 33 history conditions. Comparisons hold the current request, screenshot, tool interface, and system prompt fixed while matching session counts and screenshot sequences across tool-bearing and action-free histories. On a shared, answerability-calibrated test panel, all five evaluated multimodal endpoints exhibit higher first-response action-attempt rates after tool-bearing histories, with increases of 29.0–53.8 percentage points and significance after multiplicity correction. Actions are scored without execution. The effect survives replaying prior actions as plain text, showing that native tool-call formatting is not necessary; four endpoints also exhibit increased attempts when the current screenshot already contains the answer. An answer-first prompt gate reduces attempts under tool-bearing histories from 60.6% to 16.2% on a development panel while leaving direct-answer accuracy unchanged. These findings expose a gap between execution competence and action restraint: reliable agents must recognize when to answer, even after a conversation full of actions.

Primary area:
datasets and benchmarks
Keywords:
GUI agents, computer-use agents, tool use, tool overuse, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HRBxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper identifies a problem: when the context history of a GUI agent accumulates tool-using actions, the agent tends to invoke unnecessary tool calls. To quantify how severe this is, the authors build a paired benchmark that fixes the question and the state (screenshot) at a given point while switching only the history context. In both the synthetic (Main-S) and the external-benchmark-derived (External-R) settings, the first-response tool-attempt rate is higher after tool-trace histories than after matched action-free histories (T−N > 0) across all five LLMs. The paper also proposes mitigation strategies to alleviate unnecessary tool calling, both training-based and training-free.

Critical strengths

(1) The paper's experimental setting reflects the components of a GUI agent environment well.

(2) The experimental results appear reliable: the experiments were conducted in a controlled setting, and confidence intervals are reported.

Critical weaknesses

(1) The paper's central scenario is one in which the agent's history is filled with tool calls and the user then suddenly asks a simple question that does not require tools. It is unclear how often this happens in the real world. As an agent paper, it should better justify what real-world problems the defined issue can cause. To strengthen the justification, the authors should report how often such situations occur in practice.

(2) The benchmark consists of questions that the agent can answer directly without tools (short knowledge or calculation questions), but it is unclear what the criterion for this is and where it comes from. Figure 2's example, "What is 64 multiplied by 17?", appears to represent the benchmark's questions, yet in my view using a calculator is a reasonable choice for this question. The authors should clarify the criterion and show that it would be widely accepted.

(3) The related work shows that tool overuse is already a well-studied topic, so it is unclear what novelty these controlled-setting results offer over prior work. If there is novelty, the authors should state it clearly. To distinguish itself as an agent paper, the work should also present failure cases or problem scenarios from real-world deployment.

(4) The paper emphasizes that it is about GUI agents (the title begins with "GUI"), but no experiment tests whether the effect is specific to the image modality. For example, there is no text-only condition or screenshot-removed condition, so it is impossible to tell whether the effect stems from the GUI/visual context or is a general property of histories containing tool calls.

(5) In Section 7, the reward gives the lowest value whenever the agent attempts a tool call, so it is unsurprising that the agent stops calling tools. This is not really a mitigation: the trained agent may suffer a serious side effect, namely never calling tools even when they are necessary. The paper does not evaluate this on required-tool tasks for the RL model.

(6) The writing quality needs to be improved substantially.

Questions for authors

(1) As far as I understand, there are four channels through which the agent's tool attempt is detected (a native provider function call, a serialized tool-call block naming a tool, a standalone action-record line, and executable control code). Is there a specific reason for choosing these four?

(2) Could you also report the results in the H0 (no-history) setting? This would be useful for comparison.

Writing Suggestion: The three paragraphs in Related Work should be merged into two, as they contain a lot of redundant content. The space freed up this way could be used to expand Section 3. Section 3 is quite abstract, which makes the details hard to follow. Eq. 2 appears to be a key definition, so placing it in the appendix does not seem appropriate. Including more mathematical equations would make the method easier to present, and the inputs and outputs would be clearer than they are now.

Reviewer_AtYrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors introduced GUI-HITO, a benchmark for measuring history-induced tool overuse in multimodal GUI agents: a failure mode where agents unnecessarily attempt computer-use actions after conversations containing prior tool use, even when the current request requires only a direct textual answer. Five LLMs are evaluated: GPT-5.6-Sol, Claude Sonnet 5, Kimi K3, Qwen3.8-Max, and DeepSeek V4 Flash; all showing significantly elevated tool-attempt rates after tool-bearing histories. Two mitigation strategies are proposed: a prompting-based approach that reduces tool-trace attempts from 60.6% to 16.2% while preserving answer accuracy, and a GRPO-based reinforcement learning approach that reduces the gap from 17.75 to 2.93 percentage points.

Critical strengths
  1. The general direction of the problem studied in the paper is interesting and of practical importance.

  2. The paper is well written and easy to follow.

Critical weaknesses
  1. The phenomenon tested in this paper is pretty intuitive and easily solvable using mostly providing in-context instructions to the LLM aka prompting. The prompting-based mitigation strategy explored by the authors confirms the trivialness of the task. On a side note, the authors do not provide the actual prompt text for any of the three prompt interventions. It's a reproducibility gap and makes it harder to understand if there was more room for improvement with prompt tuning.

  2. The authors posed prompting-based and GRPO-based approaches as mitigation strategies, however, based on the task, these are just intuitive steps which can be taken to handle such issues and I don't find them novel. Given the problem is pretty intuitive, a more meaningful contribution would involve investigating why the phenomenon occurs mechanistically, for instance through attention pattern analysis, representation probing, or activation-level studies, rather than applying surface-level interventions that confirm the problem is easy to address.

  3. The boundary between tool-required and answer-directly tasks could be better studied. The authors presented the example "What is 10% of 200?" as a motivating example for knowledge tasks, however, in practice enabling an LLM with the calculator tool might just increase the reliability and correctness of such math tasks. The paper treats all tool use on no-call targets as failures, but this binary framing ignores cases where tool use, while unnecessary, is actually beneficial or harmless. A more nuanced analysis of when overuse truly harms task outcomes, error propagation, or user experience degradation, would strengthen the contribution.

  4. More realistic and harder scenarios could be studied such as more complex trajectories, ambiguous tool-necessity boundaries, real-world multi-step tasks, and diverse interaction surfaces and datasets.

  5. The paper relies entirely on automated broad-attempt detection with no human evaluation or qualitative error analysis of what kinds of unnecessary actions the models attempt or why. A deeper qualitative analysis, potentially with blinded human raters, would add analytical depth that the paper currently lacks.

  6. [Minor] The paper doesn't seem to follow ICLR format exactly. The citations appear in blue and reference to sections, tables and figures appear in red. Not sure if that's the correct format for ICLR.

Questions for authors

Please refer to the weaknesses section.

Reviewer_mMxDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an overlooked problem, history-induced tool overuse (HITO), and introduces GUI-HITO, a benchmark for investigating its causes. On GUI-HITO, all four evaluated model endpoints attempt more actions even when the current screenshot already contains the answer. These findings appear to show a gap between execution competence and action restraint: the models can carry out actions but do not reliably recognize when no action is needed.

Critical strengths
  1. The problem is somewhat novel. The paper points to a failure mode that has received little attention: interaction history can lead GUI agents to keep acting even when the answer is already on the screen. Asking whether an agent knows when not to act is a reasonable complement to standard evaluations of task completion. However, the motivation and the definition of "overuse" are not well established (see Weaknesses), which limits how much this contribution adds.
  2. It provides a targeted benchmark. GUI-HITO offers a controlled setting for measuring this behavior, and the effect appears across four model endpoints.
Critical weaknesses

1.The motivation is unconvincing. The paper says a query like "10% of 200" should be answered directly rather than with a tool. However, a calculator call costs little and is often more reliable, especially for large numbers. Without a quantitative cost and accuracy comparison, or a clear line between benign and harmful tool use, it is unclear why such tool use counts as "overuse." The paper also does not explain why it studies only GUI agents. These gaps weaken the case that the problem is real and worth solving. 2.The causal claims are confounded. The interventions change the content and the role of the history at the same time. The observed effect could therefore come from imitating behavior, imitating format, inferring from the history that a tool call is expected, or long-context distraction, and the paper does not separate these. In addition, the T-group history encourages actions while the N-group history suppresses them, which may exaggerate the gap. Together, these issues make the central finding hard to interpret.

Questions for authors
  1. Can you quantitatively compare answering directly with calling a tool, in terms of cost (tokens, latency) and accuracy? Please include cases with large or complex numbers, where a calculator may be both more reliable and more efficient than reasoning. Also, do you distinguish benign tool use from harmful tool use? If so, how is "overuse" operationally defined?
  2. Why do you study HITO only in the GUI-agent setting? Is the problem more severe there, or was it simply only examined in that domain? Do you have any evidence, even preliminary, on whether the phenomenon occurs in other agentic settings, such as general function calling or coding agents?
  3. Your interventions change the content and the role of the context at the same time. Can you run controlled ablations that vary one factor at a time? Such ablations would show which of the following drives the effect: Behavioral imitation. Format imitation. Inferring from the history that a tool call is expected. Long-context effects that make the model overlook a directly answerable request.
  4. The T-group history encourages actions and the N-group history suppresses them. How do you make sure this comparison is fair and does not artificially lower the tool-use rate? Could you add a neutral-history control, and report whether the N condition also suppresses tool calls when they are actually needed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.