acceptodds
Under review as a conference paper at ICLR 2027

Absorbed in Inertia: Activation Analysis for Computer-Use Agents

Abstract

Computer-use agents have become increasingly capable of executing tasks on live desktops through natural-language instructions, based on trajectories of screenshots, actions, and reasoning. We discover that they can stealthily exhibit *inertia*, in which they repeat fruitless actions despite recognizing that these actions are ineffective. We hypothesize that inertia is reflected in the agent's internal state, i.e., the activation values of the agent's underlying model, and propose a protocol to measure the relationship between the two. Extensive analysis of high-dimensional activation states shows that inertia corresponds to an *absorbing region of activation space*, where activation values become stale across actions and even after attempts to steer them. We conjecture that drastically changing the agents' activations by re-initializing them is necessary to escape inertia. Specifically, we propose R³ (Reset, Reroute, Restore), which temporarily resets the agent's context trajectory to escape the absorbing region and then restores the historical context to effectively complete the task. Our approach yields 17–55% lower measured inertia across models relative to unmodified agents. These results suggest that changing the context can interrupt recurrence more effectively than directly steering the resulting activations. Our code is available at https://anonymous.4open.science/r/vlm-agent-defense-D076.

Primary area:
interpretability and explainable AI
Keywords:
agentic, llm, cua, computer-use, computer-use-agent, mechanistic, steering, harness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V4g3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines computer-use agents that perform an action again even though their own reasoning has already said the action failed, and it names this behavior inertia. Across four open agents, the activations before each action shift only slightly while the repetition lasts, and steering these activations does not lower it consistently. The proposed R3 clears the context once repetition is detected, looks for an action that alters the screen, and afterwards puts the earlier history back. Over 97 OpenComputer tasks, R3 lowers recurrence for every one of the four agents, while every task-score interval contains zero.

Critical strengths
  1. Recurrence is measured in a simple and repeatable way, since it relies on exact matches of screens and actions rather than on the judgment of an LLM (Section 2.1). The acknowledgment labels are validated against a second judge and against human annotation (Appendix B.3).
Critical weaknesses
  1. It is unclear what is gained in practice by reducing inertia, since lower recurrence is not accompanied by a higher task score (Table 1). For OpenCUA with steering, recurrence and task score even rise at the same time.

  2. Wan et al., Mitigating Conversational Inertia in Multi-Turn Agents (arXiv 2602.03664, ICML 2026), reduces the novelty of this paper; it is not cited here, and the two works overlap at three levels. On the phenomenon, Wan et al. describe inertia as an agent copying its earlier responses rather than adjusting to new feedback, which is similar to the repeated action examined here. On the analysis, Wan et al. find that from the same state a longer context produces stronger inertia than a shorter one, and the replay experiment carries out the same type of comparison between the full history and a fresh context (Table 2). On the method, their Clip Context empties the history once the context hits a threshold, which is also what the Reset step and the Permanent Reset baseline do. Wan et al. work with text-based agents and do not include the failure acknowledgment label, the activation analysis and steering, the recurrence trigger, or the Reroute and Restore steps. The paper should make this difference explicit and compare R3 against Clip Context.

  3. The activation analysis yields no result that the input and the reasoning text do not already account for. The limited activation change during recurrence is consistent with attention copying from a repeated context, a mechanism the paper mentions but does not disentangle (Appendix A). The acknowledgment readout is read after reasoning that has already stated the failure, so separating the two classes is the expected outcome (Figure 4). In Figure 3, the activation change of GUI-Owl also remains around 30% for most of its recurrence run.

  4. The evaluation is too narrow to tell whether the findings carry over to the latest agentic models that reason at length before acting. Each of the four agents is an open-source computer-use model whose visible reasoning is short (Appendix B.3), and the mitigation is tested on a single benchmark. Whether long-thinking agentic models display similar inertia remains unknown.

Questions for authors
  1. Could the authors provide Table 1 under a single fixed setting shared by all agents and all methods? Both the R3 threshold and the steering strength vary from agent to agent, and the threshold is tuned on the evaluation tasks (Appendix D.2).

  2. How well does the recurrence detector transfer outside the two benchmarks? It demands screens that match pixel for pixel, with only the clock region masked (Appendix B.2). Screens in real applications and in other domains frequently include dynamic content, which makes recurrence difficult to define this way and may leave many repeated actions undetected. Since this detector is what triggers R3, it restricts the generality of the method. Is there a definition of recurrence that does not rely on screens being exactly identical?

Reviewer_QrLdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper primarily studies the phenomenon in which an agent acknowledges that its previous action failed but continues to repeat the action, which the authors call inertia. Through an analysis of model activations, the authors propose that the model is in an absorbing region of activation space when inertia occurs. The authors then propose R^3 to help the model avoid inertia and conduct experiments on OpenComputer to validate the approach.

Critical strengths

The paper provides a relatively thorough analysis of inertia, including an in-depth examination of the model’s activation states.

Critical weaknesses
  1. The paper lacks a baseline that explicitly tells the agent not to use action xxx in the next prompt after it performs an incorrect action. This approach may already be effective, could preserve the KV cache, and might also have a positive effect on the final task score.
  2. Although R^3 can significantly reduce the incidence of inertia, it does not have a significant positive effect on task scores. The paper should analyze the reasons behind this in greater depth, including whether the actions produced by rerouting contribute positively to task progress.
Questions for authors

None

Reviewer_bD9rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies computer-use agents that repeat screen-action transitions while their reasoning acknowledges failure. It operationalizes this behavior as inertia, analyzes pre-action residual activations across four models and two benchmarks, and compares linear activation steering with context reset. The proposed R3 controller resets history after repeated unchanged-screen actions, searches for a screen-changing action, restores earlier history after success, and terminates if the search is exhausted. Full-trajectory mitigation is evaluated on OpenComputer.

Critical strengths

The behavioral phenomenon is interesting and practically relevant. Separating observable recurrence from expressed failure acknowledgment is a useful diagnostic design, and the paper includes cross-model analysis, task-level activation readouts, and matched-state next-action replay. The appendix contains meaningful controls, including selective reset, coordinate tolerance, full-history search, and continuation after search exhaustion. I also appreciate the explicit reporting of mixed task-score outcomes and the distinction between proposing another action, changing the screen, and completing the task.

Critical weaknesses

The absorbing-region interpretation remains underdetermined. Similar screens, repeated actions, and similar retained histories can naturally produce persistent activations. A linear readout of recurrence or explicit acknowledgment shows that the activation contains relevant information, but not that an identified internal region sustains the loop. Likewise, failure of the tested linear steering directions does not establish that a drastic reset is necessary. In the replay comparison, steering preserves the original reasoning while reset regenerates it; the interventions therefore differ in more than activation displacement. The context-length caveat in Appendix C.5 is also consequential. I read these findings as suggestive correlates of recurrence rather than a demonstrated mechanism.

Reduced recurrence has not yet translated into demonstrated recovery or task utility. Table 1 reports task-score changes of +0.60, +1.20, −2.50, and −0.90 points under R3. A screen-changing action can be unrelated to the goal, and stopping an unsuccessful search prevents subsequent repetitions by ending the trajectory. This matters especially for EvoCUA, where 19.59% of R3 trajectories terminate after search exhaustion. The continuation control is useful and confirms that termination contributes to the reduction, but does not establish useful recovery. A success-versus-cost comparison against a simple stop/retry controller would clarify what benefit R3 delivers beyond limiting wasted interaction.

Questions for authors

Which result distinguishes an inertia-specific internal mechanism from activation similarity induced by repeated inputs, retained history, or similar reasoning?

Under matched triggers, search budgets, and stopping rules, does resetting history improve successful recovery over retaining history? Does restoring history improve completion over leaving it reset?

What is the task-success and interaction-cost trade-off relative to simply stopping or retrying at the same recurrence trigger?

How often does the acknowledgment specifically identify the subsequently repeated action as ineffective?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.