acceptodds
Under review as a conference paper at ICLR 2027

AnyAct: Universal Action for Self-Evolving Agents

Abstract

As large language models (LLMs) advance, AI agents are increasingly deployed in open-world environments to tackle complex sequential tasks (e.g., document processing, cross-application collaboration), relying heavily on actions ranging from GUI operations to semantic APIs. However, three core challenges persist: the "scale dilemma" of massive tool ecosystems exceeding LLM context windows, the "non-stationarity" of tool quality due to updates or outages, and the "heterogeneity" of feedback formats (pixels, text, structured data) creating information silos. To address these, we propose AnyAct, a universal action layer that unifies available capabilities into a self-evolving action space, enabling agents to operate efficiently and reliably in large-scale, dynamic tool ecosystems. AnyAct's core design focuses on two objectives: constructing this action space via hierarchical progressive retrieval (filtering task-relevant actions) and test-time reliability evolution (pruning unreliable actions), and enabling reliability-aware action orchestration through a heterogeneous observation grounding module that unifies multi-modal feedback. Additionally, it defines a hybrid action space (primitive + semantic actions) and optimizes for a balance between task success rate and execution cost. Evaluations on LiveMCPBench and OSMCP (a new benchmark we developed for multi-granularity action collaboration) demonstrate state-of-the-art performance. AnyAct delivers substantial performance gains over baseline methods across various LLM base models on LiveMCPBench and improvements are particularly notable for models with constrained native capabilities. On OSMCP, it achieves 77.27% overall success with only 50 steps, which is half the steps required by most competitors.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool Learning, Self-Evolving Agents, Computer-Use Agents, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EjfXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AnyAct, a universal action framework for LLM-based agents that addresses challenges in large-scale and dynamic tool environments. It integrates hierarchical tool retrieval, test-time reliability evolution, and heterogeneous observation grounding to enable reliable and efficient coordination of GUI, CLI, and MCP actions.

Critical strengths
  1. This paper focuses on the meaningful problem of agent self-evolving tools, pointing out the issues of scale dilemma, non-stationary and heterogenity of feedback formats.

  2. The paper provides useful implementation details and reliable reproducibility.

Critical weaknesses
  1. The methodological novelty of AnyAct appears somewhat limited, as its core components primarily integrate established techniques like hierarchical tool retrieval, heuristic reliability scoring, and VLM-based observation parsing. In particular, the proposed self-evolution mechanism relies on manually predefined thresholds and heuristic rules to update.

  2. The evaluation of test-time reliability evolution is insufficient. As claimed in paper that adapting to dynamic tool reliability is a central motivation of AnyAct, the experiments lack controlled scenarios involving tool failures, recovery, or API changes. It remains unclear how effectively the proposed mechanism adapts to evolving tool environments.

  3. The methods compared on OSMCP employ different action interfaces, tool capabilities, and step budgets, making it difficult to determine whether the performance gains stem from the proposed framework or the additional MCP tool capabilities. Moreover, the ablation results in Table 2 between AnyAct and AnyAct w/o GUI uses different step budgets, preventing a fair evaluation of the contribution of GUI actions.

  4. The proposed benchmark OSMCP lacks dedicated evaluation settings or metrics to assess multi-granularity action collaboration, making it unclear whether OSMCP provides substantial evaluation value beyond an MCP-augmented version of OSWorld.

Questions for authors
  1. How sensitive is the proposed reliability evolution mechanism to manually specified thresholds and penalty parameters?

  2. How does AnyAct enable previously demoted tools to recover their reliability scores? If a tool is excluded from the retrieved Top-K actions due to repeated failures, how can the system detect its recovery without further executions?

  3. In Table 2, do all baseline methods have access to the same 122 MCP tools as AnyAct? If not, could the authors clarify the differences and is the comparison still fair enough?

11 comments

Thank you for your careful review and constructive comments. We appreciate your recognition of the difficulty of acting with changing tools and different forms of feedback, as well as your positive feedback on the implementation details for reproducing AnyAct. Below, we address each point in turn.


Methodological novelty and hand-set parameters (W1 & Q1)

Thank you for asking what evolves when the thresholds are set by hand. To answer this, we state what a tool call changes at test time and vary one parameter at a time.

What is distinctive. On a real desktop, an agent must find relevant tools in a large catalog, obtain the arguments those tools require, and keep working when an interface fails. AnyAct retrieves the semantic menu, preserves GUI and shell access, and passes observations from one interface into the next tool call. Updating the menu from execution status requires no extra model call or judge.

What evolves during execution (controlled MCP outage, DeepSeek-V3-0324). The parameters stay fixed. What changes is each tool’s weight and thus its place in the menu. A file writer still matched the request after its service failed. Three consecutive failures lowered its weight to 0.90 and moved it outside the Top-15 even though its rolling success rate remained above 40%.

The rolling success rate captures intermittent failures. Repeated execution errors can lower a tool’s rank even if occasional successful calls reset its streak.

Parameter sensitivity (controlled ledger test, DeepSeek-V3-0324). We changed one parameter at a time while tools competed for the same two menu slots. The first outcome measures calls to the unavailable target. The second includes failures after choosing an alternative.

Parameter changeCalled the unavailable toolAll failed decisions
Reliability floor, 0.1 → 0.44.17% → 37.50%62.50% → 91.67%
Success-rate threshold, 0.2 → 0.637.50% → 25.00%95.83% → 75.00%
Penalty per consecutive failure, 0.05 → 0.233.33% → 29.17%87.50% → 75.00%

The floor had the largest effect on calls to the failed tool. Across 24 decisions per setting, the failed tool appeared once at the lower floor and nine times at the higher floor. DeepSeek called it each time it appeared. Raising the success-rate threshold also reduced failures by letting the rule react earlier in a decline. Increasing the streak penalty had a smaller effect on target calls over the tested range.

Tool failures and recovery (W2 & Q2)

Thank you for asking for direct evidence of adaptation. We followed tools through a healthy phase, an outage and repair, and tracked the agent’s choices at each stage.

Tool failures (controlled MCP outage, DeepSeek-V3-0324). All four rules faced the same requests in the 12-service outage test. The last column gives the separate four-service recovery result, among tools still excluded by their own penalty at repair.

RuleFailed-target calls during outage (12 services, of 576) ↓Correct calls while healthy (12 services, of 288) ↑Verified reuse after exclusion (4 services)
Updates off535 (92.88%)258 (89.58%)—
Success rate only491 (85.24%)256 (88.89%)3/11
Failure streak only156 (27.08%)258 (89.58%)0/17
Full rule156 (27.08%)257 (89.24%)2/18

The full rule reduced the failed-target call rate by 65.80 percentage points. Eleven services improved and one tied. The streak term alone matched the full rule during these abrupt outages. Correct calls to healthy targets remained similar across the four rules.

How an excluded tool recovers (four-service MCP recovery test, DeepSeek-V3-0324). Under the full rule, 2 of 18 eligible tool–order pairs re-entered the Top-15 and returned a verified correct value within 12 relevant requests after repair. All 18 had passed a separate health check and were still excluded by their own penalty at repair. Both recoveries involved geocoding in different orders. On a request for Vienna, the tool ranked 14th while its weight was still 0.90. DeepSeek selected it and received the correct coordinates. The successful call raised its weight to 1.00.

Action interfaces, tool capabilities and step budgets in Table 2 (W3 & Q3)

Thank you for pointing out the limits of the comparison in Table 2. We agree that action access and the controller’s contribution should be evaluated separately.

Did all submitted baselines have the same tools?. The end-to-end GUI baselines used their native GUI interfaces. The other external frameworks used their reported interfaces and budgets. Both AnyAct variants had access to the same 122 application MCP tools.

Matched comparison (OSMCP, DeepSeek-V3-0324 with Qwen3-VL-30B-A3B-Instruct). All systems used the same shell, application MCP tools, initial desktops and evaluators, with a common cap of 50 controller decisions and 50 tool actions. Both GUI-enabled systems used the same vision model. The desktop ReAct baseline [1] saw the full tool catalog each round. AnyAct retrieved up to 15 MCP tools from that catalog.

MethodMean final-state score
ReAct + MCP54.04%
AnyAct w/o GUI56.57%
AnyAct65.15%

With the controller and budget fixed, GUI access raised the mean final-state score by 8.59 percentage points. With GUI, shell and MCP access fixed, AnyAct scored 11.11 points above the ReAct baseline.

Reviewer_WeLsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AnyAct, an agent action framework that unifies GUIs, Shell, and MCP (Model Context Protocol) tools. The approach employs hierarchical retrieval to filter relevant tools, dynamically adjusts tool reliability based on execution history, and converts diverse forms of feedback into a unified information representation. It demonstrates strong task-completion performance on LiveMCPBench and OSMCP—a benchmark built upon OSWorld. 44236_AnyAct_Universal_Action_

Critical strengths
  1. The problem addressed is of practical value. It reflects the realities of an actual tool ecosystem by accounting for challenges such as the vast number of available tools, service instability, and the diversity of feedback formats.
  2. The framework design is complementary. It leverages semantic tools to enhance execution efficiency while retaining foundational interaction methods—such as GUIs—as a supplement, thereby balancing efficiency with broad functional coverage.
  3. The study provides comprehensive empirical results. It validates the tool retrieval and reliability modules across multiple foundation models and establishes the OSMCP evaluation environment, which encompasses seven distinct categories of applications.
Critical weaknesses
  1. It is recommended to explain the initialization and resetting of execution history, the impact of task sequencing, and how to distinguish between tool failures and errors in the arguments passed to the model.
  2. Existing comprehensive comparisons primarily focus on self-constructed OSMCP; it is recommended to include representative frameworks from public benchmarks for comparison and to explain the sources of any observed advantages.
  3. The framework primarily employs conventional tools for retrieval and reliability management; it is recommended that its unique architectural features be further clarified.
Questions for authors

ref weakness

7 comments

Thank you for your thoughtful feedback. We appreciate your recognition that application tools can shorten a task while GUI actions keep other routes open, as well as your positive feedback on the evaluation across several models and desktop applications. Below, we address the three weaknesses you raised.


History, task order and argument errors (W1)

Thank you for asking what the agent remembers between tasks. We state exactly what is reset and what carries over, report three task orders, and explain how an argument error reaches the history.

History and reset. Each task starts with a fresh workspace. Tool history starts empty for each condition and task order and carries across tasks in that run. Records are keyed by backend, server and tool. With updates off, the menu uses relevance alone.

Effect of task order (LiveMCPBench, DeepSeek-V3-0324). We evaluated the original task order and two random orders. The table reports pass rates for separate updates-on and updates-off runs.

Task orderUpdates onUpdates off
Original order84.95%56.99%
Random order A81.72%59.14%
Random order B86.02%60.22%
Mean across orders84.23%58.78%

Across orders, the observed pass rates spanned 4.30 percentage points with updates on and 3.23 with updates off.

Tool failures and argument errors. A schema check can reject an argument before dispatch. In that case the history stays unchanged. Once a call reaches the backend, a rejected argument and an outage can produce the same failure flag. The controller can still use the error text within the task. In the Excel example (LiveMCPBench, DeepSeek-V3-0324), the reader reported that A1:Z100 exceeded the valid range A1:D16. The controller retried with the valid range and completed the task.

Public comparisons and sources of improvement (W2)

Thank you for suggesting a comparison on public benchmarks. We ran two controls on LiveMCPBench. The first asks which retriever supplies the most useful menu. The second asks how well each planner uses an identical menu.

Retrieval with a fixed executor (LiveMCPBench, DeepSeek-V3-0324). All methods used the same BGE encoder, execution budget and full tool catalog, with cross-task updates off. Menu sizes, recall and token costs come from a separate instrumented evaluation.

RetrieverTask pass rateReference-tool recallMean tools shownAgent tokens/task
Plain BGE Top-1555.91%51.46%15.0035,744
ToolDreamer [1]49.46%62.03%15.0044,392
AnyAct hierarchical retrieval60.22%76.09%9.1638,443

AnyAct covered more of the annotated solution with a shorter menu. With the same executor, it also achieved the highest pass rate and used 13.4% fewer tokens per task than ToolDreamer.

Planning with a fixed menu (LiveMCPBench, DeepSeek-V3-0324). All three methods received the same BGE Top-15 menu, shell and call budgets.

PlannerTask pass rateAgent tokens/task
ReAct [2]37.63%46,558
ToolTree [3]23.66%45,020
AnyAct executor55.91%35,744

AnyAct also achieved the highest pass rate and the lowest token cost when every planner saw exactly the same tools. This locates a second gain in how the controller uses the menu.

Architectural contribution (W3)

Thank you for asking what is distinctive about AnyAct. Its universal action layer keeps the tool menu compact while preserving ways to act when APIs change or fail. At task entry, AnyAct retrieves relevant semantic tools. Across tasks, it adjusts their ranks using the execution status each call already returns, with no additional model inference or external judge. GUI and command-line (CLI) actions remain available across applications.

AnyAct grounds screenshots, shell output and API results in a shared task history, so one interface can supply arguments for another. In Writer, both variants chose the same image-insertion API. Only the GUI-enabled agent found 1.png on the desktop and supplied the correct path. In GIMP, the agent read a photo’s path from the screen and used the CLI to rename it.

GUI ablation (OSMCP, DeepSeek-V3-0324 with Qwen3-VL-30B-A3B-Instruct). With the controller, MCP tools, CLI and action budget fixed, GUI access raised the mean final-state score by 8.59 percentage points.

Interfaces available to AnyActMean final-state score
MCP tools and CLI56.57%
MCP tools, CLI and GUI65.15%

References

[1] S. Sengupta et al., “ToolDreamer: Instilling LLM reasoning into tool retrievers,” in Proc. 19th Conf. European Chapter of the Association for Computational Linguistics (EACL), 2026.

[2] S. Yao et al., “ReAct: Synergizing reasoning and acting in language models,” in Proc. 11th Int. Conf. Learning Representations (ICLR), 2023.

[3] S. Yang et al., “ToolTree: Efficient LLM agent tool planning via dual-feedback Monte Carlo tree search and bidirectional pruning,” in Proc. 14th Int. Conf. Learning Representations (ICLR), 2026.

Reviewer_fNkuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AnyAct, a universal action layer for agents operating in large, dynamic tool ecosystems. It unifies GUI actions, command-line operations, and semantic APIs into a hybrid action space, and constructs a task-conditioned action subset via hierarchical progressive retrieval (LLM-based server shortlisting, BM25 filtering, dense reranking) performed once at task initialization. A test-time reliability evolution module updates per-tool reliability from backend execution status accumulated across tasks (with cold-start handling and a reliability floor), and a heterogeneous observation grounding module converts GUI screenshots, terminal logs, and structured API responses into a shared knowledge representation for reliability-aware action orchestration. The framework is evaluated on LiveMCPBench across multiple LLM backbones and on a newly introduced benchmark OSMCP (built from a subset of OSWorld applications, preserving its instructions and evaluators), where it reports an overall score of 77.27% under the reported system configuration (50 steps, roughly half the steps of most competitors).

Critical strengths

Practical problem with a coherent, well-motivated design. The framework directly targets tool scale, changing service availability, and heterogeneous observations. Hierarchical filtering keeps the candidate tool set within context budget, while GUI and command-line actions provide a fallback when APIs fail. For open-world agents, managing context overhead while staying robust to unreliable interfaces is a real, pressing bottleneck; the framework addresses it directly, and the pieces fit together coherently rather than being an ad-hoc system stack.

Broad evaluation with transparent scope, including a new benchmark. Experiments cover API use and computer operation, multiple backbones, retrieval and evolution variants, tool-switching cases, and description perturbations; OSMCP preserves the selected OSWorld instructions and evaluators and documents its catalog and task list, and the paper discusses catalog development, action granularity, and untested transfer. That lets me judge where the gains come from and how far they generalize, even though some transfer remains untested.

Critical weaknesses

Execution status provides limited evidence of tool quality and recovery. The reliability signal in Section 3.1.2 is backend execution-status flags, which mainly indicate whether a call executes normally; they do not establish semantic correctness or separate argument errors from service faults. Aggregating scores per tool may penalize interfaces that fail only for particular tasks or arguments, and a nonzero reliability floor does not guarantee that tools dropped from Top-K are invoked again after recovery. Figure 2 and the rate-limiting case do not isolate these effects. Reliability evolution is one of the paper's two central mechanisms, yet its core claim — that the agent adapts to non-stationary tool quality — is not directly demonstrated.

Grounding and hierarchical retrieval lack targeted validation. Section 3.2 has no controlled comparison against raw screenshots, ordinary screenshot summaries, or alternative knowledge aggregation, and does not quantify extraction errors or their downstream effects; task success alone cannot establish that initialization-time retrieval preserves all necessary tools. Figure 3 measures only local Top-10 stability and application-server coverage, and Appendix B.5 indicates that the candidate limit retains all cached tools, making dense and hybrid retrieval identical in that setting without server planning or task execution. Both the grounding module and the single-shot retrieval claim are load-bearing for the paper's efficiency and success-rate arguments, and the current evidence does not separate their contribution from overall system integration.

Questions for authors

What are the exact task-run settings for K, the server-filtering threshold (N_thresh), the BM25 candidate count, the dense encoder, and the grounding VLM and its prompts? Can Section 3.2 specify the knowledge schema, keyframe selection, text truncation, and context aggregation (comparable in detail to the reliability defaults in Appendix B.4)?

Is Equation (1) a conceptual objective or directly optimized — if optimized, what solver is used and how is λ chosen? For Equations (2)–(3), are similarity scores mapped to a nonnegative range so that multiplying by lower reliability consistently lowers ranking scores?

In Figure 1, does "Code Start" mean "Cold Start"? Do the documentation assessment, "High Doc Score & Low Success Rate," and "Re-Eval" branches concern session-boundary cache maintenance, and do they affect the reliability score used in Equations (4)–(6)?

Suggestions for additional evidence (would materially affect assessment if answered with new experiments)

Can you run controlled outages/recovery, argument errors, and semantic errors, alongside success-rate-only and failure-count-only controls, to isolate what the reliability mechanism actually recovers from?

Can you add grounding ablations, critical-tool recall, and success/cost measurements across tool scales and K values (with server planning and task execution, not only cached-tool retrieval)?

Can you compare against ToolDreamer and ToolTree under matched catalogs, backbones, and comparable call budgets? If interfaces or state replay prevent a fair comparison, please state those constraints explicitly.

6 comments

Thank you for your thoughtful and detailed review of our work. We appreciate your recognition that hierarchical filtering keeps the tool menu small and that GUI and shell actions give the agent another way forward when an API fails, as well as your positive feedback on the OSMCP task list and evaluators. Below, we address the weaknesses, suggestions and questions you raised.


Execution status, tool quality and recovery (W1 & Suggestion 1)

Thank you for this helpful suggestion. We agree that reliability evolution is a central mechanism and should be tested directly, so we ran controlled outage and recovery experiments.

How AnyAct responds to errors (LiveMCPBench, DeepSeek-V3-0324). Within a task, the controller can use an error message to correct its next call. An Excel reader rejected A1:Z100 and reported the valid range A1:D16. The controller retried with that range, read the data and completed the task. Across tasks, AnyAct updates each tool’s weight from the status the runtime already returns. Updating the weight requires no extra model call or judge.

Argument and semantic errors (controlled file-reader test, DeepSeek-V3-0324). A local schema rejection leaves the tool’s history unchanged. After dispatch, an invalid argument and a service outage can produce the same failure flag. When we injected invalid arguments, the full rule had 11 failed decisions in 36 requests, compared with 16 with updates off, 17 with the success-rate term alone and 11 with the streak term alone. The full rule also excluded the target on 3 of 12 eligible valid requests. On all three, the agent chose another reader and returned the correct value.

Outage experiment (controlled MCP outage, DeepSeek-V3-0324). We followed 16 target tools across 12 services. Each tool received six healthy requests, 12 during an outage and 12 after repair, in three request orders. All four rules saw the same requests. The first two columns cover the outage experiment. The last column reports verified reuse in the separate four-service recovery test described below.

RuleFailed-target calls during outage (12 services, of 576) ↓Correct calls while healthy (12 services, of 288) ↑Verified reuse after exclusion (4 services)
Updates off535 (92.88%)258 (89.58%)—
Success rate only491 (85.24%)256 (88.89%)3/11
Failure streak only156 (27.08%)258 (89.58%)0/17
Full rule156 (27.08%)257 (89.24%)2/18

The full rule made 379 fewer calls to the failed target than updates off, with similar healthy-phase accuracy. Eleven services improved and one tied. The file writer illustrates why the failure streak mattered. After six successes followed by three failures, its success rate was still 67%, above the 40% threshold. The third failure lowered its weight to 0.90 and it fell outside the next Top-15.

Intermittent failures give a different result. In a ranking replay with fixed BGE scores, we constructed a history in which the tool succeeded on only 34% of calls but never failed three times in a row. The streak-only rule kept it in the Top-15 in all 450 cases where it was initially visible. Adding the success-rate term moved it out in 388 of those cases. Occasional successes reset the streak, but the rolling success rate still captures the decline.

Recovery after exclusion (four-service MCP recovery test, DeepSeek-V3-0324). We followed tools that passed a health check after repair but still ranked outside the Top-15 because of their own penalty. Within the next 12 relevant requests, 2 of 18 eligible tool–order pairs re-entered the menu, were called and returned a verified correct result under the full rule.

Both full-rule recoveries involved the same geocoding tool in different orders. A query for Vienna put it in 14th place while its weight was still 0.90. DeepSeek called it and received the correct coordinates. Only then did the weight return to 1.00.

Grounding, critical-tool recall and single-shot retrieval (W2 & Suggestion 2)

Thank you for pointing out what a task score leaves open. We agree that grounding and retrieval should be validated separately from the full system, so we followed a visual value through to the saved file and measured initial-menu coverage, catalog scale and K.

Visual grounding (OSMCP, DeepSeek-V3-0324 with Qwen3-VL-30B-A3B-Instruct). All four conditions used the same state evaluator. Qwen3-VL either read pixels directly or first converted them to text. Each condition produced the same type of decision record for the controller.

Visual inputVisual records keptTasks with full creditAgent tokens/task
Raw screenshotsAll51.01%27,883
Ordinary summaryAll57.58%27,432
Task-directed extractionAll57.58%23,145
Task-directed extractionLatest only65.15%22,809

With all records retained, extraction and ordinary summaries each improved full-credit task success by 6.57 percentage points over raw screenshots. Extraction used 15.6% fewer tokens than ordinary summaries. Keeping only the latest extracted record gave the highest full-credit rate and the lowest mean token cost.

From a screenshot to a saved cell (controlled Calc screenshot test, DeepSeek-V3-0324 with Qwen3-VL-30B-A3B-Instruct). On fixed screenshots, task-directed extraction wrote 62.50% of target cells correctly, compared with 54.17% for raw screenshots and 50.00% for ordinary summaries. In a separate replay, we located 21 wrong values in intermediate extraction records. Replaying the records unchanged corrected none of the saved cells. Replacing just the wrong value, then letting DeepSeek regenerate the write arguments, corrected 13 of 21. This traces the error from what the agent read to what it wrote.

Initial reference-tool coverage (LiveMCPBench, DeepSeek-V3-0324). With the full catalog at K = 15, 63.6% of tasks passed when the initial menu contained every tool in the annotated solution. Even when a reference tool was missing, 52.6% still passed. We therefore tested when a second search was worth its cost.

Catalog size and K (LiveMCPBench, DeepSeek-V3-0324). One-shot retrieval fixes the menu at task entry. On-demand retrieval permits two additional searches. Both use the same shell, executor, budget and judge, with evolution off. The smaller catalog retains tools matching reference names across the benchmark.

CatalogRetrieval policyPass rate at K = 5K = 15K = 30
152 toolsOne-shot62.37%67.74%65.59%
525 toolsOne-shot54.84%59.14%64.52%
525 toolsOn-demand58.06%62.37%60.22%

At K = 15, moving to the full catalog lowered one-shot success by 8.60 percentage points and raised mean agent tokens per task from 26,954 to 36,669. Within the full catalog, raising K from 5 to 30 improved initial reference-tool recall from 63.28% to 79.08%. On-demand retrieval led at the two smaller budgets, but one-shot led by 4.30 points at K = 30. One-shot used 18.6%, 13.3% and 6.9% fewer tokens per task at the three K values, respectively.

What Figure 3 measures (LiveMCPBench ranking diagnostic, BGE encoder). Figure 3 measures Top-10 stability and application-server coverage. Its candidate limit retained every cached tool. To isolate lexical filtering, we compared dense ranking with BM25 prefiltering followed by the same dense ranker. The prefilter raised reference-tool recall from 41.74% to 46.67% in the full catalog. In the smaller catalog the rates were similar (53.52% vs. 52.89%). The benefit appeared when the prefilter had more competing tools to remove.

ToolDreamer, ReAct and ToolTree under matched settings (Suggestion 3)

Thank you for suggesting these baselines. We agree that a better menu and a better planner can each raise task success, so we separated the two comparisons.

Retrieval with a fixed executor (LiveMCPBench, DeepSeek-V3-0324). Both the encoder and downstream executor were fixed. Each retriever could supply up to 15 MCP tools. These baseline runs were separate from the catalog and K sweep. Menu sizes, recall and token costs come from a separate instrumented evaluation.

RetrieverTask pass rateReference-tool recallMean tools shownAgent tokens/task
Plain BGE Top-1555.91%51.46%15.0035,744
ToolDreamer [1]49.46%62.03%15.0044,392
AnyAct hierarchical retrieval60.22%76.09%9.1638,443

AnyAct covered more reference tools with a shorter menu. It used 13.4% fewer tokens per task than ToolDreamer and had the highest task pass rate.

Planning with a fixed menu (LiveMCPBench, DeepSeek-V3-0324). Each planner received the same BGE Top-15 menu and shell. The AnyAct row here uses the plain BGE configuration from the retrieval comparison.

PlannerTask pass rateAgent tokens/task
ReAct [2]37.63%46,558
ToolTree [3]23.66%45,020
AnyAct executor55.91%35,744

With retrieval fixed, AnyAct scored 18.28 percentage points above ReAct and 32.26 points above ToolTree. ToolTree reached the generation cap on 73.12% of tasks.

Baseline implementations. The repository linked in Appendix H of the ToolDreamer paper [1] returned HTTP 404 when we checked. We used its Figure 8 prompt to generate ideal tool descriptions, retrieved candidates with the shared BGE encoder and fused the rankings. This evaluates the paper’s inference-time variant without its alignment-trained weights. All planners had the same caps of 50 tool calls and 50 model generations per task. ReAct [2] retained its native loop of up to seven action–observation rounds. ToolTree [3] counted branch evaluations against the same budget.

References

[1] S. Sengupta et al., “ToolDreamer: Instilling LLM reasoning into tool retrievers,” in Proc. 19th Conf. European Chapter of the Association for Computational Linguistics (EACL), 2026.

[2] S. Yao et al., “ReAct: Synergizing reasoning and acting in language models,” in Proc. 11th Int. Conf. Learning Representations (ICLR), 2023.

[3] S. Yang et al., “ToolTree: Efficient LLM agent tool planning via dual-feedback Monte Carlo tree search and bidirectional pruning,” in Proc. 14th Int. Conf. Learning Representations (ICLR), 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We thank the Area Chair and reviewers for their time and constructive feedback. We are glad that the reviewers recognized the "coherent, well-motivated design" of AnyAct for a practical problem (fNku), described its framework design as "complementary" with "comprehensive empirical results" (WeLs), and appreciated its "useful implementation details and reliable reproducibility" (EjfX). Below we summarize our contributions and the new results from the rebuttal.

Contributions. AnyAct is a universal action layer for self-evolving agents. It unifies GUI, shell and MCP actions in one action space. It has three parts: (1) hierarchical progressive retrieval, which builds a compact semantic menu at task entry, (2) test-time reliability evolution, which reweights each tool from the execution status the runtime already returns, with no extra model call, and (3) cross-modal observation grounding, which turns pixels, shell text and API results into one shared task history. We also introduce OSMCP: 198 OSWorld tasks with 122 MCP tools labeled by granularity (45 atomic, 69 compound, 8 workflow).

On LiveMCPBench, AnyAct outperforms the Naive baseline (ReAct with MCP-Zero retrieval) on four backbones by 14.60–42.84 pp, reaching 77.42–93.55%. On DeepSeek-V3, reliability evolution alone raises the pass rate from 58.06% to 84.95%. On OSMCP, AnyAct surpasses the strongest baseline, UiPath Screen Agent, by 4.88 pp (77.27% vs. 72.39%) while using half its step budget (50 vs. 100 steps).

New results during the rebuttal. The reviewers asked four main questions. Does reliability evolution really adapt to failing tools, and do they come back? Are grounding and single-shot retrieval validated on their own? Are the comparisons fair? What do the architecture and OSMCP add? The new results answer each one.

Reliability evolution adapts to failing tools. In a controlled outage on 16 tools across 12 services (DeepSeek-V3), calls to the failed tool fall from 92.88% to 27.08%. Both update terms are needed. When a tool's success rate drifts down to 34% without a three-failure streak, the streak term alone keeps it in the menu in all 450 cases. Adding the success-rate term removes it in 388. Over the original and two shuffled task orders, the mean pass rate is 84.23% with updates and 58.78% without.

Tools come back after repair. We repeat one fixed task per tool, so no other task can bring the tool back. We push each weight to the 0.20 floor and keep the service down for 48 more hours. After a silent outage, all 12 verified tools return correct results on the first request after repair. With 96 requests during the outage, geocoding, file writing and presentation creation succeed on requests 9, 10 and 10, and the median over the 12 tools is 9.5.

Grounding improves both accuracy and cost. On all 198 OSMCP tasks (DeepSeek-V3 with Qwen3-VL), task-directed extraction reaches 65.15% full credit. Raw screenshots reach 51.01% and ordinary summaries 57.58%. Tokens fall by 18.2% against raw screenshots.

Single-shot retrieval is robust. On LiveMCPBench (DeepSeek-V3), tasks whose initial menu missed a reference tool still pass 52.6% of the time, against 63.6% with complete coverage. At 525 tools, raising K from 5 to 30 lifts the pass rate from 54.84% to 64.52%.

Both our retrieval and our planner outperform existing methods. We separate the two on LiveMCPBench (DeepSeek-V3). With a fixed executor, hierarchical retrieval passes 60.22%, against 49.46% for ToolDreamer and 55.91% for plain BGE Top-15. With a fixed menu, the AnyAct executor passes 55.91%, against 37.63% for ReAct and 23.66% for ToolTree.

GUI access and the controller each add value. On OSMCP (DeepSeek-V3 with Qwen3-VL), all systems share the same controller, tools and 50-action cap. GUI access adds 8.59 pp (65.15% vs. 56.57%). AnyAct is 11.11 pp above ReAct with the same tools (54.04%).

The components reinforce each other, and OSMCP measures it. GUI and shell stay available when retrieval misses, and reliability reweights the same menu without extra inference. On OSMCP, 17.17% of tasks use at least two MCP granularities, and 11.11% combine GUI and MCP actions.

Reviewer WeLs found the paper "a solid piece of work" and raised the rating from 2 to 3 after our responses. Reviewer EjfX asked for controlled failure and recovery scenarios, matched comparisons on OSMCP, and evidence for the value of OSMCP. In response, we added the outage and stress tests, the same-controller comparison and the multi-granularity usage statistics. Reviewer fNku asked for direct evidence on reliability, grounding and retrieval, and for comparisons with other methods. In response, we added the outage and recovery experiments, the grounding ablations, the coverage, scale and K analyses, and the ToolDreamer, ReAct and ToolTree comparisons. We have uploaded a revised manuscript with the changes marked in blue, and we will incorporate all new results into the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.