- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AnyAct, a universal action framework for LLM-based agents that addresses challenges in large-scale and dynamic tool environments. It integrates hierarchical tool retrieval, test-time reliability evolution, and heterogeneous observation grounding to enable reliable and efficient coordination of GUI, CLI, and MCP actions.
Critical strengths
-
This paper focuses on the meaningful problem of agent self-evolving tools, pointing out the issues of scale dilemma, non-stationary and heterogenity of feedback formats.
-
The paper provides useful implementation details and reliable reproducibility.
Critical weaknesses
-
The methodological novelty of AnyAct appears somewhat limited, as its core components primarily integrate established techniques like hierarchical tool retrieval, heuristic reliability scoring, and VLM-based observation parsing. In particular, the proposed self-evolution mechanism relies on manually predefined thresholds and heuristic rules to update.
-
The evaluation of test-time reliability evolution is insufficient. As claimed in paper that adapting to dynamic tool reliability is a central motivation of AnyAct, the experiments lack controlled scenarios involving tool failures, recovery, or API changes. It remains unclear how effectively the proposed mechanism adapts to evolving tool environments.
-
The methods compared on OSMCP employ different action interfaces, tool capabilities, and step budgets, making it difficult to determine whether the performance gains stem from the proposed framework or the additional MCP tool capabilities. Moreover, the ablation results in Table 2 between AnyAct and AnyAct w/o GUI uses different step budgets, preventing a fair evaluation of the contribution of GUI actions.
-
The proposed benchmark OSMCP lacks dedicated evaluation settings or metrics to assess multi-granularity action collaboration, making it unclear whether OSMCP provides substantial evaluation value beyond an MCP-augmented version of OSWorld.
Questions for authors
-
How sensitive is the proposed reliability evolution mechanism to manually specified thresholds and penalty parameters?
-
How does AnyAct enable previously demoted tools to recover their reliability scores? If a tool is excluded from the retrieved Top-K actions due to repeated failures, how can the system detect its recovery without further executions?
-
In Table 2, do all baseline methods have access to the same 122 MCP tools as AnyAct? If not, could the authors clarify the differences and is the comparison still fair enough?
11 comments
Tool failures and recovery (W2 & Q2)
Thank you for asking for direct evidence of adaptation. We followed tools through a healthy phase, an outage and repair, and tracked the agent’s choices at each stage.
Tool failures (controlled MCP outage, DeepSeek-V3-0324). All four rules faced the same requests in the 12-service outage test. The last column gives the separate four-service recovery result, among tools still excluded by their own penalty at repair.
| Rule | Failed-target calls during outage (12 services, of 576) ↓ | Correct calls while healthy (12 services, of 288) ↑ | Verified reuse after exclusion (4 services) |
|---|---|---|---|
| Updates off | 535 (92.88%) | 258 (89.58%) | — |
| Success rate only | 491 (85.24%) | 256 (88.89%) | 3/11 |
| Failure streak only | 156 (27.08%) | 258 (89.58%) | 0/17 |
| Full rule | 156 (27.08%) | 257 (89.24%) | 2/18 |
The full rule reduced the failed-target call rate by 65.80 percentage points. Eleven services improved and one tied. The streak term alone matched the full rule during these abrupt outages. Correct calls to healthy targets remained similar across the four rules.
How an excluded tool recovers (four-service MCP recovery test, DeepSeek-V3-0324). Under the full rule, 2 of 18 eligible tool–order pairs re-entered the Top-15 and returned a verified correct value within 12 relevant requests after repair. All 18 had passed a separate health check and were still excluded by their own penalty at repair. Both recoveries involved geocoding in different orders. On a request for Vienna, the tool ranked 14th while its weight was still 0.90. DeepSeek selected it and received the correct coordinates. The successful call raised its weight to 1.00.
Action interfaces, tool capabilities and step budgets in Table 2 (W3 & Q3)
Thank you for pointing out the limits of the comparison in Table 2. We agree that action access and the controller’s contribution should be evaluated separately.
Did all submitted baselines have the same tools?. The end-to-end GUI baselines used their native GUI interfaces. The other external frameworks used their reported interfaces and budgets. Both AnyAct variants had access to the same 122 application MCP tools.
Matched comparison (OSMCP, DeepSeek-V3-0324 with Qwen3-VL-30B-A3B-Instruct). All systems used the same shell, application MCP tools, initial desktops and evaluators, with a common cap of 50 controller decisions and 50 tool actions. Both GUI-enabled systems used the same vision model. The desktop ReAct baseline [1] saw the full tool catalog each round. AnyAct retrieved up to 15 MCP tools from that catalog.
| Method | Mean final-state score |
|---|---|
| ReAct + MCP | 54.04% |
| AnyAct w/o GUI | 56.57% |
| AnyAct | 65.15% |
With the controller and budget fixed, GUI access raised the mean final-state score by 8.59 percentage points. With GUI, shell and MCP access fixed, AnyAct scored 11.11 points above the ReAct baseline.
Thank you for your careful review and constructive comments. We appreciate your recognition of the difficulty of acting with changing tools and different forms of feedback, as well as your positive feedback on the implementation details for reproducing AnyAct. Below, we address each point in turn.
Methodological novelty and hand-set parameters (W1 & Q1)
Thank you for asking what evolves when the thresholds are set by hand. To answer this, we state what a tool call changes at test time and vary one parameter at a time.
What is distinctive. On a real desktop, an agent must find relevant tools in a large catalog, obtain the arguments those tools require, and keep working when an interface fails. AnyAct retrieves the semantic menu, preserves GUI and shell access, and passes observations from one interface into the next tool call. Updating the menu from execution status requires no extra model call or judge.
What evolves during execution (controlled MCP outage, DeepSeek-V3-0324). The parameters stay fixed. What changes is each tool’s weight and thus its place in the menu. A file writer still matched the request after its service failed. Three consecutive failures lowered its weight to 0.90 and moved it outside the Top-15 even though its rolling success rate remained above 40%.
The rolling success rate captures intermittent failures. Repeated execution errors can lower a tool’s rank even if occasional successful calls reset its streak.
Parameter sensitivity (controlled ledger test, DeepSeek-V3-0324). We changed one parameter at a time while tools competed for the same two menu slots. The first outcome measures calls to the unavailable target. The second includes failures after choosing an alternative.
The floor had the largest effect on calls to the failed tool. Across 24 decisions per setting, the failed tool appeared once at the lower floor and nine times at the higher floor. DeepSeek called it each time it appeared. Raising the success-rate threshold also reduced failures by letting the rule react earlier in a decline. Increasing the streak penalty had a smaller effect on target calls over the tested range.