acceptodds
Under review as a conference paper at ICLR 2027

ParaAgent: Reinforcing Parallel Acting in Open-World Tool Environments

Abstract

Language model agents are increasingly deployed in open-world tool environments, which require balancing exploring unknown capabilities and exploiting known ones. Existing methods face a performance–efficiency tradeoff: they either rigidly decouple exploration and execution or interleave them without coordination. We argue that the key lies not in whether to decouple or interleave them, but in how to coordinate them across granularities. We introduce ParaAct, a structured parallel-action loop that combines phase-level Exploration  Execution with action-level parallelism. To learn this loop, ParaAgent combines multi-agent cold-start demonstrations with reinforcement learning under multi-level advantage decoupling, making planning structure explicit and supervising it with step-, phase-, and trajectory-level rewards. Learning is supported by our ToolEnv, a scalable simulator grounded in 50,011 realistic tool interfaces. On two open-world tool benchmarks, ParaAgent achieves the best average success among all baselines, including GPT-4.1 systems, with the largest gains on multi-tool tasks. Behavioral analyses show that these gains stem from this action organization, highlighting its importance for capable and efficient open-world agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Open-world tool use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_azwXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the performance–efficiency tradeoff of LLM agents for open-world tool use. It presents PARAACT, a structured parallel action loop alternating between parallel exploration for capability gaps and DAG-based parallel execution of dependent tools. PARAAGENT learns this loop via two-stage training: multi-agent demonstration SFT for cold start, plus reinforcement learning with decoupled hierarchical rewards at step, phase and trajectory levels. A simulated environment TOOLENV built from real API data supports low-cost training. Tested on ToolBench and API-Bank, the 4B PARAAGENT surpasses many larger baselines including GPT-4.1, achieving strong multi-tool performance within ~3.74 turns per task. Improvements stem from structured action planning instead of more interactions or bigger models. Ablations validate the proposed training and reward design. Limitations are the 4B policy and reliance on simulation; future work scales models and extends to skill ecosystems.

Critical strengths
  1. The paper proposes an effective parallel planning method that improves model efficiency and tool-calling performance. The framework design and task decomposition demonstrate a degree of novelty.

  2. The detailed experimental analysis supports the design choices and demonstrates the contributions of individual modules.

Critical weaknesses
  1. The experiments use only one relatively small model, leaving the method’s generalizability across model sizes and capabilities unclear.
Questions for authors
  1. How does the proposed method perform on widely used tool-use benchmarks such as BFCL and τ-bench? Evaluation on these benchmarks would help assess its generalizability to more challenging tasks.

  2. The proposed method involves additional training, whereas the part of baselines do not. Can the authors provide comparisons under matched training conditions to clarify whether the improvements come from the proposed framework or the additional training?

4 comments
AuthorsOpenReview ↗

Response to Reviewer azwX (part 1)

Q1: The experiments use only one relatively small model, leaving the method’s generalizability across model sizes and capabilities unclear.

We thank the reviewer for the suggestion. Given the time constraints, we add experiments on Qwen3-1.7B, under the same settings as our 4B model. Our method remains effective, improving average success from 4.97 to 29.70 after SFT and further to 42.33 after RL, demonstrating its generalizability across model sizes. We will add these results to the revised paper and include larger models.

ModelMethodI1 Suc.I1 PathI2 Suc.I2 PathI3 Suc.I3 PathToolBench Avg. Suc.ToolBench Avg. PathAPI-Bank Suc.API-Bank Acc.
1.7BBackbone10.1315.074.7810.070.005.194.9710.116.0015.27
1.7B+ SFT27.85 (+17.72)39.43 (+24.35)31.74 (+26.96)40.90 (+30.83)29.51 (+29.51)22.40 (+17.21)29.70 (+24.73)34.24 (+24.13)16.00 (+10.00)32.82 (+17.55)
1.7B+ SFT & RL41.56 (+31.43)45.44 (+30.36)46.09 (+41.30)42.09 (+32.03)39.34 (+39.34)27.54 (+22.35)42.33 (+37.36)38.36 (+28.25)20.00 (+14.00)48.09 (+32.82)
4BBackbone22.3631.1324.3528.4519.6710.9022.1323.4916.0035.88
4B+ SFT35.44 (+13.08)40.17 (+9.04)35.65 (+11.30)39.76 (+11.31)24.59 (+4.92)32.68 (+21.78)31.90 (+9.77)37.54 (+14.04)20.00 (+4.00)43.51 (+7.63)
4B+ SFT & RL45.36 (+23.00)47.25 (+16.11)56.09 (+31.74)46.62 (+18.17)47.54 (+27.87)33.74 (+22.84)49.66 (+27.53)42.54 (+19.04)24.00 (+8.00)58.02 (+22.14)

Table 1. Model-size comparison (Success: Pass@1; Δ vs. backbone).

Q2: How does the proposed method perform on widely used tool-use benchmarks such as BFCL and τ-bench? Evaluation on these benchmarks would help assess its generalizability to more challenging tasks.

Thank you for the suggestion. Since BFCL and τ-bench (tool states are bound to specific tasks) are difficult to adapt to dynamic tool discovery and evolving execution feedback, we add MCPBench [1] and LiveMCPBench [2], two large-scale benchmarks built on real, running MCP servers. For both, we verified and configured all accessible tool interfaces, adapted them to the open-world setting, and used their official evaluation protocols.

  • MCPBench ( 66 tasks, 222 tools, and 27 servers). Ours achieves the highest task completion score across all models, outperforming the same-size Qwen3-4B baselines by 22.2 points. Notably, it also achieves the highest planning effectiveness at a lower cost.
  • LiveMCPBench ( 90 tasks, 652 tools, and 6 domains). ParaAgent-4B outperforms all 4B/32B baselines and GPT-4.1 ReAct, ranking second only to GPT-4.1 DFS while achieving the best balance between performance and cost.

Overall, these results demonstrate ParaAgent's effectiveness and generalizability to unseen real-world MCP servers, achieving a better performance–efficiency trade-off.

MethodTask Completion ↑Tool Hit ↑Planning ↑Avg. Turns ↓
GPT-4.1 + ReAct37.0%54.4%27.3%11.47
Qwen3-32B + ReAct29.4%47.4%22.8%10.45
Qwen3-4B + ReAct21.4%38.7%17.7%15.22
GPT-4.1 + DFS37.2%57.9%27.8%12.11
Qwen3-32B + DFS28.0%46.5%20.8%10.93
Qwen3-4B + DFS21.3%36.9%17.3%15.82
Qwen3-32B + Parallel29.0%49.4%32.5%8.25
Qwen3-4B + Parallel21.7%34.1%22.3%9.47
ToolOmni-4B26.8%45.3%20.3%12.53
DeepAgent-32B20.9%44.7%19.5%12.11
ParaAgent-4B43.9%58.4%44.5%4.02

Table 2. MCPBench performance.

Model / MethodPass@1 ↑Tool Hit ↑Avg. Turns ↓Total Tokens ↓
GPT-4.1 + ReAct40.00%43.19%5.4619,875
Qwen3-32B + ReAct23.33%27.51%4.8117,436
Qwen3-4B + ReAct21.11%31.08%10.0680,698
GPT-4.1 + DFS45.56%42.61%6.8234,769
Qwen3-32B + DFS24.44%32.85%5.0818,368
Qwen3-4B + DFS21.11%27.79%11.54118,224
Qwen3-32B + Parallel32.22%33.25%4.4917,914
Qwen3-4B + Parallel24.44%27.59%7.2456,570
ToolOmni-4B20.00%31.61%9.5619,559
DeepAgent-32B34.44%38.99%8.5968,913
ParaAgent-4B41.11%38.53%3.9118,694

Table 3. LiveMCPBench performance (Pass@1).

AuthorsOpenReview ↗

Response to Reviewer azwX (part 2)

Q3: The proposed method involves additional training, whereas the part of baselines do not. Can the authors provide comparisons under matched training conditions to clarify whether the improvements come from the proposed framework or the additional training?

Thanks for the suggestion. To better understand where the gains come from, we conduct three controlled analyses:

  • (1) Training-controlled comparison (Table 4). We train ReAct-4B with RL using the same RL data, settings, and outcome reward as ParaAgent. (i) ParaAgent achieves higher performance at lower cost under matched training conditions. (ii) It also outperforms ToolOmni-4B and DeepAgent-32B (SOTA models trained with RL ). Thus, with comparable training conditions, the framework still wins.
  • (2) Training-free comparison (Table 5). Under the same prompting, ParaAct on Qwen3-4B/32B outperforms ReAct with fewer turns, whereas naive parallel underperforms ReAct. On Qwen3-4B, ParaAct approaches DFS with fewer turns. These results show that the framework can improve the performance–interaction trade-off without training.
  • (3) Joint view of framework × training (Table 6). Training also improves ReAct and narrows the overall ToolBench gap, but benefits ParaAct more: ParaAgent gains +17.5 Avg with fewer turns, while RL increases ReAct's turns. ParaAct's advantage grows on the hardest subset, and its API-Bank deficit turns into a gain, showing that our training method amplifies the framework's structural advantages.

Overall, our framework and training are complementary: ParaAct provides basic gains, and ParaAgent training realizes them fully, especially on complex tasks. We will add these in the revision.

MethodToolBench I1 ↑I2 ↑I3 ↑Avg. ↑Turns ↓Path ↑API-Bank Suc. ↑Acc. ↑Turns ↓
ToolOmni-4B29.128.321.326.211.0139.120.046.6 ± 0.010.22
DeepAgent-32B39.737.827.935.113.3431.924.059.9 ± 2.77.76
ReAct-4B + direct RL57.1762.6144.2654.686.6732.122.0056.495.00
ParaAgent-4B58.465.755.759.93.3042.924.058.0 ± 0.04.19

Table 4. RL-trained baselines (Success: Pass@2).

Model / MethodToolBench I1 ↑I2 ↑I3 ↑Avg. ↑Turns ↓API-Bank Suc. ↑Acc. ↑Turns ↓
Qwen3-32B / ReAct46.848.327.941.05.1622.053.44.34
Qwen3-32B / DFS50.647.439.345.87.0722.056.54.57
Qwen3-32B / Parallel35.733.026.231.64.7510.048.14.33
Qwen3-32B / ParaAct52.5359.1349.1853.614.4720.057.254.22
Qwen3-4B / ReAct33.836.123.030.95.7614.046.24.44
Qwen3-4B / DFS47.553.042.647.730.8118.049.68.11
Qwen3-4B / Parallel31.032.224.629.34.7818.028.24.18
Qwen3-4B / ParaAct48.1051.3027.8742.424.0018.042.374.37

Table 5. Training-free comparison (Success: Pass@2).

SettingMethodToolBench Avg ↑I3 ↑Turns ↓API-Bank Suc ↑Acc ↑Turns ↓
Training-freeReAct30.923.05.7614.046.24.44
Training-freeParaAct42.427.94.0018.042.44.37
Δ+11.5+4.9−1.76+4.0−3.8−0.07
Matched RLReAct54.744.36.6722.056.55.00
Matched RLParaAct59.955.73.3024.058.04.19
Δ+5.2+11.4−3.37+2.0+1.5−0.81

Table 6. Framework × training (Success: Pass@2).

References

[1] Wang, Zhenting, et al. MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers. arXiv:2508.20453, 2025.

[2] Mo, Guozhao, et al. LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? arXiv:2508.01780, 2025.

AuthorsOpenReview ↗

Follow-up on Author Rebuttal

Dear Reviewer azwX,

We would appreciate it if you could take another look at our responses and let us know whether your concerns have been addressed.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Second Follow-up on Rebuttal

Dear Reviewer azwX,

We would like to kindly follow up once again on our rebuttal. We have added experiments on a smaller 1.7B model, two real-world MCP benchmarks, and controlled comparisons under matched training conditions to address your questions.

With the discussion period nearing its end, we would greatly appreciate your feedback on whether these results resolve your concerns.

Thank you for your time and consideration.

Reviewer_4YKXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how tool agents coordinate exploration, execution, and parallel tool calls. ParaAct alternates between exploration and execution, using parallel capability search during exploration and dependency-aware scheduling during execution. ParaAgent learns this structure through demonstration-based fine-tuning and RL, with gains on complex multi-tool tasks and using fewer interaction rounds.

Critical strengths
  1. The motivation is clear. Exploration is about finding missing capabilities, while execution requires managing dependencies between calls. Treating these stages differently is a sensible approach to a real problem.

  2. The structured protocol makes the training targets more concrete. Capability slots, dependency plans, and execution layers provide explicit behavior that can be checked and rewarded. Deployment still uses a single policy model, so it does not retain the full cost of the multi-agent demonstration system.

  3. The larger gains on multi-tool tasks fit the paper’s motivation. The same-reward comparison of normalization methods and the reward ablations also provide useful evidence for parts of the training design. Together, these results support the practical value of the full system.

Critical weaknesses
  1. The dependency supervision may not fit every task instance. The HARD/SOFT distinction and outcome gate help address this concern, and the phase-reward ablation shows an overall benefit. However, it remains unclear how reliably tool-level dependencies capture the dependencies between actual calls in a given task.

  2. The strongest gains are concentrated on multi-tool tasks. ParaAgent trails the strongest baseline on ToolBench’s single-tool subset and ties the best success rate on API-Bank. These results support its value for coordinating multiple tools, while evidence of broader gains across tool-use settings is more limited.

Questions for authors

Please see the weakness part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4YKX (part1)

Q1: The dependency supervision may not fit every task instance..... it remains unclear how reliably tool-level dependencies capture the dependencies between actual calls in a given task.

Thank you for the reviewer's comments. We would like to further clarify that task-level dependency is already considered at two levels:

  • (1) Representation Level. The global tool dependency graph distinguishes necessary (HARD) and task-dependent conditional (SOFT) dependencies. Thus, rather than simply labeling tool pairs as binary, we encode both necessary and conditional dependencies, providing a basis for capturing task-specific variations. Types are labeled independently by three strong LLMs (Kimi-K2.6, Qwen3-235B, DeepSeek-V4-Pro; GPT-5.1 resolves ties) and re-checked from task and early trajectories.
  • (2) Execution Level. In training, the local graph combines applicable HARD edges with SOFT edges declared by the model, supervising both planning and execution. Since SOFT edges can vary across tasks and phases, the reference graph adapts accordingly. Importantly, the outcome gates graph construction, constraining self-consistent but invalid SOFT dependencies. This enables fine-grained, phase-level dependency graphs that adapt online.

Manual verification. We manually annotate 240 sampled tool pairs each from within and outside execution trajectories. For in-trajectory pairs, the graph achieves 85.42% overall accuracy, with at least 82.5% across all dependency types, supporting its reliability.

Dependency TypeOut-of-Trajectory (240)In-Trajectory (240)
HARD (80)72.50%88.75%
SOFT (80)71.25%82.50%
NONE (80)67.50%85.00%
Overall70.42%85.42%

Table 1. Dependency graph validation (%).

Q2: The strongest gains are concentrated on multi-tool tasks.... These results support its value for coordinating multiple tools, while evidence of broader gains across tool-use settings is more limited.

Thank you for the suggestion. We consider our single-tool performance acceptable, as our 4B model matches GPT-4.1 and outperforms all other baselines, while achieving the best overall performance–efficiency trade-off.

We further add MCPBench [1] and LiveMCPBench [2], two large-scale tool-use benchmarks built on real, running MCP servers. For both, we verified and configured all accessible tool interfaces and extended them to the open-world setting.

  • MCPBench (66 tasks, 222 tools, and 27 servers). Ours achieves the highest task completion score across all models, outperforming the same-size Qwen3-4B baselines by 22.2 points. Notably, it also achieves the highest planning effectiveness at a lower cost.
  • LiveMCPBench ( 90 tasks, 652 tools, and 6 domains). ParaAgent-4B outperforms all 4B/32B baselines and GPT-4.1 ReAct, ranking second only to GPT-4.1 DFS while achieving the best balance between performance and cost.

Overall, these results demonstrate ParaAgent's effectiveness and generalizability to unseen real-world MCP servers, achieving a better performance–efficiency trade-off.

Model / MethodTask Completion ↑Tool Hit ↑Planning ↑Avg. Turns ↓
GPT-4.1 + ReAct37.0%54.4%27.3%11.47
Qwen3-32B + ReAct29.4%47.4%22.8%10.45
Qwen3-4B + ReAct21.4%38.7%17.7%15.22
GPT-4.1 + DFS37.2%57.9%27.8%12.11
Qwen3-32B + DFS28.0%46.5%20.8%10.93
Qwen3-4B + DFS21.3%36.9%17.3%15.82
Qwen3-32B + Parallel29.0%49.4%32.5%8.25
Qwen3-4B + Parallel21.7%34.1%22.3%9.47
ToolOmni-4B26.8%45.3%20.3%12.53
DeepAgent-32B20.9%44.7%19.5%12.11
ParaAgent-4B43.9%58.4%44.5%4.02

Table 2. MCPBench performance.

Model / MethodPass@1 ↑Tool Hit ↑Avg. Turns ↓Total Tokens ↓
GPT-4.1 + ReAct40.00%43.19%5.4619,875
Qwen3-32B + ReAct23.33%27.51%4.8117,436
Qwen3-4B + ReAct21.11%31.08%10.0680,698
GPT-4.1 + DFS45.56%42.61%6.8234,769
Qwen3-32B + DFS24.44%32.85%5.0818,368
Qwen3-4B + DFS21.11%27.79%11.54118,224
Qwen3-32B + Parallel32.22%33.25%4.4917,914
Qwen3-4B + Parallel24.44%27.59%7.2456,570
ToolOmni-4B20.00%31.61%9.5619,559
DeepAgent-32B34.44%38.99%8.5968,913
ParaAgent-4B41.11%38.53%3.9118,694

Table 3. LiveMCPBench performance (Pass@1).

[1] Wang, Zhenting, et al. MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers. arXiv:2508.20453, 2025.

[2] Mo, Guozhao, et al. LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? arXiv:2508.01780, 2025.

AuthorsOpenReview ↗

Follow-up on Author Rebuttal

Dear Reviewer 4YKX,

We would appreciate it if you could take another look at our responses and let us know whether your concerns have been addressed.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Second Follow-up on Rebuttal

Dear Reviewer 4YKX,

We would like to kindly follow up on our previous message. We have provided additional dependency-graph validation and results on two real-world MCP benchmarks to address your concerns.

As the discussion period is nearing its end, we would appreciate your feedback on whether these new results have addressed your remaining concerns.

Thank you again for your constructive review.

Reviewer_EAPzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Tool-using agents must both find tools and use them, and existing methods either fix the tool set up front or mix searching and calling without structure. The paper solves the issue with 2 steps: It proposes PARAACT, a loop that alternates "search" and "call" phases and parallel searches for missing capabilities and parallel calls scheduled by a dependency graph. It also proposes PARAAGENT, which learns this loop from multi-agent demonstrations and then RL training. In general: the paper follows some paradigm of work custom chain of thought + RL post-training

Critical strengths
  • Clear framing. The idea of structural having a loop of exploration and execution conditioned on prerequisites is easy to follow and well illustrated
  • Substantial resources. The paper builds a 50,011-tool library, a 242K-pair tool simulator and a 147K-edge dependency graph which is large scale and could be useful for the community
  • Experiments design is sound. Table 2 shows the effect of SFT, RL, separate reward normalization, and each reward component.
Critical weaknesses

Major:

  • Name and institute leaked in supplement
  • Novelty is incremental. The proposed method is conventional, self-define an chain-of-thought loop, then post-train on it. Planning tool calls as a dependency graph and running independent calls in parallel is the core idea of LLMCompiler (Kim et al., ICML 2024) and GAP (Wu et al., 2025, which also uses SFT + RL); neither is cited. Normalizing each reward separately follows GDPO (Liu et al., 2026), and the hierarchical rewards are hand-designed weights (Table 7) without sensitivity analysis.
  • Reproducibility is a concern. The released code does not contain the method. PARAGACT folder does not contain the core method code. The PARAGAGENT folder is a generic RL agent framework (verl / Agent-R1 with GSM8K, ReTool and search examples). It has none of the paper's own pieces: no <plan> parsing, no step or phase rewards (L1–L4), no separate reward normalization, no answer judge, no TOOLENV simulator or dependency graph, and no multi-agent data pipeline. The inference script uses ordinary function calling with a default HotpotQA-style question.
  • Potential Training Set Leakage. The released RL sample (5,504 of 16,512 prompts) contains 14 queries identical to StableToolBench test queries, with the same gold tools. Seven of them are from I3, which has only 61 tasks, so 11% of the hardest split appears in one third of the training data. I3 is exactly where the paper reports its largest gain (55.7 vs 45.9 for the best baseline), and the training reward directly rewards hitting those gold tools (Eq. 6). Minor:
  • Small and inconsistent evaluation numbers.
    • API-Bank has 50 tasks, so the "tie for best" (24.0) is 12 tasks, and the ablation gap of 24 vs 14 is 5 tasks.
    • Table 1 reports Pass@2 but Table 2 and Fig. 1 report Pass@1, and the same full model scores 59.9 vs 49.7. With greedy decoding, best-of-two runs is an unusual headline metric, and the 10-point gap suggests a noisy evaluation environment. There are no error bars on success.
  • The comparison does not show that the structure is what helps.
    • ParaAgent is trained on ToolBench tasks (8,665 RL and 4,471 SFT prompts) and on a simulator fitted to ToolBench API responses (105,877 pairs). Most baselines are untrained prompting setups.
    • The claim that "gains come from how actions are structured rather than from model scale" needs a flat ReAct-style agent trained with the same data and RL, and PARAACT prompting on GPT-4.1 or Qwen3-32B without training. Neither is reported.
Questions for authors
  • Can the authors release the code for the hierarchical rewards, the separate normalization, the PARAACT inference loop, TOOLENV and the dependency graph?
  • Can authors provide explicit results of multiple LLMs of (1) comparing PARAACT on non-finetuned backbone with ReAct on non-finetuned backbone; (2) comparing PARAACT on non-finetuned backbone with ReAct on fine-tuned backbone; (3) comparing PARAACT on non-finetuned backbone with PARAACT on PARAAGENT fine-tuned backbone? etc?
  • Why is Pass@2 the main metric under greedy decoding, and why does the same model score 59.9 (Pass@2) vs 49.7 (Pass@1)? What are the Pass@1 numbers for all baselines?
  • How does the method differ from LLMCompiler and GAP, and how does it compare with them empirically?
  • How accurate are the dependency-graph edges and the simulator responses against human labels or real API calls?
8 comments
AuthorsOpenReview ↗

Response to Reviewer EAPz (part 1)

Q0: Potential Training Set Leakage

Thank you for checking the training data. We apologize for packaging an RL sample drawn from an earlier candidate pool. The reported ParaAgent results use the final RL training file.

Source of the sample. The earlier candidate pool contains 16,512 tasks (ToolBench 9,220 / Toucan 5,407 / Simia 1,885). The original supplement used a one-third sample drawn with seed 42. The revised supplement includes the pool and a script that reproduces all 5,504 sampled records byte-for-byte.

Actual RL training file. The final file contains 16,384 tasks (ToolBench 8,665 / Toucan 4,840 / Simia 2,879). After lowercasing, removing punctuation, and collapsing whitespace, we found exact matches for 0 of the 14 flagged queries and 0 of the 765 StableToolBench test queries in this file. The revised supplement includes the file and the check script.

Comparison with the contaminated variant. We trained a "contaminated" variant under the same settings, replacing our original ToolBench subset with the 9,220-task subset while keeping Toucan and Simia, for a total of 16,939 tasks. The contaminated variant achieves high scores on ToolBench, while both models perform similarly on benchmarks without overlap (API-Bank, MCPBench, and LiveMCPBench).

RL training dataI1 Suc. ↑I1 Path ↑I2 Suc. ↑I2 Path ↑I3 Suc. ↑I3 Path ↑ToolBench Avg. Suc. ↑ToolBench Avg. Path ↑API-Bank Suc. ↑MCPBench Task Completion ↑LiveMCPBench Pass@1 ↑
Contaminated variant (16,939)59.9250.9169.5748.3649.1840.0059.5546.4222.0044.4044.44
Ours (16,384, final RL data)45.3647.2556.0946.6247.5433.7449.6642.5424.0043.9041.11

Table 0. Final RL data vs. contaminated variant (%). Suc. is Pass@1 on ToolBench and API-Bank.

Q1: Can the authors release the code for the hierarchical rewards, the separate normalization, the PARAACT inference loop, TOOLENV and the dependency graph?

Thank you for raising this. The training and inference code remains available in the revised supplementary code package, which includes the following components, along with the training scripts and configurations.

  • Action parsing: src/paraagent/paraact/protocol.py
  • Step and phase rewards (L1–L4), trajectory reward, and answer judge: `src/paraagent/rewards/tool_reward.py
  • Separate reward normalization: src/paraagent/training/trainer/ppo/core_algos.py
  • ParaAct inference loop: src/paraagent/paraact/loop.py (CLI paraact)
  • ToolEnv simulator: src/paraagent/toolenv/

The training datasets and the tool dependency graph are available separately at https://anonymous-hf.com/a/l6z9f7vulx2t/.

AuthorsOpenReview ↗

Response to Reviewer EAPz (part 2)

Q2: Can authors provide explicit results of multiple LLMs of (1) comparing PARAACT on non-finetuned backbone with ReAct on non-finetuned backbone; (2) comparing PARAACT on non-finetuned backbone with ReAct on fine-tuned backbone; (3) comparing PARAACT on non-finetuned backbone with PARAACT on PARAAGENT fine-tuned backbone? etc?

Thanks for the suggestion. To disentangle the effects of the ParaAct framework and ParaAgent training, we compare the combinations of action framework and backbone training:

  • (1) Training-free comparison (Table 1). Under the same prompting, ParaAct on Qwen3-4B/32B outperforms ReAct with fewer turns, whereas naive parallel underperforms ReAct. On Qwen3-4B, ParaAct approaches DFS with fewer turns. These results show that the framework can improve the performance–interaction trade-off without training.
  • (2) Comparison across training conditions (Table 2). We train ReAct-4B with RL using the same RL data, settings, and outcome reward as ParaAgent.
    • (i) Training-free ParaAct vs. trained ReAct. RL-trained ReAct-4B achieves higher scores than training-free ParaAct, while ParaAct uses fewer turns, indicating that the framework is the primary factor driving efficiency gains.
    • (ii) Training-free ParaAct vs. trained ParaAct. With the ParaAct framework held fixed, training raises average success from 42.42% to 59.9% and reduces average turns from 4.00 to 3.30.
    • (iii) Trained ParaAct vs. trained ReAct. Under the matched RL conditions, ParaAgent achieves higher average success with fewer turns.
  • (3) Joint view of framework and training (Table 3). Training also improves ReAct and narrows the overall ToolBench gap, but benefits ParaAct more: ParaAgent gains +17.5 Avg while reducing turns, whereas RL increases ReAct's turns. ParaAct's advantage widens on the hardest subset, and its API-Bank deficit turns into a gain, showing that our training method amplifies the structural benefits.

Overall, our framework and training are complementary: ParaAct provides basic gains, and while ParaAgent training further amplifies them. We will add all details in the revision.

Model / MethodToolBench I1 ↑I2 ↑I3 ↑Avg. ↑Turns ↓API-Bank Suc. ↑Acc. ↑Turns ↓
Qwen3-32B / ReAct46.848.327.941.05.1622.053.44.34
Qwen3-32B / DFS50.647.439.345.87.0722.056.54.57
Qwen3-32B / Parallel35.733.026.231.64.7510.048.14.33
Qwen3-32B / ParaAct52.5359.1349.1853.614.4720.057.254.22
Qwen3-4B / ReAct33.836.123.030.95.7614.046.24.44
Qwen3-4B / DFS47.553.042.647.730.8118.049.68.11
Qwen3-4B / Parallel31.032.224.629.34.7818.028.24.18
Qwen3-4B / ParaAct48.1051.3027.8742.424.0018.042.374.37

Table 1. Training-free comparison (Success: Pass@2).

MethodToolBench I1 ↑I2 ↑I3 ↑Avg. ↑Turns ↓Path ↑API-Bank Suc. ↑Acc. ↑Turns ↓
ReAct-4B + direct RL57.1762.6144.2654.686.6732.122.0056.495.00
Qwen3-4B / ParaAct48.151.327.8742.42431.0518.042.374.37
ParaAgent-4B58.465.755.759.93.3042.924.058.04.19

Table 2. Training comparison (Success: Pass@2).

SettingMethodToolBench Avg ↑I3 ↑Turns ↓API-Bank Suc ↑Acc ↑Turns ↓
Training-freeReAct30.923.05.7614.046.24.44
Training-freeParaAct42.427.94.0018.042.44.37
Δ+11.5+4.9−1.76+4.0−3.8−0.07
Matched RLReAct54.744.36.6722.056.55.00
Matched RLParaAct59.955.73.3024.058.04.19
Δ+5.2+11.4−3.37+2.0+1.5−0.81

Table 3. Framework × training (Success: Pass@2).

AuthorsOpenReview ↗

Response to Reviewer EAPz (part 3)

Q3: Why is Pass@2 the main metric under greedy decoding, and why does the same model score 59.9 (Pass@2) vs 49.7 (Pass@1)? What are the Pass@1 numbers for all baselines?

Thanks for the comment. Pass@2 counts a task as solved if at least one of two runs succeeds.

Why Pass@2 > Pass@1 under greedy decoding. Even with greedy decoding, trajectories may differ because the tool environment can vary across runs (network or service variability for online APIs, or LLM-generated responses from the official ToolBench simulator). Following prior works [1-3], we evaluate greedy agents over multiple runs. As shown in Table4, ParaAgent achieves 48.9 and 49.7 Pass@1 on ToolBench (mean 49.3) and 59.9 Pass@2 because the runs can succeed on different tasks.

The table below reports Pass@1 for all baselines. We will add these details to the final version to improve readability.

ParadigmAction / BackboneTB Run 1TB Run 2TB Mean ± SDTB Pass@2API Run 1API Run 2API Mean ± SDAPI Pass@2
Exploration-then-ExecutionReAct / GPT-4.123.222.622.9 ± 0.434.118.018.018.0 ± 0.020.0
DFS / GPT-4.124.423.724.1 ± 0.533.920.018.019.0 ± 1.420.0
DFS / T-LLaMA-7B8.97.18.0 ± 1.311.22.02.02.0 ± 0.02.0
Parallel / GPT-4.118.217.918.0 ± 0.228.022.020.021.0 ± 1.422.0
Parallel / DTA-8B9.78.29.0 ± 1.113.62.02.02.0 ± 0.02.0
ReWOO / GPT-4.125.828.527.2 ± 1.937.010.010.010.0 ± 0.012.0
ToolOmni / ToolOmni-4B18.318.318.3 ± 0.026.220.020.020.0 ± 0.020.0
Exploration-and-ExecutionReAct / GPT-4.142.835.639.2 ± 5.153.022.020.021.0 ± 1.422.0
ReAct / Qwen3-32B30.929.630.2 ± 0.941.020.022.021.0 ± 1.422.0
ReAct / Qwen3-4B18.124.321.2 ± 4.430.914.014.014.0 ± 0.014.0
DFS / GPT-4.142.636.439.5 ± 4.452.322.020.021.0 ± 1.422.0
DFS / Qwen3-32B34.835.435.1 ± 0.445.820.022.021.0 ± 1.422.0
DFS / Qwen3-4B29.133.931.5 ± 3.447.718.018.018.0 ± 0.018.0
Parallel / GPT-4.147.131.739.4 ± 10.955.522.020.021.0 ± 1.422.0
Parallel / Qwen3-32B19.622.421.0 ± 2.031.610.010.010.0 ± 0.010.0
Parallel / Qwen3-4B22.112.517.3 ± 6.829.316.02.09.0 ± 9.918.0
DeepAgent / DeepAgent-32B21.323.522.4 ± 1.635.124.022.023.0 ± 1.424.0
OursParaAct / ParaAgent-4B48.949.749.3 ± 0.659.922.024.023.0 ± 1.424.0

Table 4. Success on ToolBench and API-Bank: Pass@1 (runs, mean ± SD) and Pass@2 (%).

[1] Wang, Ziyi, et al. Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents. ACL 2026.

[2] Yao, Shunyu, et al. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, 2024.

[3] Yang, John, et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their constructive feedback. We highlight new results in light blue in the revised paper.

  1. Generalization to live MCP benchmarks (4YKX, azwX). ParaAgent-4B achieves the best balance between performance and efficiency on two benchmarks built on live MCP servers (Appendices D.1, E.5).
  2. Generalization across model scales (azwX). The same training pipeline substantially improves a smaller 1.7B backbone (Appendix E.6).
  3. Framework vs. training (EAPz, azwX). ParaAct outperforms ReAct without training. Under matched RL, ParaAgent still leads with about half the turns (Section 5, Table 2).
  4. Related methods (EAPz). We add LLMCompiler as a baseline, which ParaAgent substantially outperforms (Table 1), and discuss LLMCompiler and GAP in Related Work.
  5. Metric (EAPz). To improve readability, we now report Pass@2 for all success rates in the main text, and per-run Pass@1 and Pass@2 for all methods in Appendix E.4.
  6. Graph and simulator reliability (EAPz, 4YKX). The dependency graph reaches*85.42% agreement with human labels, and ToolEnv outperforms MirrorAPI on real API responses (Appendices E.2–E.3).
  7. Training data (EAPz). Corrected anonymous code and data have been uploaded.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.