acceptodds
Under review as a conference paper at ICLR 2027

MCPDISCO: SEESAW-MITIGATED TUNING FOR MCP-NATIVE AGENTS

Abstract

The Model Context Protocol (MCP) is rapidly emerging as the substrate of the agentic web: a unified client-server standard that decouples LLM agents from an open-ended, ever-growing space of external tools. As agents become MCP-native, they face a decision-making regime that is qualitatively different from ad hoc function calling—one that requires jointly reasoning over a vast tool ecosystem and producing syntactically precise tool invocations. In this paper, we present \modelname, the first work to perform reinforcement learning (RL) tuning of MCP-native agents at ecosystem scale, spanning 70 heterogeneous MCP servers and 527 tools. We formalize the agent as a Thought-Markov Decision Process (Thought-MDP) that makes internal reasoning a first-class citizen alongside external actions, and we identify a reasoning-tool seesaw effect: under jointly optimized rewards, gains on reasoning tokens systematically trade off against tool-call correctness due to conflicting gradient directions. To resolve this, we propose 1) a disentangled mask reward mechanism that decouples reasoning quality from tool syntax and 2) reasoning-focused gradient surgery that densifies task-relevant reasoning. \modelname achieves higher success with fewer chain-of-thought tokens. On the out-of-distribution MCP-Atlas and GAIA benchmarks, \modelname maintains monotone gains, confirming that the learned behavior generalizes beyond the training server distribution. Model weight and implementation are available. https://anonymous.4open.science/r/MCPDisCo/README.md.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MCP, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hs2COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper trains an MCP-native agent (Qwen3-4B) with RL on a live pool of 70 MCP servers and 527 tools, where the agent has to search for the right tool and then call it correctly. The authors argue that when a reward for reasoning quality and a reward for tool-call correctness are simply added together, the two objectives pull against each other (a "reasoning-tool seesaw"). To address this, they apply each reward only to its own tokens (reasoning tokens vs. tool-call tokens) and project away the part of the reasoning gradient that conflicts with the tool gradient. On LiveMCPBench, success goes from 15.79% to 26.32% with slightly shorter chains of thought, and the paper reports small gains on MCP-Atlas and GAIA.

Critical strengths
  1. A good, timely idea. Giving reasoning tokens and tool-call tokens separate credit is simple and intuitive, and how reasoning and tool use interact during agentic RL is a question that matters beyond MCP.

  2. Training on real MCP infrastructure. RL on 70 live servers is closer to real deployment than most tool-use RL work, and the authors plan to release data, code and weights.

Critical weaknesses
  1. The seesaw itself is not convincingly shown.
  • There is no evidence of the trade-off in behavior (reasoning improving while tool correctness drops); the only evidence is a gradient-angle plot where both settings stay around 90 degrees, over the first 100 of 512 steps.
  • Li et al. (2026) already report reasoning-tool interference in agentic RL, which narrows the novelty.
  1. The experiments are too thin to support the claims.
  • The main result is 15 vs. 25 solved tasks out of 95 from a single run, and ablation variants differ by about 2 tasks.
  • The training-data section is missing ("Section ??") while training uses the same 70 servers as evaluation, and the paper contradicts itself on whether gradient surgery is needed.
Questions for authors
  1. Can you show the seesaw directly, e.g., reasoning reward and tool-call correctness over the full training run for the integrated vs. disentangled settings?
1 comment
AuthorsOpenReview ↗

Response to Reviewer hs2C

Thanks for your time, we respond to your concerns one-by-one.

Q: More analysis on seesaw.

A: For your information, we train with one-sided reward only/both rewards and track the changes of the different components. The curves are saved in https://anonymous.4open.science/r/seesaw_figs-8702/

From the visualizations, we can see that one-sided reward impairs the performance of the other side, justifying the seesaw effect. In the original paper, we visualize the gradient conflict windows during training and the cumulative negative-cosine integral in Figure 2 (a) and (b). The curves show that our designs impair the conflict, while also giving rise to the final performance correspondingly.

Q: Experiments are too thin (evaluation set is small).

A: We have evaluated MCPDisCo on LiveMCP (70 servers, 527 tools, 9 domains, 95 tasks), MCPAtlas (36 servers, 220 tools, 500 tasks), and GAIA (139 tasks for L1+L2). Overall, our method shows consistent improvements over all datasets, for which we think the empirical validation is significant enough.

We also additionally did the significant test, evaluating each method five times using different seeds. Results show significance with p<0.001.


Despite the mentioned concerns above, we would like to emphasize the generalization hierarchy of mcp calling over simple tool call (discussed in RQ3) and the OOD generalization ability of the trained policy, which is an important claim of our paper and barely mentioned by the reviewers.

Hope these can get you through the concerns. We welcome further interaction should any misunderstanding remain. Also, respectively, we merit a rejudgment on score, as a higher rating would increase the likelihood of sharing this work with the broader research community.

Reviewer_aegZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains a Qwen3-4B agent with GRPO on 70 live MCP servers from LiveMCPBench. A reasoning reward combines KeyBERT keyword density and the coherence between adjacent sentences. A tool reward scores the JSON format, the server and tool names, and the arguments against ground-truth calls. Token masks send each reward to its own span, and an asymmetric PCGrad step projects the reasoning gradient off the tool gradient. On the 95 LiveMCPBench tasks, success rises from 15.79% to 26.32%, and CoT falls from 1762 to 1663 tokens. The authors report OOD gains on MCP-Atlas and GAIA against ToolACE-8B and Hammer-7B.

Critical strengths
  1. The work runs RL on live, heterogeneous MCP servers and releases the 8k training set, the code, and the weights. Few public artifacts exist for MCP-native RL, so the release has practical value.

  2. Token-masked credit assignment is a simple and cheap design. It stops the reasoning reward from acting on JSON tokens. If a stronger evaluation confirms it, practitioners can adopt it with no extra cost.

Critical weaknesses
  1. The evidence cannot support the claims -> LiveMCPBench has 95 tasks, and the paper reports one greedy run with no seeds or intervals. The main gain is 10 tasks, and the ablation gaps are 2 and 4 tasks. -> Training uses the same 70-server pool as the evaluation. The data section is a broken reference (“Section ??”), so a reader cannot check for task leakage. -> The OOD results contradict the text. MCP-Atlas accuracy moves from 1.9% to 4.0%, and GAIA L2 falls from 0.105 to 0.070. The integrated-reward row that the text cites is absent from Tables 5 and 6.

  2. The paper does not show the seesaw mechanism or its cure -> Both conditions keep a mean gradient angle near 90°, which is the normal case in high dimensions. Disentanglement cuts cumulative conflict by 5.6%. -> No curve shows the reasoning reward rising while the tool reward falls. -> Section 2 states that gradient surgery is not necessary, but Section 4.4 applies it. -> The shorter CoT is not a side effect. D_rel divides by the reasoning length, so it acts as a length penalty. CoT counts also exclude trajectories that do not complete.

Questions for authors
  1. How did you build the 8k dataset, and do its queries or ground-truth calls overlap the LiveMCPBench tasks?
  2. What are the mean success rates and the intervals over three or more seeds for Base, -mask, -surgery, and the full method?
  3. Under integrated reward, do the training curves show R_reasoning rising while R_tool falls?
  4. What do integrated reward and SFT on the same 8k data score on MCP-Atlas and GAIA?
  5. Does CoT still shrink if D_rel counts keyword tokens without division by the reasoning length?
2 comments
AuthorsOpenReview ↗

Response to Reviewer aegZ

Thanks for your time. We respond to the concerns one-by-one.

Q: Evaluation set is small. Lack of multiple seeds. OOD test results are not satisfying enough.

A:

  • We have evaluated MCPDisCo on LiveMCP (70 servers, 527 tools, 9 domains, 95 tasks), MCPAtlas (36 servers, 220 tools, 500 tasks), and GAIA (139 tasks for L1+L2). Overall, our method shows consistent improvements over all datasets, for which we think the empirical validation is significant enough.
  • As for the "multiple seeds" concern, we append our evaluation over multiple seeds here. We evaluate each method five times using different seeds, the means and variances are listed below. From the results, we can see that the improvements are steady and significant with p<0.001.

MethodSeed 1Seed 2Seed 3Seed 4Seed 5MeanVar
Base15.7914.7416.8413.6815.7915.3681.44
MCPDisCo (-mask)24.2122.1123.1622.1124.2123.161.1
MCPDisCo(-surgery)22.1124.2124.2123.1621.0522.9481.88
MCPDisCo26.3228.4226.3226.3224.2126.3182.22

  • For the OOD results, the improvements are also significant with p<0.001.

Q: Details on dataset building.

A: We synthesized the data with real servers using gpt-oss-20b. The final training data was filtered using multi-dimensional gating including format, tasks success, diversity, etc.

Q: More analysis on seesaw.

A: For your information, we train with one-sided reward only/both rewards and track the changes of the different components. The curves are saved in https://anonymous.4open.science/r/seesaw_figs-8702/

From the visualizations, we can see that one-sided reward impairs the performance of the other side, justifying the seesaw effect. In the original paper, we visualize the gradient conflict windows during training and the cumulative negative-cosine integral in Figure 2 (a) and (b). The curves show that our designs impair the conflict, while also giving rise to the final performance correspondingly.

Q: Lack of direct comparison on "SFT" and "integrated reward"

A: We did have a comparison with "integrated reward," dubbed as MCPDisCo (-mask). For your information, we have appended a comparison with "SFT" here.


Methods (Success Rate)SFTIntergrated Reward (MCPDisCo -mask in the paper)MCPDisCo
Office16.1316.1325.81
Leisure21.4335.7121.43
Travel33.3316.6750.00
LifeStyle13.3340.0013.33
Finance7.1421.4321.43
Shopping22.2222.2233.33
Overall17.8924.2126.32

Results show consistent improvements.

Q: Concern on how to attribute the reasoning density increasement.

A: We have appended results with "w/o length penalty by division." Results are shown below.


Methods#Cot Tokens# ChainSuccess Rate#Cot Tokens# ChainSuccess Rate
Office2032.311.6819.351803.3914.5225.81
Leisure1414.666.6414.291349.218.9321.43
Travel2774.6711.3324.002715.5817.1750.00
LifeStyle1516.169.0726.671331.1712.1313.33
Finance1748.4012.3621.431665.2916.2921.43
Shopping991.107.4444.44818.678.1133.33
Overall1708.888.6323.161663.5313.3126.32

From the results, we can see that the "division" term (left three columns) did provide some contribution to the reasoning density. However, our designed component improves the reasoning density too.


Despite the mentioned concerns above, we would like to emphasize the generalization hierarchy of mcp calling over simple tool call (discussed in RQ3) and the OOD generalization ability of the trained policy, which is an important claim of our paper and barely mentioned by the reviewers.

Hope these can get you through the concerns. We welcome further interaction should any misunderstanding remain. Also, respectively, we merit a rejudgment on score, as a higher rating would increase the likelihood of sharing this work with the broader research community.

Reviewer_aegZOpenReview ↗

Thankyou for the rebuttal

Thankyou for the rebuttal but some of my concerns still remain and will keep my scores as is. The response does not provide confidence intervals or enough detail to assess the reported significance tests, and the evaluation still covers only 95 LiveMCPBench tasks.

Reviewer_6hehOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MCPDisCo, a method to disentangle conflicting gradients from different reward components in agentic RL training, especially tool-use and reasoning rewards. The authors first show the phenomenon of conflicting gradients in RL training and use token masks to isolate tokens so that each token is optimized by only one reward component. Then, gradient surgery is applied to further remove conflicting gradient components. MCPDisCo improves the success rate on LiveMCPBench from 15.79% to 26.32%, while slightly reducing the number of reasoning tokens.

Critical strengths

Important problem. The paper studies the interference between the abilities of reasoning and tool use in agentic RL, which is a meaningful and practically relevant problem, as agents are now co-improving in both reasoning and tool-use abilities.

Simple and intuitive method. MCPDisCo proposes reward decomposition into individual axes, combined with token-masked rewards to separate different axes and gradient surgery to remove conflict gradients. The methods are straightforward and well-motivated, and potentially applicable to other agentic RL settings.

Critical weaknesses

Insufficient comparisons with existing RL methods. The experiments mainly compare MCPDisCo with its own ablations, while missing some important RL baselines: (1) First, Table 1 does not include a standard GRPO baseline using both reasoning and tool rewards. (2) Second, the paper lacks comparisons with existing credit-assignment methods like ActFocus (https://arxiv.org/pdf/2605.14558) and DART (https://arxiv.org/abs/2602.00994), or gradient-analysis methods like GradAlign (https://arxiv.org/abs/2602.21492). A performance comparison of existing credit-assignment methods and some analysis of gradient analysis are needed to distinguish the contribution of MCPDisCo from existing works.

Limited experimental evidence. The evaluation set is small: in the main experiments, it uses only 95 tasks in LiveMCPBench, and no results across multiple seeds. Therefore, it is unclear how stable the reported improvements are, especially within individual domains, as the number of tasks there is even smaller. More importantly, the evidence for the reasoning–tool seesaw mechanism is not convincing: Figure 1 shows only a small decrease (5.6%) in gradient conflict, with the gradient angle around 90. Therefore, it is unclear whether this small change actually contributes to the larger improvement in task success. More analysis of how gradient conflict affects performance and training dynamics would strengthen this part.

Limited evaluation across models. All experiments are run on Qwen3-4B. This narrow evaluation scope makes it hard to tell whether the improvement of MCPDisCo can generalize to other model sizes and families. Since the paper claims to address a general reasoning-tool inference problem, testing more models with different backbones, and perhaps on different evaluation benchmarks, can strengthen the claim.

Questions for authors

Can the authors provide more direct evidence for the reasoning–tool seesaw effect? For example, they could train with one-sided reward only (e.g., only the reasoning reward or only the tool reward) and track the reward change of the other side during training. This would demonstrate whether optimizing one objective actually hurts the other. I think it is a more direct showcase for the seesaw effect than the gradient direction.

See Weakness 2: Currently, the experiments only report single-seed results without repeated runs. If the authors could train the model multiple times and report the performance mean and variance, that would be nice. I understand that the compute cost or time required for retraining may make this infeasible, and I think re-evaluating the same trained model multiple times is acceptable to reduce evaluation variance.

See Weakness 3: More model backbones and sizes are needed to demonstrate the generalizability of MCPDisCo. Currently, it is unclear whether the seesaw effect only occurs in the tested 4B model or is a general phenomenon even in larger models. Including these experiments would be meaningful for drawing solid conclusions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6heh

Thanks for your time. We respond to the concerns one-by-one.

Q. Table 1 does not include a standard GRPO baseline using both reasoning and tool rewards.

A: We do have included the standard GRPO with both rewards baseline, which is dubbed as "MCPDisCo (-mask)". Results show that our method improves over this baseline.

Q: Lack of comparisons with "ActFocus", "DART", and "GradAlign".

A: For your information, we append comparisons with "ActFocus", "DART", and "GradAlign" here.


Methods (Success Rate)ArtForceDARTGradAlignMCPDisCo
Office9.6812.9025.8125.81
Leisure35.7135.7114.2921.43
Travel25.008.338.3350.00
LifeStyle33.3340.0013.3313.33
Finance7.147.1428.5721.43
Shopping33.3311.1122.2233.33
Overall21.0518.9420.0026.32

Results show that our methods consistently outperform the mentioned baselines.

Q: Evaluation set is small. No results across multiple seeds.

A: We have evaluated MCPDisCo on LiveMCP (70 servers, 527 tools, 9 domains, 95 tasks), MCPAtlas (36 servers, 220 tools, 500 tasks), and GAIA (139 tasks for L1+L2). Overall, our method shows consistent improvements over all datasets, for which we think the empirical validation is siginificant enough.

As for the "multiple seeds" concern, we append our evaluation over multiple seeds here.

We evaluate each method five times using different seeds, the means and variances are listed below.


MethodSeed 1Seed 2Seed 3Seed 4Seed 5MeanVar
Base15.7914.7416.8413.6815.7915.3681.44
MCPDisCo (-mask)24.2122.1123.1622.1124.2123.161.1
MCPDisCo(-surgery)22.1124.2124.2123.1621.0522.9481.88
MCPDisCo26.3228.4226.3226.3224.2126.3182.22

From the results, we can see that the improvements are steady and significant with p<0.001.

Q: More evidence for the reasoning-tool seesaw. More analysis of how gradient conflict affects performance and training dynamics.

A: For your information, we train with one-sided reward only/both rewards and track the changes of the different components. The curves are saved in https://anonymous.4open.science/r/seesaw_figs-8702/

From the visualizations, we can see that one-sided reward impairs the performance of the other side, justifying the seesaw effect. In the original paper, we visualize the gradient conflict windows during training and the cumulative negative-cosine integral in Figure 2 (a) and (b). The curves show that our designs impair the conflict, while also giving rise to the final performance correspondingly.

Q: Testing more models with different backbones, and perhaps on different evaluation benchmarks

A: We have included different evaluation benchmarks (LiveMCP for in-domain test, and MCPAtlas/GAIA for OOD generalization test). Results show that MCPDisCo works on different benchmarks and have generalization ability to unseen servers and general agent tasks. (Due to the limit of resources, we could not train on different backbones.)


Despite the mentioned concerns above, we would like to emphasize the generalization hierarchy of mcp calling over simple tool call (discussed in RQ3) and the OOD generalization ability of the trained policy, which is an important claim of our paper and barely mentioned by the reviewers.

Hope these can get you through the concerns. We welcome further interaction should any misunderstanding remain. Also, respectively, we merit a rejudgment on score, as a higher rating would increase the likelihood of sharing this work with the broader research community.

Thank you for the additional experiments, and they address most of my questions. I appreciate the new baseline comparisons and reward analyses. Also, the repeated evaluations and additional benchmarks strengthen the results. However, two concerns remain: (1) it is unclear whether the five seeds are independent training runs or only repeated evaluations of the same trained model. (2) The cross-model generalization concern is still unresolved. Overall, the rebuttal strengthens the paper, but these concerns still exist, so I will keep my score at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.