- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes EvoMCP, a framework that combines iterative training-data construction with supervised fine-tuning to improve MCP-based tool-use agents. In each round, feedback from the current student’s rollouts guides the selection of learning targets and verified seed tasks. The framework then modifies task instructions, environment states, or both, while maintaining consistency with the corresponding verifiers. A teacher model executes the resulting tasks to produce demonstrations. For each fixed task, model-related execution failures are addressed through reusable recovery skills, while successful trajectories are audited and potentially regenerated to reduce tool-use errors and avoidable work. Selected verifier-passing demonstrations are used to fine-tune the student after removing teacher-only guidance. Accepted tasks serve as seeds for subsequent task generation, and the updated student provides fresh feedback for the next round.
Experiments using Qwen3.6-27B on Toolathlon, MCPMark, and MCP-Atlas report improvements over the initial student and a static SFT baseline across three evolution rounds. The paper also presents component ablations and analyses of task validity, teacher-trajectory quality, and skill-guided recovery to assess the contributions of its data-construction pipeline.
Critical strengths
-
Practical attention to training-data validity. The pipeline explicitly checks consistency between generated task instructions, environment states, and verifiers. It also distinguishes model-behavior failures from task or infrastructure failures before applying recovery. These are sensible design choices for constructing executable MCP training data.
-
Controlled evidence for guided recovery. The reported improvement supports the usefulness of recovery guidance for obtaining additional successful demonstrations beyond simply allocating more execution attempts.
Critical weaknesses
1. The main evaluation does not separate training feedback from held-out testing.
Section 5.1 and Appendix E.1 state that the same benchmark tasks are used both to guide the next round of data generation and to evaluate the updated student. My concern is that the model is being tested on tasks that have already shaped its training data. The gains therefore show benchmark-guided adaptation, but do not tell us how well the method generalizes beyond that feedback pool. I would like to see results on a separate held-out test set that is excluded from both feedback and data construction.
2. The baseline comparisons do not sufficiently establish the advantage of the proposed adaptive training strategy.
The public-model scores in Table 1 use different evaluation setups and do not directly compare training methods. In Table 2, the only baseline followed through all three rounds is Static SFT, which keeps reusing the same seed data. Feedback-free generation is only tested at R1, so it remains unclear how much the later gains depend on refreshed student feedback beyond the benefit of new training examples. Matching SFT steps alone also does not ensure comparable data-generation costs. I would like to see the feedback-free control extended across rounds, alongside few existing adaptive data-generation baselines with comparable student, teacher, and compute settings.
3. The incremental novelty over existing adaptive data-generation and trajectory-quality methods is insufficiently established.
Much of the pipeline has close precedents. Confucius [1] uses student uncertainty to guide data generation and iterative SFT, while CoEvolve [2] updates training tasks or data based on model feedback. The paper acknowledges most of these connections, but I am less convinced about what its specific MCP integration adds. The ablations show that the components help within this pipeline, without establishing an advantage over existing alternatives. For example, comparing skill-guided recovery with per-task reflection, and trajectory regeneration with step-quality-filtered SFT, would help clarify the contribution. As it stands, I see the work mainly as an integration of established ideas, with the value of its particular design choices still needing stronger evidence1.
References
[1] Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum. AAAI 2024.
[2] CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution. ACL 2026.
Questions for authors
-
Comparison with related methods. Could the authors compare EvoMCP with a representative prior data-construction approach under a comparable MCP training setup? This would help clarify what the proposed pipeline adds. The comparison should use the same student and teacher, with comparable generation and training budgets.
-
Held-out evaluation. Appendix A.3 and Figure 14 mention held-out environment variants, but I could not find aggregate student results on that split. Could the authors report these results, specify the split size, and clarify whether it was excluded from both feedback generation and training-data construction? If a held-out student evaluation is outside the intended scope, could the authors explain why benchmark-guided evaluation is sufficient to support the paper’s main claims?