acceptodds
Under review as a conference paper at ICLR 2027

Model Context Agent: Validated Agentic Workflow via Agentic Planning

Abstract

The Model Context Protocol (MCP) has become the standard for connecting large language models to external tools, but it standardizes only how a service is accessed, leaving every client to rediscover how it should be used. That burden falls on a reactive loop that chains tool calls one at a time and judges its own success; on realistic multi-step tasks it is unreliable, costly, and offers no guarantee the goal was met. We argue that the service should ship its own expert. We introduce the Model Context Agent (MCA), an abstraction in which planning, execution, validation are provided by the server rather than delegated to a general-purpose model, and a server-agnostic engine that realizes it: a grounded planner compiles a goal into an ordered tool-call sequence, a deterministic executor replays it with no model in the loop, a verifier confirms the outcome by re-querying the live system instead of trusting a self-report, and a task generator distills verified runs into a knowledge base the planner reuses. We instantiate the same MCA on four production servers spanning 127 tools—GitHub, PostgreSQL, Playwright, and Yahoo Finance—and evaluate it on 125 multi-step tasks under twelve backbones from five providers. Matched against a ReAct loop with the same backbone, tasks, and verifier, it verifies 71% of tasks against 40%, is significantly ahead in 34 of 48 campaigns, and uses up to 12.6 times fewer tokens. On hard suites, the knowledge base an MCA distills from its own verified runs adds 7–12 verified tasks per backbone on PostgreSQL and GitHub and cuts replans by up to 62%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, tool use, Model Context Protocol, agentic planning, execution verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qS9MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to use a service-specific agent with deterministic executors, built in verifiers and planner, and a knowledge of service call patterns to replace MCP based tool calling using general purpose LLMs. The overarching idea is that the best agent to interact with a service's tools and APIs is one defined by the service. To illustrate this, the authors define a model context agent - an agent shipped by the expert, with access to deterministic executors and LLM based verifiers and planners. The MCA is provided with a user goal, generates an ordered plan (including tools, arguments, post-hooks, etc - effectively generating a program). Built into the planner is a live catalog (up to 3 exemplars from the knowledge base), and guardrails that are derived from failures from earlier runs. The plan is validated before it is executed - so an invalid plan (with missing arguments, for ex.) would not be executed. This plan is then executed by a deterministic executor (no LLM involved). If a step fails, executor just returns the plan annotated with the outcome, and the planner re-plans. Finally, the execution of the task is verified through format, static and dynamic checks, and verified only if ALL checks pass. Dynamic checks are written by an LLM, but are grounded on the tool's implementation - and are restricted to helper functions from that check the state of the system via deterministic checks. The number of re-planning turns is bounded and running out of turns fails the task.

In addition to the MCA, the paper proposes knowledge base construction over time, building it from existing, verified runs. If a run fails initially and is fixed at a later stage, the mistakes serve as grounding guidance that steers the models away from common failure modes. This is effectively converting episodic memory to procedural memory.

The paper compares this method against a ReAct based agent that uses the MCP tools directly, and compare MCA against the baseline on 125 tasks, using 12 different models, and reporting the execution and verification rate. The results show that MCA outperforms the ReAct agent on most tasks, and uses fewer tokens. The benefit of the knowledge base is seen primarily in long horizon, harder tasks. The conclusions drawn from these experiments are that the workflow adds reliability to this process, verification is best done with tools and processes provided by the service, and that having a constantly updated tool-use specific memory helps reduce error rates.

Critical strengths
  • Proposes a solid argument for a service-centric architecture for a tool use agent, and the system proposed has a well-defined architecture that bounds the number of retries per stage, proposes deterministic executors and relies on tools for verification while checking the actual state instead of relying on LLM's claims/judgements. The general idea of having a service specific agent that is an expert in using the service makes sense.
  • The experiments and analyses done go into a fair bit of detail around the planning step, how the retries are bounded, how the KB is constructed, issues with retrieval, how the KB benefit scales with difficulty, and benchmarks it against 12 different models.
Critical weaknesses
  • The primary issue I take with the current experimental setup is the lack of separation of cause - for lack of better phrasing. The system proposed introduces components that the ReAct system lacks:
    • A dedicated, retry-able planning step
    • A knowledge base of common failures
    • Access to service defined verifiers It would be good to get a sense of the lift provided by each part of this system. How much was because of the planning phase vs the verifiers that were made available, vs the KB?

Additionally, from page 16, it looks like the context provided to the ReAct agent might be different from what MCA has access to (Appendix A.9, page 30) - the ReAct agent only sees the rendered text. This makes the baseline a little weak - It would be interesting to provide the ReAct agent with an Agent as a Judge style verifier instead, instead of a simple react loop.

  • The same model plans and generates the claims to verify, but the model might miss something that needs to be verified. We need human calibration on the automatically generated checks to ensure that the generated checks were sufficient, preferably with false positive / false negative rates on subset of completed tasks.

  • There's only one reported run per task - which does not give us an idea of run-to-run variance, especially with the verification. There is some variance clearly (we see this in the hard PostGresSQL tasks), but it's not entirely clear how much. While it might be hard to rerun everything, can we generate this on a subset?

Questions for authors

See weaknesses for the questions that affect my assessment.

Additional questions: Of the four domains, only finance permits replanning after failed checks (instead of failing the task). Why is this the case? Can we report the results from both (was it considered verified?) vs (was the answer correct?) perspectives for finance?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qS9M (Part 1/3)


Thank you for the positive assessment, for finding the service-centric argument solid and the bounded, deterministic architecture well defined, and for the concrete suggestions! We hope the following results and clarifications can address your concerns.

What we ran. Beside the MCA we ran three ablations (−retrieval, −guardrails, −recovery layer), derived a fourth from the full runs' first attempts (−replanning). We also ran five baselines on the same tools and backbone, each under the loop’s 20-turn budget: (1) the paper’s ReAct loop, unchanged; (2) ReAct shown the decoded file contents; (3) ReAct with the MCA’s guardrails and retrieved precedents; (4) ReAct with an agent-as-a-judge; and (5) LangGraph’s plan-and-execute template. We re-ran the paper's GLM-5.3-Flash and DeepSeek V4.1 Flash campaigns with the paper's settings on GitHub (28 tasks) and PostgreSQL (22 tasks). To make the comparison more reliable, every run is scored twice. The first score comes from our verifier (the paper's metric; on PostgreSQL the MCA run's checks grade the loop's final database, as in the paper; on GitHub the verifier writes checks for each run). The second score comes from checkers (named as task checkers in the revised paper): read-only checks of the live state, generated by Claude Opus 5.5 from the task prompts alone. The task checkers are not ground truth but offer a second, unbiased perspective: written by a model that is neither backbone, blind to our verifier, and the same for every arm. After the runs, one stale GitHub task (its repositories had moved) was dropped, so GitHub is scored on 27 tasks. p is the exact two-sided McNemar test, paired by task and backbone.

W1. Separation of cause.

Our component analysis shows that the MCA’s gain comes from validated, model-free execution with repair.. Plan-and-execute keeps the planning step but executes each step with an LLM agent, and it verifies only 16 of 44 PostgreSQL runs, against the MCA’s 40 (p < 10⁻⁶). Without bounded replanning, the MCA verifies 33 of 54 GitHub runs instead of 45 (p < 0.001).

Table A. Change in verified runs when one part is removed or replaced (task checker change in brackets; p by verifier). GitHub: 54 runs (27 tasks × GLM, DeepSeek), MCA 45 verified (43 task checker). PostgreSQL: 44 runs (22 tasks × GLM, DeepSeek), MCA 40 (34).

PartGitHubPostgreSQL
Executor and plan language → plan-and-execute (stands in for −executor)−17 (−11), p < 0.001−24 (−21), p < 10⁻⁶
Bounded replanning−12 (−12), p < 0.001−4 (−4), p = 0.12
Recovery layer−12 (−5), p = 0.004+3 (0)
Knowledge base (retrieval)−1 (−1); first plan −8 (−8), p = 0.08−2 (+1)
Guardrails−2 (−4); first plan −4 (−7)0 (+1)
  • The planning step pays off only with the MCA's execution. Plan-and-execute keeps a retry-able planner but runs each step with an LLM agent: it reported PostgreSQL results it never queried, and on GitHub ran out of turns in 19 of its 22 task checker failures. In the MCA the plan is validated against the live catalog, run without a model, and returned to the planner, annotated, when a step fails; that repair alone rescues 12 of the 21 failed GitHub first plans.
  • The knowledge base and guardrails make the first plan right. Without them, first-plan verified successes on GitHub fall from 33 to 25 (p = 0.08) and 29, and replanning repairs most of the difference at more replans and calls (runs needing a repair: 18 → 27 and 20). Where a replan cannot supply the missing knowledge, on the hard suites, retrieval adds 9–10 verified tasks per backbone on PostgreSQL (shipped corpus) and 7–12 on GitHub (grown corpus; the paper's Table 3), and 40 → 53 on the covered hard GitHub tasks (p = 0.007).
  • The layers back each other up. Without recovery, first plans succeed in 21 of 54 instead of 33 (p = 0.008). Replanning then repairs most runs (by task checker the final loss is 5), but repaired plans drop steps the first attempt completed, so fewer runs end on one plan the verifier can confirm.
  • Verifiers. The MCA's verifier decides acceptance after execution and does not feed the planner (except on finance, see Q), so by design it adds no success of its own; it adds an auditable verdict (W2). Your agent-as-a-judge suggestion measures what an in-loop verifier adds to ReAct (part 2).
AuthorsOpenReview ↗

Response to Reviewer qS9M (Part 2/3)


W1 (continued): ReAct with the MCA's context, and with an agent-as-a-judge.

Given the MCA's context or an agent-as-a-judge, ReAct still verifies fewer tasks than the MCA on both servers, significantly on PostgreSQL (26–27 vs. 40 of 44, p < 0.001), and every ReAct variant needs 4.3–6.6× the MCA's model calls.

Table B. Same runs as Table A; p vs. the MCA by verifier.

ArmGitHub verified (p)GitHub task checkerPostgreSQL verified (p)PostgreSQL task checkerLLM calls, GH / PG
MCA45434034133 / 108
ReAct, the paper's loop30 (0.004)2623 (< 10⁻⁴)34822 / 463
ReAct (shown decoded file contents)34 (0.03)4423 (< 10⁻⁴)34730 / 463
ReAct + MCA guardrails, precedents40 (0.33)4527 (< 0.001)32723 / 474
ReAct + agent-as-a-judge39 (0.15)4526 (< 0.001)37879 / 674
Plan-and-execute (LangGraph)28 (< 0.001)3216 (< 10⁻⁶)131,295 / 1,065
  • Context. We acknowledge that the paper's GitHub loop saw only the rendered text of file resources (Figure 10); Figure 5's caption, which says otherwise, is wrong and will be corrected. After shown the decoded contents, the loop rises from 30 to 34 verified runs (26 → 44 by task checker), and the MCA still verifies more (45, p = 0.03) with 5.5× fewer calls. The same loop also received the MCA's own guardrails and its top-3 retrieved precedents, verbatim (the "guardrails, precedents" row).
  • Agent-as-a-judge. An LLM judge with read-only tools must approve before the loop may stop; its calls come on top of the loop's 20 turns. On PostgreSQL it sent back 20 of the 56 answers it judged, and the arm verifies 3 more runs than the plain loop, still far below the MCA (26 vs. 40). On GitHub the arm verifies 39 vs. 34, but the judge itself changed no outcome: it returned FAIL on 2 of the 48 answers it judged, and neither run recovered. The arm makes 6.2× and 6.6× the MCA's calls.
  • At a matched budget. Within 10 model calls the MCA has all 45 GitHub and 40 PostgreSQL verified successes; ReAct + agent-as-a-judge has 14 and 3 (by task checker: 43 and 34 vs. 15 and 6). A successful MCA run takes a median of 2 calls, a successful judge-loop run 15–16.
ArmGitHub: ≤ 4 / ≤ 10 calls / full budgetPostgreSQL: ≤ 4 / ≤ 10 / full budget
MCA45 / 45 / 4540 / 40 / 40
ReAct0 / 17 / 341 / 6 / 23
ReAct + MCA guardrails, precedents0 / 17 / 403 / 6 / 27
ReAct + agent-as-a-judge0 / 14 / 390 / 3 / 26
Plan-and-execute0 / 0 / 280 / 2 / 16

W2. Calibration of the generated checks.

The MCA acts as the service's expert: it carries out each task and verifies it from its plan and trace, and the task checkers of part 1, a second perspective rather than a ground truth, confirm 84% of its acceptances; the verifier accepts 92% of the MCA runs that meet the goal.

  • Why re-query the live state. The tasks act on live repositories, databases, pages and prices that change over time, so an answer key written in advance goes stale (one GitHub task did during the rebuttal: its repositories had moved). A read-only query of the live state after the run is the closest available test of success; the task checkers test it the same way.
  • Why three layers. Deterministic format checks (every planned step ran and returned a result) and static checks (arguments and results obey fixed rules: each tool's schema, and on finance invariants and bounds) cover execution and form, so model-written dynamic checks cover only the task's content.
  • Calibration. A model can still leave a requirement unchecked, as the reviewer note.
MCA runsPostgreSQL (44)GitHub (54)Both (98)
False positive: accepted, fails the task checker8 of 40 accepted (20%)6 of 45 (13%)14 of 85 (16%)
False negative: meets the task checker, rejected2 of 34 (6%)4 of 43 (9%)6 of 77 (8%)
Same verdict as the task checker34 (77%)44 (81%)78 (80%)
  • The false positives are narrow. On PostgreSQL the MCA left the database in the required state in all 44 runs (272 of 272 state requirements); the 8 accepted runs missed only report deliverables, mostly listings that came back empty or incomplete. On GitHub the 6 are plan choices the checks inherited, such as listing 3 of 11 search results or forking the wrong repository.
  • The false negatives are mostly a single model-written check with a wrong expected value, or one that could not resolve a count or an issue number; in the PostgreSQL cases every deterministic check passed.
AuthorsOpenReview ↗

Response to Reviewer qS9M (Part 3/3)


W3. Run-to-run variance.

A second full round of runs reproduces the outcomes: under fixed task checkers the MCA keeps the same outcome in all 44 PostgreSQL runs and in 93% of the 54 GitHub runs scored, and its verified totals move by at most 3 on PostgreSQL, GitHub and finance. Round 2 re-ran every task with the same arms and settings (444 runs: the MCA and its matched loop on PostgreSQL, the browser and finance, and the MCA on GitHub, all under GLM-5.3-Flash and DeepSeek V4.1 Flash).

Table C. MCA verified tasks: the paper's runs → round 1 (Nov 10–12) → round 2 (Nov 12–13).

Server (tasks)GLM-5.3-FlashDeepSeek V4.1 FlashSame result, rounds 1–2
GitHub (28)19 → 21 → 2223 → 24 → 2493% (task checker, 27 tasks)
PostgreSQL (22)20 → 18 → 1920 → 22 → 20100% (task checker)
Finance (40)38 → 36 → 3738 → 38 → 4091% (verifier)
Browser (35)20 → 14 → 1420 → 26 → 1866% (own checks)
  • The two rounds differ by at most 2 verified tasks per server and backbone, except DeepSeek on the live-web browser (8).
  • By our verifier, the MCA's lead on PostgreSQL and finance replicates: PostgreSQL 40 vs. 23 and 39 vs. 19 of 44 (p < 10⁻⁴ in both rounds), finance 74 vs. 63 and 77 vs. 53 of 80 (p = 0.027 and p < 10⁻⁶). By the task checkers, PostgreSQL is 34 vs. 34 and 34 vs. 33.
  • The browser is noisy for both arms: Live pages change run to run, and about a third of tasks change outcome between rounds in both the MCA and the loop (by their own checks, MCA 40 → 32 and loop 45 → 46 of 70).
  • Earlier replications point the same way. In the paper’s campaigns, we ran PostgreSQL under MiniMax-M3 three times per arm. The MCA verified 18, 21 and 21 of 22 tasks, against 16, 13 and 16 for the loop. The MCA’s outcome was identical across all three runs on 17 of 22 tasks, compared with 10 for the loop. The hard PostgreSQL suite was likewise run three times per arm and backbone, and Table 3 of the paper reports the majority vote.

Q. Finance: why replan on a failed check, and verified vs. answer correct.

On finance the MCA verifies more runs than ReAct in the paper's campaigns and in fresh runs (454 vs. 287 of 480; 74 vs. 63 of 80, p = 0.027), and on the fixed-date tasks of the paper's GLM and DeepSeek campaigns it also answers more questions correctly by MCP-Universe's evaluators (62 vs. 52 of 72, p = 0.006).

Why replanning. A finance question is computable only if the fetched series supports it (for example, enough daily bars for a 50-day mean), and only a dynamic check can see that. The finance port therefore returns a failed check to the planner, at most three times; the other servers keep the verifier off the planner's feedback path. In the paper's campaigns, without its check-triggered replans the MCA still verifies 362 of 480 runs against the loop's 287 (p < 10⁻⁶). This rationale was lost from Appendix A.1; the revision will state it in the merged workflow section (§2.2–2.3) and restores it in Appendix A.1.

Table D. Answers are written by the run's own backbone.

FinanceRuns per armVerified, MCA vs. ReActAnswer correct, MCA vs. ReAct
Paper's campaigns, all tasks480454 vs. 287 (362 without check-triggered replans)–
Paper's GLM and DeepSeek campaigns, 18 fixed-date tasks72–62 vs. 52 (p = 0.006)
Fresh runs, matched loop (20 turns, same thinking)8074 vs. 63, p = 0.027 (57 without check-triggered replans)56 vs. 50 (p = 0.31)
  • The MCA's pipeline returns the fetched data; one LLM call by the same backbone then writes a short program that computes the answer from it. Tasks without a fixed date are graded against live market data, so for the paper's runs we compare on the 18 fixed-date tasks of the four GLM and DeepSeek campaigns (GLM-5.3-Flash, GLM-5.3, DeepSeek V4.1 Flash, V4 Pro); the fresh runs were graded right after each task.
  • "Verified" on finance confirms that the evidence an answer needs was fetched. On the fresh runs this does not yet give significantly more correct answers (56 vs. 50), so the revision will report both columns and rests the finance claims on answer correctness.

We will incorporate these details and make it more clear in the revised paper. Thanks again for your insightful suggestion!

Reviewer_ykiNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper proposes the Model Context Agent (MCA), a framework that achieves a higher success rate and lower token consumption than the ReAct framework by adopting plan–execute–verify components. In addition, a task generator distills verified runs into a knowledge base for the planner to reuse. The authors point out that current agents such as ReAct operate as interactive loops that call tools sequentially, which leads to three limitations: (i) unreliability on multi-step tasks, (ii) resending an ever-growing history at every turn, and (iii) task success being self-reported by the model.
  • The authors address these problems with three principles: (P1) a service should provide a domain expert in addition to its tools; (P2) the model should not be involved in execution or verification, except for writing the plan and the verification code; and (P3) task success should be judged against the actual state of the service.
  • The framework consists of four components: a planner, an executor, a verifier, and a task generator. The planner turns the task into an ordered tool-call plan. The executor runs the plan deterministically without an LLM, with bounded replanning when a step fails: it sends the plan, annotated with each step's outcome, back to the planner within a limited budget. The verifier then confirms the outcome by querying the live service rather than relying on self-assessment. The task generator creates new goals and stores the attempts that passed the verifier without replanning in the knowledge base.
  • The paper reports that MCA outperforms ReAct, verifying 71% of tasks versus 40% overall. It is significantly ahead in 34 of 48 campaigns and behind in none, and it consumes fewer tokens in 45 of 46 campaigns while taking less time. The evaluation covers 1,500 runs across 4 MCP servers with 127 tools, 125 tasks, and 12 backbone models.
Critical strengths
  • This paper proposes MCA, a framework that shifts from the existing approach, in which the agent accessing an MCP server carries out the entire task on its own, to one in which a domain expert agent on the MCP server performs the task. This addresses two problems of the existing approach: the history growing ever larger, and the agent judging its own execution results regardless of whether the execution actually succeeded.
  • The claim that an MCP server should provide a domain expert agent along with its tools can be regarded as a new discussion in the MCP context. It seems to share a similar perspective with A2A, but appears to be specialized for MCP servers.
  • The experiments are extensive (1,500 runs) and were conducted on real live services, which makes the results reliable. They also show that the framework can be applied to diverse settings such as REST APIs, SQL databases, and browsers.
  • Out of 48 campaigns, MCA is significantly better than ReAct in 34 and significantly worse in none, showing consistent results across servers and backbones.
Critical weaknesses

W1. My main concern is that the novelty of the work needs to be further verified against prior work. Papers such as ReWOO (Xu et al., 2023) also propose frameworks that perform planning and binding, but since the paper provides no comparison with them, the differences are not clearly shown. LLMCompiler (Kim et al., ICML 2024) also executes tool calls without a model after planning. It would be helpful if the authors could explain in more detail how their work differs from these approaches, as well as from HuggingGPT (Shen et al., NeurIPS 2023). In addition, the proposed method is not compared with the plan-and-execute agent templates provided by LangChain or LangGraph, and its performance is evaluated only against ReAct. The related work section states that, on the MCPMark benchmark, a plain tool-calling loop outperformed ReAct and Codex for GPT-5; since ReAct is the only baseline used, the baselines are insufficient. It should be made clear whether the performance gain comes from the similar planner → executor → replanner structure in general or from contributions unique to MCA. Furthermore, there is no ablation of components other than the knowledge base and replanning (guardrails, deterministic executor, recovery layer), so it is impossible to isolate which component the performance gain comes from.

W2. The shipped corpus and the grown corpus, which are the key comparison conditions in the experiments, are not sufficiently defined in the method section (Section 2) and first appear in the caption of Table 3 and in the results section. In particular, the rectification pass that builds the grown corpus is effectively a separate algorithm, yet the main text only repeats a brief description of the same idea twice. Its detailed settings (replan budget, stopping condition, selection criteria, etc.) appear only in Appendix A.6, and the settings for PostgreSQL are not described at all. I would ask the authors to explicitly define both corpora in Section 2.4 and to describe the rectification pass as part of the method.

Minor:

  • Sections 2.2 and 2.3 convey almost the same information.
  • Overall, the notation is not easy to follow. In the Setting of Section 2.3, rr is defined as the reward, whereas in line 12 of Algorithm 1 it refers to the number of replans, as it also does in the inequality r≤Rr \le R in the Termination paragraph on page 4. In addition, πθ(ak∣u,Tσ,a<k,o<k)\pi_\theta(a_k \mid u, T_\sigma, a_{<k}, o_{<k}) is defined in the Setting but never appears again; I believe it would be better to avoid introducing a definition or formulation that is not used in the rest of the paper. In line 11 of Algorithm 1, 1m\mathbf{1}^{m} in the expression if c≠1m then return \textbf{if } c \neq \mathbf{1}^{m} \textbf{ then return } appears for the first time without being defined. Also, many new symbols are introduced in Algorithm 2, such as AσA_{\sigma}, α\alpha, KαK_{\alpha}, GG, and safeσ\mathrm{safe}_{\sigma}, but most of them are used only within Algorithm 2 and its accompanying description, which adds notational burden without much benefit.

Note to authors and AC: The weaknesses have been condensed and grouped for clarity. This is an editorial change only — the content and my overall evaluation remain the same as in the original version.

Questions for authors

Q1. The paper states that "the prevailing agent is a per-turn policy πθ\pi_\theta that also decides when to stop and reports its own success." Could the authors provide references for this claim? Concrete agent systems should be cited as the point of comparison.

Q2. On page 4, the finance port relaxes oracle independence by replanning on failed checks, and the rationale is deferred to Appendix A.1; however, I could not find it there. Since this may affect the comparability of the results, please briefly state the reason and its impact in the main text.

Q3. In the token cost comparison, it is not specified whether the tokens spent on verifier-generation calls were counted toward MCA or ReAct on each server. The baseline description in Section 3.1 is ambiguous about how the verifier was applied on PostgreSQL and Playwright, and Section 3.2 only mentions that ReAct incurs no verification cost on PostgreSQL by design. Please clarify how the verifier was generated and applied on each server, and to which arm its cost was attributed.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ykiN (Part 1/3)

Thank you for the detailed review! We hope the following results and clarifications can address your concerns.

What we ran. Beside the MCA we ran three ablations (−retrieval, −guardrails, −recovery layer), derived a fourth from the full runs' first attempts (−replanning). We also ran five baselines on the same tools and backbone, each under the loop’s 20-turn budget: (1) the paper’s ReAct loop, unchanged; (2) ReAct shown the decoded file contents; (3) ReAct with the MCA’s guardrails and retrieved precedents; (4) ReAct with an agent-as-a-judge; and (5) LangGraph’s plan-and-execute template. We re-ran the paper's GLM-5.3-Flash and DeepSeek V4.1 Flash campaigns with the paper's settings on GitHub (28 tasks) and PostgreSQL (22 tasks). To make the comparison more reliable, every run is scored twice. The first score comes from our verifier (the paper's metric; on PostgreSQL the MCA run's checks grade the loop's final database, as in the paper; on GitHub the verifier writes checks for each run). The second score comes from checkers (named as task checkers in the revised paper): read-only checks of the live state, generated by Claude Opus 5.5 from the task prompts alone. The task checkers are not ground truth but offer a second, unbiased perspective: written by a model that is neither backbone, blind to our verifier, and the same for every arm. After the runs, one stale GitHub task (its repositories had moved) was dropped, so GitHub is scored on 27 tasks. p is the exact two-sided McNemar test, paired by task and backbone.

W1. Is the gain from the shared planner → executor → replanner structure, or from the MCA?

The gain comes from the MCA's own mechanisms: LangGraph's plan-and-execute template, run over the same tools, backbone and budget, verifies 16 vs. 40 of 44 PostgreSQL runs (p < 10⁻⁶) and 28 vs. 45 of 54 GitHub runs (p < 0.001).

Table A. GitHub: 27 tasks × GLM, DeepSeek (54 runs). PostgreSQL: 22 tasks × GLM, DeepSeek (44 runs).

ArmGitHub verified (p vs. MCA)GitHub task checkerPostgreSQL verified (p)PostgreSQL task checkerLLM calls, GH / PG
MCA45434034133 / 108
Plan-and-execute (LangGraph)28 (< 0.001)3216 (< 10⁻⁶)131,295 / 1,065
MCA − guardrails43 (0.79)3940 (1)35135 / 112
MCA − recovery layer33 (0.004)3843 (0.25)34152 / 103
MCA − retrieval44 (1)4238 (0.69)35142 / 107
MCA − replanning (first attempt)33 (< 0.001)3136 (0.12)30–
ReAct shown decoded file contents34 (0.03)4423 (< 10⁻⁴)34730 / 463
ReAct + MCA guardrails, precedents40 (0.33)4527 (< 0.001)32723 / 474
ReAct + agent-as-a-judge39 (0.15)4526 (< 0.001)37879 / 674

By the task checkers, no new arm differs significantly from the MCA except −replanning on GitHub (p < 0.001) and plan-and-execute (GitHub p = 0.019, PostgreSQL p < 10⁻⁶).

  • Why the generic structure falls short. Plan-and-execute's LLM executor and replanner track progress themselves: it reported PostgreSQL results it never queried, and on GitHub, where its step agent spends a model call on every tool call, 19 of its 22 task checker failures ran out of the 20-turn budget. It used 9.7–9.9× the MCA's model calls.
  • The ablations we runned. The deterministic executor cannot be removed without an LLM taking over each step, which is what plan-and-execute does, so that arm stands in for −executor (it also replaces the MCA's planner and plan language). Without replanning the MCA verifies 33 instead of 45 GitHub runs (p < 0.001; 31 vs. 43 by task checker, p < 0.001). Without the recovery layer it also verifies 33 (p = 0.004), mostly a loss in what the verifier can confirm: repaired plans drop steps the first attempt completed, and by task checker the loss is 5.
  • The layers back each other up. Guardrails and retrieval change the final count little because replanning repairs what they would have prevented: without them, first-plan verified successes on GitHub fall from 33 to 29 and 25 (p = 0.08), and the runs needing a repair rise from 18 to 20 and 27. Without recovery the first plan succeeds in 21 (p = 0.008).
  • Supported ReAct. With the MCA's guardrails and precedents or an agent-as-a-judge, ReAct still verifies fewer tasks than the MCA on both servers, significantly on PostgreSQL, at 4.3–6.6× its model calls.
AuthorsOpenReview ↗

Response to Reviewer ykiN (Part 2/3)


W1 (continued): how the MCA differs from ReWOO, HuggingGPT, LLMCompiler and LangGraph.

All four plan ahead, and the first three bind earlier outputs into later inputs; none of them validates the plan against the server's live catalog, repairs it from the current state of a stateful service, decides success on live state, or ships with the service, and to our knowledge no prior agent combines these four properties.

ReWOOHuggingGPTLLMCompilerLangGraph plan-and-executeMCA
Plansteps with evidence variablestasks with <resource>-id dependenciesDAG with placeholdersnatural-language stepstool calls with bindings $s_j.path into nested earlier results
Executed bytool workersexpert modelsparallel dispatcher, no modelan LLM agent per stepdeterministic executor, no model
Plan checked against the live tool catalog before any callnonononoyes
After a failureno replanningno replanningan LLM joiner replans (stateless calls)an LLM replanner revises the plancontinues from the current state with completed writes marked; re-creations and not-yet-visible writes absorbed
Success decided byan LLM solver's answeran LLM's responsean LLM joineran LLM replannerre-querying live state with invariants and format checks, off the planner's feedback path
Ships with the service (guardrails, verified precedents from its own task generator)nonononoyes

We will add this comparison to §4 and as a row of its comparison table (Table 4).

  • Why these are not our baselines. ReWOO targets token efficiency and LLMCompiler parallel function calling, both over stateless calls, and neither is an MCP agent. A port to a stateful MCP server must add the rows above that they lack, so it would test our re-implementation of the MCA. LangGraph's template is the generic structure in runnable form, so we add it as the plan-and-execute baseline in Table A.
  • Why ReAct. MCP-Universe, whose tasks and prompts we adopt, evaluates every model with ReAct as "the most popular framework" (§4.1 of that paper).
  • MCPMark's plain tool-calling loop. Six of the paper's 48 campaigns already run it through native tool calling (§3.1). Among the three compared campaigns, MCA verifies more in all three: 18 vs. 7 (GPT-5.6 Luna, PostgreSQL), 8 vs. 7 (GPT-5.6 Luna, browser), 20 vs. 6 (Claude Sonnet 5, browser).

W2. Shipped and grown corpora, and rectification.

We will define both corpora in the knowledge-base section (§2.4, which becomes §2.3 after the §2.2/§2.3 merge) and specify rectification as part of the method, in three steps with its settings on every server; the rectified entries are what lift the hard GitHub suite, 69 → 81 under GLM-5.3-Flash (p = 0.043) and 40 → 53 on the 65 tasks they cover (p = 0.007).

  • The shipped corpus is Algorithm 2's output: goals from the task generator, planned with retrieval off and a replan budget of 0, admitted only if executed end to end and verified, at most 3 entries per tool signature (41 browser entries passed a relaxed gate, Table 7).
  • The grown corpus adds a rectification pass: it re-runs rejected generations under a wide replan budget and admits a sequence only after it re-executes from scratch on a clean state and verifies.
ServerReplan budget and stopping ruleAdmitted if verified afterEntries added
GitHub10; stops when a step fails the same way 3 times≥ 2 replans32
PostgreSQL10; each replan on a clean database≥ 1 replan145
Browser15; replans on any failure, including a failed check2–15 replans87
AuthorsOpenReview ↗

Response to Reviewer ykiN (Part 3/3)


Minor.

  • §2.2 and §2.3 will be merged into one subsection, "MCA Workflow".
  • r denotes only the reward. The replan count will be Algorithm 1's loop counter k (line 12: k = 0), and π_θ's turn index becomes ℓ.
  • m, the number of checks, and 1ᵐ will be defined at Algorithm 1's lines 10–11.
  • π_θ will be cited (see Q1) and used where the baseline is defined in §3.1.
  • Algorithm 2's own symbols (𝒜_σ, α, K_α, G, safe_σ) will be replaced by words.

Q1. References for "the prevailing agent … decides when to stop and reports its own success."

The sentence describes the loop of ReAct and of the MCP benchmarks we build on, and we will cite them there: ReAct (Yao et al., 2023b), MCP-Universe (Luo et al., 2025) and MCPMark (Wu et al., 2026). In each, the loop ends when the model emits an answer instead of an action, and that answer is its report of the outcome; software agents stop the same way, through SWE-agent's submit and OpenHands' finish actions (Yang et al., 2024; Wang et al., 2025).

Q2. Finance: why replan on a failed check, and its impact.

In the paper's campaigns, replanning on failed checks does not create finance's lead: without any check-triggered replan the MCA still verifies 362 of 480 runs against the loop's 287 (p < 10⁻⁶).

  • Reason. A finance question is computable only if the fetched series supports it (for example, enough daily bars for a 50-day mean), and only a dynamic check can see that. The finance port therefore returns a failed check to the planner, at most three times; the other servers keep the verifier off the planner's feedback path. This rationale was lost from Appendix A.1 by mistake.
  • Impact. In the paper's runs, 92 of the MCA's 454 verified finance runs needed a check-triggered replan (Table 5). In the new runs such replans matter more: 17 of its 74 verified runs needed one (57 without them, against the matched loop's 63), and 11 of its 56 correct answers (the loop answers 50 of 80 correctly) come from such runs. Answer correctness is therefore the measure we report for finance.
  • The revision will state the reason in the merged workflow section and restores it in Appendix A.1; §3.2 will report both strict counts.

Q3. How the verifier was generated and applied, and who paid for it.

The MCA pays for writing its checks on every server and the loop only on GitHub, and the MCA still uses fewer tokens in 45 of the 46 campaigns with token accounting. The verifier writes its dynamic checks from a run's plan and trace (a loop's calls are first bridged into the same schemas); format and static checks are deterministic. On PostgreSQL a failed check may be rewritten once before the verdict (a check-repair round, used in 16 of the 44 new MCA runs). How each server grades the loop:

ServerHow the loop is gradedCheck-writing tokens charged toTokens, ReAct ÷ MCA (Table 2)
GitHubthe same verifier writes checks for each arm's runboth arms2.4–12.6×
PostgreSQLthe MCA run's check battery runs on the loop's final databaseMCA only0.4–1.7×
Playwrightthe MCA run's verifier is applied to the loop's traceMCA only3.8–8.1×
FinanceMCP-Universe's answer evaluators; our verifier on the loop's trace as an evidence-only checkMCA only1.4–4.9×
  • On PostgreSQL, writing checks is 59–68% of the MCA's tokens. On planning tokens alone the loop spends 2.8–4.8× the MCA's under 10 of 11 backbones (1.06× for GPT-5.6 Luna's native loop), and in the new runs it makes 4.0–4.6× more LLM calls.
  • We will state this per server in §3.1 and in Table 2's caption.

We will incorporate these details and make it more clear in the revised paper. Thanks again for your insightful suggestion!

Reviewer_RxF5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Model Context Agents (MCAs), which provide planning, execution, and validation as server-side capabilities. The core workflow comprises agentic planning, which compiles a user goal into an ordered tool-call plan; deterministic execution, which replays the plan in order with no model in the execution loop; and three-layer validation against the live system, using format, static, and dynamic checks to verify the outcome. A task generator distills verified runs into a knowledge base for reuse by the planner. Across GitHub, PostgreSQL, Playwright, and Yahoo Finance, with 12 models and 125 tasks, the authors report a verified-task rate of 71%, compared with 40% for their ReAct baseline, with lower token usage in most configurations. Retrieval and replanning ablations further examine the roles of precedents and recovery.

Critical strengths
  1. The paper addresses a gap between MCP tool descriptions and correct service use that individual schemas do not fully specify dependencies between tools, state constraints, or common operational rules. MCA incorporates this knowledge into planning, execution, and validation provided by the service, and reuses verified precedents for subsequent tasks. Structured plans and explicit result bindings make call order and data transfer amenable to programmatic checking and execution. This concrete design for encapsulating and reusing service expertise is valuable for building inspectable, maintainable MCP agents.

  2. MCA represents dependencies through structured plans and result bindings, checks arguments, tool names, and bindings before invocation, and programmatically distinguishes errors that should halt execution from transient errors that can be retried. This gives call order, data transfer, and recovery conditions explicit execution rules, reducing reliance on the model to maintain these constraints at every turn. It is a useful engineering contribution for tool tasks involving dynamic identifiers, exact content transfer, and stateful writes.

Critical weaknesses
  1. The paper uses passing verification as its main measure of task success. But the dynamic checks are generated by a model and may miss or misinterpret user requirements. Querying the live system can test these checks, but does not show that all relevant requirements were checked. The same verifier selects knowledge-base entries and scores the experiments. The examples show that it can catch some errors, but do not sufficiently establish that accepted runs meet the original user goals. The paper therefore needs stronger evidence for its reliability claim.

  2. The experiments show that the full MCA system outperforms the ReAct baseline. However, MCA adds more than advance planning: it also uses service rules, retrieved examples, result bindings, and error recovery. The paper does not establish that the two systems receive comparable support in these respects. Using the same model rules out a difference in model capability, but does not show which design choices explain the gain. The ablations test retrieval and replanning within MCA, but do not test a ReAct system with similar support. The results therefore do not establish that planned execution is generally better than reactive tool use.

Questions for authors
  1. Table 3 compares the shipped corpus with a corpus expanded through further runs. Given these results and the task-coverage analysis, for which tasks and under what conditions is continued improvement in service expertise supported?
  2. Compared with work such as LLMCompiler that separates planning from execution, what is MCA's most important new design choice or insight? Please use a concrete mechanism or result to explain its value for tool use on stateful services.
  3. Section 3.3 uses the equal 70/125 results of GPT-5 nano with MCA and GLM-5.3-Flash with ReAct to suggest that the workflow can compensate for a weaker backbone. Changing the backbone also changes the model that generates the checks. How is comparability of the acceptance criteria established for these results? Is the cross-model conclusion supported by analysis under common task criteria?
5 comments
AuthorsOpenReview ↗

Response to Reviewer RxF5 (Part 1/3)


Thank you for the detailed review!We hope the following results and clarifications can address your concerns.

What we ran. Beside the MCA we ran three ablations (−retrieval, −guardrails, −recovery layer), derived a fourth from the full runs' first attempts (−replanning). We also ran five baselines on the same tools and backbone, each under the loop’s 20-turn budget: (1) the paper’s ReAct loop, unchanged; (2) ReAct shown the decoded file contents; (3) ReAct with the MCA’s guardrails and retrieved precedents; (4) ReAct with an agent-as-a-judge; and (5) LangGraph’s plan-and-execute template. We re-ran the paper's GLM-5.3-Flash and DeepSeek V4.1 Flash campaigns with the paper's settings on GitHub (28 tasks) and PostgreSQL (22 tasks). To make the comparison more reliable, every run is scored twice. The first score comes from our verifier (the paper's metric; on PostgreSQL the MCA run's checks grade the loop's final database, as in the paper; on GitHub the verifier writes checks for each run). The second score comes from checkers (named as task checkers in the revised paper): read-only checks of the live state, generated by Claude Opus 5.5 from the task prompts alone. The task checkers are not ground truth but offer a second, unbiased perspective: written by a model that is neither backbone, blind to our verifier, and the same for every arm. After the runs, one stale GitHub task (its repositories had moved) was dropped, so GitHub is scored on 27 tasks. p is the exact two-sided McNemar test, paired by task and backbone.

W1. Is "verified" reliable?

We acknowledge the reviewer's concern regarding the reliability of MCA verifier.

An MCA is the service's expert: it carries out the task and verifies it from its plan and trace, re-querying the live service after execution. Against the task checkers, the verifier accepts 92% of the MCA runs that meet the goal, and 84% of its acceptances meet every requirement.

  • Why re-query. Tasks act on live repositories, databases, pages and prices that change over time. The concrete example is the GitHub task(t06) we dropped because its repositories had moved. Therefore, a fixed measurement that meets all user goal doesn't exist. A read-only query after the run is the closest available test of success, and our task checkers test the same way.
  • Three layers MCA provides. MCA's verifier consists of three parts: (1) Deterministic format checks confirm each step ran and returned a result; and (2) static checks test arguments and results against fixed rules (tool schemas; on finance, invariants and bounds); (3)Model-written dynamic checks further tests the task's content correctness.

We used independent task checkers to measure how often the verifier cannot catch a requirement that none of its checks tests and Table 1 & 2 shows the results.

Table 1. GitHub: 27 tasks × GLM, DeepSeek (54 runs). PostgreSQL: 22 tasks × GLM, DeepSeek (44 runs).

MCA runsPostgreSQL (44)GitHub (54)Both (98)
Accepted by the verifier404585
… that meet every task checker requirement32 (80%)39 (87%)71 (84%)
task checker passes the verifier accepts32 of 34 (94%)39 of 43 (91%)71 of 77 (92%)

Table 2. How the verifier checks each kind of requirement, and where wrong runs got through.

What the task asks forHow the verifier checks itWrong runs it accepted
Things the run creates: repositories, branches, files, pull requests, workflows (GitHub, 310 requirements)Re-reads them from the live service (99% of these requirements)none
The database's final state: tables, rows, indexes (PostgreSQL, 272)Re-queries the database (91%)none
Choices that depend on the data, e.g. "fork the repository with the fewest open issues" (GitHub, 38)Often takes the answer from the plan itself, so a wrong choice passes6
Results the user asked to see, e.g. "list the objects so I can see the view" (PostgreSQL, 326)Usually checks only that the call returned something8
  • Overall accuracy. Of the 85 runs the verifier accepted, 71 (84%) meet every requirement. Of the 77 runs that meet every requirement, it accepted 71 (92%). It rejected the other 6, five of them because a model-written check was faulty (for example, one that could not read a count).
  • The misses are narrow. On PostgreSQL the database state was always right; the 8 accepted runs that miss an task checker missed only report deliverables, mostly empty or incomplete listings. On GitHub the 6 are plan choices the checks inherited, such as listing 3 of 11 search results or forking the wrong repository.
  • On the shared verifier: it admits corpus entries only from the task generator's offline runs on archetype goals (retrieval off). The knowledge base was frozen in every evaluation run. The task checker columns below do not use the verifier.
AuthorsOpenReview ↗

Response to Reviewer RxF5 (Part 2/3)


W2. Comparable support for the loop.

We appreciate the reviewer for identifying that the MCA contributes more than advance planning (service rules, retrieved examples, result bindings and error recovery). Without testing a loop that receives comparable support, the paper cannot show which of these design choices accounts for the gain. To address this, we ran three sets of comparisons on the same tasks and backbones (Table A): (i) the paper's loop and the loop given the MCA’s support, in three variants: the decoded file contents, the MCA’s service rules and retrieved examples, or an agent-as-a-judge; (ii) LangGraph’s plan-and-execute; and (iii) four ablations of the MCA, each removing one component.

Table A. GitHub: 27 tasks × GLM, DeepSeek (54 runs). PostgreSQL: 22 tasks × GLM, DeepSeek (44 runs).

ArmGitHub verified (p vs. MCA)GitHub task checkerPostgreSQL verified (p)PostgreSQL task checkerLLM calls, GH / PG
MCA45434034133 / 108
ReAct, the paper's loop30 (0.004)2623 (< 10⁻⁴)34822 / 463
ReAct (shown decoded file contents)34 (0.03)4423 (< 10⁻⁴)34730 / 463
ReAct + MCA guardrails, precedents40 (0.33)4527 (< 0.001)32723 / 474
ReAct + agent-as-a-judge39 (0.15)4526 (< 0.001)37879 / 674
Plan-and-execute (LangGraph)28 (< 0.001)3216 (< 10⁻⁶)131,295 / 1,065
MCA − retrieval44 (1)4238 (0.69)35142 / 107
MCA − guardrails43 (0.79)3940 (1)35135 / 112
MCA − recovery layer33 (0.004)3843 (0.25)34152 / 103
MCA − replanning (first attempt)33 (< 0.001)3136 (0.12)30–

By the task checkers, no new arm differs significantly from the MCA except −replanning on GitHub (p < 0.001) and plan-and-execute (GitHub p = 0.019, PostgreSQL p < 10⁻⁶).

  • The support we gave the loop. It saw the decoded file contents that the MCA's executor sees (the paper's GitHub loop saw only rendered text), the MCA's own guardrails and its top-3 retrieved precedents, verbatim, and, in a third arm, an LLM judge with read-only tools that must approve before the loop may stop. Recovery and repair live inside an executor, which a loop does not have, so we test them in the MCA (−recovery, −replanning); bindings are not ablated, and plan-and-execute shows a planner–executor without any of these mechanisms.
  • What explains the gain is validated, model-free execution with repair, not planning ahead as such. Plan-and-execute shares plan-ahead decomposition but runs each step with an LLM agent: it trails the MCA on both servers (Table A), reported PostgreSQL results it never queried, and on GitHub ran out of turns in 19 of its 22 task checker failures. Inside the MCA, removing replanning costs 12 verified GitHub runs (p < 0.001; 12 by task checker, p < 0.001). Removing the recovery layer also costs 12 (p = 0.004), mostly in what the verifier can confirm: repaired plans drop steps the first attempt completed, and by task checker the loss is 5. Q2 shows how the parts back each other up.
  • We will state the comparison as measured: a higher verified rate than every ReAct variant (significantly on PostgreSQL, and against the plain loop on GitHub), task checker success not significantly different from theirs at 4.3–6.6× fewer model calls, and a large lead at any budget up to 10 calls.

At a matched budget the MCA is far ahead. Table B counts the verified successes reached within k model calls, read off the full-budget runs; the MCA's count includes its own verifier calls.

ArmGitHub: ≤ 4 / ≤ 10 calls / full budgetPostgreSQL: ≤ 4 / ≤ 10 / full budget
MCA45 / 45 / 4540 / 40 / 40
ReAct0 / 17 / 341 / 6 / 23
ReAct + MCA guardrails, precedents0 / 17 / 403 / 6 / 27
ReAct + agent-as-a-judge0 / 14 / 390 / 3 / 26
Plan-and-execute0 / 0 / 280 / 2 / 16

The task checkers give the same picture: within 4 calls the MCA reaches 43 of its GitHub and 34 of its PostgreSQL successes, and no ReAct variant more than 2. A successful MCA run takes a median of 2 model calls, a successful loop run 11–16.

AuthorsOpenReview ↗

Response to Reviewer RxF5 (Part 3/3)


Q1. For which tasks does continued growth help?

Growth pays off where the new entries cover the task's archetype: on the 65 hard GitHub tasks whose archetypes the 32 rectified entries cover, verified counts rise 40 → 53 (GLM-5.3-Flash, p = 0.007) and 39 → 49 (DeepSeek V4.1 Flash, p = 0.021). The paper's Table 3 and Appendix A.6 support these two conditions:

ConditionEvidence
1. The new entries cover the task's archetypehard GitHub: the 65 covered tasks 40 → 53 and 39 → 49; the 40 uncovered tasks 29 → 28 and 26 → 23
2. The failure comes from missing knowledge (a result shape, a cross-tool constraint, a site's URL idiom), not from state seen only at run timehard browser, where element ids exist only once a page loads: growth mainly cuts replans per task (−20 to −31%) and raises first-plan successes (7 → 11, 5 → 11); verified 14 → 17 and 13 → 18 (n.s.)

The new runs show the same mechanism on the standard GitHub suite: retrieval raises first-plan verified successes from 25 to 33 of 54 (p = 0.08) and cuts the runs that need a repair from 27 to 18. We will state these conditions as the scope of the knowledge-base claims.

Q2. What is new relative to LLMCompiler?

The decisive choice is repairing a validated plan against the live state: on GitHub, bounded replanning turns 12 of the 21 failed first plans into verified successes (45 vs. 33 of 54, p < 0.001), while a generic planner–executor–replanner (LangGraph's plan-and-execute) verifies 28 (p < 0.001), and on PostgreSQL 16 of 44 to the MCA's 40 (p < 10⁻⁶).

LLMCompilerMCA
Plana DAG of function calls with placeholdersordered tool calls whose arguments bind into nested earlier results ($s_j.path), validated against the live MCP catalog before any call
Executionparallel dispatch of stateless functionsin order, with no model; 3 attempts per call; a recovery layer absorbs re-creations and writes not yet visible
After a failurean LLM joiner finishes or replans from the observationsthe plan, annotated with each step's outcome, returns to the planner, which continues from the current state with completed writes marked
Successthe joiner's answerre-queried on live state, off the planner's feedback path
Knowledgenone from the serviceservice guardrails and verified precedents that ship with the server, built by its own task generator

On a stateful service these mechanisms are what make repair work: a replan that restarted from scratch would re-create the repository the first attempt made. The parts back each other up (GitHub, new runs; verified, with the task checker count in brackets):

MCA variantFirst plan aloneRescued by replanningFinalRuns replannedLLM calls
full33 (31)12 (12)45 (43)18133
− retrieval25 (23), p = 0.0819 (19)44 (42)27142
− guardrails29 (24)14 (15)43 (39)20135
− recovery layer21 (18), p = 0.00812 (20)33 (38)30152
  • Each layer raises what the first plan gets right, and replanning repairs most of the remaining failures, at the price of more replans and calls (p compares each first-plan count with the full MCA's).
  • Without the recovery layer, repaired plans drop steps the first attempt completed, so fewer runs end on one plan the verifier can confirm (33 vs. 45, p = 0.004).
  • Bindings move data by reference: a copied file travels as $s5.content, so the MCA never re-types it. Re-typed copies account for the loop's only false acceptance on GitHub and for one of its failures.
  • Precedents supply result shapes: on hard GitHub, the shipped corpus cuts GLM-5.3-Flash's first attempts that fail on a binding error from 87 to 23 of 105.

LLMCompiler targets parallel calls of stateless functions and is not an MCP agent. A port to a stateful MCP server would need exactly these mechanisms, so Table A tests the generic structure with LangGraph's plan-and-execute template.

Q3. The 70/125 comparison.

We agree that verified counts of different backbones should not be compared, because each backbone writes the checks that accept its own runs. GPT-5 nano's MCA and GLM-5.3-Flash's loop therefore reached their 70 of 125 under different verifiers. We did not establish that their acceptance criteria are comparable, and no analysis under common task criteria supports the cross-model conclusion. In §3.3 we will remove this comparison and correct the claim it supported, that the workflow can compensate for a weaker backbone.

We will incorporate these details and make it more clear in the revised paper. Thanks again for your insightful suggestion!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Acknowledgement to Reviewers and Revision Summary


We sincerely thank all reviewers again for their valuable comments and precious time.

What the MCA contributes to MCP. We are glad all three reviewers credit our core contribution, that an MCP server should ship its own expert, not just its tools: Reviewer ykiN regards it as "a new discussion in the MCP context", Reviewer qS9M finds the service-centric argument solid, and Reviewer RxF5 values the design for packaging and reusing service expertise.

For MCP, MCA propose a new paradigm. Today each client must infer how to use a server from its tool schemas alone and grade its own work. With an MCA, a client sends a goal and the server acts as

  • expert: knowledge no schema states (service rules, verified precedents) ships with the server, serves every client and grows from verified runs;
  • supervisor: a model only writes the plan and its checks; the server validates the plan, runs it without a model and decides success on its live state, with a replayable record.

We have revised our paper accordingly. Our revision summary is as follows (section, table and appendix numbers refer to the revised paper):

[Section 2.2, Algorithms 1 and 2] We merged the submitted Sections 2.2 and 2.3, made the loop counter k count replans, and replaced Algorithm 2's own symbols with words; instead of defining r, π_θ's turn index, m and 1ᵐ, as our reply proposed, we dropped them, following the suggestion of Reviewer ykiN.

[Sections 2.2 and 3.2, Appendices A.1 and A.12] We explained why finance replans on a failed check, and added strict counts without such replans (362 of 480 runs against the loop's 287; in the new runs, 57 of 80 against 63) and answer correctness, on which the finance comparison now rests, following the suggestions of Reviewers ykiN and qS9M.

[Sections 2.2–2.3 and Appendix A.3] We defined the shipped and the grown corpus and the cap of 3 entries per tool signature, and described rectification step by step, with each server's settings in Table 9, following the suggestion of Reviewer ykiN.

[Sections 2.2 and 3.2, Appendices A.1 and A.12] We explained why finance replans on a failed check, and added strict counts without such replans (362 of 480 runs against the loop's 287; in the new runs, 57 of 80 against 63) and answer correctness, on which the finance comparison now rests, following the suggestions of Reviewers ykiN and qS9M.

[Section 3.1, Table 2 and Appendix A.13] We stated for each server how the loop is graded and which arm pays for writing checks, following the suggestion of Reviewer ykiN.

[Section 3.2 and Appendix A.6] We added the two conditions under which knowledge-base growth helps: the new entries cover the task's archetype, and the failure comes from missing knowledge, following the suggestion of Reviewer RxF5.

[Sections 3.3 and 5] We restated the result as measured: the MCA verifies more tasks than every ReAct variant (significantly on PostgreSQL, and against the plain loop on GitHub), with task-checker success not significantly different from that of the loops given the MCA's support, at 4.4–6.6× fewer model calls, following the suggestion of Reviewer RxF5.

[Section 3.3, Appendices A.8 and A.9] We added new experiments: four MCA ablations and four baselines on GitHub and PostgreSQL (the paper's loop, ReAct with the MCA's guardrails and precedents, ReAct with an agent-as-a-judge, and plan-and-execute), graded by our verifier and by task checkers, with first-plan success, replans and model calls, following the suggestions of Reviewers RxF5, ykiN and qS9M.

[Section 3.4] We removed the 70/125 comparison across backbones and now state that verified counts are comparable only within a backbone, following the suggestion of Reviewer RxF5.

[Section 4, Table 5 and Appendix A.9] We added a discussion of agents that plan ahead (ReWOO, HuggingGPT, LLMCompiler, LangGraph's plan-and-execute) and a comparison with them, following the suggestions of Reviewers RxF5 and ykiN.

[Appendix A.9] We added the successes reached within 4 and within 10 model calls, following the suggestions of Reviewers RxF5 and qS9M.

[Appendix A.10] We added a calibration of the verifier against the task checkers, with false-positive and false-negative rates per server, following the suggestions of Reviewers RxF5 and qS9M.

[Appendix A.11] We added two new evaluation rounds on all four servers and the earlier PostgreSQL replications as run-to-run variance test, following the suggestion of Reviewer qS9M.

[Appendices A.8 and A.9, AI use statement] We documented the 50 task checkers, the baseline implementations and every arm's settings, and disclose that Claude Opus 5.5 wrote the checkers and mapped their requirements, following the suggestions of Reviewers RxF5, ykiN and qS9M.

Please let us know if anything is still unclear. We are happy to answer more questions if needed. Thanks again for all reviewers' constructive suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.