- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes to use a service-specific agent with deterministic executors, built in verifiers and planner, and a knowledge of service call patterns to replace MCP based tool calling using general purpose LLMs. The overarching idea is that the best agent to interact with a service's tools and APIs is one defined by the service. To illustrate this, the authors define a model context agent - an agent shipped by the expert, with access to deterministic executors and LLM based verifiers and planners. The MCA is provided with a user goal, generates an ordered plan (including tools, arguments, post-hooks, etc - effectively generating a program). Built into the planner is a live catalog (up to 3 exemplars from the knowledge base), and guardrails that are derived from failures from earlier runs. The plan is validated before it is executed - so an invalid plan (with missing arguments, for ex.) would not be executed. This plan is then executed by a deterministic executor (no LLM involved). If a step fails, executor just returns the plan annotated with the outcome, and the planner re-plans. Finally, the execution of the task is verified through format, static and dynamic checks, and verified only if ALL checks pass. Dynamic checks are written by an LLM, but are grounded on the tool's implementation - and are restricted to helper functions from that check the state of the system via deterministic checks. The number of re-planning turns is bounded and running out of turns fails the task.
In addition to the MCA, the paper proposes knowledge base construction over time, building it from existing, verified runs. If a run fails initially and is fixed at a later stage, the mistakes serve as grounding guidance that steers the models away from common failure modes. This is effectively converting episodic memory to procedural memory.
The paper compares this method against a ReAct based agent that uses the MCP tools directly, and compare MCA against the baseline on 125 tasks, using 12 different models, and reporting the execution and verification rate. The results show that MCA outperforms the ReAct agent on most tasks, and uses fewer tokens. The benefit of the knowledge base is seen primarily in long horizon, harder tasks. The conclusions drawn from these experiments are that the workflow adds reliability to this process, verification is best done with tools and processes provided by the service, and that having a constantly updated tool-use specific memory helps reduce error rates.
Critical strengths
- Proposes a solid argument for a service-centric architecture for a tool use agent, and the system proposed has a well-defined architecture that bounds the number of retries per stage, proposes deterministic executors and relies on tools for verification while checking the actual state instead of relying on LLM's claims/judgements. The general idea of having a service specific agent that is an expert in using the service makes sense.
- The experiments and analyses done go into a fair bit of detail around the planning step, how the retries are bounded, how the KB is constructed, issues with retrieval, how the KB benefit scales with difficulty, and benchmarks it against 12 different models.
Critical weaknesses
- The primary issue I take with the current experimental setup is the lack of separation of cause - for lack of better phrasing. The system proposed introduces components that the ReAct system lacks:
- A dedicated, retry-able planning step
- A knowledge base of common failures
- Access to service defined verifiers It would be good to get a sense of the lift provided by each part of this system. How much was because of the planning phase vs the verifiers that were made available, vs the KB?
Additionally, from page 16, it looks like the context provided to the ReAct agent might be different from what MCA has access to (Appendix A.9, page 30) - the ReAct agent only sees the rendered text. This makes the baseline a little weak - It would be interesting to provide the ReAct agent with an Agent as a Judge style verifier instead, instead of a simple react loop.
-
The same model plans and generates the claims to verify, but the model might miss something that needs to be verified. We need human calibration on the automatically generated checks to ensure that the generated checks were sufficient, preferably with false positive / false negative rates on subset of completed tasks.
-
There's only one reported run per task - which does not give us an idea of run-to-run variance, especially with the verification. There is some variance clearly (we see this in the hard PostGresSQL tasks), but it's not entirely clear how much. While it might be hard to rerun everything, can we generate this on a subset?
Questions for authors
See weaknesses for the questions that affect my assessment.
Additional questions: Of the four domains, only finance permits replanning after failed checks (instead of failing the task). Why is this the case? Can we report the results from both (was it considered verified?) vs (was the answer correct?) perspectives for finance?
5 comments
Response to Reviewer qS9M (Part 2/3)
W1 (continued): ReAct with the MCA's context, and with an agent-as-a-judge.
Given the MCA's context or an agent-as-a-judge, ReAct still verifies fewer tasks than the MCA on both servers, significantly on PostgreSQL (26–27 vs. 40 of 44, p < 0.001), and every ReAct variant needs 4.3–6.6× the MCA's model calls.
Table B. Same runs as Table A; p vs. the MCA by verifier.
| Arm | GitHub verified (p) | GitHub task checker | PostgreSQL verified (p) | PostgreSQL task checker | LLM calls, GH / PG |
|---|---|---|---|---|---|
| MCA | 45 | 43 | 40 | 34 | 133 / 108 |
| ReAct, the paper's loop | 30 (0.004) | 26 | 23 (< 10⁻⁴) | 34 | 822 / 463 |
| ReAct (shown decoded file contents) | 34 (0.03) | 44 | 23 (< 10⁻⁴) | 34 | 730 / 463 |
| ReAct + MCA guardrails, precedents | 40 (0.33) | 45 | 27 (< 0.001) | 32 | 723 / 474 |
| ReAct + agent-as-a-judge | 39 (0.15) | 45 | 26 (< 0.001) | 37 | 879 / 674 |
| Plan-and-execute (LangGraph) | 28 (< 0.001) | 32 | 16 (< 10⁻⁶) | 13 | 1,295 / 1,065 |
- Context. We acknowledge that the paper's GitHub loop saw only the rendered text of file resources (Figure 10); Figure 5's caption, which says otherwise, is wrong and will be corrected. After shown the decoded contents, the loop rises from 30 to 34 verified runs (26 → 44 by task checker), and the MCA still verifies more (45, p = 0.03) with 5.5× fewer calls. The same loop also received the MCA's own guardrails and its top-3 retrieved precedents, verbatim (the "guardrails, precedents" row).
- Agent-as-a-judge. An LLM judge with read-only tools must approve before the loop may stop; its calls come on top of the loop's 20 turns. On PostgreSQL it sent back 20 of the 56 answers it judged, and the arm verifies 3 more runs than the plain loop, still far below the MCA (26 vs. 40). On GitHub the arm verifies 39 vs. 34, but the judge itself changed no outcome: it returned FAIL on 2 of the 48 answers it judged, and neither run recovered. The arm makes 6.2× and 6.6× the MCA's calls.
- At a matched budget. Within 10 model calls the MCA has all 45 GitHub and 40 PostgreSQL verified successes; ReAct + agent-as-a-judge has 14 and 3 (by task checker: 43 and 34 vs. 15 and 6). A successful MCA run takes a median of 2 calls, a successful judge-loop run 15–16.
| Arm | GitHub: ≤ 4 / ≤ 10 calls / full budget | PostgreSQL: ≤ 4 / ≤ 10 / full budget |
|---|---|---|
| MCA | 45 / 45 / 45 | 40 / 40 / 40 |
| ReAct | 0 / 17 / 34 | 1 / 6 / 23 |
| ReAct + MCA guardrails, precedents | 0 / 17 / 40 | 3 / 6 / 27 |
| ReAct + agent-as-a-judge | 0 / 14 / 39 | 0 / 3 / 26 |
| Plan-and-execute | 0 / 0 / 28 | 0 / 2 / 16 |
W2. Calibration of the generated checks.
The MCA acts as the service's expert: it carries out each task and verifies it from its plan and trace, and the task checkers of part 1, a second perspective rather than a ground truth, confirm 84% of its acceptances; the verifier accepts 92% of the MCA runs that meet the goal.
- Why re-query the live state. The tasks act on live repositories, databases, pages and prices that change over time, so an answer key written in advance goes stale (one GitHub task did during the rebuttal: its repositories had moved). A read-only query of the live state after the run is the closest available test of success; the task checkers test it the same way.
- Why three layers. Deterministic format checks (every planned step ran and returned a result) and static checks (arguments and results obey fixed rules: each tool's schema, and on finance invariants and bounds) cover execution and form, so model-written dynamic checks cover only the task's content.
- Calibration. A model can still leave a requirement unchecked, as the reviewer note.
| MCA runs | PostgreSQL (44) | GitHub (54) | Both (98) |
|---|---|---|---|
| False positive: accepted, fails the task checker | 8 of 40 accepted (20%) | 6 of 45 (13%) | 14 of 85 (16%) |
| False negative: meets the task checker, rejected | 2 of 34 (6%) | 4 of 43 (9%) | 6 of 77 (8%) |
| Same verdict as the task checker | 34 (77%) | 44 (81%) | 78 (80%) |
- The false positives are narrow. On PostgreSQL the MCA left the database in the required state in all 44 runs (272 of 272 state requirements); the 8 accepted runs missed only report deliverables, mostly listings that came back empty or incomplete. On GitHub the 6 are plan choices the checks inherited, such as listing 3 of 11 search results or forking the wrong repository.
- The false negatives are mostly a single model-written check with a wrong expected value, or one that could not resolve a count or an issue number; in the PostgreSQL cases every deterministic check passed.
Response to Reviewer qS9M (Part 3/3)
W3. Run-to-run variance.
A second full round of runs reproduces the outcomes: under fixed task checkers the MCA keeps the same outcome in all 44 PostgreSQL runs and in 93% of the 54 GitHub runs scored, and its verified totals move by at most 3 on PostgreSQL, GitHub and finance. Round 2 re-ran every task with the same arms and settings (444 runs: the MCA and its matched loop on PostgreSQL, the browser and finance, and the MCA on GitHub, all under GLM-5.3-Flash and DeepSeek V4.1 Flash).
Table C. MCA verified tasks: the paper's runs → round 1 (Nov 10–12) → round 2 (Nov 12–13).
| Server (tasks) | GLM-5.3-Flash | DeepSeek V4.1 Flash | Same result, rounds 1–2 |
|---|---|---|---|
| GitHub (28) | 19 → 21 → 22 | 23 → 24 → 24 | 93% (task checker, 27 tasks) |
| PostgreSQL (22) | 20 → 18 → 19 | 20 → 22 → 20 | 100% (task checker) |
| Finance (40) | 38 → 36 → 37 | 38 → 38 → 40 | 91% (verifier) |
| Browser (35) | 20 → 14 → 14 | 20 → 26 → 18 | 66% (own checks) |
- The two rounds differ by at most 2 verified tasks per server and backbone, except DeepSeek on the live-web browser (8).
- By our verifier, the MCA's lead on PostgreSQL and finance replicates: PostgreSQL 40 vs. 23 and 39 vs. 19 of 44 (p < 10⁻⁴ in both rounds), finance 74 vs. 63 and 77 vs. 53 of 80 (p = 0.027 and p < 10⁻⁶). By the task checkers, PostgreSQL is 34 vs. 34 and 34 vs. 33.
- The browser is noisy for both arms: Live pages change run to run, and about a third of tasks change outcome between rounds in both the MCA and the loop (by their own checks, MCA 40 → 32 and loop 45 → 46 of 70).
- Earlier replications point the same way. In the paper’s campaigns, we ran PostgreSQL under MiniMax-M3 three times per arm. The MCA verified 18, 21 and 21 of 22 tasks, against 16, 13 and 16 for the loop. The MCA’s outcome was identical across all three runs on 17 of 22 tasks, compared with 10 for the loop. The hard PostgreSQL suite was likewise run three times per arm and backbone, and Table 3 of the paper reports the majority vote.
Q. Finance: why replan on a failed check, and verified vs. answer correct.
On finance the MCA verifies more runs than ReAct in the paper's campaigns and in fresh runs (454 vs. 287 of 480; 74 vs. 63 of 80, p = 0.027), and on the fixed-date tasks of the paper's GLM and DeepSeek campaigns it also answers more questions correctly by MCP-Universe's evaluators (62 vs. 52 of 72, p = 0.006).
Why replanning. A finance question is computable only if the fetched series supports it (for example, enough daily bars for a 50-day mean), and only a dynamic check can see that. The finance port therefore returns a failed check to the planner, at most three times; the other servers keep the verifier off the planner's feedback path. In the paper's campaigns, without its check-triggered replans the MCA still verifies 362 of 480 runs against the loop's 287 (p < 10⁻⁶). This rationale was lost from Appendix A.1; the revision will state it in the merged workflow section (§2.2–2.3) and restores it in Appendix A.1.
Table D. Answers are written by the run's own backbone.
| Finance | Runs per arm | Verified, MCA vs. ReAct | Answer correct, MCA vs. ReAct |
|---|---|---|---|
| Paper's campaigns, all tasks | 480 | 454 vs. 287 (362 without check-triggered replans) | – |
| Paper's GLM and DeepSeek campaigns, 18 fixed-date tasks | 72 | – | 62 vs. 52 (p = 0.006) |
| Fresh runs, matched loop (20 turns, same thinking) | 80 | 74 vs. 63, p = 0.027 (57 without check-triggered replans) | 56 vs. 50 (p = 0.31) |
- The MCA's pipeline returns the fetched data; one LLM call by the same backbone then writes a short program that computes the answer from it. Tasks without a fixed date are graded against live market data, so for the paper's runs we compare on the 18 fixed-date tasks of the four GLM and DeepSeek campaigns (GLM-5.3-Flash, GLM-5.3, DeepSeek V4.1 Flash, V4 Pro); the fresh runs were graded right after each task.
- "Verified" on finance confirms that the evidence an answer needs was fetched. On the fresh runs this does not yet give significantly more correct answers (56 vs. 50), so the revision will report both columns and rests the finance claims on answer correctness.
We will incorporate these details and make it more clear in the revised paper. Thanks again for your insightful suggestion!
Response to Reviewer qS9M (Part 1/3)
Thank you for the positive assessment, for finding the service-centric argument solid and the bounded, deterministic architecture well defined, and for the concrete suggestions! We hope the following results and clarifications can address your concerns.
What we ran. Beside the MCA we ran three ablations (−retrieval, −guardrails, −recovery layer), derived a fourth from the full runs' first attempts (−replanning). We also ran five baselines on the same tools and backbone, each under the loop’s 20-turn budget: (1) the paper’s ReAct loop, unchanged; (2) ReAct shown the decoded file contents; (3) ReAct with the MCA’s guardrails and retrieved precedents; (4) ReAct with an agent-as-a-judge; and (5) LangGraph’s plan-and-execute template. We re-ran the paper's GLM-5.3-Flash and DeepSeek V4.1 Flash campaigns with the paper's settings on GitHub (28 tasks) and PostgreSQL (22 tasks). To make the comparison more reliable, every run is scored twice. The first score comes from our verifier (the paper's metric; on PostgreSQL the MCA run's checks grade the loop's final database, as in the paper; on GitHub the verifier writes checks for each run). The second score comes from checkers (named as task checkers in the revised paper): read-only checks of the live state, generated by Claude Opus 5.5 from the task prompts alone. The task checkers are not ground truth but offer a second, unbiased perspective: written by a model that is neither backbone, blind to our verifier, and the same for every arm. After the runs, one stale GitHub task (its repositories had moved) was dropped, so GitHub is scored on 27 tasks. p is the exact two-sided McNemar test, paired by task and backbone.
W1. Separation of cause.
Our component analysis shows that the MCA’s gain comes from validated, model-free execution with repair.. Plan-and-execute keeps the planning step but executes each step with an LLM agent, and it verifies only 16 of 44 PostgreSQL runs, against the MCA’s 40 (p < 10⁻⁶). Without bounded replanning, the MCA verifies 33 of 54 GitHub runs instead of 45 (p < 0.001).
Table A. Change in verified runs when one part is removed or replaced (task checker change in brackets; p by verifier). GitHub: 54 runs (27 tasks × GLM, DeepSeek), MCA 45 verified (43 task checker). PostgreSQL: 44 runs (22 tasks × GLM, DeepSeek), MCA 40 (34).