acceptodds
Under review as a conference paper at ICLR 2027

Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents

Abstract

Language agents increasingly act through external tools and interactive environments, making the interface between the model and its executable capabilities part of the decision process. Model Context Protocol (MCP) and command-line interfaces (CLI) can expose the same semantic operations through markedly different model-facing representations. We study whether these representational differences change agent behavior when executable capabilities are matched. We introduce Matched Interface Comparison and Analysis (MICA), which renders MCP and CLI from a shared semantic action contract and routes decoded actions through the same runtime. Across six models and three interactive environments, matched interfaces produce substantial differences in task performance and interaction cost. These effects extend beyond surface validity to semantic action selection, failure pathways, and recovery. Matched-state probes localize sensitivity to capability presentation, realization, and feedback. Paired natural trajectories reveal structured divergence in 94.3% of 2,004 episodes and show that early differences propagate through subsequent interaction. This analysis identifies interface realization as a direct intervention boundary. We therefore introduce Decoupled Action Realization and Targeted Repair (DART), which preserves validated semantic commitments while repairing interface-specific realization. DART improves benchmark score in all six evaluated settings by up to 9.0 points and reduces MCP token cost in all three environments by up to 27.1k tokens per task. Together, these results establish interface representation as part of the closed-loop decision environment shaping agent behavior. Code is available in the Supplementary Material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Agents, Tool Use, Interface Design

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_ndMbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MICA, a framework for studying how interface representations affect language agents under matched executable capabilities. MICA constructs MCP and CLI interfaces from a shared semantic action contract and uses a shared runtime to isolate the effect of interface design. Experiments across six models and three environments show that interface representations influence task performance, interaction cost, semantic decisions, and failure patterns. The paper further introduces DART, a repair mechanism that separates semantic action selection from interface realization and improves recovery from realization failures.

Critical strengths
  1. Capability-matched interface construction

MICA generates both interfaces from a shared semantic action contract and uses a shared validation and execution pipeline, reducing capability and execution-related confounds in the comparison.

  1. Complementary local and trajectory-level analysis

Matched-state factorial probes separate presentation, realization, and feedback effects, while paired trajectories characterize subsequent divergence. Together, they provide a more detailed understanding of how interface differences affect agent behavior beyond endpoint performance.

Critical weaknesses
  1. Limited generalization beyond controlled interface representations

MICA studies MCP-style and CLI-style interfaces under carefully controlled matched semantic contracts and shared execution pipelines. While this design enables clean attribution of interface effects, it remains unclear whether the observed behaviors transfer to real-world MCP servers, terminal environments, and production agent scaffolds with additional tooling and orchestration layers.

  1. Limited attribution of DART’s improvements

DART combines several components, including semantic commitment formation, phase-specific prompting, and interface-specific realization repair. Since the evaluation compares DART against the Base policy without isolating these components, it remains unclear whether the gains primarily come from commitment preservation, additional reasoning steps, or other intervention effects.

Questions for authors
  1. Regarding generalization beyond controlled interfaces

Could the authors clarify how they expect the findings from MICA’s controlled MCP/CLI comparison to transfer to deployed agent systems with native MCP servers, terminal environments, and additional orchestration layers? (See Weakness 1.)

  1. Regarding the source of DART’s improvements

Could the authors provide additional ablations or analysis to isolate the contribution of semantic commitment preservation from other components in DART, such as additional reasoning steps or realization repair? (See Weakness 2.)

  1. Can the observed interface effects be linked to identifiable task or interface characteristics?

While the paper analyzes action selection, failure pathways, and component interactions, it remains unclear whether these analyses reveal general patterns that predict when MCP-style or CLI-style representations are likely to be more effective.

Reviewer_jDgfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper demonstrates that LLM-based agent performance can vary across MCP and CLI interfaces. By unifying the interfaces with an intermediate specification schema with fixed predeclared semantics with respect to the task environment (MICA). With Decoupled Action Realization and Targeted Repair (DART), the paper attempts to show that interface-specific syntactic repairs (called realization repairs) can improve performance even without altering action semantics . DART iterates and commits to predeclared semantics (contracts) before initiating repair loops. The reported results show that DART successfully improve Gemini-2.5-Flash-Base performance under both CLI and MCP interfaces across 3 benchmark environments (ALFWorld, WebShop, ScienceWorld).

Critical strengths
  • The presentation is clear and easy to follow
  • The paper is thorough in experimental design for empirical evidence
Critical weaknesses
  • Despite the soundness, the paper seems to merely document a known problem that performance can vary across textual interfaces, which is partly the motivation for protocols like MCP.
  • All the RQs have largely predictable answers, so confirming empirical results offer limited insights. Critically, the results do not identify which properties from the interfaces produce which effects in a way that would let a practitioner predict or design better interfaces.
Questions for authors

Can you attribute which properties from the interfaces produce which effects? Correspondingly, how should design future protocols to avoid them?

Reviewer_Vq5UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  1. The paper asks whether exposing the same capabilities through an MCP-style vs. a CLI-style interface changes how LLM agents behave. Both interfaces are built from one shared action contract and run through the same validator and executor (MICA), so only the model-facing side differs.

  2. Across 6 models and 3 environments (ALFWorld, WebShop, ScienceWorld), they find sizable score and token gaps whose direction depends on model and environment. For example, Qwen3-14B on ALFWorld gets 20.1 with MCP vs. 56.7 with CLI.

  3. A 2×2×2 probe that swaps presentation, realization and feedback at fixed states points to realization as the main source of the gap, and 94.3% of paired trajectories diverge early.

  4. They propose DART. After a failure, it has the model pick a canonical JSON action and then re-express it in the interface. On Gemini-2.5-Flash it improves score in all 6 settings (+0.8 to +9.0).

Critical strengths
  1. The controlled setup is clean and reusable. Building both interfaces from one contract and swapping one component at a time is the right way to separate representation from capability, and others could apply it to real systems.

  2. The analysis goes beyond final scores to look at failure stages, recovery and trajectories. It's also transparent, with full prompts released and every extra call counted in the token cost.

Critical weaknesses
  1. This isn't really MCP vs. CLI, and the gaps aren't checked against noise. "MCP" here is JSON text written into the reply and "CLI" is a single command line, so the comparison is basically JSON vs. shell-quoted text, and part of the gap comes from strict decoders (apostrophes breaking the quoting, for example). The results look like single greedy runs, Table 1 has no CIs, 13 of the 18 gaps are under 5 points, and the 94.3% divergence figure has no null baseline.

  2. The mechanism and DART claims don't hold up. The probes don't match the real episodes: for Qwen3-32B on ALFWorld, CLI realization fails 82% of the time in the probes but only 5.7% in real runs. The biggest gap (Qwen3-14B on ALFWorld) comes with identical failure rates under both interfaces. DART claims to keep the original decision, but its own prompt tells the model to choose a new action, and it's tested on one model with no call-matched retry baseline.

Questions for authors
  1. How were the probe states chosen, and why are probe failure rates so different from the real episodes?

  2. How many runs per task? Can you add CIs and a null baseline, such as the same interface run twice or a paraphrased version of one interface?

  3. For DART, how many triggers were there per setting, and how often does the new committed action match the one that failed?

  4. What causes the Qwen3-14B ALFWorld collapse under MCP when the failure rates are the same?

  5. Can you compare DART against a plain extra retry and against directly executing the validated JSON action?

Reviewer_1gnqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper renders MCP and CLI interfaces from a shared semantic action contract and routes both through the same execution pipeline, then compares them across 6 models and 3 environments. It finds that interface representation substantially changes task performance and token cost, but in no consistent direction; the response format component has the largest effect. Based on this, the authors propose DART, which first commits to a validated semantic action and then repairs only its interface-specific format. DART improves scores in all 6 settings on Gemini-2.5-Flash.

Critical strengths

The question is meaningful.

The combination of factorial probes and trajectory analysis gives a finer-grained picture than endpoint scores alone.

The experiments are large-scale, statistically sound, and reproducible.

Critical weaknesses

Both "interfaces" are custom text formats, so the conclusions may not generalize to real MCP and CLI.

The differences mainly reflect whether the model can produce the format correctly, which is not sufficient evidence that the model's decisions change.

The divergence analysis has neither a null baseline nor counterfactual interventions.

Questions for authors

Same as weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.