- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AgentMT, a metamorphic-testing framework for measuring the consistency of LLMs’ behaviour when the same request (in terms of intent) is requested differently. The prompts are sourced from three agent benchmarks (AI Agent Permissions, AgentCIBench, TRAJECT-Bench), and are transformed along tone, language, and formulation (specifically sentence reordering and paraphrasing differently). Agent executions are logged as typed events and converted into operation graphs, and consistency is measured at three levels: graph similarity, LLM-judged output similarity, and consequence consistency. Experiments with Gemini 2.5 Flash, GPT 4.1 Mini, and Claude Haiku 4.5 show that rewordings reduce graph similarity relative to repeated-run baseline across all 27 model-dataset-transformation settings.
Critical strengths
S1. The paper evaluates what the agent does under intent-preserving rewordings (which permissions it requests, what information it discloses and to which channels, and which tools it calls), rather than only whether the task succeeds. To my knowledge, defining metamorphic relations over these behavioral consequences is novel.
S2. The experiments cover a good breadth of benchmarks, transformation types, and interaction settings (single-turn, multi-turn clarification, and coding agents run through Claude Code and Codex CLI). I particularly appreciated the analysis comparing operation-graph and output similarity, which shows that agents can produce similar final outputs while behaving differently along the way.
Critical weaknesses
W1. This paper offers limited new insights into a well-documented phenomenon of “sensitivity of LLM behavior to prompt perturbations”. The paper’s measurement contributions in the agentic setting are novel, but its main findings largely confirm what existing literature would predict: rewording changes behavior beyond run-to-run noise. Beyond aggregate breakdowns by transformation type, the paper does not offer new takeaways such as explaining why particular rewordings cause particular behavioral changes, or which tasks and actions are most at risk (which practitioners may care about).
W2. The main experiments use smaller, cheaper and/or older models (Gemini 2.5 Flash, GPT-4.1 Mini, Claude Haiku 4.5) with limited comparison to newer and more capable models. Only one flagship frontier model (GPT-6 Astra) is used on a subset of AgentCIBench, where it shows high operational consistency. From this, the authors conclude that “[s]tronger model capability thus substantially reduces sensitivity…” when this is too low of a sample size to conclude with the stated confidence. Having more frontier models across different model families (e.g. newer Gemini Pro or Claude Sonnet/Opus models) would strengthen their conclusion. In absence of that, I would suggest the authors weaken this claim.
W1 is the main driver of my rating, with W2 as an additional concern. The paper’s additional findings (e.g. the effects of clarification) are, in my view, incremental confirmations of known prompt sensitivity rather than substantial new insight.
Questions for authors
Q1. How confident are the authors that the rewordings mean the same thing? For example, something like “Are you able to {task}?” could reasonably be answered by “yes” without doing anything, and “This time, {task}” could imply a previous failure to do {task}. The paper says two authors checked the prompts, but gives no details on what the criteria for their verification were.
Q2. How often did agents ask a question or refuse to act under each template, and how is that counted?
10 comments
Response to Reviewer bmxW (Part 2)
W2. Evaluation Across Frontier Models.
We agree that our original conclusion that stronger model capability substantially reduces sensitivity was too broad given the limited frontier-model evaluation. To address this concern, we conducted additional experiments with two frontier models from different model families, Claude Opus 5.5 and Gemini 4 Carbon, alongside the previously evaluated GPT-6 Astra. We used the same 15 AgentCIBench tasks, prompt transformations, repeated-run baselines, and evaluation metrics to ensure a consistent comparison.
| Model | S–S | Tone | Language | Formulation |
|---|---|---|---|---|
| GPT-6 Astra | 0.965 | 0.942 | 0.964 | 0.975 |
| Claude Opus 5.5 | 0.965 | 0.965 | 0.939 | 0.928 |
| Gemini 4 Carbon | 0.919 | 0.825 | 0.828 | 0.889 |
The results reveal distinct operational consistency patterns across frontier models. GPT-6 Astra maintains high graph similarity (0.942–0.975) across all transformations. Claude Opus 5.5 also exhibits relatively high consistency, although its graph similarity decreases under language (0.939) and formulation (0.928) variations. In contrast, Gemini 4 Carbon exhibits larger reductions under tone and language variations, from its repeated-run baseline of 0.919 to 0.825 and 0.828, respectively.
Interestingly, all three models maintain high output similarity (0.967–1.000) despite their differences in operational consistency. This further supports our finding that similar final outputs do not necessarily imply consistent operational behaviors.
Together, these additional results suggest that operational consistency is model-dependent and does not necessarily improve uniformly with stronger model capability. We revise RQ3 to emphasize the observed differences across frontier models rather than attributing higher operational consistency solely to stronger model capability. This analysis further broadens our analysis across three frontier-model families and provides a more nuanced understanding of operational consistency.
Response to Reviewer bmxW (Part 3)
Q1. Validation of Intent-Preserving Transformations.
We thank the reviewer for highlighting the potential ambiguity in expressions such as "Are you able to {TASK}?" and "This time, {TASK}?" We agree that these expressions may admit different interpretations in isolation. However, their interpretation should also consider the context of AI agents, which are designed to understand and execute user requests rather than merely answer questions like LLMs.
For "Are you able to {TASK}?", although it can literally be interpreted as a capability question, it is commonly used as an indirect request for action. To examine how agents actually interpret such expressions, we analyzed all 15 tone templates across 116 tasks, three models, and three repeated executions, yielding 1,044 executions per template. Our observations show that agents generally interpret these expressions as actionable requests rather than simply answering whether they possess the required capability. For example, 71.0% of executions under "Are you able to ...?" proceeded with consequential actions, compared with 82.3% for "Can you ...?" and 85.7% for "Could you ...?". Our inspection did not identify cases where agents merely answered "yes" without attempting the requested task.
For "This time, {TASK}", we acknowledge that this expression can imply prior interaction, but it does not necessarily indicate that the task previously failed or introduce additional operational requirements. Such expressions reflect natural human communication patterns, where contextual or emphatic wording may be used to convey requests. Our tone templates are informed by naturally occurring user expressions from WildChat and prior linguistic studies, rather than being constructed specifically to represent requests following failed executions.
Transformation design. As described in Section 3.2 and Appendix A.1, we employ several measures to preserve the original user intent:
- Verbatim task preservation. Each tone template contains a fixed {TASK} slot into which the original request is inserted without modifying its content.
- Preservation of operational requirements. The transformations do not modify requested actions, entities, recipients, task constraints, or authorization scope.
- Controlled linguistic variations. The transformations modify only predefined linguistic expressions surrounding the original request, rather than introducing new conditions.
Human validation. To further assess intent preservation, we conducted a blinded human audit of 100 tone variants, independently evaluated by two annotators with relevant domain expertise. Each variant was assessed against four criteria: (1) preservation of task intent, (2) preservation of recipients and objects, (3) preservation of authorization scope, and (4) whether the transformed prompt remained an execution request. Deviations were additionally classified as none, minor, or major.
Both annotators judged all four criteria to be preserved in 93/100 cases, with criterion-level agreement of 98%, 100%, 100%, and 96%, respectively. Neither annotator identified any major deviations. More specifically, among 38 samples involving context-sensitive expressions, both annotators confirmed task-intent and authorization-scope preservation in all cases.
We also conducted a separate human validation for 210 language variants, covering 30 tasks across each of seven languages. The same two annotators independently evaluated task intent, recipients/objects, authorization scope, and tone. Both annotators were fluent in English, and each could read at least one additional language included in our evaluation. To support multilingual assessment, we employed Claude for source-blind literal back-translation into English, supplemented by additional translation tools when necessary. The annotators agreed on the joint preservation judgment in 204/210 cases (97.14%), with criterion-level agreement ranging from 98.10% to 100%.
These results provide empirical support for our intent-preserving transformation design.
Response to Reviewer bmxW (Part 1)
We thank the reviewer for the detailed comments and for recognizing the novelty of evaluating agents' operational behaviors. We are confident to address the concerns and answer the questions raised by the reviewer.
W1. Novel Insights Beyond General Prompt Sensitivity.
We thank the reviewer for recognizing the novelty of our measurement contributions and for encouraging more actionable insights. We agree that the general observation that prompt rewording changes LLM behavior is already well established. However, our contribution goes beyond demonstrating prompt sensitivity by investigating operational consistency as a distinct reliability property of AI agents.
Unlike conventional evaluations focusing on model responses or task success, AgentMT systematically examines whether intent-equivalent requests lead to consistent permission decisions, information disclosure, and execution trajectories. Our MT-based formulation enables such comparisons without requiring a unique correct execution trajectory.
Beyond this methodological contribution, our study provides several empirical insights that are not directly implied by existing prompt-sensitivity findings.
Which tasks and actions are most at risk?
To provide more actionable insights, we conducted additional analyses using the benchmark-defined violation indicators and task characteristics.
First, we identified
which operational violations are more likely to be introduced by different prompt transformations. The results are shown as follows.On AIP, tone transformations show the highest rate of newly introduced unnecessary permissions (3.3%). On AgentCIBench, language transformations are associated with the highest rates of missing required information (15.7%) and unwanted disclosures (6.5%). These results reveal that different linguistic variations are associated with different types of task-level violations.
Second, we examined
which task characteristics are associated with greater operational inconsistency. On AIP, tasks involving four or more data fields exhibit a mean graph-similarity reduction of 0.153, compared with 0.071 for tasks involving one to three fields. On AgentCIBench, model–task combinations with frequent non-action exhibit a substantially larger reduction (0.229) than those in which agents consistently act (0.061). In contrast, TRAJECT-Bench shows similar reductions across different target-tool counts (0.090–0.093), suggesting that tool count alone is not a reliable indicator of inconsistency. These findings provide more specific insights into which operational requirements and task characteristics deserve closer attention, beyond the general observation that rewording affects agent behavior.Which rewordings are associated with behavioral changes?
We additionally analyzed individual tone templates and their effects on agents' decisions to act, seek clarification, or refuse. For example, "Are you able to ...?" increases the clarification rate from 6.4% to 16.0%, while reducing the action rate from 84.3% to 71.0%. More challenging expressions, such as "Why can't you ...?", are associated with substantially higher refusal rates (36.5%, compared with 8.0% for original requests). These results identify specific linguistic constructions associated with changes in agents' execution decisions, beyond aggregate differences across transformation families.
Our contribution is not simply that rewording changes agent behavior, but that operational inconsistency is distinct from output inconsistency, can involve concrete task-level violations, and exhibits identifiable patterns.