acceptodds
Under review as a conference paper at ICLR 2027

CloudGym: Can AI Agents Manage Shared Cloud Infrastructure?

Abstract

Managing cloud infrastructure requires substantial expertise and operational effort, motivating automation with AI agents. As cloud infrastructure is inherently shared, cloud management agents must coordinate with other participants whose actions can invalidate their plans or block their operations. Existing evaluations largely focus on static plans or isolated execution, leaving coordination in shared environments insufficiently tested. We present CloudGym, a benchmark for evaluating cloud management agents in live, shared cloud infrastructure, with 102 cases spanning 19 AWS services and 65 resource types. Each case introduces controlled concurrent participants whose actions create incompatible requirements, ambiguous targets, or temporary execution conflicts. Semantic oracles use initial and final cloud states and the record of activated participants to check whether outcomes satisfy the task intent under a specified resolution policy. To support reliable evaluation, an automated pipeline generates these cases from existing management tasks and certifies their behavior through execution against AWS. Our evaluation shows that frontier agents perform poorly on CloudGym, with the top-performing model, Opus 5.5, achieving an overall pass rate of 34.3% compared with 97.1% in isolated execution. This gap highlights coordination as a major obstacle to agentic cloud management. We open-source the CloudGym dataset and evaluation framework to enable future research.

Primary area:
datasets and benchmarks
Keywords:
AI agents, agentic cloud management, cloud infrastructure, benchmark, concurrent agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_U5PrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CLOUDGYM, a benchmark for evaluating cloud-management agents in live, shared environments with concurrent interference from other actors. The key motivation is that real cloud management is inherently multi-party: actions by one operator can invalidate another operator's plans, introduce ambiguity, or temporarily block execution.

Critical strengths

•Addresses important evaluation gap: Cloud infrastructure management is inherently multi-party, yet existing cloud-management benchmarks largely evaluate isolated execution or static artifact generation. The benchmark directly evaluates coordination under concurrent infrastructure modifications rather than purely intent realization in isolation. This is a meaningful capability that current agent evaluations appear to miss.

•Strong benchmark design: The benchmark executes against real AWS infrastructure rather than simulated plans or static configuration artifacts. Distractors operate through actual cloud operations, and final correctness is evaluated using cloud-state snapshots rather than generated code.

•Demonstrated difficulty: Experiments reveal substantial performance degradation under shared-environment conditions, suggesting meaningful challenge.

Critical weaknesses

•Limited theoretical grounding. The paper is motivated by scenarios in which multiple actors modify a shared cloud environment and interfere with one another's objectives. The broader literature [1, 2, 3] would view such settings through the lens of conflict detection, management and resolution. Surprisingly, this connection is neither explicitly discussed nor used to ground the benchmark design resulting a work that appears somewhat disconnected from prior work that studies similar phenomena under different terminology.

•Single-agent adaptation rather than multi-agent coordination. Although CLOUDGYM is framed as a multi-actor setting, evaluation is performed through a single agent interacting with scripted distractors that do not reason, communicate, or pursue evolving objectives. Consequently, the benchmark primarily measures robustness, recovery, and policy-aware adaptation to environmental changes rather than coordination or conflict resolution among multiple decision making entities.

•Policy retrieval dominates conflict reasoning. Since each benchmark instance defines a resolution policy that agents can obtain through consultation, many tasks reduce to detecting interference and applying the retrieved policy. This places substantial emphasis on information retrieval and policy compliance, while providing more limited evaluation of real cloud-based multi-agent dynamics such as reasoning about competing objectives, priority arbitration, or conflict-resolution strategies.

•Metrics not aligned with benchmark’s motivation. CLOUDGYM is motivated as a benchmark for shared environments with concurrent interference, yet the reported metrics focus almost exclusively on final task success and interaction cost. Additional metrics measuring conflict detection accuracy, policy-retrieval success, recovery after interference, adaptation latency, or correctness of conflict-resolution decisions would provide a more direct assessment of the capabilities the benchmark aims to evaluate.

The main concern is construct validity rather than implementation quality. The benchmark has clear value, but its claims about measuring coordination capabilities appear stronger than what the environments and metrics currently support. A narrower positioning as a benchmark for robustness to concurrent environmental changes and wider evaluation metrics would substantially strengthen the paper.

Minor: •Limited statistical robustness: Results are reported from only two independent trials per condition. While the large performance gaps support the main conclusions, reporting additional runs and variability measures (e.g., standard deviation or confidence intervals) would strengthen confidence in the robustness of the results. References: 1.del Prever, P. B., D’Oro, S., Bonati, L., Polese, M., Tsampazi, M., Lehmann, H., & Melodia, T. (2025). Pacifista: Conflict evaluation and management in open ran. IEEE Transactions on Mobile Computing, 24(10), 10590-10605. 2.Adamczyk, C., & Kliks, A. (2023). Conflict mitigation framework and conflict detection in o-ran near-rt ric. IEEE Communications Magazine, 61(12), 199-205. 3.Li, L., Liu, L., Huang, S., Lv, S., Lin, K., & Zhu, S. (2022). Agent-based multi-tier SLA negotiation for intercloud. Journal of Cloud Computing, 11(1), 16.

Questions for authors

•How does CLOUDGYM relate to existing conflict-resolution and coordination frameworks, and what distinguishes it from established conflict-management settings in multi-agent systems?

•In the current formulation, the evaluated agent interacts with benchmark-controlled distractors that execute predefined actions. To what extent do the authors view CLOUDGYM as measuring multi-agent coordination versus robustness to environmental interference and policy-aware adaptation? More specifically, what capabilities does CLOUDGYM evaluate that would not already be captured by a benchmark focused on dynamic environment adaptation?

•Can the authors provide additional theoretical justification for why the SC/IA/EC taxonomy form the appropriate decomposition of the problem space? Are the categories intended to be mutually exclusive and collectively exhaustive, and how do they relate to established taxonomies of conflict, resource contention, and coordination failures in multi-agent systems?

•How much of the benchmark difficulty comes from conflict reasoning versus discovering and applying the predefined resolution policy? Have you considered settings without explicit policy retrieval to better isolate conflict-resolution capabilities?

4 comments

We thank the reviewer for highlighting relevant work on conflict management and coordination. We will address the concerns and questions below.

R3.1:How does CloudGym relate to prior conflict-management research?

We thank the reviewer for sharing this work with us.

PACIFISTA [1] and Adamcyzk and Kliks [2] investigate conflict detection and mitigation among control applications in O-RAN environments. Li et al. [3] study multi-agent negotiation mechanisms for intercloud service-level agreements. These work address important coordination problems involving multiple participants operating over shared resources.

CloudGym addresses a related but distinct research problem. From a workload perspective, the cited O-RAN works focus on interactions among radio-network control parameters, whereas cloud infrastructure management involves a different type of shared state. Cloud infrastructure can be viewed as a heterogeneous, mutable resource graph. In this graph, resources can be created, modified, and deleted, and relationships among resources (such as dependencies) can change. Moreover, many individual resources have their own lifecycle states and operational constraints. Correctness often depends on invariants spanning multiple resources. In summary, cloud infrastructure is a heterogeneous environment with dynamic topology, resource configurations, and lifecycle semantics.

From a task perspective, O-RAN conflict-management frameworks focus on interactions among applications controlling radio-network parameters. However, CloudGym evaluates general-purpose AI agents performing infrastructure-management tasks across heterogeneous AWS services and resources. These tasks involve creating, modifying, and managing cloud infrastructure while other participants introduce concurrent changes. CloudGym evaluates whether agents can detect such changes, reconcile competing management requirements, and complete their tasks under interference. Furthermore, rather than proposing a conflict-detection or resolution mechanism, CloudGym provides a benchmark for evaluating these capabilities in a live cloud environment.

We acknowledge that these works provide valuable context for the conflict-management challenges motivating CloudGym. We will add them to the discussion and related work of the paper.

R3.2: Why SC/IA/EC? Are they mutually exclusive and exhaustive?

We motivate our taxonomy from the lifecycle of a cloud coordination framework. Effective coordination requires reasoning about potentially conflicting management intents during planning, maintaining unambiguous interpretations as the cloud state evolves, and adapting operations to interference during execution. These motivate three categories of coordination challenges: SC, IA, and EC, respectively.

  1. Semantic Conflict (SC): two participants have incompatible intended outcomes, requiring the agent to resolve the conflict according to a coordination policy.
  2. Intent Ambiguity (IA): concurrent changes that make different interpretations of an intent exist, such as when a resource reference becomes non-unique or a state-dependent target requires clarification.
  3. Execution conflict (EC): The participants' intended outcomes are compatible, but concurrent operations encounter cloud-provider constraints that prevent successful execution, requiring the agent to adapt its action.

Each CloudGym case is constructed around one of these 3 interference types. And each case belongs to exactly one category. Thus, the categories are mutually exclusive for the benchmark suite by construction. However, we do not claim that they are mutually exclusive or collectively exhaustive for all real-world scenarios, where multiple forms of interference may coexist. Unlike existing conflict taxonomies that classify interactions, SC/IA/EC primarily focus on challenges an agent must address to complete its task. We will clarify this motivation and the scope of the taxonomy in the revised manuscript.

R3.3: What coordination capabilities does CloudGym actually evaluate? What metrics directly measure coordination-related behavior?

CloudGym evaluates whether an agent can complete its cloud-management task while accounting for concurrent operations and respecting the resolution policy.

Our three experiment conditions contain different challenges. (i) Control evaluates the agent's ability to complete the original management task without any interference. (ii) Prompted policy introduces concurrent changes while providing the resolution policy upfront, evaluating whether an agent can detect and reconcile interference and complete the main task. (iii) Consulted policy additionally requires the agent to identify when consultation is necessary, obtain the relevant policy, and incorporate it into its actions. CloudGym evaluates these capabilities, not negotiation among independently reasoning agents.

Regarding metrics, final-state pass rates measure end-to-end success, while differences between conditions quantify the additional difficulty introduced by interference and consultation. For finer-grained evidence, we analyzed execution traces (R2.2).

  • No recorded observation of distractor: a requirement tied to a distractor failed, and there is no evidence to prove that the agent observed the corresponding change.
  • Distractor requirement failed despite exposure: a requirement tied to a distractor failed, although the agent received at least one response exposing the change.
  • Collateral change to an initial-state resource: the agent modified or deleted a resource that existed before the run and that the oracle requires to be preserved.
  • Operation error: the agent did not complete the run.
  • Main intent failed: every distractor requirement was satisfied, but the main intent was not accomplished.
Failure TypePrompted (ii)Consulted (iii)
No recorded observation of distractor33.1%36.6%
Distractor requirement failed despite exposure43.6%40.3%
Collateral change to initial-state resource0.0%0.8%
Operation error1.2%3.2%
Main intent failed only22.1%19.1%

Among failed runs, 33.1% under prompted policy and 36.6% under consulted policy showed no evidence that the relevant distractor change was observed. In contrast, 43.6% and 40.3% had the change exposed to the agents, but agents failed the corresponding resolution requirements.

We will include these analyses and evaluate coordination capabilities in the revised manuscript.

R3.4: Is the difficulty primarily policy retrieval rather than conflict reasoning?

Condition (ii) directly addresses this question by providing the resolution policy upfront, without consultation. Based on the result reported in Q3 of the General Reply, comparing condition (ii) to condition (i), the pass rate reductions ranged from 41.7 percentage points to 63.2 percentage points, while from condition (ii) to condition (iii), the reduction only ranged from 1.0 percentage point to 23.5 percentage points. This indicates that obtaining the policy introduces additional difficulty, but policy retrieval alone cannot explain the substantial performance degradation under interference.

The failure breakdown in R2.2 provides further evidence. Under condition (ii), 33.1% of failed runs showed no recorded observation of the relevant distractor change, while 43.6% failed distractor-specific requirements despite the change being exposed to the agent. Together, these account for 76.7% of failures, even though the resolution policy was already available. This suggests that detecting and appropriately handling interference remains very challenging independently of policy retrieval.

We want to clarify that CloudGym evaluates the ability to detect and reconcile interference under specified coordination policies, rather than independently negotiating or inventing such policies. Providing an explicit policy also makes the expected resolution well-defined for objective evaluation.

R3.5: Statistical reliability.

We thank the reviewer for raising this concern. Please refer to Q3 of our General Reply for the detailed statistical analysis.

We report 95% confidence intervals using a case-level bootstrap stratified by taxonomy category. Across all five evaluated models, the pass-rate reductions from control (i) to consulted policy (iii) range from 62.7 to 67.6 percentage points, with all paired confidence intervals excluding zero. These results support our primary finding that concurrent interference substantially degrades agent performance.

Thank you for the detailed responses. The bootstrap CIs, the third trial, the distractor-activation rates and the failure breakdown are welcome. However, my main concern was construct validity, and the rebuttal confirms it rather than resolving it. No revised PDF has been uploaded, so none of the promised changes can be checked. Framing. The authors now state that CloudGym “is not intended as a multi-agent benchmark” and does not evaluate “negotiation among independently reasoning agents”. Yet the title, abstract and conclusion still present the results as evidence about coordination. Scripted distractors do not reason or respond, and the agent is never told that others are acting. The benchmark measures robustness to unannounced state changes and compliance with a given policy. The claims should be reframed to match. Policy retrieval. Condition (ii) hands the policy to the agent, so it measures compliance, not conflict reasoning. A no-policy condition, which I asked for, was not run. In condition (iii), the tool returns the policy only if the agent cites the correct fingerprint after one of its own API responses has exposed it. Otherwise it gives an empty reply with no feedback. The (ii)→(iii) drop therefore partly measures whether agents discover this unstated convention. Taxonomy. Labelling each case with one category doesn’t show the categories are distinct. Under the paper’s definitions, SC cannot overlap with the others, but IA and EC can, and the IA definition is still too weak to check. The “planning / interpretation / execution” motivation is not related to established notions of concurrent conflict, e.g., EC is resource contention; SC covers write–write conflicts and invariant violations; the O-RAN works I cited distinguish direct, indirect and implicit conflicts. Positioning the taxonomy against these would show what CloudGym covers, what it leaves out, and whether the taxonomy is exhaustive, which was my original question. Metrics are unchanged. No coordination-specific metric was added. The proxy and snapshot logs should make detection rate, time to detection and consultation correctness directly measurable, but only end-to-end pass rate and a post-hoc failure taxonomy are reported. The engineering is solid, and the sharp degradation under concurrent changes is a useful finding. But the paper claims to measure coordination, and its design measures something narrower. I maintain my rating of 2.

We thank the reviewer for the follow-up. We have uploaded our revised manuscript. Please refer to our general post for a summary of changes.

Framing

We acknowledge that the term coordination may have caused some misunderstanding about CloudGym's scope. We have clarified its goals and non-goals (Sections 1 & 2).

Goal. CloudGym evaluates whether an individual cloud-management agent can detect, reason about, and resolve interference introduced by other participants (e.g., human operators, AI agents, and CI/CD pipelines) in shared cloud infrastructure. These are essential capabilities for coordination in shared cloud environments.

Non-goal. CloudGym is not intended to evaluate full-fledged multi-agent coordination, such as negotiation or multi-turn interactions among independently reasoning agents. Instead, we use scripted, event-driven distractors to introduce controlled and reproducible interference, allowing us to isolate and compare agents' capabilities under equivalent conditions.

We would also like to clarify the reviewer's statement that "the agent is never told that others are acting." While CloudGym deliberately does not reveal specific concurrent activities in advance, agents are informed of the shared infrastructure context through their instructions and available tools. Under the consulted-policy condition, the consultation tool is explicitly described as contacting the "platform team that owns the shared infrastructure." Crucially, this lack of prior knowledge reflects real-world cloud management. Independent teams, CI/CD pipelines, etc. do not necessarily announce their operations to every other participant. Agents must therefore discover concurrent changes, reason about their implications, and resolve conflicts without assuming prior knowledge of others' actions.

Policy retrieval

The reviewer’s original question was "Have you considered settings without explicit policy retrieval to better isolate conflict-resolution capabilities?" This asks about a setting without explicit policy retrieval, which is precisely what our experimental condition (ii) evaluates. In this condition, the resolution policy is provided upfront, and no policy retrieval is required.

However, in the recent follow-up, the reviewer appears to instead ask for a setting without any predefined resolution policy, which is a different experimental question.

For semantic conflicts and intent ambiguity, removing the resolution policy can leave the correct outcome underdetermined. For example, consider two teams managing the same AWS subnet. The application team requires the subnet's default route to use an Internet Gateway, while the security team requires it to use a NAT Gateway. These requirements are mutually incompatible, and neither team inherently has priority. Without a resolution policy, the agent has no basis for deciding which requirement should prevail. An oracle accepting only one configuration would implicitly impose an arbitrary priority rule.

In real-world cloud management, such decisions depend on organizational policies, resource ownership, or clarification from the responsible parties. Allowing an agent to choose arbitrarily would not constitute correct infrastructure management. Therefore, evaluating such a setting would require an alternative arbitration criterion to define correctness. CloudGym instead evaluates whether agents can reconcile concurrent management intents under specified resolution policies, allowing their outcomes to be evaluated objectively.

Taxonomy

Our taxonomy is cloud-specific. It is neither intended as a general-purpose conflict taxonomy across domains nor as a classification of low-level concurrency conflicts. Instead, it captures coordination challenges at different levels of cloud management, from high-level management intents to low-level cloud operations.

Specifically, Semantic Conflict (SC) concerns incompatible high-level intents, which may exist independently of concurrent execution. Such conflicts cannot be adequately characterized by traditional operation-level concurrency notions, as the fundamental issue is the incompatibility of the desired outcomes rather than the interleaving of operations. Intent Ambiguity (IA) concerns the interpretation of intents under changing cloud states, consisting of both high-level target ambiguity and lower-level state-dependent parameter selection. Execution Conflict (EC) is more closely related to traditional concurrency conflicts, where compatible operations temporarily interfere due to cloud-provider execution constraints. These distinctions are illustrated in Figure 3.

We do not claim that these categories are mutually exclusive or exhaustive across all cloud-management scenarios. Instead, they provide a practical decomposition of the coordination challenges CloudGym evaluates. Each benchmark case isolates one category by construction, although multiple categories may co-occur in real-world environments.

Reviewer_afAFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CloudGym, a benchmark that tests cloud-management agents on live AWS infrastructure shared with other parties. It has 102 cases across 19 AWS services and 65 resource types. In each case, scripted "distractors" act as concurrent participants. They are launched by state-based or API-based triggers, and an API proxy can briefly hold the agent's requests so that race conditions reproduce reliably. The distractors create three kinds of interference:

  • Semantic conflicts (SC): the two intents cannot both hold.
  • Intent ambiguity (IA): the target of the agent's intent becomes unclear.
  • Execution conflicts (EC): an operation is temporarily blocked but can succeed later.

Success is checked by Rego oracles over the initial state, the final state and the set of distractors that actually fired. The oracles enforce a resolution policy, which the agent can obtain through an MCP consultation tool. Cases are generated by an LLM agent from IaC-Eval seeds and certified on real AWS with a control run and an interference run.

Results cover Claude Code and Codex with five model backends:

  • Isolated tasks: 82.8–97.1% pass.
  • Policy given in the prompt: 19.6–55.4%.
  • Policy available only through consultation: 18.6–34.3% (best: Opus 5.5, 34.3%).
  • EC cases are much easier than SC and IA. Restricting agents to the SDK, CLI or IaC changes results only modestly.
Critical strengths
  1. A new and practically relevant evaluation axis. Existing benchmarks test agents in isolation or check static IaC. CloudGym tests them under concurrent changes in a live environment, and the SC/IA/EC taxonomy is simple and useful. The three conditions — control, prompted policy, consulted policy — run on the same cases, which separates coordination from base task ability. The drop from about 97% to about 34% is a striking, actionable result.
  2. Careful evaluation infrastructure. The pieces fit well:
    • Live-state snapshots cover every interface (SDK, CLI, IaC).
    • The proxy gives deterministic control over short-lived races.
    • Oracles conditioned on which distractors fired accept multiple valid final states.
    • The code is released.
Critical weaknesses
  1. The benchmark's validity is not established, so the headline gap is hard to interpret. The cases, distractors, resolution policies, oracles and test fixtures are all LLM-generated. Certification checks only that the control run passes and that the distractors fire; the interference-run outcome is explicitly not gated. The paper never shows that a correct coordination strategy passes each oracle under interference. There is no reference solution, no human audit, and no false-negative estimate.

    Even with the policy given in the prompt, SC pass rates stay at 5.4–40.2%. This could reflect genuinely poor coordination, but it could also come from:

    • the distractor firing after the agent's last check,
    • strict "no unnecessary change" checks against the initial state,
    • unstated conventions, such as having to query a fingerprinted resource through the consultation tool.

    Without a failure breakdown (detection vs. resolution vs. oracle mismatch) and a validity audit, the 97% → 34% gap cannot be attributed to coordination ability.

  2. The empirical evidence is thin for a benchmark paper.

    • Few runs and small categories. There are only 2 trials per case on live, nondeterministic AWS, and no confidence intervals. From Table 1, I estimate the split at about 46 SC, 38 IA and 18 EC cases, so EC results rest on 18 cases. The models under the consulted policy span only 28.4–34.3%, which is likely within noise.
    • Scaffold confounded with model family. Claude models run in Claude Code and GPT models in Codex.
    • Missing details. The main text is about 9 pages with no appendix. It omits per-category statistics, the generator model, certification yield, consultation rates per model (only Haiku's 11.3% is given), the fraction of runs in which distractors actually fired, and the AWS cost of running the benchmark.
Questions for authors
  1. For each case, does a reference solution (human or scripted) that coordinates correctly pass the oracle under interference? How many cases were checked by humans?
  2. Can you break down failures under the prompted and consulted conditions into: distractor not detected, wrong resolution, collateral change flagged against the initial state, and operational error?
  3. In what fraction of evaluated runs did each distractor actually fire? If a trigger never fires, the case reduces to the control task, which could inflate pass rates for some agents.
  4. Is the agent told that the environment is shared and when the consultation tool should be used? What does the tool return for queries that do not reference a distractor's fingerprinted resource?
  5. Please report the variance across the two trials, or more trials, and confidence intervals per category. Can you separate scaffold effects from model effects?
  6. The IA definition (∃S≠S′\exists S \neq S' with both satisfying ϕA∧ϕB\phi_A \wedge \phi_B) holds for almost any pair of intents and does not seem to capture ambiguity. Could you clarify the intended formalization?
2 comments

R2.4: What instructions and consultation mechanisms are available to agents?

Agents are informed of the shared-infrastructure context through the available instructions and tools. In the consulted condition, the consultation tool is described as contacting "the platform team that owns the shared infrastructure." In the prompted condition, the resolution policy is provided upfront and refers to resources maintained by other participants.

Agents are not explicitly informed that other participants will perform operations during their tasks. Detecting concurrent changes from the cloud state is a necessary part of building a reliable cloud management agent.

The consultation tool returns the relevant policy only when three conditions hold: (1) the distractor's change has completed, (2) the query cites the correct fingerprint, and (3) one of the agent's own API responses has already returned a fingerprint after the change. Every query opens a thread and returns its thread ID. If the conditions are not met, the reply list is empty, and no resolution policy is given. A reply that becomes available later is delivered with the agent's next tool call.

We will include the complete agent prompt and tool descriptions in the appendix of the revised manuscript.

R2.5: Regarding statistical reliability.

Please refer to Q3 of the General Reply for the statistical analysis.

Confidence interval per category based on two trials of data, as requested:

Category(i) -> (ii)(ii) -> (iii)(i) -> (iii)
SC−69.3 [−77.8, −60.4]−15.7 [−22.2, −9.8]−85.0 [−90.7, −78.9]
IA−50.0 [−60.3, −39.5]−16.6 [−24.7, −8.4]−66.6 [−76.8, −55.5]
EC−9.4 [−22.8, −0.6]−1.7 [−6.1, +2.2]−11.1 [−23.9, −2.8]

Regarding scaffold differences, CloudGym is designed to evaluate end-to-end coding agents performing cloud infrastructure management tasks. We therefore evaluate agents with their respective coding-agent environment, which provides the tools and execution capabilities necessary for infrastructure operations. Importantly, each agent uses the same scaffold across all three experimental conditions. Therefore, the substantial performance degradation observed cannot be attributed to changes in the scaffold. Our primary conclusion concerns this degradation under concurrent interference.

R2.6: What exactly constitutes IA, Intent Ambiguity?

We thank the reviewer for raising this question to us. We would like to clarify that IA concerns ambiguity in the interpretation of an intent, rather than simply the existence of multiple valid cloud states satisfying both intents.

Particularly, CloudGym implements primarily two types of IA:

  1. Target ambiguity: which resource the intent refers to.
    • Phrase: "the account's public hosted zone of example53.com"
    • Change: another participant creates a second public zone with the same name.
    • Effect: the phrase no longer picks one zone.
  2. Derived value ambiguity: a value computed from state that can change.
    • Phrase: "the next free CIDR in vpc X"
    • Change: another participant reserves the range the agent has read as free.
    • Effect: the value the agent computed earlier is now wrong.

R2.7: Additional experimental and implementation details.

Per-category case statistics are shown below.

SCIAECAll
Case counts463818102
Distinct Terraform resource types43353865
Resource types the task changes, median3363
Intent length, median words64.564.583.569

Benchmark generation: The benchmark cases were generated using Opus 5 through Claude Code.

Consultation rates: The table below reports the percentage of runs in which each model invoked the consultation tool at least once under condition (iii)

ModelAsked / RunsConsultation rate (of all runs)
Haiku 4.523 / 20411.3%
Opus 5.5103 / 20450.5%
Fable 596 / 20447.1%
GPT-5.544 / 20421.6%
GPT-5.6 Sol107 / 20452.5%

AWS execution cost: AWS cost of running the benchmark is estimated to be less than $0.013 per run, as most of the resources are billed based on usage and the lifetimes during execution are very short for each case. This estimate excludes model inference cost.

Thanks for the constructive feedback and for recognizing the potential of CloudGym We discuss the reviewer's concerns and questions below.

R2.1: How do we validate oracle correctness under interference?

Please refer to Q2 of our General Reply for a detailed description of our validation methodology.

In brief, we validate the 102 case-specific oracles using 2660 positive and negative test fixtures during generation. Additionally, our manual quality audit of 32 sampled cases found that the oracle and human assessment agreement is 31 out of 32 (96.9%). These results provide additional evidence supporting the consistency of our semantic oracles.

R2.2: What causes agent failures under interference?

As requested, we try to break the failures under the prompted and consulted into five categories at best effort:

  • No recorded observation of distractor: a requirement tied to a distractor failed, and there is no evidence to prove that agent observed the corresponding change.
  • Distractor requirement failed despite exposure: a requirement tied to a distractor failed, although the agent received at least one response exposing the change.
  • Collateral change to an initial-state resource: the agent modified or deleted a resource that existed before the run and that the oracle requires to be preserved.
  • Operation error: the agent did not complete the run.
  • Main intent failed: every distractor requirement was satisfied, but the main intent was not accomplished
Failure TypePrompted (ii)Consulted (iii)
No recorded observation of distractor33.1%36.6%
Distractor requirement failed despite exposure43.6%40.3%
Collateral change to initial-state resource0.0%0.8%
Operation error1.2%3.2%
Main intent failed only22.1%19.1%

R2.3: How reliably do distractors trigger?

We analyzed distractor activation across 2040 evaluation runs under the prompted and consulted conditions. At least one distractor fired in 99.7% of runs, while all configured distractors fired in 92.7% (over all runs). Overall, 97.4% of configured distractor executions were successfully triggered (over all distractors). The per-model breakdown is presented below:

ModelRuns where ≥1 firedRuns where all firedShare of distractors fired
Haiku 4.599.0%89.0%96.4%
Opus 5.599.5%91.2%96.0%
Fable 599.8%91.9%97.2%
GPT-5.5100.0%95.1%98.7%
GPT-5.6 Sol100.0%96.6%98.9%
All99.7%92.7%97.4%

We also examined the timing of distractor operations relative to agent completion. In 95% of runs included in the timing analysis, the final distractor landed at least 7.8 seconds before completion.

Reviewer_S9v2OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

CloudGym is a benchmark of 102 live AWS tasks (19 services, 65 resource types) that tests whether cloud-management agents can complete an intent while other parties change the same infrastructure.

Critical strengths
  • Live infrastructure with concurrent actors in shared infrastructure are realistic that no existing benchmark covers.
  • Difficult engineering and well motivated problem.
  • Released artifact
Critical weaknesses
  • The final verdict is also checked by the semantic oracle, not a human. How do we trust the oracle? Unclear if there are false positives?
  • Alternate baselines with explicit rules around resource sharing are missing
  • The paper is framed around shared cloud infrastructure and coordination among concurrent participants, but only one agent is actually evaluated (with the others being detractors)
Questions for authors

This is a valuable piece of work. Please see my concerns in Weaknesses.

In addition:

  • What is the oracle’s agreement with human experts on a sample of traces from the interference condition?
  • How diverse are the generated tasks? The paper reports 102 cases, 19 services, and 65 resource types, but does not clearly establish how many distinct underlying management intents are represented or how much structural similarity exists between cases.
  • How many distractors are present in each task? Is the benchmark primarily evaluating one concurrent change at a time, or does it include scenarios with multiple simultaneous participants and overlapping conflicts?
  • The experiments use only two independent trials per case and configuration. Given the stochasticity of LLM-based agents and the variability of live cloud operations, this is a limited basis for interpreting small performance differences. For example, the differences between models achieving 28.9%, 33.3%, and 34.3% overall pass rates may not be statistically meaningful.
1 comment

We thank the reviewer for the constructive feedback and for recognizing the value of CloudGym. We address the concerns and questions below.

R1.1: Oracle correctness and human validation

We validate oracle correctness through automated checks during case generation and live AWS certification, supplemented by human validation on a sampled population as mentioned in Q2.

Across the 102 cases, we use 2660 test fixtures. These fixtures test whether the oracles accept valid outcomes and reject incorrect ones. Please refer to Q2 of our General Reply for more details on the validation methodology.

R1.2: Regarding rule-based or policy-aware baseline

We thank the reviewer for suggesting alternative baselines with explicit resource-sharing rules. Our Prompted Policy condition (ii) already evaluates agents with resolution policies provided upfront in the agent prompt. We evaluate contemporary coding agents using their respective execution environments, which serve as our primary end-to-end baselines for cloud infrastructure management. This differs from introducing a separate rule-based coordination mechanism that enforces resource-sharing constraints. We thank the reviewer for raising this, and evaluating agents augmented with such mechanisms is a promising direction for future work.

R1.3: Regarding multi-agent setup

We would like to clarify that CloudGym evaluates a single AI agent handling concurrent intents in a shared cloud environment, rather than multi-agent collaboration. Please refer to Q1 of the General Reply for the design rationale.

R1.4: Benchmark case diversity

The 102 cases are derived from 93 distinct IaC-Eval source tasks: 84 are used once and 9 twice, where the second case pairs the same task with different distractors. The benchmark spans 19 AWS services and 65 resource types. Every case has its own intent description (102 unique). The median Jaccard word overlap of these intent description pairs is 0.094 (different intents share very few words in common).

To quantify structural diversity, we compare the resource types changed by the main task (initial-to-target delta) and those potentially changed by distractors. Among all 5151 case pairs, the mean pairwise Jaccard similarity is 0.074 (median 0) with a 95th percentile of 0.41. Only 3.4% of case pairs have a similarity greater than 0.5, indicating that high overlap in resource-type change patterns is uncommon.

These results indicate diversity in both the underlying management tasks and the interference introduced by distractors. We will include the detailed results in the appendix of the revised manuscript.

R1.5: Evaluation and distractors

Each case contains several distractors: at least 4 in every SC and IA case and 2 in every EC case. Every distractor is a distinct participant with its own role and responsibility (for example, a migration engineer, an observability engineer in the same case), so SC and IA cases generally involve four other participants. Within a case, all distractors belong to the same interference category.

Distractor operations are introduced sequentially rather than simultaneously at best effort. Their triggers are distributed across the agent's execution, and live certification checks that distractor changes occur before the reference agent's final action and are separated by at least 15 seconds or an intervening agent call.

R1.6: Statistical implications of model scores

Please refer to Q3 of our General Reply for the detailed statistical analysis. We calculate 95% confidence intervals using a case-level bootstrap stratified by taxonomy category (10000 resamples), with comparisons paired by case.

Across all five evaluated models, the pass rate decreases by 62.7-67.6 pp from the control to the consulted condition, with all paired 95% CI excluding zero. These results support our primary finding that concurrent interference substantially degrades agent performance, independently of the relative ranking of closely performing models.

Comments on the paper

5 comments

Q3 How robust are the evaluation results?

In this question, we use (i), (ii), and (iii) to refer to the same setup as section 3 of the submission.

Pass rates in 95% confidence intervals (CIs)

We calculate 95% CIs from a case-level bootstrap stratified by taxonomy category (10000 resamples). Both trials of each case are resampled together, and comparisons between experiment conditions are paired by case.

Model(i) Control(ii) Prompted(iii) Consulted
Haiku 4.582.8 [76.0, 89.2]19.6 [13.7, 26.0]18.6 [14.2, 23.5]
Opus 5.597.1 [94.1, 99.5]55.4 [46.6, 63.7]34.3 [26.5, 42.2]
Fable 596.1 [92.6, 98.5]52.0 [44.1, 59.8]28.4 [22.5, 34.3]
GPT-5.596.6 [93.6, 99.0]38.7 [31.4, 46.1]28.9 [22.5, 35.3]
GPT-5.6 Sol96.6 [93.1, 99.0]45.6 [38.2, 53.4]33.3 [27.0, 40.2]

Paired change in pass rate, in percentage points in 95% CIs

Model(i) -> (ii)(ii) -> (iii)(i) -> (iii)
Haiku 4.5−63.2 [−70.6, −55.4]−1.0 [−5.4, +3.4]−64.2 [−71.1, −56.9]
Opus 5.5−41.7 [−50.5, −33.3]−21.1 [−29.4, −13.2]−62.7 [−70.1, −54.9]
Fable 5−44.1 [−52.5, −35.8]−23.5 [−31.4, −15.7]−67.6 [−74.0, −60.8]
GPT-5.5−57.8 [−65.7, −50.0]−9.8 [−16.7, −2.9]−67.6 [−74.0, −60.8]
GPT-5.6 Sol−51.0 [−58.8, −43.1]−12.3 [−19.6, −5.4]−63.2 [−70.1, −56.4]

Across five models, the pass rate decreases by 62.7-67.6 percentage points from (i) to (iii), with all paired bootstrap CIs excluding zero. This provides further evidence of substantial performance degradation across all evaluated models, while accounting for case-level uncertainty.

Withholding resolution policies leads to an additional performance decline for four of the five models, with their paired 95% bootstrap confidence intervals excluding zero. For Haiku 4.5, the additional difference is small (−1.0 percentage points), and its confidence interval [−5.4, +3.4] includes zero.

These confidence intervals quantify uncertainty across benchmark cases based on the two trials per case in our original evaluation.

Q2 How do we validate the quality and realism of generated benchmark cases?

As described in Section 2.5, CloudGym incorporates multiple validation mechanisms throughout case generation. These include stage-gated consistency checks, fixture testing, and live AWS certification. In particular:

  • Oracle validation: across the 102 benchmark cases, we have generated 2660 test fixtures (both positives and negatives). The fixtures test whether the oracle accepts valid resolutions and rejects incorrect outcomes.
  • Policy validation: check to ensure the resolution policies do not directly reveal the expected solution.
  • Live certification: control and interference runs verify that tasks are executable, distractors can be triggered, and their effects are observable during agent execution.

While these automated checks help establish the technical correctness and consistency of generated cases at best effort, they cannot fully assess scenario realism or agreement with human judgment. We manually inspected cases during benchmark generation and conducted additional systematic human quality audits as follows:

We examined 32 cases, with 14 SC, 12 IA, and 6 EC cases, approximately matching the benchmark's category distribution (randomly selected ~30% from each category) and also covering roughly 30% of the full benchmark suite. A researcher on our team with experience in cloud infrastructure reviewed the intents, distractors, resolution policies, and execution traces.

The audit focused on three aspects:

  • Scenario realism: Whether the main intent and distractors represent plausible cloud management scenarios.
  • Coordination validation: Whether the distractor presents the correct taxonomy type? Whether the distractors introduce the intended coordination challenge at an appropriate point during execution?
  • Oracle agreement: Whether the human assessment of the observed final cloud state agrees with the semantic oracle's outcome?

This methodology gives the following result:

MetricSC (n=14)IA (n=12)EC (n=6)Total
Realism14/1412/126/6100%
Coordination validity13/1412/126/696.9%
  - Correct type of conflict?14/1412/126/6100%
  - Appropriate timing?13/1412/126/696.9%
Oracle agreement13/1412/126/696.9%

All 32 reviewed scenarios contain plausible cloud management scenarios and the intended types of coordination challenges. Interference timing was considered appropriate in 31 of 32 cases. In the remaining case (case ID 350), three distractors were triggered by a batch of parallel discovery reads, and their changes landed within 0.2 seconds of each other. The fourth distractor landed 3.6 seconds later.

The audit also found oracle agreement in 31 of 32 cases. The disagreement arose in an SC case (ID 430), where one of four distractors failed 11 seconds into execution due to a transient AWS error. It left a subnet marked for external access without completing its full configuration. The agent completed the main task and satisfied the requirements of the three successfully completed distractors. The reviewer considered the unconnected subnet a violation of the resolution policy, whereas the oracle accepted the run because the fourth distractor did not complete successfully. Under the oracle's rule, which we clarified in the manuscript (section 2.3), an incomplete distractor does not impose additional requirements on the evaluated agent. We consider this interpretation reasonable, since completing another participant's partially failed operation is not expected for the evaluated agent.

AuthorsOpenReview ↗

General Reply

We thank all the reviewers for the constructive feedback. This post intends to address the common concerns regarding the submission raised by all the reviewers.

Q1 Why does CloudGym use distractors instead of a multi-agent setup?

We would like to clarify that CloudGym is not intended as a multi-agent benchmark. Our goal is to evaluate how an individual AI agent handles concurrent management intents in a shared cloud environment. We chose to model other participants through a distractor design for two main reasons:

(1) To represent a shared cloud environment. Cloud infrastructure is typically shared by multiple participants performing concurrent operations. These participants are not necessarily AI agents. They might also be human operators, automated management systems, or CI/CD. Therefore, evaluating coordination in a shared cloud environment does not inherently require multiple AI agents. CloudGym uses distractors to represent these participants and their operations in the live cloud environment.

(2) To ensure reproducibility and comparability. We considered using multiple autonomous AI agents during the benchmark's design. However, independently operating AI agents introduce additional sources of randomness. Their decisions could change the type and timing of interference across evaluation runs, making it difficult to determine whether differences in performance arise from the evaluated agent or the other participants. CloudGym controls the interference conditions and expected resolution for each case by using event-driven distractors. This allows us to evaluate different agents against comparable challenges.

Supporting multiple evaluated agents as a team is a separate interesting extension that we have already identified as future work in Section 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.