- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CLOUDGYM, a benchmark for evaluating cloud-management agents in live, shared environments with concurrent interference from other actors. The key motivation is that real cloud management is inherently multi-party: actions by one operator can invalidate another operator's plans, introduce ambiguity, or temporarily block execution.
Critical strengths
•Addresses important evaluation gap: Cloud infrastructure management is inherently multi-party, yet existing cloud-management benchmarks largely evaluate isolated execution or static artifact generation. The benchmark directly evaluates coordination under concurrent infrastructure modifications rather than purely intent realization in isolation. This is a meaningful capability that current agent evaluations appear to miss.
•Strong benchmark design: The benchmark executes against real AWS infrastructure rather than simulated plans or static configuration artifacts. Distractors operate through actual cloud operations, and final correctness is evaluated using cloud-state snapshots rather than generated code.
•Demonstrated difficulty: Experiments reveal substantial performance degradation under shared-environment conditions, suggesting meaningful challenge.
Critical weaknesses
•Limited theoretical grounding. The paper is motivated by scenarios in which multiple actors modify a shared cloud environment and interfere with one another's objectives. The broader literature [1, 2, 3] would view such settings through the lens of conflict detection, management and resolution. Surprisingly, this connection is neither explicitly discussed nor used to ground the benchmark design resulting a work that appears somewhat disconnected from prior work that studies similar phenomena under different terminology.
•Single-agent adaptation rather than multi-agent coordination. Although CLOUDGYM is framed as a multi-actor setting, evaluation is performed through a single agent interacting with scripted distractors that do not reason, communicate, or pursue evolving objectives. Consequently, the benchmark primarily measures robustness, recovery, and policy-aware adaptation to environmental changes rather than coordination or conflict resolution among multiple decision making entities.
•Policy retrieval dominates conflict reasoning. Since each benchmark instance defines a resolution policy that agents can obtain through consultation, many tasks reduce to detecting interference and applying the retrieved policy. This places substantial emphasis on information retrieval and policy compliance, while providing more limited evaluation of real cloud-based multi-agent dynamics such as reasoning about competing objectives, priority arbitration, or conflict-resolution strategies.
•Metrics not aligned with benchmark’s motivation. CLOUDGYM is motivated as a benchmark for shared environments with concurrent interference, yet the reported metrics focus almost exclusively on final task success and interaction cost. Additional metrics measuring conflict detection accuracy, policy-retrieval success, recovery after interference, adaptation latency, or correctness of conflict-resolution decisions would provide a more direct assessment of the capabilities the benchmark aims to evaluate.
The main concern is construct validity rather than implementation quality. The benchmark has clear value, but its claims about measuring coordination capabilities appear stronger than what the environments and metrics currently support. A narrower positioning as a benchmark for robustness to concurrent environmental changes and wider evaluation metrics would substantially strengthen the paper.
Minor: •Limited statistical robustness: Results are reported from only two independent trials per condition. While the large performance gaps support the main conclusions, reporting additional runs and variability measures (e.g., standard deviation or confidence intervals) would strengthen confidence in the robustness of the results. References: 1.del Prever, P. B., D’Oro, S., Bonati, L., Polese, M., Tsampazi, M., Lehmann, H., & Melodia, T. (2025). Pacifista: Conflict evaluation and management in open ran. IEEE Transactions on Mobile Computing, 24(10), 10590-10605. 2.Adamczyk, C., & Kliks, A. (2023). Conflict mitigation framework and conflict detection in o-ran near-rt ric. IEEE Communications Magazine, 61(12), 199-205. 3.Li, L., Liu, L., Huang, S., Lv, S., Lin, K., & Zhu, S. (2022). Agent-based multi-tier SLA negotiation for intercloud. Journal of Cloud Computing, 11(1), 16.
Questions for authors
•How does CLOUDGYM relate to existing conflict-resolution and coordination frameworks, and what distinguishes it from established conflict-management settings in multi-agent systems?
•In the current formulation, the evaluated agent interacts with benchmark-controlled distractors that execute predefined actions. To what extent do the authors view CLOUDGYM as measuring multi-agent coordination versus robustness to environmental interference and policy-aware adaptation? More specifically, what capabilities does CLOUDGYM evaluate that would not already be captured by a benchmark focused on dynamic environment adaptation?
•Can the authors provide additional theoretical justification for why the SC/IA/EC taxonomy form the appropriate decomposition of the problem space? Are the categories intended to be mutually exclusive and collectively exhaustive, and how do they relate to established taxonomies of conflict, resource contention, and coordination failures in multi-agent systems?
•How much of the benchmark difficulty comes from conflict reasoning versus discovering and applying the predefined resolution policy? Have you considered settings without explicit policy retrieval to better isolate conflict-resolution capabilities?
4 comments
R3.3: What coordination capabilities does CloudGym actually evaluate? What metrics directly measure coordination-related behavior?
CloudGym evaluates whether an agent can complete its cloud-management task while accounting for concurrent operations and respecting the resolution policy.
Our three experiment conditions contain different challenges. (i) Control evaluates the agent's ability to complete the original management task without any interference. (ii) Prompted policy introduces concurrent changes while providing the resolution policy upfront, evaluating whether an agent can detect and reconcile interference and complete the main task. (iii) Consulted policy additionally requires the agent to identify when consultation is necessary, obtain the relevant policy, and incorporate it into its actions. CloudGym evaluates these capabilities, not negotiation among independently reasoning agents.
Regarding metrics, final-state pass rates measure end-to-end success, while differences between conditions quantify the additional difficulty introduced by interference and consultation. For finer-grained evidence, we analyzed execution traces (R2.2).
- No recorded observation of distractor: a requirement tied to a distractor failed, and there is no evidence to prove that the agent observed the corresponding change.
- Distractor requirement failed despite exposure: a requirement tied to a distractor failed, although the agent received at least one response exposing the change.
- Collateral change to an initial-state resource: the agent modified or deleted a resource that existed before the run and that the oracle requires to be preserved.
- Operation error: the agent did not complete the run.
- Main intent failed: every distractor requirement was satisfied, but the main intent was not accomplished.
| Failure Type | Prompted (ii) | Consulted (iii) |
|---|---|---|
| No recorded observation of distractor | 33.1% | 36.6% |
| Distractor requirement failed despite exposure | 43.6% | 40.3% |
| Collateral change to initial-state resource | 0.0% | 0.8% |
| Operation error | 1.2% | 3.2% |
| Main intent failed only | 22.1% | 19.1% |
Among failed runs, 33.1% under prompted policy and 36.6% under consulted policy showed no evidence that the relevant distractor change was observed. In contrast, 43.6% and 40.3% had the change exposed to the agents, but agents failed the corresponding resolution requirements.
We will include these analyses and evaluate coordination capabilities in the revised manuscript.
R3.4: Is the difficulty primarily policy retrieval rather than conflict reasoning?
Condition (ii) directly addresses this question by providing the resolution policy upfront, without consultation. Based on the result reported in Q3 of the General Reply, comparing condition (ii) to condition (i), the pass rate reductions ranged from 41.7 percentage points to 63.2 percentage points, while from condition (ii) to condition (iii), the reduction only ranged from 1.0 percentage point to 23.5 percentage points. This indicates that obtaining the policy introduces additional difficulty, but policy retrieval alone cannot explain the substantial performance degradation under interference.
The failure breakdown in R2.2 provides further evidence. Under condition (ii), 33.1% of failed runs showed no recorded observation of the relevant distractor change, while 43.6% failed distractor-specific requirements despite the change being exposed to the agent. Together, these account for 76.7% of failures, even though the resolution policy was already available. This suggests that detecting and appropriately handling interference remains very challenging independently of policy retrieval.
We want to clarify that CloudGym evaluates the ability to detect and reconcile interference under specified coordination policies, rather than independently negotiating or inventing such policies. Providing an explicit policy also makes the expected resolution well-defined for objective evaluation.
R3.5: Statistical reliability.
We thank the reviewer for raising this concern. Please refer to Q3 of our General Reply for the detailed statistical analysis.
We report 95% confidence intervals using a case-level bootstrap stratified by taxonomy category. Across all five evaluated models, the pass-rate reductions from control (i) to consulted policy (iii) range from 62.7 to 67.6 percentage points, with all paired confidence intervals excluding zero. These results support our primary finding that concurrent interference substantially degrades agent performance.
Thank you for the detailed responses. The bootstrap CIs, the third trial, the distractor-activation rates and the failure breakdown are welcome. However, my main concern was construct validity, and the rebuttal confirms it rather than resolving it. No revised PDF has been uploaded, so none of the promised changes can be checked. Framing. The authors now state that CloudGym “is not intended as a multi-agent benchmark” and does not evaluate “negotiation among independently reasoning agents”. Yet the title, abstract and conclusion still present the results as evidence about coordination. Scripted distractors do not reason or respond, and the agent is never told that others are acting. The benchmark measures robustness to unannounced state changes and compliance with a given policy. The claims should be reframed to match. Policy retrieval. Condition (ii) hands the policy to the agent, so it measures compliance, not conflict reasoning. A no-policy condition, which I asked for, was not run. In condition (iii), the tool returns the policy only if the agent cites the correct fingerprint after one of its own API responses has exposed it. Otherwise it gives an empty reply with no feedback. The (ii)→(iii) drop therefore partly measures whether agents discover this unstated convention. Taxonomy. Labelling each case with one category doesn’t show the categories are distinct. Under the paper’s definitions, SC cannot overlap with the others, but IA and EC can, and the IA definition is still too weak to check. The “planning / interpretation / execution” motivation is not related to established notions of concurrent conflict, e.g., EC is resource contention; SC covers write–write conflicts and invariant violations; the O-RAN works I cited distinguish direct, indirect and implicit conflicts. Positioning the taxonomy against these would show what CloudGym covers, what it leaves out, and whether the taxonomy is exhaustive, which was my original question. Metrics are unchanged. No coordination-specific metric was added. The proxy and snapshot logs should make detection rate, time to detection and consultation correctness directly measurable, but only end-to-end pass rate and a post-hoc failure taxonomy are reported. The engineering is solid, and the sharp degradation under concurrent changes is a useful finding. But the paper claims to measure coordination, and its design measures something narrower. I maintain my rating of 2.
We thank the reviewer for the follow-up. We have uploaded our revised manuscript. Please refer to our general post for a summary of changes.
Framing
We acknowledge that the term coordination may have caused some misunderstanding about CloudGym's scope. We have clarified its goals and non-goals (Sections 1 & 2).
Goal. CloudGym evaluates whether an individual cloud-management agent can detect, reason about, and resolve interference introduced by other participants (e.g., human operators, AI agents, and CI/CD pipelines) in shared cloud infrastructure. These are essential capabilities for coordination in shared cloud environments.
Non-goal. CloudGym is not intended to evaluate full-fledged multi-agent coordination, such as negotiation or multi-turn interactions among independently reasoning agents. Instead, we use scripted, event-driven distractors to introduce controlled and reproducible interference, allowing us to isolate and compare agents' capabilities under equivalent conditions.
We would also like to clarify the reviewer's statement that "the agent is never told that others are acting." While CloudGym deliberately does not reveal specific concurrent activities in advance, agents are informed of the shared infrastructure context through their instructions and available tools. Under the consulted-policy condition, the consultation tool is explicitly described as contacting the "platform team that owns the shared infrastructure." Crucially, this lack of prior knowledge reflects real-world cloud management. Independent teams, CI/CD pipelines, etc. do not necessarily announce their operations to every other participant. Agents must therefore discover concurrent changes, reason about their implications, and resolve conflicts without assuming prior knowledge of others' actions.
Policy retrieval
The reviewer’s original question was "Have you considered settings without explicit policy retrieval to better isolate conflict-resolution capabilities?" This asks about a setting without explicit policy retrieval, which is precisely what our experimental condition (ii) evaluates. In this condition, the resolution policy is provided upfront, and no policy retrieval is required.
However, in the recent follow-up, the reviewer appears to instead ask for a setting without any predefined resolution policy, which is a different experimental question.
For semantic conflicts and intent ambiguity, removing the resolution policy can leave the correct outcome underdetermined. For example, consider two teams managing the same AWS subnet. The application team requires the subnet's default route to use an Internet Gateway, while the security team requires it to use a NAT Gateway. These requirements are mutually incompatible, and neither team inherently has priority. Without a resolution policy, the agent has no basis for deciding which requirement should prevail. An oracle accepting only one configuration would implicitly impose an arbitrary priority rule.
In real-world cloud management, such decisions depend on organizational policies, resource ownership, or clarification from the responsible parties. Allowing an agent to choose arbitrarily would not constitute correct infrastructure management. Therefore, evaluating such a setting would require an alternative arbitration criterion to define correctness. CloudGym instead evaluates whether agents can reconcile concurrent management intents under specified resolution policies, allowing their outcomes to be evaluated objectively.
Taxonomy
Our taxonomy is cloud-specific. It is neither intended as a general-purpose conflict taxonomy across domains nor as a classification of low-level concurrency conflicts. Instead, it captures coordination challenges at different levels of cloud management, from high-level management intents to low-level cloud operations.
Specifically, Semantic Conflict (SC) concerns incompatible high-level intents, which may exist independently of concurrent execution. Such conflicts cannot be adequately characterized by traditional operation-level concurrency notions, as the fundamental issue is the incompatibility of the desired outcomes rather than the interleaving of operations. Intent Ambiguity (IA) concerns the interpretation of intents under changing cloud states, consisting of both high-level target ambiguity and lower-level state-dependent parameter selection. Execution Conflict (EC) is more closely related to traditional concurrency conflicts, where compatible operations temporarily interfere due to cloud-provider execution constraints. These distinctions are illustrated in Figure 3.
We do not claim that these categories are mutually exclusive or exhaustive across all cloud-management scenarios. Instead, they provide a practical decomposition of the coordination challenges CloudGym evaluates. Each benchmark case isolates one category by construction, although multiple categories may co-occur in real-world environments.
We thank the reviewer for highlighting relevant work on conflict management and coordination. We will address the concerns and questions below.
R3.1:How does CloudGym relate to prior conflict-management research?
We thank the reviewer for sharing this work with us.
PACIFISTA [1] and Adamcyzk and Kliks [2] investigate conflict detection and mitigation among control applications in O-RAN environments. Li et al. [3] study multi-agent negotiation mechanisms for intercloud service-level agreements. These work address important coordination problems involving multiple participants operating over shared resources.
CloudGym addresses a related but distinct research problem. From a workload perspective, the cited O-RAN works focus on interactions among radio-network control parameters, whereas cloud infrastructure management involves a different type of shared state. Cloud infrastructure can be viewed as a heterogeneous, mutable resource graph. In this graph, resources can be created, modified, and deleted, and relationships among resources (such as dependencies) can change. Moreover, many individual resources have their own lifecycle states and operational constraints. Correctness often depends on invariants spanning multiple resources. In summary, cloud infrastructure is a heterogeneous environment with dynamic topology, resource configurations, and lifecycle semantics.
From a task perspective, O-RAN conflict-management frameworks focus on interactions among applications controlling radio-network parameters. However, CloudGym evaluates general-purpose AI agents performing infrastructure-management tasks across heterogeneous AWS services and resources. These tasks involve creating, modifying, and managing cloud infrastructure while other participants introduce concurrent changes. CloudGym evaluates whether agents can detect such changes, reconcile competing management requirements, and complete their tasks under interference. Furthermore, rather than proposing a conflict-detection or resolution mechanism, CloudGym provides a benchmark for evaluating these capabilities in a live cloud environment.
We acknowledge that these works provide valuable context for the conflict-management challenges motivating CloudGym. We will add them to the discussion and related work of the paper.
R3.2: Why SC/IA/EC? Are they mutually exclusive and exhaustive?
We motivate our taxonomy from the lifecycle of a cloud coordination framework. Effective coordination requires reasoning about potentially conflicting management intents during planning, maintaining unambiguous interpretations as the cloud state evolves, and adapting operations to interference during execution. These motivate three categories of coordination challenges: SC, IA, and EC, respectively.
Each CloudGym case is constructed around one of these 3 interference types. And each case belongs to exactly one category. Thus, the categories are mutually exclusive for the benchmark suite by construction. However, we do not claim that they are mutually exclusive or collectively exhaustive for all real-world scenarios, where multiple forms of interference may coexist. Unlike existing conflict taxonomies that classify interactions, SC/IA/EC primarily focus on challenges an agent must address to complete its task. We will clarify this motivation and the scope of the taxonomy in the revised manuscript.