acceptodds
Under review as a conference paper at ICLR 2027

InfraMind: Infrastructure-Aware Multi-Agent Orchestration

Abstract

Multi-agent LLM systems spend additional computation on reasoning and collaboration, but shared serving queues can prevent that computation from producing answers before a deadline. This creates a coupled decision: the reasoning workflow determines the load, while changing load determines which reasoning choices remain affordable. We introduce InfraMind, a hierarchical policy that adapts workflow planning and execution to serving conditions. At arrival, a planner selects the collaboration graph using task information, load, and the time budget. Before each agent call, an executor jointly selects the model and reasoning strategy using refreshed telemetry and remaining time. A shared latency penalty connects the two policies. Our analysis characterizes why observing load matters when congestion reverses the preferred computation. Across five benchmarks on a shared five-model deployment, InfraMind improves low-load accuracy by up to percentage points over the strongest evaluated baseline. On 500 MATH questions with matched EDF scheduling and per-request deadlines of –, it increases correct, on-time answers from to . With 15 models and Azure timings replayed on the same benchmark and deadline tiers, this fraction rises from to over MasRouter. These results support adapting reasoning decisions to the serving capacity available throughout a workflow.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Multi-Agent LLMs, LLM Orchestration, LLM Serving, Infrastructure-Aware Reasoning, Adaptive Reasoning, Model Routing, Test-Time Compute, Resource-Aware Inference, Reinforcement Learning, Deadline-Aware Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e7S3OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes InfraMind, a multi-agent orchestration framework that uses serving load and time budgets when deciding how much reasoning to perform. A planner selects the collaboration graph when a request arrives, and an executor chooses the model and reasoning prompt before each agent call using refreshed serving telemetry. The two policies share a latency penalty during training, and pending calls are scheduled by deadline. Experiments on five benchmarks measure accuracy, latency, and correct answers delivered on time. In the 15-model study with replayed Azure arrival timings, the paper reports a 4.7x improvement in correct, on-time answers over MasRouter.

Critical strengths
  1. The paper identifies a real limitation of task-only orchestration: a model or reasoning strategy that is useful in isolation can be a poor choice when its queue is congested. Adapting the requested computation to the available serving capacity is a useful direction. Measuring correct, on-time answers also captures the practical objective better than reporting accuracy and latency separately.

  2. The common-EDF experiment on MATH provides evidence that orchestration changes have value beyond changing the queue scheduler. Delivered accuracy improves from 20.4% for MasRouter to 33.8% for InfraMind under this control. The executor telemetry ablation is also helpful for understanding why observing the serving state matters.

Critical weaknesses
  1. The comparisons do not consistently match scheduling and training conditions. In Table 1, InfraMind uses EDF on MATH, HumanEval, and MMLU-Pro, while the baselines use FCFS. The common-EDF MATH experiment shows that the benefit persists there, but does not resolve the scheduling contribution on the other two tasks. In the 15-model study, Appendix D reports 15 training epochs for InfraMind versus five for MasRouter, additional load and budget training for InfraMind, and a zero cost-quality coefficient for MasRouter. These differences may favor InfraMind and make the size of the policy's advantage difficult to judge.

  2. The added value of the planner and shared penalty remains unclear. The telemetry ablation supports using serving-state inputs, but does not establish why the full hierarchy is needed over a simpler load-aware extension of MasRouter. Section 5.3 evaluates load-conditioned planning, the shared penalty, and joint model-strategy selection together. The discussion would benefit from less repeated motivation and a clearer explanation of what these choices contribute individually.

The writing in this paper omits context needed for human readers, while creating unnecessarily long, repetitive basic claims and presenting long lists of experimental numbers with little interpretation.

Questions for authors
  1. How do the HumanEval and MMLU-Pro comparisons change when all methods use the same queue scheduler, as in the common-EDF MATH experiment?

  2. In the 15-model experiment, does the advantage persist when MasRouter receives comparable training effort and an objective suited to the same deadlines?

  3. How much does the load-conditioned planner contribute when the executor is already load-aware? Would the original task-driven planner paired with the proposed executor achieve similar results?

Reviewer_hLEgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InfraMind proposes infrastructure-aware orchestration for multi-agent LLM workflows. At request arrival, a planner chooses the collaboration structure, agent count, and roles using task information, serving load, and a latency budget. Before each agent call, an executor selects a model and prompting strategy based on updated queue, latency, cache, and budget information. The two policies share a latency penalty during reinforcement learning, while deadline-based scheduling prioritizes waiting calls. Experiments evaluate correctness, latency, and correct answers delivered before deadlines. The approach improves timely delivery, but its quality tradeoffs and advantages over alternative resource allocations require clearer evaluation and stronger user-facing quality controls.

Critical strengths
  1. The paper addresses an important interaction between agent reasoning decisions and serving congestion.

  2. Jointly adapting workflow structure, model selection, and reasoning effort offers useful flexibility under changing resource availability.

  3. The evaluation includes real concurrent serving, multiple benchmarks, bursty arrivals, and a comparison with matched queue scheduling.

Critical weaknesses
  1. Separate the latency improvement from the accuracy sacrificed. The MMLU-Pro ablation illustrates this concern: adding infrastructure awareness reduces raw accuracy from 61.0% to 57.8%, while increasing correct, on-time answers from 21.8% to 28.0%. This is a meaningful tradeoff, but its acceptability depends on the application. Please report accuracy–latency tradeoff curves and compare methods at matched accuracy levels. Breaking down accuracy changes by task difficulty would also reveal whether difficult requests disproportionately bear the cost of reducing congestion.

  2. Compare routing adaptation with changing resource allocation. The main deployment keeps five models resident on two B200 GPUs with fixed placements. When a preferred model becomes congested, InfraMind adapts the requested computation. Another option is to allocate more capacity to that model, potentially preserving its answer quality. Please compare against additional replicas or alternative GPU allocations under the same hardware or monetary budget. The larger heterogeneous model-pool experiment does not resolve this question because it does not evaluate demand-driven replication of heavily used models.

  3. Make quality an explicit serving requirement. This concern extends beyond measuring average accuracy. A policy that balances correctness against latency does not ensure that a particular application or request class receives its required quality. Users may accept longer waiting times more readily than a less reliable answer, especially if the quality change happens automatically during congestion. Please explain how callers can specify an acceptable quality target or restrict model substitution, and how the system behaves when both quality and latency requirements cannot be satisfied. Evaluation with such restrictions would clarify its practical applicability.

Questions for authors
  1. How much latency or delivered-accuracy improvement remains when methods are compared at matched overall accuracy?

  2. Can replication or resource reallocation achieve comparable benefits while preserving model choice under the same resource budget?

  3. How can users prevent quality degradation under congestion, and what happens when their quality and latency requirements conflict?

Reviewer_9gm4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InfraMind targets the problem how multi-agent systems serve requests efficiently under a given per-request budget, which is so called "useful and affordable" in this paper. The core observation is the agent graph and both reasoning effort and model choice of each agent should be determined and adjusted according to the serving load (infrastructure load). Therfore, InfraMind introduces learned planner and executor policies to determine the agent graph and select the reasoning effort and model choice based on the serving load. InfraMind formalizes this problem and trains them jointly under a shared optimization objective. Evaluation shows that InfraMind can outperform baselines in completing more tasks within the specified budget and reducing average task completion latency.

Critical strengths

1.This paper targets an important and relevant problem, how to make multi-agents systems efficient to complete tasks with models of different sizes. And determining and adjusting multi-agent orchestration based on serving load is a sensible perspective to make multi-agent systems both useful and affordable. This perspective is interesting and practical. 2.Compared with the evaluated baselines, InfraMind can complete more tasks with specified budgets and shorten the average task latency substantially.

Critical weaknesses

1.The core design of this paper is multi-agent orchestration according to serving load. However, there are some existing works exploring this direction. Both Aragog and Chimera adjust multi-agent orchestration based on serving load, in aspects of both agent graph (workflow) and model choice. This paper lacks correct reference and discussion of these papers. 2.The ablations cannot sufficiently tell why to make each design choices. Even though these designs are coupled as paper describes, ablation for each components (planner and executor) are required at least. And there are more algorithmic choices requesting validation.

Questions for authors

1.What is the difference between this paper and Aragog and Chimera, what is the most key design of this paper beyond Aragog and Chimera? 2.Can you validate the effects of planner and executor respectively? 3.Can you clarify how to migrate already trained planner and executor as deployed models change, is training from scratch required? 4.In this evaluation protocol of Sec 5.1, why not use EDF for baselines and why use different scheduling policies for InfraMind?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.