- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EcoMAS, a framework that explains where the uncertainty of the final output comes from in dynamically orchestrated LLM-based multi-agent systems, and uses this information to improve the calibration of the orchestration policy. The authors define system output uncertainty as the second-order Tsallis entropy over semantic clusters and show that it decomposes exactly into stepwise contributions from agent selection, called the orchestration contribution, and from agent responses, called the agent execution contribution. These contributions are estimated without bias through paired sampling, which compares three continuation branches that fix the context, the selected agent, and the response, respectively. The same samples are then reused to estimate an orchestration policy gradient that reduces the semantic calibration error, defined as the distance to the true-answer distribution. The paper reports estimation accuracy, attribution faithfulness, and improvements in calibration and accuracy on MMLU-Pro, MATH-500, and ChaosNLI.
Critical strengths
S1. Splitting uncertainty into "which agent was selected" and "what that agent produced" is an intuitive formulation. Interpreting second-order Tsallis entropy as the probability that two independent runs produce semantically different answers, and splitting it into orchestration and agent-execution contributions, provides a useful unit for analyzing MAS behavior.
S2. The paired sampling estimator is simple and comes with theoretical guarantees. It estimates each contribution without bias using only three continuations per step, so the cost grows quadratically with the number of steps instead of exponentially. The variance analysis based on the binary same-cluster kernel is also helpful for choosing the sampling budget.
S3. Reusing the explanation samples for the calibration gradient is an elegant connection. The observation that the conditional semantic change defining the orchestration contribution coincides with the baseline-subtracted term of the calibration gradient allows samples generated for explanation to be reused for optimization at no extra cost.
Critical weaknesses
W1. The exact decomposition is not specific to Tsallis entropy, and the novelty is overstated. Each contribution is defined as the reduction in uncertainty at a step, so the fact that the contributions sum to the total uncertainty is simply a telescoping sum. The same decomposition holds for Shannon entropy as well. The claim of being "the first to establish an exact, nonnegative decomposition" is therefore overstated, and the actual contribution lies in the unbiased estimation and sample reuse that Tsallis-2 makes possible. The paper should reposition its contribution accordingly.
W2. The evaluation is aligned with EcoMAS's own definitions and objective, which favors the proposed method. The faithfulness experiment makes the top-ranked steps deterministic and measures the remaining uncertainty, which is essentially the same quantity that EcoMAS estimates. This shows that EcoMAS predicts uncertainty reduction well, but not that it explains MAS behavior. In the calibration experiment, only EcoMAS directly optimizes the evaluation metric, while the compared methods optimize task reward, so a baseline that optimizes the same objective with a standard policy gradient is needed. Moreover, on MMLU-Pro and MATH-500 the target distribution is one-hot, so this objective essentially reduces to accuracy optimization.
W3. What is being explained and how the experiments are configured are unclear, which hinders reproducibility. The contributions and their estimator are averages of "the t-th step" over many trajectories, yet under dynamic orchestration the t-th step involves different agents and contexts in different trajectories. The case-study figures, however, present the contributions as if they explained a single run, and how these run-specific contributions are defined and estimated is not described. It is also not possible to tell from the paper which model the orchestrator trained with policy gradient is, how it is parameterized and trained, or how the baselines were converted into step-level rankings.
Questions for authors
Q1. Since the same decomposition holds for Shannon entropy, what is the reason for choosing Tsallis-2 beyond ease of estimation? How much do the step rankings change compared with Shannon-based contributions?
Q2. How are the run-specific contributions shown in the case studies defined and estimated, and how do they relate to the population-level contributions? Does EcoMAS remain superior under independent criteria other than uncertainty reduction, such as predicting answer flips or changes in correctness?
Q3. What are the architecture and parameterization of the orchestrator, and what are its training settings? How does EcoMAS compare with a baseline that optimizes the same calibration objective with a standard policy gradient?