acceptodds
Under review as a conference paper at ICLR 2027

Explaining and Calibrating Dynamically Orchestrated LLM MAS through Stepwise Uncertainty Decomposition

Abstract

Motivated by the need for trustworthiness and interpretability, reliable uncertainty decomposition and calibration techniques provide transparency into the black-box behavior of Large Language Models (LLMs). However, extending this explainability to LLM-based Multi-Agent Systems (MAS) remains challenging, as stochastic interactions among agents and system-level control introduce complex, nested uncertainty. To address this challenge, we propose Explanation and Calibration through Orchestration for Multi-Agent Systems (EcoMAS), a framework that reveals stepwise information contributions in dynamically orchestrated MAS through uncertainty decomposition and optimizes orchestration policies to improve calibration. We model system output uncertainty using second-order Tsallis entropy and derive an exact decomposition into stepwise information contributions from orchestration and agent execution. We develop a paired sampling algorithm that efficiently produces unbiased estimates of these contributions with theoretical finite-sample concentration bounds. We further derive an unbiased calibration gradient estimator from the semantic increments underlying information attribution, reusing the explanation samples for orchestration policy optimization. We evaluate EcoMAS on MMLU-Pro, MATH-500, and ChaosNLI to assess estimation accuracy, explanation faithfulness, and orchestration calibration. The results show that paired sampling accurately estimates information contributions with small sampling budgets. Case studies and intervention evaluations qualitatively illustrate how the attributions explain answer revision and reinforcement, while quantitative faithfulness evaluations demonstrate that EcoMAS reliably identifies steps that strongly influence system output uncertainty. Finally, comparisons of orchestration optimization methods show that EcoMAS reduces calibration error while improving task accuracy, highlighting the association between semantic calibration and MAS output quality.

Primary area:
interpretability and explainable AI
Keywords:
Multi-Agent System, Explainability, Uncertainty Decomposition

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_uwxrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EcoMAS, a framework that explains where the uncertainty of the final output comes from in dynamically orchestrated LLM-based multi-agent systems, and uses this information to improve the calibration of the orchestration policy. The authors define system output uncertainty as the second-order Tsallis entropy over semantic clusters and show that it decomposes exactly into stepwise contributions from agent selection, called the orchestration contribution, and from agent responses, called the agent execution contribution. These contributions are estimated without bias through paired sampling, which compares three continuation branches that fix the context, the selected agent, and the response, respectively. The same samples are then reused to estimate an orchestration policy gradient that reduces the semantic calibration error, defined as the distance to the true-answer distribution. The paper reports estimation accuracy, attribution faithfulness, and improvements in calibration and accuracy on MMLU-Pro, MATH-500, and ChaosNLI.

Critical strengths

S1. Splitting uncertainty into "which agent was selected" and "what that agent produced" is an intuitive formulation. Interpreting second-order Tsallis entropy as the probability that two independent runs produce semantically different answers, and splitting it into orchestration and agent-execution contributions, provides a useful unit for analyzing MAS behavior.

S2. The paired sampling estimator is simple and comes with theoretical guarantees. It estimates each contribution without bias using only three continuations per step, so the cost grows quadratically with the number of steps instead of exponentially. The variance analysis based on the binary same-cluster kernel is also helpful for choosing the sampling budget.

S3. Reusing the explanation samples for the calibration gradient is an elegant connection. The observation that the conditional semantic change defining the orchestration contribution coincides with the baseline-subtracted term of the calibration gradient allows samples generated for explanation to be reused for optimization at no extra cost.

Critical weaknesses

W1. The exact decomposition is not specific to Tsallis entropy, and the novelty is overstated. Each contribution is defined as the reduction in uncertainty at a step, so the fact that the contributions sum to the total uncertainty is simply a telescoping sum. The same decomposition holds for Shannon entropy as well. The claim of being "the first to establish an exact, nonnegative decomposition" is therefore overstated, and the actual contribution lies in the unbiased estimation and sample reuse that Tsallis-2 makes possible. The paper should reposition its contribution accordingly.

W2. The evaluation is aligned with EcoMAS's own definitions and objective, which favors the proposed method. The faithfulness experiment makes the top-ranked steps deterministic and measures the remaining uncertainty, which is essentially the same quantity that EcoMAS estimates. This shows that EcoMAS predicts uncertainty reduction well, but not that it explains MAS behavior. In the calibration experiment, only EcoMAS directly optimizes the evaluation metric, while the compared methods optimize task reward, so a baseline that optimizes the same objective with a standard policy gradient is needed. Moreover, on MMLU-Pro and MATH-500 the target distribution is one-hot, so this objective essentially reduces to accuracy optimization.

W3. What is being explained and how the experiments are configured are unclear, which hinders reproducibility. The contributions and their estimator are averages of "the t-th step" over many trajectories, yet under dynamic orchestration the t-th step involves different agents and contexts in different trajectories. The case-study figures, however, present the contributions as if they explained a single run, and how these run-specific contributions are defined and estimated is not described. It is also not possible to tell from the paper which model the orchestrator trained with policy gradient is, how it is parameterized and trained, or how the baselines were converted into step-level rankings.

Questions for authors

Q1. Since the same decomposition holds for Shannon entropy, what is the reason for choosing Tsallis-2 beyond ease of estimation? How much do the step rankings change compared with Shannon-based contributions?

Q2. How are the run-specific contributions shown in the case studies defined and estimated, and how do they relate to the population-level contributions? Does EcoMAS remain superior under independent criteria other than uncertainty reduction, such as predicting answer flips or changes in correctness?

Q3. What are the architecture and parameterization of the orchestrator, and what are its training settings? How does EcoMAS compare with a baseline that optimizes the same calibration objective with a standard policy gradient?

Reviewer_xNgCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EcoMAS, which is used to explain and calibrate the dynamic scheduling LLM multi-agent system. The final output is mapped to semantic categories, and the uncertainty of the answer distribution is characterized using second-order Tsallis entropy, and it is precisely decomposed into the contribution of each agent's choice and the contribution of the agent's execution at each step. To avoid enumerating all subsequent execution trajectories, the paper designs a paired sampling estimator, proves its unbiasedness, and provides bounds in a finite sample set. Based on this, the authors further utilize the incremental conditional semantic distribution to construct the gradient for the calibration target, eliminate the correlation bias through leave-one-out estimation, and enable the samples from the explanation stage to be reused for the optimization of the scheduling strategy.

Critical strengths
  1. The research motivation is clear. The final output of the dynamic multi-agent system is influenced by both the scheduling decisions and the randomness of agent generation. The paper quantifies the contributions of "selecting which agent" and "what content the agent generates", which can provide more detailed diagnostic information than simply reporting the final answer entropy.
  2. The author utilized the equivalent relationship between the second-order Tsallis entropy and the variance of one-hot semantic features to decompose the final output uncertainty into non-negative incremental conditional distributions. Subsequently, the square norm of the same increment was used for attribution, and its inner product with the error of the target distribution was used to construct the optimization gradient.
  3. The experiment provided evidence from three aspects: estimation accuracy, attribution effect, and downstream optimization. The finite execution tree experiment directly compared the estimated values with the precise targets, enabling a more targeted examination of the estimator. The intervention experiment investigated whether the attribution ranking could identify the steps that affect the uncertainty of the system output. The final scheduling optimization experiment covered three tasks and three backbones, and the improvement in semantic calibration error showed good consistency.
Critical weaknesses
  1. EcoMAS directly optimizes the squared distance between the system answer distribution and the reference distribution. However, the existing scheduling baseline may not optimize the same objective. Therefore, a lower semantic calibration error can support the effectiveness of the entire training scheme, but it does not fully explain that the benefits mainly come from paired sampling, conditional semantic increment, or its specific gradient estimation method.
  2. Theoretical contributions measure the reduction in expected conditional uncertainty brought about by observing a certain step selection or answer under the original random execution distribution. The experiment, on the other hand, changes the high-ranked steps to deterministic execution, such as fixing the selection of the agent with the highest probability or using greedy decoding, and measures the remaining uncertainty of the new system. These two have a reasonable connection, but they are not the same. Determinization alters the execution distribution, and continuous intervention may also change the subsequent context and the applicability of the original attribution ranking.
  3. The computational cost in a real reasoning environment is still unclear. Pairwise sampling avoids exponential enumeration, which is an important advantage, but its overall cost remains O(BT2)O(BT^2), and multiple positions of each main trajectory need to be re-generated for subsequent execution. For longer processes or expensive models, this may result in significant overhead. The appendix has clearly stated that the over 30,000-fold acceleration comes from the computations on the finite diagnostic tree, and this cannot be used to determine the end-to-end efficiency.
  4. The experimental scale and statistical report have limited the assessment of the stability of performance advantages. The scheduling optimization experiments used 300 training questions and 100 test questions for each dataset, and each test question was run 10 times. This setup can provide preliminary evidence, but the main result table lacks question-level confidence intervals or significance analysis. Some accuracy differences are very small, for example, Qwen2.5 on MATH-500, the accuracy rates of EcoMAS and Puppeteer are 0.742 and 0.738 respectively.
Questions for authors
  1. Under the same strategy parameterization, training data, and generation budget, how will the standard score-function policy gradient and other reasonable estimators differ in terms of their gradient variance, convergence speed, and final performance when optimizing the same distribution matching objective?
  2. Could the author report the actual number of LLM calls, the number of generated tokens, the training time and inference time, and compare the quality of explanations with the optimization benefits under the same generation budget?
  3. Could the author provide a paired statistical analysis based on the questions, and examine the results under different training divisions and random seeds for strategy training?
Reviewer_eoMdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies multi-agent LLM systems where an orchestrator picks one agent per step, and splits the uncertainty of the final answer exactly into per-step contributions from agent choice and agent response. These contributions are estimated without bias by rerunning continuations from each step and checking whether the final answers fall in the same cluster. The same samples give an unbiased gradient for training the orchestrator to match a target answer distribution. On MMLU-Pro, MATH-500 and ChaosNLI with three 7B-20B models, the method beats baselines on a faithfulness test and on calibration error.

Critical strengths
  • The paper analysis is interesting: decomposing and attributing the final uncerainty of the MAS to individual agents is relevant, and an important timely topic.
  • The paper proposes a detailed theoretical development of the decomposition of uncertainty. It seems grounded and relevant (although I did not check in detail).
Critical weaknesses
  • The method assumes a narrow agentic setting, in contradiction with most agentic applications, which strongly weaken the paper. Assumption 1 rules out uncertainty coming from outside the context, and assumes full access to the context of previous agent steps. But a large part of agentic uncertainty comes from outside the context, in particular tool calls [1], or from the split of context during agent communication or delegation [2]. The paper assumes that the agentic task can be re-run many times, but it is almost always not possible when the agents take action in the words (which is the case in practice of almost all agentic applications). The paper also assumes that the number of agentic steps is fixed (T). The studied agentic stochasticity is severely limited and not representative of agentic uncertainty.
  • The experimental setting is also weak: The setup is closer of LLM and reasoning UQ than of agent UQ [3]. The experiments are all based on QA settings, which are far from applications of MAS. The paper presents EcoMAS as a framework to study MAS. But what it does is to give the same original question at each hand-off (with the previous agent outputs). The canonical use-case of MAS is an orchestrator to split a complex task into sub-tasks given to several sub-agents. The results say little about how uncertainty behaves in MAS. Additional points are missing and weaken the experimental setting: Calibration is not evaluated OOD, either on different splits within the same benchmark, or from one benchmark to another. Overall, the experimental contributions are quite shallow.

References

  • [1] Ye et al., Uncertainty Quantification for LLM Function-Calling, TMLR, 2026
  • [2] Emde et al., Lost in Communication: Uncertainty Propagation in Multi-Agent Systems, ICML-W 2026
  • [3] Oh et al., Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities, ACL 2026
Questions for authors
  • In Table 2, are the methods roughly the same test-time computational cost (number of tokens)? Same question for the training computational cost?
Reviewer_St9BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies uncertainty attribution and calibration in dynamically orchestrated LLM multi-agent systems (MAS). Existing uncertainty quantification methods score individual responses, and learned orchestrators are typically trained on task reward. The authors argue that earlier selections and responses reshape later contexts, so local response variability cannot reveal a step’s contribution to final-answer uncertainty; calculating this contribution exactly requires enumerating exponentially many continuations; and reward-based training does not control how probability is distributed across semantic answers. To address this, they propose EcoMAS, which measures final-answer uncertainty with second-order Tsallis entropy, attributes it to each step’s agent selection and agent response via the expected squared change in the conditional answer distribution, estimates these contributions with continuation branches that fix a prefix and resample the remainder, and reuses the same branches to estimate a policy gradient that moves the system’s answer distribution toward a target distribution. They theoretically show that, under Markov and fixed-clustering assumptions, the contributions sum exactly to the total uncertainty, that both the contribution estimates and the gradient estimator are unbiased, and that the contribution estimates concentrate at rate O(B−1/2)O(B^{-1/2}). Across three benchmarks and three backbones, they report that EcoMAS attributions are accurately estimated with small budgets and identify the steps that most reduce system uncertainty, and that EcoMAS orchestration achieves the lowest calibration error and generally the highest accuracy among learned orchestration baselines.

Critical strengths
  1. The paper targets a timely and practically important problem. As LLM-based MAS are deployed for complex reasoning, knowing which orchestration decisions and agent responses drive the uncertainty of the final answer is a prerequisite for diagnosing and trusting these systems, and it complements existing work on failure attribution in MAS. Framing the question at the level of the final answer distribution, rather than the variability of individual responses, correctly accounts for the fact that earlier steps reshape the context of later ones.
  2. The framework is clean and self-consistent, and its derivations are rigorous. Using the variance form of second-order Tsallis entropy, the paper decomposes final-answer uncertainty exactly into nonnegative contributions from agent selection and agent execution (Thm. 2), and estimates each contribution without bias using a single continuation per prefix (Prop. 1). Separating which agent is chosen from what it produces is a natural lens for analyzing dynamically orchestrated MAS.
  3. The evaluation spans diverse answer spaces, including a genuinely distributional target. Beyond multiple-choice and math, ChaosNLI uses human label distributions as the target, a setting where matching a distribution rather than a single label is meaningful. Results are reported on three backbones.
Critical weaknesses
  1. On two of the three benchmarks, the “calibration error” is the Brier score and is largely related to accuracy. For MMLU-Pro and MATH-500, q⋆q^\star is one-hot, so by the identity in App. B.7, both the objective and CErr reduce exactly to the Brier score (1−qθ(c))2+∑s≠cqθ(s)2(1-q_\theta(c))^2+\sum_{s\neq c}q_\theta(s)^2, which is zero only when the system always answers correctly. By the Murphy decomposition [1], the Brier score combines reliability, i.e., calibration, with resolution and an uncertainty term, so a more accurate system obtains a lower score even without being better calibrated. The association between lower CErr and higher accuracy reported in the abstract and Sec. 4.5 is therefore not evidence linking calibration to output quality, and only ChaosNLI tests distributional alignment; the authors should report standard calibration metrics (e.g., ECE using qθ(y^)q_\theta(\hat y) as confidence, and confidence on wrong answers) or rename the objective.
  2. The faithfulness evaluation is circular. EcoMAS scores each step by the expected reduction in the system uncertainty H(S)H(S), and RQ3 evaluates a ranking by how quickly H(S)H(S) decreases when the top-ranked steps are made deterministic. Because the method and the metric measure the same quantity, EcoMAS’s advantage in Table 1 is largely expected by construction and says little about whether its attributions explain the system’s behavior, especially since the baselines were not designed to predict this quantity. An evaluation that does not rely on H(S)H(S), such as checking whether high-attribution steps coincide with the annotated decisive errors in Who&When [2], would be more convincing.
  3. The experiments do not isolate the source of the gains in RQ4. EcoMAS differs from the baselines simultaneously in its objective, its gradient estimator, and possibly its compute budget, and on ChaosNLI it is trained directly on the human distribution that is also used for evaluation. Without ablations that vary one factor at a time, Table 2 cannot show that the gains come from the proposed estimator, which is the paper’s claimed contribution. Moreover, Uniform is the only untrained reference, so it is unclear how much of the gap reflects training rather than the pretrained orchestrator's initial ability.
  4. The case studies conflict with the theoretical definition of the attributions. By Eq. 2, AtA_t averages over all contexts and agent selections that can occur at step tt, so it measures how much step tt matters on average, yet Figs. 1–3 attach these values to specific decisions in a single run (e.g., “Choose Agent: Social Institutions”). The narratives in Sec. 4.3, including the pattern that answer revision contributes less than subsequent reinforcement, are therefore not supported by the displayed values, which may overstate or understate the effect of the depicted decision. Explaining a single run would require a pointwise quantity such as ∥μt(ct,rt)−μt(ct)∥2\|\mu_t(c_t,r_t)-\mu_t(c_t)\|^2 at the realized (ct,rt)(c_t,r_t). Still, such pointwise contributions do not sum to H(S)H(S) along a trajectory, so the exact decomposition and the explanation of individual runs cannot both hold in the current framework.
  5. The efficiency and finite-sample guarantees do not hold at practical budgets. With T=5T=5, δ=0.05\delta=0.05, and B=32B=32, the bound in Thm. 3 is about 0.61, much larger than nearly all estimated contributions, so the guarantees do not support any experimental conclusion. The efficiency claim (over 30,000× speedup, Table 5) is made against exhaustive enumeration, whereas the natural baseline is nested Monte Carlo at the same budget; the end-to-end cost is also unreported, although the branches require roughly 9× the agent calls of plain sampling at T=5T=5.

[1] Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4), 595–600.

[2] Zhang, S., Yin, M., Zhang, J., Liu, J., Han, Z., Zhang, J., Li, B., Wang, C., Wang, H., Chen, Y., & Wu, Q. (2025). Which agent causes task failures and when? On automated failure attribution of LLM multi-agent systems. In Proceedings of the 42nd International Conference on Machine Learning, 76583–76599.

Questions for authors
  1. How do two simple baselines perform in RQ3: ranking steps by position (contributions in Figs. 1–3 concentrate in later steps) and leave-one-step-out resampling? Could the authors also provide an evaluation that does not rely on H(S)H(S), e.g., on Who&When? (see Weakness 2)
  2. Could the authors report a 2×2 ablation that crosses the objective (accuracy reward vs. Lcal\mathcal{L}_{\text{cal}}) with the gradient estimator (main-trajectory REINFORCE vs. the branch-based estimator) at matched LLM-call budgets, together with the performance of the pretrained orchestrator before training? On ChaosNLI, which supervision signal does each baseline receive, and which reference distribution does BOT-Orch use? (see Weakness 3)
  3. How were the values in Figs. 1–3 computed? Are the contributions averages over trajectories or pointwise quantities along the displayed run, and how were the per-step values H(S∣Ct)H(S\mid C_t) obtained, given that Algorithm 1 and App. D.4 only estimate AtℓA_t^\ell and the overall H(S)H(S)? In Fig. 1, the contributions sum to 0.763 against H(S∣C1)=0.879H(S\mid C_1)=0.879; is the remaining gap of 0.116 estimation error? If not, it would imply that the final answer is not determined by the last step, contrary to Assumptions 2–3. (see Weakness 4)
  4. What are the empirical values of the semantic-boundary probability βtℓ\beta_t^\ell, and does the Bernstein bound in Thm. 4 give a meaningful guarantee at B=32B=32? How does the paired estimator compare with nested Monte Carlo at the same number of LLM calls? It would also help to report the practical computational cost of attribution and orchestration training. (see Weakness 5)
  5. An interesting phenomenon is that most Qwen2.5-7B entries in Table 1 are exact small-denominator fractions (e.g., all five ChaosNLI HAUC values are multiples of 1/80; on MATH-500, 0.7333 = 11/15 and 0.5833 = 7/12), unlike the Olmo and GPT-OSS rows. Could the authors clarify how many questions and repetitions underlie these entries and how they were aggregated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.