acceptodds
Withdrawn from ICLR 2027

Predictive Coordination of LLM Agents Via Semantic Failure Attribution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Teams of LLM agents often perform worse than a single agent, and recent benchmarks trace the losses to broken expectations and unkept commitments rather than to weak individuals. A coordinator that wants to avoid such failures must learn whom to trust. We prove it cannot learn this from outcomes: in a sequential-stage model of collaboration, a weak agent with a compatible partner and a strong agent with an incompatible partner induce identical success rates for every plan, so any estimator of individual competence suffers an error bounded away from zero, for any number of episodes and under any adaptive policy. Semantic attribution breaks the tie. If failures are labeled by cause through any invertible confusion channel , the parameters become identifiable and estimation is -consistent at a price , so the language model need only be informative, not accurate. On this result we build LPC (LLM-native Predictive Coordination), which gives an LLM coordinator belief filtering over competence and pairwise compatibility, predictive plan scoring, posterior-sampling exploration, and a computable re-coordination threshold as tools under typed contracts. We then measure for four LLMs on 96 multi-agent coding failures whose cause is fixed by construction. All are informative, but all share one bias: they identify a faulty patch almost perfectly and miss the interaction failure, blaming the implementer for what is in fact a broken agreement, which is precisely the failure outcome data also cannot see. Driven by these measured channels, LPC closes 65% to 70% of the reputation-to-oracle gap, beating plan-level Thompson sampling by a margin that grows with population size.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, multi-agent systems, agent coordination, predictive coordination, semantic attribution, Bayesian belief modeling, compatibility modeling, Thompson sampling, POMDPs

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_YjnzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the problem of multi-agent coordination among LLM agents to solve coding tasks. They consider a specific setting where problems can be decomposed into an (implement, test, integrate) pipeline, with agents being allocated roles by a central coordinator.

They propose the LLM-Native Predictive Coordination (LPC) framework as a means to solve the above problem. The framework consists of a) a semantic failure attribution module, b) partner-interaction ability modelling, c) planning, and d) fallbacks in case of alarms.

The authors also provide theoretical results that motivate the LPC framework in the proposed setting, importantly showing that outcome-only feedback cannot distinguish individual competence from pairwise compatibility.

Via experimentation on synthetically generated problems, the authors validate their framework against various baselines, including those that learn from outcome-only feedback.

Critical strengths

The paper is clearly written and easy to follow. The theoretical results are interesting and motivate the different components of LPC. The attribution study with LLMs also provides evidence for using LLMs as judges in multi-agent coding applications.

Critical weaknesses

Problem Setting is not validated

  1. The paper presents a restricted class of coordination problems where each task is split into an (implement, test, integrate) pipeline. — How reflective is this pipeline of coding tasks that are being tackled in the real world by teams of LLM agents?

  2. If A and B agree on an API specification but A does not follow it faithfully in the implementation, why should that be attributed to compatibility? I argue that it is an issue with A’s implementation capabilities.

  3. The current generation of LLMs is strong at implementation given accurate task specs [1]; hence, the primary challenge is in generating the specs. I argue that if two agents A and B had to discuss and agree on a spec, this would highlight compatibility failures. However, this is not evaluated in this framework and hence doesn’t capture the main coordination challenge encountered by agents in the real world.

  4. In coding especially, we already have an estimate of each agent’s intrinsic ability (public leaderboards); hence, do we need semantic failure attribution? In this case, compatibility can be inferred purely from outcome-based rewards.

Method is not scalable; not experimentally validated; evaluations are synthetic

  1. Confusion matrix M

    1. How would one estimate this on complex coding tasks? Do humans have to evaluate each output of LLM-extract?

    2. The paper reports no results on human effort or the accuracy of estimating M on real coding tasks.

  2. The modelling parameters a, b, k depend on the context x; how would this scale in the real world where the coding problems are open-ended?

    1. Again, the paper would benefit from results supporting this.
  3. It is unclear how LLM-Rollout can be implemented accurately. Does a single LLM generate the exact rollout of two LLM agents A and B given estimated parameters?

    1. Again, given a complex coding task and two LLMs that contain billions of parameters, how does one “generate” the entire interaction faithfully?

    2. Can the authors run an experiment showcasing that this is feasible?

  4. The criticisms in point 1 hold for \pi and \tau^* as well. (I recommend avoiding reuse of the symbol \pi.)

  5. Although the paper focuses on coordination of LLM agents, none of the main results (Table 1, Figure 4) evaluate LLM agents and test solely on synthetic benchmarks.

    1. Can you take an established coding dataset/benchmark, validate the proposed LPC method and verify whether the results observed on the synthetic tasks hold?
  6. The authors also introduce components in their method that are not used in the experiments (ex: CVaR).

Questions for authors

Please refer to the Weaknesses.

Reviewer_3Ht5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes LPC, a framework for coordinating LLM agents by modeling individual competence and pairwise compatibility. The authors establish an identifiability result showing that outcome-only observations cannot distinguish these factors, while noisy semantic failure attribution can restore identifiability under certain conditions. LPC integrates LLM-based failure attribution with Bayesian belief updates, predictive planning, and adaptive re-coordination.

The approach is evaluated through failure attribution on 96 constructed coding failures and coordination experiments in a simulated environment. The results show improvements over several baselines, but the complete framework is not evaluated end-to-end on real multi-agent tasks.

Critical strengths
  1. Interesting theoretical perspective. The paper provides a formal analysis of why outcome-only observations cannot distinguish individual competence from pairwise compatibility. This is a useful perspective on failure attribution in multi-agent systems.

  2. Clear framework and controlled evaluation. The combination of semantic attribution and probabilistic coordination is conceptually coherent, and the controlled experiments help isolate the contributions of individual components.

Critical weaknesses
  1. Unclear practical significance of compatibility modeling.

I am not convinced that learning pairwise compatibility is a sufficiently meaningful or actionable objective in realistic multi-agent systems.

In closed environments, where agents are known and controllable, improving individual agents or their coordination protocols may be more effective than continuously learning and avoiding incompatible pairings. In open environments, compatibility may be difficult to estimate reliably due to limited interactions, while the coordinator may have little freedom to select or replace agents.

Furthermore, compatibility is inherently task-dependent. Although the paper introduces context-conditioned parameters, its coarse contextual modeling and simulated evaluation do not establish whether compatibility estimates generalize across heterogeneous tasks. These issues challenge the practical motivation of the proposed framework.

  1. Questionable scalability and feasibility of compatibility estimation.

Estimating pairwise compatibility already requires learning a quadratic number of potential relationships as the agent population grows. For larger teams, higher-order interactions may introduce additional complexity that pairwise modeling cannot capture.

More importantly, reliable compatibility estimation depends on informative failure attribution. Although the theory allows noisy attribution, it assumes a known, invertible attribution channel. In realistic systems, failures may have multiple interacting causes, and obtaining sufficiently accurate and stable attribution labels could be difficult and expensive.

The paper's attribution experiments are limited to small, constructed coding failures, while the coordination gains are demonstrated primarily through simulation. Consequently, the work does not convincingly establish that the proposed approach remains feasible or beneficial at realistic scales.

Questions for authors
  1. In what realistic deployment scenarios is learning pairwise compatibility preferable to improving individual agents or their coordination protocols? How does the framework remain useful when the coordinator cannot freely select or replace agents?

  2. How does the framework scale as the number of agents, team size, and task diversity increase? In particular, what evidence supports the feasibility of obtaining sufficiently informative failure attribution and reliable compatibility estimates in real-world environments?

Reviewer_d3HROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LPC, a multi-agent coordination framework that combines LLM-based failure attribution with Bayesian inference to guide coordination decisions by estimating individual agent competence and pairwise compatibility. The authors theoretically analyze the non-identifiability of agent competence from task outcomes alone and evaluate the method through constructed coding failure cases and simulation experiments.

Critical strengths

S1. The paper provides a theoretical analysis of the non-identifiability of individual competence and pairwise compatibility and proposes using informative but noisy failure attribution to address this issue. S2. The paper includes failure attribution experiments with real LLMs, along with component ablations and robustness analyses in a simulated environment.

Critical weaknesses

W1. Lack of validation with real LLM agent interactions. The main performance experiments rely entirely on a simulator that adopts the same factorized competence-compatibility model assumed by LPC. These experiments demonstrate the method's effectiveness under its own modeling assumptions but do not establish whether those assumptions hold in real LLM agent interactions. Moreover, core components such as LLM-based plan generation, predictive rollouts, and dynamic re-coordination have not been evaluated end-to-end. W2. Insufficient comparisons with related methods. Although the paper discusses existing partner modeling and LLM coordination methods, the experiments primarily compare LPC against simplified reputation-based approaches and generic Thompson sampling. The lack of direct comparisons with recent related work makes it difficult to assess LPC's advantages over existing methods.

Questions for authors

Q1. Can the authors provide at least a small-scale experiment involving real LLM agent interactions to validate the proposed competence-compatibility model and demonstrate whether the complete LPC framework improves actual multi-agent coordination performance? Q2. Why does the paper not compare LPC with closely related methods? How do the current experiments support LPC's advantages over these approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.