- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on the problem of multi-agent coordination among LLM agents to solve coding tasks. They consider a specific setting where problems can be decomposed into an (implement, test, integrate) pipeline, with agents being allocated roles by a central coordinator.
They propose the LLM-Native Predictive Coordination (LPC) framework as a means to solve the above problem. The framework consists of a) a semantic failure attribution module, b) partner-interaction ability modelling, c) planning, and d) fallbacks in case of alarms.
The authors also provide theoretical results that motivate the LPC framework in the proposed setting, importantly showing that outcome-only feedback cannot distinguish individual competence from pairwise compatibility.
Via experimentation on synthetically generated problems, the authors validate their framework against various baselines, including those that learn from outcome-only feedback.
Critical strengths
The paper is clearly written and easy to follow. The theoretical results are interesting and motivate the different components of LPC. The attribution study with LLMs also provides evidence for using LLMs as judges in multi-agent coding applications.
Critical weaknesses
Problem Setting is not validated
-
The paper presents a restricted class of coordination problems where each task is split into an (implement, test, integrate) pipeline. — How reflective is this pipeline of coding tasks that are being tackled in the real world by teams of LLM agents?
-
If A and B agree on an API specification but A does not follow it faithfully in the implementation, why should that be attributed to compatibility? I argue that it is an issue with A’s implementation capabilities.
-
The current generation of LLMs is strong at implementation given accurate task specs [1]; hence, the primary challenge is in generating the specs. I argue that if two agents A and B had to discuss and agree on a spec, this would highlight compatibility failures. However, this is not evaluated in this framework and hence doesn’t capture the main coordination challenge encountered by agents in the real world.
-
In coding especially, we already have an estimate of each agent’s intrinsic ability (public leaderboards); hence, do we need semantic failure attribution? In this case, compatibility can be inferred purely from outcome-based rewards.
Method is not scalable; not experimentally validated; evaluations are synthetic
-
Confusion matrix M
-
How would one estimate this on complex coding tasks? Do humans have to evaluate each output of LLM-extract?
-
The paper reports no results on human effort or the accuracy of estimating M on real coding tasks.
-
-
The modelling parameters a, b, k depend on the context x; how would this scale in the real world where the coding problems are open-ended?
- Again, the paper would benefit from results supporting this.
-
It is unclear how LLM-Rollout can be implemented accurately. Does a single LLM generate the exact rollout of two LLM agents A and B given estimated parameters?
-
Again, given a complex coding task and two LLMs that contain billions of parameters, how does one “generate” the entire interaction faithfully?
-
Can the authors run an experiment showcasing that this is feasible?
-
-
The criticisms in point 1 hold for \pi and \tau^* as well. (I recommend avoiding reuse of the symbol \pi.)
-
Although the paper focuses on coordination of LLM agents, none of the main results (Table 1, Figure 4) evaluate LLM agents and test solely on synthetic benchmarks.
- Can you take an established coding dataset/benchmark, validate the proposed LPC method and verify whether the results observed on the synthetic tasks hold?
-
The authors also introduce components in their method that are not used in the experiments (ex: CVaR).
Questions for authors
Please refer to the Weaknesses.