- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper describes a framework that uses LLMs to assist in automated synthesis of control policies for multi-agent systems from user prompts translated to formal temporal logic specifications. In particular, LLMs are used to translate user intent described in natural language to temporal logic (specifically GR1 fragment). Then to handle multi-agent aspect, assume-guarantee framework is used, where contracts for individual agents are generated using LLMs. A variety of checks are implemented using symbolic tools from formal methods literature. Policies are finally synthesized for individual agents from the decomposed GR1 specifications using tools for reactive synthesis. The framework is implemented and evaluated on some simple case studies.
Critical strengths
Overall methodology is sound. Basically a neurosymbolic architecture where LLMs help in tasks that are challenging for humans (writing formal specs, decomposing spec into assume guarantee contracts) and classical symbolic tools are used to assure "correct by construction" guarantees (for instance, extracting finite-state control policy from GR1 spec for an individual agent).
Critical weaknesses
The framework puts together multiple pieces that have been studied separately and well understood. I did not find any technical novelty in the method itself. Such a paper should IMO be evaluated on the basis of how strong the experimental evaluation is. All three case studies seem relatively simple. The agents are symmetric, and the main coordination requirement is some form of mutual exclusion. This is exactly the case where I would know how to write a contract for an agent myself. The fact that it then scales in number of agents is unsurprising. What would be really cool if the authors demonstrate that the LLM is able to generate contracts for "complex" tasks where there is some subtlety in finding the right assumptions and guarantees.
Questions for authors
- Is there an example of an LLM generated contract that you found surprising/interesting.
- There are no baselines. Robot learning is developing rapidly with RL / learning from demonstrations / VLMs / VLAs ... Which of these are relevant for the high-level problem (say the Charging Problem), and how do they compare with your method (or where do they actually fail in practice)
Thank you for the rebuttal. It does address some of the concerns raised in my review, and I am happy to raise my score to 3.