- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper focuses on the problem of “tacit understanding”, the ability of a pair of agents, after repeated interaction, to predict the coordination patterns of one another in a way that generalises to unseen tasks and results in a reduced need for explicit communication.
They state that “tacit understanding” requires 3 properties: formation, specificity and generalisation. Next, they introduce a decentralised framework to enable tacit understanding for a pair of LLM agents and benchmark their method on the Overcooked, Hanabi and CoBlock environments.
Critical strengths
This paper raises an important question about understanding how deeply a pair of agents model each other after repeated interaction.
Also, the experiments indicate some promising results across three established coordination benchmarks.
Critical weaknesses
Poor writing; a lot of missing details
-
The writing exposition is poor, making it difficult to follow the method and experiments.
-
For example: Eq (3) introduces \rho, which is never used again in the main paper.
-
Many symbols are introduced without defining what they represent, e.g., m_train, etc., in Eq 9.
-
-
The paper misses key details, making it difficult to properly assess the contributions of the paper. This also makes reproducibility of the results challenging. Some examples are:
-
“neutralised history” — how is this computed?
-
“directional interventions” — how are they computed?
-
“cell” — how is this computed? Who defines the mapping from states to cells? Is this scalable?
-
Thresholds m_train, m_val, etc. — what are the exact values used?
-
Eq(10): how are the predictions converted to text? Can you provide an example of this?
-
Baselines and Ablations:
- The baselines miss key implementation details (e.g, how is memory retrieval implemented?), making it difficult to interpret the proposed contributions — the same applies to ablations.
-
Tacit understanding not well defined; not validated
-
The distinction between “tacit” understanding and basic partner modelling is not sufficiently motivated.
-
Why do formation, specificity and generalisation imply tacit understanding? Can you satisfy the GTU criteria but perform worse in coordination?
-
How does tacit understanding differ from regular partner modelling? Under a standard prediction model, we would still assume that a) zeroing out the history and b) replacing it with that of another partner would change its predictions? Additionally, we would expect it to generalise to test scenarios.
-
The proposed action prediction model is a fairly standard predictor of partner actions. Which aspects of the method make it better at tacit understanding?
-
More importantly, the paper does not present any results comparing tacit understanding of the method against the baselines. p_ID,d is defined on Line 370, but not reported in the main paper. Does the proposed method lead to greater tacit understanding (as defined by the authors) compared to the baselines?
Statistical validity; missing key ablations
-
Table 1 reports many mixed results, with the proposed method losing out to the baselines in some tasks. Also, the numbers often show small gains with overlapping error bars. How significant are the results?
-
What is the motivation for the specialised architecture in Eq (6)? How does it compare against a standard MLP that takes context as input and outputs a probability distribution over usage labels?
Questions for authors
Please see the Weaknesses.