acceptodds
Under review as a conference paper at ICLR 2027

Learning to Coordinate Through Strategic Game Slices

Abstract

Coordination under private information requires agents to infer partners’ preferences and select actions using those inferences. We study these capabilities in a finite negotiation game with private preferences and mixed incentives. A reference-policy oracle supplies posterior judgments and counterfactual action values, enabling matched diagnostic interventions. For Qwen3-4B-Instruct-2507, these probes reveal errors in both preference inference and action selection: supplying an oracle judgment does not reliably eliminate action errors. We use the same oracle to construct game slices and train a shared model on direct action selection, preference inference, and planning. With approximately 6M generated response tokens, joint slice training retains 47 of 72 meetings in a 24-game, four-agent sequential calendar suite adapted from CalBench, compared with 37 for both the unchanged model and complete-game self-play. These results provide initial evidence that targeted supervision from controlled strategic games can improve coordination in an environment with different interaction mechanics.

Primary area:
reinforcement learning
Keywords:
large language model agents, multi-agent coordination, partner modeling, belief-conditioned planning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UWkfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors focus specifically on coordination scenarios, and find that on Qwen3-4B-Instruct-2507 (individual LLM) has errors through both preference inference and action selection/planning. Giving oracle judgement changes the model’s actions, but also doesn’t make planning necessarily better. The authors turn oracle-labeled game slices into training data for training with GRPO. The authors also extend on a multi-party negotiation game, CalBench, to include private preferences and investigation actions, evaluations show that supervised training from controlled game slices can transfer to games with other interaction mechanics.

Critical strengths

The authors break down a general problem (agents coordinate poorly) into two isolated clear branches: preference inference and action selection and extend a commitment-based negotiation game to involve private preferences and information gathering. The design to be able to isolate the two problems is nice through comparing planning with the judgement (based on preference judgement from history and available information) vs with an oracle judgement. The authors evaluate game slice games with various numbers of players and also extend to CalBench, with different game dynamics.

Critical weaknesses

The paper results rely on one single base model. It would be interesting for the first portion of the results (the evaluation of preference inference and action selection) on various models (both open sourced and closed sourced and of various sizes), and look at findings to see if perhaps there are interesting patterns or if models are generally better at one than the other (and perhaps more recent models are more capable at planning and inference than older models). The authors should also perform training on a few other models as well. Then this might also help with slightly alleviating burden on same vs different model negotiation (if the findings hold amongst multiple models after training, different models can be substituted in perhaps with not much effects to the results perhaps; or this could make it easier to examine more effects on multi model negotiation).

After slice training with GRPO, the trained model retains more meetings on CalBench, though it would also be interesting to see if the trained model also just performs better on the first experiments described comparison to the oracle evaluation of preference inference and planning, to see if there are improvements specifically to both or one that helps to better explain the improvement in performance on CalBench compared to other models. Also since the authors have isolated between the preference inference and planning, is there any difference when training with these in isolation? (if training on preference inference would increase performance of preference inference evaluations?)

There are some other relevant work that might be good to cite: works on similar theoretical formulations of multi-agent/agent+human collaboration: [1] Hadfield-Menell, D., Russell, S. J., Abbeel, P., & Dragan, A. (2016). Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29.

[2] Büning, T. K., George, A. M., & Dimitrakakis, C. (2022, June). Interactive inverse reinforcement learning for cooperative games. In International Conference on Machine Learning (pp. 2393-2413). PMLR.

[3] Ma, R., Qu, J., Bobu, A., & Hadfield-Menell, D. (2025). Flexible Agent Alignment with Goal Inference from Open-Ended Dialog. arXiv preprint arXiv:2508.15119.

[4] Dai, A., He, Z., Zhang, Z., Pentland, A., & Pei, J. (2026). Interactive Task Alignment as a POMDP. arXiv preprint arXiv:2607.16412.

[5] Zhi-Xuan, T., Ying, L., Mansinghka, V., & Tenenbaum, J. B. (2024). Pragmatic instruction following and goal assistance via cooperative language-guided inverse planning. arXiv preprint arXiv:2402.17930.

Questions for authors

Interested to know more about the performances of other models as well for the first evaluation on preference inference and planning (are there any trends that have to do with scale or family model or capabilities with the more recent models). How well do the training results generalize to training for other models? How well do trained models perform on the first evaluation on preference inference and planning? Is there a way to isolate training for preference inference data vs planning data, and would that increase the performance of one or both accordingly? How well would this generalize to less constrained/less formated/templated multi-agent games?

Reviewer_uttcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the following two abilities in multi-agent coordination under imperfect information and potentially contradictory incentives, (1) inferring and reasoning about the preferences of other agents, (2) acting according to the beliefs established in the first part. The authors construct a finite multi-party negotiation game with private preferences and irreversible actions, where agents interact through a pre-determined scheme. The authors also incorporate a reference-policy oracle which generates Bayesian posterior judgments and counterfactual action values, effectively distinguishing errors in inference of preferences from errors in planning and actions, and further create game slices with the oracle for training. Evaluation-wise, the setting successfully reveals failures in both preference inference and action selection. Training-wise, a shared Qwen3-4B-Instruct-2507 model is trained on all these capabilities with around 6 million generated response tokens and then evaluated on both the negotiation environment and CalBench, where the evaluation results suggest meaningful improvement over the raw baseline and the complete-game self-play baseline.

Critical strengths

(1) The research problem is clear and meaningful, and the authors provide a neat formulation and distinguishment protocol for the different capabilities. Multi-agent coordination is a popular and important area of study, and the fine-grained separation of inference and planning is a meaningful methodological contribution which could help in interpreting failures. Through thorough ablation studies, the paper does a good job in separating different capabilities with controlled environment and parameters, making reported results credible and informative.

(2) The authors also demonstrate valuable transfer from the given settings to a different coordination scenario. Through the evaluation on CalBench, the authors show that the learnt capabilities through game slices in the their negotiation environment could improve performance on other tasks, demonstrating the transfer value of the created finite multi-party negotiation environment and the value of the game slices created in that synthetic environment.

Critical weaknesses

(1) The experiments do not adequately separate the influence from game slicing and oracle supervision. Compared between the game slicing version and the complete-game self-play, there are two distinctions, where the first is the origin of the signal (i.e. oracle-scored objectives vs. full game results), and the second is the training data distribution (i.e. game slices vs. full games). From the evaluation, we only know that oracle-scored objectives on game slices outperform traditional full-game self-play. It would be helpful and interesting to distinguish the effects from game slicing and the effects from more dense oracle-scored rewards.

(2) The claims on the transferability is not very well supported. The scale of all experiments and the transfer evaluation on CalBench is relatively small. All the results are based on a single 4B model and the evaluation is established on a small benchmark. The results themselves are well supported, but the scalability and transferability of the proposed methods remain uncertain, which may limit the significance and impact of the entire work.

Questions for authors

Please refer to the weaknesses. I'm willing to increase the score to 3 if Q1 and the first part of Q2 could be addressed.

(1) Could you please design some additional ablations to distinguish the effects from game slicing and the effects from more dense oracle-scored rewards? (See Weakness 1)

(2) Could you please test the transferability on some larger-scale benchmarks and compare with state-of-the-art results on a similar scale model? If possible (though I understand it may not be realistic), could you test the proposed methods on larger models (ideally from another model family, e.g. Nemotron)?

Reviewer_rW7BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed an LLM reinforcement learning method, aiming at leveraging the ability / performance of an LLM in multi-agent incomplete-information games (negotiation game of 2, 3 or 4 players). The paper also show evidences of the transferrability of the model tuned by this method to an unseen envrionment.

In detail, the paper make use of an oracle in generating a training set of "game slices" which is a set of QA tasks and apply GRPO on this set of tasks. The oracle is a policy obtained from an iterative solver, ensuring an equilibirum, on a finite decisoin tree model of the game with preferece updated by Bayesian inference. The authors first make a diagnosis on the LLM's failures, which turns out to be on preference inference and on decision making after knowing correct preference. With this insights, they built a training set. They first cut the task into "game slices" (several key-turns in a game played by the oracle), the split them into single decisions ("snapshots"). The training set is then made of three types of such questions: 1. a full-decision task at a turn, 2. a preference-inference task given game status, 3. a decision generation task given game status with oracle-generated preference. Then GRPO is applied to the model on the training set, on three task settings: type 1, type 1+3, and type 1+2+3.

In experiment, the paper compared the trained models on three types and a self-played RL model. Results show that the method outperforms self-played RL on all testing tasks (oracles not given in the test experiments. Tasks are the negotiation game of 2, 3, 4 players and an unseen sequential calender coordination task of different interaction types). In detail, each of the three trained models may behave the best in different types of games.

Critical strengths
  1. The paper defined the problem in a clear way and gave a concrete method to solve the problem.

  2. The method-construction motivation is clear and evident. The method can be directly inferred from the diagnosis which shows .

  3. Experiment shows the trained model can transfer to other multi-agent environments.

Critical weaknesses
  1. From the result, type 1+2+3 is not always the best in performance, I am not sure whether such a training could leverage the model's ability on both preference inference and decision making with correctly inferred preference (I see no such tests). It is unclear what really contributes such an increase of performance.

  2. The reference self-played RL is on full-game trajectory, it is not excluded that the performance increase is due to inference-decision decomposition or simply condensing the reward on important game slices.

  3. In tests, I only read the self-played games. Are there other types of results, such as "gameslice-AP vs gameslice-joint", "gameslice-A vs Self-play", "gameslice-Joint vs oracle"? Afterall, the volume of test games is relatively small.

  4. Only one base (Qwen3-4B-instruct-2507) model is trained using the method (and also on self-play). Qwen3-4B is relatively small, but more importantly, I think the authors should at least try various models on various scales (could be all below 4B), if computing resource is really a limit, in order to ensure the results are not influenced by any special properties of Qwen.

  5. The method requires an oracle, meaning the environment may be simple enough to admit one. This may limit the application. Or the authors may show how it performs with an estimated non-optimal oracle.

  6. Could there be a comparison with purely prompt engineering method, such as given a well-written game guidance? This may show how necessary the RL method should be applied in this environment.

Questions for authors

I wrote my major concerns in the Weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.