- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the challenge of imitation-learning an individual agent's policy from the trajectories of multiple agents. There is an asymmetry between trajectory data of a group of agents and action-label data in terms of how much of each can be collected. Therefore, a learning pipeline that learns a latent representation from trajectory data in advance and then connects this latent action representation to action labels has the potential to make learning more efficient and to appropriately separate the policies of individual agents. The paper extends an existing observation-only latent-action learning pipeline to the multi-agent setting, in which the influences of multiple agents are mixed within a joint transition, and tackles the problem of extracting a target-specific latent action. The contribution of this paper lies in clarifying this problem setting in multi-agent environments, and in proposing Controlled Predictive Comparisons (PC) for latent-action learning in order to make such a pipeline work.
The proposed method (PC) learns a latent action that characterizes the behavior of a specific target agent through controlled predictive comparisons. For each transition (query), a learned dynamics model makes two predictions from the same history and the same partner codes, changing only the target code: one prediction uses the query’s target code, and the other replaces it with the target code of a reference transition (donor) retrieved from the training data by nearest observation history. The difference between the two predictions isolates the prediction change associated with the target agent. Repeating this comparison across partner settings captures how the target’s effect depends on individual partners and their combinations, and these structured differences are compressed into a latent action. This makes it possible to obtain a code that represents an individual agent’s behavior from multi-agent observations.
Critical strengths
-
Adapting to the multi-agent setting the idea of comparing predictions in which only the target agent’s latent code is changed while the history and partner codes are held fixed, and extending this comparison across partner subsets so that the representation retains how the target’s effect depends on individual partners and their combinations, is novel.
-
Broad experiments and comparisons. In addition to multiple baselines such as Base, duplicated Base, LAPO/LAOM, BC, and IDM, the paper examines the effects of the features and of the encoder structure. The scope of evaluation is also wide, including numerical observations, RGB observations, and an eight-agent setting.
Critical weaknesses
-
The benefits of the predictive comparisons are not statistically supported relative to the complexity of the method. The proposed method consists of many components, including the inverse encoder, donor retrieval, predictive contrasts, a shared predictor, encoders, history prediction, and an action decoder. However, as the paper itself states, the effects of coordinates and matching vary across seeds and budgets, and the corrected tests do not establish a uniform structural advantage. In addition, duplicated Base remains a strong competitive baseline in MaMuJoCo, adapted Base outperforms both adapted PC variants in the eight-agent setting, and BC and IDM outperform the Frozen latent policies on RGB Ant at the evaluated label budget.
-
There are concerns about the validity of donor-code recombination. There is no guarantee that target code changing from query to donor corresponds to a realizable observation sequence. Therefore, the conditions under which the prediction difference produced by recombination can be interpreted as the effect of a meaningful replacement of the target agent are unclear. Although the existing diagnostics partially address this issue, their relationship to prediction reliability in the main downstream tasks is not sufficiently shown.
Questions for authors
Q1. Contribution of the PC comparison operation itself. Across the extensive ablation studies, many of the comparisons involving Predictive Comparisons (PC) do not reach statistical significance. Which experimental results provide clear evidence that the performance gains can be attributed specifically to the PC operation, rather than to other components of the proposed architecture?
Q2. Validity of donor-code recombination. As the paper notes, nearest-history retrieval alone does not guarantee that donor codes keep their meaning when transported or that the predictor handles recombined inputs. On the main downstream tasks, can the authors evaluate whether the recombined inputs correspond to realizable transitions, and show how this affects control performance?