acceptodds
Withdrawn from ICLR 2027

Learning to Adapt to Humans Without Human Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Any intelligent agent operating in an open-ended environment with multiple co-players must be able to adapt to behaviors it has not seen before. In safety-critical settings such as self-driving, this adaptation should happen continuously and rapidly, as a function of recent context. We ask whether such adaptation can be learned without any human data. To this end, we train a memory-based driving policy with reinforcement learning against a diverse population of fully synthetic co-players. The resulting policies adapt over repeated encounters with held-out human driving replays, improving their performance as they gain experience. We ablate the training procedure and demonstrate the importance of memory. Analysis of the attention map reveals an interpretable mechanism behind the agent's adaptation. Our results suggest a path toward agents that not only generalize across diverse partners, but also adapt beyond their training population to entirely new co-players through interaction.

Primary area:
reinforcement learning
Keywords:
Multi Agent Reinforcement Learning, Autonomous Driving, Meta Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sApsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors aim at understanding how much can memory based RL driving policies trained solely on synthetic data generalize to human policies. They thus train such policies on synthetic data and evaluate how much these adapts to human policies. They show that these memory based policies are bad at such tasks when memory is deactivated. Then they use attention map to try to interpret how agents adapt.

Critical strengths

The paper is overall clear in the main problem it is addressing: Can we train RL based driving policy only on synthetic data and easily generalize (in few shot) to human data?

I did not need to ask an LLM to understand what the paper does and how globally (which is rare nowadays), even if some details are not clearly exposed.

Critical weaknesses

I have trouble extracted the main takeaway message(s). What is the core takeaway message from this paper that I can apply to other AI domains? Or that give me a new perspective on this specific problem?


My main concern is that I don't feel like the hours spent on the paper brought me any clear takeaway message that I can reuse in discussions with other researchers or within my own research.

First, I was not convinced that ablating out of the training set human data is a necessity. Second, there is nothing compared to. Other baselines would help understand the advantage of one method over another one. You ablate memory from a memory-based RL policy to then deduce that it is a crucial part of it (am I really learning something here).

I am not saying that the research pursued has no value, simply that I was not able to extract it.


I am extremely doubtful about the conclusions on using attention maps to extract interpretations. This is not a core cause of my judgement of the overall paper, but I would like to bring the authors to think about the following: Attention maps do not allow you to draw conclusions on an interpretation of what is happening. It tells you that a specific part of the input is important in the decision process, but not why is this input part important. In the second reference below, importance maps show the ball as a relevant part of the input, however, the agent is clearly using the enemy's position (spuriously correlated) to infer the ball's position (if it does infer the ball position). Interpreting the reasoning behind the heatmap is still an ad-hoc interpretation that can be proven to be wrong (causality checks, at least, would be needed here).

Di Langosco, L. L., Koch, J., Sharkey, L. D., Pfau, J., & Krueger, D. (2022, June). Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning (pp. 12004-12019). PMLR.

Delfosse, Q., Sztwiertnia, S., Rothermel, M., Stammer, W., & Kersting, K. (2024). Interpretable concept bottlenecks to align reinforcement learning agents. Advances in Neural Information Processing Systems, 37, 66826-66855.


Clarity is not perfect though: "We ask whether such adaptation can be learned without any human data." -> but you use memory over replayed episodes of human data, so somehow it's learned with human data.


The scope is quite narrow though (only autonomous driving), there is no study of the distance between the synthetic policies and the human ones (which would allow me to bring a broader conclusion). This is a bit redundant with the takeaway message(s) not being easy to identify.

Questions for authors

Not saying non of this is present in the full text of the paper. But after reading it fully, I was confused and tried to use only the abstracts, contributions and figures to answer these, but could not:

  1. In Figure 1 left, why are there so many ego agents? (even if caption only talks about a single one).
  2. Policies are getting ego-centric states and goal, but how are they encoded? A frozen causal transformer is use? Not trained on your data? But pretrained? So only natural language policy is used? (that should go in your method, Figure2).
  3. Co plauyers are pre-scripted policies, or are they independant copies of the ego?
  4. Are you training a memory based system? And then demonstrate the importance of memory?
Reviewer_BA4xOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Population Play (APP) to train a driving policy for in-context adaptation to unseen co-players. A co-player population is first trained with PPO in PufferDrive on nuPlan scenes, conditioned on reward weights, entropy coefficient and discount factor. A transformer ego policy is then trained with PPO against the frozen population over repeated trials of the same scene, retaining memory across trials. Evaluation is performed on 539 held-out scenes, with all other vehicles replaying recorded human trajectories. Return on adaptable scenes improves across trials without parameter updates. Resetting memory between trials removes most of this improvement at co-player entropy bounds e = 0.10 and e = 0.20. Attention maps show which events from earlier trials receive attention.

Critical strengths
  1. Memory-reset control. Clearing memory between trials removes most of the improvement at co-player entropy bounds e = 0.10 and e = 0.20. The same scenes are used with and without memory, controlling for apparent gains caused by selecting scenes with poor first-trial performance. Without scene selection, improvement remains at e = 0.05, 0.10 and 0.20. Evaluation covers 539 held-out scenes, 20 rollouts per scene and three training seeds.

  2. Memory without cross-trial credit. Earlier observations can be used without previous rewards, previous actions or an explicit trial-boundary signal being provided as input. Rewards from later trials are not credited to actions in earlier trials. The policy can therefore learn to use earlier experience without early actions being rewarded for gathering information that helps in later trials.

Critical weaknesses
  1. The main claim is not supported by the evaluation. The paper claims that the ego adapts to human drivers, but in every trial the other vehicles replay the same recorded trajectories from the same initial state. Improvement across trials can therefore come from memorizing the replay, that is, when and where each vehicle appears, without inferring anything about the drivers. The authors acknowledge in Section 5 that these two explanations cannot be separated. Without an evaluation in which the scene or the trajectories change across trials, the results support memory-based improvement on a repeated scene but do not establish adaptation to human drivers.

  2. The contribution of the co-player population is not tested. The first stage of APP builds a diverse co-player population, but no comparison is reported with an ego trained against a single fixed co-player or against a population without reward and discount variation. The only population setting varied is the entropy bound, and each bound uses a separately trained co-player network, so the entropy trend does not isolate diversity either. This trend also rests on three ego-training seeds, with ΔR ranging from roughly 0.2 to 1.8 at e = 0.10, and the memory-reset control is reported only for e = 0.10 and e = 0.20, not for the e = 0.05 configuration used in Figure 1 or for e = 0.50. Without these ablations, it is unclear whether population diversity is needed for the reported adaptation.

  3. The interpretability claim is supported by a single example. The paper claims that cross-trial attention reveals which experiences drive adaptation, but the evidence is the attention map of one episode. In the second and third trials the ego attends to frames of its earlier collisions, and in the third it approaches slowly and reaches its goal. No attention results across scenes are reported. As Section 5 acknowledges, high attention to a past collision does not show that this memory caused the change in driving. The claim is therefore not established; an intervention on the attended memories would be needed to test it.

Questions for authors
  1. Can the scene and the behavior of the other vehicles be varied independently across trials, for example co-players with the same conditioning placed in different scenes, comparing retained and reset memory?
  2. What does the memory-reset control give for e = 0.05 and e = 0.50, and what are the per-trial success and collision rates?
Reviewer_Xrj6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-stage framework for learning in-context adaptation to unseen co-players without human behavioral demonstrations. The method trains a diverse population of synthetic co-players and then trains a memory-based ego policy through repeated trials. Experiments on held-out human driving replays show improved performance across trials without parameter updates.

Critical strengths
  • The paper addresses the important research challenge of enabling adaptive behavior in multi-agent environments.
  • The paper provides a detailed analysis of the proposed method's adaptation behavior across repeated trials.
Critical weaknesses
  • Although the paper discusses three relevant research directions—Generalizing to Unseen Partners, Learning to Adapt In Context, and Diverse Agents in Driving—the discussion remains largely at a high conceptual level. While the authors highlight general differences in research objectives and evaluation settings, they do not sufficiently clarify the specific methodological distinctions between APP and the most closely related prior approaches. Consequently, it is difficult to identify the precise novelty of the proposed framework relative to existing work.
  • Although the paper discusses three relevant research directions (Generalizing to Unseen Partners, Learning to Adapt In Context, and Diverse Agents in Driving), the discussion remains largely at a high conceptual level. While the authors highlight general differences in research objectives and evaluation settings, they do not sufficiently clarify the specific methodological distinctions between APP and the most closely related prior approaches. Consequently, it is difficult to identify the precise novelty of the proposed framework relative to existing work.
  • The paper discusses three relevant lines of work (Generalizing to Unseen Partners, Learning to Adapt In Context, and Diverse Agents in Driving), but the discussion remains somewhat high-level. While the authors point out differences in research goals and evaluation settings, they do not sufficiently clarify the specific methodological distinctions between APP and the most closely related prior approaches. It is difficult to identify the precise novelty of the proposed framework relative to existing work.
  • Related to the previous concern, the paper lacks meaningful comparisons with relevant existing methods. The experiments primarily focus on analyzing the proposed method's performance across repeated trials and conducting ablation studies on its components, rather than benchmarking APP against established approaches.
  • The paper also lacks direct comparisons with relevant existing methods. The experiments mainly focus on evaluating APP over repeated trials and examining the contributions of individual components through ablation studies.
  • The third claimed contribution, Interpretable Adaptation, seems to be more of a qualitative analysis of attention patterns than a methodological contribution. Moreover, the proposed method does not appear to include any specific design choices aimed at improving interpretability.
  • While the paper emphasizes learning to adapt "without any human data", the proposed method focuses more on general multi-agent adaptation, with limited evidence of how it specifically adapts to human behavior.
Questions for authors

see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.