acceptodds
Under review as a conference paper at ICLR 2027

From Task-Oriented Agents to Personalized Teammates: Understanding Continual Adaptation in Cooperative Games

Abstract

In cooperative games, task performance alone does not determine whether an AI teammate is a good fit for a player. Motivated by the data available to game developers, we study personalization from abundant gameplay logs and sparse, noisy post-match endorsements, without requiring the target player to participate in training. Gameplay logs describe how the player acts, whereas endorsements provide imperfect evidence of how they want a teammate to cooperate. This creates a continual adaptation challenge: preference learning and subsequent task optimization pursue different objectives and can interfere with one another. Through controlled experiments in Overcooked, we isolate the effects of preference initialization, training partners, and reward design. Our analysis shows that, in multi-agent on-policy learning, task optimization can lead to different behavioral adaptations under different training-partner distributions. The same task objective can improve a learned division of labor under one interaction condition, yet replace it under another. Guided by this analysis, we initialize the AI from endorsements, learn an interactive player proxy from gameplay logs, and continue task-reward RL with that proxy. This simple procedure improves both task performance and preference satisfaction in our benchmark without requiring preference reward.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Human–AI Cooperation, Preference Learning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_EHQrOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies personalized AI teammates in cooperative games using historical gameplay logs and sparse, potentially noisy endorsements, without requiring online interaction with the target player during training. The authors distinguish between how a player acts and how they prefer their teammate to cooperate, which is insightful. Using controlled experiments in Overcooked, the authors studied how preference initialization, training partners, and reward design influence adaptation.

Critical strengths
  1. The distinction between how a user behaves and how they want a teammate to behave is important. Using gameplay logs to model the former and endorsements to learn the latter provides a intuitive approach to personalization.

  2. The experiments separately examine preference initialization, training partners, and reward design. The finding that the training partner influences whether task-oriented RL preserves previously learned preferences is informative.

Critical weaknesses
  1. In Distant Tomato Style C, task performance improves substantially while preference satisfaction deteriorates. Although this limitation is acknowledged, a more systematic analysis of task & preference alignment is needed to establish when adaptation reinforces the preferred behavior and when it replaces it.

  2. The target player is jointly trained with a teammate using a specific preference reward. Proxy-RL may therefore recover their learned cooperation rather than preserve user preferences. Testing different preferences with the same target player would help clarify this

Questions for authors
  1. Does Proxy-RL remain effective when the target-user policy is trained independently of the preferred teammate style? Can you evaluate multiple teammate preferences while holding the user policy fixed?

  2. What determines whether task-only RL preserves or overrides the preference learned during SFT? Can you systematically vary the conflict between task reward and preference reward?

3 comments

We thank the reviewer for recognizing the importance of distinguishing player behavior from preferred teammate behavior, as well as the insights from our analysis of training partners and preference retention. To address the concerns raised, we conducted an additional experiment with an independently trained player and further analyzed the behavioral changes underlying the task–preference trade-offs.

Response to W2 and Q1: Multiple preferences with an independently trained player

We conducted the fixed-player comparison you suggested. In Many Orders, we use one task-trained FCP policy as the target player, rather than selecting a player from a preference-specific co-training pair. Styles A–C share this target player and the same 72-trajectory gameplay archive.

For each training seed, we reuse the same learned player proxy across styles. We also compare Proxy-RL with Proxy-RL (w/o SFT), which starts directly from the FCP base without preference fine-tuning. Both RL conditions use the same proxy, task-reward protocol, 10M-step budget, and checkpoint-selection rule.

The table reports mean ± standard deviation across three training seeds, rounded to three significant figures. Offline trajectories provide reference means. The same three policies trained without preference SFT are evaluated under all three preference specifications.

MethodA: Pref.A: Env.B: Pref.B: Env.C: Pref.C: Env.
Offline Trajectories510233366125167165
Preference SFT278 ± 95.9134 ± 52.8175 ± 56.348.0 ± 20.018.7 ± 7.3618.7 ± 7.36
Proxy-RL (w/o SFT)742 ± 21.9325 ± 8.83253 ± 7.78325 ± 8.83−18.3 ± 13.0325 ± 8.83
Proxy-RL710 ± 136328 ± 37.5409 ± 147201 ± 8.60212 ± 53.8175 ± 38.3

Proxy-RL improves both returns over its SFT initialization and exceeds the offline-reference means across all three styles. In Styles B and C, it achieves the highest mean preference returns among the compared methods. Its preference advantage over the no-SFT condition occurs despite lower task returns.

These results directly address the co-training concern: preference-dependent adaptation also occurs when the player has not been co-trained for the corresponding style. The preference supervision can influence subsequent cooperation, rather than simply reproducing a single cooperation pattern associated with that player.

Response to W1 and Q2: Task progress and preference retention

We extended the Distant Tomato analysis to examine the behavioral changes behind the task and preference returns. For each style, we compare its actual SFT initialization with subsequent task-reward RL using the learned player proxy. Each style retains its original target player and preference definition. The results below use the n=4 feedback condition.

The table reports mean ± standard deviation across three training seeds, rounded to three significant figures.

MethodA: Pref.A: Env.B: Pref.B: Env.C: Pref.C: Env.
Preference SFT492 ± 29875.3 ± 53.5628 ± 98.1329 ± 41.8−52.2 ± 12.281.7 ± 15.5
Proxy-RL2340 ± 24.8412 ± 3.86648 ± 43.0342 ± 11.5−341 ± 195257 ± 6.16
Proxy-RL+Pref.2300 ± 29.4418 ± 2.36664 ± 19.6360 ± 12.0−111 ± 26.7144 ± 44.3

We further examine the changes in cooperative behavior during task-reward RL.

Improving execution within the preferred cooperation. In Style A, both task and preference returns increase across all three seeds. The AI's onion-potting frequency rises from 9.67 to 66.7 events per episode, while the player's soup deliveries increase from 3.95 to 20.6. The AI continues to make almost no deliveries. Thus, improved task performance is accompanied by a stronger expression of the preferred division of labor.

Increasing activities penalized by the preference. Style C presents a different outcome. Task return increases from 81.7 to 257, while preference return decreases from −52.2 to −341, with the same direction observed across all three seeds. We examine the changes in AI event frequencies and their contributions under the original preference weights.

AI eventPreference weightSFTProxy-RLΔ\DeltaPref.
Potting−106.9321.1−142
Onion potting−103.0711.7−85.8
Soup delivery−100.5503.28−27.3
Tomato potting+107.209.52+23.2

The table reports mean AI event counts per episode and their contributions to the change in preference return from SFT to Proxy-RL. The two negatively weighted potting-related terms account for approximately 228 points of the 289-point mean preference decrease. Although positively weighted tomato potting also increases, its contribution is insufficient to offset these losses.

Style C penalizes five of its six nonzero event terms, covering onion pickup, potting, soup pickup, and delivery. The penalties therefore span several activities involved in completing the task, and some can accumulate on the same potting action. Across all scored events, task-reward recovery adds approximately 312 penalty points, compared with a gain of only 23.2 points from the positively weighted tomato-potting term. In Style C, the accumulated penalties outweigh the gains in positively scored behavior, so better task execution comes at the expense of preference performance. Preference initialization and a matched partner alone do not prevent this conflict.

Dear Reviewer EHQr,

Thank you again for your thoughtful feedback. We have provided detailed responses and additional experimental results addressing your main concerns. As the discussion period approaches its end, we would greatly appreciate your thoughts on our responses and whether any further clarification would be helpful.

Thank you for your time and consideration.

Reviewer_V95cOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to personalize a task-trained AI teammate when a game platform has gameplay logs and a small number of potentially incorrect post-match endorsements, but cannot interact with the target player during training. The proposed method, Proxy-RL, first fine-tunes the AI on the actions of endorsed teammates. It then learns a proxy for the target player from that player’s gameplay logs and continues training the AI with the proxy using only the game’s task reward. Experiments across three Overcooked layouts and synthetic preference styles show that Proxy-RL often improves both task return and the specified preference score. Controlled comparisons also show that the training partner affects whether task-oriented reinforcement learning preserves the cooperation style acquired from endorsements.

Critical strengths
  1. The paper makes a useful distinction between how a player acts and how that player wants a teammate to act. This distinction motivates the separate uses of gameplay logs and endorsements.
  2. The controlled experiments examine preference initialization, training-partner choice, and reward design separately. The finding that task-reward learning can preserve or replace a preferred division of labor depending on the partner is informative.
  3. The evaluation reports both task performance and preference score, and includes noisy endorsements and a case where the two objectives conflict. These results provide a more nuanced picture than task return alone.
Critical weaknesses
  1. The synthetic preferences may not capture the complexity of individual player preferences. Each experimental style is specified by researcher-defined weights over a limited set of game events and, in some cases, task reward. These rules are used to construct correct endorsements and to evaluate preference satisfaction. This controlled design is useful for isolating mechanisms, but it does not establish that the method can infer why a real player endorses a teammate. Real endorsements may reflect several context-dependent factors, including outcomes, competence, role allocation, and social expectations. The injected incorrect endorsements also model only part of this uncertainty. Consequently, improvements in the specified preference score should not yet be interpreted as evidence of improved satisfaction for real players.
  2. The learned player proxy is evaluated in a setting that may be substantially easier than modeling a real person. For each experimental condition, the target player is a simulated policy, and the proxy is fitted to that policy’s recorded actions. A real player may change strategies across teammates, situations, and time, and may act inconsistently even in similar states. The proxy-fitting variants test additional stochasticity and checkpoint diversity, but do not establish how proxy error affects the final personalized teammate. Because the paper’s central mechanism depends on the training partner, robustness to imperfect or shifting player models is especially important.
  3. The data and computation required per player are not fully characterized. Proxy-RL requires player-specific gameplay logs, endorsement-based fine-tuning, proxy fitting, and further environment-interaction training. The appendix reports some hardware and phase-level timing measurements, but not a complete per-player training cost or a comparison of cost against gains at a common performance target. The experiments also use 72 logged trajectories per target player; it is unclear how performance changes when an individual player has substantially less usable history. These questions matter for the proposed platform-side setting, where personalization may need to scale to many players.
Questions for authors
  1. What aspects of real post-match endorsements do the event-weighted preference styles and randomly injected incorrect labels aim to represent? Which plausible endorsement mechanisms fall outside this construction?
  2. Can preference satisfaction be evaluated using feedback that is independent of the hand-specified reward used to construct the endorsement labels—for example, held-out human judgments or a distinct preference-generation process?
  3. How accurately does the learned proxy reproduce the target player’s behavior in held-out interactions, particularly with teammates absent from the logged data? How do task and preference returns change as proxy fidelity is deliberately reduced?
  4. How sensitive is Proxy-RL to the number and diversity of gameplay trajectories available for an individual player? What is the minimum history needed for a useful proxy?
  5. What is the complete per-player training cost, including preference fine-tuning, proxy fitting, and reinforcement-learning rollouts? How does that cost compare with the baselines, and how would it scale to a large player population?
5 comments

Thank you for recognizing the distinction between player behavior and teammate preferences, and the value of the controlled comparisons. We added experiments on player-history requirements and held-out proxy prediction, and clarify below what the preference evaluation and cost measurements establish.

Response to W1 and Q1: What the endorsement construction represents

Our construction captures a setting in which players care about how a teammate cooperates, but the platform receives only sparse, match-level, and sometimes misleading approval. The event weights specify the cooperation requirements, such as recipe choices and task allocation. A trajectory-level endorsement indicates approval of an interaction without identifying the particular actions that prompted it.

This feedback format is motivated by post-match commendation systems such as Kudos in Brawl Stars. The event-based representation follows the modeling approach used in HSP [1] and ZSC-Eval [2]. The score threshold makes the intended meaning of a clean endorsement explicit, allowing us to examine learning from limited feedback against a known cooperation preference.

The incorrectly endorsed examples are actual rollouts involving the simulated target player and its teammates, drawn from the same gameplay archive. We change their endorsement labels, not their actions. This captures the learning problem created when a recorded like does not indicate the intended approval, as in the sarcastic use discussed in our Introduction. Random sampling controls the amount of this labeling error. The construction does not explicitly represent why a player gives a misleading endorsement or how social context changes that decision. We identify these as extensions of the current feedback model.

Response to Q2: Independent preference evaluation

We have not collected independent human preference judgments or used a separate preference-generation process. Our experiments test whether an agent can learn a specified cooperation preference from limited endorsements. The reward parameters and numerical preference scores are withheld from the personalization learner.

Using the same preference criterion to evaluate new interactions directly measures whether that learning objective has been achieved. This is also the design of DPO's controlled sentiment experiment: a sentiment classifier generates synthetic preference pairs and evaluates the reward of new outputs [3]. Our results therefore support the controlled comparisons of preference initialization, training partners, and rewards. We plan to extend the evaluation to held-out human judgments in future work, examining whether the observed cooperation patterns are also preferred by real players.

Response to W2: Player variability and imperfect proxies

We address the concern that personalization may depend on an idealized player model by evaluating the learned proxy itself and the AI subsequently trained with it. The target players make state-dependent, stochastic decisions. Their behavior can change as different teammates alter the interaction, rather than following a fixed action sequence. The proxy is fitted separately from recorded actions. What remains fixed is the underlying player policy.

Our new experiments measure prediction accuracy in fresh interactions, including teammates absent from the fitting logs, and evaluate downstream personalization after reducing the available player history. This lets us assess the usefulness of an approximate training partner through its measured prediction quality and the resulting cooperation with the original player, as detailed in our response to Q3.

[1] Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. ICLR 2023. [2] Zsc-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination. NeurIPS 2024. [3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

Response to Q3: Proxy accuracy with unseen teammates and downstream effects

We added direct measurements of proxy prediction quality and linked them to the performance of the AI trained with each proxy. These experiments use the original target-player policies for Many Orders Styles A–C.

For each target player, we collected ten fresh episodes with each of the 36 original logging-pool teammates and 15 additional FCP teammate checkpoints absent from the proxy-fitting logs. The target player generated the test actions, and all corresponding proxies were evaluated on the same recorded states and actions.

Action accuracy is the percentage of recorded player actions correctly predicted by the proxy's highest-probability action. We compare four histories: 72 trajectories from 36 teammate groups, 36 trajectories retaining one episode from each of those groups, 36 trajectories from 18 groups with two episodes each, and 18 trajectories from 9 groups with two episodes each.

The table reports mean and standard deviation across three training seeds, rounded to three significant figures. The ACC columns evaluate the player proxy; the return columns evaluate the personalized AI trained with that proxy and paired with the original target player.

StyleHistory: groups × episodesACC: original partner pool (%)ACC: held-out teammates (%)Pref. returnEnv. return
A36 × 244.3 ± 0.30037.2 ± 0.2001080 ± 36.9412 ± 15.6
A36 × 142.5 ± 0.45035.7 ± 0.5601100 ± 8.28418 ± 2.49
A18 × 242.6 ± 0.29036.2 ± 0.1101100 ± 17.5419 ± 6.68
A9 × 233.6 ± 0.39032.4 ± 0.530432 ± 137147 ± 53.3
B36 × 244.3 ± 0.35037.9 ± 0.4301120 ± 22.0428 ± 8.60
B36 × 143.1 ± 0.20036.4 ± 0.4601100 ± 19.7421 ± 7.04
B18 × 241.9 ± 0.28034.8 ± 0.420769 ± 479338 ± 118
B9 × 239.4 ± 0.18034.7 ± 0.360784 ± 490345 ± 121
C36 × 256.4 ± 0.13049.8 ± 0.170296 ± 1.01237 ± 0.470
C36 × 154.9 ± 0.23048.0 ± 0.230287 ± 5.83232 ± 3.68
C18 × 253.1 ± 0.53045.9 ± 0.890281 ± 20.3225 ± 15.9
C9 × 251.1 ± 0.20041.7 ± 0.550269 ± 5.12221 ± 1.25

Although accuracy declines on held-out teammates, it remains above the theoretical 16.7% reference for uniform guessing over all six action labels, showing predictive ability in new teammate interactions. For context, the stochastic target policies' own highest-probability actions match their sampled actions on 48.0%, 49.1%, and 59.6% of held-out steps in Styles A–C, respectively.

With one episode from each of the 36 teammate groups, held-out accuracy decreases by approximately 1.5–1.8 percentage points relative to the full history. Nevertheless, both downstream mean returns remain within 3% of the full-history results across all three styles. The proxy therefore continues to support effective personalization despite this reduction in prediction accuracy.

The narrower histories show greater performance losses or reduced stability in some styles. In particular, the 9 × 2 condition yields lower prediction accuracy and lower mean task and preference returns than the full history across all three styles. These results highlight the value of retaining a broad range of player–teammate experiences. They also show that prediction accuracy and downstream cooperation should be evaluated together: a modest change in average one-step accuracy need not imply a similarly small change in the usefulness of the training partner.

Response to W3 and Q4: Amount and composition of player history

The four conditions in the Q3 table examine both history size and the range of teammates represented in that history. Only the logs used to fit the player proxy change; the endorsement data, preference-SFT initialization, and downstream RL protocol remain fixed.

Retaining one episode from each of the 36 teammate groups preserves near-full-history performance across all three styles. The equal-size 18 × 2 history is less consistent, while the smaller 9 × 2 history produces larger overall reductions. Thus, maintaining broad coverage of cooperation experiences matters alongside the number of recorded episodes.

Among the tested conditions, 36 trajectories spanning all 36 logging-partner groups are the smallest history that keeps both mean returns within 3% of the full-history results across all three styles. The comparison between the two 36-trajectory histories further shows that data requirements should account for history composition, rather than trajectory count alone.

Response to W3 and Q5: Training cost and CPU/GPU execution

The main benchmark uses 72 gameplay trajectories per player. Every run retains the same budget of 10 million environment steps, using 200 parallel environments and 125 rollout-update iterations. The experiments ran on H100 GPUs, with two to three training jobs sharing each GPU concurrently.

Proxy-fitting historyCompleted RL runsMean wall-clock time per RL runRollout share of recorded iteration time
36 groups × 1 episode92.10 hours82.2%
18 groups × 2 episodes92.10 hours82.2%
9 groups × 2 episodes92.08 hours76.8%

These per-run times were measured with multiple jobs progressing concurrently. For comparison, the recorded 20-epoch proxy-fitting stage in the 36-group, one-episode condition takes a median of 50.2 seconds. The timing breakdown identifies rollout collection as the dominant stage. Overcooked simulation and frozen player-proxy inference run in CPU subprocesses, while the learning AI's batched action/value inference and PPO updates run on the GPU. Rollout time also includes communication, data transfer, and synchronization between these components.

Reducing the fitting history changes proxy training but leaves the 10-million-step interaction budget unchanged. Base-policy training can be shared across players, whereas preference SFT, proxy fitting, and subsequent task RL form the player-specific training stages. The measurements above characterize proxy fitting and downstream RL in our current implementation. For larger player populations, improved rollout scheduling, batched policy inference, and CPU/GPU coordination could reduce per-player wall-clock overhead.

Reviewer_cqsgOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to train AI models to be good teammates for particular human preferences, specifically using the type of information that is actually available to game developers (sparse, noisy data on approved teammate trajectories, and rich data on how that player likes to play themselves). The authors run an experiment by developing synthetic human preferences for how to be played with and training RL models in accordance with these, then generating sequences of game data and sparse likes by having these simulated humans play against multiple different simulated teammates (also RL models). Once this data is generated (which is supposed to represent the human data that game developers have access to), the authors train a proxy human on this data. Finally, they fine-tune a final RL agent on the historical liked trajectories of this synthetic human and then do continual learning against the proxy human model to learn how to coordinate with it optimally.

Critical strengths

I think this paper studies an important topic. Being good at a game can mean much more than simple how good you are at winning, especially in a multi-agent scenario.

Critical weaknesses

I found this paper hard to parse. I wish the authors had discussed the experiment setup more clearly as it took me a while to understand.

Questions for authors

Humans who like to be played against a certain way will likely change their play style to represent this; if I'm understanding correctly, the way that you model the correlation between human preference in teammate play style and trajectory data (other than differences in likes) is that the original simulated human is trained with another model which is trying to optimize for that human's preferences. Why is this a good way to represent how preferences influence play style?

2 comments

Thank you for highlighting the importance of cooperation beyond game scores. We have clarified the benchmark construction in the appendix and added an experiment with a fixed, independently trained player.

1. Response to the concern about experimental clarity

For each preference configuration, we train a simulated player together with a teammate guided by the corresponding preference reward. This gives the player experience coordinating with that style. We then fix the selected player and collect its interactions with different teammates. These logs and the available endorsements provide the data for training a personalized AI teammate.

Appendix B.1 now explains the purpose of this construction, and Appendix C.2 specifies the co-training objectives. We also summarize the new fixed-player experiment at the end of the experimental results and provide its full setup in the appendix.

2. Response to the question about player behavior and teammate preferences

Your understanding of the construction is correct. We agree that a player's expectations of a teammate can be reflected in their own play. For example, a player who prefers a teammate that consistently prioritizes tomato recipes may adjust ingredient preparation and pot use to coordinate with that routine.

Co-training allows this coordination to develop through interaction. The preference reward specifies the teammate's style, and the player gains experience working with that style. We use this construction to represent a player whose behavior has developed through prior cooperation with a particular kind of teammate. The player's own actions and the expected teammate behavior can therefore be related without being identical.

Additional evaluation with an independently trained player. We retain the same three Many Orders preference specifications but use one task-trained FCP player without prior coordination training for those styles. The target player and gameplay logs are fixed across styles. We run three training seeds per style and reuse the corresponding single proxy across styles. We compare task-reward RL with and without preference-SFT initialization, using the same proxy and training budget.

The results are summarized below. Policy results report mean and standard deviation over three training seeds, rounded to three significant figures. Offline trajectories provide reference means. The same Proxy-RL (w/o SFT) policies are evaluated under all three styles.

MethodA: Pref.A: Env.B: Pref.B: Env.C: Pref.C: Env.
Offline Trajectories510233366125167165
Preference SFT278 ± 95.9134 ± 52.8175 ± 56.348.0 ± 20.018.7 ± 7.3618.7 ± 7.36
Proxy-RL (w/o SFT)742 ± 21.9325 ± 8.83253 ± 7.78325 ± 8.83−18.3 ± 13.0325 ± 8.83
Proxy-RL710 ± 136328 ± 37.5409 ± 147201 ± 8.60212 ± 53.8175 ± 38.3

Proxy-RL exceeds the offline-reference means in both task and preference return across all three styles. It also improves both returns over its SFT initialization. In Styles B and C, it achieves the highest mean preference returns among the evaluated methods. These results show that preference supervision can guide subsequent task learning even without preference-specific player co-training. The full setup and results are reported in the new appendix section, “Personalization with an Independently Trained Player.”

Dear Reviewer cqsg,

Thank you again for your thoughtful feedback. We have provided detailed responses and additional experimental results addressing your main concerns. As the discussion period approaches its end, we would greatly appreciate your thoughts on our responses and whether any further clarification would be helpful.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.