- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies personalized AI teammates in cooperative games using historical gameplay logs and sparse, potentially noisy endorsements, without requiring online interaction with the target player during training. The authors distinguish between how a player acts and how they prefer their teammate to cooperate, which is insightful. Using controlled experiments in Overcooked, the authors studied how preference initialization, training partners, and reward design influence adaptation.
Critical strengths
-
The distinction between how a user behaves and how they want a teammate to behave is important. Using gameplay logs to model the former and endorsements to learn the latter provides a intuitive approach to personalization.
-
The experiments separately examine preference initialization, training partners, and reward design. The finding that the training partner influences whether task-oriented RL preserves previously learned preferences is informative.
Critical weaknesses
-
In Distant Tomato Style C, task performance improves substantially while preference satisfaction deteriorates. Although this limitation is acknowledged, a more systematic analysis of task & preference alignment is needed to establish when adaptation reinforces the preferred behavior and when it replaces it.
-
The target player is jointly trained with a teammate using a specific preference reward. Proxy-RL may therefore recover their learned cooperation rather than preserve user preferences. Testing different preferences with the same target player would help clarify this
Questions for authors
-
Does Proxy-RL remain effective when the target-user policy is trained independently of the preferred teammate style? Can you evaluate multiple teammate preferences while holding the user policy fixed?
-
What determines whether task-only RL preserves or overrides the preference learned during SFT? Can you systematically vary the conflict between task reward and preference reward?
3 comments
Response to W1 and Q2: Task progress and preference retention
We extended the Distant Tomato analysis to examine the behavioral changes behind the task and preference returns. For each style, we compare its actual SFT initialization with subsequent task-reward RL using the learned player proxy. Each style retains its original target player and preference definition. The results below use the n=4 feedback condition.
The table reports mean ± standard deviation across three training seeds, rounded to three significant figures.
| Method | A: Pref. | A: Env. | B: Pref. | B: Env. | C: Pref. | C: Env. |
|---|---|---|---|---|---|---|
| Preference SFT | 492 ± 298 | 75.3 ± 53.5 | 628 ± 98.1 | 329 ± 41.8 | −52.2 ± 12.2 | 81.7 ± 15.5 |
| Proxy-RL | 2340 ± 24.8 | 412 ± 3.86 | 648 ± 43.0 | 342 ± 11.5 | −341 ± 195 | 257 ± 6.16 |
| Proxy-RL+Pref. | 2300 ± 29.4 | 418 ± 2.36 | 664 ± 19.6 | 360 ± 12.0 | −111 ± 26.7 | 144 ± 44.3 |
We further examine the changes in cooperative behavior during task-reward RL.
Improving execution within the preferred cooperation. In Style A, both task and preference returns increase across all three seeds. The AI's onion-potting frequency rises from 9.67 to 66.7 events per episode, while the player's soup deliveries increase from 3.95 to 20.6. The AI continues to make almost no deliveries. Thus, improved task performance is accompanied by a stronger expression of the preferred division of labor.
Increasing activities penalized by the preference. Style C presents a different outcome. Task return increases from 81.7 to 257, while preference return decreases from −52.2 to −341, with the same direction observed across all three seeds. We examine the changes in AI event frequencies and their contributions under the original preference weights.
| AI event | Preference weight | SFT | Proxy-RL | Pref. |
|---|---|---|---|---|
| Potting | −10 | 6.93 | 21.1 | −142 |
| Onion potting | −10 | 3.07 | 11.7 | −85.8 |
| Soup delivery | −10 | 0.550 | 3.28 | −27.3 |
| Tomato potting | +10 | 7.20 | 9.52 | +23.2 |
The table reports mean AI event counts per episode and their contributions to the change in preference return from SFT to Proxy-RL. The two negatively weighted potting-related terms account for approximately 228 points of the 289-point mean preference decrease. Although positively weighted tomato potting also increases, its contribution is insufficient to offset these losses.
Style C penalizes five of its six nonzero event terms, covering onion pickup, potting, soup pickup, and delivery. The penalties therefore span several activities involved in completing the task, and some can accumulate on the same potting action. Across all scored events, task-reward recovery adds approximately 312 penalty points, compared with a gain of only 23.2 points from the positively weighted tomato-potting term. In Style C, the accumulated penalties outweigh the gains in positively scored behavior, so better task execution comes at the expense of preference performance. Preference initialization and a matched partner alone do not prevent this conflict.
Dear Reviewer EHQr,
Thank you again for your thoughtful feedback. We have provided detailed responses and additional experimental results addressing your main concerns. As the discussion period approaches its end, we would greatly appreciate your thoughts on our responses and whether any further clarification would be helpful.
Thank you for your time and consideration.
We thank the reviewer for recognizing the importance of distinguishing player behavior from preferred teammate behavior, as well as the insights from our analysis of training partners and preference retention. To address the concerns raised, we conducted an additional experiment with an independently trained player and further analyzed the behavioral changes underlying the task–preference trade-offs.
Response to W2 and Q1: Multiple preferences with an independently trained player
We conducted the fixed-player comparison you suggested. In Many Orders, we use one task-trained FCP policy as the target player, rather than selecting a player from a preference-specific co-training pair. Styles A–C share this target player and the same 72-trajectory gameplay archive.
For each training seed, we reuse the same learned player proxy across styles. We also compare Proxy-RL with Proxy-RL (w/o SFT), which starts directly from the FCP base without preference fine-tuning. Both RL conditions use the same proxy, task-reward protocol, 10M-step budget, and checkpoint-selection rule.
The table reports mean ± standard deviation across three training seeds, rounded to three significant figures. Offline trajectories provide reference means. The same three policies trained without preference SFT are evaluated under all three preference specifications.
Proxy-RL improves both returns over its SFT initialization and exceeds the offline-reference means across all three styles. In Styles B and C, it achieves the highest mean preference returns among the compared methods. Its preference advantage over the no-SFT condition occurs despite lower task returns.
These results directly address the co-training concern: preference-dependent adaptation also occurs when the player has not been co-trained for the corresponding style. The preference supervision can influence subsequent cooperation, rather than simply reproducing a single cooperation pattern associated with that player.