- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the phenomenon that online reinforcement learning may erase transfer-domain ability gained through offline imitation learning. 20 PPO updates can reduce cross-hero winrates from 38.9% to 29.2% without a gain in the source task. The authors introduce capability-support distillation or CSD. It distills the frozen policy on reward-free states from three sentinel heroes. CSD achieves highest teacher agreement on sentinel heroes and additional 4 held-out heroes, and improves winrates to baseline level, which is better than unconstrained PPO. The authors define decision rentention and trajectory retention and argue they are not equal.
Critical strengths
S1. The evaluation methodology is solid. Training seeds are grouped, truncations are listed out separately, sentinel games and held-out games details are all listed. And numbers are all reported with confidence intervals.
S2. The reporting of the analysis are very honest. To reduce the noisiness of the results, held-out evaluations are repeated after frozen results (run r4), the numbers are detailedly reported even though it weakens the original assumptions.
S3. The problem setup to study how online RL affects the capabilities learned by offline RL is interesting and connects recent related work.
Critical weaknesses
W1. The claimed forgetting and adaptation signals are weak. PPO−BC cross-hero win rate drop has 4/6 seeds negative, and in the replicate run r4 it largely disappears. On the source hero, BC has a win rate of 98.6%, and after 20 PPO updates, there is no real source task gain, and the result only differs by 1-2 games. The forgetting may be a random drift or noise in the experiment setup.
W2. Hero roles likely explain the sentinel failures and the held-out successes. The source hero (112, Luban) is a marksman; the sentinels (121 Mi Yue, 140 Guan Yu, 123 Lu Bu) are fighters; three of four held-out heroes (199 Gongsun Li, 169 Hou Yi, 133 Di Renjie) are marksmen. Role-swapped pairs suggest BC is largely marksman-to-marksman transfer: BC wins 17/18 of 112 v 121 [derived from Table 1, Table 3, Figure 4] but 0/18 of 121 v 112, and wins 16/18 of 133 v 123 but none of 123 v 133. The held-out heroes therefore tests transfer to heroes that are similar to the source, not generalization beyond the sentinel. This is also consistent with the experiment result that CSD only matching BC on held-out heroes.
W3. The main claim is a known phenomenon as compounding error. In this specific setup, the authors claim that the gap is due to shifting away from teacher's states, but no experiment supports this and no fix is proposed to fill in the gap.
W4. Limited statistical significance. Authors transparently reported in section 8.
Questions for authors
-
How were the eight heroes chosen from the 20 supported by HoK Arena, and how was the sentinel/held-out assignment decided? Were other matchups evaluated?
-
The source hero and three of the four held-out heroes appear to be marksmen, while the sentinels are fighters. How do the authors separate an effect of the sentinel memory from BC’s transfer to similar playing-style heroes? Reporting per-matchup results for all methods would help clarify this.
-
How do the results change with longer adaptation or stronger opponents, where PPO yields a real source-task gain?
2 comments
Thank you for your response. Due to the weaknesses I will keep my score.
Reply to W1-W4 and Q1-Q3: the rule behind the hero split, and wins per held-out matchup for every method
Thank you for the careful reading. We agree with all four weaknesses.
W1. You are right. The abstract's drop from 38.9% to 29.2% comes from the first run only, and it is negative in four of six seeds (sign test p = 0.375). On the held-out heroes the change is −18.1 points in the first run and −2.8 in the second; pooled it is −10.4 [−27.1, 2.1], which includes zero (Table 4). On the source hero the pre-trained policy already wins 142 of 144 games, so PPO has almost nothing to gain there (140 of 144 afterwards). The data do not rule out drift. Section 6.2 and Appendix B report the pooled interval; the abstract leads with the first-run number.
W2 and Q2. The paper does not report hero roles, and it should. The per-match records in the supplement confirm the counts you derived for the pre-trained policy (BC): 17 of 18 games won as 112 v 121, 0 of 18 as 121 v 112, 16 of 18 as 133 v 123 and 0 of 18 as 123 v 133. Wins per held-out matchup for every method, from the same records, both runs pooled (36 games per cell; sentinel matchups: Section 6.3, Figure 4):
So 62 of BC's 74 held-out wins come from heroes 133 and 169, which you label marksmen like the source hero. We agree that the held-out set mostly tests transfer to heroes close to the source, and that the sentinel versus held-out contrast is confounded with role. One qualifier: BC wins none of its 36 games with 199, also a marksman by your labels.
The closest the paper comes to isolating the stored sentinel states is CSD against soft distillation: same frozen teacher, weight, batch size and sampler seed. The states differ in hero and in origin (the teacher's own games, against sequences from the offline corpus). Pooled, that difference is +6.2 win-rate points [0.7, 13.9] (three of six seeds positive) and +5.71 reward [−1.19, 13.41] (Table 4), so the paper does not establish it. In the table it comes from 193 and 199 (5 and 7 more wins; four of the seven come from one training seed), where BC is weakest, and not from 133 and 169 (2 and 1 fewer). 193 v 140 is also the mirror of a sentinel matchup, as is 133 v 123 (Section 5). These counts are descriptive only. We cannot separate the stored states from role similarity with this design.
Q1. By a hash rule, in which roles play no part. The seven non-source heroes were sorted by the SHA-256 hash of "20260913:" followed by the hero id, and the first three became sentinels; Section 5 lists them in that order. The rule is in the experiment plan in the supplement and was written before any trained CSD policy was evaluated on sentinel or held-out heroes; BC, PPO and hard-replay results on these matchups were already known (Section 5). It put the three heroes you label fighters in the sentinel set. For the eight heroes we have no selection rule to give you: the experiment plans fix them as the opponent cycle of every run and record no reason. No other matchup appears in any run reported in the paper or its appendix: the per-match files cover 112 against each of the eight heroes and the eight cross-hero pairs of Section 5, fifteen distinct pairs. The hyperparameter screen on seed 101 (Section 8) is not among these files.
Q3. We have not measured this. Every method gets 20 updates against the built-in AI (Section 8), and at 98.6% the source task leaves almost no room for a gain, so these experiments cannot show that regime.
W3. We agree. That matching the teacher on stored states need not reproduce its game outcomes is what compounding error predicts, and Section 3 cites that literature; the paper reports it in one more setting: after online fine-tuning, on heroes that give no reward. The explanation in Section 7, a shift away from the teacher's states, is not tested: we never compared the states the fine-tuned policy visits with the teacher's. The paper proposes no fix.
W4. Agreed; Section 8 lists these limits.
One clarification and two corrections. Agreement with the teacher is measured only on the stored sentinel states (Table 2); on the held-out heroes the paper reports game outcomes only. Section 4.2 says the replay methods draw "the same sequence positions"; for CSD, whose memory holds other games, this can only mean the same sampler seed. And the supplement's README says every number in the paper can be regenerated from the per-match records. That is not true for Table 2 and the agreement numbers quoted from it, which are computed from the stored sentinel states and the checkpoints; neither those files nor that output are in the supplement.
We are not uploading a revised PDF. Settling W2, Q3 and W4 needs new experiments (a role-balanced split, a source task with room to improve, more seeds).