acceptodds
Under review as a conference paper at ICLR 2027

Decision Retention Is Not Trajectory Retention: Preserving Cross-Hero Capabilities During Offline-to-Online Game Reinforcement Learning

Abstract

Offline imitation can produce behavior that transfers beyond the demonstrated task, while narrow online reinforcement learning may erase those untrained capabilities. We study this problem in the licensed Honor of Kings Gamecore using a corpus of 1.78 million decisions from one controlled hero. Across six training seeds and 1,422 Gamecore matches, twenty source-only PPO updates reduce cross-hero scheduled win rate from 38.9% to 29.2% (four of six seeds negative) without a reliable source-reward gain. Source-policy distillation restores cross-hero reward and win rate to near their pre-adaptation levels. We introduce capability-support distillation (CSD), which stores reward-free states and legal-action constraints from sentinel heroes and distills the frozen pre-adaptation policy during online learning. On 39,094 sentinel states, CSD raises teacher agreement from 75.78% under source distillation to 79.35%. In a one-time held-out evaluation released by a hashed decision file fixed before any held-out CSD game, CSD improves over unconstrained PPO by 14.59 reward points [3.21, 26.47] and 20.83 percentage points [8.33, 33.33]; a second independent held-out run shows the same direction with smaller gaps, and pooled over both runs the advantage is 10.63 reward points [2.68, 19.82] and 13.2 percentage points [5.6, 21.5]. Pooled, CSD matches but does not exceed the pre-adaptation policy, and its reward differences from behavioral cloning, hard replay, and source distillation include zero. On sentinel heroes, where the five methods win one of 270 games, CSD agrees with the teacher most closely yet loses fewer games and reaches the step cap more often than the teacher. Decision retention on stored states is therefore measurable but is not by itself evidence of trajectory retention.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
reinforcement learning, catastrophic forgetting, policy distillation, offline-to-online learning, game AI

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iBXLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the phenomenon that online reinforcement learning may erase transfer-domain ability gained through offline imitation learning. 20 PPO updates can reduce cross-hero winrates from 38.9% to 29.2% without a gain in the source task. The authors introduce capability-support distillation or CSD. It distills the frozen policy on reward-free states from three sentinel heroes. CSD achieves highest teacher agreement on sentinel heroes and additional 4 held-out heroes, and improves winrates to baseline level, which is better than unconstrained PPO. The authors define decision rentention and trajectory retention and argue they are not equal.

Critical strengths

S1. The evaluation methodology is solid. Training seeds are grouped, truncations are listed out separately, sentinel games and held-out games details are all listed. And numbers are all reported with confidence intervals.

S2. The reporting of the analysis are very honest. To reduce the noisiness of the results, held-out evaluations are repeated after frozen results (run r4), the numbers are detailedly reported even though it weakens the original assumptions.

S3. The problem setup to study how online RL affects the capabilities learned by offline RL is interesting and connects recent related work.

Critical weaknesses

W1. The claimed forgetting and adaptation signals are weak. PPO−BC cross-hero win rate drop has 4/6 seeds negative, and in the replicate run r4 it largely disappears. On the source hero, BC has a win rate of 98.6%, and after 20 PPO updates, there is no real source task gain, and the result only differs by 1-2 games. The forgetting may be a random drift or noise in the experiment setup.

W2. Hero roles likely explain the sentinel failures and the held-out successes. The source hero (112, Luban) is a marksman; the sentinels (121 Mi Yue, 140 Guan Yu, 123 Lu Bu) are fighters; three of four held-out heroes (199 Gongsun Li, 169 Hou Yi, 133 Di Renjie) are marksmen. Role-swapped pairs suggest BC is largely marksman-to-marksman transfer: BC wins 17/18 of 112 v 121 [derived from Table 1, Table 3, Figure 4] but 0/18 of 121 v 112, and wins 16/18 of 133 v 123 but none of 123 v 133. The held-out heroes therefore tests transfer to heroes that are similar to the source, not generalization beyond the sentinel. This is also consistent with the experiment result that CSD only matching BC on held-out heroes.

W3. The main claim is a known phenomenon as compounding error. In this specific setup, the authors claim that the gap is due to shifting away from teacher's states, but no experiment supports this and no fix is proposed to fill in the gap.

W4. Limited statistical significance. Authors transparently reported in section 8.

Questions for authors
  1. How were the eight heroes chosen from the 20 supported by HoK Arena, and how was the sentinel/held-out assignment decided? Were other matchups evaluated?

  2. The source hero and three of the four held-out heroes appear to be marksmen, while the sentinels are fighters. How do the authors separate an effect of the sentinel memory from BC’s transfer to similar playing-style heroes? Reporting per-matchup results for all methods would help clarify this.

  3. How do the results change with longer adaptation or stronger opponents, where PPO yields a real source-task gain?

2 comments
AuthorsOpenReview ↗

Reply to W1-W4 and Q1-Q3: the rule behind the hero split, and wins per held-out matchup for every method

Thank you for the careful reading. We agree with all four weaknesses.

W1. You are right. The abstract's drop from 38.9% to 29.2% comes from the first run only, and it is negative in four of six seeds (sign test p = 0.375). On the held-out heroes the change is −18.1 points in the first run and −2.8 in the second; pooled it is −10.4 [−27.1, 2.1], which includes zero (Table 4). On the source hero the pre-trained policy already wins 142 of 144 games, so PPO has almost nothing to gain there (140 of 144 afterwards). The data do not rule out drift. Section 6.2 and Appendix B report the pooled interval; the abstract leads with the first-run number.

W2 and Q2. The paper does not report hero roles, and it should. The per-match records in the supplement confirm the counts you derived for the pre-trained policy (BC): 17 of 18 games won as 112 v 121, 0 of 18 as 121 v 112, 16 of 18 as 133 v 123 and 0 of 18 as 123 v 133. Wins per held-out matchup for every method, from the same records, both runs pooled (36 games per cell; sentinel matchups: Section 6.3, Figure 4):

133 v 123169 v 199193 v 140199 v 169
BC3032120
PPO232547
Hard replay2530911
Soft distillation322935
CSD3028812

So 62 of BC's 74 held-out wins come from heroes 133 and 169, which you label marksmen like the source hero. We agree that the held-out set mostly tests transfer to heroes close to the source, and that the sentinel versus held-out contrast is confounded with role. One qualifier: BC wins none of its 36 games with 199, also a marksman by your labels.

The closest the paper comes to isolating the stored sentinel states is CSD against soft distillation: same frozen teacher, weight, batch size and sampler seed. The states differ in hero and in origin (the teacher's own games, against sequences from the offline corpus). Pooled, that difference is +6.2 win-rate points [0.7, 13.9] (three of six seeds positive) and +5.71 reward [−1.19, 13.41] (Table 4), so the paper does not establish it. In the table it comes from 193 and 199 (5 and 7 more wins; four of the seven come from one training seed), where BC is weakest, and not from 133 and 169 (2 and 1 fewer). 193 v 140 is also the mirror of a sentinel matchup, as is 133 v 123 (Section 5). These counts are descriptive only. We cannot separate the stored states from role similarity with this design.

Q1. By a hash rule, in which roles play no part. The seven non-source heroes were sorted by the SHA-256 hash of "20260913:" followed by the hero id, and the first three became sentinels; Section 5 lists them in that order. The rule is in the experiment plan in the supplement and was written before any trained CSD policy was evaluated on sentinel or held-out heroes; BC, PPO and hard-replay results on these matchups were already known (Section 5). It put the three heroes you label fighters in the sentinel set. For the eight heroes we have no selection rule to give you: the experiment plans fix them as the opponent cycle of every run and record no reason. No other matchup appears in any run reported in the paper or its appendix: the per-match files cover 112 against each of the eight heroes and the eight cross-hero pairs of Section 5, fifteen distinct pairs. The hyperparameter screen on seed 101 (Section 8) is not among these files.

Q3. We have not measured this. Every method gets 20 updates against the built-in AI (Section 8), and at 98.6% the source task leaves almost no room for a gain, so these experiments cannot show that regime.

W3. We agree. That matching the teacher on stored states need not reproduce its game outcomes is what compounding error predicts, and Section 3 cites that literature; the paper reports it in one more setting: after online fine-tuning, on heroes that give no reward. The explanation in Section 7, a shift away from the teacher's states, is not tested: we never compared the states the fine-tuned policy visits with the teacher's. The paper proposes no fix.

W4. Agreed; Section 8 lists these limits.

One clarification and two corrections. Agreement with the teacher is measured only on the stored sentinel states (Table 2); on the held-out heroes the paper reports game outcomes only. Section 4.2 says the replay methods draw "the same sequence positions"; for CSD, whose memory holds other games, this can only mean the same sampler seed. And the supplement's README says every number in the paper can be regenerated from the per-match records. That is not true for Table 2 and the agreement numbers quoted from it, which are computed from the stored sentinel states and the checkpoints; neither those files nor that output are in the supplement.

We are not uploading a revised PDF. Settling W2, Q3 and W4 needs new experiments (a role-balanced split, a source task with room to improve, more seeds).

Thank you for your response. Due to the weaknesses I will keep my score.

Reviewer_9ZddOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary
  • This paper investigates a continual RL method called CSD for transfer learning a model trained on one hero to another in the game of Honor of Kings. In CSD, the authors fine-tune the source model with PPO updates both on the new hero and the source model’s decision distributions at a set of new states.
    • The authors find higher performance when using CSD relative to unconstrained PPO or other baselines
Critical strengths

None

Critical weaknesses
  • It is very unclear how CSD differs from other continual learning methods, or what it’s pros and cons are. The authors provide a paragraph in the related work summarizing other continual learning methods, and provide another paragraph describing how CSD works, but provide no comparison between CSD and prior work, and either theoretical or intuitive justification of this method over these other methods. The only experimental results compare CSD against simple baselines, and ignore all continual learning methods.
    • It doesn’t seem like CSD improves performance over any other baselines provided. In Figure 1A, CSD does not have any signifcant performance differences over the baselines, and the 95% CIs overlap between every method. In Figure 1B, none of the models ever win except for a single hard replay win. Truncation could easily indicate that the model was going to lose after 3000 steps, and just hasn’t yet.
Questions for authors
  • How does CSD compare and contrast against other continual learning methods, both theoretically and empirically?
    • Are there any statistically significant improvements that CSD has over other simple and continual learning methods?
2 comments
AuthorsOpenReview ↗

No other continual-learning method is compared; the win-rate gain over plain PPO has sign-test p = 0.063

Thank you for the review. Most of it is right, and we take your points in turn.

The setup. Our description let two things run together, so to be exact: PPO collects reward with one hero only (112, the hero in the offline data). Three other heroes supply stored game states, with no reward, outcome or expert-action label (Section 4.3, Figure 2). The aim is to keep what the pre-trained policy could already do with other heroes while it is fine-tuned on the first, not to move it to a new hero.

How CSD differs from continual-learning methods. CSD is policy distillation from a frozen copy of the pre-trained policy, and Section 2.2 says that part is not new. The one change is where the distillation states come from: three stored games per training seed played with other heroes, in place of states of the hero being trained. Our soft-distillation baseline uses the same teacher and the same weight on source-hero states, so it is the closest control for that change (its states also come from the offline corpus, not from the teacher's own games). The intuition is that source-hero states never show the policy the situations that arise with other heroes. We have no theory for it.

The continual-learning methods cited in Section 2.2 protect behaviour the agent was trained on earlier, by replaying that data, distilling on its states or constraining weights. Here the behaviour to keep was never trained: it belongs to heroes that gave no data and no reward, so the only material is states the pre-trained policy visits with them. We cannot give pros and cons against those methods, because the paper compares with none of them.

Comparison with continual-learning methods. You are right that there is none beyond the replay and distillation controls. Section 8 lists the missing comparison with weight-space methods such as EWC; the replay and distillation methods of Section 2.2 are not compared either, and we have no such results to report.

"It doesn't seem like CSD improves performance over any other baselines provided." Mostly right. Against the pre-trained policy, hard replay and soft distillation, CSD's pooled reward differences include zero, and the paper says so (abstract, Section 6.2). One pooled win-rate interval, against soft distillation, excludes zero (+6.2 points [0.7, 13.9]), but only three of six seeds are positive. The exception is plain PPO. That comparison is paired within training seed, and pooled over both runs the paired difference is +10.63 reward [2.68, 19.82] and +13.2 win-rate points [5.6, 21.5], positive in five of six seeds (Table 4). Figure 1a draws each method's own interval, which hides the pairing; that was our choice of figure.

To your question on significance: for game outcomes, no. With six seeds the exact sign test for the win-rate difference from PPO gives p = 0.063 (Section 6.2), and Section 5 notes that a bootstrap on six values is optimistic. The paper reports two CSD effects in all six seeds: higher agreement with the teacher on the stored states than soft distillation, and fewer finished sentinel games than PPO (Section 6.2). Neither is a win, and the files behind the first are not in the supplement.

Truncation in Figure 1b. We agree that a truncated game may be a loss that has not happened yet. The paper cannot tell, and it never counts one as a success: win rate is wins divided by all scheduled games (Section 5). Figure 1b is the negative half of the result. On those heroes almost no method wins, the pre-trained policy included (one win in 270 games over the five methods). CSD matches the teacher's decisions most closely there, yet it wins no game: it loses fewer and reaches the step cap more often.

No revised PDF accompanies this reply. The comparison you ask for needs new experiments, and we do not have them.

Thanks for the response. I will keep my score of 1.

Reviewer_P72sOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies "offline-to-online RL". A policy trained offline on one hero is able to control a collection of heroes slightly better than the same policy after RL fine-tuning, and the paper proposes CSD to preserve that capability. CSD generates a lot of states using other heroes and adds a regularization term to keep the policy from deviating too far from the reference imitation policy on those states.

Critical strengths
  • The paper compares to decent baselines in Table 3 and reports confidence intervals.
  • The authors have good discussions on the limitations of the work.
Critical weaknesses
  • The paper is poorly written and hard to follow. Concepts like "sentinel heroes" are used before being clearly defined; Table 3 could use better column names than Run r3, Run r4, etc.
  • The motivation of the problem setup is a bit ill-formed. If the policy is only trained on data from one source hero during offline training and then RL fine-tuned on the same source hero, it is reasonable to NOT use the policy to control any other hero because it is probably a weak policy to control those heros (such as the sentinel heros) anyway. The numbers in the paper also show this, as both pretrained and fine-tuned policies rarely win when controlling the sentinel heroes.
  • At its core, this paper says that adding distillation on a broader range of states (adding sentinel states) helps preserve behaviors that the pretrained policy has. As the authors have acknowledged, this is not a new technique and its effect has been demonstrated in many prior works.
  • The experimental results are weak because of short training (20 PPO updates), wide confidence intervals in Table 3, etc.
Questions for authors

If both the BC pretraining and RL finetuning are conducted on a single source hero, what's the motivation to let the policy preserve the ability to control other heros.

1 comment
AuthorsOpenReview ↗

Why keep the other heroes: our answer, and agreement on the writing, the novelty and the short training

Thank you for the review. We start with your question, which is also your second weakness, then take the other three; we accept all four.

Why preserve the ability to control other heroes? Not for practical use: a policy trained on one hero would not be used to control the others. We used it as a small case of a more general situation, in which a pre-trained policy can do more than the one task it is later fine-tuned on, and fine-tuning may remove the rest. You are right about the sentinel heroes: in the main evaluation the pre-trained policy wins none of its 54 games with them (Figure 1b), so there is little there to preserve. With the four held-out heroes it wins about half of its games (51.4% over both runs, most of the wins with two of the four heroes), and that is the ability the paper tracks. After plain PPO the figure is 41.0%; the interval of that difference includes zero (Table 4). And the paper has no experiment in a setting where someone needs that ability, for example a policy pre-trained on many heroes and then fine-tuned on one. We accept that point.

Writing. Agreed. "Sentinel heroes" appears in the abstract and the introduction before Section 5 defines it, and Table 3 uses our internal run labels. For this discussion: sentinel heroes are the three non-source heroes whose game states the method may store, without rewards or expert-action labels. Held-out heroes are the other four: the policy never controls them in training, and no CSD game with them was seen before the method was fixed. They do appear as opponents, and baseline results with them were known earlier (Section 5). "Run r3" is the first held-out evaluation and "Run r4" is a repeat of it on fresh games.

Novelty. Agreed. Distilling a frozen teacher on a wider set of states is a known technique, and Section 2.2 says our contribution is not the KL term or the frozen teacher. What the paper claims is narrower than a new technique: the added states come from heroes the policy was never trained on, with no reward or expert-action label, and the evaluation keeps agreement on those states apart from game outcomes.

Results. Agreed: every method gets 20 PPO updates, there is one distillation weight and six training seeds, and the pooled intervals in Table 3 are wide. Section 8 lists these limits. The one difference whose pooled intervals exclude zero on both reward and win rate is CSD against plain PPO (+13.2 win-rate points [5.6, 21.5], Table 4); it is positive in five of six seeds, and the exact sign test gives p = 0.063 (Section 6.2). Against the pre-trained policy and the two replay baselines the pooled reward differences include zero.

We have not uploaded a revised PDF. The definitions above are for this discussion; the main points need a different setting and new experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.