acceptodds
Withdrawn from ICLR 2027

Scale Buys Imitation, Not the Dividend: Post-Imitation RL Adds a Size-Invariant Gain in an Imperfect-Information Game

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Does the gain from reinforcement learning on top of an imitation model grow with the model? The answer decides whether compute goes into a larger base or into the RL stage, and it is rarely measured, because the gain is small against the variance of the tasks where it matters. We call the gain one RL stage adds over its own clone the RL dividend and measure it on a controlled ladder: five transformer clones (8M to 508M parameters) of one 1.4-billion-decision expert corpus in a four-player imperfect-information game (Riichi mahjong), one belief-state-distillation PPO recipe held fixed, a rules-accurate simulator, replicated runs, and reading rules fixed in advance. The dividend does not scale. From 22.5M to 155M parameters it is +80 to +110 Elo at every rung (25 PPO updates; 30 at 155M, which reads +64 at its 25th), replicated by an independent second run at two rungs, and the three post-RL policies finish within four points of even against each other on the same tables. One RL stage on the 22.5M model (about five GPU-hours on one consumer card, an hour of it PPO) beats the 155M clone on 59% [56, 62] of shared tables; scaling the clone seven-fold and training it longer buys 55% [52, 58], and scaling it from 22.5M to 56M buys nothing (49%). PPO does the same thing at every size (entropy collapses from 1.4 to 0.5 within five updates, 16-23% of held-out human decisions move, two independent 508M runs agree on 97.3% of decisions). At the ends of the range the dividend is run-dependent: 0, +36 and +53 Elo at 8M; +45, +77 and +99 at 508M, where two of the runs differ by 58 Elo on the same tables and one keeps rising to +110 with fifteen further updates. In two small imperfect-information games with exact metrics the dividend is bounded by the teacher's headroom, nil against a near-Nash teacher, growing with capacity against a near-deterministic exploitable one and growing less against a noisy one, the regime a human teacher is in; a ladder of language models pretrained on one corpus (Pythia, 14M to 410M), cloned from one noisy teacher on a synthetic task, takes the same dividend at every size. We also report how we got the shape wrong first: three silent infrastructure defects (stale trajectories from 384 games in flight, a value-head gate dropped in a refactor, mislabelled discard tokens in the deployed encoder) sat behind a spurious inverted-U with every training diagnostic inside its healthy band; two reproduced it on the repaired stack, one had shifted every Elo of the earlier one, and null readings on a fixed instrument and a code audit caught them. Single-run RL scaling curves are not evidence. Scale buys imitation; it does not buy the dividend.

Primary area:
reinforcement learning
Keywords:
scaling laws, reinforcement learning post-training, RL fine-tuning, imitation learning, behavior cloning, PPO, imperfect-information games, multi-agent games, evaluation methodology, reproducibility, pre-registration, Riichi mahjong

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_iYx8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces the RL dividend, the elo of the post RL policy - elo of the BC clone (before RL) and studies it across different games (Mahjong, leduce, and liars dice) and across model sizes (1M to 500M).

Critical strengths

The main observation of the paper is that the increase in performance due to RL (from BC) is roughly constant (in ELO delta). The observation is interesting.

Critical weaknesses

The paper is impossible to read. I firmly believe that encouraging people to read this paper is a disservice. Every paragraph is an information dump and there is no vision in the paper. Out of the 3 paragraph I sampled pangram reports 100% AI. Beyond this, the toy experiments figure 2 are way too small. Given that it's easy to measure exploitability in liars dice, perhaps the authors should pick a variant that is not solvable by almost any method. Beyond this, there is no reason to believe that the same RL parameters need to be used across different model size and the paper provides no clear vision of what the scaling law actually looks like.

Questions for authors

why does table 1 have the same column twice?

Reviewer_RnR4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the experimental scaling of RL after imitation learning with different sizes of models. It does so on a game called Riichi mahjong and on two medium-sized standard benchmark card games. In the latter, the explotability can be computed and the teacher's sub-optimality controlled.

Critical strengths

The paper addresses an important question of how one should allocate compute, but this is only important if one accepts the premise that imitation learning followed by RL is a good approach to computing strategies for games. Personally I don't believe that: game-theoretic techniques have proven to be better.

The selection of benchmark games seems good. I like the addition of the two smaller games on which more controlled experiment can be conducted.

The authors seem to be up to speed with scaling laws in imitation+RL.

Critical weaknesses

The paper is very poorly written -- probably largely by AI. Many sentences are more like long lists than actual sentences. Many concepts are not explained or defined. Results sentences are so dragged on that they are in many cases ununderstandable. It was really hard to understand any of the results.

The paper lacks clear conclusions, section roadmaps, and summarization.

It was hard to learn anything from the paper given how poorly it is written.

Questions for authors

There are many places where things are not explained. I present here, quoted verbatim, a small sampling of them. Hopefully the authors can explain these to me.

On the fixed instrument, little past 8M.

Mortal anchors

The same tables put the three within four points of even.

Reviewer_tDkqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The work studies how RL gains after imitation learning depend on model size. A family of different-scale (8M-508M) models were trained on Majong with PPO using a common human corpus. The paper reports each resulting agent's gain compared to its imitation-only starting point, with approximately similar gains across the middle model sizes, and more variation between runs at the smallest and largest sizes. Additional experiments use Leduc Hold’em, Liar’s Dice, and Pythia language models (14M-410M) on a synthetic sorting task.

Critical strengths
  1. Study about where to spend more compute: a smaller model with more RL vs. a bigger model training addresses the practical problem.
  2. The evaluation has meaningful protocols. Checkpoints are selected using the first half of an evaluation sweep and evaluated using the second half. The authors also report repeated training runs and direct comparisons.
Critical weaknesses
  1. The comparison uses different RL budgets. The +80 to +110 Elo gain across the middle sizes reports the 155M result after 30 updates, while the other model sizes were from the 25th update. The same 155M run gains +64 Elo at update 25.

  2. Across different sizes, the models see different amounts of imitation data and have different training pipelines. The critic’s batch size also varies, and a confidence-weighting setting is only enabled at 8M and 508M but disabled at the in-between sizes. These differences make a big difference, since the small and large models behave differently from the middle. Additionally, the paper reports that the 508M encoder is frozen during PPO. Overall, the models are not quite comparable, which makes it very difficult to draw any conclusion.

  3. The paper is very hard to follow.

Questions for authors
  1. Can the authors compare models with the same imitation data training and pipeline, with only tuning the model sizes as a knob?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.