- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes KAIKA, a population-based framework for reinforcement-learning post-training using comparisons between continuations from the same state. In chess games, it adds an engine-labelled DPO term to a policy-gradient objective. In chess puzzles, its DUEL variant replaces the policy-gradient term with comparisons between successful and failed attempts, using summed sequence log-probabilities. The authors report improvements over their GRPO baselines and attribute part of the puzzle gains to avoiding length normalization that can penalize shared correct prefixes.
Critical strengths
This work studies an interesting research topic and is transparent about negative results and limitations.
Critical weaknesses
-
Insufficient methodological novelty The paper acknowledges that single-policy DUEL with constant pair weights reduces to a DisCO-style discriminative objective with summed scores. Length-normalization corrections and all-success–failure preference pairing also have established precedents. Meanwhile, the more distinctive components lack demonstrated benefits: increasing population size or encouraging diversity does not improve average puzzle accuracy, and SPEC/UNIFIED do not consistently outperform DUEL. The incremental contribution beyond existing objectives and normalization corrections therefore remains unclear.
-
Writing needs improvement. The paper introduces several variants (e.g., KAIKA, DUEL, SPEC, and UNIFIED) alongside a full objective that differs from the implementations supporting the headline results. Their relationships and experimental roles are spread across the main text and appendices, making it difficult to identify the essential algorithm and connect each claimed contribution to its supporting evidence. The presentation would benefit from a more focused account of the validated method, a clearer distinction between the full framework and evaluated subsets, and a sharper explanation of what is new relative to prior work.
Questions for authors
Please see the weaknesses part.
At the current stage, I don't think the paper meets the standard for acceptance at a top-tier ML conference, and I tend toward rejection. However, I am open to revisiting my assessment based on the authors’ response and subsequent discussion.
4 comments
Reply to Reviewer Cxf5 (2 of 4)
W1c, more members and diversity. In chess the score against GRPO rises with the number of slots. With 8 rows per slot it goes from 0.5599 at 2 slots to 0.6604 at 16 against GRPO on 32 games and 64 rows, and with the step held at 32 games and 64 rows it stays between 0.6309 and 0.6503 (Table 3 in the general response). In Blotto, training the members lowers the exploitability of KAIKA's Nash (maximin) mixture against all 231 legal opponent allocations from 0.449 to 0.225 (general response, point 6).
Agreed for the 32M puzzle model. Four members with founding raise the mean pass@1 at 5M from 0.0811 to 0.0832 and lower it at 32M from 0.1504 to 0.1460 (App. Table 29). These are the last two rows of Table 5.
Table 5 of the revision, all rows: from one policy trained by the pair loss to DUEL on chess puzzles (ten training runs of 600 steps), each row adding its named change; a cell is the row's gain over the base as a multiple of that GRPO arm's, with the training runs on which the row is higher.
| token-mean GRPO, 5M | token-mean GRPO, 32M | GRPO (sum), 5M | GRPO (sum), 32M | |
|---|---|---|---|---|
| One policy, per-token scores, constant weight | 1.26 (7/10) | 1.15 (7/10) | 0.95 (4/10) | 0.88 (1/10) |
| + Fisher weight | 1.45 (9/10) | 1.25 (8/10) | 1.09 (6/10) | 0.96 (5/10) |
| + summed scores | 1.58 (9/10) | 1.53 (10/10) | 1.19 (7/10) | 1.18 (9/10) |
| + retain weight 0.05 (from 0.001) | 1.69 (10/10) | 1.60 (10/10) | 1.27 (8/10) | 1.24 (9/10) |
| + four members with founding (DUEL) | 1.84 (9/10) | 1.48 (10/10) | 1.38 (8/10) | 1.14 (6/10) |
Against GRPO (sum) with its KL coefficient raised to 0.05, the last two rows give 0.99 (4/10) and 1.08 (6/10) at 5M, 1.14 (8/10) and 1.05 (6/10) at 32M.
The diversity controls also lower the mean (App. Table 32), and DUEL trains no diversity term. App. Table 33 sweeps the members.
W1d, the two other variants. Neither variant beats DUEL consistently (submitted Table 6, now App. Table 37). Both give each member a policy-gradient term. DUEL is the puzzle method of §3.2, and both variants are now in App. C.3.
Reply to Reviewer Cxf5 (3 of 4)
W1e, beyond the correction. Beyond the normalisation correction, the pair loss on one policy is above GRPO with summed aggregation at 32M at equal coefficients. One policy with the pair loss, the Fisher weight and summed scores gains 1.18 times as much as GRPO (sum) when retain weight and KL coefficient are both at 0.001 (9/10 training runs), and 1.14 times when both are at 0.05 (8/10). At 5M the same two multiples are 1.19 and 0.99 (App. Tables 28 and 29).
App. Table 29, six of its rows: one policy (summed scores, Fisher weight) and DUEL against GRPO (sum), ten training runs of 600 steps; greedy pass@1 of the row and of the GRPO arm, the mean difference per training run, its paired t and the training runs on which the row is higher.
| Size | Row | GRPO arm | pass@1, row | pass@1, arm | Difference | t | Higher |
|---|---|---|---|---|---|---|---|
| 5M | one policy, retain 0.001 | GRPO (sum) | 0.0796 | 0.0759 | +0.0037 | +1.45 | 7/10 |
| 5M | one policy, retain 0.05 | GRPO (sum), KL 0.05 | 0.0811 | 0.0813 | -0.0002 | -0.10 | 4/10 |
| 5M | DUEL | GRPO (sum), KL 0.05 | 0.0832 | 0.0813 | +0.0020 | +0.74 | 6/10 |
| 32M | one policy, retain 0.001 | GRPO (sum) | 0.1480 | 0.1394 | +0.0086 | +4.15 | 9/10 |
| 32M | one policy, retain 0.05 | GRPO (sum), KL 0.05 | 0.1504 | 0.1435 | +0.0070 | +3.40 | 8/10 |
| 32M | DUEL | GRPO (sum), KL 0.05 | 0.1460 | 0.1435 | +0.0025 | +0.77 | 6/10 |
At KL 0.05 DUEL's gain is 1.08 and 1.05 times that of GRPO (sum) (Table 4 in the general response) and 1.21 and 1.18 times that of Dr. GRPO (8/10 each).
App. Table 30, all rows: Dr. GRPO on chess puzzles, ten training runs of 600 steps; its greedy pass@1 and its gain over the base as a multiple of each GRPO arm's, then DUEL's gain as a multiple of Dr. GRPO's with the mean difference per training run and its paired t. In brackets, the training runs on which the first-named arm is higher; no multiple where Dr. GRPO's gain is under 0.007.
| Size | KL | lr | pass@1 | GRPO | GRPO (sum) | GRPO (sum), KL 0.05 | DUEL's multiple | Difference | t |
|---|---|---|---|---|---|---|---|---|---|
| 5M | 0.05 | 1e-5 | 0.0787 | 1.52 (8/10) | 1.15 (6/10) | 0.90 (5/10) | 1.21 (8/10) | +0.0045 | +3.22 |
| 5M | 0.05 | 2e-5 | 0.0704 | 0.95 (6/10) | 0.71 (2/10) | 0.56 (0/10) | 1.94 (9/10) | +0.0129 | +3.96 |
| 5M | 0.05 | 4e-5 | 0.0584 | 0.12 (0/10) | 0.09 (0/10) | 0.07 (0/10) | +0.0248 | +9.96 | |
| 5M | none | 1e-5 | 0.0714 | 1.02 (5/10) | 0.77 (2/10) | 0.60 (0/10) | 1.81 (10/10) | +0.0118 | +5.55 |
| 32M | 0.05 | 1e-5 | 0.1378 | 1.25 (9/10) | 0.97 (4/10) | 0.89 (3/10) | 1.18 (8/10) | +0.0082 | +2.25 |
| 32M | 0.05 | 2e-5 | 0.1201 | 0.76 (1/10) | 0.59 (1/10) | 0.54 (0/10) | 1.95 (10/10) | +0.0259 | +11.83 |
| 32M | 0.05 | 4e-5 | 0.0922 | -0.02 (0/10) | -0.01 (0/10) | -0.01 (0/10) | +0.0538 | +13.80 | |
| 32M | none | 1e-5 | 0.1342 | 1.15 (8/10) | 0.89 (3/10) | 0.82 (1/10) | 1.28 (9/10) | +0.0118 | +3.31 |
In chess the pair loss raises KAIKA's score against GRPO on each of its five training runs (Table 2 in the general response), with learning members by 0.1005 at T = 0.7 and 0.0544 at T = 0.3.
Reply to Reviewer Cxf5 (4 of 4)
W2a, too many variants. The submission presented four variants and an objective that its headline results did not train (submitted App. C, Eq. 15). §3.1 and §3.2 define two algorithms, KAIKA for games and DUEL for puzzles, as equations before any result, and App. Table 6 lists what each result trains.
Tables 1 to 3 test KAIKA (the table's first row), and Tables 4 and 5 test DUEL on chess puzzles. That objective is the variant of App. A.7 (Eq. 13), whose three chess arms are in the table's second row with one training run per arm of 100 steps, and no result in Section 4 trains it. The three engine-free KAIKA arms, which support neither method, left the paper (listed on p. 16 of the revision).
W2b, framework and subsets. The submission's sentence on this was wrong (submitted §7 and App. C). KAIKA, the learner of the headline chess results, is the method (§3.1), and it is above GRPO on every training run at each equal temperature from 0.1 to 0.9 (Table 1, Figure 2a). App. Figure 5 lists the components of each arm. Section 3.3 states what each pair loss compares and where it departs from all-pairs DPO on one policy's group (Eq. 7). With reward-gap weights, at the reference and for equal lengths, that pair gradient is a positive multiple of the GRPO gradient, and for rollouts that other members sampled it is not member k's on-policy gradient. In KAIKA a played row's pair gradient in the logits is not a rescaling of its GRPO gradient when a third move has positive probability, even at the reference (App. A.6). On novelty, see W1a and general response, point 5. App. F discusses population methods such as NeuPL.
Reply to Reviewer Cxf5 (1 of 4)
We thank the reviewer for the careful report.
W1a, a DisCO-style objective. The DisCO relation covers one restricted contrast term. The submission stated it at l. 133 to 135 and l. 224 and 225 for one policy and constant pair weights, with summed scores where DisCO's J1 has token-averaged ones (§2.1). That restricted case gains 1.15 and 1.13 times GRPO (sum) at retain weight 0.001 (7/10 training runs each; App. Table 28).
App. Table 28, its rows for one policy trained by the pair loss with summed scores and its 600-step DUEL rows (chess puzzles, ten training runs of 600 steps): greedy pass@1 and the gain over the base as a multiple of each GRPO arm's, with the training runs on which the row is higher. The first row of each size is the restricted case.
What is new is how the pairs are built and how a population trains on them (general response, point 5). In KAIKA, LoRA members of one trained policy play one another without search, and low-rated members are replaced by that policy plus noise (§3.1). Beside a full-weight GRPO step on all slots' played moves, the shared policy applies the DPO logistic loss to pairs from the population's games: a player's played move against an unplayed branch move at that position, ordered by the engine. Each member takes its own Adam step on its policy gradient minus the mean of all slots' gradients (Eq. 4). DUEL has four members, each founded by one GRPO step from one parent (§3.2). It trains each member on verifier-ordered pairs across all members' rollouts, weighted by a Fisher-scoring step in the pair's margin, with no GRPO term after founding, and holds each member to the frozen base model by a retain term on its own rollouts.
W1b, precedents. Both have precedents, cited in §2.1 and §2.2. DUEL is compared with both baselines that remove length normalisation, GRPO with summed aggregation and Dr. GRPO (general response, point 4). The updates still differ: with 1 of 16 rollouts passing, the weight of a puzzle is 1 for DUEL, 0.24 for GRPO with summed aggregation and 0.06 for Dr. GRPO, at zero margins and one shared length constant (§3.3).