acceptodds
Under review as a conference paper at ICLR 2027

KAIKA: Reinforcement Learning from Pairwise Comparisons of Continuations

44 views

Abstract

Reinforcement-learning post-training reduces a model’s behaviour to one scalar reward per response; Group Relative Policy Optimization (GRPO), the standard method, standardises these rewards within a group of responses to the same prompt. Many post-training outcomes are comparisons, a game won against an opponent or a puzzle solved where another attempt failed, and the scalar discards which continuation beat which. We ask whether a learner told only which of two continuations of the same state was better can post-train a model beyond GRPO. KAIKA is a population of policies that learns from such comparisons, with log-probabilities summed over each continuation: in games, a winner-vs-loser DPO term added to the policy gradient, pairing the played move with an alternative from the same position; on puzzles, where the comparisons replace the policy gradient, each successful attempt against each failed one. In chess, starting from a Stockfish-distilled 230M-parameter model, KAIKA gains +156 Elo over the base against GRPO’s +79 at T = 0.7 (paired score difference +0.0986, 95% CI [+0.0599, +0.1374], 5/5 seeds) and wins head-to-head on all 10 seeds; at T = 0.3 the paired difference is +0.0404 ([−0.0098, +0.0906]). The advantage over GRPO persists when every population member is an exact copy of the trained policy. On chess puzzles solved by 5M- and 32M-parameter language models, the comparison update exceeds GRPO with per-token-mean aggregation by +0.0098 at 5M (4/4 seeds) and +0.0127 at 32M (5/6 seeds), and by +0.0167 on three 32M seeds trained for 600 steps. Ablations locate part of the effect: replacing the summed log-probabilities with per-token means lowers accuracy on 3/3 seeds, by 0.0063 on average. Because an attempt ends at its first wrong move, failures are shorter than successes, and token averaging weights the correct opening moves a failure shares with a success more heavily in the failure; summing cancels their direct contribution. Neither population size (1 to 16 members), a curvature-based pair weight, nor a partial-credit reward adds measurable accuracy. Over training, the per-move rate of illegal moves rises by 41–91% under GRPO against 2–24% under KAIKA.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, LLM post-training, pairwise comparisons, GRPO, preference optimization, policy gradient, population-based training, self-play, chess

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Cxf5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes KAIKA, a population-based framework for reinforcement-learning post-training using comparisons between continuations from the same state. In chess games, it adds an engine-labelled DPO term to a policy-gradient objective. In chess puzzles, its DUEL variant replaces the policy-gradient term with comparisons between successful and failed attempts, using summed sequence log-probabilities. The authors report improvements over their GRPO baselines and attribute part of the puzzle gains to avoiding length normalization that can penalize shared correct prefixes.

Critical strengths

This work studies an interesting research topic and is transparent about negative results and limitations.

Critical weaknesses
  1. Insufficient methodological novelty The paper acknowledges that single-policy DUEL with constant pair weights reduces to a DisCO-style discriminative objective with summed scores. Length-normalization corrections and all-success–failure preference pairing also have established precedents. Meanwhile, the more distinctive components lack demonstrated benefits: increasing population size or encouraging diversity does not improve average puzzle accuracy, and SPEC/UNIFIED do not consistently outperform DUEL. The incremental contribution beyond existing objectives and normalization corrections therefore remains unclear.

  2. Writing needs improvement. The paper introduces several variants (e.g., KAIKA, DUEL, SPEC, and UNIFIED) alongside a full objective that differs from the implementations supporting the headline results. Their relationships and experimental roles are spread across the main text and appendices, making it difficult to identify the essential algorithm and connect each claimed contribution to its supporting evidence. The presentation would benefit from a more focused account of the validated method, a clearer distinction between the full framework and evaluated subsets, and a sharper explanation of what is new relative to prior work.

Questions for authors

Please see the weaknesses part.

At the current stage, I don't think the paper meets the standard for acceptance at a top-tier ML conference, and I tend toward rejection. However, I am open to revisiting my assessment based on the authors’ response and subsequent discussion.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer Cxf5 (1 of 4)

We thank the reviewer for the careful report.

W1a, a DisCO-style objective. The DisCO relation covers one restricted contrast term. The submission stated it at l. 133 to 135 and l. 224 and 225 for one policy and constant pair weights, with summed scores where DisCO's J1 has token-averaged ones (§2.1). That restricted case gains 1.15 and 1.13 times GRPO (sum) at retain weight 0.001 (7/10 training runs each; App. Table 28).

App. Table 28, its rows for one policy trained by the pair loss with summed scores and its 600-step DUEL rows (chess puzzles, ten training runs of 600 steps): greedy pass@1 and the gain over the base as a multiple of each GRPO arm's, with the training runs on which the row is higher. The first row of each size is the restricted case.

SizeRowpass@1token-mean GRPOGRPO (sum)GRPO (sum), KL 0.05
5Msummed scores, constant weight, retain 0.0010.07871.52 (8/10)1.15 (7/10)0.90 (3/10)
5Msummed scores, Fisher weight, retain 0.0010.07961.58 (9/10)1.19 (7/10)0.93 (4/10)
5Msummed scores, constant weight, retain 0.050.08141.71 (9/10)1.29 (7/10)1.01 (5/10)
5Msummed scores, Fisher weight, retain 0.050.08111.69 (10/10)1.27 (8/10)0.99 (4/10)
5MDUEL0.08321.84 (9/10)1.38 (8/10)1.08 (6/10)
32Msummed scores, constant weight, retain 0.0010.14541.46 (10/10)1.13 (7/10)1.04 (5/10)
32Msummed scores, Fisher weight, retain 0.0010.14801.53 (10/10)1.18 (9/10)1.09 (9/10)
32Msummed scores, constant weight, retain 0.050.14961.58 (10/10)1.22 (8/10)1.12 (7/10)
32Msummed scores, Fisher weight, retain 0.050.15041.60 (10/10)1.24 (9/10)1.14 (8/10)
32MDUEL0.14601.48 (10/10)1.14 (6/10)1.05 (6/10)

What is new is how the pairs are built and how a population trains on them (general response, point 5). In KAIKA, LoRA members of one trained policy play one another without search, and low-rated members are replaced by that policy plus noise (§3.1). Beside a full-weight GRPO step on all slots' played moves, the shared policy applies the DPO logistic loss to pairs from the population's games: a player's played move against an unplayed branch move at that position, ordered by the engine. Each member takes its own Adam step on its policy gradient minus the mean of all slots' gradients (Eq. 4). DUEL has four members, each founded by one GRPO step from one parent (§3.2). It trains each member on verifier-ordered pairs across all members' rollouts, weighted by a Fisher-scoring step in the pair's margin, with no GRPO term after founding, and holds each member to the frozen base model by a retain term on its own rollouts.

W1b, precedents. Both have precedents, cited in §2.1 and §2.2. DUEL is compared with both baselines that remove length normalisation, GRPO with summed aggregation and Dr. GRPO (general response, point 4). The updates still differ: with 1 of 16 rollouts passing, the weight of a puzzle is 1 for DUEL, 0.24 for GRPO with summed aggregation and 0.06 for Dr. GRPO, at zero margins and one shared length constant (§3.3).

AuthorsOpenReview ↗

Reply to Reviewer Cxf5 (2 of 4)

W1c, more members and diversity. In chess the score against GRPO rises with the number of slots. With 8 rows per slot it goes from 0.5599 at 2 slots to 0.6604 at 16 against GRPO on 32 games and 64 rows, and with the step held at 32 games and 64 rows it stays between 0.6309 and 0.6503 (Table 3 in the general response). In Blotto, training the members lowers the exploitability of KAIKA's Nash (maximin) mixture against all 231 legal opponent allocations from 0.449 to 0.225 (general response, point 6).

Agreed for the 32M puzzle model. Four members with founding raise the mean pass@1 at 5M from 0.0811 to 0.0832 and lower it at 32M from 0.1504 to 0.1460 (App. Table 29). These are the last two rows of Table 5.

Table 5 of the revision, all rows: from one policy trained by the pair loss to DUEL on chess puzzles (ten training runs of 600 steps), each row adding its named change; a cell is the row's gain over the base as a multiple of that GRPO arm's, with the training runs on which the row is higher.

token-mean GRPO, 5Mtoken-mean GRPO, 32MGRPO (sum), 5MGRPO (sum), 32M
One policy, per-token scores, constant weight1.26 (7/10)1.15 (7/10)0.95 (4/10)0.88 (1/10)
+ Fisher weight1.45 (9/10)1.25 (8/10)1.09 (6/10)0.96 (5/10)
+ summed scores1.58 (9/10)1.53 (10/10)1.19 (7/10)1.18 (9/10)
+ retain weight 0.05 (from 0.001)1.69 (10/10)1.60 (10/10)1.27 (8/10)1.24 (9/10)
+ four members with founding (DUEL)1.84 (9/10)1.48 (10/10)1.38 (8/10)1.14 (6/10)

Against GRPO (sum) with its KL coefficient raised to 0.05, the last two rows give 0.99 (4/10) and 1.08 (6/10) at 5M, 1.14 (8/10) and 1.05 (6/10) at 32M.

The diversity controls also lower the mean (App. Table 32), and DUEL trains no diversity term. App. Table 33 sweeps the members.

W1d, the two other variants. Neither variant beats DUEL consistently (submitted Table 6, now App. Table 37). Both give each member a policy-gradient term. DUEL is the puzzle method of §3.2, and both variants are now in App. C.3.

AuthorsOpenReview ↗

Reply to Reviewer Cxf5 (3 of 4)

W1e, beyond the correction. Beyond the normalisation correction, the pair loss on one policy is above GRPO with summed aggregation at 32M at equal coefficients. One policy with the pair loss, the Fisher weight and summed scores gains 1.18 times as much as GRPO (sum) when retain weight and KL coefficient are both at 0.001 (9/10 training runs), and 1.14 times when both are at 0.05 (8/10). At 5M the same two multiples are 1.19 and 0.99 (App. Tables 28 and 29).

App. Table 29, six of its rows: one policy (summed scores, Fisher weight) and DUEL against GRPO (sum), ten training runs of 600 steps; greedy pass@1 of the row and of the GRPO arm, the mean difference per training run, its paired t and the training runs on which the row is higher.

SizeRowGRPO armpass@1, rowpass@1, armDifferencetHigher
5Mone policy, retain 0.001GRPO (sum)0.07960.0759+0.0037+1.457/10
5Mone policy, retain 0.05GRPO (sum), KL 0.050.08110.0813-0.0002-0.104/10
5MDUELGRPO (sum), KL 0.050.08320.0813+0.0020+0.746/10
32Mone policy, retain 0.001GRPO (sum)0.14800.1394+0.0086+4.159/10
32Mone policy, retain 0.05GRPO (sum), KL 0.050.15040.1435+0.0070+3.408/10
32MDUELGRPO (sum), KL 0.050.14600.1435+0.0025+0.776/10

At KL 0.05 DUEL's gain is 1.08 and 1.05 times that of GRPO (sum) (Table 4 in the general response) and 1.21 and 1.18 times that of Dr. GRPO (8/10 each).

App. Table 30, all rows: Dr. GRPO on chess puzzles, ten training runs of 600 steps; its greedy pass@1 and its gain over the base as a multiple of each GRPO arm's, then DUEL's gain as a multiple of Dr. GRPO's with the mean difference per training run and its paired t. In brackets, the training runs on which the first-named arm is higher; no multiple where Dr. GRPO's gain is under 0.007.

SizeKLlrpass@1GRPOGRPO (sum)GRPO (sum), KL 0.05DUEL's multipleDifferencet
5M0.051e-50.07871.52 (8/10)1.15 (6/10)0.90 (5/10)1.21 (8/10)+0.0045+3.22
5M0.052e-50.07040.95 (6/10)0.71 (2/10)0.56 (0/10)1.94 (9/10)+0.0129+3.96
5M0.054e-50.05840.12 (0/10)0.09 (0/10)0.07 (0/10)+0.0248+9.96
5Mnone1e-50.07141.02 (5/10)0.77 (2/10)0.60 (0/10)1.81 (10/10)+0.0118+5.55
32M0.051e-50.13781.25 (9/10)0.97 (4/10)0.89 (3/10)1.18 (8/10)+0.0082+2.25
32M0.052e-50.12010.76 (1/10)0.59 (1/10)0.54 (0/10)1.95 (10/10)+0.0259+11.83
32M0.054e-50.0922-0.02 (0/10)-0.01 (0/10)-0.01 (0/10)+0.0538+13.80
32Mnone1e-50.13421.15 (8/10)0.89 (3/10)0.82 (1/10)1.28 (9/10)+0.0118+3.31

In chess the pair loss raises KAIKA's score against GRPO on each of its five training runs (Table 2 in the general response), with learning members by 0.1005 at T = 0.7 and 0.0544 at T = 0.3.

AuthorsOpenReview ↗

Reply to Reviewer Cxf5 (4 of 4)

W2a, too many variants. The submission presented four variants and an objective that its headline results did not train (submitted App. C, Eq. 15). §3.1 and §3.2 define two algorithms, KAIKA for games and DUEL for puzzles, as equations before any result, and App. Table 6 lists what each result trains.

Tables 1 to 3 test KAIKA (the table's first row), and Tables 4 and 5 test DUEL on chess puzzles. That objective is the variant of App. A.7 (Eq. 13), whose three chess arms are in the table's second row with one training run per arm of 100 steps, and no result in Section 4 trains it. The three engine-free KAIKA arms, which support neither method, left the paper (listed on p. 16 of the revision).

W2b, framework and subsets. The submission's sentence on this was wrong (submitted §7 and App. C). KAIKA, the learner of the headline chess results, is the method (§3.1), and it is above GRPO on every training run at each equal temperature from 0.1 to 0.9 (Table 1, Figure 2a). App. Figure 5 lists the components of each arm. Section 3.3 states what each pair loss compares and where it departs from all-pairs DPO on one policy's group (Eq. 7). With reward-gap weights, at the reference and for equal lengths, that pair gradient is a positive multiple of the GRPO gradient, and for rollouts that other members sampled it is not member k's on-policy gradient. In KAIKA a played row's pair gradient in the logits is not a rescaling of its GRPO gradient when a third move has positive probability, even at the reference (App. A.6). On novelty, see W1a and general response, point 5. App. F discusses population methods such as NeuPL.

Reviewer_C52kOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes KAIKA, an RL post training method, that maintains a population of policies, trains them with a pairwise term (Duel) in addition to the policy-gradient term. The pairwise term compares two sampled completions thorugh summed token log-ratio scores. This method is tested on two chess setups : puzzles with a 230M distilled model, and full games with chess language models evaluated heads to heads on a Stockfish ladder, with reported improvement over GRPO on puzzle accuracy and in play.

Critical strengths

S1: The experimental design sounds correct, same-seeded runs, along with ablations.It isolates correctly the fact that the introduced pairwise term itself has an impact on the current problem. A reproducibility repository is also provied, making claims checkable.

S2: The authors are honest about parts of the gain it cannot explain (Intro and conclusion), included a limitation section, relates relates Duel to DisCO, and summed aggregation to Dr. GRPO.

Critical weaknesses

W1: Very limited scope : The paper title claims RL from pairwise comparisons of continuations. This is only demostrated though on a very limited setup that is chess, making results, if they hold, look like a trick or a small fix on a single domain.

W2: Writing and presentation: The paper starts with an abstract where half of it is extensive results, no preliminaries or background section to introduce the setup and notations. Section 3 start opens directly with "KAIKA keeps K=8K=8" without any prior definitions or formalism, while GRPO is first formally defined at Eq.2 in page 4. Internal labels are used without definition ("sketch-B split", "F1/F2/F3", "KAIKA-H"), and "KAIKA" names four different learners (Table 2, Fig. 5).

W3: Main results are not shown to hold under matched comparators and evaluation settings: Chess gains seem to depend on sampling temperature and decoding. At T=0.3T=0.3, with greedy decoding on the Stockfish ladder, and at equal entropy, the confidence intervals include zero or parity (l 307,318, Table 18). The chess GRPO baseline is also not matched to KAIKA: it trains twice as many policy-gradient rows, about a third of them off-policy branch moves, with 1 advantage group vs 4 (Table 16). With the pair term turned off, copies of θ\theta still beat GRPO (0.5531, Table 4), so part of the gap comes from the training recipe and not from the pairwise term. Against GRPO trained for 600 steps the margin shrinks toward parity (l 2213), and on puzzles it also drops against GRPO with summed aggregation (Table 11).

W4: Incremental novelty, and the named components are not where the gain comes from: At constant pair weight, single-policy Duel is DisCO's J1J_1 with summed-token scores and summed aggregation with a constant normaliser is Dr. GRPO. Combining a group pairwise loss with GRPO was also already done by Lanchantin et al. The paper cites their GroupDPO but not this combination. The population has "no measurable effect" on puzzles (l 440) and collapses to copies of θ\theta in chess (l 1070). The pair weight EabE_{ab} is almost constant at the default β\beta, and making it active lowers the accuracy (Table 7). Part of the puzzle gain also comes from summed aggregation.

Questions for authors

With copies of θ\theta and the pair term off, KAIKA still beats GRPO (0.5531, Table 4). What differs between that arm and GRPO, and which of those differences explains the gap?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer C52k (1 of 4)

We thank the reviewer for the careful report, and answer the points in the order of the review. Points 1 to 7 and Tables 1 to 4 are in the general response. Every other table below is copied from the revised paper.

Summary

KAIKA's shared policy adds to its GRPO step a preference loss on engine-ordered pairs of moves at one position, on chess games at 230M (§3.1). DUEL trains its members on every passed-versus-failed pair of their shared attempts, with a retain term toward the base model. They have no GRPO term after founding (§3.2).

W1, scope

KAIKA trains the 230M chess model on engine credit, and Qwen2.5-1.5B-Instruct on game outcomes in Colonel Blotto and Nim (§4.4). DUEL trains 5M- and 32M-parameter models on a puzzle verifier (§4.3). With the game result as reward, DUEL also gains more than GRPO in chess (§4.4; general response, point 7). Every task here is a game or a puzzle with a verifiable reward, and tasks that are not games are the next test (§5.1).

W2a, background and definitions

GRPO was the submission's Eq. 2, on p. 4, and Section 3 began with "KAIKA keeps" (submitted l. 153). The revision defines the setting, the notation and GRPO (Eq. 1) in Preliminaries, §2.2. Figure 1, at the start of Section 3, says which results come from which algorithm, and the abstract says what each algorithm does before giving its results.

W2b, labels and names

The internal labels the reviewer lists are not in the revision. KAIKA and DUEL are defined separately in §3.1 and §3.2 (the submission used "KAIKA" for four learners in its Table 2 and Figure 5). App. Table 6 and Figure 5 list what each result trains. In App. B a plain "KAIKA" is the submitted configuration, as the appendix says at its start, and Table 3's caption says KAIKA as defined. DUEL on chess games uses KAIKA's shared policy and LoRA members (§4.4).

W3a, temperature and decoding

The submitted runs score 0.5268 against GRPO at T = 0.3, 0.5127 at equal entropy with GRPO cooled and 0.5234 with KAIKA heated (7/10 training runs each; submitted Table 1 and l. 318, now App. Table 8). KAIKA as defined leaves pairs whose branch move has probability zero out of the pair loss (general response, point 1). On the same ten seeds, against the same GRPO training runs, it scores 0.5915 at T = 0.3, and 0.5861 and 0.5945 at equal entropy. On every training run it is above GRPO at each equal temperature from 0.1 to 0.9 (Table 1, Figure 2a), at equal move entropy and with greedy play on 128 held-out openings (App. Table 8). With greedy play it scores 0.5278 (6/10) on the eight training openings (16 distinct games; Table 1).

App. Table 8, the rows against same-seed GRPO (32 games and 64 rows a step for both), seeds 1234 and 2 to 10, 1,024 games per match. In brackets, the training runs above 0.5. All kept pairs is the submitted configuration.

DecodingKAIKAall kept pairs
T = 0.70.6333 (10/10)0.6009 (10/10)
T = 0.30.5915 (10/10)0.5268 (7/10)
greedy, 128 held-out openings0.5747 (10/10)0.5088 (5/10)
equal entropy, GRPO cooled0.5861 (10/10)0.5127 (7/10)
equal entropy, KAIKA heated0.5945 (10/10)0.5234 (7/10)
each side at its best T0.5748 (10/10)0.5050 (6/10)

On the Stockfish ladder, replayed for KAIKA as defined, its greedy paired difference over GRPO is 52 Elo (9/10; App. Table 15(c)).

App. Table 15, mean fitted Elo on the Stockfish ladder at 12,000 nodes a move, ten training runs: panel (a), the submitted runs, and panel (c), KAIKA as defined with the same GRPO runs and the base replayed, at T = 0.7 and greedy (T = 0).

PanelTBaseKAIKAGRPOKAIKA minus GRPOabove 0
(a)0.7119513601283+7710/10
(a)0140014591448+115/10
(c)0.711201360.51290+70.510/10
(c)013751500.51448.5+529/10
AuthorsOpenReview ↗

Reply to Reviewer C52k (2 of 4)

W3b, rows and advantage groups

Both learners have 32 games and 64 rows a step (App. Table 11, read from the step records of the ten KAIKA training runs of Table 1). GRPO trains on all 64 rows, about 21 of them branch moves. KAIKA takes at most 8 rows from each slot, and its shared policy trains its GRPO term on the 32 played rows of all slots, with a mean of 23.3 pairs a step in its pair loss (submitted Table 16; general response, point 2). Each learner makes about 2,800 engine evaluations a step (median). Both standardise in one pooled group a step (§2.2), and "Adv. groups" in submitted Table 16 is "Row positions" in App. Table 9, the number of a slot's 8 row positions that hold a policy-gradient row.

The one-policy arm is GRPO on 32 played rows a step chosen by KAIKA's rule, with no pair loss and no members, on KAIKA's 32 games a step (App. B.8). Against it KAIKA scores 0.5796 at T = 0.7 (10/10) and 0.5407 at T = 0.3 (9/10; Table 1). KAIKA is above GRPO without branch rows on every training run. With credit standardised over the step or per position, and 32 games and 64 rows a step on both sides, it scores between 0.5707 and 0.6216 at T = 0.7 and 0.3 (5/5 each; Table 1, App. Table 8).

W3c, clone members with the pair loss off

Clone members (copies of the shared policy) with the pair loss off score 0.5531 in submitted Table 4, so part of the gap is the recipe. With the same members KAIKA as defined scores 0.5645 against GRPO with the pair loss off and 0.6317 with it on (5/5 each; Table 2). In Table 2 the pair loss raises the score against GRPO on each of its five training runs. One policy trained on played rows reproduces most of the recipe's part (Q1).

W3d, longer GRPO training

On the two training runs tested, the margin narrows against GRPO trained for 600 steps. At T = 0.7 the submitted configuration scores 0.5889 and 0.5488 against 600-step GRPO on seeds 1234 and 2. With both learners trained for 1,000 steps it scores above GRPO in 19 of 20 seed-temperature cells. GRPO is ahead on seed 2 at T = 0.3, and on seed 2 after 1,000 steps with both players greedy.

App. Table 16, the submitted runs (300 steps) against same-seed GRPO trained for n steps, 1,024 games, both players at T.

SeedTn = 300n = 375n = 600
12340.70.64600.60990.5889
12340.30.58010.55660.5327
20.70.56400.54790.5488
20.30.47070.48930.4614

App. Table 14(b), both learners trained for 1,000 steps, the submitted configuration's score against same-seed GRPO with both players at T.

SeedT = 0.00.10.20.30.40.50.60.70.80.9Mean
12340.54050.52340.52780.54790.57180.57760.60250.62550.64310.66210.5822
20.46580.57080.56300.57320.58200.61180.63330.63570.66360.68950.5989
AuthorsOpenReview ↗

Reply to Reviewer C52k (3 of 4)

W3e, the puzzle margin

The margin is smaller against GRPO with summed aggregation, where, with each method at its own setting, DUEL's gain over the base is 1.38 and 1.14 times that baseline's at 5M and 32M (Table 4). The general response, point 4, covers the raised coefficient and the one-policy pair loss.

W4a, DisCO and Dr. GRPO

The submission stated the DisCO relation at l. 133 to 135 and l. 224 and 225 for one policy and constant pair weights, with summed scores where DisCO's J1 has token-averaged ones (§2.1), and the training algorithms differ. Dr. GRPO makes two changes, a constant normaliser and no standardisation (§2.2). GRPO (sum) makes the first and keeps the standardisation. Dr. GRPO is a trained baseline in the revision, and DUEL's gain over the base is 1.21 and 1.18 times its gain at KL coefficient 0.05 at 5M and 32M (8/10 training runs each; App. Table 30).

App. Table 30, the rows at learning rate 1e-5 (ten training runs of 600 steps on chess puzzles, greedy pass@1). In brackets, the training runs on which the first-named arm is higher.

SizeKLDr. GRPO pass@1Dr. GRPO's multiple of GRPO (sum)DUEL's multiple of Dr. GRPODUEL minus Dr. GRPO
5M0.050.07871.15 (6/10)1.21 (8/10)+0.0045
5Mnone0.07140.77 (2/10)1.81 (10/10)+0.0118
32M0.050.13780.97 (4/10)1.18 (8/10)+0.0082
32Mnone0.13420.89 (3/10)1.28 (9/10)+0.0118

W4b, GroupDPO with a GRPO term

The submission cited the GroupDPO of Lanchantin et al. (submitted l. 137) without its GRPO term, and the revision cites that term in §2.1. Lanchantin et al. test adding a GRPO term with weight 0.01 or 0.001 to all-pairs training of one policy. Beside a full GRPO step, KAIKA pairs a played move with a branch move (general response, point 5). DUEL's pairs cross members, with no GRPO term after founding (§3.2).

W4c, the population

In chess the score against GRPO with the same games and rows rises with the number of slots, from 0.5807 at 2 slots to 0.6503 at 8, and is 0.6452 at 16 (Table 3). With the step held at GRPO's 32 games and 64 rows, the slots share the rows and the score stays between 0.6309 and 0.6503. In Blotto the exploitability of KAIKA's Nash (maximin) mixture against all 231 legal opponent allocations falls from 0.449 to 0.225 (general response, point 6).

The submission's word "collapses" (submitted l. 1070) was unclear. Two learning members choose different greedy moves at 13.8 per cent of 842 positions (App. B.11). We have not established what their own weights add over clone members in chess strength. The clone-member arm against the learning-member arm scores 0.5004 at T = 0.7 and 0.5026 at T = 0.3 (2/5 training runs above 0.5 each; §4.2, App. B.11). On puzzles four members with founding score above one policy at 5M and below it at 32M, on one shared pool of 16 rollouts per puzzle (§4.3; last two rows of Table 5 below). At 32M the members add no accuracy over one policy.

Table 5 of the revision. From one policy trained by the pair loss to DUEL on chess puzzles, ten training runs of 600 steps, each row adding its named change. A cell is the row's gain over the base as a multiple of token-mean GRPO's or GRPO (sum)'s, with the training runs on which the row is higher.

RowGRPO, 5MGRPO, 32MGRPO (sum), 5MGRPO (sum), 32M
One policy, per-token scores, constant weight1.26 (7/10)1.15 (7/10)0.95 (4/10)0.88 (1/10)
+ Fisher weight1.45 (9/10)1.25 (8/10)1.09 (6/10)0.96 (5/10)
+ summed scores1.58 (9/10)1.53 (10/10)1.19 (7/10)1.18 (9/10)
+ retain weight 0.05 (from 0.001)1.69 (10/10)1.60 (10/10)1.27 (8/10)1.24 (9/10)
+ four members with founding (DUEL)1.84 (9/10)1.48 (10/10)1.38 (8/10)1.14 (6/10)
Reviewer_KBArOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces KAIKA, a method that explores training policies through pairwise comparisons of subsequent trajectories originating from the same state. The core idea is to increase the probability of "better" trajectories relative to "worse" ones, while constructing training signals from samples shared across multiple policies—an approach intended to minimize interference during updates when trajectories happen to share correct steps. The primary contributions include the specific implementation of this training scheme, along with a systematic analysis of loss formulations, multi-policy sampling, and decoding configurations. The authors evaluate the method primarily through chess gameplay and puzzle-solving, supplemented by several smaller-scale game experiments. The results indicate that certain configurations outperform the paper’s GRPO baseline; however, this advantage diminishes when stronger baselines are used or when decoding randomness is controlled. Furthermore, the independent benefit of multi-policy collaboration remains unclear. Overall, while the paper offers valuable experimental observations, the validation of the method’s stability and generalizability remains insufficient.

Critical strengths

The core concept of "comparison signals" is well-motivated and supported by preliminary experimental evidence. Directly comparing the relative merits of different continuations from the same board state represents a clear and promising approach for model training. In ablation experiments conducted in the context of chess—with a temperature setting of 0.7—incorporating a pairwise loss consistently yielded performance gains across all five random seeds used for training.

Investigating the underlying sources of these performance improvements is of substantial value. The authors further examined key factors such as sequence-sum baselines and decoding entropy matching, finding that the observed advantages either diminished or disappeared entirely. Such analyses are crucial for determining whether the performance gains truly stem from the comparison mechanism itself, or rather from factors like loss normalization and sampling configurations.

Critical weaknesses

The gains over stronger baselines are not sufficiently robust. In experiments using 32M-parameter chess models, DUEL shows an average improvement of 1.27 percentage points over the original GRPO. However, when compared against a "GRPO+sum" variant—which incorporates sequence-level summation—this advantage shrinks to just 0.39 percentage points, with only three out of six random seeds yielding positive gains. This suggests that the loss normalization method accounts for a significant portion of the observed improvement; the paper fails to provide sufficient evidence that the newly introduced comparison mechanism offers consistent, incremental benefits.

Furthermore, the core claims regarding the population-based framework lack adequate validation. Although the paper emphasizes multi-policy collaboration as a key component, the performance difference between single-policy and multi-policy setups is marginal, and "learning" agents show no clear advantage over "cloned" (static) agents. Moreover, the authors acknowledge that training the full-scale population-based model has not yet succeeded in the context of chess. Consequently, the reported positive results stem primarily from simplified variants, which are insufficient to substantiate the value of the full framework and its underlying population mechanisms. This creates a substantial gap between the paper’s central claims and the empirical evidence provided.

Questions for authors

After aligning the sampling budget and regularization strength, can the performance gains of DUEL over GRPO+sum be consistently reproduced across multiple random seeds?

Can direct evidence be provided—within the context of a full-population objective—to demonstrate that multi-policy collaboration yields independent benefits compared to a single-policy approach or a collection of cloned policy members?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer KBAr (1 of 2)

We thank the reviewer for the careful report. The points are answered in the order of the review.

Summary, baselines and decoding

On puzzles the margin is smaller against the stronger baselines (general response, point 4). In chess KAIKA as defined is above GRPO on all ten training runs at each equal temperature from 0.1 to 0.9 (Table 1 in the general response, Figure 2a), at equal move entropy, with each side at its best temperature (App. Table 8) and with greedy play on held-out openings (Table 1). Tables 1, 4 and 5 use ten training runs each, and Section 4 covers four tasks and four models.

W1, summing at 32M

In the ten training runs of 600 steps, DUEL's gain at 32M is 1.14 times that of GRPO with summed aggregation (6/10 training runs; Table 4 in the general response), so summing explains much of DUEL's gain there. Greedy pass@1 at 32M is 0.0928 for the base, 0.1460 for DUEL, 0.1288 for GRPO and 0.1394 for GRPO (sum) (App. Table 28).

Table 5 of the revision. From one policy trained by the pair loss to DUEL (chess puzzles, ten training runs of 600 steps), each row adding its named change: the row's gain over the base as a multiple of token-mean GRPO's and of GRPO (sum)'s, with the training runs on which the row is higher.

GRPO, 5MGRPO, 32MGRPO (sum), 5MGRPO (sum), 32M
One policy, per-token scores, constant weight1.26 (7/10)1.15 (7/10)0.95 (4/10)0.88 (1/10)
+ Fisher weight1.45 (9/10)1.25 (8/10)1.09 (6/10)0.96 (5/10)
+ summed scores1.58 (9/10)1.53 (10/10)1.19 (7/10)1.18 (9/10)
+ retain weight 0.05 (from 0.001)1.69 (10/10)1.60 (10/10)1.27 (8/10)1.24 (9/10)
+ four members with founding (DUEL)1.84 (9/10)1.48 (10/10)1.38 (8/10)1.14 (6/10)

The pair loss on one policy adds to summing at 32M (Q1). At 5M its two multiples of GRPO (sum) are 1.19 and 0.99, with retain weight and KL coefficient both at 0.001 and both at 0.05 (App. Tables 28 and 29).

W2a, one policy against several

The score against GRPO rises with the number of slots, and in Blotto training lowers the exploitability of the Nash (maximin) mixture against all 231 legal opponent allocations (Q2).

W2b, the submitted objective

The submission's sentence on this was wrong (submitted §7, App. C). KAIKA, the learner of the headline chess results, is the method: a shared policy and seven members, each learning by its own Adam step (§3.1, Eq. 4), and it is above GRPO on every training run. The objective the submission set beside it is a separate variant with one training run per arm of 100 steps (App. A.7), and no result in Section 4 trains it.

Q1, aligned settings

App. Table 29, rows at equal coefficients. Mean difference in greedy pass@1 per training run over ten training runs of 600 steps, with its paired t and the training runs on which the row is higher. One policy is trained by the pair loss with summed scores and the Fisher weight, at retain weight 0.001 against GRPO (sum) and 0.05 against GRPO (sum), KL 0.05.

GRPO (sum), differencethigherGRPO (sum), KL 0.05, differencethigher
One policy, 5M+0.0037+1.457/10-0.0002-0.104/10
DUEL, 5M+0.0074+2.268/10+0.0020+0.746/10
One policy, 32M+0.0086+4.159/10+0.0070+3.408/10
DUEL, 32M+0.0066+1.916/10+0.0025+0.776/10

Both learners have 16 rollouts per puzzle, 4 drawn by each DUEL member (general response, point 2). The KL coefficient of GRPO (sum), 0.001, and DUEL's retain weight, 0.05, weigh different losses, so equal coefficients are not equal strength (general response, point 4). With the KL coefficient of GRPO (sum) raised to 0.05, DUEL's gain is 1.08 and 1.05 times that arm's (6/10 training runs each; Table 4 in the general response). One policy with the pair loss, the Fisher weight and summed scores is above GRPO (sum) at 32M at equal coefficients, with a gain of 1.18 times that arm's when retain weight and KL coefficient are both at 0.001 (9/10) and 1.14 times when both are at 0.05 (8/10; App. Tables 28 and 29). At 32M DUEL at half its learning rate gains 1.19 times as much as GRPO with summed aggregation at its own setting (10/10; Table 4). With each method at the searched setting with its highest test pass@1 (post hoc; App. Table 31), DUEL's multiple is 1.08 at both sizes (6/10 and 8/10 training runs).

AuthorsOpenReview ↗

Reply to Reviewer KBAr (2 of 2)

Q2, several policies

Against GRPO with the same games and rows, KAIKA's score rises from 0.5807 at 2 slots to 0.6503 at 8, and is 0.6452 at 16 (Table 3 in the general response).

  1. In Blotto, with the shared policy fixed and no pair loss, the exploitability of the Nash (maximin) mixture against all 231 legal opponent allocations falls from 0.449 to 0.225 (§4.4), while that of one policy alone rises from 0.647 to 0.841 (App. Table 42). The result is the population's alone (general response, point 6).

App. Table 42, its mean columns: exploitability of the Nash mixture over k of the 8 policies (the shared policy and seven members, fixed shared policy; probe; mean over every subset of size k and over seeds 309, 311 and 313) at step 199 and at step 0, and the same Nash solver over k random allocations with no learning (mean of 10,000 draws); lower is better.

kSubsetsMean, step 199Nash solver, k random allocationsStep 0
180.8411.0000.647
2280.6611.0000.559
3560.5080.9960.529
4700.4020.9600.511
5560.3320.8820.494
6280.2850.7890.479
780.2510.6890.464
810.2250.6020.449
  1. With the step held at 32 games and 64 rows for both learners, KAIKA's score stays between 0.6309 and 0.6503 (5/5 training runs each).

  2. Against the one-policy arm, which has no members and no pair loss, KAIKA scores 0.5796 (10/10) and 0.5407 (9/10; Table 1).

  3. With clone members (copies of the shared policy) the score rises the same way, from 0.5416 to 0.6754 against GRPO (Table 3 in the general response). Against GRPO at T = 0.7 the learning-member arm is 0.0186 higher than the clone-member arm (higher on 3/5 training runs), and at T = 0.3 the clone-member arm is higher (App. Table 50 below). In their direct duel the clone-member arm scores 0.5004 at T = 0.7 and 0.5026 at T = 0.3 (2/5 training runs above 0.5 each; §4.2, App. B.11). What the members' own weights add in chess strength is not established (general response, point 6).

App. Table 50. Learning members against clone members, KAIKA as defined, on Table 2's five training runs with the pair loss on (seeds 1234, 2 to 5): the shared policy's score over 1,024 games against the same seed's GRPO or the base, both players at T, and the learning-minus-clone difference by training seed, with SD and paired t over the five training runs and the training runs on which learning members score higher.

AgainstTLearningClone12342345MeanSDtHigher
GRPO0.70.65030.6317+0.0312-0.0010+0.0547-0.0049+0.0127+0.01860.0247+1.683/5
GRPO0.30.59800.6004+0.0337-0.0459-0.0225-0.0044+0.0273-0.00230.0335-0.162/5
the base0.70.74230.7421+0.0063+0.0024+0.0015-0.0186+0.0093+0.00020.0109+0.044/5
the base0.30.65270.6586+0.0205-0.0132-0.0283+0.0044-0.0127-0.00590.0187-0.702/5
  1. In one training run (seed 4; App. B.11, App. Table 49), on 4,000 held-out positions scored by Stockfish, some member has a better top move than the shared policy at 471. The per-position best of the eight, an upper bound chosen by the engine and not servable, loses 40.6 centipawns a move against 52.6 for the shared policy, and their average 51.0. The moves are there, and what is left is selecting among the policies and retaining them.

Thank you for your detailed response, but I am maintaining my score.

AuthorsOpenReview ↗

Reply to Reviewer KBAr's follow-up

Thank you for the reply.

"After aligning the sampling budget and regularization strength, can the performance gains of DUEL over GRPO+sum be consistently reproduced across multiple random seeds?"

Table 4: DUEL's mean gain over the base on chess puzzles (greedy pass@1; ten training runs of 600 steps) as a multiple of each GRPO arm's mean gain; in brackets, the training runs on which DUEL is higher. GRPO (sum) is GRPO with summed aggregation at its KL coefficient 0.001; the lr 5 × 10⁻⁶ column halves its learning rate (lr), the KL 0.05 column raises that coefficient to 0.05, and the lower rows halve DUEL's learning rate.

SizeGRPOGRPO (sum)GRPO (sum), lr 5 × 10⁻⁶GRPO (sum), KL 0.05
5M1.84 (9/10)1.38 (8/10)1.44 (8/10)1.08 (6/10)
32M1.48 (10/10)1.14 (6/10)1.04 (6/10)1.05 (6/10)
5M, DUEL lr 5 × 10⁻⁶1.66 (9/10)1.25 (8/10)1.29 (9/10)0.97 (4/10)
32M, DUEL lr 5 × 10⁻⁶1.54 (10/10)1.19 (10/10)1.08 (8/10)1.09 (6/10)

Halving the learning rate of DUEL and of GRPO (sum) at its own coefficient, to 5 × 10⁻⁶, raises both gains at 32M and lowers both at 5M. The submission had 3/6 at 32M against GRPO (sum).

"Can direct evidence be provided ... to demonstrate that multi-policy collaboration yields independent benefits compared to a single-policy approach or a collection of cloned policy members?"

In Blotto, with the shared policy fixed and no pair loss, each new member takes a founding step toward an allocation no policy played (App. D.1), then trains on game outcomes and is selected. The result is the population's alone. On the probe (each policy's saved allocation distribution), the mixture's exploitability falls from 0.449 to 0.225, while that of one policy alone rises (Figure 4). The exploitability of the Nash mixture over k of the 8 policies falls with each added policy on every training run, from 0.841 to 0.225, and is below the Nash solver over k random allocations at every k (App. Table 42).

In chess the clone-member arm against the learning-member arm (0.5004 at T = 0.7, 0.5026 at T = 0.3; 2/5 above 0.5 each; App. B.11) isolates the members' own weights. Against GRPO at 5M, each added part raises the gain. The multiple rises from 1.26 to 1.84 over the rows of Table 5, from one policy trained by the pair loss with per-token scores and a constant pair weight to DUEL. At 32M one policy at retain weight 0.05 is higher than DUEL, 1.60 against 1.48 over GRPO and 1.24 against 1.14 over GRPO (sum). That policy samples all 16 rollouts of a puzzle; each DUEL member samples 4.

On every run KAIKA is above 0.5 against GRPO at each equal temperature from 0.1 to 0.9 (Figure 2a), at equal move entropy and with each side at its best temperature (Table 1).

We would be grateful to know which of these concerns remains decisive.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1 of 4)

We are grateful to all the reviewers for their time and suggestions, we have made the following changes in response.

Overall changes

A. As per reviewers C52k and Cxf5, the present revision adds Preliminaries in §2.2 and defines KAIKA and DUEL separately in §3.1 and §3.2, before any result. The submission used "KAIKA" for four learners (submitted Table 2, Figure 5).

B. An overall algorithmic fix, the submitted configuration kept probability-zero branch move pairs in its pair loss. We have now left them out, and this is KAIKA as defined in §3.1 and App. A.4.

C. Reviewers C52k and KBAr have pointed out the sensitivity of the results (submitted Table 1, now App. Table 13) to both temperature and decoding. In an attempt to respond to this, we have retrained KAIKA with the fix in change B and scored 20 different models (10 GRPO and 10 KAIKA training runs, seeds 1234 and 2 to 10) with the aim of measuring this sensitivity in both of the models. Each same-seed pair plays 1,024 games at every temperature from 0.1 to 0.9 and with greedy play. The submitted runs scored 0.5268 against GRPO at T = 0.3 (7/10 training runs). KAIKA as defined is above GRPO on every training run at each equal temperature from 0.1 to 0.9 (Table 1, Figure 2a).

D. We added a new chess baseline, the one-policy arm, and five training runs that measure what the pair loss adds (Tables 1 and 2).

E. The submitted Table 16 counted "Adv. groups", which reviewer C52k raised. We now state one step budget for both learners (§4.1, App. Table 9) and we added Table 3 on the number of slots (§4.2).

F. As per reviewers KBAr, C52k and Cxf5, we added ten training runs of 600 steps on puzzles against GRPO with summed aggregation at three settings and against Dr. GRPO (Tables 4, 5 and App. Table 30). The submission had Tables 6 and 11.

G. We removed the three engine-free KAIKA arms and the submitted App. H archive (both listed on p. 16 of the revision), and Figure 4 now plots Blotto exploitability against the number of KAIKA's eight policies in the Nash (maximin) mixture.

Table 1 of the revision. KAIKA as defined against same-seed GRPO on chess games, ten training runs, 1,024 games per match. In brackets, the training runs on which KAIKA is higher. Higher is better.

T = 0.7T = 0.3
KAIKA against GRPO, score0.6333 (10/10)0.5915 (10/10)
KAIKA, Elo gain over the base+180+107
GRPO, Elo gain over the base+84+50
KAIKA's gain as a multiple of GRPO's (from unrounded gains)2.14 (10/10)2.13 (10/10)
Against the one-policy arm (32 games, 32 rows a step), score0.5796 (10/10)0.5407 (9/10)
Against GRPO without branch rows, credit standardised over the step (seeds 1234, 2 to 5)0.5943 (5/5)0.5707 (5/5)
The same, credit standardised per position (seeds 1234, 2 to 5)0.6216 (5/5)0.6058 (5/5)

The table's last two rows: at equal move entropy the score is 0.5861 with GRPO cooled and 0.5945 with KAIKA heated (10/10 each). With each side at its best temperature it is 0.5748, and with greedy play on 128 held-out openings 0.5747 (10/10 each).

Table 1, lower part. Both players at temperature T (T = 0 is greedy play on the eight training openings, 16 distinct games per match).

T00.10.20.30.40.50.60.70.80.9
score0.52780.58930.58530.59150.58970.60090.61330.63330.62980.6578
training runs above 0.56/1010/1010/1010/1010/1010/1010/1010/1010/1010/10

We list the terms on p. 15 and define gains, multiples and the k/n counts on p. 16, and the table on pp. 15 and 16 gives the numbering of the submitted version beside that of the revision.

1. KAIKA as defined

KAIKA is a shared policy and seven LoRA members that play one another (§3.1). The shared policy takes a GRPO step on the played moves of all eight slots and adds a pair loss, DPO's logistic loss, on two moves at one position that the engine orders, while each member takes its own Adam step and the lowest-rated members are replaced. A branch move comes from the mover's sampling distribution, and when that distribution holds only the played move the trainer takes another legal move, of probability zero. KAIKA leaves such a pair out of its pair loss, and over its ten training runs these pairs are 26.9 per cent of the kept pairs (App. A.4). The submitted runs kept those pairs and changed nothing else, and in one such training run (seed 1234) they are 33.3 per cent of the kept pairs. App. Table 8 gives both versions, KAIKA and the submitted configuration, on the same ten seeds, against the same GRPO training runs.

AuthorsOpenReview ↗

General response (2 of 4)

2. One budget for both learners

Both learners have one budget in the headline comparisons, 32 games and 64 rows a chess step (§4.1) and 16 rollouts per puzzle, 4 drawn by each DUEL member (§4.3). Each of KAIKA's slots contributes at most 8 of the 64 rows (App. Table 11). GRPO trains on all 64, about 21 of them branch moves, and KAIKA's shared policy trains its GRPO term on the 32 played rows, with a branch move entering only its pair loss (§4.1). Table 3 grows KAIKA's step with the number of slots (point 6).

3. Chess games

KAIKA scores 0.6333 against GRPO at the training temperature T = 0.7 and 0.5915 at T = 0.3 (10/10 training runs each), and its Elo gain over the base is 2.14 and 2.13 times GRPO's (Table 1). On every training run it is above GRPO at each equal temperature from 0.1 to 0.9, at equal move entropy and with each side at its best temperature (Table 1, Figure 2a, App. Table 8). With greedy play it scores 0.5278 (6/10) on the eight training openings (16 distinct games) and 0.5747 (10/10) on 128 held-out openings (Table 1). Against the one-policy arm, which is GRPO on 32 played rows with no pair loss and no members, it scores 0.5796 (10/10) and 0.5407 (9/10; Table 1). In Table 2 the pair loss raises the score against GRPO on each of its five training runs, with learning members by 0.1005 at T = 0.7 and 0.0544 at T = 0.3 (from unrounded scores).

Table 2 of the revision. KAIKA's arms on seeds 1234 and 2 to 5 of Table 1: the shared policy's score against same-seed GRPO over 1,024 games, both players at T. In brackets, training runs above 0.5. Clone members are players with their own games and no gradient step.

MembersPair lossT = 0.7T = 0.3
learningon0.6503 (5/5)0.5980 (5/5)
learningoff0.5498 (5/5)0.5437 (5/5)
cloneon0.6317 (5/5)0.6004 (5/5)
cloneoff0.5645 (5/5)0.5410 (5/5)
one-policy arm, 32 rows0.5606 (5/5)0.5617 (5/5)

4. Puzzles and stronger GRPO baselines

A multiple is DUEL's gain over the base divided by the GRPO arm's. On chess puzzles DUEL's gain is 1.84 times token-mean GRPO's at 5M and 1.48 times at 32M (Table 4). Against GRPO with summed aggregation, GRPO (sum), with each method at its own setting, the multiple is 1.38 and 1.14, and DUEL is higher on 9, 10, 8 and 6 of the ten training runs. At 32M DUEL is 0.0066 above GRPO (sum) in pass@1, which is 14 per cent of that arm's gain (App. Table 29).

Table 4 of the revision. DUEL's mean gain over the base on chess puzzles (greedy pass@1, ten training runs of 600 steps) as a multiple of each GRPO arm's mean gain. In brackets, the training runs on which DUEL is higher. GRPO (sum) has KL coefficient 0.001.

SizeGRPO (token-mean)GRPO (sum)GRPO (sum), lr 5e-6GRPO (sum), KL 0.05
DUEL, 5M1.84 (9/10)1.38 (8/10)1.44 (8/10)1.08 (6/10)
DUEL, 32M1.48 (10/10)1.14 (6/10)1.04 (6/10)1.05 (6/10)
DUEL at learning rate 5e-6, 5M1.66 (9/10)1.25 (8/10)1.29 (9/10)0.97 (4/10)
DUEL at learning rate 5e-6, 32M1.54 (10/10)1.19 (10/10)1.08 (8/10)1.09 (6/10)

Reviewer KBAr asked for aligned settings. The KL coefficient of GRPO (sum), 0.001, and DUEL's retain weight, 0.05, weigh different losses, so equal coefficients are not equal strength (§4.3). With the KL coefficient of GRPO (sum) raised to 0.05, 50 times its own, the multiple is 1.08 and 1.05 (6/10 training runs each), and the paired t is 0.74 and 0.77 (App. Table 29). One policy with the pair loss, the Fisher weight and summed scores is above GRPO (sum) at 32M at equal coefficients, with a gain of 1.18 times that arm's when retain weight and KL coefficient are both at 0.001 (9/10) and 1.14 times when both are at 0.05 (8/10). At 5M the same two multiples are 1.19 and 0.99 (App. Tables 28 and 29). Against Dr. GRPO at KL 0.05 DUEL's multiple is 1.21 and 1.18 (8/10 each; App. Table 30). With each method at the searched setting with its highest test pass@1 (post hoc; App. Table 31), DUEL's multiple is 1.08 at both sizes (6/10 and 8/10 training runs).

AuthorsOpenReview ↗

General response (3 of 4)

5. DisCO and Lanchantin et al.

The DisCO identity covers one restricted contrast term. It is stated at submitted l. 133 to 135 and l. 224 and 225 for one policy and constant pair weights, with summed scores where DisCO's J1 has token-averaged ones (§2.1), so it does not identify the training algorithms. That restricted case gains 1.15 and 1.13 times GRPO (sum) at retain weight 0.001 (7/10 training runs each; App. Table 28). As defined in §3.2, DUEL adds a Fisher pair weight, a retain term on each member's own rollouts, and members, each founded by one GRPO step from one parent, that pool their attempts. In Table 5, with per-token scores, the Fisher weight raises one policy's multiple at 5M from 1.26 to 1.45. At retain weight 0.05 the restricted case reaches 0.0814 and 0.1496, against DUEL's 0.0832 and 0.1460, and with the Fisher weight it gives nearly the same, 0.0811 and 0.1504 (App. Table 28).

As §2.1 describes, Lanchantin et al. train one policy on every correct-by-incorrect pair of its responses to a prompt, and they test adding a GRPO term with weight 0.01 or 0.001 on the same responses. Beside a full-weight GRPO step, KAIKA applies the DPO logistic loss to pairs from its population's games, each pair being a player's played move against an unplayed branch move at that position, ordered by the engine (§3.1). DUEL's pairs cross its members, with no GRPO term after founding (§3.2).

6. The population

KAIKA's score against GRPO rises with the number of slots (Table 3), and in Colonel Blotto training lowers the exploitability of KAIKA's Nash (maximin) mixture from 0.449 to 0.225 (§4.4). In Table 3 we train KAIKA as defined at 8 rows per slot, so the step grows from 8 games and 16 rows at 2 slots to 64 and 128 at 16. Against GRPO with the same games and rows, KAIKA's score rises from 0.5807 to 0.6503 at 8 slots and is 0.6452 at 16 (5/5 training runs each). GRPO trained on those games and rows scores 0.4725 at 2 slots and 0.5271 at 16 against GRPO on 32 games and 64 rows, where KAIKA scores 0.5599 and 0.6604 (Table 3). With KAIKA and GRPO both on 32 games and 64 rows a step, where the slots share the rows, the score stays between 0.6309 and 0.6503 (5/5 training runs each; Table 3).

Table 3 of the revision. Score against GRPO at T = 0.7 with 2 to 16 slots, mean of five training runs (seeds 1234, 2 to 5) of KAIKA as defined. In brackets, those above 0.5. GRPO (32/64) is Table 1's GRPO on 32 games and 64 rows a step.

Score ofagainst same-seed2 slots4 slots8 slots16 slots
8 rows per slot, so games/rows a step8/1616/3232/6464/128
KAIKAGRPO (32/64)0.5599 (5/5)0.5892 (5/5)0.6503 (5/5)0.6604 (5/5)
KAIKA, clone membersGRPO (32/64)0.5416 (5/5)0.5772 (5/5)0.6317 (5/5)0.6754 (5/5)
GRPO, same games/rowsGRPO (32/64)0.4725 (1/5)0.5113 (3/5)that GRPO itself0.5271 (5/5)
KAIKAGRPO, same games/rows0.5807 (5/5)0.5868 (5/5)0.6503 (5/5)0.6452 (5/5)
Step held at 32 games and 64 rows, so rows per slot321684
KAIKAGRPO (32/64)0.6309 (5/5)0.6386 (5/5)0.6503 (5/5)0.6320 (5/5)

Clone members (copies of the shared policy) still play their own games and rows, and at 8 rows per slot their score rises the same way, from 0.5416 to 0.6754 against GRPO on 32 games and 64 rows. The clone-member arm against the learning-member arm isolates the members' own weights, and it gives 0.5004 at T = 0.7 and 0.5026 at T = 0.3 (2/5 training runs above 0.5 each; §4.2, App. B.11). That part is not established in chess strength. On puzzles, four members with founding raise one policy's multiple of token-mean GRPO's gain at 5M from 1.69 to 1.84 and lower it at 32M from 1.60 to 1.48 (Table 5). Since they share one pool of 16 rollouts per puzzle, they add no data (§4.3).

AuthorsOpenReview ↗

General response (4 of 4)

6. The population (continued)

In Blotto KAIKA serves its eight policies, the shared policy and seven members, as the Nash (maximin) mixture against all 231 legal opponent allocations. The exploitability of one policy alone rises from 0.647 to 0.841, and each added policy lowers the mixture's (Figure 4, App. Table 42). The seven members train on a fixed shared policy with no pair loss, so the result is the population's alone. Each new member takes a founding step toward an allocation no policy played (App. D.1), then trains on game outcomes and is selected. In one training run each, the mixture reaches 0.224 with game-outcome steps alone and 0.197 with the founding step alone, and it stays at 0.534 with neither (App. Table 51). That table's clone-member row, where every slot is the base model, was not run and gives the base's 0.550.

When we run the same Nash solver over eight random allocations, with no learning, it averages 0.602, against 0.225 for the trained policies (App. Table 42). Over 48 random allocations it has a median of 0.051, below KAIKA's snapshot archive at 0.082 to 0.155 (App. Table 41). On the evaluation instrument, which is the allocations played in evaluation games, KAIKA's mixture has exploitability 0.194 to 0.228 over three training runs. One policy trained by GRPO, with 64 games a step against a fixed opponent and 64 rows, has 0.651 to 0.888, and when we give GRPO the same Nash solver over its three policies it has 0.532 (App. Table 41). With a trained shared policy KAIKA's served mixture has 0.344 to 0.372 on that instrument, and the same Nash solver over the best 8 of 48 random allocations has a median of 0.143 (App. Table 41).

7. Nim and the engine

In Nim KAIKA scores 0.5656 at T = 0.7 and 0.5719 at T = 0.3 against GRPO (3/5 training runs each; App. Table 44). As defined in §3, DUEL calls no engine, and its label is the verifier's pass or fail, or the result of a chess game. We also trained DUEL on chess games with the game result as both learners' reward. Engine-free GRPO learns from the same 32 game results a step on 128 rows, and against it DUEL scores 0.5557 at T = 0.7 and 0.5543 at T = 0.3 (5/5 training runs each). Its gain over the base at T = 0.7 is 2.06 times engine-free GRPO's (4/5; App. Table 23). Against GRPO with engine credit it scores 0.4858 (2/5) and 0.4940 (3/5; §4.4). Finally, the DUEL runs that call an engine, one training run each, are the "Engine credit" columns of App. Table 24.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.