- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Language models can state a distribution they are asked to sample from yet fail to sample from it, and the error lies in their output probabilities, so prompting and decoding fixes only partly help. The paper trains this ability with GRPO, whose standard reward fails here: a single group-level score makes all centered advantages zero. Their fix is the witness advantage — a per-rollout reward derived from the MMD witness function, computed in closed form from the group's outcome counts — which rewards outcomes the group under-produces and penalizes over-produced ones, and provably makes the expected update follow the negative gradient of the distance to the target.
Critical strengths
Practical value at negligible cost. A few GPU-hours on one GPU markedly improve sampling fidelity on unseen families and real-world opinion tasks while preserving general capabilities, where supervised cross-entropy achieves better fit only by destroying them.
Critical weaknesses
(1) Single small model, narrow scope. Only Qwen2.5-1.5B is ever trained; larger models are evaluated but not trained.
(2) AI-authored proofs and figures, unverified in depth. Per the authors' own AI Use Statement, proofs, code, figures, and prose are AI-assisted; I verified the main-text derivations but not every appendix proof. Combined with default-tool figures and load-bearing details deferred to the appendix, the submission carries a real verification burden.
Questions for authors
-
Cross-entropy transfers better to unseen parameters at a known capability cost — is that damage method-inherent or just a budget artifact? Please compare at matched budget.
-
Since GAPO is implementable in your harness, can you report it empirically instead of only criticizing it analytically?
-
Please confirm the AI-assisted proofs in Appendices C–E were independently human-verified, and by whom.
-
Refer to "Critical Weaknesses"
8 comments
Q1. "Cross-entropy transfers better to unseen parameters at a known capability cost — is that damage method-inherent or just a budget artifact? Please compare at matched budget."
We ran the matched-budget comparison. It shows that cross-entropy's capability loss is not a budget artifact. The loss follows the learning rate, and on the larger models it remains even at the best learning rate (, Page 8, and , Page 39).
First, the loss starts early. In the submitted run, IFEval has already dropped after 10 of the 110 training steps. At that point, cross-entropy is still twice as far from the target distributions as the witness advantage (, Page 39). Therefore, stopping training earlier does not prevent the loss.
Second, the loss depends on the learning rate. At the lower learning rate used for the witness advantage and with matched compute, cross-entropy keeps its MMLU and IFEval scores on Qwen2.5-1.5B (, Page 8).
However, a low learning rate does not fix cross-entropy in general. On the three new models, we tuned the learning rate of both methods in the same way. For each method, we picked the setting that fit the target distributions best while keeping MMLU intact on a validation split (, Page 38). Even with this tuning, cross-entropy significantly lowered MMLU or IFEval on every new model. In contrast, the witness advantage kept both scores on all four models (, Page 39).
The table shows the change from the untrained model in percentage points. A dagger marks a statistically significant drop.
| Model | Cross-entropy, MMLU | Cross-entropy, IFEval | Witness (ours), MMLU | Witness (ours), IFEval |
|---|---|---|---|---|
| Qwen2.5-1.5B | −0.5 | −1.4 | 0.0 | −1.2 |
| Llama-3.2-3B | −1.6† | −1.7 | −0.2 | −1.2 |
| Qwen3-4B | −1.6† | −1.1 | −0.3 | +0.4 |
| Qwen3.5-4B | +0.5 | −5.3† | +0.6 | −1.3 |
Moreover, cross-entropy is sensitive to its learning rate. On Qwen3.5-4B, its accuracy on the MMLU validation split falls from 0.741 to 0.413 over the three candidate rates, while the witness advantage stays between 0.732 and 0.743 (, Page 38).
Finally, the two methods need different supervision. Cross-entropy needs every outcome of a target listed with one fixed response string (, Page 7, and , Page 33). The witness advantage only scores the model's own samples, and it matches cross-entropy on unseen families on Qwen2.5-1.5B and Llama-3.2-3B (, Page 8, and , Page 40).
Q2. "Since GAPO is implementable in your harness, can you report it empirically instead of only criticizing it analytically?"
Yes. We implemented GAPO as published, and the witness advantage outperforms it on all three evaluation sets (, Page 8, and , Page 39). Both methods train Qwen2.5-1.5B with the same prompts, group size, and number of steps, with three seeds (, Page 39). The table shows the sampling error.
| Method | Training targets | Unseen parameters | Unseen families |
|---|---|---|---|
| Untrained model | 0.460 | 0.477 | 0.588 |
| GAPO | 0.253 | 0.417 | 0.538 |
| Witness advantage (ours) | 0.115 | 0.169 | 0.380 |
GAPO removes 13% of the error on unseen parameters and 8% on unseen families, and the witness advantage removes 65% and 35%.
Thank you for the response. I’m not fully convinced that my main concerns have been addressed, and I’ll keep my original score.
We thank the reviewer for the careful review and for verifying the main-text derivations. We are glad that you find the method of "practical value at negligible cost", with a few GPU-hours that "markedly improve sampling fidelity on unseen families and real-world opinion tasks while preserving general capabilities". We respond to the weaknesses (W) and the questions (Q) below.
W1 and Q4. "Single small model, narrow scope. Only Qwen2.5-1.5B is ever trained; larger models are evaluated but not trained."
Following your comment, we trained three more models, and our findings generalize to all of them (Table 2 and Sec. 4.3, Page 9). The new models are Llama-3.2-3B-Instruct, Qwen3-4B-Instruct, and Qwen3.5-4B. We use the settings of the submission and three seeds per model, and only the learning rate is chosen per model (App. H.1, Page 38).
The table shows each score before and after training. Error is the median excess TV, where lower is better. Trained values are means over three seeds.
Training lowers the error of every model on all three sets. General capability metrics (MMLU and IFEval) also barely change on any of the four models.
Furthermore, we extended our experiments to mixed-strategy games. Their prompts state no distribution, and we did not train on them. Our method raises the score of all four models in both games (Sec. 4.4, Page 10).
W2 and Q3. "[...] Combined with default-tool figures and load-bearing details deferred to the appendix, the submission carries a real verification burden. [...] Please confirm the AI-assisted proofs in Appendices C–E were independently human-verified, and by whom."
Yes. The authors verified every proof in Appendices B to E line by line. We clarified this in our AI use statement (AI Use Statement, Page 11). Moreover, our code checks the formal results numerically. It compares the expected update with the automatic gradient of the squared MMD, and it enumerates every possible group for small group sizes (App. C.4, Page 24, and App. D.3, Page 28).
We also agree that a reader should be able to verify the main claims without the appendix. Therefore, the revision moves the following into the main text:
All changes are marked in blue. We also checked every figure for legibility at its printed size, and we will gladly revise any figure that remains unclear.