acceptodds
Under review as a conference paper at ICLR 2027

Sample What You Say: Aligning Language Models to Sample the Distributions They State

Abstract

Language models are increasingly used to sample from a specified distribution, for instance, to simulate survey respondents or generate synthetic data. Instruction-tuned models can state such a distribution correctly and still fail to sample from it. Prompting and changes to decoding reduce this mismatch only partly, which motivates training with policy optimization. Group relative policy optimization (GRPO) is a natural fit for this problem because it already samples a group of rollouts per prompt, and the group's empirical distribution can be compared with the target. However, scoring the group as a whole gives every rollout the same reward. Group-relative centering then sets all advantages to zero, and the model receives no learning signal. To give each rollout its own signal, we introduce the *witness advantage*, a per-rollout advantage derived from maximum mean discrepancy (MMD). It trains a model to match a target distribution over a finite set of outcomes. The MMD between the model's distribution and the target has a witness function that measures how over- or under-produced each outcome is. Each rollout's advantage estimates the negative witness at its outcome, so a rollout is rewarded for an outcome the group under-produces and penalized for one it over-produces. The witness advantage is computed in closed form from the group's outcome counts, and we use it as the reward in GRPO. On unseen target distributions, training with the witness advantage substantially reduces the total variation distance to the target while largely preserving the model's general capabilities.

Primary area:
generative models
Keywords:
language models, sampling, distribution matching, reinforcement learning, GRPO, maximum mean discrepancy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BN2SOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Language models can state a distribution they are asked to sample from yet fail to sample from it, and the error lies in their output probabilities, so prompting and decoding fixes only partly help. The paper trains this ability with GRPO, whose standard reward fails here: a single group-level score makes all centered advantages zero. Their fix is the witness advantage — a per-rollout reward derived from the MMD witness function, computed in closed form from the group's outcome counts — which rewards outcomes the group under-produces and penalizes over-produced ones, and provably makes the expected update follow the negative gradient of the distance to the target.

Critical strengths

Practical value at negligible cost. A few GPU-hours on one GPU markedly improve sampling fidelity on unseen families and real-world opinion tasks while preserving general capabilities, where supervised cross-entropy achieves better fit only by destroying them.

Critical weaknesses

(1) Single small model, narrow scope. Only Qwen2.5-1.5B is ever trained; larger models are evaluated but not trained.

(2) AI-authored proofs and figures, unverified in depth. Per the authors' own AI Use Statement, proofs, code, figures, and prose are AI-assisted; I verified the main-text derivations but not every appendix proof. Combined with default-tool figures and load-bearing details deferred to the appendix, the submission carries a real verification burden.

Questions for authors
  1. Cross-entropy transfers better to unseen parameters at a known capability cost — is that damage method-inherent or just a budget artifact? Please compare at matched budget.

  2. Since GAPO is implementable in your harness, can you report it empirically instead of only criticizing it analytically?

  3. Please confirm the AI-assisted proofs in Appendices C–E were independently human-verified, and by whom.

  4. Refer to "Critical Weaknesses"

8 comments

We thank the reviewer for the careful review and for verifying the main-text derivations. We are glad that you find the method of "practical value at negligible cost", with a few GPU-hours that "markedly improve sampling fidelity on unseen families and real-world opinion tasks while preserving general capabilities". We respond to the weaknesses (W) and the questions (Q) below.


W1 and Q4. "Single small model, narrow scope. Only Qwen2.5-1.5B is ever trained; larger models are evaluated but not trained."

Following your comment, we trained three more models, and our findings generalize to all of them (Table 2\textrm{\color{blue}Table 2} and Sec. 4.3\textrm{\color{blue}Sec. 4.3}, Page 9). The new models are Llama-3.2-3B-Instruct, Qwen3-4B-Instruct, and Qwen3.5-4B. We use the settings of the submission and three seeds per model, and only the learning rate is chosen per model (App. H.1\textrm{\color{blue}App. H.1}, Page 38).

The table shows each score before and after training. Error is the median excess TV, where lower is better. Trained values are means over three seeds.

ModelError, training targetsError, unseen parametersError, unseen familiesMMLUIFEval
Qwen2.5-1.5B0.460 → 0.1150.477 → 0.1690.588 → 0.3800.584 → 0.5840.519 → 0.507
Llama-3.2-3B0.770 → 0.1220.821 → 0.1800.607 → 0.3920.589 → 0.5860.789 → 0.777
Qwen3-4B0.737 → 0.1480.841 → 0.2220.803 → 0.3570.700 → 0.6960.882 → 0.886
Qwen3.5-4B0.442 → 0.0780.579 → 0.0990.484 → 0.4020.731 → 0.7370.880 → 0.867

Training lowers the error of every model on all three sets. General capability metrics (MMLU and IFEval) also barely change on any of the four models.

Furthermore, we extended our experiments to mixed-strategy games. Their prompts state no distribution, and we did not train on them. Our method raises the score of all four models in both games (Sec. 4.4\textrm{\color{blue}Sec. 4.4}, Page 10).


W2 and Q3. "[...] Combined with default-tool figures and load-bearing details deferred to the appendix, the submission carries a real verification burden. [...] Please confirm the AI-assisted proofs in Appendices C–E were independently human-verified, and by whom."

Yes. The authors verified every proof in Appendices B to E line by line. We clarified this in our AI use statement (AI Use Statement\textrm{\color{blue}AI Use Statement}, Page 11). Moreover, our code checks the formal results numerically. It compares the expected update with the automatic gradient of the squared MMD, and it enumerates every possible group for small group sizes (App. C.4\textrm{\color{blue}App. C.4}, Page 24, and App. D.3\textrm{\color{blue}App. D.3}, Page 28).

We also agree that a reader should be able to verify the main claims without the appendix. Therefore, the revision moves the following into the main text:

  • A proof sketch of Proposition 1 (Sec. 3.2\textrm{\color{blue}Sec. 3.2}, Page 5).
  • An example prompt that asks the model to sample from a stated distribution, and how we read the sampled outcome from its answer (Sec. 4.1\textrm{\color{blue}Sec. 4.1}, Page 7).
  • The comparisons with GAPO and with cross-entropy (Table 1\textrm{\color{blue}Table 1}, Page 8).
  • The results on other models (Sec. 4.3\textrm{\color{blue}Sec. 4.3}, Page 9).

All changes are marked in blue. We also checked every figure for legibility at its printed size, and we will gladly revise any figure that remains unclear.

Q1. "Cross-entropy transfers better to unseen parameters at a known capability cost — is that damage method-inherent or just a budget artifact? Please compare at matched budget."

We ran the matched-budget comparison. It shows that cross-entropy's capability loss is not a budget artifact. The loss follows the learning rate, and on the larger models it remains even at the best learning rate (Sec. 4.2\textrm{\color{blue}Sec. 4.2}, Page 8, and App. H.2\textrm{\color{blue}App. H.2}, Page 39).

First, the loss starts early. In the submitted run, IFEval has already dropped after 10 of the 110 training steps. At that point, cross-entropy is still twice as far from the target distributions as the witness advantage (Table 11\textrm{\color{blue}Table 11}, Page 39). Therefore, stopping training earlier does not prevent the loss.

Second, the loss depends on the learning rate. At the lower learning rate used for the witness advantage and with matched compute, cross-entropy keeps its MMLU and IFEval scores on Qwen2.5-1.5B (Table 1\textrm{\color{blue}Table 1}, Page 8).

However, a low learning rate does not fix cross-entropy in general. On the three new models, we tuned the learning rate of both methods in the same way. For each method, we picked the setting that fit the target distributions best while keeping MMLU intact on a validation split (App. H.1\textrm{\color{blue}App. H.1}, Page 38). Even with this tuning, cross-entropy significantly lowered MMLU or IFEval on every new model. In contrast, the witness advantage kept both scores on all four models (Table 13\textrm{\color{blue}Table 13}, Page 39).

The table shows the change from the untrained model in percentage points. A dagger marks a statistically significant drop.

ModelCross-entropy, MMLUCross-entropy, IFEvalWitness (ours), MMLUWitness (ours), IFEval
Qwen2.5-1.5B−0.5−1.40.0−1.2
Llama-3.2-3B−1.6†−1.7−0.2−1.2
Qwen3-4B−1.6†−1.1−0.3+0.4
Qwen3.5-4B+0.5−5.3†+0.6−1.3

Moreover, cross-entropy is sensitive to its learning rate. On Qwen3.5-4B, its accuracy on the MMLU validation split falls from 0.741 to 0.413 over the three candidate rates, while the witness advantage stays between 0.732 and 0.743 (Table 10\textrm{\color{blue}Table 10}, Page 38).

Finally, the two methods need different supervision. Cross-entropy needs every outcome of a target listed with one fixed response string (Sec. 4.1\textrm{\color{blue}Sec. 4.1}, Page 7, and App. F.4\textrm{\color{blue}App. F.4}, Page 33). The witness advantage only scores the model's own samples, and it matches cross-entropy on unseen families on Qwen2.5-1.5B and Llama-3.2-3B (Table 1\textrm{\color{blue}Table 1}, Page 8, and Table 14\textrm{\color{blue}Table 14}, Page 40).


Q2. "Since GAPO is implementable in your harness, can you report it empirically instead of only criticizing it analytically?"

Yes. We implemented GAPO as published, and the witness advantage outperforms it on all three evaluation sets (Table 1\textrm{\color{blue}Table 1}, Page 8, and Table 12\textrm{\color{blue}Table 12}, Page 39). Both methods train Qwen2.5-1.5B with the same prompts, group size, and number of steps, with three seeds (App. H.3\textrm{\color{blue}App. H.3}, Page 39). The table shows the sampling error.

MethodTraining targetsUnseen parametersUnseen families
Untrained model0.4600.4770.588
GAPO0.2530.4170.538
Witness advantage (ours)0.1150.1690.380

GAPO removes 13% of the error on unseen parameters and 8% on unseen families, and the witness advantage removes 65% and 35%.

Thank you for the response. I’m not fully convinced that my main concerns have been addressed, and I’ll keep my original score.

Reviewer_2S6mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper motivates that sampling from the distribution a model itself states is important, as LLMs are used to simulate survey respondents or generate synthetic data. It designs a witness advantage for GRPO and shows experimental results against training-based and training-free methods.

Critical strengths
  1. The reward design is straightforward and intuitive, and the subsequent experimental results are convincing on current setups.
  2. The paper provides theoretical insight into the proposed reward, showing that its expected update follows the negative gradient of MMD 2^2 and explaining why simpler alternatives (the group-scalar reward and the sign witness) fail.
  3. The writing is clear, and the paper is easy to follow.
Critical weaknesses
  1. The experimental settings do not strongly support the motivation: most targets are stated explicitly in the prompt and could be sampled exactly by a random number generator or a tool call. Evaluating on scenarios that genuinely require model-generated randomness, such as mixed-strategy games [1], would better demonstrate the method’s practical value.
  2. Comparisons with the closest prior methods are missing. GAPO [2] and the calibration fine-tuning of [3] address similar problems but are not evaluated empirically. I suggest the authors provide a discussion about these related works in detail.
  3. The current experimental setups are restricted to older family of open-source model (Qwen2.5). The claim the authors provided in the paper could be strengthened if they could test on latest open-source model with stronger reasoning capabilities such as Qwen3.5.

[1] West, Peter, and Christopher Potts. "Base models beat aligned models at randomness and creativity." arXiv preprint arXiv:2505.00047 (2025).

[2] Anschel, Oron, et al. "Group-aware reinforcement learning for output diversity in large language models." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

[3] Baldelli, Davide, et al. "LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents." arXiv preprint arXiv:2601.06973 (2026).

Questions for authors

Please see the weakness section above.

3 comments

We thank the reviewer for the thoughtful feedback. We are glad that you find the reward design "straightforward and intuitive", the experimental results "convincing", and the paper "easy to follow". We respond to the weaknesses (W) below.


W1. "The experimental settings do not strongly support the motivation: most targets are stated explicitly in the prompt and could be sampled exactly by a random number generator or a tool call. Evaluating on scenarios that genuinely require model-generated randomness, such as mixed-strategy games [1], would better demonstrate the method's practical value."

Following your suggestion, we evaluated mixed-strategy games, and training with the witness advantage improves the play of every model (Sec. 4.4\textrm{\color{blue}Sec. 4.4}, Page 10, and Table 2\textrm{\color{blue}Table 2}, Page 9). We follow the setting of the paper you mention [1]. Each model plays rock-paper-scissors (RPS) and a seven-place hide-and-seek (H&S) against an opponent that plays the best response to the model's most likely move. The prompt states no distribution, and no game appears in training.

The table shows the net score, which is the win rate minus the loss rate in percent, before and after training. A player who always makes the same move scores −100. A higher score means the model's play is harder to predict and take advantage of.

ModelRPSH&S
Qwen2.5-1.5B−51.8 → −30.9−6.8 → 14.8
Llama-3.2-3B−43.7 → −17.927.8 → 43.7
Qwen3-4B−82.9 → −12.8−81.5 → 39.8
Qwen3.5-4B−26.8 → −13.319.4 → 44.2

The scores also rise for every model when the prompt states the equilibrium strategy (Table 15\textrm{\color{blue}Table 15}, Page 40). Moreover, two tasks of the submission already require more than reading a stated distribution (Sec. 4.4\textrm{\color{blue}Sec. 4.4}, Page 10). The implicit opinion prompts do not provide probabilities, so the model must infer the distribution from the context, and training removes 56% to 69% of the error there. In the sibling task with uniform targets, the model must build the valid outputs before it samples among them.


W2. "Comparisons with the closest prior methods are missing. GAPO [2] and the calibration fine-tuning of [3] address similar problems but are not evaluated empirically. I suggest the authors provide a discussion about these related works in detail."

Thanks for the suggestion. We address all the papers below.

For GAPO, we had already discussed this work in the related work. Additionally, we ran an empirical comparison with GAPO, and the witness advantage outperforms it on all three evaluation sets (Table 1\textrm{\color{blue}Table 1}, Page 8, and Table 12\textrm{\color{blue}Table 12}, Page 39). We implement GAPO as published and train it on Qwen2.5-1.5B with the same prompts, group size, and number of steps, with three seeds (App. H.3\textrm{\color{blue}App. H.3}, Page 39). The table shows the sampling error, which is the median excess TV, where lower is better.

MethodTraining targetsUnseen parametersUnseen families
Untrained model0.4600.4770.588
GAPO0.2530.4170.538
Witness advantage (ours)0.1150.1690.380

GAPO removes 13% of the error on unseen parameters and 8% on unseen families, and the witness advantage removes 65% and 35%. The GAPO reward compares the group with a uniform target, so it trains toward a uniform distribution for every prompt.

For [3], "LLMs Can't Play Hangman" by Baldelli et al. (arXiv:2601.06973), the revision now discusses the paper in the related work (App. A\textrm{\color{blue}App. A}, Page 17). That paper shows that an agent whose only state is the public conversation cannot keep hidden information consistent with its replies, and it addresses the problem with a private working memory in the agent architecture. Our work is complementary, because we train the model so that its samples follow a stated distribution.

For calibration fine-tuning, the closest work is by the same first author, "Probabilistic Calibration Is a Trainable Capability in Language Models" (arXiv:2605.11845). It fine-tunes with a supervised loss, and our supervised cross-entropy baseline represents this kind of training.

On Qwen2.5-1.5B, the witness advantage matches this baseline on unseen families (Table 1\textrm{\color{blue}Table 1}, Page 8). Moreover, the witness advantage preserves general capabilities. On all four models that we train, its MMLU and IFEval scores do not differ significantly from those of the untrained model. In contrast, the cross-entropy baseline significantly lowers one of the two scores on each of the three new models (Table 13\textrm{\color{blue}Table 13}, Page 39).

W3. "The current experimental setups are restricted to older family of open-source model (Qwen2.5). The claim the authors provided in the paper could be strengthened if they could test on latest open-source model with stronger reasoning capabilities such as Qwen3.5."

Following your suggestion, we trained Qwen3.5-4B, and the witness advantage improves its sampling and keeps its general capabilities (Table 2\textrm{\color{blue}Table 2} and Sec. 4.3\textrm{\color{blue}Sec. 4.3}, Page 9). We also trained Qwen3-4B-Instruct and Llama-3.2-3B-Instruct. All runs use the settings of the submission and three seeds, and only the learning rate is chosen per model (App. H.1\textrm{\color{blue}App. H.1}, Page 38). The table shows each score before and after training.

ModelError, training targetsError, unseen parametersError, unseen familiesMMLUIFEval
Qwen3.5-4B0.442 → 0.0780.579 → 0.0990.484 → 0.4020.731 → 0.7370.880 → 0.867
Qwen3-4B0.737 → 0.1480.841 → 0.2220.803 → 0.3570.700 → 0.6960.882 → 0.886
Llama-3.2-3B0.770 → 0.1220.821 → 0.1800.607 → 0.3920.589 → 0.5860.789 → 0.777

On Qwen3.5-4B, training removes 82% of the error on the training targets and 83% on unseen parameters. General capability metrics (MMLU and IFEval) also barely change on any of the four models. Furthermore, Qwen3.5-4B improves in both mixed-strategy games (W1).

Thanks for the author's reply. Some doubts have been clarified, and I will raise my score to 3 points.

Reviewer_DjFvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to train language models to sample from the distributions specified in their prompts. It introduces the witness advantage, a reward that guides GRPO to increase under-produced outcomes and decrease over-produced ones. Theoretical analysis and experiments show that the method improves distribution matching and generalizes to unseen target distributions.

Critical strengths
  1. The reward has a clear theoretical motivation, including an analysis of its expected update and the bias from group centering.
  2. The experiments extend beyond synthetic distributions to opinion distributions and structured outputs, supporting the method’s usefulness in more realistic settings.
Critical weaknesses

The training results are limited to Qwen2.5-1.5B-Instruct. Although the paper evaluates larger models as baselines, it does not test the proposed training method on them. It remains unclear whether the sampling gains and capability preservation extend to larger models or other model families.

Questions for authors

Have you tested witness training on a larger model or a different model family?

2 comments

We thank the reviewer for the positive feedback. We are glad that you find that the reward has "a clear theoretical motivation" and that the experiments on opinion distributions and structured outputs support "the method's usefulness in more realistic settings". We respond to the weakness (W) and the question (Q) below.


W and Q. "The training results are limited to Qwen2.5-1.5B-Instruct. [...] It remains unclear whether the sampling gains and capability preservation extend to larger models or other model families. [...] Have you tested witness training on a larger model or a different model family?"

Following your suggestion, we trained three more models, and both the sampling gains and the capability preservation extend to all of them (Table 2\textrm{\color{blue}Table 2} and Sec. 4.3\textrm{\color{blue}Sec. 4.3}, Page 9). The new models are Llama-3.2-3B-Instruct, Qwen3-4B-Instruct, and Qwen3.5-4B. We use the settings of the submission and three seeds per model, and only the learning rate is chosen per model (App. H.1\textrm{\color{blue}App. H.1}, Page 38). In the table, error is the median excess TV, where lower is better.

ModelError, training targetsError, unseen parametersError, unseen familiesMMLUIFEval
Qwen2.5-1.5B0.460 → 0.1150.477 → 0.1690.588 → 0.3800.584 → 0.5840.519 → 0.507
Llama-3.2-3B0.770 → 0.1220.821 → 0.1800.607 → 0.3920.589 → 0.5860.789 → 0.777
Qwen3-4B0.737 → 0.1480.841 → 0.2220.803 → 0.3570.700 → 0.6960.882 → 0.886
Qwen3.5-4B0.442 → 0.0780.579 → 0.0990.484 → 0.4020.731 → 0.7370.880 → 0.867

Training removes 75% to 84% of the error on the training targets and 65% to 83% on unseen parameters, and it lowers the error on unseen families for every model. Moreover, the untrained models differ widely in their error on unseen families, yet the four trained models reach a similar error. General capability metrics (MMLU and IFEval) also barely change on any of the four models.

Furthermore, the gains extend to tasks where distributions are not stated. In two mixed-strategy games that we did not train on, our method raises the score of all four models (Sec. 4.4\textrm{\color{blue}Sec. 4.4}, Page 10, and Table 15\textrm{\color{blue}Table 15}, Page 40).

Dear Reviewer DjFv,

As the deadline for the discussion approaches, we wanted to follow up to ensure that our responses have addressed your concerns. If there are any additional questions or points for discussion, we would be happy to address them promptly.

We look forward to your feedback and sincerely appreciate your time and effort in reviewing our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We are encouraged that the reviewers find the reward "straightforward and intuitive" (2S6m) with "a clear theoretical motivation" (DjFv), the experimental results "convincing" (2S6m), and the method of "practical value at negligible cost" (BN2S). We also appreciate that they value experiments that "extend beyond synthetic distributions" (DjFv) and writing that is "clear" and "easy to follow" (2S6m).

We ran every experiment that the reviewers asked for, and we respond to each reviewer individually. The new experiments support the submission's claims. The uploaded revision marks all changes in blue. Its three main additions are:

  • Three more trained models from two model families, including Qwen3.5 (Table 2\textrm{\color{blue}Table 2} and Sec. 4.3\textrm{\color{blue}Sec. 4.3}, Page 9). The witness advantage reduces sampling error across all four models on training targets, unseen parameters, and unseen families, while preserving their general capabilities.
  • An empirical comparison with GAPO (Table 1\textrm{\color{blue}Table 1}, Page 8). With the same model, prompts, and training steps, GAPO removes 8% of the sampling error on unseen families, while our method removes 35%.
  • Mixed-strategy games, where the model must produce the randomness (Sec. 4.4\textrm{\color{blue}Sec. 4.4}, Page 10). Although we have not directly trained on these games, training with the witness advantage raises the score of all four models in both games.

To address the presentation concerns of Reviewer BN2S, the main text now includes an example of the prompts we give the model during training and evaluation (Sec. 4.1\textrm{\color{blue}Sec. 4.1}, Page 7), together with a proof sketch of our main result (Sec. 3.2\textrm{\color{blue}Sec. 3.2}, Page 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.