- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this work, the authors study a modification of the GRPO loss to train models to sample from prompted distributions. They highlight a challenge with using GRPO (no advantage over rollouts, therefore no signal) and propose a modification, Jackknife Credit Assignment, to get training signal. Through experiments, they demonstrate that JCA results in samples closer to the target distributions measured in TV.
Critical strengths
The authors provide an approach that properly addresses the issue of 0 advantage in GRPO training settings. The empirical results highlight the efficacy of this approach.
Critical weaknesses
I'm generally confused about the overall motivation of this paper. Why do we want LLMs to sample from target distributions? We can simply call an actual mathematical package, such as numpy.random, to get the samples we need. The intro makes the question clear, but not the motivation behind this question. Additionally, the authors talk about demographics in the intro, but the rest of the paper is about sampling from various distributions you could find in a Probability 101 textbook.
The critical tension highlighted in this paper seems obvious: if you average your reward signal over rollouts, of course GRPO will result in 0 gradient. Why not compute TV per-rollout? What is the point of averaging across rollouts/the multiset in the first place?
Also, why do we expect 0 TV in the finite sample regime (the reward is 1-TV)? For instance, with a Bernoulli r.v. with p=0.5, with 2 flips, we don't want to train a model to output only [0,1] or [1,0]. [0,0] and [1,1] have equal likelihoods, but if you were to compute empirical distributions these would be much worse in terms of TV and would be penalized.
Nit: in Section 6.6, you dive straight into the results, with no explanation of what persona prompts are, or any sort of justification for the experiment. In general I would like to see justification for the evaluations you run and why they help emphasize the impact of your contributions.
Overall its unclear to me what the exact problem is here and what you are trying to solve.
Questions for authors
What are credit signal and pass count? These terms aren't defined and I'm a little confused about what your results say about them.
4 comments
3. Finite-sample TV: the Bernoulli(0.5) two-flip example (core)
The reviewer is correct. Plug-in TV rewards on finite pools prefer balanced pools over iid sampling; we confirm this from two directions.
The toy experiment uses exactly the reviewer's setup: Bernoulli(0.5), each rollout a length-2 list, policy a softmax over the 4 list types {00,01,10,11}, group-level TV reward plus JCA, 5 seeds × 800 steps. The policy biases toward balanced lists: {01,10} mass 0.688 (iid 0.5), entropy 1.230 (iid 1.386).
LLM measurements confirm this as well: after JCA training, Bernoulli(0.7) per-list #1 count variance is 0.30 (3 seeds; iid expectation 5.04, base 2.12); balancing pressure is measurable at production scale. The new baseline (per-rollout TV) pushes this to the extreme: variance 0.00, per-position TV 0.112, deterministic balancing templates (see point 2). The finite-sample bias of pooled TV rewards is a real force that dominates training behavior.
The bias magnitude can be bounded by the iid ceiling gap. A perfect sampler at m=384 has expected TV 0.940 (balanced pool 1.0, gap about 0.06), while base distortion is 0.3 to 0.5; large-gain families are dominated by distortion correction. The residual pressure on near-perfect families (Bernoulli base 0.97) is exactly the balancing the reviewer predicted. Full iid ceiling table (mean TV score of a perfect sampler):
| Pool size m | 24 | 42 | 384 | 1008 |
|---|---|---|---|---|
| 15-family mean | 0.766 | 0.823 | 0.940 | 0.963 |
The degeneracy is a finite-sample property of the scorer, not of credit assignment and not of format.
First, same toy, same JCA, swapping in a joint-aware scorer (TV over list-type frequencies): the policy converges to iid ({01,10} mass 0.494, entropy 1.385, TV→0.020); at pool size m=384 the TV scorer's pressure also weakens (0.548).
Second, the SFT control (same format/model/budget, teacher = iid target samples): Bernoulli count variance 5.82 (iid expectation 5.04), uniform dispersion ratio 0.994, lag-1 -0.016, independence fully preserved. Cross-entropy is a proper scoring rule with no finite-sample balancing bias, while all plug-in TV reward methods collapse (JCA 0.30, oracle 0.30, W1-diff 2.53, per-rollout 0.00). The degeneracy is reward-induced, unrelated to the list format.
Third, JCA is black-box with respect to the scorer: proper or joint-aware scorers plug in directly, and additive scorers reduce exactly to GRPO via Prop 2.
The paper's claim is pooled histogram fidelity (training objective matches evaluation); it never claimed iid sampling. The Limitations section already states independence as a complementary measurement axis. The revision tightens the abstract and main text to batch-level distribution matching within the trained format, and adds a finite-sample bias analysis subsection (with the toy figure and ceiling table).
4. Terminology: credit signal and pass count
credit signal (Fig 4b) is the standard deviation of sample-level credit (before the count penalty), measuring within-group gradient signal strength; broadcast being identically 0 is the empirical counterpart of Prop 1. pass count is the number of families (/15) passing goodness-of-fit tests (discrete χ², continuous KS) at α=0.01. §6.1 has the convention but no explicit definition; the revision adds definitions at first occurrence.
5. §6.6 missing setup (nit)
Accepted. The revision adds: the MCQ/persona probes come from Bad Dice's downstream constraints (answer-position bias, demographic attribute ratios); the purpose is to test whether list-format training transfers to natural-language constrained generation; full prompts go to the appendix. The persona-format matching training results (occupation 69/31 pass, city near-uniform pass) are merged into this section as direct evidence for training in the deployment format.
Reviewer Response
Thank you for your response. I believe my critique of the fundamental motivation of the paper still stands, for instance, if we know the target population (and we do as we specify it in the prompt), why not just prompt with persona A and persona B using an external random generator? For instance, if our target is 70-30, just prompt with persona A 70% of the time and persona B 30% of the time?
We agree that for a known, orchestrable attribute, external RNG routing is simpler, and the paper does not claim otherwise. But this observation does not bear on the contribution. The same argument would dismiss any verifiable testbed: GSM8K answers are known, but does the existence of a calculator negate the value of LLMs solving arithmetic problems? The distribution families are a controlled setting for studying group-level credit assignment, exactly as the intro states (numpy is the right tool when only numerical samples are needed and the pipeline can call code); they are not a claim to be the best production persona generator. That a simpler engineered pipeline exists for a task is not an objection to a mechanism contribution.
The mechanism targets group-level rewards where the premise "the target is known" does not hold. Diversity / coverage of a generated set has no target distribution to sample from and no per-rollout decomposition: the only signal is the group score, broadcasting it yields zero advantage after within-group centering (Prop 1), and JCA constructs sample-level credit from the black-box group scorer. External RNG has no analog here, and this is the setting the method is built for. The persona probe shows the mechanism operates in a natural-language format; the revision will make the testbed-versus-application scope explicit.
We thank the reviewer for the direct criticism. The finite-sample point is correct, and we confirmed it with new experiments; the contribution still stands after this concession, as detailed below.
1. Motivation: why not just use numpy.random?
We agree on the narrow case: if only numerical samples are needed and code can be called, numpy is the right tool, and the paper explicitly does not claim to replace it.
The research question is whether a model can learn to make distribution-conforming choices inside its generated response. Settings where choice is entangled with text generation and no external tool is available do exist: prompt-only synthesis pipelines (persona attributes must be generated jointly with text), answer-position balancing when writing questions, mixed strategies for agents. The Bad Dice audit (ACL 2026) documents these downstream distortions; our persona probe reproduces them: asked explicitly for occupation 70/30, base outputs 97/3.
The textbook distribution families are a controlled testbed for verifiable rewards (analogous to GSM8K for arithmetic reasoning), not an end application. The mechanism contribution, black-box credit assignment for group-level rewards, applies to group rewards such as diversity/fairness/simulator where no numpy oracle exists.
The new experiment grounds the motivation: JCA training directly in the natural persona format (group reward = occupation 70/30 plus city-uniform TV). Occupation base 96.6%/3.4% drops to 69.4%/30.6% (TV 0.994, χ² passes); city base heavily skewed becomes near-uniform (TV 0.955, passes); full table in jm33 Q1. The demographic motivation in the intro now has a corresponding experiment: JCA repairs the downstream constraint distortion documented by Bad Dice in natural-language generation, and the mechanism is not limited to numerical lists. The revision rewrites the intro motivation paragraph and adds the probe setup and rationale to §6.6 (see point 5).
2. "The tension is obvious" / why not compute TV per rollout?
Identifying the cancellation is easy; escaping the deadlock is not. Removing within-group centering (global baseline, cross-group EMA-standardized broadcast) collapses 2 of 3 seeds (57% degenerate steps, z-scores ±7.5); clip±3 does not rescue it; the collapse is structural (missing within-group content contrast), not scale. The scale-matched control (new experiment, credit scaled to JCA's SD, ±0.005 vs ±0.004, 3 seeds) trains stably (600/600 effective steps, vs 255/600 unscaled), but aligned TV 0.690±0.053 remains far below JCA 0.858. Shared group credit, even scale-matched, does not learn the information in sample-level contrasts.
We ran per-rollout TV (new baseline, standard GRPO, matched budget 600 steps, 2 seeds): aligned TV 0.925±0.005, 6.5/15 pass, pooled score above JCA's 0.858. But independence diagnostics reveal the cost:
Per-rollout TV maximizes finite-sample balancing pressure on the small n=24 pool; the model satisfies the reward with deterministic balanced templates rather than faithful sampling. Pooled TV is blind to this: per-rollout scores higher on pooled TV yet is a worse sampler.
This answers why pooling: balancing pressure decays with pool size roughly as 1/√m (toy m=32 balanced bias 0.688, m=384 0.548; ceiling table n=24 iid upper bound 0.766, m=384 0.940). Pooling at m=384 pulls the reward from per-list balancing back to batch-level distribution matching. JCA's group-level credit retains more randomness under this pressure (variance 0.30 vs 0.00).
For this task, which is decomposable per rollout, standard GRPO plus per-rollout TV is a simpler strong baseline; the revision adds it to Table 2 and reports its independence cost. JCA's contribution is positioned at general group-level rewards (diversity/fairness/simulator, where no per-rollout decomposition exists); Prop 2 guarantees that the decomposable case reduces exactly to GRPO. The zero-gradient deadlock (Prop 1) still holds for genuinely group-level rewards.
The contribution is not the observation of "zero gradient" itself, but the black-box credit construction, the additive reduction (Prop 2), the estimator analysis, and the empirical boundary of finite-sample bias (including this new baseline).