acceptodds
Under review as a conference paper at ICLR 2027

Rolling Fair Dice: Group-Level Credit Assignment for LLMs

Abstract

Large language models (LLMs) are widely used to sample from target distributions, yet audits show they fail goodness-of-fit tests on most standard distribution families. We turn this into a training problem with a verifiable reward: the total-variation (TV) distance between a rollout group's pooled empirical distribution and the target. This reward breaks GRPO-style training: broadcast to every member, it makes the centered advantage identically zero and the policy gradient vanish (Proposition 1). We introduce two remedies: JCA (Jackknife Credit Assignment) credits each sample with its leave-one-out contribution to the group score, a black-box estimator for any group-level reward, and a differentiable -diff objective reweights the empirical CDF by likelihood ratios. Under decomposable rewards, JCA reduces exactly to standard GRPO (Proposition 2). On 15 Bad-Dice families, JCA lifts a Qwen-4B model from 0.59 to 0.86 TV score and -diff to 0.93; a 9B model reaches 0.93±0.01, within two points of the strongest 2026 frontier model, without any test-time reasoning. Fine-tuning on samples from the target itself reaches the 0.96 ceiling, pricing what reward-only access costs; training generalizes zero-shot to held-out families (0.93) but stays anchored to list formats and does not transfer to natural-format tasks.

Primary area:
reinforcement learning
Keywords:
probabilistic instruction following, credit assignment, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aPGDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors study a modification of the GRPO loss to train models to sample from prompted distributions. They highlight a challenge with using GRPO (no advantage over rollouts, therefore no signal) and propose a modification, Jackknife Credit Assignment, to get training signal. Through experiments, they demonstrate that JCA results in samples closer to the target distributions measured in TV.

Critical strengths

The authors provide an approach that properly addresses the issue of 0 advantage in GRPO training settings. The empirical results highlight the efficacy of this approach.

Critical weaknesses

I'm generally confused about the overall motivation of this paper. Why do we want LLMs to sample from target distributions? We can simply call an actual mathematical package, such as numpy.random, to get the samples we need. The intro makes the question clear, but not the motivation behind this question. Additionally, the authors talk about demographics in the intro, but the rest of the paper is about sampling from various distributions you could find in a Probability 101 textbook.

The critical tension highlighted in this paper seems obvious: if you average your reward signal over rollouts, of course GRPO will result in 0 gradient. Why not compute TV per-rollout? What is the point of averaging across rollouts/the multiset in the first place?

Also, why do we expect 0 TV in the finite sample regime (the reward is 1-TV)? For instance, with a Bernoulli r.v. with p=0.5, with 2 flips, we don't want to train a model to output only [0,1] or [1,0]. [0,0] and [1,1] have equal likelihoods, but if you were to compute empirical distributions these would be much worse in terms of TV and would be penalized.

Nit: in Section 6.6, you dive straight into the results, with no explanation of what persona prompts are, or any sort of justification for the experiment. In general I would like to see justification for the evaluations you run and why they help emphasize the impact of your contributions.

Overall its unclear to me what the exact problem is here and what you are trying to solve.

Questions for authors

What are credit signal and pass count? These terms aren't defined and I'm a little confused about what your results say about them.

4 comments

We thank the reviewer for the direct criticism. The finite-sample point is correct, and we confirmed it with new experiments; the contribution still stands after this concession, as detailed below.

1. Motivation: why not just use numpy.random?

We agree on the narrow case: if only numerical samples are needed and code can be called, numpy is the right tool, and the paper explicitly does not claim to replace it.

The research question is whether a model can learn to make distribution-conforming choices inside its generated response. Settings where choice is entangled with text generation and no external tool is available do exist: prompt-only synthesis pipelines (persona attributes must be generated jointly with text), answer-position balancing when writing questions, mixed strategies for agents. The Bad Dice audit (ACL 2026) documents these downstream distortions; our persona probe reproduces them: asked explicitly for occupation 70/30, base outputs 97/3.

The textbook distribution families are a controlled testbed for verifiable rewards (analogous to GSM8K for arithmetic reasoning), not an end application. The mechanism contribution, black-box credit assignment for group-level rewards, applies to group rewards such as diversity/fairness/simulator where no numpy oracle exists.

The new experiment grounds the motivation: JCA training directly in the natural persona format (group reward = occupation 70/30 plus city-uniform TV). Occupation base 96.6%/3.4% drops to 69.4%/30.6% (TV 0.994, χ² passes); city base heavily skewed becomes near-uniform (TV 0.955, passes); full table in jm33 Q1. The demographic motivation in the intro now has a corresponding experiment: JCA repairs the downstream constraint distortion documented by Bad Dice in natural-language generation, and the mechanism is not limited to numerical lists. The revision rewrites the intro motivation paragraph and adds the probe setup and rationale to §6.6 (see point 5).

2. "The tension is obvious" / why not compute TV per rollout?

Identifying the cancellation is easy; escaping the deadlock is not. Removing within-group centering (global baseline, cross-group EMA-standardized broadcast) collapses 2 of 3 seeds (57% degenerate steps, z-scores ±7.5); clip±3 does not rescue it; the collapse is structural (missing within-group content contrast), not scale. The scale-matched control (new experiment, credit scaled to JCA's SD, ±0.005 vs ±0.004, 3 seeds) trains stably (600/600 effective steps, vs 255/600 unscaled), but aligned TV 0.690±0.053 remains far below JCA 0.858. Shared group credit, even scale-matched, does not learn the information in sample-level contrasts.

We ran per-rollout TV (new baseline, standard GRPO, matched budget 600 steps, 2 seeds): aligned TV 0.925±0.005, 6.5/15 pass, pooled score above JCA's 0.858. But independence diagnostics reveal the cost:

  • Bernoulli per-list #1 count variance 0.00: every list has identical counts, a deterministic balancing template (iid expectation 5.04, JCA 0.30, base 2.12);
  • uniform per-position marginal TV 0.112: each position is nearly fixed (JCA 0.567, base 0.594, iid reference 0.737);
  • list-mean dispersion ratio 0.010 (iid 1), lag-1 autocorrelation -0.71.

Per-rollout TV maximizes finite-sample balancing pressure on the small n=24 pool; the model satisfies the reward with deterministic balanced templates rather than faithful sampling. Pooled TV is blind to this: per-rollout scores higher on pooled TV yet is a worse sampler.

This answers why pooling: balancing pressure decays with pool size roughly as 1/√m (toy m=32 balanced bias 0.688, m=384 0.548; ceiling table n=24 iid upper bound 0.766, m=384 0.940). Pooling at m=384 pulls the reward from per-list balancing back to batch-level distribution matching. JCA's group-level credit retains more randomness under this pressure (variance 0.30 vs 0.00).

For this task, which is decomposable per rollout, standard GRPO plus per-rollout TV is a simpler strong baseline; the revision adds it to Table 2 and reports its independence cost. JCA's contribution is positioned at general group-level rewards (diversity/fairness/simulator, where no per-rollout decomposition exists); Prop 2 guarantees that the decomposable case reduces exactly to GRPO. The zero-gradient deadlock (Prop 1) still holds for genuinely group-level rewards.

The contribution is not the observation of "zero gradient" itself, but the black-box credit construction, the additive reduction (Prop 2), the estimator analysis, and the empirical boundary of finite-sample bias (including this new baseline).

3. Finite-sample TV: the Bernoulli(0.5) two-flip example (core)

The reviewer is correct. Plug-in TV rewards on finite pools prefer balanced pools over iid sampling; we confirm this from two directions.

The toy experiment uses exactly the reviewer's setup: Bernoulli(0.5), each rollout a length-2 list, policy a softmax over the 4 list types {00,01,10,11}, group-level TV reward plus JCA, 5 seeds × 800 steps. The policy biases toward balanced lists: {01,10} mass 0.688 (iid 0.5), entropy 1.230 (iid 1.386).

LLM measurements confirm this as well: after JCA training, Bernoulli(0.7) per-list #1 count variance is 0.30 (3 seeds; iid expectation 5.04, base 2.12); balancing pressure is measurable at production scale. The new baseline (per-rollout TV) pushes this to the extreme: variance 0.00, per-position TV 0.112, deterministic balancing templates (see point 2). The finite-sample bias of pooled TV rewards is a real force that dominates training behavior.

The bias magnitude can be bounded by the iid ceiling gap. A perfect sampler at m=384 has expected TV 0.940 (balanced pool 1.0, gap about 0.06), while base distortion is 0.3 to 0.5; large-gain families are dominated by distortion correction. The residual pressure on near-perfect families (Bernoulli base 0.97) is exactly the balancing the reviewer predicted. Full iid ceiling table (mean TV score of a perfect sampler):

Pool size m24423841008
15-family mean0.7660.8230.9400.963

The degeneracy is a finite-sample property of the scorer, not of credit assignment and not of format.

First, same toy, same JCA, swapping in a joint-aware scorer (TV over list-type frequencies): the policy converges to iid ({01,10} mass 0.494, entropy 1.385, TV→0.020); at pool size m=384 the TV scorer's pressure also weakens (0.548).

Second, the SFT control (same format/model/budget, teacher = iid target samples): Bernoulli count variance 5.82 (iid expectation 5.04), uniform dispersion ratio 0.994, lag-1 -0.016, independence fully preserved. Cross-entropy is a proper scoring rule with no finite-sample balancing bias, while all plug-in TV reward methods collapse (JCA 0.30, oracle 0.30, W1-diff 2.53, per-rollout 0.00). The degeneracy is reward-induced, unrelated to the list format.

Third, JCA is black-box with respect to the scorer: proper or joint-aware scorers plug in directly, and additive scorers reduce exactly to GRPO via Prop 2.

The paper's claim is pooled histogram fidelity (training objective matches evaluation); it never claimed iid sampling. The Limitations section already states independence as a complementary measurement axis. The revision tightens the abstract and main text to batch-level distribution matching within the trained format, and adds a finite-sample bias analysis subsection (with the toy figure and ceiling table).

4. Terminology: credit signal and pass count

credit signal (Fig 4b) is the standard deviation of sample-level credit (before the count penalty), measuring within-group gradient signal strength; broadcast being identically 0 is the empirical counterpart of Prop 1. pass count is the number of families (/15) passing goodness-of-fit tests (discrete χ², continuous KS) at α=0.01. §6.1 has the convention but no explicit definition; the revision adds definitions at first occurrence.

5. §6.6 missing setup (nit)

Accepted. The revision adds: the MCQ/persona probes come from Bad Dice's downstream constraints (answer-position bias, demographic attribute ratios); the purpose is to test whether list-format training transfers to natural-language constrained generation; full prompts go to the appendix. The persona-format matching training results (occupation 69/31 pass, city near-uniform pass) are merged into this section as direct evidence for training in the deployment format.

Reviewer_aPGDOpenReview ↗

Reviewer Response

Thank you for your response. I believe my critique of the fundamental motivation of the paper still stands, for instance, if we know the target population (and we do as we specify it in the prompt), why not just prompt with persona A and persona B using an external random generator? For instance, if our target is 70-30, just prompt with persona A 70% of the time and persona B 30% of the time?

We agree that for a known, orchestrable attribute, external RNG routing is simpler, and the paper does not claim otherwise. But this observation does not bear on the contribution. The same argument would dismiss any verifiable testbed: GSM8K answers are known, but does the existence of a calculator negate the value of LLMs solving arithmetic problems? The distribution families are a controlled setting for studying group-level credit assignment, exactly as the intro states (numpy is the right tool when only numerical samples are needed and the pipeline can call code); they are not a claim to be the best production persona generator. That a simpler engineered pipeline exists for a task is not an objection to a mechanism contribution.

The mechanism targets group-level rewards where the premise "the target is known" does not hold. Diversity / coverage of a generated set has no target distribution to sample from and no per-rollout decomposition: the only signal is the group score, broadcasting it yields zero advantage after within-group centering (Prop 1), and JCA constructs sample-level credit from the black-box group scorer. External RNG has no analog here, and this is the setting the method is built for. The persona probe shows the mechanism operates in a natural-language format; the revision will make the testbed-versus-application scope explicit.

Reviewer_GuZqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to train LLMs to generate numerical samples that better match a specified distribution when the reward is computed over a pooled set of outputs. The authors show that broadcasting the same pooled reward to every rollout makes GRPO’s centered advantages zero. They propose Jackknife Credit Assignment (JCA), which assigns each sample a contribution based on the change in group score when it is removed, and then aggregates these contributions into rollout-level advantages. They also introduce W1 diff, a differentiable surrogate based on likelihood-weighted empirical CDFs. Experiments on 15 distribution families show substantial improvements in pooled distributional fidelity, including transfer to held-out families. Additional experiments show that transfer is strongest within list generation and is more limited under independent requests and natural-language formats.

Critical strengths
  1. The paper identifies a clear failure mode and provides an intuitive intervention. The cancellation of broadcast rewards under within-group centering is straightforward but practically relevant. JCA offers a simple way to obtain differentiated credit from a pooled scoring function without requiring reward derivatives.

  2. The improvements within the studied setting are substantial. On Qwen3.5-4B, JCA improves the mean TV score from 0.585 to 0.858, while W1 diff reaches 0.926. The main experiments include three training seeds, a larger model, and held-out distribution families, providing useful evidence beyond a single configuration.

  3. The paper makes a useful effort to establish the boundaries of its results. The replacement-credit and global-baseline controls go beyond comparison with a baseline that necessarily has zero distributional gradient. The authors also report limited transfer across output formats and explicitly distinguish pooled histogram fidelity from independent sampling. I appreciate this calibration of the claims.

Critical weaknesses
  1. Important details of the continuous-distribution evaluation are missing. The TV score uses 20 equal-width bins over a truncated support, but I could not find the truncation endpoints for each family or a precise description of how out-of-range values are handled. These choices directly affect the target probabilities and measured scores, particularly for heavy-tailed distributions. The current description is therefore insufficient to independently reproduce the main numbers. Sensitivity to bin resolution would also help establish that the improvements are not specific to the chosen discretization.

  2. Matched requests do not ensure matched effective sample budgets. Appendix C states that all parsed values are pooled, including multiple values returned to nominally single-sample requests. For Uniform(0,1), the base and trained models produce 1,367 and 2,629 values from 1,000 requests, respectively. Consequently, the comparison changes both the number of observations and the weight assigned to individual responses. Since empirical TV depends on sample size, the independent-request improvement cannot be attributed entirely to better sampling fidelity without a count-controlled comparison.

  3. The goodness-of-fit pass counts require stronger statistical justification. The chi-squared and KS tests are applied to pooled outputs, while samples within a list may be dependent. The observation that 25 of 42 uniform lists begin with 0.45 makes position-level behavior particularly relevant. The authors acknowledge this limitation, but the nominal significance level does not by itself establish that the tests are calibrated under this sampling structure. This does not invalidate the pooled TV improvements, but it weakens the additional evidence provided by the pass counts.

  4. The credit-assignment comparison does not fully isolate the effect of the attribution rule. The global baseline uses running normalization, whereas JCA produces much smaller removal credits, and both are combined with a count penalty. Differences in effective update scale and the relative strength of the penalty may therefore contribute to the observed gap. A scale-matched and reasonably tuned global-baseline comparison would make the evidence for the benefit of sample-level attribution more convincing.

Questions for authors
  1. Could you provide the complete evaluation specification for each distribution? In particular, please list the truncation endpoints, bin boundaries, parsing rules, and treatment of out-of-range values. Do the main improvements remain under alternative bin resolutions or shifted boundaries?

  2. How much of the independent-request improvement remains after controlling the effective sample count? Please consider reporting results with one predetermined sample per response, alongside equal-sized subsamples of the pooled outputs. Per-family count compliance and parsing rates would help interpret the comparison.

  3. Can the existing outputs support a position-level and dependence-aware analysis? For example, how do the marginal distributions vary across list positions, and how would accounting for within-list dependence affect the goodness-of-fit conclusions?

  4. How sensitive is the global-baseline comparison to reward scaling and learning rate? A comparison with matched update magnitudes, together with the relative gradient contributions of the distribution reward and count penalty, would help clarify the benefit attributable to JCA.

2 comments

We thank the reviewer for the careful protocol-level reading. All four points raise testable methodological questions, and we respond to each with new analyses and controlled experiments. New numbers come from discussion-period experiments and will be merged into the revision.

W1/Q1 Missing evaluation specification for continuous distributions

We acknowledge the description was insufficient. The full specification follows; the revision will include it as an appendix table.

FamilyTruncated supportFamilyTruncated support
Uniform(0,1)[0,1] naturally boundedChi-square(5)[0,25]
Gaussian(0,1)[-4,4]F(5,10)[0,10]
Beta(2,2)[0,1] naturally boundedGamma(2,2)[0,10]
Exponential(1)[0,10]Weibull(1.5,1)[0,8]
Cauchy(0,1)[-10,10]Laplace(0,1)[-8,8]
Student-t(3)[-8,8]Logistic(0,1)[-8,8]

Truncation is stated explicitly in the prompt; the target distribution is renormalized after truncation (TV target masses and KS reference samples both come from this truncated CDF; 1000 reference samples). Continuous TV uses 20 equal-width bins; target masses are normalized adjacent-CDF differences, empirical masses are normalized histogram counts. Out-of-range values are discarded as invalid (tolerance 1e-6): no clipping, no overflow bin; invalid counts are logged per family (bad_items). Discrete families use exact category frequencies; χ² merges adjacent bins until expected ≥5; Poisson is truncated to 0..20 and renormalized; α=0.01.

Binning sensitivity (new analysis, recomputed on stored samples) covers bins ∈ {10,20,40,80}, half-bin shift, and target equal-mass bins, six configurations in total. The main ordering is stable across all of them:

ConfigbasenaiveJCAW1-difforacleSFT
b100.5590.6330.8670.9540.9440.965
b200.5320.5850.8440.9190.9250.953
b400.5110.5550.8200.8830.8890.936
b800.4890.5270.7860.8370.8380.912
b20 half-bin shift0.5310.5830.8440.9220.9160.954
b20 equal-mass0.5290.5730.8190.8900.8860.935

Values are means over the 12 numerical families. W1-diff and oracle differ by ≤0.01, within each other's noise bands; their relative order flips across configurations, consistent with the paper reporting them as statistically equivalent. The improvement is not a discretization artifact; the KS test itself has no bin dependence and is reported alongside.

W2/Q2 Matching request counts ≠ matching effective sample budgets

We acknowledge the protocol description flaw: 1367 vs 2629 does change both the observation count and the response weight. We applied three layers of control, using a rerun that logs the first value, all values, and counts per response on fresh sampling draws (one self-consistent dataset).

First, count-matched subsampling (per family, subsample to min(n_base, n_trained), 200 repetitions): trained mean 0.563, nearly identical to pooled 0.564 (base 0.405). The improvement does not come from sample count.

Second, first-value-only (strictly one prespecified sample per response): base 0.373 → JCA 0.457 (+0.084). The improvement holds under the strictest protocol, but the magnitude is smaller than pooled (+0.159); part of the pooled gain does come from the multi-value effect.

Third, per-family compliance/parse table: JCA's parse rate is 1.00 across all families (base 0.78 to 0.99), but multi-value rates are substantial on heavy-tailed families (gaussian 80%, logistic 52%, t3 53%, laplace 42%, cauchy 41%). After training the model often outputs multiple values under a single-sample request; this is a genuine format-compliance defect.

Per-family results under the first-value protocol are mixed: heavy-tailed families improve substantially, discrete families regress:

FamilyTV base first-valueTV JCA first-valueDirection
laplace0.1770.602↑
logistic0.3850.728↑
gaussian0.2800.493↑
f_5_100.2780.515↑
binomial0.8920.658↓
poisson0.5560.281↓
bernoulli0.7760.709↓

Under the indep protocol the mean improvement holds but per-family results are mixed, and the trained model's multi-value behavior inflates the pooled numbers. The revision will report all three protocols per family (pooled / count-matched / first-value) with the compliance table, and tighten the abstract's indep claim.

W3/Q3 Dependence calibration of GoF pass counts

We acknowledge that pooled tests have miscalibrated nominal levels under within-list dependence. The new analysis follows.

Per-position marginals: reconstruct list boundaries, per-position TV (m=42) against an iid reference. JCA beats base on 11/12 numerical families (Weibull 0.324→0.690) but stays below the iid reference (about 0.8); pooled fidelity does not equal per-position correctness.

Within-list dependence: lag-1 autocorrelation base -0.62, JCA -0.57 (uniform), adjacent-repeat rate 0; Bernoulli per-list count variance base 2.12, JCA 0.30 (iid expectation 5.04). Training amplifies balancing on easy families (an inherent bias of finite-sample TV rewards; see the toy analysis in aPGD §3), but within-list negative correlation already exists in base. Attribution control: SFT (iid teacher, same format and budget) variance 5.82 (iid expectation 5.04), dispersion ratio 0.994, lag-1 -0.016, independence fully preserved. The degeneracy is induced by the plug-in TV reward, not by the list format; the per-rollout TV baseline collapses hardest (variance 0.00).

Dependence-robust test: one random value per list (lists independent, n=42 effective samples), χ²/KS, 200 repetitions. JCA passes 7/15 families at rate ≥0.96 (11/15 at ≥0.8); base only 2/15 (both easy families); hard families (chi2 0.25, weibull 0.36) still show detectable deviation. The improvement holds under the dependence-robust protocol; the pooled 3.3/15 is a strict verdict under high power plus dependence inflation.

List-level bootstrap (resampling lists, 500 draws) gives 95% CIs for pooled TV, e.g., Weibull base [0.335,0.359], JCA [0.687,0.768], non-overlapping.

The revision will label pass count explicitly as pooled and high-power, add a calibrated-protocol column, and note explicitly where independence claims fail.

W4/Q4 Credit-assignment comparison does not isolate the attribution rule

We acknowledge the scale confound: measured JCA sample-level credit SD median 1.24e-3, global broadcast z-score across-step SD 1.46 (range ±7.5); at the same lr the effective update scales differ by about 1000×.

Existing evidence: the clip±3 control does not rescue the collapse (s0 kills 9 families, s2 kills 3); the collapse is structural (missing within-group content contrast), not unbounded scale.

The new controlled experiment is a scale-matched global: broadcast credit multiplied by 8.5e-4 so the sample-level SD matches JCA (1.24e-3), lr/λ unchanged, 3 seeds.

Unscaled globalScale-matched global (new)JCA
aligned TV0.694±0.0460.690±0.0530.858±0.013
Effective steps (s0)255/600600/600600/600
Effective families (s0)11/1515/1515/15
Advantage range±7.5±0.005±0.004

Scale matching removes the collapse but not the gap to JCA. The gap is attributable to the attribution rule (within-group sample-level content contrast), not the effective update scale; this is robust across three seeds. Global credit is constant within a group, so its rollout-level SD equals its sample-level SD; JCA's rollout-level SD does not exceed its sample-level SD, so this matching is lenient toward global and the direction of the conclusion is robust. On lr sensitivity: scaling advantages by c at fixed lr is equivalent to an lr×c adjustment for that channel; the gscale experiment matches the update magnitude of the distribution channel. The count-penalty channel is unscaled; its relative contribution follows.

Channel decomposition (training logs): JCA's late-stage count penalty averages 1e-4 (n_dev→0), distribution credit 1.2e-3 dominates; global late-stage penalty 0.007 to 0.016. naive's distribution credit is identically 0 throughout (1800 steps), penalty channel only, TV flat at 0.637.

Reviewer_jm33OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JCA and W1 diff to help LLMs generate samples that match a target distribution. Both improve sampling fidelity on 15 distribution families, but the gains are much smaller on other output formats.

Critical strengths
  1. Clear problem. JCA addresses the zero-advantage problem in GRPO when all samples share the same reward.
  2. Strong results. JCA improves the 4B model's TV score from 0.585 to 0.858, while W1 diff reaches 0.926.
Critical weaknesses
  1. Unclear use case. Why train an LLM for this instead of using a random number generator?
  2. Poor transfer. Gains do not carry over to MCQ answer positions or occupation proportions, and city-selection performance gets worse.
  3. Unclear contribution. W1 diff performs better, but it is unclear whether the gain comes from the loss itself or better credit assignment.
Questions for authors
  1. Practical value: Why use an LLM instead of a random number generator? Can you show any real downstream benefits?
  2. Generalization: Why does the method fail to transfer to other output formats? Why does city-selection performance get worse?
  3. SFT comparison: Appendix F shows that SFT achieves a TV distance below 10−610^{-6} on two tasks. Why use RL instead, and how do the methods compare on the main tasks?
  4. W1 diff vs. JCA: How much of W1 diff's improvement comes from its loss rather than the proposed credit assignment?
  5. Sample independence: Does the method ensure independent samples and correct distributions at each list position, or only a matching overall distribution?
  6. Training cost: Are the extra scoring calls in JCA and the target CDF required by W1 diff worth the cost?
3 comments

We thank the reviewer for the clear summary of the problem and the positive reading of the results. We respond to each point below; all new numbers come from discussion-period experiments.

Q1 Practical value: why use an LLM instead of a random number generator?

We agree with the reviewer's premise: if only numerical samples are needed and the pipeline can call code, numpy.random is the right tool, and the paper does not claim to replace it (the intro states this; the revision will strengthen it).

The paper studies a different setting: the choice must be made inside the generation process, with the model deciding distribution-conforming values while it writes content. Three concrete scenarios:

  1. Tool-free, prompt-only synthesis pipelines. Persona attributes (occupation/city) must be generated jointly with the biography text; an external RNG cannot intervene in the generation.
  2. Question writing with answer-position balancing. The model writes the stem and decides the answer position in the same act; the position distribution is part of the generation behavior.
  3. Mixed-strategy sampling for agents. Strategy selection and action expression happen in the same generation step.

The Bad Dice audit (ACL 2026) documents these downstream distortions. Our persona probe reproduces the occupation distortion: asked explicitly for 70/30, the base model outputs 97/3.

The 15 textbook distribution families are a controlled testbed for verifiable rewards, not an end application, much as arithmetic reasoning research uses GSM8K. The mechanism contribution (JCA) applies to any group-level reward: diversity, fairness, simulator feedback. These settings have no numpy oracle, and reward-only training is the only path.

The new experiment grounds this motivation: we ran JCA training directly in the natural persona format (group reward = TV of occupation 70/30 plus city uniformity, leave-one-persona-out credit, 600 steps). Results on 400 evaluation samples:

Probebaseafter persona-JCAtarget
Occupation (engineer/designer)96.6% / 3.4% (TV 0.734)69.4% / 30.6% (TV 0.994, p=0.82, pass)70 / 30
City (four cities)168/29/11/102 (TV 0.629)87/90/70/94 (TV 0.955, p=0.27, pass)uniform
MCQ position0.810.83 (no transfer)uniform

Parse rates are 73.5% for occupation and 85.3% for city (exact keyword match). The lack of MCQ transfer shows format specificity is symmetric. The demographic motivation in the intro now has a corresponding experiment: JCA repairs the downstream distortion documented by Bad Dice in a natural-language format, and the mechanism is not limited to numerical lists.

Q2 Transfer: why do other formats not transfer? Why does city get worse?

On city: TV score = 1 − TV distance, higher is better (Table 5 caption and §3 definition); 0.63→0.68 is a small improvement. What did not transfer is MCQ answer position (0.81→0.81) and occupation ratio (97/3→97/3).

Mechanistic evidence comes from the new analysis. List training improves the pooled marginal distribution, not the per-position conditional distribution. We reconstructed the sample boundaries of each list and computed per-position marginal TV: JCA beats base on 11 of the 12 numerical families (e.g., Weibull 0.324→0.690) but stays below the iid reference at the same sample size (about 0.8); within-list lag-1 autocorrelation is about -0.5 for both base and trained models. Training corrects the overall proportions without correcting per-position iid sampling, consistent with the list-format training objective.

The repair path is now experimentally verified: train in the deployment format. After list training the occupation ratio does not move (97/3); after persona-format training it passes at 69/31 (see Q1). Same JCA mechanism, different training format, transfer repaired. The revision will upgrade "format is a first-class design dimension of distribution training" from a discussion claim to an experimentally supported conclusion, and scope the abstract to batch-level distribution matching within the trained format.

Q3 SFT comparison: what is the <1e-6 in Appendix F? Why RL?

The <1e-6 in Appendix F comes from a supervised CE reference on a 4-action softmax toy. We added an SFT control on the LLM: lists sampled directly from the target distribution serve as the teacher, trained on Qwen3.5-4B at matched budget (600 steps, LoRA rank 32, same prompts).

MethodInformation accessaligned TVpass / 15
SFT (3 seeds)target sampler (oracle)0.947±0.00212.3
W1-diff (3 seeds)target CDF0.926±0.00410.7
JCA (3 seeds)score queries only0.858±0.0133.3
Perfect iid sampler (simulated)n/a0.963 (noise ceiling)n/a

SFT requires a target sampler (oracle access); RL needs only the reward. SFT is near the ceiling (0.947 vs 0.963). JCA's gap of about 0.10 to the ceiling, and W1-diff's gap of about 0.03, are the price of not using a target sampler. In the motivating applications (diversity/fairness/simulator group rewards) no teacher sampler exists, and reward-only is the only feasible path.

SFT has a second advantage: independence preservation (see the diagnostic table in Q5). At the same format and budget, SFT's Bernoulli count variance is 5.82 (iid expectation 5.04), uniform dispersion ratio 0.994 (iid 1), lag-1 autocorrelation -0.016 (iid 0), while all plug-in TV reward RL methods collapse. Cross-entropy is a proper scoring rule with no finite-sample balancing bias. The RL gap is therefore not only in TV numbers but also in sampling quality. The revision adds an SFT row to Table 1 and makes the information-access tiers and independence diagnostics explicit.

Q4 Does W1-diff's gain come from the loss or from credit assignment?

W1-diff is a differentiable surrogate of the likelihood-reweighted empirical CDF; gradients flow through the likelihood weights, and it contains no credit-assignment step. The paper's original information-matched control is oracle-bin (sample-level residual credit + target bin probabilities) at 0.929±0.005, comparable to W1-diff's 0.926±0.004, which had suggested information access as a confound. The new decomposition shows that information is not the main factor; the optimization mechanism and signal strength are.

MethodCredit mechanismTarget informationaligned TV
JCA+TVjackknife (group-level, SD 1.2e-3)scores only0.858±0.013 (3 seeds)
JCA+W1 (new)jackknife (group-level)CDF0.780±0.039 (2 seeds)
oracle-binbin residual (sample-level, SD 2.5e-2)target bins0.929±0.005
W1-diffdifferentiable likelihood reweightingCDF0.926±0.004
Per-rollout TV+GRPO (new)standard GRPO (rollout-level, SD 0.016+)scores only0.925±0.005 (2 seeds)

First, W1-diff's gain does not come from the CDF information itself: giving JCA the same CDF (JCA+W1) clearly hurts (0.780 < 0.858). The gain comes from the optimization mechanism. Differentiable gradients are a strong, low-variance signal; jackknife leave-one-out credit is a weak O(1/m) signal (a 20× SD difference), compounded by the metric mismatch between the W1 reward and TV evaluation. Second, signal strength and granularity dominate: oracle sample-level residuals, per-rollout strong signals, and differentiable gradients all reach about 0.92, with or without target information; JCA+TV is the most restricted interface (scores only + weak group-level credit), and 0.858 is the price paid for black-box generality. Third, metric alignment also matters: JCA+W1 optimizes W1 but is evaluated with TV bins (mismatch), while oracle-bin optimizes TV bin residuals (aligned) and reaches 0.929.

The per-rollout baseline carries an independence cost (Bernoulli count variance 0.00, deterministic templates; see aPGD §2): a high pooled TV does not equal faithful sampling. JCA's value is in black-box access: no CDF, no derivatives, no per-rollout decomposition required; any group-level scorer plugs in. For non-decomposable group rewards (diversity/fairness/simulator) there is no alternative path.

Q5 Sample independence: does the method guarantee per-position correctness or only aggregate matching?

Direct answer: only aggregate (pooled) matching; per-position marginals and independence are not guaranteed. The Limitations section states this; we now add quantification. Diagnostics on a representative seed (s0):

MethodBernoulli per-list #1 variance (iid 5.04)uniform dispersion ratio (iid 1)uniform lag-1 (iid 0)uniform per-position TV (iid ref 0.737)
base2.120.093-0.6170.594
JCA0.410.074-0.5610.567
Per-rollout TV0.000.010-0.7110.112
SFT5.820.994-0.0160.748

JCA's Bernoulli variance averaged over three seeds is 0.30; s0 is 0.41. The finite-sample TV reward does exert balancing pressure; per-rollout pushes it to the extreme (deterministic templates), while SFT preserves independence fully.

Under dependence-calibrated GoF (one random value per list, lists independent, n=42, 200 repetitions), JCA passes 7/15 families at rate ≥0.96 (11/15 at ≥0.8), versus only 2/15 for base. The improvement holds under a dependence-robust protocol.

Balancing pressure is a property of finite-sample plug-in scorers, not of credit assignment. In the toy experiment (Bernoulli(0.5), n=2 lists, same JCA), the pooled TV reward biases the policy toward balanced lists ({01,10} mass 0.688, iid 0.5); switching to a joint-aware scorer converges to iid (0.494, entropy 1.385, iid 1.386); at pool size m=384 the pressure weakens (0.548). JCA is black-box with respect to the scorer and can plug into a joint-distribution-aware scorer. The revision adds this analysis and tightens the claim wording.

Q6 Training cost: are the extra scoring calls and the CDF worth it?

JCA's m scoring calls are local histogram computations, with no model or API calls. Measured at m=384 pool: 22.7 ms/step (numerical families), 7.2 ms (discrete), 0.29% of a training step (about 8 s, dominated by remote sampling and forward/backward passes). The cost is negligible.

W1-diff's CDF requirement is an information cost, not a compute cost: when the target distribution is known the CDF is free; when unknown, one falls back to JCA.

On the inference side, the trained 4B/9B models reach 0.858/0.929 with no thinking and about 800 tokens of budget; frontier thinking models need 32k to 131k tokens of reasoning budget (Appendix E measurements). The deployment cost advantage is substantial.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.