acceptodds
Under review as a conference paper at ICLR 2027

DART: Do Language Models Follow Requested Distributions?

Abstract

We ask whether language models can sample from a probability distribution that a user explicitly requests. We introduce Distribution Audit via Requested Targets (DART), an audit that asks a model for a single draw from a distribution stated in the prompt and scores the completed answer exactly over an enumerated candidate set, separately measuring whether the model attempts the task and, when it does, how closely its choice follows the requested law. Applied to 13 base and post-trained model pairs from six families, DART finds a consistent pattern: base models usually decline the task; when they answer, they typically follow the requested law more closely than their post-trained counterparts, which almost always answer yet concentrate on a single favorite outcome. Four of the six comparable chat-model endpoints place most of their probability on a single answer in a uniform ten-way draw, and the favorite shifts with the answer range rather than a fixed token. A factorial experiment shows that the residual distortion decomposes into interpretable priors over label identity and display position that predict held-out combinations but do not transfer across models or prompt templates. Interventions that flatten the measured output distribution fail to restore genuine sampling behavior. DART thus makes distribution-following measurable: current post-training trades it for unconditional compliance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, evaluation and auditing, sampling and stochastic simulation, mode collapse, post-training and alignment, probability distributions

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_DxtgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a study of the extent to which LLMs generate outputs according to predefined distributions (e.g., uniform integers between 1 and 10). The paper finds that relative to posttrained models, base models better adhere to the requested distribution, but also more often fail at correctly formatting answers. The paper also finds that activation steering does not help address calibration in the context of autoregressive generation.

Critical strengths

To the best of my knowledge, the proposed methodology and set of experiments are technically sound. Though, this is hard to entirely verify given the lack of methodological details given for activation steering.

Critical weaknesses
  • The paper and experimental methodology is substantially AI drafted, which impairs readability. For example, much of the appendix is an AI experiment log (Appendices E and F, table 18, etc).
  • The novelty of the paper is more limited than the claims. In contributions (1) and (2):

1 … DART, an audit that scores completed answers exactly over enumerated candidate sets, separately measuring whether a model attempts the task and how closely its choice follows the requested law.

This type of audit is common in the literature (e.g., West & Potts, 2025 evaluate random number generation as well).

2 … We trace the distortion across the OLMo-2 post-training pipeline and show that it is already detectable at SFT and deepens at later stages on selected probes.

West & Potts 2025 also evaluate across the stages of the Tulu post-training pipeline (DPO, SFT, and the full pipeline), with similar findings.

I am not aware of prior work that explicitly contributes (3) and (4).

  • The paper provides little to no methodological detail for how activation steering is done in contribution (4)
Questions for authors
  • What is the claimed novelty for contributions (1) and (2) relative to prior work like West & Potts?
  • How was the activation steering done?
Reviewer_gBTNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Authors systematically measure the ability of post-trained and base models to adhere to drawing randomly from a specified distribution in the prompt. For example, "pick a random number from 1 to 10". Authors show that models are systematically biased towards specific numbers or samples, and that post-training increases the distance from the requested uniform distribution.

Critical strengths

None

Critical weaknesses

I unfortunately do not believe this paper is contributing much in this crowded space. All of the contributions I identified have already been identified by prior work, some of it from several years ago.

The wording and descriptions in the paper are difficult to understand. Expressions like "the core battery" are not familiar to me at all. The writing is dense and does not make sense to me.

Questions for authors

See critical weaknesses

Reviewer_KrG4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper focuses on a clear question: if you give a language model a request to sample from a finite discrete distribution (e.g., uniform over integers 1 to 10), how well does it do? The main contribution is “distribution audit via requested targets” (DART), a method for evaluating a model’s performance on this task alongside its compliance with the request (many checkpoints refuse). The methodology is fine and comes with a reasonable argument against prior work, but it is not especially interesting or novel in its insights. The paper’s findings are fairly straightforward: post-trained models are compliant (i.e., they answer the request) but show skewed distributions that don’t match what the request asks for, while pre-trained models have lower compliance and distribute answers more according to the request. Using Olmo at 1B and 7B sizes, the method is used to localize the behavior change at the SFT stage.

The paper is highly uneven, giving extensive detail about some aspects of the work (the math behind the probability calculations) and almost none about others (e.g., section 4.5). Some of the writing is clear, but much of it reads like unedited chatbot generations that haven’t been polished for a reader outside the project. The biggest problems are (1) no experimental comparison to the alternative methods discussed early in the paper, and (2) a major confound in that the prompts are not the same for base and post-trained models.

Critical strengths

The approach pays particularly close attention to the calculation of the probabilities of each of the possible answers to the query, giving two variants and explaining the pros and cons of each. (This doesn’t make much practical difference, as the paper notes, and strikes this reviewer as a bit tedious, more of an appendix-appropriate aside.)

The experimental setup includes many models and sizes, and the main effect seems to hold up across them, if we trust the measurement.

Critical weaknesses

The paper doesn’t do a great job of motivating the main research question. If a scientist wants to do a randomized simulation, it’s not clear why they would turn to a language model. I’m willing to believe that the research question is relevant to some real world use (after all, these models are being used in all kinds of surprising ways), but the current single sentence addressing this in the opening paragraph isn’t convincing.

In many places the writing has the punchy style of a chatbot; it’s unkind to a reader who doesn’t have the full context and project-internal jargon that the authors have established in the course of the work. I found the interpretation notes paragraph (line 178) indecipherable; I couldn’t tell if these were notes about the method, the measurements, or the findings. Another example: Line 138 says “All readouts are evaluated in a single scoring pass per candidate path; no sampling is involved.” But I don’t understand how to square that with “384 free-running replicates for sampling-based validation of the deterministic readouts.” The paper doesn’t explain what “free-running replicates” are or “sampling-based validation.”

Section 4.4 is similarly difficult to follow, with unexplained terms (“residual distortion”) and strange word choice (“forces”). This sentence confesses that the experiment is incomplete and breezily softens the conclusion in a way that is basically meaningless: “No analogous decomposition was run on base models, so we attribute the structure to post-training by consistency rather than by isolation.” In fact, the “structure” (also not defined in this context) cannot be attributed to post-training based on this experiment. “Consistency” within one condition doesn’t excuse a missing comparison. The header’s conclusion about “interpretable structure” is never spelled out in a way I could interpret.

Line 36 — “Whether a model can comply with such a request is a basic question about what post-training does to a model’s behavior, yet it is rarely measured directly.” This seems to presuppose the paper’s conclusion; isn’t it a question about learning overall, not just post-training?

Line 196 — “Base models are evaluated under raw prompting; post-trained models under their chat templates, matching deployment.“ This strikes me as a significant confound. I don’t know what “matching deployment” means or why this justifies using a different prompt for the early and late model stages. A more thorough approach would evaluate both models under both conditions and try to determine whether the prompt or the training stage better explains variation in the outcome. This is arguably the most damning problem with the experiments.

Line 199 — what is a cell?

The paper includes a notion of a “descriptive collapse threshold” and sets it to 0.6. This is an overwrought, subjective choice that does absolutely no work in the paper.

Olmo is left out of table 2 for no clear reason.

Line 399 — what does it mean to “flatten” a readout or a distribution? I couldn’t make sense of section 4.5.

Section 4.6 packs in a whole bunch of additional results with almost no information about the experiments run to achieve them.

The paper discusses alternative probability estimation methods in section 2, and basically dismisses them, rather than comparing experimentally whether the results would be the same if we used those (presumably flawed) methods. So it fundamentally fails to make the case for the DART approach compared to other methods.

Questions for authors

See weaknesses; if you can explain why the prompt confound shouldn't concern me, and why none of the alternatives in table 1 should be considered in experiments, and what the deeper motivation for the research question is, then I will reconsider.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.