acceptodds
Under review as a conference paper at ICLR 2027

Random Choice in Language Models: A Linear Choice State and Low-Rank Sampling Control

125 views

Abstract

Large language models (LLMs) often make highly biased choices when asked to select an item at random. We ask whether this behavior is only an output-level failure or whether models internally represent random choice as a distinct task mode that can be read and controlled. We study this question across twelve answer domains using linear probes, sparse autoencoders (SAEs), and causal interventions on internal activations. We identify a domain-general linear representation that separates random-choice requests from deterministic ones, generalizes to unseen domains and paraphrases that do not explicitly mention randomness, extends to broader arbitrary-choice contexts, and replicates across Gemma-2, Gemma-3, Llama-3.1, Qwen2.5, and Qwen3. Causal interventions on Gemma-2-9B, with a qualitative replication on Llama-3.1-8B, reveal a functional decomposition: one direction acts as a switch into the random-choice mode, whereas a separate direction shapes the resulting answer distribution. This decomposition enables a mechanism-guided intervention that improves sampling without task-specific gradient training. We further find that a trained low-rank adaptation (LoRA) policy produces low-dimensional activation changes, motivating a low-rank representation fine-tuning (LoReFT) intervention that often reaches the trained adapter's calibration regime with substantially fewer parameters, though with lower robustness on some skewed targets. Together, these results show that random-choice behavior is supported by compact, causally relevant internal structure that can be exploited for efficient sampling control.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, large language models, activation steering, internal representations, probabilistic sampling

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_MpuEOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether biased 'random' choices in LLMs reflect an internal task state. Using 208 paired random/deterministic prompts over twelve answer domains, linear probes at the answer position separate the two request types, transfer to held-out domains, implicit paraphrases and naturalistic prompts, and replicate across Gemma-2/3, Llama-3.1 and Qwen2.5/3; selective SAE latents track the same state. On Gemma-2, adding the difference-in-means direction moves deterministic prompts into the random regime but does not raise entropy beyond its natural level, while a late SAE direction chosen by entropy attribution shapes the output distribution. The paper then builds a gated rank-1 reader×writer edit, trains conditional LoRA and rank-4 LoReFT samplers, and compares them with naive prompting and String Seed of Thought on five probabilistic instruction-following targets.

Critical strengths
  1. Lexical confounds are handled better than in most probing work. Mid-stack probes transfer from explicit to implicit requests at AUC 1.0 with no false positives on mention prompts, while early layers behave like token detectors. On naturalistic prompts with vocabulary Jaccard 0.77, the leave-scenario-out probe reaches AUC 1.00 against 0.90 for bag-of-words. Appendix D.6 adds an informative negative result: the direction does not separate stochastic sampling from arbitrary free choice (AUC 0.50 on Gemma, 0.57 on Llama), so it encodes an open-choice regime. That is a sharper finding than the title conveys.

  2. The causal tests on Gemma-2 come with proper controls. The gated edit raises held-out entropy from 1.48 to 2.20 bits and valid mass from 0.479 to 0.665 while deterministic accuracy goes from 0.96 to 0.95. Without the gate deterministic accuracy falls to 0.15, all 50 random readers and all 50 frequency-matched writers do worse, and a trained rank-1 edit reaches 2.31 bits from the mechanistic initialization against 1.76 from random initialization. The switch-versus-shape dissociation is well supported for this model.

Critical weaknesses
  1. The benchmark mixes evaluation interfaces. In Table 2, LoRA and LoReFT are trained for each target and scored by sampling from the valid-answer marginal at the first answer position, while naive prompting and SSoT are zero-shot and scored on free generations that include invalid outputs. Reaching the PRNG interval in 25/25 cells is largely a property of that readout, since the installed marginals already have analytic JS of 0.00–0.24×10⁻³. When the same adapters generate freely (Table 16), TV ranges from 0.04 to 0.39 against 0.05–0.08 for restricted draws, Gemma 3-choice is essentially no better than naive prompting (0.327 vs. 0.333), and on the reasoning model TV after the chain of thought is 0.18–0.54. SSoT is also, by its authors' account, unreliable on non-reasoning models below about 8B, so it is a weak reference here.

  2. The link from mechanism to control is thin. The writer is selected by gradient attribution on entropy, so its raising entropy is partly built in, and the paper notes that it mainly increases diffuseness. There is no comparison with a reader-gated temperature increase or logit flattening matched on valid mass, which is the obvious test of whether this particular direction matters. The edit transfers only partly to Llama, the mode direction is not a sufficient switch on Gemma-3 or Qwen3, and projecting it out of the trained LoRA branch removes 2.1% of the adapter's gain. The best-performing controller does not use the mechanism the paper identifies, so mechanism-guided control is shown for one model and as a partial effect.

Questions for authors
  1. Under free generation with invalid outputs counted, how do the target-specific LoRA and LoReFT samplers compare with SSoT and with naive prompting that states the target, across all five models?

  2. How does the gated edit compare with a reader-gated temperature increase or flattening over the valid answer set, matched on valid mass? If the two are close, what does the specific writer direction add?

  3. Is the mode direction necessary for the model's own behavior on natural random requests? Ablating it on the base model cuts valid mass to 0.14 (Appendix D.1), which mixes loss of the random mode with general damage. A targeted ablation that separates the two would strengthen the causal claim.

1 comment
AuthorsOpenReview ↗

Follow-up to Reviewer MpuE

Thank you again for your detailed review. With the discussion period ending soon, we would like to know whether our response addresses your concerns.

Following your suggestions, the revision adds a matched end-to-end comparison (Table 3; Appendix E.1) and a comparison of the gated edit with reader-gated temperature and flattening matched on valid mass (Appendix E.7). It also adds a necessity test of the mode direction (Appendix E.6) and an analysis of how the learned LoReFT subspace relates to that direction (Appendix E.8). If these address your concerns, we would be grateful if you would consider updating your score. We are happy to answer any remaining questions.

Reviewer_netMOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates internal representations of random-choice requests in language models using linear probes, sparse autoencoders, and causal interventions. It identifies a domain-general choice representation and separates choice-mode activation from distribution control. Building on these findings, the authors propose a rank-1 intervention and LoReFT-based sampling control. Experiments across multiple model families demonstrate improved calibration on held-out domains and non-uniform distributions. Rank-4 LoReFT achieves near-PRNG calibration in 20 of 25 settings under restricted-marginal evaluation, with substantially fewer trainable parameters than LoRA.

Critical strengths
  1. Interesting mechanistic insights supported by multiple analyses. The paper identifies a linearly decodable choice-state representation and distinguishes it from a distribution-shaping direction. The combination of probing, SAE analysis, and causal interventions provides stronger evidence than correlational analysis alone. Cross-domain evaluations and controlled interventions further support the generality of these findings.

  2. Effective and parameter-efficient sampling control. The proposed rank-1 intervention improves sampling entropy without additional training while largely preserving deterministic behavior. LoReFT further improves distributional calibration with substantially fewer trainable parameters than LoRA. Experiments across model families, target distributions, and intervention ranks demonstrate the effectiveness and potential generalizability of the approach.

Critical weaknesses
  1. Inconsistent training and evaluation protocols limit the main comparison. In Table 2, LoRA and LoReFT are trained for specific target distributions and evaluated using restricted valid-answer marginals, while prompting baselines use target distributions provided in prompts and are evaluated through free generation. This makes direct comparisons difficult. Matched end-to-end evaluations, particularly on unseen target distributions, would better support the claimed advantages.

  2. Limited robustness and generalizability of sampling control. The causal interventions show inconsistent effectiveness across models. The training-free rank-1 intervention recovers only 45% and 22% of the trained entropy gains on Gemma-2 and Llama-3.1, while mode steering is unreliable on Gemma-3 and Qwen3. Rank-4 LoReFT also encounters calibration failures or excessive off-support mass in 5 of 25 settings. Moreover, one intervention reduces tinyGSM8K accuracy from 0.92 to 0.67. These results raise concerns about robustness and capability preservation, motivating more systematic evaluations of calibration–performance trade-offs.

Questions for authors
  1. Could the authors provide a matched end-to-end comparison of conditional LoRA, LoReFT, and SSoT under the same generation and evaluation protocol, including invalid-output rates?

  2. How reliably can the identified reader–writer mechanism provide selective sampling control across newer model families?

1 comment
AuthorsOpenReview ↗

Follow-up to Reviewer netM

Thank you again for your review. With the discussion period ending soon, we would like to know whether our response addresses your concerns.

In the revision, all four methods are compared under one matched end-to-end protocol (Table 3; Appendix E.1). The representation and the intervention are reported separately for each model (Appendix E.2), the unstable LoReFT cells are analyzed in Appendix E.3, and capability is evaluated on five benchmarks (Appendix E.4). If these address your concerns, we would be grateful if you would consider updating your score. We are happy to answer any remaining questions.

Reviewer_EC9EOpenReview ↗
Rating
3(+2)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

Authors investigate the ability of LLMs to model specific distributions and produce samples from those distributions. They claim to identify a causal structure within the LLMs that allow them to deterministically tweak aspects of the distribution being randomly selected from. As far as I can tell, they only test selecting from uniform distributions, and so the aspect that's being tweaked with their intervention is simply "degree of randomness", compared against a PRNG.

They use a deterministic vs non-determinstic set of prompts (e.g. pick the largest) to be able to identify a "random mode" in the model. However authors simultaneously establish, that "random mode" isn't really random (models have preferred digits etc.). So I'm not quite sure what it is that they've identified. Activations in the model that indicate that randomness is being requested in the prompt?

Critical strengths

None

Critical weaknesses

Authors make bold claims about being able to tweak aspects of random distributions with their intervention, but as far as I can tell, they only test uniform distributions. As a practitioner, why should I not simply crank the temperature to near infinity, then do constrained decoding over my set of options? For uniform distributions, this should work, should it not? But also, if I'm gonna do this, I don't really need the LLM, I can just use a PRNG directly. This only breaks for non-uniform distributions, which as far as I can tell, the authors do not really test. But in fact I could also do non-uniform distributions with a PRNG. What does this add beyond using a PRNG directly. You need to explain to me why I shouldn't just use a PRNG. Right now, it's extremely unclear to me what the advantage of this method is.

Authors purposely exclude "estimating a human-choice distribution from data, as studied in distributional-alignment benchmarks". But to me, this is the only area in which LLMs can have any sort of advantage/utility.

Questions for authors

Humans are not random. This is part of the problem, and the main reason "preferred digit preference" appears. No? Humans don't make "naturally varied choices". Careful with the wording.

If this "random mode" flag that you've identified is active even when the model is picking a "preferred digit", what exactly is it that you've identified? This implies the model knows that randomness is being asked of it, but it is unable to actually produce the randomness?

Why should we expect LLMs to be able to be random at all, beyond the randomness that the sampler injects (which is already effectively a PRNG)? I'm stuck again on, why wouldn't I just use a PRNG to begin with?

1 comment
AuthorsOpenReview ↗

Follow-up to Reviewer EC9E

Thank you again for your review and for your direct questions. With the discussion period ending soon, we would like to know whether our responses address your concerns.

We would also like to add context to our answer to your main question: why the model itself should sample well when a PRNG exists.

Our paper belongs to an established line of work on whether language models can follow a specified answer distribution. The most closely related work, String Seed of Thought (Misaki & Akiba, ICLR 2026), is a prompting method that improves probabilistic instruction following, the task we evaluate in Tables 2 and 3. Zhang et al. (COLM 2024) fine-tune models toward diffuse distributions over valid answers. Gu et al. (COLING 2025) find that LLM agents struggle with probability sampling, and that coding tools help them sample from distributions given explicitly but not from implicit ones. Meister et al. (NAACL 2025) find that LLMs describe a group's opinion distribution more accurately than they simulate it. These works treat the model's own sampling as a research problem. As our introduction notes, it matters in applications such as simulating human behavior, generating diverse content and playing mixed strategies. In these applications the model makes the choice while it generates, and an external PRNG covers them only when the options and the choice point are known in advance (our response, comment 1/2).

Our contribution to this line is mechanistic. We show where a request for a random or arbitrary choice is represented in the model, and how that representation can be used to control the output distribution with a small number of parameters.

If our responses address your concerns, we would appreciate it if you would consider revisiting your score. We are happy to answer any remaining questions.

References

Reviewer_PAh7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether language models internally represent requests for random choice and whether this representation can support better sampling control. Across several model families, the authors identify a linearly decodable random-or-arbitrary-choice state that generalizes across answer domains and prompt formulations. Causal interventions distinguish activating this choice mode from shaping the resulting answer distribution. Based on this distinction, they construct a gated reader–writer intervention that partially improves random sampling while preserving deterministic behavior. They further analyze the low-dimensional activation changes produced by trained LoRA adapters and evaluate LoReFT as a smaller learned controller. Experiments examine distribution matching, held-out-domain generalization, repeated sampling, and preservation of ordinary responses.

Critical strengths

The paper distinguishes recognizing a random-choice request from successfully producing the desired distribution. This provides an informative perspective on why models can respond to randomness instructions while still exhibiting strong answer biases. The identified reader–writer structure leads to a hand-built intervention with measurable improvements on held-out domains. The comparison between mechanistic and random initialization provides additional evidence that the analysis yields useful information for constructing a compact controller.

Critical weaknesses

The major concern I have is for table 2. According to Appendix D.8, it seems that LoRA and LoReFT are evaluated by sampling from their restricted valid-answer distributions, whereas naive prompting and SSoT produce complete responses. If this is true, I find it difficult to attribute the reported performance gap entirely to the intervention proposed by the authors. Appendix D.8 only covers the further analysis of conditional LoRA and naive prompting but does not include the results for LoReFT and SSoT.

Questions for authors

Please see weakness. That is the major question I have, and I'm happy to change score if that's properly addressed.

1 comment
AuthorsOpenReview ↗

Follow-up to Reviewer PAh7

Thank you again for your review. With the discussion period ending soon, we would like to know whether our response addresses your concern about Table 2.

The revised paper separates the answer-position comparison of the two trained controllers (Table 2) from a matched end-to-end comparison of all four methods under one generation and evaluation protocol (Table 3). Appendix E.1 gives per-cell results and invalid-output rates for LoReFT, conditional LoRA, naive prompting and SSoT. You mentioned that you would be willing to change your score if this was addressed; if our response does so, we would be grateful if you would consider it. We are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.