acceptodds
Withdrawn from ICLR 2027

Dispersive Sampling: Batch Diversity via Cross-Sequence Repulsion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This paper introduces Fisher Diverse sampling, an inference-time approach to encourage the samples drawn from a language model to be semantically distinct from each other. To accomplish this, we use determinantal point processes (DPP) to couple autoregressive sampling across the batch dimension. DPPs are a standard method for defining probability distributions over sets of dissimilar elements, given pairwise similarity scores of those elements. In our setting, this requires us to specify similarity scores for each pair of possible output tokens. The challenge then is to find a numerical score that suitably captures the semantic similarity between outputs. We accomplish this by using the Fisher (information) geometry of the language model's output distribution to define a (context-sensitive) notion of similarity on the token unembeddings. Qualitative examples and empirical evaluation show that the method is highly effective at increasing the diversity of model outputs without degrading quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, test-time compute, coupled decoding, diverse sampling, reasoning, information geometry

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Ke1pOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Fisher Diverse sampling, an inference-time method that encourages a language model to generate semantically diverse samples within a batch via determinantal point processes. The Fisher information of the language model output distribution is used as a context-dependent estimation of similarity on the token unembeddings, and a replusion hyperparameter is used to control the level of next-token repulsion of the current batch element vs tokens already sampled in the earlier elements of the batch.

Critical strengths
  • The method induces token-level repulsion from the model output distribution and unembedding vectors instead of using external models
  • Interesting application of Fisher information, incrementally comparing the current to-sample token vs already-sampled tokens in the batch (though a potential risk being compute requirement)
Critical weaknesses
  • The range of model types and sizes used are very narrow; only one model Gemma-4 E2B is used.
  • The tasks used are also limited to synthetic math tasks, mainly combinatorial and enumerative.
  • L338: Regarding the baseline temperature frontier, apart from comparing directly with iid sampling, are there other comparable baselines that primarily promote batch diversity? Or even sequential iid sampling using prior model responses as input context (potentially practical though not novel given the tasks you designed).
  • It seems Fisher Diverse sampling is highly sensitive to temperature and truncation threshold, even when the task suite is already limited to the structured math problems (Table 3). How does it scale to other datasets/tasks?
  • Fisher Diverse sampling consistently underperforms against the temperature frontier on the SUM sub-task in tables 4, 6 and 7, even when all tasks are designed to allow numerous correct answers.
  • How is the pooled metric calculated in tables 4, 6 and 7? It may be biased by tasks with a larger population size.
  • L468: Unclear outperformance statement - Fisher Diverse at B=4 did not outperform the iid baseline at B=32 in terms of validity.
  • Table 5: What is the temperature used for the Fisher Diverse sampling methods, or are they temperature-tuned? Apart from comparing with iid sampling at T=1.0, were temperatures tuned in this experiment?
  • Besides the structured and narrow problems in L756-L781 used that admit large answer sets, can datasets/tasks outside of these synthetic problems be explored? They can potentially require reasoning as well.
  • Given the algorithmic complexity in L232-240, are there analyses of Fisher Diverse sampling's computational and memory efficiency, and comparison with the baselines?
  • On the datasets/tasks used, there are no illustrative examples of solution coverage and validity of Fisher Diverse sampling vs the baselines; please provide.
  • What are the reasons for showing non-consistent configurations for the Fisher, PPR and Trunc-only methods that you showed across tables 4, 6 and 7? Could an expanded and consistent set of configurations be shown?
Questions for authors

Please refer to questions, clarifications and feedback in weaknesses.

Reviewer_5dvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a sampling method that combines Determinantal Point Processes (DPP) and Fisher geometry to enhance the diversity of model outputs. Fisher geometry is used to measure the similarity between probability distributions across different contexts, while DPP serves to prevent the selection of similar tokens. Experimental results demonstrate that the proposed method improves solution coverage in multi-solution reasoning tasks.

Critical strengths
  1. Related work provides a systematic summary of methods for promoting diversity in generation, including inference-time techniques, approaches that combine multiple parallel generation streams, and methods based on embedding spaces.

  2. To promote diversity in the generation, the paper explores and utilizes context-dependent similarity measures in the unembedding space.

  3. Figures 1 and 2 intuitively demonstrate the impact of the proposed sampling method on generation, helping readers better understand the concept of Fisher geometry.

Critical weaknesses
  1. The practical contribution of the Fisher Diverse Sampling proposed in this paper remains unclear. While the authors employ Fisher geometry to measure the similarity of probability distributions across different contexts and integrate this with DPP sampling, it is not explicitly stated what specific improvements or contributions this combination offers over the existing underlying theories.

  2. The benefits of using Fisher geometry are not sufficiently demonstrated, and its general applicability is difficult to substantiate. Among the five tasks presented in Table 3, the advantage of Fisher k-DPP over the simpler PPR baseline is essentially confined to a single task (FACTOR: +0.412 vs. +0.096). For the remaining tasks, the two methods perform comparably (SUBSET: +0.243 vs. +0.241; GRAPH: +0.071 for both; CONSTR: +0.053 vs. +0.041), while neither shows any improvement on the SUM task. In other words, the marginal gains provided by context-dependent Fisher geometry are highly task-specific, making it difficult to argue for a general advantage over simpler repulsion-based methods.

  3. The paper lacks sufficient validation regarding semantic diversity. Although the core motivation of this study is to enhance semantic diversity in open-ended generation tasks, the quantitative experiments do not directly evaluate this. All five benchmark tasks are mathematical or combinatorial in nature, with outputs constrained to fixed-format numerical tuples; the evaluation metric (solution coverage) merely measures the number of distinct valid numerical solutions discovered. Therefore, the increase in coverage observed in the experiments reflects a rise in the number of distinct numerical answers, but it does not prove that the generated content is more semantically diverse.

Questions for authors
  1. As shown in Table 2, the performance of Fisher k-DPP is nearly identical to that of PPR in terms of the Pooled metric (+0.056 vs. +0.052), and it performs comparably across several specific tasks (SUBSET, GRAPH). Could the authors further explain under what conditions and why Fisher diversity sampling offers additional advantages compared to the simpler PPR method?

  2. This paper primarily evaluates solution coverage within the context of structured mathematical and combinatorial tasks, where distinct solutions are differentiated primarily by their numerical outputs. Could the authors provide experimental results for open-ended text generation tasks to demonstrate that the proposed method genuinely enhances semantic diversity, rather than simply increasing the number of distinct numerical solutions?

  3. The paper uses iid sampling as the primary baseline and only reports comparisons against the PPR and the truncation-only baseline. It lacks comparisons with other state-of-the-art diversity-promoting decoding methods (such as min-p sampling and contrastive search). Could the authors provide empirical evidence demonstrating the superiority of the proposed method over other approaches?

  4. The quantitative experiments were conducted solely using the Gemma-4 E2B model. Would the authors consider extending the evaluation to other backbone models of varying scales or architectures to verify that the proposed method generalizes beyond a single model?

Reviewer_apjAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Fisher Diverse sampling, an inference-time method for generating a batch of semantically diverse responses from a language model. Rather than sampling each response independently, the method couples generation across batch elements using a determinantal point process. Its key technical idea is to derive a context-dependent similarity measure between candidate tokens from the Fisher information geometry of the model’s predictive distribution. At each decoding step, later trajectories are repelled from earlier trajectories according to this semantic similarity, while the base model probabilities serve as candidate-quality scores.

Critical strengths
  1. The Fisher-geometric construction is novel and well motivated.
  2. The paper takes the temperature confound seriously, and the experiments include useful controls.
  3. The sample-efficiency result is promising
Critical weaknesses
  1. I tried the prompt in figure 1 with ChatGPT-6.1-sol, everytime I am getting a different answer. I guess the author might need to come up with stronger motivation beyond figure 1.
  2. Limited evidence that Fisher geometry is necessary. The results more strongly support coupled sampling than the proposed semantic geometry. The much simpler PPR baseline performs similarly on most tasks and nearly matches Fisher k-DPP in the pooled results. The clearest advantage of Fisher geometry appears only on factor pairs.
  3. The evaluation does not fully test semantic diversity. Most benchmarks involve short, structured integer tuples where different solutions naturally correspond to different tokens. Token-level repulsion is therefore already well aligned with the evaluation objective. More open-ended tasks, where semantically distinct solutions cannot be identified from token differences alone, would better support the paper’s central claim.
Questions for authors
  1. What happens if the batch order is randomly permuted at every decoding step, or if several random orderings are averaged?
Reviewer_fSzAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Fisher Diverse Sampling, an inference-time method that couples autoregressive generation across a batch to encourage diverse outputs. It combines a context-dependent similarity measure derived from the model’s Fisher geometry with DPP-based repulsion. Experiments on generated multi-solution tasks show improvements in solution coverage over temperature sampling.

Critical strengths

Interesting problem and clear motivation. Redundant outputs can limit the effectiveness of repeated sampling. Coordinating generation across a batch to discover more distinct valid solutions is a meaningful research direction.

An interesting methodological contribution. Using contextual Fisher geometry over the model’s own unembedding vectors to guide diverse sampling is a novel and appealing design. It avoids requiring an external semantic encoder or additional model training.

Critical weaknesses

The comparison with temperature sampling does not fully establish the coverage–validity trade-off. The paper appropriately evaluates eight baseline temperatures from 0.5 to 2.5 with relatively dense points, but its frontier-gap metric uses discrete operating points whose validity is at least that of the evaluated method. Consequently, the reported gap can be sensitive to temperature spacing and does not necessarily compare methods at matched validity. Full per-task coverage–validity curves and denser temperature sweeps around the relevant operating points would make the comparison more convincing.

Limited analysis of the lack of Pass@k improvements. The paper reports minimal Pass@k gains on an initial subset of standard single-answer mathematics benchmarks, but does not identify these benchmarks or provide detailed results and experimental settings. Authors attribute the limited gains to problems being either consistently solved or consistently failed by the base model, without quantitative evidence supporting this explanation. As a result, it remains unclear whether the limited gains reflect the benchmark difficulty distribution or a limitation of the proposed diversity mechanism, and when increased diversity can improve downstream problem-solving success.

The strongest results rely on task-specific hyperparameter selection. Temperature, repulsion strength, and probability thresholds are selected separately for each task. The shared-configuration results and within-task held-out evaluation partially address this concern, but it remains unclear how to select a configuration for an unseen task without task-specific validation. Moreover, under the shared configuration evaluation, Fisher sampling and the simpler PPR baseline achieve similar pooled frontier gaps, leaving the practical advantage of the more elaborate method uncertain.

Questions for authors
  1. Could the authors clarify which standard mathematics benchmarks were used in the preliminary Pass@k evaluation and report the corresponding results and experimental settings? It would also be helpful to explain why the observed improvements in diversity and multi-solution coverage did not translate into meaningful Pass@k gains.
  2. Do the main coverage gains persist with a denser temperature baseline, particularly around the relevant validity levels? How well does a configuration selected on other tasks transfer to an unseen task?
  3. What is the actual decoding overhead relative to independent sampling and PPR, including end-to-end latency, throughput, and additional memory at the evaluated batch sizes?

I would consider raise my score if the authors can address my questions clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.