- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces Fisher Diverse sampling, an inference-time method that encourages a language model to generate semantically diverse samples within a batch via determinantal point processes. The Fisher information of the language model output distribution is used as a context-dependent estimation of similarity on the token unembeddings, and a replusion hyperparameter is used to control the level of next-token repulsion of the current batch element vs tokens already sampled in the earlier elements of the batch.
Critical strengths
- The method induces token-level repulsion from the model output distribution and unembedding vectors instead of using external models
- Interesting application of Fisher information, incrementally comparing the current to-sample token vs already-sampled tokens in the batch (though a potential risk being compute requirement)
Critical weaknesses
- The range of model types and sizes used are very narrow; only one model Gemma-4 E2B is used.
- The tasks used are also limited to synthetic math tasks, mainly combinatorial and enumerative.
- L338: Regarding the baseline temperature frontier, apart from comparing directly with iid sampling, are there other comparable baselines that primarily promote batch diversity? Or even sequential iid sampling using prior model responses as input context (potentially practical though not novel given the tasks you designed).
- It seems Fisher Diverse sampling is highly sensitive to temperature and truncation threshold, even when the task suite is already limited to the structured math problems (Table 3). How does it scale to other datasets/tasks?
- Fisher Diverse sampling consistently underperforms against the temperature frontier on the SUM sub-task in tables 4, 6 and 7, even when all tasks are designed to allow numerous correct answers.
- How is the pooled metric calculated in tables 4, 6 and 7? It may be biased by tasks with a larger population size.
- L468: Unclear outperformance statement - Fisher Diverse at B=4 did not outperform the iid baseline at B=32 in terms of validity.
- Table 5: What is the temperature used for the Fisher Diverse sampling methods, or are they temperature-tuned? Apart from comparing with iid sampling at T=1.0, were temperatures tuned in this experiment?
- Besides the structured and narrow problems in L756-L781 used that admit large answer sets, can datasets/tasks outside of these synthetic problems be explored? They can potentially require reasoning as well.
- Given the algorithmic complexity in L232-240, are there analyses of Fisher Diverse sampling's computational and memory efficiency, and comparison with the baselines?
- On the datasets/tasks used, there are no illustrative examples of solution coverage and validity of Fisher Diverse sampling vs the baselines; please provide.
- What are the reasons for showing non-consistent configurations for the Fisher, PPR and Trunc-only methods that you showed across tables 4, 6 and 7? Could an expanded and consistent set of configurations be shown?
Questions for authors
Please refer to questions, clarifications and feedback in weaknesses.