acceptodds
Under review as a conference paper at ICLR 2027

On the Diversity of Analogy Making in Large Language Models

Abstract

Large Language Models (LLMs) have demonstrated remarkable potential for analogy making, a core cognitive capability that drives novelty and creativity. While prior research has extensively investigated the applications and underlying mechanisms of LLM-based analogy making, its output diversity remains largely unexplored, despite being essential for broadening cross-domain connections and fostering scientific innovation. In this work, we present a comprehensive evaluation of analogy diversity across ten state-of-the-art open- and closed-source LLMs. Our findings highlight a concerning issue of domain homogeneity, a prevalent tendency for LLMs to generate analogies from a narrow set of target domains, limiting both inter-query and intra-model diversity. Furthermore, our analysis reveals a fundamental trade-off in existing LLM diversity-enhancement methods: increasing output diversity often comes at the expense of output quality. Finally, our causal analysis of LLM information flow reveals substantial differences in the model-sensitive regions governing analogy diversity across LLMs, suggesting a potential mechanism for the observed diversity-quality trade-off. To our knowledge, this is among the first studies to systematically investigate output diversity in LLM-based analogy making.

Primary area:
interpretability and explainable AI
Keywords:
Large language models, cognitive science, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hwJyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines the trade-offs between generation quality and diversity in LLM-based analogy generation. They use three datasets (AnaloBench, Metaphoric Analogies, MUNCH) and ten open- and closed-source LLMs. The authors measure sentence-level diversity (a weighted mix of embedding cosine distance, Self-BLEU, and EAD), target-domain diversity (domains extracted by GPT-4o-mini), and LLM-based clustering of target domains with GPT-4o-mini as a judge. They showed that 1. LLMs show “domain homogeneity”. 2. Inference-time diversity methods trade quality for diversity.

Critical strengths
  • Diversity in analogy generation is an interesting question. It is critical to many very relevant domains, like autoresearch, creative writing, and even RSI in some sense.
  • This paper covers many models and diversity methods, which further shows a comprehensive evaluation of these methods.
  • Most of the details are included in the appendix.
Critical weaknesses
  • First, it is about the results; I do not think the main idea (quality and diversity trade-offs for analogy generation) is supported well by the paper. I observe that some results show they could achieve similar quality w/o sacrificing diversity. In that case, it seems like the current model does not have such trade-offs with better harnessing.
  • Second, it is about the implementation details. Basically, the authors use gpt-4o-mini as a judge model across the experiment to evaluate more capables models such as gpt5.2, grok, etc. I think either a study to show that this model is capable enough for such tasks or replacing it with more capable models would be acceptable.
Questions for authors

See the above weakness section.

1 comment

Weakness 1:

Comment: “Further justification on the quality and diversity trade-offs for analogy generation as well as trade-offs with better harnessing.”

Response:

We thank the reviewer for raising this question. We note that our reported results in the paper together with the additional experiments and human evaluation results in our previous rebuttal response reflects the quality-diversity tradeoff from multiple perspectives. Specifically, our response to the fifth question from Reviewer MrdT regarding human evaluation on the potential target domain range reveals humans could achieve a higher number of clusters and indeed LLMs have limitations on diversity generation. Additionally, we further extend our evaluation setup from distributional diversity to guided inference diversity as is elaborated in our response text to the first question by Reviewer 8cZp.

To sum up, existing methods can mitigate homogeneity, but larger diversity gains generally incur quality costs under the evaluated settings. We will clarify the distinction between distributional diversity and coordinated generation, make the empirical scope of our conclusion explicit, and incorporate the additional evidence into the revised manuscript.

Weakness 2:

Comment: “Second, it is about the implementation details... I think either a study to show that this model is capable enough for such tasks or replacing it with more capable models would be acceptable.”

Response:

We thank the reviewer for raising this important question. Regarding the validity of using GPT-4o-mini as a judge, we include two extra experiments, including adding different LLMs into comparison as well as using human evaluations on the correctness of clustering.

For adding different LLMs, we include below another two models as additional judges, namely Gemini 2.5 Flash-Lite and DeepSeek V3.2. We rate the outputs on 150 samples, evenly drawn among the three datasets across the same three prompting settings in our paper. The temperature of the judges is set to zero, and the rating ranges from 1-5.

Table 2 reports the average quality rating using these LLM judges. Among all three judges, GPT-5.2 obtains the highest mean quality, and the rankings of these two judges shares a high spearman correlation with GPT-4o mini (last row in Table 2). These results mitigate concerns about GPT-4o mini’s preference for its own model family, as both additional judges also rank GPT-5.2 highest in mean quality.

Model nameGPT-4o miniGemini 2.5 Flash-LiteDeepSeek V3.2
Claude Sonnet 4.54.1344.2414.197
Gemini 2.5 Pro3.9603.9504.104
Gemini 3.1 Pro4.0574.1894.227
Grok 4.53.9944.2773.980
GPT-5.24.3594.4374.257
Spearman correlation–0.700.80

Table 2: Mean quality rating on the model outputs (n=150 samples). The bolded number corresponds to the highest mean quality rating. We also report Spearman correlation between each judge’s five-generator ranking and that of GPT-4o mini.

For human evaluation, we select 50 distinct source–model output sets from the model outputs (T=0.7): five per generator across ten models, comprising 17 AnaloBench, 17 MA and 16 MUNCH sources. During the annotation, each annotator reviewed the source sentence and the output clusters of the 10 independent generations. For correctness, they are asked to provide a “Yes / No” answer on whether the clustering is correct; for diversity, they are asked to provide a scale between 1-5 (low to high) on cluster diversity. We collected a total of 150 judgments and calculated the accuracy and agreement in Table 5.

We observe a high accuracy (80% of clustering correctness on gpt-4o-mini) and satisfied agreement rate between the annotators (62.7% of agreement), meanwhile the diversity of the output clusters remain relatively low, with an average number of 2.85. We observe that although the annotator shared a high agreement with respect to the correctness, the diversity agreement varies between them. However, this is not undermine our observation as human reported a high expectation of target domain diversity as is showcased in our previous answer in Table 4. We will include the extra human evaluation results on clustering correctness and diversity in our paper revision.

Metricn=50 samples; 3 annotators
Accuracy (Correct + Partly correct)120/150 (80%)
Correctness agreement (binary; pairwise exact; Fleiss’ κ)94/150 (62.7%); κ=−0.167
Mean diversity rating (1–5; mean [SD])2.85 (1.14)
Diversity agreement (pairwise exact; mean weighted κ)37/150 (24.7%); κ=0.041

Table 5: Human evaluations of clustering correctness and target-domain diversity (50 sets; three annotators; 150 ratings per question).

Reviewer_8cZpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate whether LLMs can generate diverse cross-domain analogies rather than repeating the same stereotypical target domains across ten open-weight and proprietary models on AnaloBench, Metaphoric Analogies, and MUNCH. Sampling 10 analogies per input, they find that models typically collapse onto just 2 to 3 semantic target domains and show high distributional overlap across model families. They then evaluate several prompting and inference-time decoding strategies, finding a consistent trade-off between diversity and LLM-judged quality, and run a layer-wise perturbation experiment to test which internal layers affect this balance.

Critical strengths

Looking at target-domain diversity in open-ended analogy generation is a nice step beyond standard multiple-choice analogy benchmarks. Separating surface lexical variation from conceptual domain diversity via domain extraction and clustering (validated with human agreement) makes a convincing case that LLMs default to a narrow set of target domains.

Combining intra-model clustering with inter-model MAUVE comparisons across frontier models cleanly shows that domain homogeneity is shared across model families rather than unique to one post-training recipe.

Critical weaknesses
  1. If I am reading Appendix H correctly, each prompt asks the model for a single sentence, and the 10 candidates are drawn via independent sampling at T=0.7. Because each forward pass has no memory of the other 9 samples, telling a single call to "be diverse" cannot coordinate diversity across independent draws; it just shifts the prior toward whatever single domain the model deems creative.

  2. In Appendix B, the four decoding methods (Top-k, Top-p, Top-eta-sigma, and Min-p) appear to be run at T=1.5 with the base prompt, whereas the prompting baselines use T=0.7. Since tail truncation reduces variance at a fixed temperature, the higher diversity and lower quality in Section 5 seem driven by raising the temperature to 1.5 rather than by truncation itself. Also, G2 drops to a quality score of ~1.0 (the minimum on the 1 to 5 scale) in Table 1, which looks like degenerate text rather than meaningful diversity.

  3. I am not fully convinced that the layer perturbation experiment in Section 6 isolates an analogy-making region. Selecting layers that maximize (5 - Quality) * Diversity under hidden-state noise directly rewards breaking generation quality toward 1 while inflating surface variation. Without a non-analogy control task or showing that steering those layers improves the quality-diversity frontier, any layer whose corruption breaks model coherence would score high on this metric.

Questions for authors
  1. Did you test prompting the models to generate all 10 analogies from distinct domains in a single turn (or sequentially conditioned on previous outputs), and does domain homogeneity still hold when the model can see its prior choices?

  2. Did you evaluate standard un-truncated sampling at T=1.5, or Top-k / Top-p / Min-p at T=0.7? If I am misreading Appendix B and you actually ran an unconfounded temperature comparison, please let me know, as that would address my main reservation about Section 5.

  3. Could you clarify a few setup details that differ between the main text and appendix: specifically, whether the main diversity metric is pure cosine distance (Section 3) or the weighted composite with Self-BLEU and EAD (Appendix B), which sample counts were used for MA and MUNCH (Section 3 vs. Appendix A), and how the online entropy threshold and Noise Blur / looped perturbations are computed?

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Weakness 1:

Comment: “If I am reading Appendix H correctly, each prompt asks the model for a single sentence, and the 10 candidates are drawn via independent sampling at T=0.7. Because each forward pass has no memory of the other 9 samples, telling a single call to "be diverse" cannot coordinate diversity across independent draws; it just shifts the prior toward whatever single domain the model deems creative.”

Question: “Did you test prompting the models to generate all 10 analogies from distinct domains in a single turn (or sequentially conditioned on previous outputs), and does domain homogeneity still hold when the model can see its prior choices?”

Response:

We agree with the reviewer that coordination between previous generations could affect the diversity of outputs. However, evaluations conducted upon memorizing the previous outputs does not reflect the diversity built into the model’s own output distribution. In fact, existing works often differentiate diversity evaluation into distributional diversity and inference guided diversity [2]. In our work, we follow these distributional diversity setups, which are also used in prior works that measure output novelty in LLMs [1]. Furthermore, in real-world applications, accessing every decision step in LLMs is increasingly limited (e.g. when the use of analogy is embedded in a multi-step task), particularly when LLM systems become increasingly complex, where the steps are executed until when the results are aggregated and coordinated. In addition, most proprietary LLMs do not allow for accessing intermediate reasoning steps. As a result, the intermediate steps where diversity is required are often not accompanied by memories, which is mirrored in our evaluation setup that studies the distributional diversity of models.

Despite a difference in the scope of our evaluation, we follow the reviewer’s suggestion and run experiments under the guided diversity setup: the model writes 10 short analogies (at most 20 tokens each) in one forward pass, and we sample ten independent responses per source sentence, which yields 5,000 responses and 50,000 analogies in total. We use the same temperature settings (T=0.7) for four models. The results are reported in Table 6.

We observe a numerical increase in target domain clusters, with an average of 6.27 clusters out of ten. However, the issue of domain homogeneity remains notable, with GPT-5.2 achieving an average cluster of 3.85 and the linkage of 0.602, indicating that the models still suffer from outputs that lack diversity despite a switch in generation setting. Besides, the responses keep returning to the same domains: two responses of the same model to the same source sentence share 51.7% of their target domains (intra-output overlap) and the responses of two different models share 33.0% (inter-model overlap), compared with 11.1% for responses to two different source sentences (random baseline).

Therefore, although guided generation diversifies the analogies inside one response, the set of domains that the models draw from still fall short in that of humans, stays narrow, and overlaps substantially across responses and across models, which is consistent with the domain homogeneity reported in our paper. We will update our justifications of diversity evaluation in our revision and add these follow-up experiments in our appendix section.

ModelAvg. ClustersAverage linkageDomain diversityIntraoutput overlapIntermodel overlapRandom baseline
Claude Sonnet 4.56.541.0700.63153.7%35.4%10.8%
Gemini 2.5 Pro7.011.1160.65145.5%31.6%12.6%
Gemini 3.1 Pro7.801.1670.71948.7%31.6%12.7%
Grok 4.56.140.9400.60547.6%35.0%10.8%
GPT-5.23.850.6020.38562.9%31.5%8.4%
Mean6.270.9790.59851.7%33.0%11.1%

Table 6: Additional results on guided inference setup. Clusters merge at cosine distance ≤0.4; average linkage uses cross-cluster sentence-pair Euclidean distances (0 for one cluster).

References:

[1] Zhang, Y. et al. NoveltyBench: Evaluating Language Models for Humanlike Diversity.

[2] Zhang, J. et al. Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity.

AuthorsOpenReview ↗

Rebuttal Part 2

Weakness 2:

Comment: “In Appendix B, the four decoding methods... Also, G2 drops to a quality score of ~1.0 (the minimum on the 1 to 5 scale) in Table 1, which looks like degenerate text rather than meaningful diversity.”

Question: Elaboration on standard un-truncated sampling at T=1.5, or Top-k / Top-p / Min-p at T=0.7 in Appendix B

Response:

We thank the reviewer for this question. We hereby clarify that Top-k / Top-p / Min-p all belong to temperature tuning techniques, and these methods achieve a higher output diversity by directly tuning up the temperature. In other words, if we use uniform temperature, all these methods achieve similar outcomes when compared with untruncated baselines. Additionally, the selection of T=1.5 and T=0.7 for non temperature scaling methods are supported in Table 1 in the original G2 paper [1], and the authors specifically justified the validity of using varied temperature in their comparison of output diversity.

To further justify the necessity of using higher temperature in these methods, we add the additional experiment in Table 7 that evaluates the three methods when fixed to T=0.7, Top-k=20, Top-p=0.9 and Min-p=0.1 on 100 AnaloBench sources using Llama-3.1-8B-Instruct, ten outputs per cell and a shared judge. We can see that under the same temperature scale, different sampling methods has marginal difference in output quality and diversity, highlighting the necessity of increased temperatures when revealing our argument on quality-diversity tradeoff. We will add this one-model controlled study, its scoring protocol and explicit temperature settings in the appendix.

MethodQuality, T=0.7D_total, T=0.7Quality, T=1.5D_total, T=1.5
Untruncated4.1260.4281.9900.794
Top-k (k=20)4.1800.4173.9660.650
Top-p (p=0.9)4.2540.3733.0720.757
Min-p (p=0.1)4.3170.3424.0540.550

Table 7: Mean quality and sentence D_total of each decoder at T=0.7 and T=1.5 (92 complete sources).

References:

[1] Ruan Z. et al. G2: Guided Generation for Enhanced Output Diversity in LLMs. In EMNLP 2025.

Weakness 3:

Comment: Clarification on the layer perturbation experiment in Section 6 in isolating an analogy-making region. “Selecting layers... on this metric.”

Response:

We thank the reviewer for this question. In our equation that probes the sensitive regions in analogy making, we note that since quality and diversity are evaluated specifically for analogy making. Also, as our earlier responses (i.e. our responses in R1Q2 and R1Q6) provide evidence for the reliability of the LLM judge, their combination offers an interpretable measure of analogy sensitivity with the potential to identify analogy-sensitive regions. We will add this rationale to the revised manuscript.

Following the similar rationale in our clustering of target domains, we also conducted a follow-up analysis informed by recent work on representation geometry within inference-trajectory [1, 2]. We extended clustering into the model’s hidden-state during generation, applying the original layerwise cosine-distance DBSCAN setup (ε at the fifth distance percentile; min_samples=10) to saved generated-token states. We also measured mean pairwise cosine distance to quantify variation directly, since short sequences can yield all-noise cluster assignments. This diagnostic compares analogy, creative-writing and summarization prompts for one shared AnaloBench source in Llama-3.1-8B-Instruct. Specifically, we use the same source sentence from AnaloBench and modify prompts for different purposes: creative writing asks for one creative sentence inspired by it, while text summarization asks for a one-sentence summary.

In the final third of layers, analogy states were more concentrated than those under the creative prompt: mean cosine distances were 0.584 versus 0.672 for Llama, with the same ordering in equal-token-prefix comparisons (Table 7). This suggests restricted late-layer representational exploration as a possible factor in limited analogy diversity. This single-source diagnostic remains preliminary. We will include this analysis and its protocol in the revision.

ModelLayer bandSummary promptCreativity taskAnalogy task
Llama-3.1-8BEarly (1–11)0.6360.7260.704
Llama-3.1-8BMiddle (12–22)0.5810.6680.634
Llama-3.1-8BLate (23–32)0.5960.6720.584

Table 7: Mean pairwise cosine distance between generated-token states, averaged over each third of decoder layers (higher = greater geometric dispersion).

References:

[1] Park, Choe and Veitch. [The Linear Representation Hypothesis and the Geometry of Large Language Models]. ICML 2024.

[2] Ma et al. [Reasoning emerges from constrained inference manifolds in large language models].

AuthorsOpenReview ↗

Rebuttal Part 3

Weakness 4:

Question: “Could you clarify a few setup details that differ between the main text and appendix: specifically, whether the main diversity metric is pure cosine distance (Section 3) or the weighted composite with Self-BLEU and EAD (Appendix B), which sample counts were used for MA and MUNCH (Section 3 vs. Appendix A), and how the online entropy threshold and Noise Blur / looped perturbations are computed?”

Response:

We thank the reviewer for requesting these clarifications. Figure 3(a) and the Div. columns use D_total=0.5D_cos+0.25(1−Self-BLEU)+0.25D_EAD, computed on sentences or extracted target-domain descriptions. Source counts (AnaloBench/MA/MUNCH) are 340/262/300 for prompting/decoding and 300/262/300 for hidden-state experiments (253 MA sources for Mistral); invalid quality scores are excluded from the means. The looped procedure passes selected-layer states through the model once more, matches the output to their original norm, and blends it back. Entropy steering adds an editing instruction and triggers when H>τ and P_domain>0.6. Hyperparameters are: ten unperturbed completions per source; entropy over renormalized top-200 probabilities, divided by log 200; τ at the 90th percentile; Noise Blur α=0.5 for the first ten tokens; and entropy-steering α_n=0.5max(1−0.03n,0.1) within the first 40 tokens, where n counts triggers including the current one and the first token is always steered. We will align these details in the main text and appendix.

Reviewer_MrdTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper motivates that increasing the diversity of analogies language models can generate will improve their usefulness for scientific discovery. The paper employs a mixture of embedding, clustering and LLM-as-a-judge-based diversity metrics and analyzes the impact of prompting and sampling-based methods to increase diversity. The findings show that increasing diversity leads to lower analogy quality, which the authors propose could be due to the models’ analogy skills originating in lower model layers, whereas the diversity-increasing methods are applied to the input (prompting-based) or the ouput logits (sampling-based).

Critical strengths
  • The motivation on why increasing analogy diversity would be beneficial is well-motivated, although the exact impact of models’ analogical understanding on scientific discovery capabilities remains unquantified.
  • Measuring analogy diversity not only for each model separately but also comparing across models.
Critical weaknesses
  • the novel parts of the analogy diversity metrics are not sufficiently explained
  • the gpt-4o-mini judge ends up scoring the gpt-5.2 model’s outputs as highest quality —> most model families prefer their own outputs. Can the results be validated by a second family's judge model?
  • the paper could more clearly connect the approaches and their impact on diversity (e.g. different prompt types asking specifically for more diverse ouputs) to what has been found in the general text generation domain. Do these approaches consistently lead to quality trade-offs in the general domain too? Why would you expect different outcomes for the analogy domain? If you did not, this expectation could more clearly be explained in the experimental setup.
  • The data contamination analysis is not fully convincing. It can be expected that the closed-source models have seen the most common analogies. Could you provide more insight into how much the reconstruction method varies per sample? Are there samples that can be more or less fully reconstructed, especially for common analogies for which it can be assumed that models have seen them?
Questions for authors
  • Could you explain more how you adapted the standard text generation diversity metrics to the analogy domain and clearly state which parts have already been used in prior work and where your novel contribution to these metrics comes in?
  • The lack of analogy diversity in the models is framed as a limitation. How much diversity can we reasonably expect there to be? Is it sensible to demand that each generation maps to completely disjoint target domains? The 10 analogy generations per method are sampled independently, so the model cannot even avoid previously generated analogies. Would the results change if you kept all previously generated analogies in the context?
  • Wrt the human evaluation you write: “Based on a randomly sampled subset comprising 10% of the data, the human annotations agreed with the automatic clustering in 95% of the cases.” How many samples are these percentages in absolute numbers?
  • Could you provide more information on the human evaluation of the clusters? E.g., how many humans reviewed the clusters? Where did they disagree with the model?
  • Are there samples that can be more or less fully reconstructed, especially for common analogies for which it can be assumed that models have seen them?
  • Why was the looped computation method chosen over other probing methods for identifying layers relevant to analogical understanding?

Additional feedback (not necessarily affecting reviewing recommendation):

  • related work: you mention creative writing twice with separate references
  • l. 28: Our code is available
  • Virtually all of your citations should appear in parentheses but are just in plain text. Use \citep{} to put citations into parentheses (e.g. l. 51, but present throughout the whole paper)
  • l. 88: “…, using novel and multi-faceted metrics …”. No “a”
  • l. 91: “… based on both intra-query and intra-model perspectives.” Perspectives plural

If my questions are addressed I will increase my score.

5 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Weakness 1:

Comment: Explain more on the novel parts of the analogy diversity metrics

Question: “Could you explain more how you adapted the standard text generation diversity metrics to the analogy domain and clearly state which parts have already been used in prior work and where your novel contribution to these metrics comes in?”

Response:

Thank you for your question. To the best of our knowledge, our work is among the first to propose both quantitative and qualitative evaluations of diversity in analogy making of LLMs. Previous studies typically evaluate diversity in generated text, potentially capturing surface-level lexical variations rather than genuine conceptual diversity. In contrast, our approach explicitly distinguishes source and target domains, focusing on the conceptual leap involved in analogical reasoning. Through domain extraction and clustering, we evaluate conceptual diversity rather than merely lexical diversity. Moreover, clustering allows us not only to quantify diversity, but to characterize the range of conceptual domains explored by LLMs. Thus, we highlight that the domain-level evaluation of diversity in analogical reasoning and our successive findings in LLM domain homogeneity constitute the key novelty of our work. We will add the above additional descriptions emphasizing the novelty of this evaluation in our introduction.

Weakness 2:

Comment: “the gpt-4o-mini judge ends up scoring the gpt-5.2 model’s outputs as highest quality —> most model families prefer their own outputs. Can the results be validated by a second family's judge model?”

Response:

We thank the reviewer for raising this important question. We include below another two models as additional judges, namely Gemini 2.5 Flash-Lite and DeepSeek V3.2. We rate the outputs on 150 samples, evenly drawn among the three datasets across the same three prompting settings in our paper. The temperature of the judges is set to zero, and the rating ranges from 1-5.

Table 2 reports the average quality rating using these LLM judges. Among all three judges, GPT-5.2 obtains the highest mean quality, and the rankings of these two judges shares a high spearman correlation with GPT-4o mini (last row in Table 2). These results mitigate concerns about GPT-4o mini’s preference for its own model family, as both additional judges also rank GPT-5.2 highest in mean quality. We will add the descriptions in our evaluation setup section and include the detailed comparison in our appendix.

Model nameGPT-4o miniGemini 2.5 Flash-LiteDeepSeek V3.2
Claude Sonnet 4.54.1344.2414.197
Gemini 2.5 Pro3.9603.9504.104
Gemini 3.1 Pro4.0574.1894.227
Grok 4.53.9944.2773.980
GPT-5.24.3594.4374.257
Spearman correlation–0.700.80

Table 2: Mean quality rating on the model outputs (n=150 samples). The bolded number corresponds to the highest mean quality rating. We also report Spearman correlation between each judge’s five-generator ranking and that of GPT-4o mini.

Weakness 3:

Comment: Connecting the approaches and their impact on diversity to what has been found in the general text generation domain. Do these approaches consistently lead to quality trade-offs in the general domain too? Why would you expect different outcomes for the analogy domain? If you did not, this expectation could more clearly be explained in the experimental setup.

Response:

We thank the reviewers for raising this question. In fact, we are not claiming that our contribution leads to a quality trade-off in the general domain. Rather, we aim to offer a deeper understanding of LLM output diversity through the lens of exploiting analogies and extracting target domains, which is regarded as an important reasoning task in human cognition. Unlike previous works that evaluate diversity in general sentences, mixing creative ideas and their descriptions and potentially capturing surface-level lexical variations, we extract the target domains and explore clustering (alongside several previously proposed diversity metrics) to perform a dedicated quantitative diversity evaluation in analogy making. Besides revealing a quality–diversity trade-off in analogy generation, our work enables us to evaluate how diverse LLM outputs actually are – including clustering evaluation. We believe that this approach is complementary in future work to better understand diversity in text generation. Nevertheless, we will include a clearer explanation of the scope of our research in our updated manuscript.

AuthorsOpenReview ↗

Rebuttal Part 2

Weakness 4:

Comment: Provide more insights into how much the reconstruction method varies per sample.

Question: “Are there samples that can be more or less fully reconstructed, especially for common analogies for which it can be assumed that models have seen them?”

Response:

We thank the reviewer for raising this question. We note that our contamination analysis is based on [1], which tests the exact match rate of the generations when provided with different lengths of the golden label prefix. To further support our findings, we provide the per-sample results in Table 5 using the same method on 338 samples in the first dataset.

From the results, we observe that considering the ratio 0.1 of the input sentence, only 5 to 7 of the 338 references (at most 2.1%) are reconstructed exactly, and the median token accuracy is between 0 and 0.05 for every model. For example, one dataset source describes an athlete’s career beginning with her first gym visit and progressing to world-class level. Given the prefix “A journey of a thousand”, GPT-5.2 completed it with “miles begins with a single step.” This familiar expression does not appear in the documented demonstrations.

After excluding demonstration-overlap cases and suffixes of at most three words, only one or two exact reconstructions per model remain out of 321 sources (at most 0.6%). We will add the additional description in our data contamination section and include the sample-wise contamination results in our appendix.

ModelRatio 0.1Ratio 0.2Ratio 0.4Ratio 0.8
Claude Sonnet 4.566516
Gemini 2.5 Pro75928
Gemini 3.1 Pro55425
Grok 4.565719
GPT-5.254515

Table 3: Sample-wise contamination analysis on AnaloBench (n=338 samples).

References:

[1] Golchin, S. and Surdeanu, M. [Time Travel in LLMs: Tracing Data Contamination in Large Language Models]. ICLR 2024.

Weakness 5:

Question: “How much diversity can we reasonably expect there to be? Is it sensible to demand that each generation maps to completely disjoint target domains? The 10 analogy generations per method are sampled independently, so the model cannot even avoid previously generated analogies. Would the results change if you kept all previously generated analogies in the context?”

Response: We thank the reviewer for this question. We added a blinded human evaluation of 50 source–model output sets, and asked the annotators on the potential target domain range given the source sentence to make an analogy. Three annotators independently rated the broadness of target-domain clusters range on a 1–5 scale (1 = narrow; 5 = broad), yielding 150 ratings on the same 50 sets. We observe a high rating on the diversity of target domains on the source input sentence (3.93 on broadness out of 5) with a relative high inter-annotator agreement rate (84%). Our human evaluation revealed that some LLM responses lacked sufficient diversity, with annotators indicating that they could generate more diverse responses for the same cases, suggesting that humans could achieve a higher number of clusters and indeed LLMs have limitations on diversity generation. We will include this study in the results and its protocol and full statistics in the appendix.

MetricTarget-domain expected diversity (1 = very narrow; 5 = very broad)
Mean rating (SD)3.93 (0.89)
Sources rated above 3 by ≥2 annotators42/50 (84.0%)
Table 4: Expected diversity on source sentences and agreement rate among annotators.

For the independent sampling of 10 analogy generations, please refer to our response to Weakness 1 by Reviewer 8cZp.

AuthorsOpenReview ↗

Rebuttal Part 3

Weakness 6:

Question: “Wrt the human evaluation… How many samples are these percentages in absolute numbers?”

Question: “Could you provide more information on the human evaluation of the clusters? E.g., how many humans reviewed the clusters? Where did they disagree with the model?”

Response:

We thank the reviewer for this question. We agree that our original description needs clarification: it referred to a preliminary check by a single evaluator. We will distinguish that initial check from the enhanced blinded human evaluation with 3 independent annotators and 50 samples drawn from all the datasets. We will also update the earlier “10% / 95%” description with the explicit sampling procedure and the statistics reported below.

The new blinded evaluation covers 50 distinct source–model output sets from the model outputs (T=0.7): five per generator across ten models, comprising 17 AnaloBench, 17 MA and 16 MUNCH sources. During the annotation, each annotator reviewed the source sentence and the output clusters of the 10 independent generations. For correctness, they are asked to provide a “Yes / No” answer on whether the clustering is correct; for diversity, they are asked to provide a scale between 1-5 (low to high) on cluster diversity. We collected a total of 150 judgments and calculated the accuracy and agreement in Table 5.

We observe a high accuracy (80% of clustering correctness on gpt-4o-mini) and satisfied agreement rate between the annotators (62.7% of agreement), meanwhile the diversity of the output clusters remain relatively low, with an average number of 2.85. We observe that although the annotator shared a high agreement with respect to the correctness, the diversity agreement varies between them. However, this does not undermine our observation as human reported a high rating of target domain diversity as is showcased in our previous answer in Table 4. We will include the extra human evaluation results on clustering correctness and diversity in our paper revision.

Metricn=50 samples, 3 annotators
Accuracy120/150 (80%)
Correctness agreement (binary; pairwise exact; Fleiss’ κ)94/150 (62.7%) κ=−0.167
Mean diversity rating (1–5; mean [SD])2.85 (1.14)
Diversity agreement (pairwise exact; mean weighted κ)37/150 (24.7%) κ=0.041

Table 5: Human evaluations of clustering correctness and target-domain diversity (50 sets; three annotators; 150 ratings per question).

Weakness 7:

Question: “Why was the looped computation method chosen over other probing methods for identifying layers relevant to analogical understanding?”

Response:

We thank the reviewer for this question. We selected looped computation because it provides a controlled intervention on the generation process.

First, it measures generation effects [3]: we observe changes in analogy quality and diversity, complementing the representation readouts in [1, 2]. Second, it uses the model's own computation to construct the perturbation: layer activations are passed through the same frozen model again, following the idea of reusing computation [4]. Third, it enables a common comparison across depths and models: we match the reprocessed states to the original activation norm before blending and hold the mixing coefficient fixed, without training an auxiliary probe. These properties make looped computation suitable for mapping perturbation-sensitive depths. We will add this rationale and the intervention protocol to Section 6; the scope of the sensitivity analysis is discussed in our response to R2Q3.

References:

[1] Belrose, N. et al. [Eliciting Latent Predictions from Transformers with the Tuned Lens].

[2] Ghandeharioun, A. et al. [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models]. ICML 2024.

[3] Todd, E. et al. [Function Vectors in Large Language Models]. ICLR 2024.

[4] Fan, Y., Svete, A. and Lee, K. [Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers].

Weakness 8:

Comment: “- related work: you mention creative writing twice with separate references

  • l. 28: Our code is available
  • Virtually all of your citations should appear in parentheses but are just in plain text. Use \citep{} to put citations into parentheses (e.g. l. 51, but present throughout the whole paper)
  • l. 88: “…, using novel and multi-faceted metrics …”. No “a”
  • l. 91: “… based on both intra-query and intra-model perspectives.” Perspectives plural”

Response: We thank the reviewer for pointing out these inconsistencies in citation and formatting. We will go through the details and update them in our revisions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Comment

We would like to thank all the reviewers for recognizing our work in terms of topic significance, expansiveness of experiments, and the pioneering efforts in understanding the diversity of LLM’s analogy making. We would also like to thank the reviewers for offering comments that identify the ambiguity in our description as well as raising questions that deserve further clarification. We will address these comments in our following response point by point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.