- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines the trade-offs between generation quality and diversity in LLM-based analogy generation. They use three datasets (AnaloBench, Metaphoric Analogies, MUNCH) and ten open- and closed-source LLMs. The authors measure sentence-level diversity (a weighted mix of embedding cosine distance, Self-BLEU, and EAD), target-domain diversity (domains extracted by GPT-4o-mini), and LLM-based clustering of target domains with GPT-4o-mini as a judge. They showed that 1. LLMs show “domain homogeneity”. 2. Inference-time diversity methods trade quality for diversity.
Critical strengths
- Diversity in analogy generation is an interesting question. It is critical to many very relevant domains, like autoresearch, creative writing, and even RSI in some sense.
- This paper covers many models and diversity methods, which further shows a comprehensive evaluation of these methods.
- Most of the details are included in the appendix.
Critical weaknesses
- First, it is about the results; I do not think the main idea (quality and diversity trade-offs for analogy generation) is supported well by the paper. I observe that some results show they could achieve similar quality w/o sacrificing diversity. In that case, it seems like the current model does not have such trade-offs with better harnessing.
- Second, it is about the implementation details. Basically, the authors use gpt-4o-mini as a judge model across the experiment to evaluate more capables models such as gpt5.2, grok, etc. I think either a study to show that this model is capable enough for such tasks or replacing it with more capable models would be acceptable.
Questions for authors
See the above weakness section.
Rebuttal
Weakness 1:
Response:
We thank the reviewer for raising this question. We note that our reported results in the paper together with the additional experiments and human evaluation results in our previous rebuttal response reflects the quality-diversity tradeoff from multiple perspectives. Specifically, our response to the fifth question from Reviewer MrdT regarding human evaluation on the potential target domain range reveals humans could achieve a higher number of clusters and indeed LLMs have limitations on diversity generation. Additionally, we further extend our evaluation setup from distributional diversity to guided inference diversity as is elaborated in our response text to the first question by Reviewer 8cZp.
To sum up, existing methods can mitigate homogeneity, but larger diversity gains generally incur quality costs under the evaluated settings. We will clarify the distinction between distributional diversity and coordinated generation, make the empirical scope of our conclusion explicit, and incorporate the additional evidence into the revised manuscript.
Weakness 2:
Response:
We thank the reviewer for raising this important question. Regarding the validity of using GPT-4o-mini as a judge, we include two extra experiments, including adding different LLMs into comparison as well as using human evaluations on the correctness of clustering.
For adding different LLMs, we include below another two models as additional judges, namely Gemini 2.5 Flash-Lite and DeepSeek V3.2. We rate the outputs on 150 samples, evenly drawn among the three datasets across the same three prompting settings in our paper. The temperature of the judges is set to zero, and the rating ranges from 1-5.
Table 2 reports the average quality rating using these LLM judges. Among all three judges, GPT-5.2 obtains the highest mean quality, and the rankings of these two judges shares a high spearman correlation with GPT-4o mini (last row in Table 2). These results mitigate concerns about GPT-4o mini’s preference for its own model family, as both additional judges also rank GPT-5.2 highest in mean quality.
Table 2: Mean quality rating on the model outputs (n=150 samples). The bolded number corresponds to the highest mean quality rating. We also report Spearman correlation between each judge’s five-generator ranking and that of GPT-4o mini.
For human evaluation, we select 50 distinct source–model output sets from the model outputs (T=0.7): five per generator across ten models, comprising 17 AnaloBench, 17 MA and 16 MUNCH sources. During the annotation, each annotator reviewed the source sentence and the output clusters of the 10 independent generations. For correctness, they are asked to provide a “Yes / No” answer on whether the clustering is correct; for diversity, they are asked to provide a scale between 1-5 (low to high) on cluster diversity. We collected a total of 150 judgments and calculated the accuracy and agreement in Table 5.
We observe a high accuracy (80% of clustering correctness on gpt-4o-mini) and satisfied agreement rate between the annotators (62.7% of agreement), meanwhile the diversity of the output clusters remain relatively low, with an average number of 2.85. We observe that although the annotator shared a high agreement with respect to the correctness, the diversity agreement varies between them. However, this is not undermine our observation as human reported a high expectation of target domain diversity as is showcased in our previous answer in Table 4. We will include the extra human evaluation results on clustering correctness and diversity in our paper revision.
Table 5: Human evaluations of clustering correctness and target-domain diversity (50 sets; three annotators; 150 ratings per question).