- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether images helps LLMs write more divergent poems compared with giving it a preceding text. By evaluating on 1206 poems generated by two Gemini and two Qwen models, visual recoding improves lexical density and greater distance from the seed verse, but also produces less lexical diversity and no significant improvement in divergent semantic integration. The evidence shows a change in style and distance more clearly than improvement in creativity.
Critical strengths
- The paper experiments an interesting device of poem continuation, i.e., passing meaning through an image instead of a seed verse.
- The paper examines multiple levels of evaluation, including quantitative analysis of lexical features and semantic distance, and also qualitative assessment by an LLM judge.
- The paper also compares the LLM-generated verses with a translated human-written corpus.
Critical weaknesses
- Divergence is not really creativity, and greater distance from the seed or the text-based generation also does not imply stronger creativity. The paper finds that visual recoding improves lexical density and semantic distance, but reduces lexical diversity and complete clausal structure, and gets lower scores on all dimensions in the LLM evaluation. This finding does not support the claim of the paper that visual recoding is a promising direction of improving LLM creativity.
- The models chosen are not very well controlled. In the Qwen family, text generation uses qwen2.5:7b-instruct whereas the visual mode uses qwen2.5-vl-72b-instruct, which are not comparable in terms of model size. The two families also differ in terms of additional instructions.
- As the paper suggested, visual recoding may sacrifice some aspects of quality for divergence. This is an important issue and the paper should extend its analysis, perhaps by involving human judges.
- The task scope is very limited. The paper only experimented the Japanese renga, which is a very short poetic form, and may cause problems when applying metrics that are usually used in longer texts (as stated in the limitations section).
- No related work was reviewed.
Questions for authors
- The authors should justify the temperature and top_p settings for the models and LLM judge under study.
- I found the reference of the human data as "golden corpus" problematic. It looks like it is only used as a reference corpus, instead of a sample of high-quality poems.
- The definitions and criteria of the dimensions for the LLM judge are unclear.
- In Section 2.1, the prompt given to text models seems incomplete and hard to understand.