- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces C4, a cognition-inspired framework for evaluating cross-concept understanding in MLLMs by treating creative interpretation as the reversal of a cross-concept encoding process. Under this framework, a target Chinese four-character idiom is mapped through manually annotated associative bridges into indirect visual clues, and the model is asked to recover the original idiom. The authors build C4-Eval, a benchmark of 184 synthetic items and 37 web-collected human-created figures covering 84 idioms, organized into four difficulty levels and evaluated under five task settings. Experiments show a top accuracy of 50.7%, with candidate constraints substantially improving performance and open-ended answer-space search emerging as a major bottleneck.
Critical strengths
- The benchmark construction pipeline is explicit and structured. The paper makes a substantial effort to turn an otherwise subjective creative task into a reproducible evaluation protocol. The authors also distinguish image recognition, bridge reconstruction, and answer-space search through different task settings.
- The empirical analysis goes beyond a single leaderboard number. The candidate-recognition comparison and the paired T4/T5 analysis help localize model failures rather than just report overall accuracy. The large gap between candidate recognition and unconstrained recovery suggests that answer-space search is a key difficulty in this benchmark. The analysis of slot count versus bridge depth is also informative, though the authors rightly note these comparisons are not causal, since the difficulty levels differ in targets and images.
Critical weaknesses
- The construct validity of C4 as a benchmark of “creativity” is not sufficiently established.
The construct validity of C4 as a benchmark of “creativity” or “creative understanding” is not sufficiently established. The central conceptual motivation relies heavily on the associative account of creativity, particularly Mednick's notion of remote conceptual connections. However, the paper moves rather quickly from remote association to receptive creative understanding and then to a benchmark consisting almost entirely of Chinese chengyu decoding. Remote association is certainly relevant to creativity, but it is not synonymous with creativity as a broader construct, which also involves dimensions such as novelty, appropriateness, generation, flexibility, and contextual usefulness. The current evaluation therefore seems to establish that models struggle with a particular form of culturally and linguistically mediated associative decoding, rather than establishing a general deficiency in multimodal creativity.
The related-work discussion omits several directly relevant lines of research, weakening the paper's positioning of what C4 actually measures. First, it misses benchmarks of multimodal associative reasoning. [1]'s alternative operationalizations of this cognitive ability expose the key question of whether a benchmark captures general associative reasoning or merely a narrow task convention. [2] evaluates open-ended associative reasoning with human baselines, sensitivity analyses, and diversity across languages, cultures, and domains. Their inclusion of human comparisons and multi-lingual/cultural validation makes C4's lack of analogous validation especially consequential. Second, it overlooks work on associative reasoning in MLLMs such as [3] which explicitly studies this reasoning and its tension between creativity and faithfulness, and is therefore important for clarifying what capability C4 measures beyond existing evaluations of associative behavior. More broadly, the paper does not engage with recent work questioning the construct validity of applying human creativity tests to LLMs, such as [4] which shows that the link between a given test and a target creativity construct is highly test- and construct-dependent. This concern bears directly on C4: demonstrating that models struggle with one form of remote association does not by itself establish that C4 measures general multimodal creativity or "creative understanding". The authors should provide stronger construct validation and clarify the intended scope of their claims.
All targets in C4-Eval are Chinese four-character idioms, and the task design relies heavily on Chinese phonetic, lexical, cultural, and character-level associations. However, different languages have their own idioms, proverbs, fixed expressions, wordplay conventions, and culturally specific associative structures. It is unclear whether the reported failure reflects a general limitation of MLLMs in cross-concept understanding or difficulty with Chinese-specific linguistic and cultural encoding. The current evidence does not support the paper's broader framing of C4 as a general evaluation of creative understanding.
- The benchmark lacks a human baseline and does not sufficiently establish the naturalness or difficulty of its test items.
The paper evaluates ten MLLMs but does not establish how well humans can solve the same image-to-chengyu tasks under the same instructions. Consequently, a 50.7% score for the best model is difficult to interpret: it could indicate a substantial gap from human performance, near-human performance on a genuinely difficult puzzle, or simply that the benchmark contains highly artificial associations that are difficult even for humans.
Some of the intended associations appear highly non-obvious. Figure 5 relies on associations such as strawberry → grass and monk → wooden fish → wood. These are valid paths under the authors' annotation scheme, but it is not clear that they represent associations that humans would naturally infer from the image without substantial task-specific prompting or knowledge of the encoding convention. The benchmark may therefore measure the ability to reverse-engineer the authors' particular encoding system rather than a naturally occurring form of creative interpretation.
The exact evaluation prompts are not sufficiently exposed in the paper, and there is no systematic prompt-sensitivity study. The paper does report that T1-T4 use fixed prompts and that T1/T4 differences are used to examine prompt-form effects, but this does not establish robustness to alternative reasonable instructions. Given how unusual the task is, relatively small changes in instructions could plausibly have a large effect: for example, explicitly telling models to consider homophones, Chinese character decomposition, idiomatic expressions, or multi-hop associations could substantially change performance. The large improvement under candidate constraints already indicates that task formulation and answer-space restriction have a major impact. Without human baselines and prompt-sensitivity experiments, it is difficult to determine whether the reported low accuracy reflects a genuine model limitation or a benchmark-specific difficulty induced by the task interface.
[1] Li, Hong, et al. "The labyrinth of links: Navigating the associative maze of multi-modal llms." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Huang, Zimeng, et al. "MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models." Advances in Neural Information Processing Systems 38 (2026).
[3] Lyu, Xinyu, et al. "Flexac: Towards flexible control of associative reasoning in multimodal large language models." Advances in Neural Information Processing Systems 38 (2026): 2473-25
[4] Schapiro, Samuel, et al. "Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers." arXiv preprint arXiv:2605.13450 (2026).
Questions for authors
-
How well do human participants perform on T1-T4 under exactly the same images and instructions? What is the human accuracy on image-only recovery, and how does it change with the bridge hint and candidate constraints?
-
What are the exact prompts used for T1-T5? Have the authors tested reasonable alternative instructions, such as explicitly encouraging phonetic, semantic, cultural, or multi-hop associations?
-
Is there evidence that the proposed formulation transfers beyond Chinese chengyu? If not, how do the authors justify interpreting C4 as an evaluation of general MLLM creative understanding rather than Chinese-specific cross-concept decoding?
-
How does C4 differ conceptually and empirically from recent multimodal association benchmarks such as [1] and [2]? In particular, what capability does C4 measure that cannot already be captured by those evaluations?
3 comments
Response to Reviewer sLdz (Part 2/2): Exact Prompts & Human Evaluation
(Continued from Part 1/2.)
3. Exact prompts for – and prompt-sensitivity study
In our submission, the exact Chinese prompts were included in the supplementary repository (eval.jsonl), and we provide all five verbatim below (with English translations):
-
(Image-only):
你会看到一张图。请猜它对应的中文成语。只输出一个成语,不要解释。 (You will see an image. Guess the corresponding Chinese chengyu. Output only one chengyu without explanation.)
-
(Bridge hint):
你会看到一张图。这张图可能通过谐音、拆字、替换、角色联想、物品联想等跨概念方式表达一个中文成语。请根据图像线索猜成语。只输出一个成语,不要解释。 (You will see an image. This image may express a Chinese chengyu via cross-concept relations such as homophony, character decomposition, substitution, role association, or object association. Guess the chengyu from the visual clues. Output only one chengyu without explanation.)
-
(Four-way candidates):
你会看到一张图。它通过跨概念方式表达下面候选成语之一。 A. [Cand 1] B. [Cand 2] C. [Cand 3] D. [Cand 4] 请只输出最可能的成语,不要输出选项字母,不要解释。 (You will see an image expressing one of the following candidate chengyu via cross-concept relations. [A–D] Output only the most likely chengyu, without option letters or explanation.)
-
(Free answer + JSON rationale):
你会看到一张图。这张图是某个中文成语的跨概念创意表达。请先给出你认为的成语,再解释你如何从图像线索推到这个成语。必须返回 JSON,不要输出 JSON 以外的文字,格式如下: (You will see an image. This image is a cross-concept creative expression of a Chinese chengyu. First give the chengyu you believe it is, and then explain how you infer this chengyu from the visual clues. You must return valid JSON and output no text outside the JSON, in the following format:)
{ "answer": "成语 (chengyu)", "perceptual_cues": ["你看见的关键图像线索 (key visual clues you observe)"], "bridge_analysis": [ { "visual_or_textual_cue": "线索 (cue)", "bridged_concept": "被联想到的概念 (associated concept)", "idiom_part": "对应的成语片段 (aligned chengyu fragment)" } ], "final_rationale": "一句话说明整体推理 (one-sentence summary of the overall reasoning)" }(Note: parenthetical English glosses above are added here for readability; the model-facing prompt contains only the Chinese placeholder strings.)
-
(Gold answer + JSON rationale):
你会看到一张图。已知它表达的成语是“{gold_chengyu}”。请解释图像中的关键线索如何通过跨概念桥接到这个成语。必须返回 JSON,不要输出 JSON 以外的文字,格式如下: (You will see an image. Given that the chengyu it expresses is "{gold_chengyu}", explain how the key clues in the image bridge across concepts to this chengyu. You must return valid JSON and output no text outside the JSON, in the following format: [followed by the exact same JSON schema as , with
"answer"pre-filled with"{gold_chengyu}"].)
Note that already explicitly instructs models to consider homophones, character decomposition, substitution, and role/object associations, yielding modest differences ( to percentage points over ). We are running broader prompt-sensitivity experiments (step-by-step multi-hop CoT instructions, English vs. Chinese prompts, and text-only few-shot demonstrations) and will move all exact prompts and sensitivity results into the revised manuscript.
4. Independent human baseline and item naturalness
We completely agree that examples such as strawberry -> grass or monk -> wooden fish -> wood require empirical verification of whether independent humans naturally infer them without prior exposure to our annotation convention. We are actively conducting an independent human baseline across – under identical instructions alongside human naturalness/plausibility ratings, and we will update the revised manuscript with results both on the full cleaned set and stratified by human recoverability.
Thank you for your detailed response. I appreciate your acknowledgment of the construct-validity concerns. However, the main concerns remain unresolved. The human baseline and item-naturalness evaluations are still ongoing. Thus, I maintain my score.
Response to Reviewer sLdz (Part 1/2): Construct Validity & Related Work
We sincerely thank Reviewer
sLdzfor the deep, scholarly review and the highly relevant literature pointers, as well as for recognizing our structured pipeline and multi-view empirical analysis. Due to OpenReview's character limit, we organize our response in two parts: Part 1 addresses construct validity and related work, and Part 2 provides the exact T1–T5 prompts and our ongoing prompt-sensitivity and human-baseline studies.1. Calibrating construct validity and scope of claims
We fully accept your critique regarding construct validity. Moving directly from Mednick's associative theory to general multimodal "creativity" overstates what a Chinese chengyu decoding benchmark alone can establish, as creativity encompasses broader generative, flexible, and context-sensitive dimensions (Runco & Jaeger, 2012; Schapiro et al., 2026), and chengyu items intertwine conceptual bridging with Chinese phonetic, lexical, and cultural conventions.
2. Positioning with respect to recent multimodal association and creativity literature (
[1]–[4])Thank you for pointing out these directly relevant works, which we will integrate into a dedicated subsection in Related Work:
(Continued in Part 2/2 below.)