Comments
-
The motivation and downstream relevance of the benchmark are not sufficiently clear. The paper frames GEB-Bench as an evaluation of analogy, but it is not clear what useful downstream capability the benchmark is intended to represent. Many of the story motifs are implemented through formal or syntactic devices, such as mirrored token sequences, repeated sentence templates, speaker-tag patterns, acrostics, or Fibonacci-like sentence structures. Matching a visual scene such as a beehive to a story that uses a repeated sentence template therefore feels closer to discovering a hidden construction rule than to conventional structure mapping or analogical reasoning. The paper should better justify why success on these tasks is a meaningful measure of analogy.
-
The choice of motifs lacks a clear organizing principle. The benchmark uses a collection of GEB-inspired motifs, but it is not clear why these particular motifs were selected or whether they cover a principled taxonomy of relational structures. The notion of different “voices” also needs stronger justification. For example, scene-to-mathematical-statement matching can often be interpreted as concept retrieval, particularly when the mathematical candidate directly names a known theorem or principle. A clearer taxonomy of relation types and a stronger argument for the selected representations would improve the benchmark.
-
The central finding may reflect a compositional or decision-consistency gap rather than a new failure of analogy. The main result is that models can identify the source and candidate motifs correctly but still fail to select the correct match. However, related behavior has already been studied in work on compositionality gaps, generator-validator inconsistency, and mismatches between generation and recognition. Relevant work includes Press et al. (2023), Li et al. (2024), and West et al. (2024). The paper should discuss how its finding differs from these known forms of inconsistency.
-
The label-matching results weaken the claim that the models lack analogical ability. If the model first identifies the motif of each item and a deterministic rule then matches items with the same label, performance improves over direct matching for all four evaluated models. This suggests that the required relational information is already available to the models. The direct-matching failure may therefore arise from the prompt format, long candidate stories, distractor construction, or difficulties comparing several candidates simultaneously rather than from an inability to recognize the underlying analogy. The paper should include more controls to separate these explanations.
-
The scene labels appear too ambiguous to support strong conclusions. According to Table 6, model review finds unique agreement with the intended scene label for only 39.0% of scenes, while 45.2% have multiple plausible labels. This is a major concern because the benchmark assumes that each item expresses a specific intended motif. If several motifs are plausible for nearly half of the scenes, matching errors may partly reflect ambiguity in the benchmark rather than model reasoning failures. The authors should report results on a high-confidence subset with unambiguous human-agreed labels.
-
The benchmark relies heavily on generated and manually designed examples. Scenes and stories are constructed specifically to express predefined motifs. This makes it difficult to know whether models are learning general analogical relationships or responding to artifacts of the generation process. Evaluation on naturally occurring examples, or examples created independently of the benchmark construction pipeline, would provide stronger evidence for generalization.
-
The benchmark should engage more closely with existing analogy benchmarks. There is relevant prior work that appears closely related to the stated goal, including StoryAnalogy (Jiayang et al., 2023), ARN (Sourati et al., 2024), AnaloBench (Ye et al., 2024), KiVA (Yiu et al., 2025), and VOILA (Yilmaz et al., 2025). These works study story-level, abstract, visual, and multimodal analogical reasoning. A clearer comparison is needed to establish what capability GEB-Bench adds beyond these benchmarks.
-
The different target voices expose the motif with very different levels of difficulty. Mathematical statements may directly reveal a known relation or theorem, whereas stories encode the motif indirectly through narrative form. Therefore, differences between scene-to-math and scene-to-story matching do not necessarily isolate cross-modal transfer. They may simply reflect how explicitly the target representation reveals the intended motif.
-
Missing or relevant references
a. Press et al., Measuring and Narrowing the Compositionality Gap in Language Models, Findings of EMNLP, 2023.
b. Li et al., Benchmarking and Improving Generator-Validator Consistency of Language Models, ICLR, 2024.
c. West et al., The Generative AI Paradox: “What It Can Create, It May Not Understand”, ICLR, 2024.
d. Jiayang et al., StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding, EMNLP, 2023.
e. Sourati et al., ARN: Analogical Reasoning on Narratives, TACL, 2024.
f. Ye et al., AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies, EMNLP, 2024.
g. Yiu et al., KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models, ICLR, 2025.
h. Yilmaz et al., VOILA: Evaluation of MLLMs for Perceptual Understanding and Analogical Reasoning, ICLR, 2025.