acceptodds
Withdrawn from ICLR 2027

GEB-Bench: Abstract Structures Told in Many Voices

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Analogy allows models to apply relational knowledge from one situation to another. Benchmarks of equivalent representations test different ways of expressing the same underlying problem, whereas analogy often links situations involving different objects and content. To evaluate the analogy ability of models, we introduce GEB-Bench, a benchmark centered on cross-voice motif matching. A voice presents a motif as a scene, story, mathematical statement, or programmatic skeleton. The first task is identification across these voices. The second is matching scenes to stories or mathematical statements, and stories to mathematical statements, by their shared motif. A complete construction pipeline generates items in these voices from the relation specified for each motif, checks the order and repetition of story units, and assembles paired questions. We evaluate current models on our benchmark. Results show that matching performance depends on the target voice. The paired scene-to-story study reveals that (1) evaluated models still make matching errors when they separately identify the source and every candidate correctly, and (2) identification improves more consistently than scene-to-story matching across the evaluated Qwen model sizes. These findings show that successful motif identification does not ensure reliable matching across voices.

Primary area:
datasets and benchmarks
Keywords:
abstract reasoning, structural analogy, multimodal reasoning, cross-modal matching, graph operations, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_BoqZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GEB-Bench to evaluate whether models can identify abstract structures across different expression modalities and match shared structures among these forms. The benchmark contains 25 structure types, presented as scene graphs, stories, structural diagrams, and mathematical statements. The paper evaluates structure recognition and cross-format matching separately. Paired experiments reveal that even when a model correctly identifies the structures of both the source graph and all candidate stories in independent queries, direct matching still fails. The authors also test a scheme that first predicts structure labels and then matches them via fixed rules, finding that this approach improves accuracy.

Critical strengths
  • Clear and valuable research question. Distinguishing structure recognition from cross-format matching helps investigate whether models can continue to utilize the same relational concepts despite variations in content and expression format.
  • Diagnostic value of paired experiments. The authors keep the identities of the source graph and candidates consistent, analyze matching performance when recognition is correct, and report subset coverage. Appendix controls—including source-label prompting, label matching, and ambiguity filtering—further help interpret the results.
  • Sufficient disclosure of experimental materials. The appendix provides structure definitions, prompt templates, scoring rules, and review results, while also disclosing label ambiguities and auxiliary conditions for human evaluation. This information enables readers to assess the scope and applicability of the experimental conclusions.
Critical weaknesses
  • Unverified Answer Uniqueness: The benchmark assumes single correct answers without rigorous validation. Table 6 indicates that 45.2% of scenes support multiple labels, suggesting benchmark ambiguity may be misattributed as model error. Methodological flaws in human evaluation prevent it from serving as a valid double-blind check. Although Table 10 filters a cleaner subset, it represents only 14.8% of the data and remains model-dependent.

  • Asymmetric Task Information: Recognition tasks provide explicit structure definitions and menus, whereas direct matching does not. This asymmetry confounds the performance gap, making it unclear whether failures stem from flawed reasoning or missing prompt information. Furthermore, Table 9's label-matching control alters multiple variables simultaneously (prompts, call counts, selection mechanisms), preventing isolated attribution.

Questions for authors

See weakness.

Reviewer_WV4ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GEB-BENCH, a benchmark for testing whether (multimodal) models can recognize the same abstract motif across different voices. There are several motifs and the voices are generated scenes, generated stories whose form encodes the motif, math statements and 2-D program skeletons. The tasks are motif identification and cross-voice matching (scene->story, scene->math, story->math). Around 25 VLMs are evaluated. The main claims are that models often fail to match even when they identify the source and every candidate correctly.

Critical strengths
  • The idea of fixing the motif and varying the representation (image, story form, theorem, diagram) is a nice way to separate structure from surface content.

  • The evaluation is broad (around 25 VLMs along with text-only models), and the visuals are easy to follow.

Critical weaknesses

Comments

  1. The motivation and downstream relevance of the benchmark are not sufficiently clear. The paper frames GEB-Bench as an evaluation of analogy, but it is not clear what useful downstream capability the benchmark is intended to represent. Many of the story motifs are implemented through formal or syntactic devices, such as mirrored token sequences, repeated sentence templates, speaker-tag patterns, acrostics, or Fibonacci-like sentence structures. Matching a visual scene such as a beehive to a story that uses a repeated sentence template therefore feels closer to discovering a hidden construction rule than to conventional structure mapping or analogical reasoning. The paper should better justify why success on these tasks is a meaningful measure of analogy.

  2. The choice of motifs lacks a clear organizing principle. The benchmark uses a collection of GEB-inspired motifs, but it is not clear why these particular motifs were selected or whether they cover a principled taxonomy of relational structures. The notion of different “voices” also needs stronger justification. For example, scene-to-mathematical-statement matching can often be interpreted as concept retrieval, particularly when the mathematical candidate directly names a known theorem or principle. A clearer taxonomy of relation types and a stronger argument for the selected representations would improve the benchmark.

  3. The central finding may reflect a compositional or decision-consistency gap rather than a new failure of analogy. The main result is that models can identify the source and candidate motifs correctly but still fail to select the correct match. However, related behavior has already been studied in work on compositionality gaps, generator-validator inconsistency, and mismatches between generation and recognition. Relevant work includes Press et al. (2023), Li et al. (2024), and West et al. (2024). The paper should discuss how its finding differs from these known forms of inconsistency.

  4. The label-matching results weaken the claim that the models lack analogical ability. If the model first identifies the motif of each item and a deterministic rule then matches items with the same label, performance improves over direct matching for all four evaluated models. This suggests that the required relational information is already available to the models. The direct-matching failure may therefore arise from the prompt format, long candidate stories, distractor construction, or difficulties comparing several candidates simultaneously rather than from an inability to recognize the underlying analogy. The paper should include more controls to separate these explanations.

  5. The scene labels appear too ambiguous to support strong conclusions. According to Table 6, model review finds unique agreement with the intended scene label for only 39.0% of scenes, while 45.2% have multiple plausible labels. This is a major concern because the benchmark assumes that each item expresses a specific intended motif. If several motifs are plausible for nearly half of the scenes, matching errors may partly reflect ambiguity in the benchmark rather than model reasoning failures. The authors should report results on a high-confidence subset with unambiguous human-agreed labels.

  6. The benchmark relies heavily on generated and manually designed examples. Scenes and stories are constructed specifically to express predefined motifs. This makes it difficult to know whether models are learning general analogical relationships or responding to artifacts of the generation process. Evaluation on naturally occurring examples, or examples created independently of the benchmark construction pipeline, would provide stronger evidence for generalization.

  7. The benchmark should engage more closely with existing analogy benchmarks. There is relevant prior work that appears closely related to the stated goal, including StoryAnalogy (Jiayang et al., 2023), ARN (Sourati et al., 2024), AnaloBench (Ye et al., 2024), KiVA (Yiu et al., 2025), and VOILA (Yilmaz et al., 2025). These works study story-level, abstract, visual, and multimodal analogical reasoning. A clearer comparison is needed to establish what capability GEB-Bench adds beyond these benchmarks.

  8. The different target voices expose the motif with very different levels of difficulty. Mathematical statements may directly reveal a known relation or theorem, whereas stories encode the motif indirectly through narrative form. Therefore, differences between scene-to-math and scene-to-story matching do not necessarily isolate cross-modal transfer. They may simply reflect how explicitly the target representation reveals the intended motif.

  9. Missing or relevant references

a. Press et al., Measuring and Narrowing the Compositionality Gap in Language Models, Findings of EMNLP, 2023.

b. Li et al., Benchmarking and Improving Generator-Validator Consistency of Language Models, ICLR, 2024.

c. West et al., The Generative AI Paradox: “What It Can Create, It May Not Understand”, ICLR, 2024.

d. Jiayang et al., StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding, EMNLP, 2023.

e. Sourati et al., ARN: Analogical Reasoning on Narratives, TACL, 2024.

f. Ye et al., AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies, EMNLP, 2024.

g. Yiu et al., KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models, ICLR, 2025.

h. Yilmaz et al., VOILA: Evaluation of MLLMs for Perceptual Understanding and Analogical Reasoning, ICLR, 2025.

Questions for authors

Please refer to weaknesses besides the following question:

  • The paper needs to say clearly what we learn from a model scoring 70% vs 40% here that we can’t learn from existing analogy benchmarks.

  • What is the concrete use case or downstream capability that this benchmark is meant to predict? If a model goes from 40% to 70% on scene->story, what else should we expect it to be able to do?

  • Can you show direct matching vs label matching for all models, plus a version with short candidates or a neutral prompt?

  • Can you report all main results only on items where humans (independent, blind to model output and intended label, at least 2 annotators with agreement reported) assign the intended label uniquely?

Reviewer_i96MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The rough idea behind this paper seems to be something related to creating a benchmark for matching conceptually similar stuff across a few modalities. The authors present a benchmark with a few examples, and then accuracies with various models (some models getting to quite high accuracies already). The writing quality, however, is incredibly poor / incoherent, and I'm struggling to parse basic motivation for why they believe their benchmark is well-motivated compared to the vast and oversaturated space of multi-modal benchmarks, and how they construct their benchmark.

Critical strengths

The evaluation of the benchmark is done on a large set of models, which gives some information about how difficult it is for the current landscape of models.

Critical weaknesses
  • The writing is incredibly unclear and difficult to parse, even after reading paragraphs several times I have a hard time understanding the writing and overall claims. This is likely all if not highly AI-generated
  • The overall motivation seems unclear compared to existing work, and the benchmark doesn't seem particularly challenging or investigating an interesting scientific question
Questions for authors

N/A

Reviewer_QqaaOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GEB-Bench connects scenes, stories, mathematical statements, and skeleton diagrams through 25 structural motifs. It evaluates identification alongside matching between different contents and voices. The central finding is that models can identify the source and every candidate correctly in separate calls yet fail at direct matching. An additional label-based procedure improves performance. The paired diagnostic is valuable, but label validity and differences in the information provided across conditions limit the interpretation.

Critical strengths

The shared-source, shared-candidate design provides more informative evidence than comparing aggregate task scores. Requiring correct identification of all distractors strengthens this diagnostic. The construction pipeline includes executable checks on story organization, and the appendix carefully distinguishes finite realizations from broader mathematical associations. Table 9 adds useful source-label and predicted-label interventions. The disclosure of prompts, class counts, scoring rules, review outcomes, and limitations, together with the reviewed-subset analysis in Table 10, makes the work substantially easier to assess.

Critical weaknesses

My main concern is whether construction labels are sufficiently well established as evaluation targets. In Table 6, only 39.0% of scenes receive unique agreement from the model reviewer, while 45.2% admit multiple interpretations. These are not established mislabeling rates, but they motivate independent validation. The human study in D.2 does not resolve this concern: model predictions were visible throughout, 8.0% of submissions followed intended-label revelation, and each item received one submission. Table 10 is a useful partial response, but its strictest subset retains only 14.8% of questions and is still selected by a model reviewer.

Second, identification and direct matching receive different structural guidance. E.1 and J.1 specify that identification includes four motif labels and detailed voice-specific operational descriptions; direct matching contains no motif menu. Label matching additionally uses separate calls and deterministic selection. The results therefore establish a behavioral discrepancy across these conditions, but do not cleanly isolate a matching bottleneck beyond identification. The supplied-source-label intervention is informative, yet also changes the information available.

Third, some cross-voice correspondences depend on specialized conventions, such as translating tiling into sentence-template repetition or figure-ground exchange into grammatical role exchange. The appendix appropriately limits these constructions, so I am not alleging mathematical errors. However, independent discriminability of nearby motifs remains important for the intended interpretation. Finally, uneven motif counts, reused inputs, and the small strict review subset constrain generalization; motif-level uncertainty estimates would help calibrate the scaling and matching claims.

Questions for authors

Could the strict Table 10 subset and a small stratified sample of ambiguous scenes be independently adjudicated with model predictions and intended labels hidden, followed by recomputation of the central gap? For the two leading models, how much of the gap remains when direct matching receives the same operational menu as identification? This would separate explicit structural guidance from the benefit of decomposition. Could the existing outputs establish the intersection of validated items and all-correct identification, with exact counts and motif-aware uncertainty for the relevant differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.