acceptodds
Under review as a conference paper at ICLR 2027

GENIUS: Generative Fluid Intelligence Evaluation Suite

Abstract

Unified Multimodal Models (UMMs) have shown remarkable progress in visual generation. Yet, existing benchmarks predominantly assess *Crystallized Intelligence*, which relies on recalling accumulated knowledge and learned schemas. This focus overlooks *Generative Fluid Intelligence (GFI)*: the capacity to induce patterns, reason through constraints, and adapt to novel scenarios on the fly. To rigorously assess this capability, we introduce **GENIUS** (**GEN**erative Fluid **I**ntelligence Eval**U**ation **S**uite). We operationalize *GFI* through three complementary dimensions. These include *Inducing Implicit Patterns* (e.g., inferring personalized visual preferences), *Executing Ad-hoc Constraints* (e.g., visualizing abstract metaphors), and *Adapting to Contextual Knowledge* (e.g., simulating counter-intuitive physics). Collectively, these dimensions challenge models to solve problems whose governing rules are established by the immediate context. Our systematic evaluation of 14 representative models reveals significant performance deficits in these tasks. Our diagnostic analysis reveals a gap between recognizing the intended outcome and faithfully generating it: models can identify the expected output in a discriminative VQA probe, yet struggle to execute contextual rules during generation. To bridge this gap, we propose a training-free attention intervention strategy. Ultimately, **GENIUS** establishes a rigorous standard for *GFI*, guiding the field beyond knowledge utilization toward dynamic, general-purpose reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image generation, unified multimodal models, fluid intelligence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PkX9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GENIUS is a benchmark for evaluating Generative Fluid Intelligence (GFI): the ability to infer patterns, execute ad-hoc constraints, and adapt to rules that conflict with prior knowledge using the immediate multimodal context. It contains 510 samples spanning three dimensions and five tasks. Human-authored evaluation hints guide multimodal judges in assessing Rule Compliance (RC), Visual Consistency (VC), and Aesthetic Quality (AQ). The authors evaluate 14 models and investigate failures through context removal, hint-based interventions, and VQA probes. They also propose a training-free attention intervention that improves the Overall score from 26.74 to 32.92 on Bagel and from 24.71 to 30.04 on GLM-Image (Tables 2 and 4).

Critical strengths
  1. Contextual knowledge adaptation is a valuable evaluation target. Counterfactual rules and semantic disambiguation test whether models follow local conditions rather than defaulting to prior knowledge. NanoBananaPro’s RC scores of 52.97 on Prior-Conflicting Generation and 35.45 on Multi-Semantic Generation indicate that these settings remain challenging.

  2. The benchmark offers diverse tasks and concrete evaluation guidance. The tasks cover preference induction, operation transfer, conditional execution, and semantic transfer, with sample-specific RC/VC hints. Complete inputs, evaluation hints, and model outputs in the appendix clarify the intended requirements.

  3. The experiments establish the importance of context. Removing context substantially reduces RC on several tasks, while input formatting also affects performance. These controls support the benchmark’s emphasis on contextual information use.

  4. Automatic evaluation receives some validation. Ratings from five human experts and evaluation by a second multimodal judge provide stronger support than reliance on a single judge. The reported human–judge correlations are high, and model rankings are broadly consistent across judges.

Critical weaknesses
  1. Some evaluation criteria may conflict with the task objectives. The AQ rubric rewards adherence to physics, anatomy, and visual realism, whereas Prior-Conflicting Generation explicitly requires counterfactual behavior, including gravity anomalies and abnormal biological growth. Since the AQ judge receives only the output image, it may penalize intentional deviations as generation errors (Sections 2.2–2.3; Figure 1; Appendix Figure 10). The VC rubric also incorporates instruction compliance and image quality, creating overlap with RC and AQ (Appendix Figure 9). In addition, Appendix A.2 assigns zero VC to exact copies of reference images without clarifying how cases requiring an unchanged output are handled. These potential systematic biases warrant targeted validation.

  2. The need for additional terminology is insufficiently justified. The introduction and Section 2.1 introduce CI, FI, GFI, and three capability dimensions, although several tasks overlap with established in-context generation and reference-based editing settings, such as Edit Transfer[1] and VisualCloze[2]. The paper does not sufficiently explain why existing terminology is inadequate or what introducing these additional terms contributes to understanding the benchmark. Without this justification, the expanded vocabulary increases the reader’s burden without a clear conceptual benefit.

[1] Lan Chen, Qi Mao, Yuchao Gu, and Mike Zheng Shou. Edit Transfer: Learning Image Editing via Vision In-Context Relations. arXiv preprint arXiv:2503.13327, 2025. https://arxiv.org/abs/2503.13327

[2] Zhong-Yu Li, Ruoyi Du, Juncheng Yan, Le Zhuo, Qilong Wu, Zhen Li, Peng Gao, Zhanyu Ma, and Ming-Ming Cheng. VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 18969–18979, 2025. https://visualcloze.github.io/

  1. The recognition–generation comparison does not sufficiently align task demands and metrics. Section 3.2 and Figure 2(b) compare four-choice VQA accuracy with generation Overall scores. However, multiple-choice recognition provides candidate answers, while Overall additionally measures visual consistency and aesthetics; the scores are therefore not directly comparable. The illustrated distractors cover some omissions and incorrect sources, but systematic coverage of generation failures such as role swaps and attribute misbinding is not established. The experiment supports the weaker conclusion that recognizing an intended outcome is easier than generating it, but does not yet establish that models correctly understand all constraints and subsequently fail to execute them.

  2. The theory and attention visualizations do not establish the proposed failure mechanism. Theorem 4.1 provides a local output equivalence between context changes and parameter perturbations, and Theorem 4.2 expresses the constructed updates as gradient steps. However, the linear loss in Appendix G.3 is defined from the update itself; its connection to rule compliance, image quality, or meaningful task optimization remains unclear (Section 4.2; Appendix G). Figure 3 also leaves the selection and aggregation of layers, heads, and generation steps insufficiently specified, along with the correspondence between attention peaks and textual or visual regions. Since the method directly injects a relevance bias, a more concentrated attention distribution primarily confirms that the intervention changes attention. It does not independently establish that the original distribution is noise or that an implicit optimization direction has been corrected.

Questions for authors
  1. Evaluation protocol: How does AQ distinguish task-required counterfactual or non-photorealistic outputs from genuine generation errors? Should the judge receive the task context? How is the copying penalty handled when leaving the image unchanged is correct?

  2. Terminology and framing: Why is the additional terminology necessary rather than describing the tasks using established terms in in-context generation and reference-based editing? What important distinction would be lost by retaining the existing terminology?

  3. Recognition–generation alignment: Could the VQA probes include systematic role-swap and attribute-misbinding distractors, and report the proportion of samples where VQA is correct but generation fails the same semantic constraint?

  4. Mechanistic evidence: How are the attention scores in Figure 3 selected and aggregated? What establishes that suppressed tokens are irrelevant? How do the equivalent parameter updates relate to a meaningful task objective rather than an algebraic reformulation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PkX9's Review Comments (1 of 4)

Thank you for your specific suggestions on the boundaries between scoring criteria, recognition–generation alignment, and the interpretation of attention. We have completed the updates to the AQ definition and results, and added generation RC to Figure 2(b) in the revised PDF. Below, we explain the current protocol and report the results of our supplementary analyses.

W1 and Q1: The scoring boundaries between AQ and VC, and the treatment of unchanged outputs

The AQ definition update and re-evaluation are complete. The original scoring criterion treated adherence to real-world physical and anatomical laws as a universal requirement, which conflicts with the task setting of Prior-Conflicting Generation. Thank you for pointing this out. We have removed this requirement and updated the main text, appendix, and evaluation prompt accordingly. AQ focuses on composition, visual harmony, rendering clarity, and generation artifacts; surreal content, non-realistic styles, or unusual physical phenomena required by the task should not themselves lead to a deduction. AQ continues to evaluate only the output image, while RC checks whether the task rules are satisfied.

We have updated the result tables and related figures in the current manuscript using the revised AQ criteria and recomputed Overall. After the revision, Nano Banana Pro has an Overall score of 57.28. In the main results, the only change in relative ordering involves our method, which moves from eighth to sixth place, overtaking FLUX.2-dev and Gen-Searcher; the relative ordering among the baseline models remains unchanged. The main observation that attention intervention improves overall performance on both the Bagel and GLM-Image backbones is preserved.

VC measures the visual similarity between the reference content specified for each sample and the corresponding content in the generated image. For example, when a task requires preserving a reference character's identity while changing their clothing, RC checks whether the output satisfies the specified target conditions, such as the clothing requirements, while VC uses the corresponding hint to check the character appearance that should be preserved. Here, HINT refers to the sample-specific visual preservation requirements, and “follow the HINT” in the template refers to how well the output image satisfies those requirements.

Terms such as “professional,” “high-quality implementation,” and “smooth transformation” describe the quality and fidelity with which the specified reference content is preserved in the generated image. They are used within this reference-based comparison: the judge assesses the corresponding visual content identified by the VC hint. RC is evaluated independently using its own hints to assess whether the task rules are satisfied.

Regarding the copy penalty for unchanged images, we have ensured that no task has a target image that is simply an unchanged input image, so if the evaluated model copies the original image, its output is necessarily incorrect.

AuthorsOpenReview ↗

Response to Reviewer PkX9's Review Comments (2 of 4)

W2 and Q2: Overlap in editing operations and the diagnostic value of the terminology

The capability being evaluated depends on how the model determines the target operation. ARC-AGI-2 is a fluid intelligence benchmark in the understanding domain. ARC-AGI-2's final output is a transformed colored grid, but the evaluation target is to determine the transformation rule from the current examples and apply it to a new input. GENIUS applies the same rule acquisition requirement to generation driven by image–text context. The output may take the form of an edit, but the evaluation also requires determining the meanings of temporary symbols, the correspondence between operations and visual examples, and whether the current local knowledge overrides real-world priors.

Compared with the two works you mentioned:

WorkTheir contributionEvaluation focus of GENIUS
Edit TransferLearns an editing transformation from a single source–target example and applies it to a query imageOrganizes temporary bindings jointly defined by text and images, operation composition, preference disambiguation, and local overrides of priors into a systematic task collection
VisualClozeUses visual demonstrations to identify tasks, supporting multiple generation tasks and generalization to unseen tasksProvides a unified evaluation that separately measures rule induction, execution of current constraints, and contextual knowledge adaptation, and evaluates rule adherence and quality in generated outputs

We introduce GFI and its three dimensions to distinguish the rule acquisition and application requirements underlying generation operations: Implicit Pattern Induction discovers unstated regularities from examples; Ad-hoc Constraint Execution uses the bindings, conditions, and combinations defined for the current task; and Contextual Knowledge Adaptation adopts the current local knowledge or interpretation. These dimensions associate similar output operations with different reasoning requirements.

AuthorsOpenReview ↗

Response to Reviewer PkX9's Review Comments (3 of 4)

W3 and Q3: Aligning the semantic constraints of recognition, open-ended planning, and generation

Generation RC has been added to Figure 2(b) and the main text. The updated figure presents VQA accuracy, mean generation RC, and Overall together; RC is mapped from the 0–2 rating scale to 0–100.

We conducted an additional open-ended target-planning evaluation: Open-ended target-planning evaluation. To examine whether candidate answers provide additional help, each model received only the original image–text context and task and freely described the target objects, attributes, sources, spatial relationships, and operations, without receiving RC hints, candidate answers, or generated images. The evaluator used the human-authored RC hints to score the completeness of the target descriptions on a 0/0.5/1 scale. We converted these scores to a 100-point scale and report the results below.

ModelVQA accuracy (%)Open-ended planning scoreGeneration RCGeneration Overall
Nano Banana Pro94.1183.5659.8457.28
Bagel65.6848.5327.6726.72

The gap between target description and rule execution persists after candidate answers are removed. Nano Banana Pro and Bagel score 83.56 and 48.53 on the open-ended questions, respectively. Both scores are lower than their corresponding four-choice VQA accuracies, but remain higher than their mean generation RC scores (59.84 and 27.67). This indicates that even when required to describe the target independently, the models can still express more task constraints than their final images realize; the difference between recognition and generation therefore cannot be explained solely by the cues provided by candidate answers. Here, RC aligns the evaluation of target-constraint satisfaction, while Overall additionally reflects comprehensive generation performance, including visual preservation and aesthetic quality.

Regarding distractors, Figure 2(b) already includes examples of constraint omission and source/role swaps. Option B omits the style requirement, C swaps the sources of the character and clothing, and D contains an incorrect subject and omits the clothing requirement. By omitting key constraints or confusing reference sources, these options test whether the model can identify the target that satisfies all requirements simultaneously. This example therefore already covers the issues of constraint completeness and source correspondence that you suggested examining. We computed the requested paired statistics and report them below:

ModelVQA correct and RC=2 (%)VQA correct and RC=1 (%)VQA correct and RC=0 (%)VQA incorrect and RC=2 (%)VQA incorrect and RC=1 (%)VQA incorrect and RC=0 (%)
Nano Banana Pro45.8726.5421.700.001.44.49
Bagel13.4724.3827.830.892.2431.19

The paired results further show that selecting the correct target does not always translate into complete rule execution. All six proportions in the table use the full set of paired records as their denominator. Restricting the analysis to records with correct VQA answers, 51.26% and 79.49% of the generated outputs from Nano Banana Pro and Bagel, respectively, still fail to reach RC=2: they satisfy only some constraints or fail to satisfy the target rules. These results directly show that, on tasks for which the model selects the correct target, the generated image may still omit or violate rules. They support analyzing the models' capability differences through the two stages of target recognition and generation execution.

AuthorsOpenReview ↗

Response to Reviewer PkX9's Review Comments (4 of 4)

W4 and Q4: Attention evidence, intervention controls, and theoretical interpretation

Thank you for your suggestions regarding attention evidence and theoretical interpretation. We respond in three parts: what the visualization displays, the role of locating task-relevant information, and the conclusions supported by the theoretical derivation.

The attention visualization shows how context is used during generation. Figure 3 uses the image tokens during generation as queries to examine their allocation of attention to the image–text context. Appendix F.3 provides the intervention settings: layers [8,24], generation steps [10,35], and a default strength of λ=1.0. The visualization shows the change in the attention distribution before and after intervention; whether task-relevant locations contribute to performance improvements is tested by the spatial shuffling control below.

The spatial shuffling control supports the role of correctly locating task cues. We first computed the actual relevance map and then randomly shuffled the positions of the bias values within each input image, preserving the set of bias values while disrupting their correspondence with visual locations. After spatial shuffling, RC, VC, and Overall are 25.46, 17.64, and 24.65, respectively, all below the original Bagel scores of 27.67, 19.05, and 26.72. The full method reaches 33.28, 34.09, and 34.93, respectively. Compared with the shuffled control, the full method improves RC, VC, and Overall by 7.82, 16.45, and 10.28 points, respectively. This shows that locating task cues at their corresponding visual positions is an important condition for attention intervention to yield benefits.

The theory provides a parameter-update perspective for understanding the effect of context. For a given layer and intermediate generation state, Theorem 4.1 constructs a parameter perturbation that compensates for a change in context, establishing an equivalence between layer outputs. Theorem 4.2 further expresses the corresponding parameter changes as gradient updates on a constructed linear surrogate objective. This surrogate objective characterizes parameter differences and is not equivalent to the loss function of the actual generation task. The theory thus provides an explanatory framework for adjusting the influence of context, while the effect of a specific intervention on rule compliance is tested experimentally: the spatial shuffling control supports the value of locating task-relevant information, and the λ sweep shows the relationship between intervention strength, rule compliance, visual preservation, and generation quality. We will revise the corresponding main-text discussion and Figure 4 wording and caption to make this distinction explicit and align the theoretical claims with the scope of the derivation.

Thank you again for your comments. These supplementary experiments make the paper's arguments more complete. We hope the responses above address your questions; if you have any further questions, we welcome further discussion.

Reviewer_Tk4tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

GENIUS proposes a benchmark that evaluates image generation for umms, reports large gaps on current models, and adds a training-free attention intervention.

Critical strengths

The interleaved multi-image setup is clean.

The diagnostics are useful.

The evaluation is broad.

Critical weaknesses

This paper focuses on the image generation tasks for UMMs. But from my view, it seems that there have been lots of benchmarks that match the concept of GFI for image generative model, such as[1~7]? So the contribution seems to be incremental? This needs to be clarified more.

[1] Chen, K., Lin, Z., Xu, Z., Shen, Y., Yao, Y., Rimchala, J., ... & Huang, L. (2025, November). R2i-bench: Benchmarking reasoning-driven text-to-image generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 12606-12641).

[2] Sun, K., Fang, R., Duan, C., Liu, X., Li, A., & Liu, X. (2026, July). T2i-reasonbench: Benchmarking reasoning-informed text-to-image generation. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 8919-8944).

[3] Liu, Y., Huang, N., Cao, X., & Rehg, J. M. (2026). Thinking in Pictures: A Systematic Benchmark for Reasoning-driven Image Generation. arXiv preprint arXiv:2609.02864.

[4] Niu, Y., Ning, M., Zheng, M., Jin, W., Lin, B., Jin, P., ... & Yuan, L. (2025). Wise: A world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265.

[5] Chow, W., Pan, J., Liang, Y., Zhou, M., Song, X., Jia, L., ... & Chua, T. S. (2026). Weave: Unleashing and benchmarking the in-context interleaved comprehension and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15343-15353).

[6] Chen, C., Wang, Z., He, Y., Yu, Z., & Cao, N. (2026). Blending Concepts: Benchmarking Visual Metaphor Generation in Text-to-Image Models. arXiv preprint arXiv:2609.02502.

[7] Lei, J., Pu, Y., Han, X., Zhu, R., Xu, J., Wang, J., ... & Li, H. (2026). Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning. arXiv preprint arXiv:2606.24548.

Questions for authors

My major concern lies in weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Tk4t's Review Comments (1 of 2)

W1 and Q1: Differences from existing benchmarks

Thank you for pointing out the connections to related work. GENIUS primarily evaluates whether a model can acquire the rules applicable to the current task from the given image–text context and use those rules for generation. To clarify this, we organize the comparison around “where the basis for the answer comes from,” “whether rule acquisition at test time is required,” and “the role of existing knowledge in the task.”

BenchmarkMain inputs and basis for the answerDoes it test crystallized intelligence?Does it require acquiring or applying new rules at test time?Relationship to and differences from GENIUS
R2I-BenchObjects, conditions, and logical relationships specified in text, combined with knowledge such as common sense and mathematicsPrimarily focuses on world knowledge: tasks involve applying and reasoning with existing knowledge, including deductions under hypothetical conditionsSome tasks involve hypothetical conditions specified in text and deductions from rulesGENIUS adds the requirements of cross-modal rule establishment and binding, and covers other dimensions of fluid intelligence.
T2I-ReasonBenchIdioms, entities, scientific knowledge, and design goals specified in textPrimarily evaluates world knowledgeEstablishing ad-hoc rules from image–text examples is not the unifying focus of its evaluationIts idiom tasks are superficially similar in theme to our Multi-Semantic Generation, but the way meaning is determined differs: GENIUS requires models to determine from the given visual context whether a literal or metaphorical interpretation applies, and to transfer that interpretation to a new expression. The correct interpretation changes with the current context and cannot be determined solely from the expression's conventional meaning. T2I-ReasonBench primarily focuses on reasoning with world knowledge.
Thinking in Pictures / RIG-BenchVisual context, input-output demonstrations, and task instructionsDoes not test only crystallized intelligence; explicitly includes visual rule induction, analogy, and matrix reasoningRequires inducing latent rules from visual examplesGENIUS emphasizes jointly establishing ad-hoc meanings and conditional relationships through text and images. Furthermore, RIG does not treat fluid-intelligence requirements such as “using local knowledge established in the current context to override conflicting real-world priors” as a separate evaluation dimension; GENIUS measures this specifically.
WISECultural knowledge, natural science, and spatial and temporal knowledgePrimarily focuses on applying existing world knowledgeEstablishing new local rules at test time is not its central focusWISE focuses on applying real-world knowledge; GENIUS's prior-conflicting tasks require adopting local rules when they conflict with real-world knowledge.
WEAVEMulti-turn image–text interactions, historical images, and user requestsPrimarily tests memoryRetaining and reusing historical information is an important componentGENIUS centers on establishing, binding, inducing, and applying rules; multi-turn memory and rule reasoning are related but distinct evaluation requirements.
Blending Concepts / VMetaphor-BenchSource domains, target domains, and intended meanings specified in text, with the model performing cross-domain mapping and visual expressionRelies on existing conceptual knowledge and requires compositional reasoningConceptual prompts require deriving visual mappings, but acquiring ad-hoc rules from image–text examples is not its central focusThis benchmark evaluates whether models can construct the corresponding visual expression from metaphorical meanings or visual descriptions given entirely in text. GENIUS's Visual Metaphor subtask instead requires models to first acquire task-specific rules for visual expression from the current image–text examples and then transfer them to new objects. The distinction concerns acquiring and transferring rules for expression, not merely whether the final image contains a metaphor.
Are Text-to-Image Models Inductivist Turkeys? / CF-WorldReal-world laws and counterfactual world laws specified in textReal-world knowledge serves as a baseline, while the counterfactual component requires going beyond real-world priorsYes; some levels provide only the changed laws and require deriving the consequencesIts new-world rules are specified directly in text; GENIUS's Prior-Conflicting Generation requires combining image–text context to acquire, demonstrate, or disambiguate local world knowledge and then transfer it to the target object. Moreover, this task is only one part of GENIUS: we also systematically evaluate implicit pattern induction and ad-hoc constraint execution to assess fluid intelligence.
AuthorsOpenReview ↗

Response to Reviewer Tk4t's Review Comments (2 of 2)

BenchmarkMain inputs and basis for the answerDoes it test crystallized intelligence?Does it require acquiring or applying new rules at test time?Relationship to and differences from GENIUS
ARC-AGI-2A small number of input-output grid examples and a new test inputNot the primary objective; basic cognitive priors support reasoning about new tasksYes; requires interpreting symbols, composing rules, and applying them according to contextProvides a design reference for rule acquisition and transfer; GENIUS implements this requirement in natural-image generation with interleaved image–text context.
GENIUSExamples, ad-hoc definitions, preferences, conditions, and local knowledge in image–text context, together with the final generation instructionNot the primary evaluation target; existing visual and linguistic knowledge provides the foundationYes, this is a core requirement of the task design: determine the rules from the current context and then apply them to generate the targetProvides unified coverage of implicit pattern induction, ad-hoc constraint execution, and contextual knowledge adaptation, and assesses generated outputs through RC, VC, and AQ, respectively.

ARC-AGI-2 is a benchmark for testing fluid intelligence in understanding. ARC-AGI-2's final output is a transformed colored grid, but its evaluation objective is to determine the transformation rules from the current examples and apply them to a new input. Its official description specifically emphasizes symbol interpretation, rule composition, and applying rules according to context. As a simplified illustration at the protocol level: if the same test input is placed in example contexts that define different transformations, the correct output should also change with the rules; the answer is determined by the combination of the input and the current rules, rather than by a fixed pretrained answer.

GENIUS follows this design principle. Even when the final operation takes the form of modifying attributes, adjusting layouts, or transferring styles, the model must first determine what the operation means in the current task from the given examples and textual relationships. For example, the current context may bind an ad-hoc operation to a visual symbol or use examples to disambiguate the literal or metaphorical meaning of an expression; the final instruction defines a determinate generation target only after these rules have been acquired. Directly applying conventional meanings, copying a historical image, or retrieving known facts about the world cannot systematically replace this step.

Summary. Knowledge-based generation benchmarks primarily require invoking and reasoning with existing semantic and world knowledge based on the question, and typically do not require separately acquiring new rules for the current task from image–text examples; memory tasks in multi-turn generation primarily require retaining, retrieving, and reusing historical visual information. Rule induction in RIG-Bench and counterfactual reasoning in CF-World involve new rules, but they do not define and evaluate generative fluid intelligence as systematically as GENIUS. GENIUS is distinguished by organizing the entire suite around “what rules the current image–text context establishes, and whether the model can generate accordingly”: the correct output must be determined by the rules in the current context, rather than solely by pretrained priors or the reuse of historical information.

Thank you again for your comments. These comparisons make the paper's argument more complete. We hope the responses above address your questions, and we welcome further discussion if you have any additional questions.

Reviewer_BAQ4OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

To evaluate whether multimodal generative models possess Generative Fluid Intelligence (GFI) rather than merely recalling memorized priors, the authors introduce GENIUS. GENIUS comprises 510 manually curated multimodal interleaved test cases across five tasks covering implicit pattern induction, ad-hoc constraint execution, and contextual knowledge adaptation. Models are evaluated across Rule Compliance (RC), Visual Consistency (VC), and Aesthetic Quality (AQ) using dual LMM judges grounded by human-written hints. Evaluating 14 models reveals severe GFI deficits and a stark gap between discriminative recognition and generative execution, motivating a training-free attention intervention on Bagel.

Critical strengths
  1. The motivation is clear and the concept is novel. GENIUS reveals a real limitation of unified multimodal models: strong rendering of memorized concepts does not transfer to in-context rule induction or prior override.
  2. The theoretical analysis aligns with the experimental results. The account of how contextual signals steer generation is consistent with the attention intervention improving rule compliance on Bagel.
  3. The benchmark design and hybrid evaluation protocol are rigorous. The 510 expert-curated interleaved cases and the dual-judge setup, with over 0.96 human correlation, make the evaluation convincing.
  4. The diagnostic findings clearly expose the gap between recognition and generation. Across 14 models, high aesthetic scores coexist with weak rule compliance, and models recognize rules in VQA probes much better than they execute them during image generation.
Critical weaknesses
  1. The attention intervention improves rule compliance but reduces aesthetic quality. The authors should add an analysis of this trade-off, including how keyword distillation and attention bias each contribute.
  2. The VQA probe is a 4-way multiple-choice test, and several models lack native interleaved inputs. This makes it harder to separate fluid-intelligence deficits from easier option recognition and input-format constraints.1. Does the analysis and attention intervention demonstrated on Bagel generalize to other unified multimodal models?
Questions for authors
  1. Does the analysis and attention intervention demonstrated on Bagel generalize to other unified multimodal models?
2 comments
AuthorsOpenReview ↗

Response to Reviewer BAQ4's Review Comments (1 of 2)

Thank you for recognizing the motivation, task design, and diagnostic value of GENIUS. Following Reviewer PkX9's suggestion, we have updated the AQ definition and evaluation results in the paper PDF, and all results below are based on the latest results. We address the quality trade-off, component contributions, input formats, and transferability below.

W1 Quality trade-offs and component contributions of the attention intervention

The current results show that the intervention improves rule compliance and visual preservation while reducing aggregate AQ. In the latest Table 2, Bagel's Overall score increases from 26.72 to 34.93, a gain of 8.21 points. RC improves on all five tasks; AQ decreases on Implicit Pattern, Symbolic Constraint, and Prior-Conflicting. The quality trade-off is primarily reflected in the decrease in aggregate AQ.

ConditionRCVCAQOverall
Bagel27.6719.0569.0526.72
Bagel + full method33.2834.0960.6734.93

Intervention strength analysis. Appendices F.2–F.3 specify the bias strength λ, the layers [8,24], and the generation steps [10,35], with λ=1.0 as the default. We evaluated λ∈{0, 0.25, 0.5, 1.0, 1.5, 2.0} to examine the balance between RC, VC, and AQ. The results are reported below.

λRCVCAQOverall
027.6719.0569.0526.72
0.2528.5425.5664.8529.31
0.532.1030.7563.8033.21
1.033.2834.0960.6734.93
1.531.1634.7255.6633.63
2.026.8229.3452.3828.98

λ=1.0 achieves the best overall performance among the tested strengths. As λ increases, AQ decreases continuously, while RC and VC first increase and then decrease: RC peaks at λ=1.0, whereas VC peaks at λ=1.5. Weaker intervention yields smaller improvements in rule compliance; for example, at λ=0.25, RC improves by only 0.87 points. When the strength is further increased to λ=2.0, generation artifacts become more pronounced, RC falls to 26.82, below the baseline, and VC also drops from its peak to 29.34. Therefore, the intervention strength needs to balance rule compliance, visual preservation, and generation quality; λ=1.0 achieves the highest Overall score (34.93), supporting its use as the default setting.

Component contributions. Keyword distillation identifies task-relevant cues, relevance mapping locates these cues in visual tokens, and bias injection changes attention to these locations during generation. To separately examine the roles of additional textual prompts, arbitrary attention perturbations, and correct spatial localization, we evaluated the following conditions: the original Bagel, appending only the distilled keywords to the prompt, spatially shuffling the relevance map, and the full method. Spatially shuffling the relevance map means first computing the actual relevance map and then randomly shuffling the positions of the bias values within each image.

ConditionRCVCAQOverall
Bagel27.6719.0569.0526.72
Only appending distilled keywords to the prompt30.8320.3768.6429.06
Spatially shuffled relevance map25.4617.6464.0724.65
Full method33.2834.0960.6734.93

Additional textual cues provide some gains, and correct spatial localization and attention modulation further improve performance. Appending keywords alone increases Overall from 26.72 to 29.06, showing that explicitly providing task cues helps generation; the full method further increases it to 34.93, a gain of 5.87 points over keyword prompting. After spatially shuffling the relevance map, Overall drops to 24.65, and both RC and VC fall below the baseline. This control preserves the bias values but breaks their correspondence with visual locations, showing that the intervention's benefits depend on correctly locating task cues. The results support the combined effect of keyword distillation, relevance mapping, and bias injection: after distilling the cues, they still need to be located in visual tokens and used to guide attention allocation during generation.

AuthorsOpenReview ↗

Response to Reviewer BAQ4's Review Comments (2 of 2)

W2 Task differences between recognition and generation, and the effects of input format

First, the task differences between recognition and generation and the effects of input format are two separate aspects.

Regarding the multiple-choice question design, thank you for bringing this to our attention. We conducted an additional open-ended target-planning evaluation:

Open-ended target-planning evaluation. To examine whether candidate answers provide additional help, each model received only the original image–text context and task and freely described the target objects, attributes, sources, spatial relationships, and operations, without receiving RC hints, candidate answers, or generated images. The evaluator used the human-authored RC hints to score the completeness of the target descriptions on a 0/0.5/1 scale. We converted these scores to a 100-point scale and report the results below.

ModelVQA accuracy (%)Open-ended planning scoreGeneration RCGeneration Overall
Nano Banana Pro94.1183.5659.8457.28
Bagel65.6848.5327.6726.72

The gap between target description and rule execution persists after removing candidate answers. Nano Banana Pro and Bagel score 83.56 and 48.53, respectively, on the open-ended questions. Both scores are lower than the corresponding four-choice VQA accuracies but remain substantially higher than the mean generation RC scores (59.84 and 27.67). This shows that, even when required to describe the target independently, the models can still express more task constraints than their final images realize. Here, RC aligns the comparison in terms of target constraint satisfaction, while Overall additionally reflects comprehensive generation performance, including visual preservation and aesthetic quality.

Input formats already have a dedicated comparison. Appendix D.1/Figure 6 compares Edit Mode, Interleaved Mode, and Fine-Grained Interleaved Mode for Nano Banana Pro, Nano Banana, and Bagel. Edit Mode separates images and text and references images by number; the two Interleaved modes insert images at the boundaries of complete semantic units and at the positions of references within sentences, respectively. The existing results show relatively pronounced differences between Edit Mode and interleaved formats, with smaller differences between the two interleaved formats; the specific changes depend on the model. This comparison shows that input compatibility affects performance, and comparisons between models need to be interpreted in light of their respective input formats.

Q1 Can the attention intervention generalize to other models?

Appendix D.3/Table 4 has already validated the transfer effect on GLM-Image. The updated Overall score increases from 24.75 to 30.32, a gain of 5.57 points, and RC improves on all five tasks. The corresponding gain on Bagel is 8.21 points.

BackboneBaseline RCRC after interventionBaseline OverallOverall after interventionOverall gain
Bagel27.6733.2826.7234.93+8.21
GLM-Image23.3526.1124.7530.32+5.57

These results show that the intervention can improve rule compliance and overall performance in both UMMs. Its benefits are reproducible across backbones.

Thank you again for your comments. These supplementary experiments make the paper's arguments more complete. We hope the responses above address your questions; if you have any further questions, we welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.