- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GENIUS is a benchmark for evaluating Generative Fluid Intelligence (GFI): the ability to infer patterns, execute ad-hoc constraints, and adapt to rules that conflict with prior knowledge using the immediate multimodal context. It contains 510 samples spanning three dimensions and five tasks. Human-authored evaluation hints guide multimodal judges in assessing Rule Compliance (RC), Visual Consistency (VC), and Aesthetic Quality (AQ). The authors evaluate 14 models and investigate failures through context removal, hint-based interventions, and VQA probes. They also propose a training-free attention intervention that improves the Overall score from 26.74 to 32.92 on Bagel and from 24.71 to 30.04 on GLM-Image (Tables 2 and 4).
Critical strengths
-
Contextual knowledge adaptation is a valuable evaluation target. Counterfactual rules and semantic disambiguation test whether models follow local conditions rather than defaulting to prior knowledge. NanoBananaPro’s RC scores of 52.97 on Prior-Conflicting Generation and 35.45 on Multi-Semantic Generation indicate that these settings remain challenging.
-
The benchmark offers diverse tasks and concrete evaluation guidance. The tasks cover preference induction, operation transfer, conditional execution, and semantic transfer, with sample-specific RC/VC hints. Complete inputs, evaluation hints, and model outputs in the appendix clarify the intended requirements.
-
The experiments establish the importance of context. Removing context substantially reduces RC on several tasks, while input formatting also affects performance. These controls support the benchmark’s emphasis on contextual information use.
-
Automatic evaluation receives some validation. Ratings from five human experts and evaluation by a second multimodal judge provide stronger support than reliance on a single judge. The reported human–judge correlations are high, and model rankings are broadly consistent across judges.
Critical weaknesses
-
Some evaluation criteria may conflict with the task objectives. The AQ rubric rewards adherence to physics, anatomy, and visual realism, whereas Prior-Conflicting Generation explicitly requires counterfactual behavior, including gravity anomalies and abnormal biological growth. Since the AQ judge receives only the output image, it may penalize intentional deviations as generation errors (Sections 2.2–2.3; Figure 1; Appendix Figure 10). The VC rubric also incorporates instruction compliance and image quality, creating overlap with RC and AQ (Appendix Figure 9). In addition, Appendix A.2 assigns zero VC to exact copies of reference images without clarifying how cases requiring an unchanged output are handled. These potential systematic biases warrant targeted validation.
-
The need for additional terminology is insufficiently justified. The introduction and Section 2.1 introduce CI, FI, GFI, and three capability dimensions, although several tasks overlap with established in-context generation and reference-based editing settings, such as Edit Transfer[1] and VisualCloze[2]. The paper does not sufficiently explain why existing terminology is inadequate or what introducing these additional terms contributes to understanding the benchmark. Without this justification, the expanded vocabulary increases the reader’s burden without a clear conceptual benefit.
[1] Lan Chen, Qi Mao, Yuchao Gu, and Mike Zheng Shou. Edit Transfer: Learning Image Editing via Vision In-Context Relations. arXiv preprint arXiv:2503.13327, 2025. https://arxiv.org/abs/2503.13327
[2] Zhong-Yu Li, Ruoyi Du, Juncheng Yan, Le Zhuo, Qilong Wu, Zhen Li, Peng Gao, Zhanyu Ma, and Ming-Ming Cheng. VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 18969–18979, 2025. https://visualcloze.github.io/
-
The recognition–generation comparison does not sufficiently align task demands and metrics. Section 3.2 and Figure 2(b) compare four-choice VQA accuracy with generation Overall scores. However, multiple-choice recognition provides candidate answers, while Overall additionally measures visual consistency and aesthetics; the scores are therefore not directly comparable. The illustrated distractors cover some omissions and incorrect sources, but systematic coverage of generation failures such as role swaps and attribute misbinding is not established. The experiment supports the weaker conclusion that recognizing an intended outcome is easier than generating it, but does not yet establish that models correctly understand all constraints and subsequently fail to execute them.
-
The theory and attention visualizations do not establish the proposed failure mechanism. Theorem 4.1 provides a local output equivalence between context changes and parameter perturbations, and Theorem 4.2 expresses the constructed updates as gradient steps. However, the linear loss in Appendix G.3 is defined from the update itself; its connection to rule compliance, image quality, or meaningful task optimization remains unclear (Section 4.2; Appendix G). Figure 3 also leaves the selection and aggregation of layers, heads, and generation steps insufficiently specified, along with the correspondence between attention peaks and textual or visual regions. Since the method directly injects a relevance bias, a more concentrated attention distribution primarily confirms that the intervention changes attention. It does not independently establish that the original distribution is noise or that an implicit optimization direction has been corrected.
Questions for authors
-
Evaluation protocol: How does AQ distinguish task-required counterfactual or non-photorealistic outputs from genuine generation errors? Should the judge receive the task context? How is the copying penalty handled when leaving the image unchanged is correct?
-
Terminology and framing: Why is the additional terminology necessary rather than describing the tasks using established terms in in-context generation and reference-based editing? What important distinction would be lost by retaining the existing terminology?
-
Recognition–generation alignment: Could the VQA probes include systematic role-swap and attribute-misbinding distractors, and report the proportion of samples where VQA is correct but generation fails the same semantic constraint?
-
Mechanistic evidence: How are the attention scores in Figure 3 selected and aggregated? What establishes that suppressed tokens are irrelevant? How do the equivalent parameter updates relate to a meaningful task objective rather than an algebraic reformulation?
4 comments
Response to Reviewer PkX9's Review Comments (2 of 4)
W2 and Q2: Overlap in editing operations and the diagnostic value of the terminology
The capability being evaluated depends on how the model determines the target operation. ARC-AGI-2 is a fluid intelligence benchmark in the understanding domain. ARC-AGI-2's final output is a transformed colored grid, but the evaluation target is to determine the transformation rule from the current examples and apply it to a new input. GENIUS applies the same rule acquisition requirement to generation driven by image–text context. The output may take the form of an edit, but the evaluation also requires determining the meanings of temporary symbols, the correspondence between operations and visual examples, and whether the current local knowledge overrides real-world priors.
Compared with the two works you mentioned:
| Work | Their contribution | Evaluation focus of GENIUS |
|---|---|---|
| Edit Transfer | Learns an editing transformation from a single source–target example and applies it to a query image | Organizes temporary bindings jointly defined by text and images, operation composition, preference disambiguation, and local overrides of priors into a systematic task collection |
| VisualCloze | Uses visual demonstrations to identify tasks, supporting multiple generation tasks and generalization to unseen tasks | Provides a unified evaluation that separately measures rule induction, execution of current constraints, and contextual knowledge adaptation, and evaluates rule adherence and quality in generated outputs |
We introduce GFI and its three dimensions to distinguish the rule acquisition and application requirements underlying generation operations: Implicit Pattern Induction discovers unstated regularities from examples; Ad-hoc Constraint Execution uses the bindings, conditions, and combinations defined for the current task; and Contextual Knowledge Adaptation adopts the current local knowledge or interpretation. These dimensions associate similar output operations with different reasoning requirements.
Response to Reviewer PkX9's Review Comments (3 of 4)
W3 and Q3: Aligning the semantic constraints of recognition, open-ended planning, and generation
Generation RC has been added to Figure 2(b) and the main text. The updated figure presents VQA accuracy, mean generation RC, and Overall together; RC is mapped from the 0–2 rating scale to 0–100.
We conducted an additional open-ended target-planning evaluation: Open-ended target-planning evaluation. To examine whether candidate answers provide additional help, each model received only the original image–text context and task and freely described the target objects, attributes, sources, spatial relationships, and operations, without receiving RC hints, candidate answers, or generated images. The evaluator used the human-authored RC hints to score the completeness of the target descriptions on a 0/0.5/1 scale. We converted these scores to a 100-point scale and report the results below.
| Model | VQA accuracy (%) | Open-ended planning score | Generation RC | Generation Overall |
|---|---|---|---|---|
| Nano Banana Pro | 94.11 | 83.56 | 59.84 | 57.28 |
| Bagel | 65.68 | 48.53 | 27.67 | 26.72 |
The gap between target description and rule execution persists after candidate answers are removed. Nano Banana Pro and Bagel score 83.56 and 48.53 on the open-ended questions, respectively. Both scores are lower than their corresponding four-choice VQA accuracies, but remain higher than their mean generation RC scores (59.84 and 27.67). This indicates that even when required to describe the target independently, the models can still express more task constraints than their final images realize; the difference between recognition and generation therefore cannot be explained solely by the cues provided by candidate answers. Here, RC aligns the evaluation of target-constraint satisfaction, while Overall additionally reflects comprehensive generation performance, including visual preservation and aesthetic quality.
Regarding distractors, Figure 2(b) already includes examples of constraint omission and source/role swaps. Option B omits the style requirement, C swaps the sources of the character and clothing, and D contains an incorrect subject and omits the clothing requirement. By omitting key constraints or confusing reference sources, these options test whether the model can identify the target that satisfies all requirements simultaneously. This example therefore already covers the issues of constraint completeness and source correspondence that you suggested examining. We computed the requested paired statistics and report them below:
| Model | VQA correct and RC=2 (%) | VQA correct and RC=1 (%) | VQA correct and RC=0 (%) | VQA incorrect and RC=2 (%) | VQA incorrect and RC=1 (%) | VQA incorrect and RC=0 (%) |
|---|---|---|---|---|---|---|
| Nano Banana Pro | 45.87 | 26.54 | 21.70 | 0.00 | 1.4 | 4.49 |
| Bagel | 13.47 | 24.38 | 27.83 | 0.89 | 2.24 | 31.19 |
The paired results further show that selecting the correct target does not always translate into complete rule execution. All six proportions in the table use the full set of paired records as their denominator. Restricting the analysis to records with correct VQA answers, 51.26% and 79.49% of the generated outputs from Nano Banana Pro and Bagel, respectively, still fail to reach RC=2: they satisfy only some constraints or fail to satisfy the target rules. These results directly show that, on tasks for which the model selects the correct target, the generated image may still omit or violate rules. They support analyzing the models' capability differences through the two stages of target recognition and generation execution.
Response to Reviewer PkX9's Review Comments (4 of 4)
W4 and Q4: Attention evidence, intervention controls, and theoretical interpretation
Thank you for your suggestions regarding attention evidence and theoretical interpretation. We respond in three parts: what the visualization displays, the role of locating task-relevant information, and the conclusions supported by the theoretical derivation.
The attention visualization shows how context is used during generation. Figure 3 uses the image tokens during generation as queries to examine their allocation of attention to the image–text context. Appendix F.3 provides the intervention settings: layers [8,24], generation steps [10,35], and a default strength of λ=1.0. The visualization shows the change in the attention distribution before and after intervention; whether task-relevant locations contribute to performance improvements is tested by the spatial shuffling control below.
The spatial shuffling control supports the role of correctly locating task cues. We first computed the actual relevance map and then randomly shuffled the positions of the bias values within each input image, preserving the set of bias values while disrupting their correspondence with visual locations. After spatial shuffling, RC, VC, and Overall are 25.46, 17.64, and 24.65, respectively, all below the original Bagel scores of 27.67, 19.05, and 26.72. The full method reaches 33.28, 34.09, and 34.93, respectively. Compared with the shuffled control, the full method improves RC, VC, and Overall by 7.82, 16.45, and 10.28 points, respectively. This shows that locating task cues at their corresponding visual positions is an important condition for attention intervention to yield benefits.
The theory provides a parameter-update perspective for understanding the effect of context. For a given layer and intermediate generation state, Theorem 4.1 constructs a parameter perturbation that compensates for a change in context, establishing an equivalence between layer outputs. Theorem 4.2 further expresses the corresponding parameter changes as gradient updates on a constructed linear surrogate objective. This surrogate objective characterizes parameter differences and is not equivalent to the loss function of the actual generation task. The theory thus provides an explanatory framework for adjusting the influence of context, while the effect of a specific intervention on rule compliance is tested experimentally: the spatial shuffling control supports the value of locating task-relevant information, and the λ sweep shows the relationship between intervention strength, rule compliance, visual preservation, and generation quality. We will revise the corresponding main-text discussion and Figure 4 wording and caption to make this distinction explicit and align the theoretical claims with the scope of the derivation.
Thank you again for your comments. These supplementary experiments make the paper's arguments more complete. We hope the responses above address your questions; if you have any further questions, we welcome further discussion.
Response to Reviewer PkX9's Review Comments (1 of 4)
Thank you for your specific suggestions on the boundaries between scoring criteria, recognition–generation alignment, and the interpretation of attention. We have completed the updates to the AQ definition and results, and added generation RC to Figure 2(b) in the revised PDF. Below, we explain the current protocol and report the results of our supplementary analyses.
W1 and Q1: The scoring boundaries between AQ and VC, and the treatment of unchanged outputs
The AQ definition update and re-evaluation are complete. The original scoring criterion treated adherence to real-world physical and anatomical laws as a universal requirement, which conflicts with the task setting of Prior-Conflicting Generation. Thank you for pointing this out. We have removed this requirement and updated the main text, appendix, and evaluation prompt accordingly. AQ focuses on composition, visual harmony, rendering clarity, and generation artifacts; surreal content, non-realistic styles, or unusual physical phenomena required by the task should not themselves lead to a deduction. AQ continues to evaluate only the output image, while RC checks whether the task rules are satisfied.
We have updated the result tables and related figures in the current manuscript using the revised AQ criteria and recomputed Overall. After the revision, Nano Banana Pro has an Overall score of 57.28. In the main results, the only change in relative ordering involves our method, which moves from eighth to sixth place, overtaking FLUX.2-dev and Gen-Searcher; the relative ordering among the baseline models remains unchanged. The main observation that attention intervention improves overall performance on both the Bagel and GLM-Image backbones is preserved.
VC measures the visual similarity between the reference content specified for each sample and the corresponding content in the generated image. For example, when a task requires preserving a reference character's identity while changing their clothing, RC checks whether the output satisfies the specified target conditions, such as the clothing requirements, while VC uses the corresponding hint to check the character appearance that should be preserved. Here, HINT refers to the sample-specific visual preservation requirements, and “follow the HINT” in the template refers to how well the output image satisfies those requirements.
Terms such as “professional,” “high-quality implementation,” and “smooth transformation” describe the quality and fidelity with which the specified reference content is preserved in the generated image. They are used within this reference-based comparison: the judge assesses the corresponding visual content identified by the VC hint. RC is evaluated independently using its own hints to assess whether the task rules are satisfied.
Regarding the copy penalty for unchanged images, we have ensured that no task has a target image that is simply an unchanged input image, so if the evaluated model copies the original image, its output is necessarily incorrect.