acceptodds
Under review as a conference paper at ICLR 2027

When Worlds Meet: Benchmarking Implicit Causal Reasoning with IRS-Bench

Abstract

An image records a partial snapshot of a local world. Combining foreground and background snapshots therefore requires more than harmonizing pixels: the merged scene must realize the state changes and scene effects licensed by their interaction. Existing benchmarks for image composition primarily measure perceptual quality, identity preservation, and spatial alignment, leaving this merge-and-reaction problem untested. We formulate it as image-conditioned implicit causal reasoning and introduce IRS-Bench. Each instance provides a foreground image F, a background image B, and an instruction q specifying the entities and their placement while withholding the governing relation and its consequence. The source images fix entity identities; post-interaction states and scene effects must be inferred from visual evidence. To accept valid alternatives, IRS-Bench distinguishes deterministic interactions with one admissible state from intentional interactions with an admissible set. The benchmark contains 1,000 pairs across eleven leaf tasks and five causal mechanisms: physical, biological, chemical, institutional, and functional. Each item includes a governing rule, admissible outcomes, and weighted binary visual checks; 172 items also provide photographic post-interaction references. Under the ME-Ruler score, the strongest non-ReactionGraph baseline obtains 61.1 and 71.6 under GPT-5.6 SOL and Qwen-3.8-27B, respectively, across 18 systems. A training-free ReactionGraph reference pipeline obtains 77.2 and 88.8 under the same judges. These results show that perceptual coherence and causal faithfulness are distinct capabilities and that models must carry interaction constraints across visual contexts rather than merely produce plausible composites.

Primary area:
datasets and benchmarks
Keywords:
image Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6c1rOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces IRS-Bench, a benchmark designed to evaluate implicit causal reasoning in image-conditioned composition. Unlike existing image composition benchmarks that primarily focus on visual quality, spatial alignment, and identity preservation, IRS-Bench examines whether image generation models can infer and realize the causal consequences arising from interactions between foreground objects and background scenes. The benchmark contains 1,000 image pairs spanning five causal dimensions and 11 fine-grained tasks, covering physical, biological, chemical, institutional, and functional causality. To evaluate generated images, the authors introduce ME-Ruler, a rule-based evaluation framework consisting of weighted binary visual checks across five dimensions. Additionally, the paper proposes ReactionGraph, a training-free pipeline that explicitly infers causal relationships, translates them into structured constraints, and guides image generation accordingly. Experiments across a variety of image generation models demonstrate substantial performance gaps on IRS-Bench, while ReactionGraph consistently improves causal correctness over direct image composition.

Critical strengths
  1. Interesting and meaningful task formulation. The paper investigates an important yet relatively underexplored aspect of image generation: whether models can infer and realize implicit causal consequences when combining different visual contexts. Moving beyond conventional evaluations of visual fidelity and spatial consistency toward interaction-aware generation is a meaningful direction.

  2. Broad coverage of causal interactions. IRS-Bench includes 1,000 samples covering five causal dimensions and 11 fine-grained tasks. The inclusion of physical, biological, chemical, institutional, and functional interactions provides a relatively diverse evaluation setting for examining different types of causal reasoning capabilities.

  3. Fine-grained evaluation framework. The proposed ME-Ruler decomposes evaluation into weighted binary visual checks, allowing more interpretable assessments than a single holistic score. The distinction between deterministic and intentional interactions is also reasonable, as some scenarios may admit multiple valid causal outcomes rather than a unique reference.

  4. Extensive experimental comparisons. The authors evaluate a broad range of image generation and editing models, including both open-source and proprietary systems. The experiments reveal noticeable limitations of existing models in handling implicit causal interactions, and the additional analysis with ReactionGraph suggests that explicit reasoning can improve the causal consistency of generated images.

Critical weaknesses
  1. The Introduction lacks a clear motivation and research positioning. In Section 1, the overall presentation is somewhat disorganized, and the motivation is not sufficiently supported by a systematic discussion of existing studies. The authors introduce the world-merging problem and implicit causal reasoning, but do not clearly explain how the proposed task differs from existing reasoning-driven image generation and image composition tasks. A more explicit comparison with representative prior work is needed to establish the research gap and motivate the proposed benchmark.

  2. Several key concepts are not clearly introduced or defined. In Section 1, the authors introduce terms such as "image-conditioned implicit causal reasoning," "identity lock," and "state freedom" without providing sufficiently clear explanations. In particular, the distinction between identity preservation and state changes is central to the proposed task, but is not adequately illustrated in the main text. Although some formal definitions are provided in Appendix A, these concepts should be clearly explained when first introduced.

  3. The Introduction does not clearly summarize the main findings and insights of the benchmark. In the latter part of Section 1, the authors mainly describe the dataset construction, evaluation protocol, and performance improvements achieved by ReactionGraph. However, for a benchmark paper, the main contributions should also include meaningful observations about the capabilities and limitations of existing models. The Introduction would benefit from a clearer summary of the key findings revealed by IRS-Bench, rather than focusing primarily on performance comparisons.

  4. The Related Work section is incomplete and insufficiently aligned with the paper's focus. In Section 2, the discussion of image generation is largely centered on text-to-image generation, despite the fact that the proposed benchmark focuses on image-conditioned composition. The authors should provide a more comprehensive review of image-to-image generation, image editing, and multi-image composition methods. Furthermore, the discussion of reasoning-driven generation benchmarks is limited, particularly regarding image-conditioned reasoning benchmarks. Table 1 also requires a clearer explanation of how IRS-Bench differs from the most closely related benchmarks, rather than simply comparing their supported evaluation dimensions.

  5. The benchmark taxonomy lacks sufficient justification. In Section 3.1, the authors divide causal reasoning into five dimensions and 11 leaf tasks, but do not explain how this taxonomy was established. It is unclear whether the categories are based on existing causal theories or manually defined by the authors. More importantly, the authors do not discuss whether these 11 tasks adequately cover the causal reasoning capabilities they intend to evaluate. A clearer explanation of the taxonomy design and its coverage is necessary to justify the completeness and representativeness of IRS-Bench.

  6. The experimental section lacks meaningful analysis and conclusions. In Section 5, most experiments primarily present quantitative comparisons showing that ReactionGraph outperforms existing methods, with limited discussion of what can be learned from these results. As a benchmark paper, IRS-Bench should provide more systematic insights into the strengths and weaknesses of different generation models, the relative difficulty of different causal reasoning tasks, and the common failure patterns observed across models. Simply reporting performance rankings and improvements is insufficient to demonstrate the full value of the proposed benchmark.

  7. The presentation of ReactionGraph is unclear. In Section 4.2, the authors introduce ReactionGraph mainly through mathematical formulations and textual descriptions. However, the overall pipeline, including reaction-graph inference, constraint compilation, and state realization, is difficult to follow without a framework illustration. A dedicated overview figure showing the inputs, intermediate representations, and outputs of each stage would substantially improve the clarity of the method.

  8. The technical novelty of ReactionGraph is limited. The proposed ReactionGraph is essentially a multi-stage prompt-engineering pipeline built upon existing multimodal language models and image generation models. It first prompts an MLLM to infer causal relationships, converts the inferred results into structured constraints, and then uses these constraints to guide image editing. The method does not involve a new model architecture, training strategy, or fundamentally new reasoning mechanism. Although the authors organize the reasoning results into a causal graph, it is unclear whether this graph representation provides substantial benefits over directly generating a detailed natural-language editing instruction. The ablation study in Section 5.3 shows that most of the improvements already come from introducing explicit causal reasoning. Therefore, the authors should better justify the necessity of the proposed graph-based formulation and clarify its technical contributions beyond prompt engineering.

All the weaknesses listed above are based on my own comments and observations after carefully reading the entire manuscript. I used an LLM solely to polish and organize the wording of these comments.

Overall, I believe the manuscript still falls short of the expected standards of academic writing and presentation. In particular, the Introduction and Related Work sections are not well structured and lack the rigor and clarity expected of a conference paper. Many important concepts and terms are introduced without proper definitions or sufficient explanations, making the paper unnecessarily difficult to follow. In my view, the manuscript requires substantial restructuring and extensive polishing before it can meet the standards of a well-written academic paper.

Questions for authors

See the Weakness part.

12 comments
AuthorsOpenReview ↗

Response to the Overall Comments from Reviewer 6c1r

We thank Reviewer 6c1r for the detailed review and for recognizing the task formulation, the five-mechanism coverage of IRS-Bench, the interpretability of ME-Ruler, and the breadth of our evaluation. Your comments on structural organization, upfront concept definitions, and diagnostic depth provided concrete guidance for strengthening the exposition. In response to W1–W8, we have thoroughly revised and restructured the manuscript (with updates highlighted in blue in the revised PDF): we reorganized Section 1 around prior-work positioning, inline definitions of core terms, and key benchmark findings; expanded Section 2 and Table 1 to align with image-conditioned editing, composition, and reasoning benchmarks; added taxonomy justification and coverage boundaries in Section 3.1 and Appendix B; included a dedicated pipeline overview (Figure 3) and budget-matched comparisons (Table 10) to clarify ReactionGraph's diagnostic role; and expanded Section 5 with paradigm-level, mechanism/regime, and failure-mode analyses. Below, we address each point in turn and note the exact locations of the corresponding revisions.

AuthorsOpenReview ↗

Response to W1: Introduction motivation and research positioning

We greatly appreciate the reviewer's careful review of the manuscript. In the revised manuscript, we have restructured Section 1 into a clear four-step progression: (1) world-merging motivation$\rightarrow$(2) systematic positioning against four closely related benchmark paradigms$\rightarrow$(3) upfront task formulation and regime distinction$\rightarrow$(4) benchmark design, key empirical findings, and aligned contributions.

Specifically, we have added a second paragraph to Section 1 that organizes the comparison around text-to-image generation, reference-guided generation, single-image reasoning-based editing (Reason50K, PBench-Edit), and multi-image composition (OmniContext, XVerseBench, MICON-Bench)

The added paragraph states the evaluation objective and input condition of each setting. We group prior work into four categories: text-to-image generation, image-to-image generation, image editing, and multi-image composition. Text-to-image generation provides no visual anchor for entity identity; image-to-image generation emphasizes subject preservation; image editing focuses on changes within an existing scene; and multi-image composition integrates distinct visual references — OmniContext and XVerseBench emphasize subject consistency and instruction alignment, while MICON-Bench also evaluates causal story continuation.

Importantly, the revised discussion identifies the more specific gap: when independent foreground and background images are combined under placement-only instructions, interaction consequences are not systematically evaluated. This comparison gives the proposed task a clearer motivation and connects the world-merging problem introduced in the first paragraph to the task formulation that follows.

Table R1: Positioning of IRS-Bench against the four prior task families. ✓ yes, ○ partial, ✗ no.

Setting (representative work)What the model must inferImage-pair inputOutcome withheld from the textCross-source inferenceJudged on the post-interaction statePhotographic post-interaction GTMultiple admissible outcomes
I. Text-to-image generation (Commonsense-T2I, WISE, R2I-Bench, T2I-ReasonBench, T2I-CoReBench)the knowledge and relations named in the prompt✗✗✗✗✗✗
II. Image-to-image generation (DreamBench, DreamBench++)how to carry the subject into a new scene○✗✗✗✗✗
III. Image editing (Reason-Edit, Reason50K, PBench-Edit)the visual realization of a stated operation✗○✗○✗✗
IV. Multi-image composition (OmniContext, XVerseBench, MICON-Bench, MDMR-Bench, DeCo-Bench, LongBanana)how to blend consistently✓○○✗✗○
IRS-Bench (ours)the governing relation and its consequences✓✓✓✓✓✓

the table shows, ours is the only setting that satisfies every criterion. This is the gap the benchmark targets: the interaction consequence, rather than the arrangement or the operation, is what the model must supply.

For the reviewer's convenience, the revised positioning paragraph in Section 1 (Para. 2) reads as follows:

Modified sentences:

1 INTRODUCTION

....

"Existing generation benchmarks address different aspects of combining visual contexts. Text-to-image generation benchmarks assess compositional alignment and knowledge-based reasoning, but text alone provides no visual anchor for entity identity and may implicitly reveal the intended outcome. Image-to-image generation benchmarks introduces visual exemplars to evaluate subject preservation and prompt following. Image editing benchmarks further examines changes within an existing scene, including hypothetical edits in Reason50K (He et al., 2026) and physically plausible action edits in PBench-Edit (Wu et al., 2025c).Multi-image composition benchmarks extends evaluation to integrating distinct visual references. OmniContext (Wu et al., 2026a) and XVerseBench (Chen et al., 2026a) emphasize subject consistency and instruction alignment, while MICON-Bench (Wu et al., 2026b) also assesses causal story continuation. However, these studies provide limited systematic assessment of interaction consequences when independent foreground and background images are combined under placement-only instructions."

AuthorsOpenReview ↗

Response to W2: Upfront Explanation of Core Concepts

Thanks for this detailed suggestion. We have added clear explanations for the terms "image-conditioned implicit causal reasoning," "identity lock," and "state freedom" where they first appear in the main text. For convenience to the reviewer, the related sentences are shown as follows.

Modified sentences:

1 INTRODUCTION

....

"We formulate this task as image-conditioned implicit causal reasoning: inferring an unstated interaction and rendering its visual consequence from a reference image pair when the text specifies only entities and placement. An instance provides (B,F,q)(B,F,q), where BB is a background image, FF is a foreground image, and qq specifies the entities and their placement without naming the governing relation or outcome. Crucially, the task couples an identity lock with state freedom: source entities must retain their identities and counts without introducing external objects, while their post-interaction states and surrounding scene effects must evolve as licensed by the visual premises. This setting avoids a central confound in text-conditioned evaluation, where linguistic cues in the prompt can disclose the target outcome without requiring visual inference."

Reviewer_TBNSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces IRS-Bench, a benchmark for evaluating implicit causal reasoning in image-conditioned composition. The authors argue that existing image composition benchmarks primarily focus on visual quality and spatial consistency while overlooking the causal consequences of object interactions. IRS-Bench contains 1,000 image pairs covering 11 subtasks across five causal mechanisms. The paper also introduces ME-Ruler, a fine-grained evaluation framework based on weighted visual criteria, and ReactionGraph, a training-free method that explicitly infers causal relationships and translates them into image generation constraints. Experiments across various image generation models demonstrate the limitations of existing methods in capturing implicit causal effects and show that ReactionGraph can improve causal consistency.

Critical strengths
  1. The paper identifies an interesting and underexplored problem in image composition: whether generated images correctly reflect the causal consequences of object interactions. The distinction between deterministic and intentional causal relationships is particularly meaningful, as it allows the benchmark to accommodate multiple valid outcomes rather than relying on a single reference image.

  2. The proposed IRS-Bench is relatively comprehensive, covering 1,000 image pairs, 11 subtasks, and five causal mechanisms. The fine-grained ME-Ruler evaluation framework and extensive comparisons across image generation models provide useful insights into the limitations of existing methods. The controlled experiments further strengthen the empirical analysis.

Critical weaknesses
  1. The novelty of ReactionGraph is relatively limited. The framework mainly combines explicit causal reasoning, structured constraint generation, and existing image editing models. Although the results are encouraging, it remains unclear whether the proposed graph structure is necessary compared with simpler reasoning-based approaches.

  2. The proposed ME-Ruler relies heavily on multimodal language models as evaluators. Although multiple evaluators are used, their scores differ considerably, raising concerns about evaluation consistency. Furthermore, the human evaluation reported in Section 5.6 appears inconsistent with statements in the appendix indicating that the study is still in progress.

Questions for authors

Q1. Section 5.6 reports human evaluation results, while Appendix D and Appendix G indicate that the corresponding human study is still in progress. Could the authors clarify this inconsistency? Additionally, does ME-Ruler achieve similar agreement with human judgments across all five causal mechanisms?

Q2. Since ReactionGraph introduces additional reasoning and constraint generation steps, could the authors provide comparisons with simpler reasoning-augmented baselines under a similar inference budget? This would help clarify whether the improvements come specifically from the proposed graph structure.

Q3. Could the authors provide the exact prompts, model configurations, and implementation details required to reproduce ReactionGraph? Furthermore, would it be possible to include quantitative evaluations of the multi-foreground composition experiments to better support the generalization claims?

8 comments
AuthorsOpenReview ↗

Response to the Overall Comments from Reviewer TBNS

We sincerely thank Reviewer TBNS for the encouraging feedback and for appreciating our formulation of implicit causal consequences, the deterministic-versus-intentional distinction, the coverage across 1,000 pairs and 11 subtasks, and the fine-grained ME-Ruler design. We value your questions on graph necessity under comparable reasoning budgets, evaluator consistency and human-study status, reproducibility, and quantitative multi-foreground results, and we provide detailed responses below.

AuthorsOpenReview ↗

Weakness 1 & Question 2: Novelty of ReactionGraph and comparison with simpler reasoning baselines under matched budget

As a benchmark paper, a large amount of effort has been invested in constructing the proposed benchmark, in order to expose the deficiencies of existing image composition/generation models, and baseline methods are provided to promote community development. The core contribution of ReactionGraph lies in reformulating the image composition problem as a temporal inference from the initial background state to the naive composite state. Unstructured methods lack verifiable intermediate structure, whereas static graphs struggle to express the causal consequences produced by multi-image interaction. ReactionGraph compiles inference into atomic constraints that explicitly specify what should change and what should be preserved, so that causal reasoning is grounded as executable and verifiable generation requirements, thereby connecting causal reasoning with image generation and reducing causal omission or irrelevant modification; as can also be seen from Table 14, compared with other knowledge-graph methods, we achieve better results.

Following the reviewers' suggestions, we have added comparisons of ReactionGraph with simple reasoning methods, and of some related work, under comparable inference cost.

Table 10: Reasoning-budget-matched comparison on the 155-item ablation subset. Columns: Mech = mechanical causality; Opt = optical; Fluid = fluid dynamics; Thermo = thermodynamics; EM = electromagnetism; Zool = zoology; Bot = botany; Physio = physiology; Chem = chemistry; Inst = institutional; Func = functional. Per column, best in bold, second underlined, third marked with *.

SystemReasoning budget (k tokens)MechOptFluidThermoEMZoolBotPhysioChemInstFuncOverall
Reason-only~21.870.1672.66‾\underline{\text{72.66}}80.2379.5177.5867.39‾\underline{\text{67.39}}70.16*76.3259.1253.4366.7871.92‾\underline{\text{71.92}}
R2E~22.166.79*67.70*70.7575.3773.39*66.83*58.3867.4172.65‾\underline{\text{72.65}}52.9572.74*68.17*
LayerCraft~21.965.7567.6673.23*69.8063.0265.3243.7175.78‾\underline{\text{75.78}}54.6257.53*78.7666.91
LayoutAgent~21.863.5367.1872.5675.88*67.1464.0674.1471.0664.76*68.32‾\underline{\text{68.32}}68.7267.62
ReactionGraph (ours)~20.969.74‾\underline{\text{69.74}}73.5575.45‾\underline{\text{75.45}}77.27‾\underline{\text{77.27}}74.29‾\underline{\text{74.29}}75.4373.15‾\underline{\text{73.15}}71.17*80.3778.2977.38‾\underline{\text{77.38}}74.09

As reported in Table 10, ReactionGraph obtains the highest overall score (74.09), exceeding the next-best system, Reason-only, by 2.17 points and the three structured-reasoning baselines by 5.92-7.18 points. It ranks in the top two on ten of the eleven subtasks and third on physiology, where Reason-only and LayerCraft score higher. Reason-only leads on five axes-mechanism, fluid dynamics, thermodynamics, lectromagnetism, and physiology---but scores lower on chemistry (59.12) and institutional causality (53.43). Thus, ReactionGraph's advantage is not that it dominates every axis, but that it achieves stronger overall and more consistent performance across the evaluated mechanisms. Unlike the comparison systems, which express their reasoning mainly through natural-language instructions, ReactionGraph represents the interaction explicitly as a causal graph and compiles its consequences into typed constraints. Under comparable reasoning budgets, ReactionGraph outperforms both the reason-only and structured-reasoning baselines overall.

The results show that our method comprehensively outperforms existing methods, demonstrating the effectiveness of the ReactionGraph.

The related experimental results and analysis have been added in Appendix E.4 and Table 10.

AuthorsOpenReview ↗

E.4 REASONING-BUDGET-CONTROLLED BASELINES

(We have presented the following content of E.4 REASONING-BUDGET-CONTROLLED BASELINES in the revised manuscript.)

We compare five reasoning-based systems on the same 155 samples: Reason-only, R2E (He et al.,2026), LayerCraft (Zhang et al., 2025), LayoutAgent (Fan et al., 2026), and ReactionGraph. Their reasoning budgets are comparable. All five systems use GPT-5 for reasoning (temperature=0) and gpt-image-2 for image composition. Reasoning tokens are counted per sample as the sum of prompt tokens and completion tokens across reasoning and self-check calls, excluding image generation and judging. As reported in Table 10, ReactionGraph obtains the highest overall score (74.09), exceeding the next-best system, Reason-only, by 2.17 points and the three structured-reasoning baselines by 5.92–7.18

points. It ranks in the top two on ten of the eleven subtasks and third on physiology, where Reason-only and LayerCraft score higher. Reason-only leads on five axes—mechanism, fluid dynamics, thermodynamics, electromagnetism, and physiology—but scores lower on chemistry (59.12) and in-stitutional causality (53.43). Thus, ReactionGraph’s advantage is not that it dominates every axis, but that it achieves stronger overall and more consistent performance across the evaluated mechanisms. Unlike the comparison systems, which express their reasoning mainly through natural-language in-structions, ReactionGraph represents the interaction explicitly as a causal graph and compiles its consequences into typed constraints. Under comparable reasoning budgets, ReactionGraph outperforms both the reason-only and structured-reasoning baselines overall.

Reviewer_8Pi5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes IRS-Bench, a benchmark for image composition in which the composite must show the consequence implied by the interaction between a foreground and a background. Examples include a bulb lighting up in a closed circuit, copper sulfate dissolving, and sheep reacting to a wolf. Each instance is (F, B, q). The instruction q names the entities and their placement but does not state the relation or its consequence, so the consequence has to be inferred from the images. The benchmark has 1,000 pairs: 454 AI-generated, 366 web/public-dataset, and 180 author-photographed. These cover 11 leaf tasks under 5 mechanisms: physical (700), biological (215), chemical (27), institutional (33), and functional (25).

Each instance is annotated with:

  • a causal regime: 727 deterministic, with one admissible state, and 273 intentional, with an admissible set;
  • a governing rule and the admissible outcomes;
  • about 33 weighted yes/no "rulers" (33,024 in total). GPT-5 proposed these, and four annotators filtered and weighted them.

172 instances also have a photographic post-interaction reference. The score (ME-Ruler) is the weighted fraction of rulers that an MLLM judge marks as satisfied. The main judges are GPT-5.6 SOL and Qwen-3.8-27B, and Claude Sonnet 5 is a third judge in the appendix.

Critical strengths

S1. The problem is well posed, under-tested, and the task design is sensible. Withholding the consequence from q and fixing entity identities through the images removes a real confound in text-to-image reasoning benchmarks such as WISE, T2I-ReasonBench, and R2I-Bench: there, the target outcome can be read off the prompt. Existing image-conditioned composition benchmarks (OmniContext, XVerseBench, MICON-Bench) do not test consequences at all. The deterministic/intentional split with admissible sets is a reasonable way to avoid penalizing valid alternatives that differ from a single reference.

S2. The evaluation is broad, and it yields a potentially useful diagnostic signal. The evaluation covers 21 systems from four groups: a copy-paste baseline, dedicated multi-reference composition methods, open-weight generation and editing models, and proprietary systems. They are scored by three judges from different model families, with per-task results for all 11 leaf tasks. Appearance metrics, ablations, controlled variants and a user study complement these results. This breadth makes the benchmark a useful snapshot of where current systems stand. It also shows that standard appearance metrics (FID, LPIPS, CLIP-I) barely separate systems that differ substantially on this task.

Critical weaknesses

W1: Much of the score does not depend on causal reasoning. Causal correctness carries 46% of the ruler weight; the remaining 54% covers spatial, domain, naturalness, and identity criteria. Copy-paste, which does no generation, scores 36.4/47.0. Under Claude Sonnet 5 (Table 5) it beats six of the seven dedicated composition systems. Under GPT-5.6 SOL it beats four and ties SHINE and CareCom. The paper itself calls copy-paste a lower bound (Sec. 4.1). So a large share of the score is reachable with no inference at all.

The paper never reports the causal-correctness axis on its own. It also gives no breakdown into deterministic vs. intentional instances, and no results on the 172-item photographic subset. Without these, the headline claim that "perceptual coherence and causal faithfulness are distinct" is not directly shown. Table 2 cannot tell us whether the rankings come from causal rulers or from identity and spatial ones.

W2: The paper's own showcased outputs break its identity lock. Eq. 5 requires that ent(Wm)⊆ent(WF)∪ent(WB)ent(W_m) \subseteq ent(W_F) \cup ent(W_B), and Sec. 3.3 says identity covers appearance and count. Yet in the "GPT Image 2 + Ours" outputs of Figure 3:

  • the beekeeper gains a veiled hat that is in neither input;
  • the firefighters become helmeted figures spraying a hose and water jet that appear in neither input, and their faces can no longer be recognized;
  • the apple tree loses its apples and leaves.

The added hat, hose and water jet also violate ReactionGraph's own rule that "no entity may be introduced that is absent from B and F" (Sec. 4.2). If these outputs are rewarded, then annotators and judges are drawing the line between identity and state ad hoc. If they are penalized, Figure 3 showcases failures. That line is exactly what the benchmark measures, so it needs an operational definition and agreement statistics.

Questions for authors

Q1: Human study status. Is the study in Sec. 5.6 / Table 11 complete? Please reconcile it with Appendices D, F.2, and G.

Q2: Causal-only results. Can the authors report the causal-correctness axis on its own, and each other axis, for every system in Table 2?

Q3: Time horizon and identity. How is the post-interaction moment specified for deterministic items? How do the rulers treat added objects, such as the beekeeper's veil or the firefighters' hose, and the apple tree that loses its apples, given Eq. 5 and the identity definition in Sec. 3.3?

10 comments
AuthorsOpenReview ↗

Response to the Overall Comments from Reviewer 8Pi5

We sincerely thank Reviewer 8Pi5 for the thoughtful review and for recognizing that anchoring entity identities via images, adopting deterministic/intentional admissible sets, and evaluating across diverse systems and judges provide a sensible formulation and useful diagnostic signal. Your points regarding score attribution, identity-lock boundaries in qualitative examples, and consistency across the main text and appendices are sharp and constructive, and we address each of them below.

AuthorsOpenReview ↗

Table 6: Sub-dimensional results of ME-Ruler contributions on the IRS-Bench.

Table 6: Sub-dimensional results of ME-Ruler contributions on the IRS-Bench. Each cell reports GPT-5.6-SOL / Qwen3.8-27B / Claude-Sonnet 5.

SystemCausal CorrectnessIdentity PreservationSpatial ReasonablenessVisual NaturalnessDomain SpecificityBackground FidelityOverall
Heuristic Baselines
Copy-paste9.9 / 13.7 / 8.56.0 / 6.7 / 5.76.0 / 9.2 / 5.51.9 / 3.4 / 2.44.5 / 6.1 / 3.88.1 / 7.9 / 7.036.4 / 47.0 / 32.8
Text-to-Image Image Generation
OmniGen211.2 / 15.9 / 8.64.5 / 5.3 / 4.46.3 / 8.5 / 5.72.5 / 4.3 / 2.34.7 / 6.5 / 3.63.3 / 6.8 / 5.432.4 / 47.3 / 30.0
AIComposer10.6 / 15.8 / 7.84.5 / 5.9 / 4.05.3 / 9.0 / 4.52.4 / 4.4 / 2.04.6 / 6.5 / 3.47.3 / 7.6 / 5.434.6 / 49.3 / 27.1
UMO11.2 / 16.0 / 8.54.4 / 5.3 / 4.46.4 / 8.6 / 5.52.5 / 4.4 / 2.44.7 / 6.5 / 3.63.3 / 6.8 / 5.432.5 / 47.6 / 29.7
LAMIC7.3 / 10.5 / 5.42.8 / 3.2 / 2.93.3 / 4.9 / 2.91.9 / 3.1 / 1.93.2 / 4.5 / 2.43.4 / 5.6 / 5.322.0 / 31.9 / 20.8
DreamFuse15.4 / 20.0 / 12.95.7 / 6.7 / 5.79.0 / 11.4 / 8.34.3 / 6.0 / 4.06.4 / 8.2 / 5.26.2 / 7.8 / 6.647.0 / 60.1 / 42.7
SHINE12.0 / 15.8 / 9.03.8 / 4.9 / 3.46.7 / 9.2 / 5.83.1 / 4.5 / 2.54.9 / 6.4 / 3.75.9 / 6.8 / 5.636.5 / 47.6 / 29.9
CareCom11.7 / 16.7 / 9.34.7 / 6.4 / 4.47.2 / 10.4 / 6.02.5 / 4.5 / 2.34.9 / 7.0 / 3.95.5 / 7.9 / 5.836.4 / 52.9 / 31.7
Open-source Image Generation and Editing Model
Skywork UniPic 3.015.7 / 20.0 / 12.56.2 / 6.9 / 6.39.2 / 11.5 / 8.74.0 / 5.6 / 3.76.4 / 8.2 / 5.26.0 / 7.2 / 6.447.6 / 59.4 / 42.8
Qwen-Image-3.021.3 / 25.1 / 16.77.1 / 7.5 / 7.111.3 / 13.0 / 11.05.7 / 6.8 / 5.27.8 / 9.4 / 6.66.6 / 7.9 / 7.159.8 / 69.7 / 53.7
Qwen-Image-Edit-251117.6 / 22.1 / 14.56.4 / 7.1 / 6.610.2 / 12.2 / 9.74.7 / 6.3 / 4.57.0 / 9.0 / 5.95.7 / 7.0 / 6.351.7 / 63.7 / 47.6
Wan2.7-Image17.6 / 22.4 / 14.76.9 / 7.4 / 6.910.4 / 12.5 / 9.94.4 / 6.3 / 4.57.1 / 9.1 / 6.06.9 / 7.5 / 6.853.4 / 65.1 / 49.0
FireRed-Image-Edit18.4 / 23.1 / 14.66.7 / 7.3 / 6.810.4 / 12.4 / 9.84.7 / 6.2 / 4.37.2 / 9.1 / 5.76.5 / 7.6 / 6.653.7 / 65.7 / 47.9
HunyuanImage 3.0-Instruct19.5 / 24.4 / 15.37.0 / 7.5 / 7.110.7 / 12.8 / 10.45.2 / 6.6 / 4.87.4 / 9.5 / 6.26.5 / 7.8 / 6.756.3 / 68.7 / 50.5
FLUX.2-klein-9B18.3 / 23.2 / 14.46.8 / 7.3 / 6.810.2 / 12.4 / 9.84.7 / 6.4 / 4.47.1 / 9.1 / 6.06.9 / 7.7 / 6.854.1 / 66.2 / 48.2
Close-source Image Generation and Editing Model
ChatGPT Images 2.020.1 / 25.1 / 16.37.2 / 7.6 / 7.311.7 / 13.2 / 11.55.2 / 6.7 / 5.17.6 / 9.6 / 6.74.6 / 7.8 / 7.056.4 / 70.1 / 53.7
Nano Banana 222.5 / 26.6 / 17.47.1 / 7.5 / 7.111.1 / 13.1 / 10.65.0 / 6.6 / 4.88.2 / 9.9 / 6.77.3 / 7.8 / 7.061.1 / 71.6 / 53.5
Seedream 5.0 lite18.3 / 22.8 / 14.77.0 / 7.4 / 7.110.5 / 12.6 / 10.34.7 / 6.3 / 4.77.0 / 8.9 / 6.06.8 / 7.8 / 6.954.2 / 65.7 / 49.7
Proposed Method (ReactionGraph)
ReactionGraph (w/ Image 2.0)33.3 / 37.9 / 26.37.3 / 7.7 / 7.313.1 / 14.5 / 12.56.5 / 7.9 / 5.610.3 / 12.5 / 8.84.1 / 7.7 / 6.874.5 / 88.1 / 67.3
ReactionGraph (w/ Nano Banana 2)34.0 / 38.8 / 27.77.3 / 7.7 / 7.212.7 / 14.3 / 11.96.3 / 7.9 / 5.510.5 / 12.7 / 9.16.4 / 7.5 / 6.777.2 / 88.8 / 68.1
ReactionGraph (w/ Qwen-Image-3.0)30.2 / 35.3 / 24.56.9 / 7.5 / 7.012.0 / 13.9 / 11.46.3 / 7.6 / 5.59.7 / 11.8 / 8.34.0 / 7.5 / 6.369.3 / 83.7 / 63.1
AuthorsOpenReview ↗

Table 7: Sub-dimensional results of ME-Ruler contributions on the deterministic subset (DET) of IRS-Bench.

Table 7: Sub-dimensional results of ME-Ruler contributions on the deterministic subset (DET) of IRS-Bench. Each cell reports GPT-5.6-SOL / Qwen3.8-27B / Claude-Sonnet 5; the five dimension contributions plus Background Fidelity sum to Overall up to rounding.

SystemCausal CorrectnessIdentity PreservationSpatial ReasonablenessVisual NaturalnessDomain SpecificityBackground FidelityOverall
Heuristic Baselines
Copy-paste9.9 / 14.0 / 8.66.2 / 7.0 / 5.96.1 / 9.4 / 5.51.8 / 3.4 / 2.14.8 / 6.7 / 4.08.2 / 8.0 / 7.137.0 / 48.5 / 33.1
Text-to-Image Image Generation
OmniGen211.4 / 16.5 / 9.14.8 / 5.6 / 4.76.3 / 8.4 / 5.72.4 / 4.2 / 2.35.2 / 7.4 / 4.03.4 / 7.0 / 5.633.5 / 49.1 / 31.4
AIComposer10.7 / 16.3 / 8.14.8 / 6.2 / 4.35.2 / 9.2 / 4.52.3 / 4.3 / 1.75.0 / 7.4 / 3.87.5 / 7.8 / 5.535.5 / 51.3 / 27.9
UMO11.5 / 16.6 / 8.94.7 / 5.6 / 4.76.3 / 8.5 / 5.52.4 / 4.2 / 2.35.2 / 7.4 / 4.03.5 / 7.0 / 5.633.5 / 49.3 / 30.9
LAMIC7.1 / 10.4 / 5.32.6 / 3.2 / 2.72.9 / 4.5 / 2.51.7 / 2.8 / 1.53.5 / 5.1 / 2.63.4 / 5.6 / 5.521.2 / 31.6 / 20.2
DreamFuse15.7 / 20.9 / 13.55.9 / 7.0 / 6.09.0 / 11.4 / 8.44.2 / 5.8 / 3.77.2 / 9.5 / 5.96.4 / 7.9 / 6.748.4 / 62.5 / 44.2
SHINE12.0 / 16.0 / 9.03.9 / 5.1 / 3.46.7 / 9.0 / 5.73.0 / 4.4 / 2.35.4 / 7.2 / 4.16.3 / 7.0 / 5.737.2 / 48.7 / 30.3
CareCom11.7 / 17.1 / 9.54.8 / 6.6 / 4.67.1 / 10.4 / 6.02.3 / 4.4 / 2.15.3 / 7.8 / 4.35.5 / 8.0 / 6.136.8 / 54.2 / 32.5
Open-source Image Generation and Editing Model
Skywork UniPic 3.017.0 / 21.7 / 13.86.8 / 7.5 / 6.99.8 / 12.0 / 9.24.1 / 5.7 / 3.77.4 / 9.6 / 6.16.2 / 7.4 / 6.651.3 / 63.9 / 46.2
Qwen-Image-3.022.5 / 26.4 / 17.67.4 / 7.8 / 7.511.3 / 13.0 / 10.85.6 / 6.6 / 4.89.0 / 10.9 / 7.56.5 / 7.9 / 7.162.2 / 72.7 / 55.4
Qwen-Image-Edit-251118.6 / 23.5 / 15.46.8 / 7.5 / 7.010.3 / 12.3 / 9.84.6 / 6.1 / 4.28.0 / 10.5 / 6.86.0 / 7.3 / 6.654.3 / 67.2 / 49.8
Wan2.7-Image18.7 / 23.8 / 15.67.3 / 7.7 / 7.310.6 / 12.5 / 10.04.4 / 6.2 / 4.38.2 / 10.6 / 6.97.0 / 7.6 / 7.056.2 / 68.5 / 51.0
FireRed-Image-Edit19.5 / 24.9 / 15.57.1 / 7.7 / 7.210.7 / 12.7 / 10.14.6 / 6.2 / 4.18.2 / 10.8 / 6.56.5 / 7.7 / 6.856.7 / 69.9 / 50.1
HunyuanImage 3.0-Instruct20.3 / 25.9 / 16.17.4 / 7.8 / 7.410.7 / 12.8 / 10.25.0 / 6.3 / 4.48.4 / 11.0 / 6.96.7 / 7.9 / 6.858.5 / 71.8 / 51.9
FLUX.2-klein-9B19.2 / 24.5 / 15.17.1 / 7.7 / 7.210.3 / 12.4 / 9.94.7 / 6.3 / 4.08.2 / 10.7 / 6.86.9 / 7.8 / 6.956.5 / 69.3 / 49.9
Close-source Image Generation and Editing Model
ChatGPT Images 2.021.0 / 26.3 / 17.07.5 / 7.9 / 7.611.6 / 13.2 / 11.45.1 / 6.5 / 4.78.6 / 11.1 / 7.64.8 / 7.9 / 7.158.6 / 72.9 / 55.3
Nano Banana 223.9 / 29.0 / 18.77.4 / 7.9 / 7.411.3 / 13.3 / 10.75.1 / 6.6 / 4.59.4 / 11.6 / 7.67.3 / 7.9 / 7.064.4 / 76.3 / 55.9
Seedream 5.0 lite19.5 / 24.2 / 15.77.3 / 7.8 / 7.510.7 / 12.7 / 10.44.6 / 6.1 / 4.48.0 / 10.3 / 6.87.0 / 8.0 / 7.157.0 / 69.1 / 51.9
Proposed Method (ReactionGraph)
ReactionGraph (w/ Image 2.0)33.0 / 37.8 / 26.07.6 / 8.1 / 7.613.1 / 14.3 / 12.36.4 / 7.6 / 5.211.1 / 13.9 / 9.54.3 / 7.9 / 6.975.4 / 89.5 / 67.4
ReactionGraph (w/ Nano Banana 2)34.2 / 39.0 / 28.07.6 / 8.0 / 7.612.7 / 14.3 / 11.96.2 / 7.6 / 5.311.6 / 14.2 / 9.86.7 / 7.8 / 6.879.0 / 91.0 / 69.4
ReactionGraph (w/ Qwen-Image-3.0)30.7 / 35.9 / 25.07.3 / 7.9 / 7.412.0 / 13.7 / 11.26.1 / 7.3 / 5.110.6 / 13.3 / 9.04.2 / 7.8 / 6.471.0 / 85.8 / 64.1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.