acceptodds
Withdrawn from ICLR 2027

MIBE: Multi-subject Interaction Benchmark and Evaluator for Personalized Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-subject personalized image generation requires the precise rendering of all requested reference identities and their specified interactions based on a guiding prompt. However, state-of-the-art models still struggle with this process, frequently omitting subjects, failing to preserve reference appearances, or misattributing interactions. Furthermore, existing metrics designed primarily for single-subject fidelity cannot reliably capture these errors, suffering severe degradation in ranking separability and failing to align with human preference as the subject count increases. To address this gap, we introduce Multi-subject Interaction Benchmark and Evaluator (MIBE), a unified framework comprising a Multi-subject Interaction Benchmark (MIB) and a Multi-subject Interaction Evaluator (MIE). MIB systematically covers diverse relation types and scene complexities through a decoupled data regime. This consists of a 60K-pair VLM-labeled Silver Set for scalable metric training and a 4K-pair double-blind Human Evaluation Gold Set covering a diverse range of state-of-the-art generators, with the Silver Set reaching 95.1% cross-VLM preference agreement. To demonstrate the utility of this benchmark, we present MIE, a lightweight, reference-conditioned evaluator trained exclusively on the Silver Set with a dual-head ranking and diagnosis objective. MIE exhibits strong cross-generator generalization on the Gold Set, achieving 0.922 overall pairwise accuracy against human preference, including 0.982 on seen generators and 0.884 on unseen generators. By outperforming a broad spectrum of baseline metrics, including CLIP and DINO variants, MIE demonstrates that diagnostic supervision can preserve ranking separability and human alignment where traditional evaluators collapse. Ultimately, by providing data with the diagnostic and preference labels required to train robust evaluators, MIBE serves as a comprehensive framework to benchmark multi-subject image generation and guide future model alignment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Subject Personalized Generation, Multimodal Evaluators & Metrics, Human Preference Alignment, Representation Learning for Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f2poOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces the "MIB Sliver and Gold Set" benchmark, along with an evaluator (MIE) designed to assess multi-subject image generation.

Critical strengths
  1. The paper proposes a benchmark for evaluating customized multi-subject image generation.
Critical weaknesses
  1. The overall academic readability is poor; references are incomplete, and the flowcharts are ambiguous. This work bears the hallmarks of being generated by AI.

  2. The data source is too limited, covering only two models. It is recommended to incorporate more data for both evaluation and the training of the evaluator; otherwise, the system risks learning to favor specific models rather than the quality of the generated images themselves.

Questions for authors
  1. The paper appears heavily AI-generated, and the motivation is weak—it essentially serves only to provide an evaluation framework.

  2. There is a complete lack of supplementary material or visualizations of images produced by the generative models, making it impossible to assess the effectiveness of the proposed evaluator.

  3. The data sources are too sparse, originating only from "Nano Banana" and "MOSAIC"—furthermore, there is no citation provided for "MOSAIC."

1 comment

Response to Reviewer f2po

Thank you for the review. A revised PDF has been uploaded. We respond to each question below.

Q1: Writing and motivation

We revised the full manuscript for readability. The Introduction has been rewritten to state the motivation and significance more directly. In short, current metrics cannot reliably judge multi-subject generation: on MIB-Gold, PickScore and HPS v2.1 agree with human preference only 48.6% and 52.0% of the time, close to chance (Sec. 4.1). In contrast, MIE, trained on MIB-Silver, agrees with human preference on 92.2% of pairs, including 88.4% on generators unseen in training (Sec. 4.2). Using MIE, we find that even the strongest current generator (GPT-Image-1.5) satisfies all three dimensions in only 52.4% of images (Table 3). We also redrew the overview diagram (Fig. 2, p. 3). It now shows the candidate pair explicitly and omits non-essential numbers, so the main pipeline is easier to follow.

Our use of AI tools, including editing for readability, is disclosed in the AI Use Statement (p. 11) in accordance with ICLR policy. The research, experiments, and claims are the authors' own.

Q2: Visualizations

The submission includes visualizations of generated images. Because of the page limit, and since our contribution is a benchmark and an evaluator rather than a new generation method, they appear in the appendix:

  • Appendix A (Figs. 5–7, pp. 16–18): generations from different models, showing subject omission, appearance leakage, figure splitting, and action dropout.
  • Appendix E (Fig. 8, p. 23): Silver training labels, with the flaw logs written by the two VLM judges.

In the revision, the Introduction and Sec. 4 cite these figures where each failure mode is discussed (Figs. 5–7), so they are easier to find.

Q3: Data sources and MOSAIC citation

MOSAIC is now cited wherever it is used, and we rechecked the rest of the bibliography.

Only training uses two generators; evaluation uses six.

  • MIB-Silver (training): Nano Banana and MOSAIC, chosen because together their failures cover all three diagnostic dimensions.
  • MIB-Gold (evaluation): 4,020 human-labeled pairs from six generators. Four of them (GPT-Image-1.5, Flux2, Seedream 4.5, GLM) never appear in training, and 2,520 pairs compare only these unseen generators.

If MIE had only learned to favor its training generators, it would have nothing to rely on in these unseen pairs. The results show that it still generalizes:

  • Pairwise agreement with human preference (Sec. 4.2): 98.2% on seen generators and 88.4% on unseen generators, without any fine-tuning on the unseen ones.
  • On 22 new reference subjects outside the benchmark, including real photographs, the same checkpoint reaches ROC-AUC 0.827 (real) and 0.851 (synthetic) (Appendix H, Table 13).

We agree that more training generators would make MIE more robust, and these results do not fully rule out generator-specific shortcuts. The revised Limitations paragraph (Sec. 4) now states that MIB-Silver relies on two training generators and dual-VLM supervision, which may retain residual generator priors.

Reviewer_mcanOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MIBE, a benchmark and evaluator for multi-subject personalized image generation. The paper focuses on the binding problem: generated images may omit subjects, fail to preserve their appearance, or assign interactions to the wrong subjects. MIB evaluates these failures along three dimensions: Existence, Appearance, and Interaction. It contains a 60K-pair VLM-labeled Silver Set and a 4,020-pair human-annotated Gold Set, with controlled variation in subject count, composition, and interaction type. MIE is trained on the Silver Set using joint ranking and diagnostic objectives. The strongest MIE model achieves 0.922 pairwise agreement with human preference, including 0.982 on seen and 0.884 on unseen generators, substantially outperforming the reported conventional metrics.

Critical strengths
  1. The paper addresses an important evaluation gap in multi-subject personalized generation.
  2. The controlled subject hierarchical construction provides a useful way to isolate failures as compositional complexity increases. 
  3. The separation of scalable VLM-labeled training data from human Gold evaluation is well motivated.
  4. Results show strong human alignment, including reasonable transfer to unseen generators.
  5. The Existence/Appearance/Interaction decomposition provides useful diagnostics beyond a single aggregate metric.
Critical weaknesses
  1. MIE is trained entirely from VLM-generated supervision. VLM-to-VLM agreement does not establish human agreement. Per-dimension Silver-vs-human analysis would strengthen the paper.
  2. A ranking-only vs. ranking+diagnostic comparison is important to establish whether diagnostic supervision actually improves ranking.
  3. The gap between seen (0.982) and unseen (0.884) generators remains noticeable. Broader evaluation on novel generators and reference distributions would strengthen the generalization claim. 
  4. The fixed 80-subject pool and controlled white-studio setting limit real-world diversity.
  5. One reference image cannot fully distinguish identity preservation from reference replication, a limitation acknowledged by the authors.
Questions for authors
  1. How strong is Silver-to-Gold agreement per diagnostic category? In particular, does dual-VLM supervision agree equally well with humans on Existence, Appearance, and Interaction, or is the strong 95.1% VLM-to-VLM preference agreement masking systematic category-specific biases?
  2. How much of MIE's performance comes from diagnostic supervision? An ablation of ranking-only versus diagnostic-only versus joint training would more directly establish the claim that diagnostic supervision improves ranking separability and human alignment.
  3. How sensitive are the results to the particular VLM judges used for Silver annotation? Training MIE from alternative labeling models or different judge combinations could help establish that MIE learns the benchmark's underlying binding concept rather than the preferences of the two selected judges.
  4. How does MIE behave on new generation architectures with qualitatively different artifacts? The 0.884 unseen-generator result is promising, but the remaining seen/unseen gap suggests generator-specific effects remain.
  5. Could the authors provide more direct human validation of the generator-level conclusions? In particular, comparing MIE-derived All Satisfied rates with human-derived rates on a sufficiently large subset would validate the intended use of MIE as a generator benchmarking tool.
3 comments

Comment 1 of 3 — Summary; Silver labels against humans (Q1)

Thank you for the constructive review. We answer each question with numbers computed from the Silver and Gold annotation files. In brief:

  • Q1. MIE is validated against humans directly, per dimension (F1 0.93 / 0.78 / 0.74), and on the Gold pairs where both VLM judges contradict humans it follows the humans in 81 of 186 cases.
  • Q4. The seen/unseen drop is concentrated in one matchup. Within every matchup MIE's score difference predicts the human winner (AUC 0.77–0.89), and on prompts from seed groups held out of training its accuracy is 0.895, against 0.884 for training-split prompts.
  • Q5. Human-derived All Satisfied rates rank the six generators exactly as MIE does (Spearman ρ = 1.0); per dimension, generator-level means correlate at r = 0.95–0.96.
  • Q2, Q3. We have no ranking-only ablation and no alternative-judge training. We will soften the causal wording and list both as future work.

Q1. Does Silver agree with humans per category, or does 95.1% hide biases?

The 95.1% measures agreement between the two judges. Agreement with humans is measured directly on Gold:

  • MIE vs human labels, per dimension, on Gold. On the 3,690 preference-consistent pairs, using images where both annotators agree on the dimension and the default 0.5 threshold, F1 is 0.927 (Existence), 0.784 (Appearance) and 0.744 (Interaction). The default gives the highest macro-F1 (0.818) of five tested thresholds from 0.3 to 0.7 (range 0.791–0.818). Over all 8,040 Gold images, the Pearson correlation between MIE's probability and the mean human label is 0.82 / 0.69 / 0.58. For reference, human–human agreement on the same images is Fleiss' κ = 0.85 / 0.56 / 0.47, so MIE's per-dimension profile follows the reliability of the human labels: strongest on Existence, weakest on Interaction.
  • Where the judges are wrong. On 186 Gold pairs the two judges agree with each other but contradict the human consensus (154 of them GPT-Image-1.5 vs Flux.2). MIE, trained only on the judges' labels, sides with the humans on 81 of them (43.5%). A model that simply copied the judges would side with them on all 186.
  • Judge–judge agreement by category (59,852 Silver tasks; both judges give identical labels to both images): Existence 89.1%, Appearance 69.0%, Interaction 71.9%. The 95.1% preference agreement is inflated by skewed winners: each judge prefers the same generator in 97.6% and 94.4% of tasks, and chance-corrected agreement is Cohen's κ = 0.37.

The revision will report κ = 0.37 and the per-category figures beside the 95.1%, base the evaluator-validity argument on the MIE–human results above, and state that teacher–human diagnostic agreement is unmeasured. It also corrects one description: of the 57,055 Silver records in our splits, 146 carry a label from one judge only (129 in the train split), so the filter is not strictly dual-judge.

Comment 2 of 3 — Ablation, judge sensitivity, unseen generators (Q2–Q4)

Q2. Does diagnostic supervision improve ranking? MIE is trained with one joint loss (α = 1.0, β = 0.5), used for all six checkpoints. With it, a single model both ranks well (0.922 on Gold) and gives per-dimension scores that match human labels (macro-F1 0.818); the per-dimension outputs are what we validate against humans in Q5. We agree that a ranking-only (β = 0) comparison is the right way to isolate how much the diagnostic loss adds to ranking. In the revision we will soften the causal wording ("forced to ground itself", "diagnostic supervision can preserve ranking separability", "proves", "guarantees"), describe the joint loss as a design choice, and list this ablation as future work.

Q3. Sensitivity to the choice of judges. We have not retrained MIE from one judge or from other judges, so this is unmeasured and will be listed as a limitation. The one related measurement is the 186-pair result above: MIE does not inherit the judges' errors one-to-one (it follows humans on 43.5% of them).

Q4. Seen vs unseen gap and new generators. The unseen average of 0.884 combines two very different matchups (95% CIs from a bootstrap over prompt-seed groups):

Gold matchup (kept pairs)nMIE acc. (95% CI)Human winner shareAUC of MIE score differenceMinority wins MIE gets right
Nano Banana vs MOSAIC (seen)1,4110.982 (0.974–0.989)Nano Banana 98.8%0.8644 / 17
GPT-Image-1.5 vs Flux.2 (unseen)1,0710.779 (0.752–0.803)GPT-Image-1.5 90.5%0.76654 / 102
Seedream 4.5 vs GLM (unseen)1,2080.978 (0.969–0.986)Seedream 98.3%0.8924 / 20
  1. The drop is concentrated in one matchup. Seedream vs GLM, an unseen pair, is close to the seen pair (0.978 vs 0.982). The hard case is GPT-Image-1.5 vs Flux.2, where human preferences are least one-sided and annotators themselves disagree most (85.0% of its pairs pass the preference-consistency filter, against 94–96% for the other two).
  2. Within a matchup, MIE's score differences predict human preference beyond a constant generator choice. One generator wins most pairs of each matchup, so a rule that always picks it, computed post hoc from the Gold labels, scores 98.8%, 90.5% and 98.3%, above MIE's accuracy. Such a rule gives every pair the same answer (AUC 0.5) and recovers none of the 139 pairs where the weaker generator wins. MIE has AUC 0.77–0.89 inside each matchup and recovers 62 of those 139 pairs, 54 of 102 in the hard matchup. The revision will report accuracy, AUC and minority-win recall per matchup, note that two minority groups are small (17 and 20 pairs), and state this limitation of fixed-matchup Gold pairs.
  3. Prompts. The unseen split reuses Silver prompts with new generators. On the 286 pairs whose prompt comes from a seed group held out of training, accuracy is 0.895 (95% CI 0.861–0.928); over all 396 pairs whose prompt lies outside the training split it is 0.884, and for the 1,883 pairs whose prompt is inside it, 0.884. Identities are shared; new identities are tested in Appendix H (22 new subjects, AUC 0.827 with real-photo references, 0.851 with synthetic ones).

Fig. 1 needs a correction. Its baseline bars were scored on individual annotator judgments (4,991 on the unseen split), but MIE's bar (88.4%) was scored on the 2,279 consensus pairs. We rescored MIE on the same 4,991 judgments, counting ties as 0.5 for every metric: MIE gets 84.7%, and the best of the twelve similarity and preference metrics (DINO) gets 63.6%, a 21.0-point gap. The three face-based MultiHuman-Testbench bars (best 82.9%) only apply to prompts with human subjects and were scored differently; the revised figure will show them separately from the twelve metrics and state their protocol.

We have tested four unseen generators; we have not tested generators chosen for qualitatively new artifact types, and the paper will say so.

Comment 3 of 3 — Human validation of generator-level conclusions (Q5); scope

Q5. We ran this on the full Gold set rather than a subset. Two annotators label Existence, Appearance and Interaction for every Gold image, so we computed All Satisfied from human labels on the same 7,806 valid images as Table 3 (all images except those of prompt tasks that an annotator flagged as illogical; no preference filter).

Generator (n images)MIE, all 3 probabilities > 0.5Human: both annotators, all 3 dimsHuman: ≥1 annotator per dim
GPT-Image-1.5 (1,226)52.4%28.1%61.2%
Seedream 4.5 (1,226)47.2%23.2%49.2%
Nano Banana (1,451)42.0%19.0%34.9%
Flux.2-klein-9B (1,226)22.3%4.6%18.4%
GLM (1,226)1.6%0.8%4.6%
MOSAIC (1,451)0.8%0.6%2.6%
  • Ranking. MIE orders the six generators exactly as humans do under both human rules (Spearman ρ = 1.0). The order is unchanged at MIE thresholds 0.3, 0.4, 0.5 and 0.6; at 0.7 Nano Banana and Seedream swap.
  • Per dimension. Across the six generators, MIE's mean probability tracks the mean human label with Pearson r = 0.95 (Existence), 0.96 (Appearance) and 0.96 (Interaction); rank correlations are 0.94, 0.94 and 0.89.
  • Level. These comparisons do not validate MIE's rates as absolute success rates. They lie between the strict and lenient human rates for four generators and above the lenient rate for Nano Banana and Flux.2. The two human columns are aggregation rules, not uncertainty bounds.

For comparing generators, MIE therefore gives the same ranking as human labels on these six generators, and its per-dimension averages are strongly correlated with the human ones. The comparison does not establish which dimension is weakest for a given generator, nor absolute success rates. The revision bases the interpretation of Table 3 on the human-derived results: it makes the human-derived rates the primary scorecard of Table 3 and presents MIE's rate as an automatic estimate that preserves ranking but not level.

Scope (80 subjects, white studio, single reference). We agree and keep these in Limitations. Appendix H covers real-photo references for 22 new subjects. Several references per identity would help separate identity preservation from copying; we leave this to a multi-reference extension.

Revisions. (1) κ = 0.37 and per-category judge agreement beside the 95.1%, with the single-judge correction; (2) causal wording about diagnostic supervision softened, ablation and judge sensitivity listed as future work; (3) per-matchup accuracy, AUC and minority-win recall, with the fixed-matchup limitation; (4) Fig. 1 with protocols and denominators stated; (5) human-derived All Satisfied as the primary scorecard; (6) the split described as "unseen generators, shared prompt pool and identities."

We will add these analyses and corrections to the final version of the paper. We hope they support your positive view of the paper, and we are happy to answer any follow-up questions.

Reviewer_6akpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MIBE, a benchmark-and-evaluator framework for multi-subject personalized image generation. MIB contains a 60K-pair Silver Set labeled by two VLM judges and a 4,020-pair human-labeled Gold Set. Its prompts are organized by subject count, human/object composition, and relation type, while candidate quality is assessed along Existence, Appearance, and Interaction. The paper also proposes MIE, a reference-conditioned evaluator with ranking and diagnostic heads, trained on Silver and evaluated on Gold. MIE achieves 0.922 overall pairwise accuracy against human preference, including 0.884 on the unseen-generator split. The authors use the benchmark to analyze how omission, identity leakage, and interaction failures change with subject count, and compare MIE with existing metrics.

Critical strengths
  1. A useful, factorized benchmark for a real evaluation gap. MIB controls subject count, human/object composition, and relation type, while providing separate labels for existence, appearance, and interaction failures. This could help researchers identify why a multi-subject generation fails, rather than relying on one aggregate score.
  2. A scalable training set paired with human evaluation. MIE is trained on VLM-labeled Silver data and evaluated against human preferences on Gold, with 0.884 pairwise accuracy on the unseen-generator split. If the data split and annotation details support the claimed independence, this is evidence that the benchmark may enable evaluators that transfer beyond their training generators.
Critical weaknesses
  1. The Silver–Gold separation is underspecified. The paper describes hierarchical prompt seeds and a fixed pool of 80 reference identities, but does not clearly state whether prompt seeds, prompts, or reference identities overlap between Silver and Gold. If they do, the unseen-generator result may still benefit from familiarity with task templates or subjects. The absence of confidence intervals or repeated training runs also makes it difficult to judge the stability of the reported accuracy.
  2. Gold-label reliability and evaluation counts need clearer reporting. The paper reports preference-consistency filtering but does not provide per-dimension agreement for the main Gold set or a clear post-filter denominator for each split. This matters particularly for diagnostic claims: the separate reference-extension study reports low appearance agreement (κ = 0.19–0.34), while the “All Satisfied” rate is derived from MIE’s own predictions using a 0.5 threshold. The diagnostic and generator-level conclusions would be more convincing with clearer human-label reliability and threshold validation.
Questions for authors
  1. Are the Silver and Gold sets disjoint at the level of prompt seeds, exact prompts, and reference identities/images? Please report the overlap protocol and, if applicable, results on Gold tasks with held-out seeds or identities.
  2. What are the exact Gold sample counts before and after preference-consistency filtering for the seen- and unseen-generator splits? Could you also report inter-annotator agreement for each diagnostic dimension and explain how diagnostic labels are thresholded for macro-F1 and “All Satisfied”? Suggestions for future work
  • Extend the benchmark to less controlled scenes and multiple reference images per identity, which would test whether the results transfer to more varied settings and distinguish identity preservation from copying a single reference.
3 comments

Comment 1 of 3 — Summary; Silver/Gold overlap; stability

Thank you for the careful review. Below are the analyses you asked for, computed from the Gold annotations and the Silver split files. The main results:

  • Overlap. Gold shares its identity pool with Silver, and the unseen split also shares its prompt pool; no generator or candidate image of the unseen split occurs in Silver. MIE's accuracy is 0.884 on prompts outside its training split (n=396) and 0.884 on prompts inside it (n=1,883); on prompts from seed groups held out of training it is 0.895 (n=286).
  • Stability. 95% CIs over 690 prompt-seed groups are about ±1 point: overall 0.922 (0.913–0.931).
  • Gold reliability. Per-dimension agreement on the main Gold set is κ = 0.85 / 0.56 / 0.47 (Existence / Appearance / Interaction). The κ = 0.19–0.34 you cite is Appearance agreement in the separate appendix study; on main Gold, Appearance is 0.56.
  • Thresholds and All Satisfied. Human labels give the same ranking of the six generators as MIE's All Satisfied rate (Spearman ρ = 1.0), and the default 0.5 threshold gives the highest macro-F1 of the five thresholds we tested.

Q1. Are Silver and Gold disjoint?

  • Reference identities: shared by design. All 80 identities of the unseen split, and 79 of the 80 in the seen split, also occur in Silver; MIB uses a closed pool with one reference image per identity. Held-out identities are tested in Appendix H: 22 new subjects, MIE not retrained, AUC against human preference 0.827 with licensed real-photo references and 0.851 with synthetic references.
  • Prompts. The unseen split re-renders 1,260 Silver prompts with four generators that never appear in Silver. The seen split is a separate prompt draw: none of its 1,488 distinct prompts matches a Silver prompt literally, and 45 match after normalizing case, whitespace and one fixed sentence.
  • What MIE trained on. Of the 60,000 Silver tasks, 2,945 were dropped before splitting (2,943 because the two judges chose different winners). The remaining 57,055 are split by seed group (train 51,328 / val 2,847 / test 2,880; no seed and no prompt in two splits). MIE was trained for 600 optimizer steps × batch 16 = 9,600 pairs from the train split only.

Held-out test on the unseen split (rows 1–3 partition its 2,279 kept pairs):

Unseen-split pairsnMIE pairwise acc. (95% CI)
1. Prompt in the training split1,8830.884 (0.870–0.898)
2. Prompt from a Silver val/test seed group (seed never in training)2860.895 (0.861–0.928)
3. Prompt whose Silver pair was dropped for judge disagreement (never in training)1100.855 (0.787–0.917)
Rows 2 + 3: every prompt outside the training split3960.884 (0.853–0.914)

Per matchup, row 2 vs row 1: GPT-Image-1.5 vs Flux.2, 0.80 (n=138) vs 0.78 (n=878); Seedream 4.5 vs GLM, 0.99 (n=148) vs 0.98 (n=1,005). Prompts from held-out seeds score 0.895, against 0.884 for training-split prompts. Two limits: row 1 is split membership, not exposure (only 9,600 of the 51,328 train pairs were used), and identities are shared in every row, so identity familiarity is tested only by Appendix H. The revision adds this overlap protocol and table and names the split "unseen generators, shared prompt pool and identities."

Confidence intervals and repeated runs. The bootstrap resamples whole prompt-seed groups (4,000 resamples; the Level-8/6/4/2 variants of a seed move together):

n pairsseed groupsMIE acc. (95% CI)
Overall3,6906900.922 (0.913–0.931)
Seen generators1,4113750.982 (0.974–0.989)
Unseen generators2,2793150.884 (0.871–0.897)

These intervals cover test-set sampling. Each checkpoint was trained once (seed 3407), so training-seed variance is not measured; the paper will say so.

Comment 2 of 3 — Gold counts, matched-protocol comparison, per-dimension agreement

Q2a. Gold counts before and after filtering. Six annotators labeled Gold; every pair was labeled independently by two of them, with the A/B order randomized. A pair is kept when both choose the same winner.

Split / matchupRawKeptRate
Seen: Nano Banana vs MOSAIC1,5001,41194.1%
Unseen: GPT-Image-1.5 vs Flux.21,2601,07185.0%
Unseen: Seedream 4.5 vs GLM1,2601,20895.9%
Total4,0203,69091.8%

MIE's pairwise accuracies in §4.2, Table 2 and Fig. 4 use these 3,690 pairs.

A denominator correction to Fig. 1. While answering this question we found that the baseline bars of Fig. 1 (left) score individual annotator judgments (4,991 on the unseen split after removing illogical-prompt flags), whereas MIE's bar (88.4%) uses the 2,279 consensus pairs. The sentence "4,020 valid pairwise comparisons" in §4.1 of the submitted version is therefore wrong. We rescored MIE under the baselines' protocol on the same judgments:

Per-judgment agreement with humans, all judgments, ties count 0.5Unseen split (4,991)Seen split (2,950)
MIE84.7%95.4%
Best of the twelve similarity and preference metrics63.6% (DINO)74.6% (MS-SSIM)

On identical judgments and with one tie policy, MIE is 21.0 points above the best of these twelve metrics on the unseen split and 20.8 points on the seen split, so the message of Fig. 1 about similarity and preference metrics holds. (SCR's 67.3% in Fig. 1 covers only the 2,251 judgments where it is not tied; over all 4,991 it is 57.8%.) The three face-based MultiHuman-Testbench bars in Fig. 1 (best 82.9%) are defined on prompts with human subjects and were scored under a further protocol; the revised figure will show them separately from the twelve metrics and state their protocol.

Q2b. Per-dimension inter-annotator agreement on the main Gold set (two labels per image, 8,040 images; Fleiss' κ, since the two annotators rotate within the pool of six):

ExistenceAppearanceInteraction
Raw agreement92.5%80.2%74.8%
κ, all0.850.560.47
κ, seen / unseen0.82 / 0.860.59 / 0.520.52 / 0.45

Existence is reliable; Appearance and Interaction are moderate. In the revised paper, each table and figure that reports per-dimension results will include these κ values in its caption, so that Appearance and Interaction results are read with their lower label reliability in mind. The κ = 0.19–0.34 you cite belongs to a different study: the Appendix H reference-extension experiment, with eight annotators and new subjects. On the main Gold set, Appearance agreement is κ = 0.56.

Comment 3 of 3 — Thresholds, All Satisfied against humans, revisions

Q2c. Thresholds for macro-F1. A human diagnostic label is used only where both annotators of the image agree; other images are excluded for that dimension (Existence 539, Appearance 1,392, Interaction 1,815 of the 7,380 images in kept pairs). MIE counts a dimension as satisfied when σ(logit) ≥ 0.5, the default. F1 of the 4B model: Existence 0.927, Appearance 0.784, Interaction 0.744 (macro 0.818). A post hoc sweep shows that the default gives the highest macro-F1 of the five tested values, which stay within 0.791–0.818:

Threshold0.30.40.50.60.7
Macro-F10.8040.8130.8180.8090.791

Q2d. Is All Satisfied valid when it comes from MIE's own predictions? Both annotators of a pair label Existence, Appearance and Interaction for each image, so we computed All Satisfied from human labels on the same 7,806 valid images as Table 3 (all images except those of prompt tasks that an annotator flagged as illogical; no preference filter).

Generator (n images)MIE, all 3 probabilities > 0.5Human: both annotators, all 3 dimsHuman: ≥1 annotator per dim
GPT-Image-1.5 (1,226)52.4%28.1%61.2%
Seedream 4.5 (1,226)47.2%23.2%49.2%
Nano Banana (1,451)42.0%19.0%34.9%
Flux.2-klein-9B (1,226)22.3%4.6%18.4%
GLM (1,226)1.6%0.8%4.6%
MOSAIC (1,451)0.8%0.6%2.6%
  • Ranking. MIE orders the six generators exactly as humans do under both human rules (Spearman ρ = 1.0). The order is the same at MIE thresholds 0.3, 0.4, 0.5 and 0.6; at 0.7, Nano Banana and Seedream swap.
  • Per dimension. Across the six generators, MIE's mean probability tracks the mean human label with Pearson r = 0.95 (Existence), 0.96 (Appearance) and 0.96 (Interaction).
  • Level. These comparisons do not validate MIE's rates as absolute success rates: they fall between the strict and lenient human rates for four generators and above the lenient rate for Nano Banana and Flux.2. The two human columns are aggregation rules, not uncertainty bounds.

MIE therefore reproduces the human ranking of generators, overall and closely per dimension. The comparison does not establish which dimension is weakest for a given generator, so the revision makes the human-derived rates the primary scorecard of Table 3, bases its interpretation on them, and presents MIE's rate as an automatic estimate that preserves ranking but not level.

Future work. We agree that less controlled scenes and several references per identity are the right next tests; multiple references would help distinguish identity preservation from copying a single reference. Both stay in Limitations.

Revisions. (1) Overlap protocol, held-out table and the split renamed; (2) seed-group CIs and the single-training-seed statement; (3) Gold counts and κ tables; (4) Fig. 1 with protocols and denominators stated; (5) F1 protocol and threshold sweep; (6) human-derived All Satisfied as the primary scorecard. One further correction to §3.1: 146 of the 57,055 Silver records carry a label from one judge only, so the description of the dual-judge filter in §3.1.3 will be corrected.

All of the above (overlap protocol, confidence intervals, Gold counts, agreement, and the threshold check against human labels) will go into the final version of the paper. We would be grateful if you could take these results into account in your assessment, and we are happy to answer any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.