acceptodds
Withdrawn from ICLR 2027

VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such as layouts, arrows, and pose cues. However, existing benchmarks do not evaluate the joint setting in which multiple references must be composed under multiple and heterogeneous visual-instruction images. To address this gap, we introduce VIF-Bench, a benchmark of 1,241 tasks designed to assess the edge of model capabilities in this joint setting by covering: (i) multi-reference generation (up to 7) under multiple heterogeneous visual instructions (up to 6), (ii) cases where reference images can potentially compete with visual instructions (e.g., a strongly posed subject vs. a target pose), and (iii) controlled comparison of visual instructions with text descriptions at different levels of specificity. Using these capabilities, we uncover three findings: (1) models face an adherence–artifact trade-off: once models reach stronger visual instruction adherence, stronger adherence tends to coincide with more instruction artifacts in generated images, (2) visual instruction adherence tends to be lower on tasks whose reference images carry a salient state of the controlled attribute (e.g., a neon-lit subject under a light-direction instruction), most consistently for light and wind, and (3) for models that can understand visual instructions, it is often better to provide visual constraints directly rather than describe them in text; when using text, a moderate level of detail works better than an exhaustive description. VIF-Bench will be released as an open benchmark to establish a basis for fair comparison in controllable multi-reference image generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
benchmark, image generation, multimodal

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Lt7DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is a benchmark work. It evaluate image generator in a setting where multiple references composed under multiple and heterogeneous visual-instruction images. It consists 1,241 tasks with 777 reference images. It proposes three findings: (1) models face an adherence-artifact trade-off, (2) visual instruction adherence tends to be lower on tasks whose reference images carry a salient state of the controlled attribute, (3) for models that can understand visual instructions, it is often better to provide visual constraints directly rather than describe them in text.

Critical strengths

The explicit design of conflict tasks is interesting while the comparison conditions are not entirely rigorous.

Critical weaknesses

The evaluation score based on GPT-6 and Gemini and manual validation is limited. The evaluation results may be unfair and instable especially when GPT-Image is included in the evaluated generators.

Questions for authors
  1. Comparing to existing benchmarks, the authors should further explain their advancements, and these advancements must be substantial enough for the paper to be accepted again at a top-tier conference. For instance, Multi-Ref [R1].

[R1] Chen, R., Chen, D., Wu, S., Wang, S., Lang, S., Sushko, P., ... & Krishna, R. (2025, October). Multiref: Controllable image generation with multiple visual references. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 13325-13331).

  1. The reference pool contains only 777 images, and 359 synthetic references are generated by only Nano Banana and GPT-Image families, limiting the diversity of source distributions and visual styles of the benchmark.

  2. The construction pipeline relies on GPT-5 to propose labels. The potential construction bias is not sufficiently quantified and the generalization to natural controls is unclear.

  3. Why is the comparison between visual and textual instructions only conducted on a 200-task subset, as it produce a main conclusion of this benchmark.

  4. Are all 1,241 tasks include a layout instruction? If so, does this setting introduce a strong distributional bias?

Reviewer_tXsQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces VIF-Bench, a benchmark for image generation conditioned on multiple reference images and multiple heterogeneous visual instructions. It contains 1,241 tasks and 777 reference images; a task may contain up to seven reference images and six visual-instruction images, including layout boxes, 3D orientation pyramids, pose references, and wind/light arrows. The paper labels four types of reference–visual-instruction “conflict”: orientation, light, wind, and pose.

The authors evaluate eight image-generation systems with six VLM-judge criteria: Text Instruction Following, Reference Consistency, Visual Instruction Adherence, Visual Instruction Cleanliness, Scene Coherence, and Visual Quality. They report an adherence–artifact trade-off, lower visual-instruction adherence when reference images contain salient related attributes, and a preference for visual constraints over text descriptions in some settings.

Critical strengths
  1. Relevant joint benchmark setting. The benchmark combines multiple reference subjects with heterogeneous visual controls in one task, a setting that is useful for diagnosing current multimodal image-generation systems.

  2. Useful diagnostic structure and release effort. The task schema, failure categories, full judge prompt, and open-source Qwen3-VL evaluation option provide potentially useful infrastructure for future comparisons, even though the aggregate evaluation is not sufficiently validated for the paper’s stronger claims.

Critical weaknesses
  1. The central conflict construct is insufficiently novel and insufficiently validated. The four proposed conflict types substantially overlap with visual controls already evaluated in VIBE and related benchmarks. VIBE includes pose control, reorientation, lighting control, flow simulation, and multi-task visual-instruction following; MultiRef and DreamOmni3 already study combinations of multiple visual references and heterogeneous visual conditions. Table 1 only records whether prior benchmarks explicitly use a conflict label; it does not establish that analogous source–instruction inconsistencies are absent from their underlying tasks, and the paper does not provide a matched comparison using the same models, inputs, and evaluators. Moreover, light, wind, and pose conflict are defined mainly by the co-occurrence of a salient reference attribute and a corresponding control, without requiring the source and target states to be mutually incompatible. The conflict/no-conflict results therefore do not distinguish semantic conflict resolution from greater task complexity, attribute salience, or physical attribute-transfer difficulty. This is consequential because conflict is presented as the central conceptual contribution of the benchmark.

  2. The VLM evaluation and six-criterion aggregation do not justify the strength of the quantitative conclusions. The fixed prompt specifies several hard caps but does not provide a calibrated rubric for ordinary scores, so the meaning of intermediate scores remains judge-dependent. The paper does not report repeated evaluations of the same fixed image, test–retest variance, ICC, judge stochasticity, or the relevant sampling and retry procedures. Further, the six criteria are heterogeneous: they measure textual constraint satisfaction, reference fidelity, visual-control adherence, residual-mark removal, scene coherence, and perceptual quality. The paper does not establish that these criteria are commensurate, equally difficult, equally important, or sufficiently independent. Although the six scores are produced after a shared reasoning segment and in a fixed order, possible anchoring, order effects, reuse of an overall impression, and repeated penalties for the same failure mode are not examined. Reference Consistency, Visual Instruction Adherence, Visual Instruction Cleanliness, Scene Coherence, and Visual Quality may therefore be correlated or partially redundant. The unweighted arithmetic average can consequently conflate model ability with criterion difficulty, task composition, correlated scoring, and rubric-specific hard caps. Appendix G provides useful judge–human and cross-judge correlations, but it does not resolve these reliability, independence, and aggregation issues; several per-criterion correlations are modest, and no p-values, confidence intervals, standard errors, or cluster-aware uncertainty estimates are reported. These limitations directly affect the claimed model rankings, adherence–cleanliness trade-off, and conflict effects.

Questions for authors
  1. Conflict construct and prior-work comparison: Under matched models, input formats, and judges, how do the proposed conflict subsets differ from the pose, reorientation, lighting, flow, and multi-task settings already evaluated by VIBE and related benchmarks? For light, wind, and pose, why is a salient source attribute plus a corresponding control sufficient to define a conflict? Can you separately report explicitly compatible, explicitly incompatible, and salient-but-compatible source/target cases?

  2. Confounding by task difficulty: After controlling for reference count, visual-instruction count, subject category, image quality, scene complexity, and attribute salience, does the conflict/no-conflict adherence gap remain? If so, what evidence shows that it reflects reference–instruction priority resolution rather than general physical attribute-transfer difficulty or increased task complexity?

  3. Judge reliability and scoring procedure: Were fixed generated images repeatedly evaluated by the same judge? Please report the number of repeats, test–retest variance, confidence intervals, and ICC, or explain what evidence shows that model differences exceed judge stochasticity. Please also specify the model endpoints and versions, temperature/top-p/seed settings, retry and score-parsing procedures, and whether all six scores are generated in one response. Since the judge first produces free-form reasoning and then outputs criteria in a fixed order, have you tested reasoning anchoring or criterion-order effects?

  4. Criterion independence and aggregate validity: What evidence supports treating the six criteria as commensurate and equally weighted? Have you measured inter-criterion dependence or repeated penalties for shared failure modes, for example through criterion correlations, partial correlations, or factor analysis? Are model rankings and the main conclusions stable under criterion normalization, difficulty normalization, human-derived weights, or alternative aggregation rules? Please also report uncertainty for the model-level comparisons and provide the human-evaluation protocol and annotator agreement statistics for the 168-image subset.

Reviewer_Bdd8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VIF-Bench, a benchmark comprising 1,241 tasks designed to evaluate the instruction-following capabilities of image generation models when conditioned on multiple reference images and visual instructions. The authors constructed tasks containing potential conflicts between reference images and visual instructions, and employed a six-dimensional evaluation framework to assess performance. The experiments reveal a trade-off between visual instruction following and the retention of elements from the reference images.

Critical strengths
  1. This paper focuses on generation scenarios that contain both multiple reference images and diverse visual instructions.
  2. The paper compares the effectiveness of visual instructions against text-based instructions of varying detail levels, and offers valuable experimental insights.
  3. The evaluation encompasses a range of open-source and closed-source models.
Critical weaknesses
  1. The task setup in this benchmark is somewhat limited, because it lacks the free-form instructions typical of real-world user scenarios.
  2. The definition of Reference–Instruction Conflict lacks rigor. Observed performance degradation might come from the complexity of the reference images rather than the conflict itself, therefore, a distinction that remains unaddressed due to the absence of controlled experiments.
  3. The evaluation methodology lacks significant innovation. Its six dimensions largely mirror existing evaluation standards, relying on different VLMs but same scoring protocol. The paper does not demonstrate that this evaluation design offers any distinct, novel value over established approaches.
  4. The scores of newer generation models such as Flux.2 Klein or Nano Banana 2 need to be reported to reflect the capabilities of the latest SOTA models. It is recommended that the core conclusions can be validated using more recent generative models and VLM-based judges.
Questions for authors

Refer to Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.