acceptodds
Under review as a conference paper at ICLR 2027

HiP-Bench: A Benchmark for Hidden-Picture Optical Illusion Generation

14 views

Abstract

Visual generative models have demonstrated impressive capabilities in synthesizing high-quality images from multimodal inputs. However, their ability to generate hidden-picture optical illusions, where hidden targets are integrated into background images and emerge only under specific viewing conditions, remains underexplored. To fill this gap, we propose a novel benchmark for Hidden-Picture Optical Illusion Generation (HiP-Bench). Specifically, we construct a dataset of 512 samples and use 35 models to produce such illusions across three paradigms: (i) text-to-image, (ii) instruction-based image editing, and (iii) image-conditioned generation. A panel of human experts then evaluates 13,824 generated images across four dimensions, primarily focusing on illusion fidelity and shape consistency. Through five research questions, our analysis reveals the capability boundaries of existing models. Moreover, we explore Multi-Scale Distance Simulation (MDS), an automated baseline based on multimodal large language models that assesses illusion effectiveness through multi-scale visual inputs. Overall, this study establishes a reliable benchmark to facilitate and evaluate future progress.

Primary area:
datasets and benchmarks
Keywords:
multimodal datasets, evaluation, benchmarking, automated evaluation, image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_YDaiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HiP-Bench, a benchmark for evaluating Hidden-Picture Optical Illusion Generation. The benchmark contains 512 background–hidden target combinations and evaluates 35 visual generative models across three paradigms: text-to-image generation, instruction-based image editing, and image-conditioned generation. The authors conduct human evaluations of 13,824 generated images across four dimensions: Illusion Fidelity (IF), Overall Visual Quality (VQ), Scene Consistency (SC), and Illusion Shape Consistency (ISC). Through five research questions, the paper analyzes performance differences across generative paradigms, the influence of background textures and hidden target categories, and common failure modes. Additionally, the authors propose Multi-Scale Distance Simulation (MDS), an MLLM-based automated evaluation baseline that uses multi-scale visual inputs to improve alignment with human judgments.

Critical strengths
  1. A meaningful benchmark contribution. The paper addresses a relevant gap between existing understanding-oriented visual illusion benchmarks and the evaluation of hidden-picture optical illusion generation. Covering three generative paradigms makes HiP-Bench useful for assessing different approaches under a common evaluation framework.

  2. Well-motivated evaluation dimensions and substantial human annotations. The four evaluation dimensions capture important and potentially conflicting requirements of illusion generation, particularly Illusion Fidelity (IF) and Illusion Shape Consistency (ISC). The human evaluation of 13,824 generated images, together with inter-annotator agreement analysis, provides a valuable empirical foundation.

  3. Informative analysis of model limitations. Rather than relying solely on overall model rankings, the paper examines performance trade-offs across generative paradigms, background textures, and hidden target categories. The identified failure modes provide useful insights into the limitations of current generative models, while MDS offers a practical direction for automated evaluation.

Critical weaknesses
  1. Limited validation of distance-dependent human perception. Illusion Fidelity (IF) is a central evaluation dimension of HiP-Bench, emphasizing concealment quality and distance-dependent perceptibility. However, human annotators assess these effects using resized previews (128×128, 64×64, and 32×32) rather than actual changes in viewing distance. The paper does not verify whether this simulation produces judgments consistent with real-world viewing conditions, which may limit the perceptual validity of IF.

  2. Limited reliability of MDS across MLLM evaluators. Although Multi-Scale Distance Simulation (MDS) improves alignment with human judgments over direct MLLM evaluation, its effectiveness varies considerably across models. In Table 6, the SRCC values for IF are 7.1% for Qwen3-VL-8B, 31.1% for GPT-5.4, and 53.1% for Gemini-3-Pro. These relatively low and inconsistent correlations suggest that MDS remains dependent on the underlying MLLM, limiting its reliability as an automated evaluation baseline.

Questions for authors
  1. Why were only 27 of the 35 models selected for human evaluation? Could this affect the comparisons in RQ1?

  2. Are the 16 Editing and 18 CondGen validation cases used for hyperparameter tuning separate from the 512 benchmark samples?

  3. Are the model rankings for IF and ISC consistent across different random seeds?

Reviewer_wQWDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a benchmark for optical-illusion image synthesis. Specifically, it proposed an evaluation criteria for quality of optical-illusion images, provided a curated dataset of 512 combinations of background and hidden targets, proposed an automated pipeline to synthesize inputs for image-systhesis systems, conducted human evaluation, and proposed an automated evaluation method using MLLMs. Based on the evaluation results of existing image synthesis methods, this paper also performed some analysis regarding their performance and failure modes.

Critical strengths
  1. The proposed evaluation criteria is reasonable and properly reflects the core concerns regarding quality of optical-illusion images from multiple dimensions.
  2. The proposed dataset and input generation method properly captures typical setting of optical-illusion image generation, and the input generation method can be practically automated.
  3. The proposed automated evaluation method, MDS, can be practically applied. It also demonstrates increased performance in comparison to naive methods.
Critical weaknesses
  1. The proposed evaluation criteria is a set of high-level instruction that can only be reliably conducted as user study / human evaluation. Furthermore, 3 out of 4 of the metrics being used are already adopted in existing works. This makes the significance of the contribution questionable.
  2. The proposed automated evaluation method, while better than naive methods, is still notably misaligned with human evaluation, and its performance varies a lot between different MLLM backends.
  3. The discrepancy between the proposed automated evaluation method and human evaluation is not studied: when and how does automated evaluation fail?
  4. The analysis and insights provided in section 4 is largely unsuprising, and does not provide clear insights on how should optical-illusion image synthesis methods be improved.
Questions for authors
  1. (W1) Since the evaluation comprises solely of high-level question-answers regarding image contents, and can only be reliably conducted in the form of user study, what makes it important as a standalone research topic? And since 3 out of 4 metrics are being used in exising works (as shown in Table 3), please explain what are the key contributions of this work, and why they are significant.
  2. (W2) As demonstrated in Table 6, the performance of automated evaluation varies dramatically when using different MLLM backends. How do they differ? Does the proposed automated evaluation method rely on specific MLLM? Is there any criteria regarding how to choose the MLLM for automated evaluation?
  3. (W3) Please explain when and how does automated evaluation with MDS fail. For instance, does automated evaluation fail more frequently when encountering specific combinations of taxonomy classes? Does it overall tend to score higher or lower in certain metrics compared to human evaluation? What are the common reasons in the output when it fails to align with human evaluation?
  4. (W4) Could the authors explain what are the most informative findings in the analysis, and how do these findings help the development of optical-illusion image synthesis methods?
Reviewer_E81ROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HiP-Bench, a benchmark for generating hidden-picture optical illusions. In these images a target is hidden in the background up close and emerges when viewed from a distance. 32 instances for each of 4 background and 4 hidden-target types are paired into a balanced set of 512 combinations. Each paradigm gets its own inputs: GPT-4o-generated T2I prompts and editing instructions, synthesized background images for editing, and control images extracted from the targets for image-conditioned generation (CondGen). 35 models (22 T2I, 5 editing, 8 CondGen) generate the images.

For human evaluation, 13824 images from 27 models are each rated by two randomly assigned annotators, and the two ratings are averaged. They score four dimensions: Illusion Fidelity (IF, 1–5), Visual Quality (VQ, 1–5), Scene Consistency (SC, 0/1) and Illusion Shape Consistency (ISC, 1–3).

Main results:

  • T2I gets high VQ (4.29) and SC (0.97) but low IF (1.47).
  • CondGen gets the highest IF (2.03) and ISC (2.41) but the lowest VQ (3.40).
  • The best IF of any model is only 2.42/5 (SDXL + QR Code Monster).
  • Conventional metrics and MLLM judges correlate poorly with human IF/ISC.

The proposed MDS baseline works in two turns. The MLLM first scores the original image, then receives three downsampled versions and is asked to refine its judgment. This raises IF correlation, e.g., Gemini-3-Pro PLCC from 39.9 to 53.8. The statistical analysis uses stratified-bootstrap CIs, paired randomization tests and Holm correction.

Critical strengths
  1. A new and well-motivated task with careful human ground truth. Earlier hidden-picture benchmarks (IllusionBench, IllusoryVQA, HC-Bench) only test recognition. HiP-Bench is the first to evaluate generation. The input design across three paradigms is balanced and controlled, and it is supported by measured texture and shape statistics in Appendix C. The authors also show that reference-free metrics such as CLIP, VQAScore and NIQE, as well as plain MLLM judges, do not capture concealment. So human labels for IF and ISC across 13.8K images are a real contribution.
  2. A clear diagnosis of how paradigms fail. The analysis has concrete practical value for method designers. T2I treats the task as ordinary object rendering. Editing models have a narrow working range between making no change and pasting the target on top. Conditioning methods encode where the target goes but not how to express it through texture. Together with the 8-type failure taxonomy (e.g., Concept Omission is about 52–61% of low-scoring T2I and editing outputs, while Blatant Pasting is about 31% for CondGen), this points to what future methods need to fix.
Critical weaknesses
  1. The main claim about model capability is not supported for current frontier models. All 35 evaluated models are open-weight models, many from 2022–2024, such as SD 1.5, SDXL, DeepFloyd IF and InstructPix2Pix. Yet the paper itself uses frontier closed models elsewhere: Nano Banana 2 to make target images, and GPT-5.4 and Gemini-3-Pro as judges. Commercial generators and editors with strong reasoning and layout control (e.g., GPT-Image-2,2.5, Nano Banana 2, Seedream-5.0) are not evaluated. Because of this, statements like "no paradigm or model fully solves this task" and "revealing the capability boundaries of existing models" are not justified. For a benchmark paper this gap matters a lot. If frontier models already score well, the benchmark's difficulty and its findings could change significantly.
  2. The core metric (IF) and its automated proxy are not yet validated well enough for HiP-Bench to serve as a reusable benchmark. IF puts target presence and concealment on a single scale (1 = no illusion, 2 = pasted, 3 = visible outline, 4–5 = hidden). As a result, a higher mean IF can simply mean the target is shown more openly. For example, DiTs outscore U-Nets on IF largely because they omit the target less often (44.76% vs. 68.25%) and more often render it as a standalone object (32.01% vs. 14.69%), not because they hide it better.
  3. The comparison between paradigms is confounded, and the protocol has gaps.
  • The paradigms receive different information. CondGen gets an explicit spatial mask of the target, while T2I gets only text. "CondGen has higher IF" is therefore largely expected and does not show a paradigm-level difference.
  • Tuning effort is uneven. Editing and CondGen models were grid-searched on 16 and 18 cases sampled from the benchmark itself, with IF as the main objective. This risks test-set leakage. T2I models used default settings, a single seed (42) and one sample each.
  1. MDS reaches only moderate, judge-dependent IF correlation (PLCC 53.8 with Gemini-3-Pro, 32.5 with GPT-5.4 and 6.9 with Qwen3-VL-8B). It has no model-ranking validation, and it is not compared with existing concealment metrics such as Visual Anagrams' CLIP-based concealment score.
Questions for authors
  1. Can you add frontier closed-source generators and editors to all applicable paradigms, with at least a human-rated subset? If their IF is much higher than 2.42, how would your main conclusions change?
  2. Were the 16 and 18 tuning cases excluded from the reported results? How do RQ1 and RQ2 change if all models use default settings or are tuned on held-out items?
  3. How sensitive are the rankings to sampling? Please report variance over several seeds, or best-of-N results for at least the top models in each paradigm, and confidence intervals for the paradigm and category differences in RQ1 and RQ3.
  4. Does the MDS ablation (Appendix I.6) separate the effect of the downsampled views from simply adding a second reasoning turn? For example: two turns with no downsampled images, and one turn that includes the downsampled images.
  5. How was repeated exposure to the same background–target pair controlled, and does IF drift with exposure? Were images shown at the same physical size regardless of native resolution (384, 512 or 1024 px)?
Reviewer_isdHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HiP-Bench to evaluate generative models on their ability to create hidden-picture optical illusions, where a target is concealed at close range and emerges from a distance. The scope and execution of the work are genuinely solid: the authors curate 512 background–target combinations across three distinct generation paradigms, establish a rigorous human evaluation framework covering illusion fidelity, visual quality, scene consistency, and target shape consistency, and propose Multi-Scale Distance Simulation (MDS) as an automated two-stage scoring method utilizing original and downsampled views.

By evaluating 35 models and generating 17,920 images—with 13,824 images from 27 models independently rated by two human annotators—the study uncovers meaningful insights. For instance, image-conditioned generation enhances target structure and illusion fidelity at the expense of overall visual quality, and even the best-performing model achieves an average illusion fidelity of just 2.42 out of 5. Furthermore, the statistical comparisons, attribute analyses, and the breakdown of eight distinct failure modes thoroughly expose the boundaries of current methods. That said, while MDS noticeably narrows the gap between automated scores and human judgment, it still falls short of serving as a full replacement for human evaluation.

Critical strengths

A key strength is the systematic human evaluation of hidden-picture optical illusion generation. The benchmark separately evaluates illusion fidelity, visual quality, scene consistency, and target shape consistency, with each of the 13,824 images scored independently by two annotators. Standardized viewing conditions and inter-annotator agreement checks strengthen the reliability of these judgments. This provides a useful reference for evaluating concealment and scale-dependent target visibility, which conventional image metrics capture poorly.

Critical weaknesses

The perceptual validity of the core illusion metrics has received limited validation, as changes in viewing distance are primarily simulated using downsampled images. Inter-annotator agreement supports the consistency of the ratings but does not by itself establish that they accurately capture distance-dependent human perception. Since these ratings serve as the reference for both the benchmark and its automated evaluation method, this limitation weakens confidence in the validity of the resulting conclusions.

Questions for authors

Has the perceptual validity of the IF score been independently validated? Have ratings based on downsampled images been compared with target recognition performance under actual changes in viewing distance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.