- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HiP-Bench, a benchmark for evaluating Hidden-Picture Optical Illusion Generation. The benchmark contains 512 background–hidden target combinations and evaluates 35 visual generative models across three paradigms: text-to-image generation, instruction-based image editing, and image-conditioned generation. The authors conduct human evaluations of 13,824 generated images across four dimensions: Illusion Fidelity (IF), Overall Visual Quality (VQ), Scene Consistency (SC), and Illusion Shape Consistency (ISC). Through five research questions, the paper analyzes performance differences across generative paradigms, the influence of background textures and hidden target categories, and common failure modes. Additionally, the authors propose Multi-Scale Distance Simulation (MDS), an MLLM-based automated evaluation baseline that uses multi-scale visual inputs to improve alignment with human judgments.
Critical strengths
-
A meaningful benchmark contribution. The paper addresses a relevant gap between existing understanding-oriented visual illusion benchmarks and the evaluation of hidden-picture optical illusion generation. Covering three generative paradigms makes HiP-Bench useful for assessing different approaches under a common evaluation framework.
-
Well-motivated evaluation dimensions and substantial human annotations. The four evaluation dimensions capture important and potentially conflicting requirements of illusion generation, particularly Illusion Fidelity (IF) and Illusion Shape Consistency (ISC). The human evaluation of 13,824 generated images, together with inter-annotator agreement analysis, provides a valuable empirical foundation.
-
Informative analysis of model limitations. Rather than relying solely on overall model rankings, the paper examines performance trade-offs across generative paradigms, background textures, and hidden target categories. The identified failure modes provide useful insights into the limitations of current generative models, while MDS offers a practical direction for automated evaluation.
Critical weaknesses
-
Limited validation of distance-dependent human perception. Illusion Fidelity (IF) is a central evaluation dimension of HiP-Bench, emphasizing concealment quality and distance-dependent perceptibility. However, human annotators assess these effects using resized previews (128×128, 64×64, and 32×32) rather than actual changes in viewing distance. The paper does not verify whether this simulation produces judgments consistent with real-world viewing conditions, which may limit the perceptual validity of IF.
-
Limited reliability of MDS across MLLM evaluators. Although Multi-Scale Distance Simulation (MDS) improves alignment with human judgments over direct MLLM evaluation, its effectiveness varies considerably across models. In Table 6, the SRCC values for IF are 7.1% for Qwen3-VL-8B, 31.1% for GPT-5.4, and 53.1% for Gemini-3-Pro. These relatively low and inconsistent correlations suggest that MDS remains dependent on the underlying MLLM, limiting its reliability as an automated evaluation baseline.
Questions for authors
-
Why were only 27 of the 35 models selected for human evaluation? Could this affect the comparisons in RQ1?
-
Are the 16 Editing and 18 CondGen validation cases used for hyperparameter tuning separate from the 512 benchmark samples?
-
Are the model rankings for IF and ISC consistent across different random seeds?