acceptodds
Withdrawn from ICLR 2027

LEVI: Benchmarking Visual Identity Fidelity with Verifiable Low-Entropy Constructs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern generative models can produce visually plausible images while violating exact identity constraints. Evaluating such failures requires distinguishing violations of the requested identity from permissible variation in appearance. We introduce Low-Entropy Visual Identity (LEVI), which characterizes visual identities determined by compact facts and explicit rules. The low-entropy requirement concerns identity uncertainty, while appearance may remain diverse. We formulate identity fidelity in terms of the probability that a conditional image distribution assigns to the identities permitted by the prompt. Visual Identity Spaces (VIS) formalize this distinction by specifying admissible identity values and which different realizations express the same identity. Building on this formulation, we develop Verifiable Visual Grammar (VVG), a declarative language whose typed primitives and compositional constraints describe identity-bearing structures. Using this grammar, we build a benchmark for image generation. It assesses each output against the task's identity requirements, drawing evidence from the final pixels.

Primary area:
datasets and benchmarks
Keywords:
Image Generation, Visual Identity, Verifiable Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tzrpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Low Entropy Visual Identity (LEVI) as a new framework for evaluating image output from generative models. Main contributions include the introduction of LEVI, Verifiable Visual Grammar (VVG) for decomposition of the evaluation process, and a benchmark dataset including tasks across multiple domains. The method section is mainly a philosophical view of the image evaluation task, breaking it down into uncertainty of image identity and variation of realizations conditioned on a certain identity. The results provide the benchmark rankings of frontier models. The rankings somewhat resemble the gradient of model performance.

Critical strengths

The benchmark dataset (if provided) may be useful.

Critical weaknesses

No actual implementation of LEVI and VVG. The first and second terms have not been quantitatively analyzed whatsoever. The benchmark subsection in the methods section seems disconnected from the main idea. The dataset is not provided. No detailed instructions on how these experiments can be reproduced.

Questions for authors

How do you actually implement LEVI?

How exactly is identity computed in your experiments?

e^t,vt,Pt\hat{e}_t,v_t,P_t are not clearly defined in eq6. What exactly are they? E.g., domain, codomain, are they neural networks?, etc.

Reviewer_39z9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LEVI, a benchmark for testing ow accurately image generation models reproduce the visual identity while allowing the acceptable axes of variations that do not affect the visual identity. The authors split the uncertainty in generated images into two parts. First, the uncertainty about the identity and second, the variation in appearance, using the entropy chain rule. LEVI or Low-Entropy Visual Identity, is essentially the setting where identity is fully specified but the appearance can still vary. And they call the set of images that share the same identity as "Visual Identity Spaces".

To build the benchmark, there is a need for a more structured format to define the components' attributes and relationships for the task. To this end, the authors introduce VVG, Verifiable Visual Grammar, that helps define what the task is and what needs to be evaluated. The benchmark contains a total of 1375 tasks across symbolic, geometric, topological, layout, quantitative, and procedural domains. The main takeaway is that even the best model only passes ~59% of the tasks with counting and procedural being the hardest of tasks

Critical strengths

a. Moving beyond FID and CLIP score is, I think, the right direction in image generation research, especially because, as the models are getting more and more capable, these metrics are becoming saturated and give very little information. Having more benchmarks focused on specific capabilities is, I believe, always a great contribution.

b. The task design is very creative and covers a lot of failure types. The examples shown in the paper are pretty clear and also show failures that existing benchmarks would likely miss

c. By evaluating on 10 recent models, both open and closed source, and showing the per-domain breakdown for these models, these are quite informative, and the evaluations are quite informative.

Critical weaknesses

a. Is the benchmark plan to be open-sourced? I did not find a statement on whether the benchmark will be released, on whether the task references or the judging instructions will be made open source. For a benchmark paper, this statement whether you plan to open source it or not should be very important.

b. All of the shown results in the paper are based on a single VLM judge. It is hard to understand the robustness of these experiments with the single judge. Adding more VLM judges and also showing agreement with humans would benefit the claims being made in the paper. Especially because many of the tasks in the benchmark are quite challenging visually, which VLMs struggle with [1,2,3]. Adding a small discussion around this would also be beneficial to the paper.

c. Related to my point above, there's also a bunch of VLM perception benchmarks[1,4,5,6,7] that try to test many of the same properties from LEVI, but from "benchmarking a VLM" perspective. While LEVI should exist on its own (since they go from prompt to image rather than image to answer), I think adding this line of literature to the discussion would be very helpful for a reader to contextualize where LEVI stands.

d. Another relayed point: Since the judge and the top three image generation models are all from OpenAI and trained by the same company, there might be a plausible confound (because we know that LLM judges are known to prefer their own generations[8]). Repeating the evaluation with a judge from a different family and reporting the ranks again would address this potential confound.

e. After skimming through some of the cited baselines in the paper, I'm having a hard time agreeing with Table 1.

  1. Under "executable checks", both Trace-Bench and StructBench have the same mechanism that LEVI proposes for verification, where a VLM judge is provided with a checklist. I don't understand why they are not marked as benchmarks with executable checks, while LEVI is.

  2. For "verifiable construction", Strict uses exact target strings with known ground truth, and StructBench renders its target images from code. I don't understand why they are marked as not verifiable constructions.

Can the authors please provide more details here?

f. Practically, the benchmark can be seen as a VLM judge with a rubric on a set of prompts and corresponding explicit constraints the generated image should contain. This decomposition of a prompt into atomic requirements is very close to what DSG[3] has done, which is not cited by the authors. They also report per-question agreement with humans, and VQAScore and GenAI-Bench[9] report agreement with human ratings. Citing these would explain what VVG adds beyond them, and it would also improve the positioning of the paper.

[1] Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2024, December). Vision language models are blind. In Asian Conference on Computer Vision (pp. 293-309). Singapore: Springer Nature Singapore.

[2] Chen, Z., Wen, Z., Du, Y., Zhou, Y., Cui, C., Han, S., ... & Yao, H. (2026). Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation?. Advances in Neural Information Processing Systems, 38.

[3] Cho, J., Hu, Y., Baldridge, J., Garg, R., Anderson, P., Krishna, R., ... & Wang, S. (2024, May). Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In International conference on learning representations (Vol. 2024, pp. 15625-15645).

[4] Kamoi, R., Zhang, Y., Das, S. S. S., Zhang, R. H., & Zhang, R. (2024). Visonlyqa: Large vision language models still struggle with visual perception of geometric information. arXiv preprint arXiv:2412.00947.

[5] Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., ... & Krishna, R. (2024, September). Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision (pp. 148-166). Cham: Springer Nature Switzerland.

[6] Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., ... & Xie, S. (2024). Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37, 87310-87356.

[7] Paiss, R., Ephrat, A., Tov, O., Zada, S., Mosseri, I., Irani, M., & Dekel, T. (2023, October). Teaching clip to count to ten. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 3147-3157). IEEE.

[8] Panickssery, A., Bowman, S. R., & Feng, S. (2024). Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37, 68772-68802.

[9] Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., ... & Ramanan, D. (2024, September). Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision (pp. 366-384). Cham: Springer Nature Switzerland.
Questions for authors
  1. Can the authors show a few qualitative examples of a concrete VVG description for a task? Also, showing a few full examples of the grammar prompt and judging instruction would be helpful.

  2. For the rest of my questions, please refer to the weaknesses.

Reviewer_tK2GOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark and a framework for evaluating visual identity consistency in generative models. The proposed framework is called Low-Entropy Visual Identity (LEVI), and it involves the type of tasks where visual identity can be defined by an explicit set of rules. They then define the required rules as Visual Identity Spaces (VIS) and evaluate existing ten models on the tasks and show that existing models struggle at this task with the best performance of 58.69%

Critical strengths
  • The paper tackles an important and challenging issue of fine-grained visual consistency and constructs a sizeable benchmark with 1,375 tasks spanning multiple visual domians. The results suggest that current generators struggle and often violate their explicit structural or relational requirements.
  • Their proposed distinction between identity-defining requirements and acceptable appearance variation provides a useful way to organize procedural evaluation around these types of fine-grained generation tasks.
Critical weaknesses
  • The main evaluator is an automatic assessment from GPT-6-Sol with medium reasoning effort. Similar to how fine-grained generation is challenging (as demonstrated by the benchmark), fine-grained perception (for exact counts, symbolic details, topology, etc.) is also challenging and even with the decomposition of the requirements, it is very unreliable to offload evaluation to a general closed-source MLLM. The paper explicitly states that independent human agreement has not been measured, implying that there's no clear estimate or measure of how many false passes or failures there are in the reported results. This makes it very difficult to determine if the results and the model scores reliably reflect generation quality.
  • The paper mentions that the challenge set was curated post hoc partly based on multiple factors including "observed ease" (which I interpret as performance) across models. Although creating a sufficiently difficult benchmark is reasonable, it is unclear how much the reported difficulty and model rankings depend on these choices, and if this is an accurate reflection of the difficulty of these tasks as a whole.
Questions for authors

Clarifications:

  • What was the initial task pool, which models informed the post-hoc selection, and what criteria determined whether a task was too easy? How many tasks were removed for each reason?
  • How were the 120 base identity spaces selected, and what determined whether two task types were separate spaces or variations within the same space? Section 3.2 explains the general definition, but the selection and granularity of the actual catalogue remain unclear. How are different families determined? Some of it seems a bit arbitrary; for example, what's a "star family"?
  • How were the prompts, identity criteria, and vector references produced and checked in practice? The AI-use statement mentions AI-assisted curation and manual review, but there isn't sufficient details for reproducibility.
  • The novelty seems limited compared to existing procedural evaluation approaches such as Davidsonian Scene Graph and VPEval, and the paper is missing some citations / previous works around identity-consistency evaluation, especially for generative models such as NearId, ID-Sim, and Mind-the-glitch. It is also unsure how the procedural evaluation compares against these specifically identity-aligned evaluation metrics.

Additional Experiment:

  • Can you please report agreement between your proposed evaluator with independent human judgments on a representative subset, including false-pass and false-fail rates across domains and generators? Since two judging-instruction revisions were used, does the this change the scores or rankings?
  • Can you compare the proposed assessment with a simpler direct VLM judgment or other existing identity-consistency metrics on the same human-labeled subset to ablate if these atomic evaluations are needed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.