a. Is the benchmark plan to be open-sourced? I did not find a statement on whether the benchmark will be released, on whether the task references or the judging instructions will be made open source. For a benchmark paper, this statement whether you plan to open source it or not should be very important.
b. All of the shown results in the paper are based on a single VLM judge. It is hard to understand the robustness of these experiments with the single judge. Adding more VLM judges and also showing agreement with humans would benefit the claims being made in the paper. Especially because many of the tasks in the benchmark are quite challenging visually, which VLMs struggle with [1,2,3]. Adding a small discussion around this would also be beneficial to the paper.
c. Related to my point above, there's also a bunch of VLM perception benchmarks[1,4,5,6,7] that try to test many of the same properties from LEVI, but from "benchmarking a VLM" perspective. While LEVI should exist on its own (since they go from prompt to image rather than image to answer), I think adding this line of literature to the discussion would be very helpful for a reader to contextualize where LEVI stands.
d. Another relayed point: Since the judge and the top three image generation models are all from OpenAI and trained by the same company, there might be a plausible confound (because we know that LLM judges are known to prefer their own generations[8]). Repeating the evaluation with a judge from a different family and reporting the ranks again would address this potential confound.
e. After skimming through some of the cited baselines in the paper, I'm having a hard time agreeing with Table 1.
1. Under "executable checks", both Trace-Bench and StructBench have the same mechanism that LEVI proposes for verification, where a VLM judge is provided with a checklist. I don't understand why they are not marked as benchmarks with executable checks, while LEVI is.
2. For "verifiable construction", Strict uses exact target strings with known ground truth, and StructBench renders its target images from code. I don't understand why they are marked as not verifiable constructions.
Can the authors please provide more details here?
f. Practically, the benchmark can be seen as a VLM judge with a rubric on a set of prompts and corresponding explicit constraints the generated image should contain. This decomposition of a prompt into atomic requirements is very close to what DSG[3] has done, which is not cited by the authors. They also report per-question agreement with humans, and VQAScore and GenAI-Bench[9] report agreement with human ratings. Citing these would explain what VVG adds beyond them, and it would also improve the positioning of the paper.
[1] Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2024, December). Vision language models are blind. In Asian Conference on Computer Vision (pp. 293-309). Singapore: Springer Nature Singapore.
[2] Chen, Z., Wen, Z., Du, Y., Zhou, Y., Cui, C., Han, S., ... & Yao, H. (2026). Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation?. Advances in Neural Information Processing Systems, 38.
[3] Cho, J., Hu, Y., Baldridge, J., Garg, R., Anderson, P., Krishna, R., ... & Wang, S. (2024, May). Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In International conference on learning representations (Vol. 2024, pp. 15625-15645).
[4] Kamoi, R., Zhang, Y., Das, S. S. S., Zhang, R. H., & Zhang, R. (2024). Visonlyqa: Large vision language models still struggle with visual perception of geometric information. arXiv preprint arXiv:2412.00947.
[5] Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., ... & Krishna, R. (2024, September). Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision (pp. 148-166). Cham: Springer Nature Switzerland.
[6] Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., ... & Xie, S. (2024). Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37, 87310-87356.
[7] Paiss, R., Ephrat, A., Tov, O., Zada, S., Mosseri, I., Irani, M., & Dekel, T. (2023, October). Teaching clip to count to ten. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 3147-3157). IEEE.
[8] Panickssery, A., Bowman, S. R., & Feng, S. (2024). Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37, 68772-68802.
[9] Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., ... & Ramanan, D. (2024, September). Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision (pp. 366-384). Cham: Springer Nature Switzerland.