- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors addressed a new task of group-level aesthetic evaluation. Compared to the extensively studied single-image or pairwise aesthetic evaluation, group-wise aesthetic assessment has meaningful applications in real-world situations. The topic itself is interesting and the authors contributed a new dataset and a new benchmark, as well as a new model. Extensive experiments have been conducted and compared with existing MLLMs.
Critical strengths
- Group-level aesthetic assessment is a meaningful task, which would have many valuable applications in smart photography.
- The authors conducted systematic study, contributing VisGroup dataset, VisGroup-Bench benchmark and a new model visual triager.
- Model-agnostic visual triager is interesting, which significantly reduces the burden of MLLM post-training.
- Extensive experiments and comparisons were conducted to demonstrate the effectiveness of the proposed model.
Critical weaknesses
- While group-level aesthetic assessment is a new topic, the motivation behind the work is still not very clear. What is specific purpose, to rank the images in the group according to the aesthetic quality, or to understand the story behind the image group? More explanations are needed to make the motivation easily understood.
- The key finding that there is a severe sensitivity to image positioning in MLLMs is not new, which has been studied in other related researches.
- What is purpose of reordering input images according to perceived difficulty?
- More visual illustrations can be included in the experimental section.
Questions for authors
- Provide a more explanation of the motivation of the proposed work, preferably to combine it with specific application scenarios.
- What is specific purpose of reordering images according to perceived difficulty? A clear illustration with visual examples can make the paper easily understood.
2 comments
References
- Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, Xiaohong Liu, Guangtao Zhai, Shiqi Wang, and Weisi Lin. Towards open-ended visual quality comparison. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. (Co-Instruct / MICBench)
- Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Identifying and mitigating position bias of multi-image vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
- Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. MileBench: Benchmarking MLLMs in long context. In Conference on Language Modeling (COLM), 2024.
- Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David M. Chan. Visual Haystacks: A vision-centric needle-in-a-haystack benchmark. In International Conference on Learning Representations (ICLR), 2025.
We sincerely thank you for your review!
W1 and Q1. Motivation and photographic applications
As illustrated at the beginning of Section 3, photograph understanding is essentially a group understanding task in many real-world applications from professional portfolio or album curation to daily image selection for social media, etc. A photographer’s style, intent, and visual language often become apparent only through a collection of works.
Therefore the goal is to understand a photographic collection as a whole, rather than only judge its individual images. A portfolio can contain individually strong photographs while lacking a coherent visual style. Conversely, photographs of different subjects can belong together because they share a consistent use of lighting, color, composition, or mood. Individual quality scores do not describe these relationships.
This distinction matters in portfolio curation and editorial selection. A photographer may need to understand which visual qualities recur across a body of work. An editor may need to compare two photographers' portfolios against a desired style. These decisions require identifying patterns across collections and explaining differences between them.
VisGroup-Bench evaluates these underlying abilities through group-style summaries, comparisons between collections, and matching groups by authorship or shared aesthetic characteristics, as illustrated in Figure 4. Figure 1 further supports this motivation by showing that additional images improve style and authorship judgments. We will make the connection between these tasks and photographic applications more explicit.
W2. Relationship to earlier positional-bias studies
Thank you for bringing this into attention. Positional sensitivity is studied in a few existing works. The major focus and main contribution of our work is not the positional sensitivity, but introducing group aesthetics understanding task with a dedicated dataset to enable systematic study of this problem.
Positional sensitivity is an interesting observation in our context, and it is different from existing works on our task in the following ways:
Substantially more images than in similar tasks, such as Co-Instruct / MICBench, with 2 to 4 images per input in the comparison training data and 3 or 4 in MICBench for aesthetics understanding. We have upto 100 images per query.
Collection-level judgment rather than per-image answers or needle retrieval. Tian et al. (2025) treat each image in PQA as a semantically independent single-image question. MileBench (2024) and Visual Haystacks (2025) ask the model to retrieve a needle, such as a password or an anchor object, among distractor images. Our task has no single target image, and the answer aggregates related evidence across the whole collection.
Subtle aesthetic cues rather than salient content. Tian et al. (2025), MileBench (2024), and Visual Haystacks (2025) probe identifiable content, such as objects, text, or answers to general visual questions. In our task, the evidence lies in fine-grained lighting, color, composition, and mood that differ only subtly between photographs of a coherent collection.
W3 and Q2. Purpose of difficulty-based reordering
This is because, we discovered that (Section 4.2) the model has stronger perception capabilities at certain positions (the beginning and end of the sequence) in the input sequence. This motivated difficulty-based reordering. We place the images that are more difficult to perceive, in terms of visual aesthetics, into these positions so that they are better perceived. Here, difficulty measures how well the model is able to perceive an image's aesthetic attributes.
We added one example in Appendix J for intuitive illustration. In this example, the images share the flaw of having out-of-focus subjects, but the blur is subtler in some images and therefore harder for the model to perceive. Our visual triager places these more difficult images toward the end of the sequence (instead of the middle) where the model has stronger perceptual capabilities. After reordering, the model’s prediction changes from the incorrect “underexposed subject with motion blur” (A) to the correct “out-of-focus subject” (B).
Better perception of the difficult images therefore leads to better performance. As shown in Table 3, reordering improves accuracy from 80.6% to 82.2% on single-group questions and from 79.5% to 82.9% on multi-group questions.
W4. Additional visual illustrations
Thank you for this suggestion! We will move some of the visual illustrations from the Appendix to the paper in the final version, and add more figures for better illustration!