acceptodds
Under review as a conference paper at ICLR 2027

Aesthetic Visual Reasoning with Multiple Images

Abstract

Aesthetic evaluation in photography often depends on collections rather than single images, as a photographer’s style emerges through recurring patterns and themes across multiple works. This motivates moving beyond single-image assessment toward group-level aesthetic modeling, which better captures holistic characteristics and supports applications such as photo recommendation, education, and style transfer. Existing Multimodal Large Language Models (MLLMs), however, largely focus on single-image or pairwise comparisons, which limits their group-level reasoning due to two key limitations: the absence of datasets with group-level aesthetic annotations and insufficient multi-image understanding in current models. To address these gaps, we introduce the first dataset for group-level aesthetic reasoning in photography, analyze the behavior of MLLMs in multi-image contexts, and uncover some unique properties of aesthetic reasoning across image groups. Building on these insights, we fine-tune a multi-image MLLM and introduce a lightweight visual triager that re-orders images based on their perceived difficulty, addressing the model’s sensitivity to image order and improving inference on challenging images. Experiments demonstrate that our approach significantly outperforms existing models.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Image Aesthetics Assessment, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uUDFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors addressed a new task of group-level aesthetic evaluation. Compared to the extensively studied single-image or pairwise aesthetic evaluation, group-wise aesthetic assessment has meaningful applications in real-world situations. The topic itself is interesting and the authors contributed a new dataset and a new benchmark, as well as a new model. Extensive experiments have been conducted and compared with existing MLLMs.

Critical strengths
  1. Group-level aesthetic assessment is a meaningful task, which would have many valuable applications in smart photography.
  2. The authors conducted systematic study, contributing VisGroup dataset, VisGroup-Bench benchmark and a new model visual triager.
  3. Model-agnostic visual triager is interesting, which significantly reduces the burden of MLLM post-training.
  4. Extensive experiments and comparisons were conducted to demonstrate the effectiveness of the proposed model.
Critical weaknesses
  1. While group-level aesthetic assessment is a new topic, the motivation behind the work is still not very clear. What is specific purpose, to rank the images in the group according to the aesthetic quality, or to understand the story behind the image group? More explanations are needed to make the motivation easily understood.
  2. The key finding that there is a severe sensitivity to image positioning in MLLMs is not new, which has been studied in other related researches.
  3. What is purpose of reordering input images according to perceived difficulty?
  4. More visual illustrations can be included in the experimental section.
Questions for authors
  1. Provide a more explanation of the motivation of the proposed work, preferably to combine it with specific application scenarios.
  2. What is specific purpose of reordering images according to perceived difficulty? A clear illustration with visual examples can make the paper easily understood.
2 comments

We sincerely thank you for your review!

W1 and Q1. Motivation and photographic applications

As illustrated at the beginning of Section 3, photograph understanding is essentially a group understanding task in many real-world applications from professional portfolio or album curation to daily image selection for social media, etc. A photographer’s style, intent, and visual language often become apparent only through a collection of works.

Therefore the goal is to understand a photographic collection as a whole, rather than only judge its individual images. A portfolio can contain individually strong photographs while lacking a coherent visual style. Conversely, photographs of different subjects can belong together because they share a consistent use of lighting, color, composition, or mood. Individual quality scores do not describe these relationships.

This distinction matters in portfolio curation and editorial selection. A photographer may need to understand which visual qualities recur across a body of work. An editor may need to compare two photographers' portfolios against a desired style. These decisions require identifying patterns across collections and explaining differences between them.

VisGroup-Bench evaluates these underlying abilities through group-style summaries, comparisons between collections, and matching groups by authorship or shared aesthetic characteristics, as illustrated in Figure 4. Figure 1 further supports this motivation by showing that additional images improve style and authorship judgments. We will make the connection between these tasks and photographic applications more explicit.

W2. Relationship to earlier positional-bias studies

Thank you for bringing this into attention. Positional sensitivity is studied in a few existing works. The major focus and main contribution of our work is not the positional sensitivity, but introducing group aesthetics understanding task with a dedicated dataset to enable systematic study of this problem.

Positional sensitivity is an interesting observation in our context, and it is different from existing works on our task in the following ways:

  1. Substantially more images than in similar tasks, such as Co-Instruct / MICBench, with 2 to 4 images per input in the comparison training data and 3 or 4 in MICBench for aesthetics understanding. We have upto 100 images per query.

  2. Collection-level judgment rather than per-image answers or needle retrieval. Tian et al. (2025) treat each image in PQA as a semantically independent single-image question. MileBench (2024) and Visual Haystacks (2025) ask the model to retrieve a needle, such as a password or an anchor object, among distractor images. Our task has no single target image, and the answer aggregates related evidence across the whole collection.

  3. Subtle aesthetic cues rather than salient content. Tian et al. (2025), MileBench (2024), and Visual Haystacks (2025) probe identifiable content, such as objects, text, or answers to general visual questions. In our task, the evidence lies in fine-grained lighting, color, composition, and mood that differ only subtly between photographs of a coherent collection.

W3 and Q2. Purpose of difficulty-based reordering

This is because, we discovered that (Section 4.2) the model has stronger perception capabilities at certain positions (the beginning and end of the sequence) in the input sequence. This motivated difficulty-based reordering. We place the images that are more difficult to perceive, in terms of visual aesthetics, into these positions so that they are better perceived. Here, difficulty measures how well the model is able to perceive an image's aesthetic attributes.

We added one example in Appendix J for intuitive illustration. In this example, the images share the flaw of having out-of-focus subjects, but the blur is subtler in some images and therefore harder for the model to perceive. Our visual triager places these more difficult images toward the end of the sequence (instead of the middle) where the model has stronger perceptual capabilities. After reordering, the model’s prediction changes from the incorrect “underexposed subject with motion blur” (A) to the correct “out-of-focus subject” (B).

Better perception of the difficult images therefore leads to better performance. As shown in Table 3, reordering improves accuracy from 80.6% to 82.2% on single-group questions and from 79.5% to 82.9% on multi-group questions.

W4. Additional visual illustrations

Thank you for this suggestion! We will move some of the visual illustrations from the Appendix to the paper in the final version, and add more figures for better illustration!

References

  1. Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, Xiaohong Liu, Guangtao Zhai, Shiqi Wang, and Weisi Lin. Towards open-ended visual quality comparison. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. (Co-Instruct / MICBench)
  2. Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Identifying and mitigating position bias of multi-image vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
  3. Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. MileBench: Benchmarking MLLMs in long context. In Conference on Language Modeling (COLM), 2024.
  4. Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David M. Chan. Visual Haystacks: A vision-centric needle-in-a-haystack benchmark. In International Conference on Learning Representations (ICLR), 2025.
Reviewer_XYVxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a critical gap in Multimodal Large Language Models (MLLMs) regarding group-level aesthetic visual reasoning. The authors introduced VisGroup, a large-scale dataset featuring 1.4 million image groups, and utilized an efficient O(n) negative phrase mining technique with LLaMA-3.3 to extract fine-grained aesthetic labels. Through systematic analysis, the paper is the first to uncover the "Lost in the Middle" (positional bias) phenomenon in long visual contexts. To mitigate this, the authors developed AVIS-7B, a model based on Qwen2.5-VL trained via Curriculum Learning, and introduced the Visual Triager—a lightweight, test-time optimization tool. Experiments demonstrate that AVIS significantly outperforms general-purpose models like GPT-4o in tasks such as group style summarization, photographer identification, and aesthetic comparison.

Critical strengths

S1. The study successfully transitions aesthetic evaluation from single/pairwise assessment to high-level group-level reasoning. By scientifically validating that vision models suffer from positional bias in long sequences (similar to long-context LLMs), it provides a crucial foundation for future multi-image research.

S2. The dataset construction is both massive (1.4M groups) and logically sound. The use of embedding similarity to reduce label-mining complexity from O(n**2) to O(n), combined with professional metadata (Author, Challenge) and LLM-refined critiques, ensures both the professional quality and the diversity of the data.

S3. The Visual Triager is a highly flexible, model-agnostic solution. The Spearman Rank Correlation Coefficient (SRCC) analysis in the Appendix rigorously proves that "single-image difficulty" is an effective proxy for predicting performance in long sequences, making the reordering strategy highly effective without requiring architectural changes.

Critical weaknesses

W1. Two-Pass Inference and Latency Overhead: The Visual Triager requires an initial pass to score all images before reordering and final reasoning. This two-pass mechanism introduces additional response latency, which may be a bottleneck in real-time interactive scenarios or when processing extremely large image sets. Future work could explore integrating "difficulty-awareness" or "importance-awareness" directly into a single forward pass (e.g., through attention mechanism distillation) to achieve one-pass inference efficiency.

W2. Teacher-Model Bias and Architectural Incrementalism: The annotation pipeline relies heavily on LLaMA-3.3, which may cause the AVIS model to inherit aesthetic biases or hallucinations from the "teacher" model. Additionally, the core architecture remains an incremental fine-tuning of existing backbones rather than a fundamental breakthrough in handling long visual sequences at the structural level. Suggest authors to incorporate human-in-the-loop alignment to correct inherited biases, and explore specialized attention mechanisms designed for long visual contexts to address the "Lost in the Middle" issue at the architectural root.

Questions for authors

Latency Overhead:This two-pass mechanism introduces additional response latency, which may be a bottleneck in real-time interactive scenarios or when processing extremely large image sets.

2 comments

We sincerely thank you for your review!

W1 and Q1. Inference latency

Thanks for bringing this into attention! The Triager indeed adds an additional inference stage, but it does not repeat the full multi-image reasoning pass. It uses a lightweight Qwen2.5-VL-3B to score images, which supports efficient batch-inference in parallel. We show the latency comparison between AVIS with and without the Visual Triager under transformers and vLLM evaluation environments. We test the latency under different input image sizes (N = 50, 75, 100). We report the average latency per sample over 10 runs.

Number of Input ImagesInference BackendAVIS (s)Additional Inference Time with Triager (s)
50Transformers0.496+0.079
75Transformers0.691+0.121
100Transformers1.047+0.149
50vLLM0.244+0.056
75vLLM0.357+0.081
100vLLM0.495+0.094

The Triager introduces only a small inference latency overhead, demonstrating its computational efficiency.

W2 (part 1). Teacher-model bias

Thanks for the advice!

Our datasets are built on existing human annotations and ground-truth metadata, rather than LLM judgments generated from scratch. Human-written photographic critiques from DP Challenge provide the aesthetic observations, while metadata records authorship, challenge assignments, and gallery memberships. The role of LLMs is to organize them into QA data. This grounding provides a strong basis for annotation quality.

  1. The benchmark is already verified through human review. As stated in Section 3.3, the 1,291 questions in VisGroup-Bench passed the human review. Reviewers (2-3 reviewer for each question) with photography experience examined the benchmark samples, and we retained only samples whose answers were accepted by the reviewers.

  2. The human validation audit for the data: We show the pass rate per category on the table during the review. As mentioned, we count a question as valid only if all reviewers reach an agreement towards accepting it, otherwise it will be rejected. The agreement rate represents the proportion of questions for which all reviewers unanimously agree on the decision to accept or reject.

    Photo/aesthetic categoryPass rate
    Emotion & Intent97.83%
    Aesthetic Style92.02%
    Color & Tone94.50%
    Composition97.06%
    Focus98.40%
    Lighting94.30%
    Low-level Visuals95.41%
    Narrative & Symbolism97.41%
    Photography Techniques94.44%
    Post-processing92.72%
    Scene & Culture95.18%
    Subject & Interaction95.73%
    Technical Flaws93.08%
    Photographer identity (deterministic)100.00%
    Agreement Rate97.94%
    Overall Pass Rate95.98%

W2 (part 2). Architectural incrementalism

Thanks for the advice! Our main contribution is introducing group aesthetics understanding task with a dedicated dataset to enable systematic study of this problem, rather than the design of a specific model architecture. Therefore we focus on the dataset and benchmark, as indicated by the selected Primary Area and stated in the Introduction. In line with this scope, we propose the Visual Triager as a lightweight, model-agnostic method that can be seamlessly integrated at inference time as a plug-in, therefore offers stronger generalization across different LLMs to support the task, not as a new model architecture.

AuthorsOpenReview ↗

Follow-up Summary of Our Initial Response

Dear Reviewer XYVx,

We thank you again for your thoughtful review!

We would like to follow up and briefly summarize how our responses could help to address your main concerns at this moment:

1. Latency: In the new table in the rebuttal, we show that, under different inference backends and different numbers of input images, the Visual Triager adds only a small amount of additional time per sample in the tested settings with batched image scoring with a lightweight 3B model.

2. Annotation quality: Our benchmark is actually human reviewed (Sec. 3.3). And in the human validation audit: the human validation yielded a 95.98% overall pass rate and 97.94% agreement rate, providing evidence of annotation reliability.

3. Contribution scope: Our main contribution is introducing group aesthetics understanding task with the dedicated dataset. Therefore the Visual Triager is introduced as a model-agnostic method that offers stronger generalization across different LLMs to support the task, not as a new specific model at architectural level. In our response to reviewer vQ2f, we also show how it works with different models at difference sizes and different families (we attached the table below).

ModelMethodSingle-GroupMulti-GroupAverage
InternVL3-1BBaseline47.637.842.7
InternVL3-1B+ Visual Triager47.840.144.0
InternVL3-2BBaseline48.334.841.5
InternVL3-2B+ Visual Triager50.335.042.7
Qwen3.8-27BBaseline63.938.351.1
Qwen3.8-27B+ Visual Triager65.039.352.2
GPT-5.6 SolBaseline56.958.957.9
GPT-5.6 Sol+ Visual Triager58.759.058.9

We would appreciate your consideration of these clarifications in your final assessment, and would be happy to address any remaining concerns! Thank you again!

Best, Authors

Reviewer_vQ2fOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on group-level aesthetic visual reasoning via MLLM, which is an important and challenging in image aesthetic quality assessment. The paper introduced a large scale dataset (VisGroup) with aesthetic labels extracted from structural metadata in photography forums. The dataset enabled evaluating multi-image reasoning across key aesthetic dimensions like photographic style and storytelling. The dataset was also used to finetune a MLLM backbone (Qwen2.5-VL-Instruct 7B) for group level aesthetic reasoning, which achieved good prediction accuracy on multiple aesthetic dimensions. To mitigate the "Lost in the middle" positional bias in long visual sequence, the paper trained another MLLM (Qwen2.5-VL-3B-Instruct) to predict perceptual difficulty, and used it to reorder images based on predicted difficulties to place harder images near context boundaries.

Critical strengths

The introduced VisGroup dataset contains 14M image pairs and comprehensive aesthetic annotations, which has good values for benchmarking aesthetic quality models.

The validation of the “Lost in the Middle” effect in multi-image aesthetic reasoning is insightful, and the proposed difficulty-based reordering showed positive improvements.

Critical weaknesses

The dataset curation pipeline depends heavily on LLMs (LLaMA-3-70B / LLaMA-3.3-70B) at almost every step: summarizing user comments, extracting key phrases, generating critiques, and creating/filtering MCQs. While automated filtering is described, the paper lacks a detailed human validation audit quantifying the ground-truth error rate.

Small technical novelty. The design and implementation of AVIS model and visual triager are relatively simple and straightforward.

Questions for authors

It would be good to include a human verification section to evaluate the accuracy of LLM labels, which is critical for a benchmarking dataset.

Is the "Lost in the Middle" effect just observed in the AVIS model, or is it common for other MLLMs? Is the proposed reordering method also effective on other models?

4 comments

We sincerely thank you for the review!

W1 and Q1. Ground-truth sources and human verification

Our datasets are built on existing human annotations and ground-truth metadata, rather than LLM judgments generated from scratch. Human-written photographic critiques from DP Challenge provide the aesthetic observations, while metadata records authorship, challenge assignments, and gallery memberships. The role of LLMs is to organize them into QA data. This grounding provides a strong basis for annotation quality.

  1. The benchmark is already filtered and verified through human review. As stated in Section 3.3, the 1,291 questions in VisGroup-Bench are filtered and verified through the human review. Reviewers (2-3 reviewers for each question) with photography experience examined the benchmark samples, and we retained only samples whose answers were accepted by the reviewers.

  2. The human validation audit for 2,000 randomly sampled data: We show the pass rate per category in the table below. As mentioned, we count a question as valid only if all reviewers reach an agreement towards accepting it, otherwise it will be rejected. The agreement rate represents the proportion of questions for which all reviewers unanimously agree on the decision to accept or reject.

    Photo/aesthetic categoryPass rate
    Emotion & Intent97.83%
    Aesthetic Style92.02%
    Color & Tone94.50%
    Composition97.06%
    Focus98.40%
    Lighting94.30%
    Low-level Visuals95.41%
    Narrative & Symbolism97.41%
    Photography Techniques94.44%
    Post-processing92.72%
    Scene & Culture95.18%
    Subject & Interaction95.73%
    Technical Flaws93.08%
    Photographer identity (deterministic)100.00%
    Agreement Rate97.94%
    Overall Pass Rate95.98%

W2. Technical novelty

Our main contribution is introducing group aesthetics understanding task with a dedicated dataset to enable systematic study of this problem, rather than the design of a specific model architecture. Therefore we focus on the dataset and benchmark, as indicated by the selected Primary Area and stated in the Introduction. In line with this scope, we propose the Visual Triager as a lightweight, model-agnostic method that can be seamlessly integrated at inference time as a plug-in, therefore offers stronger generalization across different LLMs to support the task, not as a new model architecture.

Q2. Positional bias and effectiveness on other models

Thanks for the question, and yes! Positional bias is not specific to AVIS, and our method can improve the model performance generally. We have results of different models across different sizes, families.

Additional evidence of position sensitivity. We conducted a controlled experiment to examine whether recognizing an image’s aesthetic attributes becomes harder when the image appears in the middle of a visual context, as we did in Section 4.2, Figure 5. We sampled 285 positively annotated image–attribute pairs. In the 20-image condition reported below, each sequence contains one target image and 19 distractors. We move the target to different positions while keeping its pixels, identifying label, question, and the distractors’ relative order unchanged.

Each question asks whether the target has its annotated attribute, so the correct answer is “Yes.” Before the model generates any answer text, we obtain its probabilities for the answer tokens “Yes” and “No” and compute:

difficulty = p(No) / [p(Yes) + p(No)]

This score measures the model’s preference for the incorrect answer after normalizing over the two answer choices. Higher difficulty therefore means less confidence in the correct answer. We report this probability-based diagnostic only for open-source models, because our proprietary-model interface does not expose the required log-probabilities for both answers.

Columns indicate the target image’s position, values are mean difficulty (%) over the same 285 pairs. We use the input image number=20 for the experiment.

ModelPosition 1Position 5Position 10Position 11Position 16Position 20
InternVL3-2B66.0371.0272.6671.9868.2060.44
Qwen3.8-27B43.3446.1247.6246.9947.4746.27

For both models, difficulty increases near the middle.

The follow-up experiments evaluate the Visual Triager re-ordering strategy on both open-source and proprietary models.

ModelMethodSingle-GroupMulti-GroupAverage
InternVL3-1BBaseline47.637.842.7
InternVL3-1B+ Visual Triager47.840.144.0
InternVL3-2BBaseline48.334.841.5
InternVL3-2B+ Visual Triager50.335.042.7
Qwen3.8-27BBaseline63.938.351.1
Qwen3.8-27B+ Visual Triager65.039.352.2
GPT-5.6 SolBaseline56.958.957.9
GPT-5.6 Sol+ Visual Triager58.759.058.9

Thanks for adding the human verification data, which addressed my concern about data quality. Changed my rating to weak acceptance.

We are glad that your concern has been addressed, and thank you again for your review!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.