acceptodds
Under review as a conference paper at ICLR 2027

IMPROVING MULTIMODAL MODELS IN AUTOMATED ARTWORK EVALUATION WITH OPTIMIZED SHOT SELECTION

Abstract

Automated artwork evaluation with multimodal large language models (MLLMs) is useful for art education and analysis, but few-shot performance can be unre liable when the examples supplied in context are not representative. We study an optimized shot-selection strategy for improving MLLM-based artwork evalu ation. Specifically, we compare random selection with K-Means and K-Medoids clustering, selecting representative artwork examples as few-shot demonstrations. Experiments on ratings of 337 student artworks show that K-Medoids generally provides more stable correlations with expert ratings and is especially effective for Color Richness and Picture Organization. The results also show that the num ber of shots has dimension-specific effects: additional examples can help some dimensions while harming others. These findings indicate that representative, outlier-resistant shot selection is an important component of reliable automated artwork evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Artwork evaluation, Few‑shot selection, K‑Medoids, In‑context learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_cE2ZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates few shot prompting to help multimodal LLMs rate art produced by school children. An experiment is carried out where the llm query for a particular artwork is augmented by 1, 3 or 5 human-rated artworks from a pool. The choice of the few-shots is done (1) randomly, (2) with k-means where the datapoint closest to the cluster mean is selected and (3) k-medioids. The result is compared against human expert ratings using pearson's correlation. The results seem to indicate that k-mediods helps the most, and that the effect of number of few-shots depends on the num of dimensions.

Critical strengths
  1. the problem is definitely worthwhile studying.
  2. the experiment results are sliced according to dimension, instead of just aggregates
Critical weaknesses
  1. The level of investigation seems quite superficial, almost like mediocre undergraduate work. Many interesting angles are completely ignored. To name a few: * what about similiarity of the target artwork with expertly graded datapoints? That would essentially rediscover RAG I suspect, but even so, it probably carries a lot more significance towards affecting grading results. * Teaching the LLM about the score range - cluster in score space not just image space * what happens when the model gets it wrong? * How much do human experts agree to begin with etc etc

  2. The paper is actually unclear in its key implelmentation decisions. We never get to hear what are the image features being used? Raw pixels? embeddings?

  3. Unclear what do the authors even mean by "stable"?

  4. The baselines are too weak and related work is missing. The only baseline is random selection. The obvious baselines from the few shot demonstration-selection literature are not there: per-query similarity retrieval (kNN over CLIP embeddings, e.g., Liu et al., 2022, "What Makes Good In-Context Examples for GPT-3?"), diversity- or coverage-based selection (e.g., vote-k, Su et al., 2023), and selection stratified by rating so that demonstrations span the score range. Also, no multimodal selection work and does not compare against supervised aesthetics or art-assessment regressors (e.g., NIMA-style models trained on the same 337 images). Without these, the practical value of what they are proposing is unknown.

Questions for authors
  1. have you thought about showing a different few-shot set depending on the input?
  2. what are the image feaatures you use?
  3. what is your definition of stable, according to which k-mediods is best?
Reviewer_VkU6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies demonstration selection for multimodal evaluation of student artwork. It compares random selection, nearest-centroid K-Means exemplars, and K-Medoids at one, three, and five shots, reporting Pearson correlations between ChatGPT-4 and expert scores across nine dimensions in a dataset of 337 artworks. The intended contribution is an empirical finding that representative, especially medoid-based, selection improves reliability, while the value of additional demonstrations varies by dimension.

Critical strengths
  1. The paper addresses a relevant educational application through inexpensive and interpretable selection methods. An empirical study of this kind could offer useful practical knowledge without requiring an algorithmic invention.

2.Reporting nine dimensions and several shot counts reveals meaningful trade-offs and failures, including cases where adding demonstrations worsens correlation. This supports a nuanced investigation of the selection problem.

Critical weaknesses
  1. Limited methodological novelty. The method mainly applies existing clustering algorithms to demonstration selection for artwork evaluation. The paper does not clearly explain what new insight it adds to ICL or establish its advantage over existing multimodal selection methods, such as MMICES.

  2. The results do not fully support the main claims. K-Means outperforms K-Medoids on Picture Organization at every nonzero shot setting. Also, the lack of repeated runs and controlled outlier experiments leaves the claimed stability and robustness insufficiently supported.

  3. The experimental setup needs clarification. The zero-shot results differ across selection methods, even though no demonstrations are used. Without explaining this difference and clarifying whether other settings are consistent, it is difficult to attribute the improvements to demonstration selection.

Questions for authors
  1. Were all conditions evaluated on the same artworks and expert targets under the same model/prompt settings? Please explain the differing zero-shot rows and specify the split, features, rubric, and inference protocol.

  2. Can you reconcile the Picture Organization conclusion, §6.2's zero-shot values, and the Line Texture discussion with Tables 3–5, and state the resulting supported claim?

  3. What evidence establishes repeatability and resistance to outliers? Can existing matched-run uncertainty be supplied, or should these claims be narrowed?

  4. Please clarify expert-rating reliability/aggregation and the student-data consent or exemption, de-identification, and external-model handling arrangements.

Reviewer_aRAYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares different ways to provide in-context samples to MLLMs for art evaluation. The ways include random pick, k-means centroids, and K-Medoids selection. The author uses 337 elementary students' artwork to evaluate the best way to provide representative samples. They found that K-Medoids selection yields a better alignment with the expert rating using a correlation score. It also shows that given more examples may not be necessary to improve performance.

Critical strengths
  1. The art evaluation is a hard question, as there is no ground-truth label. Even breaking it down into metrics is hard.

  2. In-context learning is a promising way to improve alignment between expert and MLLM judges.

Critical weaknesses

To be honest, this submission is like a course project. It would be better to submit to other venues instead of ICLR.

  1. Figure 1 is not self-explanatory. I have difficulty understanding what the author wants to convey.

  2. The data selection is predefined. Only the random, K-means, and K-Medoids clustering algorithms are included. Those methods are trivial methods for ICLR audience.

  3. No dataset information is provided, such as how experts evaluate all images.

  4. The conclusion can only be applied to ChatGPT 4. Later models like ChatGPT 6 may respond very differently to few-shot learning.

Questions for authors

I don't have any questions for the authors, as the paper's content is easy to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.