- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates few shot prompting to help multimodal LLMs rate art produced by school children. An experiment is carried out where the llm query for a particular artwork is augmented by 1, 3 or 5 human-rated artworks from a pool. The choice of the few-shots is done (1) randomly, (2) with k-means where the datapoint closest to the cluster mean is selected and (3) k-medioids. The result is compared against human expert ratings using pearson's correlation. The results seem to indicate that k-mediods helps the most, and that the effect of number of few-shots depends on the num of dimensions.
Critical strengths
- the problem is definitely worthwhile studying.
- the experiment results are sliced according to dimension, instead of just aggregates
Critical weaknesses
-
The level of investigation seems quite superficial, almost like mediocre undergraduate work. Many interesting angles are completely ignored. To name a few: * what about similiarity of the target artwork with expertly graded datapoints? That would essentially rediscover RAG I suspect, but even so, it probably carries a lot more significance towards affecting grading results. * Teaching the LLM about the score range - cluster in score space not just image space * what happens when the model gets it wrong? * How much do human experts agree to begin with etc etc
-
The paper is actually unclear in its key implelmentation decisions. We never get to hear what are the image features being used? Raw pixels? embeddings?
-
Unclear what do the authors even mean by "stable"?
-
The baselines are too weak and related work is missing. The only baseline is random selection. The obvious baselines from the few shot demonstration-selection literature are not there: per-query similarity retrieval (kNN over CLIP embeddings, e.g., Liu et al., 2022, "What Makes Good In-Context Examples for GPT-3?"), diversity- or coverage-based selection (e.g., vote-k, Su et al., 2023), and selection stratified by rating so that demonstrations span the score range. Also, no multimodal selection work and does not compare against supervised aesthetics or art-assessment regressors (e.g., NIMA-style models trained on the same 337 images). Without these, the practical value of what they are proposing is unknown.
Questions for authors
- have you thought about showing a different few-shot set depending on the input?
- what are the image feaatures you use?
- what is your definition of stable, according to which k-mediods is best?