- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper tries to enhance the capability of existing MLLMs to identify human-aligned subjective characteristics such as memorability, funniness, and aesthetic appeal of images. It introduces a new benchmark to evaluate the MLLM's capabilities. Further, it also proposes a post-training procedure to enhance the MLLMs alignment, and shows results on the updated alignments.
My current recommendation is to reject the paper. The authors choose a very interesting and fundamental research question. However, the provided benchmark, method and results and their discussion are underwhelming and generally not living up to the announced contributions. The announced benchmark is a simple subset of existing datasets, no new data is provided. The fine-tuning method is a composition of very standard methods such as Lora. Finally, outdated models are used for evaluation and the analysis is very limited.
Critical strengths
- S1: The paper is generally well written and pleasant to read. The problem is nicely introduced and discussed with sufficient context.
Critical weaknesses
-
Benchmark itself:
a) The benchmark itself is composed of a subset of 4 existing datasets. There is no novel data collected, nor are new evaluations / annotations added beyond what is already available. Only an additional prompt to help the MLLM have dataset context is added. Claiming this as a new benchmarks is an overstatement.
b) A clear reason on why this joint benchmark is more effective than the individual datasets is also missing.
c) The benchmark is heavily downsampled from its original sized datasets. Sufficient analysis on fair representation of the diverse inputs is missing i.e. for the test sets. Additionally, there are overlaps between the datasets on the image level e.g. LaMem has images from AVA. Information on how this is handled is missing. -
Evaluation: There are several concerns regarding evaluation:
a) Many of the evaluated methods seem outdated: GPT-4o, GPT 5 or Claude Haiku 4.5 are released almost (or over) a year ago. In the context of fast moving MLLM capability development, the evaluations might not be representative.
b) The models are prompted to be zero shot regression models / classifiers and only the received scores are evaluated. Given the fact that the MLLMs can provide exensive justifications on how they come up with a score / conclusion, the authors missed out on e.g. analyzing in more details what bias the models carry and how they come up with their scores. Additional analysis on e.g. on the influence of a thinking budget is neither provided nor discussed. -
Post-training method: The proposed post-training method is very standard (i.e. Lora finetuning) and does not provide novel insight.
-
The post-training results show some improvements on the benchmark, however they are not impressive. Generalization across other benchmarks of the same tasks could have been one way of showing more robust results. However, the discusion on how the post-training is influencing
-
User study: The results are less impressive than I would expect from the proclaimed advantage of a now human-aligned model. Also, Age, Gender, Cultural background, etc. are very important information to provide when measuring subjective human feedback.
Questions for authors
See weaknesses.