acceptodds
Withdrawn from ICLR 2027

From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While Multimodal Large Language Models (MLLMs) are adept at answering “what” is in an image—identifying objects and describing scenes—they often lack the ability to understand “how” an image feels to a human observer. This gap is most evident when considering subjective cognitive properties, such as what makes an image memorable, funny, aesthetically pleasing, or emotionally evocative. To systematically address this challenge, we introduce CogIP-Bench, a comprehensive benchmark for evaluating MLLMs on such image cognitive properties. Our evaluation reveals a significant gap: current models are poorly aligned with human perception of these nuanced properties. We then demonstrate that a post-training phase can effectively bridge this gap, significantly enhancing the model’s alignment with human judgments. Furthermore, we show that this learned cognitive alignment is not merely predictive but also transferable to downstream creative tasks. By integrating our cognitively-aligned MLLM into an image generation pipeline, we can guide the synthesis process to produce images that better embody desired traits, such as being more memorable or visually appealing. Our work provides a benchmark to measure this human-like perception, a post-training pipeline to enhance it, and a demonstration that this alignment unlocks more human-centric AI.

Primary area:
generative models
Keywords:
MLLM, Cognition, Human Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FFiPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper tries to enhance the capability of existing MLLMs to identify human-aligned subjective characteristics such as memorability, funniness, and aesthetic appeal of images. It introduces a new benchmark to evaluate the MLLM's capabilities. Further, it also proposes a post-training procedure to enhance the MLLMs alignment, and shows results on the updated alignments.

My current recommendation is to reject the paper. The authors choose a very interesting and fundamental research question. However, the provided benchmark, method and results and their discussion are underwhelming and generally not living up to the announced contributions. The announced benchmark is a simple subset of existing datasets, no new data is provided. The fine-tuning method is a composition of very standard methods such as Lora. Finally, outdated models are used for evaluation and the analysis is very limited.

Critical strengths
  • S1: The paper is generally well written and pleasant to read. The problem is nicely introduced and discussed with sufficient context.
Critical weaknesses
  • Benchmark itself:
    a) The benchmark itself is composed of a subset of 4 existing datasets. There is no novel data collected, nor are new evaluations / annotations added beyond what is already available. Only an additional prompt to help the MLLM have dataset context is added. Claiming this as a new benchmarks is an overstatement.
    b) A clear reason on why this joint benchmark is more effective than the individual datasets is also missing.
    c) The benchmark is heavily downsampled from its original sized datasets. Sufficient analysis on fair representation of the diverse inputs is missing i.e. for the test sets. Additionally, there are overlaps between the datasets on the image level e.g. LaMem has images from AVA. Information on how this is handled is missing.

  • Evaluation: There are several concerns regarding evaluation:
    a) Many of the evaluated methods seem outdated: GPT-4o, GPT 5 or Claude Haiku 4.5 are released almost (or over) a year ago. In the context of fast moving MLLM capability development, the evaluations might not be representative.
    b) The models are prompted to be zero shot regression models / classifiers and only the received scores are evaluated. Given the fact that the MLLMs can provide exensive justifications on how they come up with a score / conclusion, the authors missed out on e.g. analyzing in more details what bias the models carry and how they come up with their scores. Additional analysis on e.g. on the influence of a thinking budget is neither provided nor discussed.

  • Post-training method: The proposed post-training method is very standard (i.e. Lora finetuning) and does not provide novel insight.

  • The post-training results show some improvements on the benchmark, however they are not impressive. Generalization across other benchmarks of the same tasks could have been one way of showing more robust results. However, the discusion on how the post-training is influencing

  • User study: The results are less impressive than I would expect from the proclaimed advantage of a now human-aligned model. Also, Age, Gender, Cultural background, etc. are very important information to provide when measuring subjective human feedback.

Questions for authors

See weaknesses.

Reviewer_2T1WOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CogIP-Bench, a comprehensive benchmark for evaluating Multimodal Large Language Models on image cognitive properties, covering aesthetics, funniness, emotional valence, and memorability. Each benchmark instance is a triplet of an image, a cognition-specific prompt, and a human-annotated score, drawn from human-annotated source datasets and sampled uniformly across score buckets to obtain a balanced distribution. Their evaluation reveals a significant gap, in that current models are poorly aligned with human perception of these nuanced properties. They then demonstrate that a post-training phase can effectively bridge this gap, significantly enhancing the model's alignment with human judgments. Furthermore, they show that this learned cognitive alignment is not merely predictive but also transferable to downstream creative tasks, since integrating their cognitively-aligned model as the backbone of the Qwen-Image pipeline guides the synthesis process to produce images that better embody desired traits, such as being more memorable or visually appealing. Their work provides a benchmark to measure this human-like perception, a post-training pipeline to enhance it, and a demonstration that this alignment unlocks more human-centric AI.

Critical strengths
  • The benchmark addresses a real and previously unmeasured gap, and it is built carefully. The authors assemble four subjective dimensions from established human-annotated source datasets, bucket and uniformly resample the scores to avoid a degenerate score distribution, and write dimension-specific prompts that make the task unambiguous.

  • The transfer experiment is the most consequential part of the paper, and it is instrumented far more seriously than the typical benchmark paper. Rather than stopping at score prediction, the authors swap their aligned model into the Qwen-Image backbone, hold the seed fixed, and evaluate with five automated preference metrics, specialist cognition predictors, and a randomised user study with bootstrap confidence intervals.

  • The paper also reports negative results honestly instead of burying them, including the finding that GRPO drives Gemma3 into prediction collapse toward central scores, together with a plausible mechanism.

Critical weaknesses
  • The quantitative claims are statistically fragile, and the paper does not establish that they are real. The test split holds only 300 instances per dimension; the results come from a single split with no cross-validation, no confidence intervals, and no significance testing, and many of the headline gains in Spearman correlation are small enough to sit inside the noise band at that sample size. These differences still carry improvement arrows and are tallied into an SFT Gains column, giving them more weight than they warrant. The problem is compounded by selective aggregation in Section 4.4, where the claim that improvements hold across most cognitive dimensions is computed over the eighteen non-memorability metrics, thereby excluding the dimension where fine-tuning actively breaks Llama, whose memorability error more than doubles.

  • The generation evidence does not support the strength of the claim built on it. On the general prompts, four of the five automated metrics move by negligible amounts, and only ImageReward moves substantially; on the cognition prompts, the memorability and aesthetics scores barely change.

  • The evaluation is also not clean: the prompts were written by GPT-5 explicitly to elicit each property, and the same specialist models produce the cognition scores of the generated images used as reference points in Table 1, so the loop is largely model-judged rather than human-judged. The user study, which is the only human signal, reaches significance on just two of the four dimensions, with emotion and funniness both failing to clear the threshold, yet the main text states that images from the aligned model consistently achieved higher preference rates and were preferred 1.2 times more often.

Questions for authors
  • What is the human-human agreement on each of the four dimensions, either from the source datasets or from your own re-annotation? Without a ceiling, it is hard to judge whether the reported correlations are close to achievable or far from it.

  • Given that GRPO dominates SFT on Qwen2.5-VL-7B across all eight metrics, what would change the decision to keep SFT as the primary method? Would a hybrid SFT-then-GRPO schedule, or per-backbone method selection, change the headline numbers?

  • Why keep memorability as a headline dimension when no model learns it under SFT and two of three backbones degrade on it? Is there a formulation, a training scale, or an input signal such as a saliency prior that makes it learnable?

Reviewer_thJwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CogIP-Bench samples existing human-annotated datasets to evaluate MLLMs on aesthetics, funniness, emotional valence, and memorability. The paper compares them with specialist predictors, applies LoRA SFT with categorical-then-numeric outputs and numerical soft labels, and transfers the fine-tuned Qwen2.5-VL backbone to Qwen-Image. The experiments cover score prediction, generation, and general-capability retention, with human preferences and ablations providing further comparisons. The contribution is this shared evaluation and transfer study, rather than a new soft-labeling loss (§§3–5).

Critical strengths
  • Table 1 reports both MAE and Spearman, distinguishing score errors from ranking errors. Comparing specialist, open, and API models under the same protocol exposes task differences: memorability remains difficult even for models that perform well elsewhere.
  • Generation transfer, seven general-capability tests, and output/loss ablations examine several effects of fine-tuning. The 32-participant study includes participant-level uncertainty (Appendix E), while Tables 5–10 report mixed human preferences and backbone-dependent GRPO failures alongside improvements.
Critical weaknesses
  • The memory and emotion results need a narrower interpretation. Qwen SFT memorability Spearman falls from .108 to .077 (Table 1), and its generation proxy rises from 7.760 to 7.770 (Table 4). Immediate image preference measures a different outcome from delayed recognition or recall. All participants judge the same 30 pairs per dimension, so the participant bootstrap leaves uncertainty across prompts unmeasured. Emotion and funniness preferences are nonsignificant in the reported analysis. Higher mean signed valence also need not mean better rendering of negative-emotion prompts (Fig. 4). Qwen's positive GRPO memorability result uses a different model from the SFT transfer experiment.
  • Several details needed to interpret the gains are missing or ambiguous. Section 3 gives sampling totals without source IDs, source-split handling, duplicate checks, or validation/model-selection rules. For HumorDB, the split should keep original/modified image families together. Table 9's Freeze LLM=True needs reconciliation with the Freeze-LLM ablation in Table 5; frozen base weights with trainable LoRA could explain the difference. It is consequently unclear which updated components enter Qwen-Image. Fig. 7 also labels aesthetics 3.491 as low, although its rule assigns values ≤3.5 to very low; the actual training targets need checking.
  • The experiments need more models to enhance the findings.
Questions for authors
  • Can existing outputs provide paired per-prompt uncertainty and emotion results grouped by intended valence? Which findings support behavioral memory improvement beyond predictor scores and immediate preference?
  • Could the authors provide split IDs/rules, image-family grouping, model-selection settings, and trainable modules for Tables 5/9, identify the weights transferred to Qwen-Image, and reconcile Fig. 7 with the training targets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.