- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MuseFACET, a benchmark for evaluating the capability of MLLM models to understand and interpret different objects and different characteristics in images belonging to museum collections. The authors introduce two tasks with relative evaluation frameworks, and evalaute 14 different models. The results highlight common struggles among the models, including different performance on images from different geographical sources.
Critical strengths
The proposed benchmark is collected in a thorough and sound way. It is particularly interesting that the authors include museums from different regions of the world. They also include human evaluation, which has been proposed with an iterative protocol.
Critical weaknesses
While reading the beginning of the paper, I was genuinely confused about the usage of the word "collection". I was expecting that the tasks of this benchmark would indeed include understanding of these image collections from a set point of view. There are indeed papers and tasks emerging on this topic, but it seems that the authors are not aware of such literature. I believe that the framing of the paper, as it is, is confusing, because the proposed tasks are actually performed on single images and it's just the aggregation of the results that then supports the usage of the word "collection".
I recommend the authors to get more familiar with existing literature on image-to-text generation for collections of images. Just to mention a few:
[1] Riccio, P., Galati, F., Schweighofer, K., Garcia, N., & Oliver, N. M. (2026, March). Imageset2text: Describing sets of images through text. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 11, pp. 8731-8739).
[2] Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., ... & Yeung-Levy, S. (2024, June). Describing differences in image sets with natural language. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24199-24208). IEEE.
[3] Deng, B., Peng, S., Genova, K., Wetzstein, G., Snavely, N., Guibas, L., & Funkhouser, T. (2025, October). Visual chronicles: Using multimodal llms to analyze massive collections of images. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 12769-12778). IEEE.
[4] Liu, M., Zhong, Z., Li, J., Franchi, G., Roy, S., & Ricci, E. (2026). Organizing unstructured image collections using natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8907-8918).
[5] Van Noord, N. (2023, October). Prototype-based dataset comparison. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 1944-1954). IEEE.
The first 4 papers in particular also propose new tasks in the context of understanding of image collections, and they all rely on MLLMs. [1] also proposes a dataset of artistic collections and has the task of providing interpretation of such collections. It would be interesting if such collection-specific models/frameworks could be somehow included into the evaluation of the benchmark. If unfeasible, it would still be relevant to contextualize this paper better within existing literature around these topics. Existing literature might also consider more art-specific tasks already proposed in the literature, considering for example
[6] Ramos, P., Gonthier, N., Khan, S., Nakashima, Y., & Garcia, N. (2025, February). No annotations for object detection in art through stable diffusion. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 6228-6237). IEEE.
or also, on a similar intent, the following paper also analyzes the understanding of art history by MLLMs
[7] Strafforello, O., Soydaner, D., Willems, M., Maerten, A. S., & De Winter, S. (2025, September). Have large vision-language models mastered art history?. In International Conference on Image Analysis and Processing (pp. 524-544). Cham: Springer Nature Switzerland.
And finally a very recent paper (makes sense that the authors were not aware of this one) is also art-specific and discusses the different interpretations that artistic images could have, could still be a nice addition/inspiration.
[8] Schaerf, L., Purificato, A., Riccio, P., Silvestri, F., & Garcia, N. (2026, September). Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-relational Representations. In European Conference on Computer Vision (pp. 273-290). Cham: Springer Nature Switzerland.
Overall, the confusing usage of the world "collection" and the lack of relevant literature in this paper makes me suggest that the authors should really make a further effort to ground their paper better in existing literature. Not only considering MLLMs but maybe also some of the models already proposed for image collection understanding and possibly adapting existing recently proposed tasks would really make a big different in how this paper positions itself and its contributions.
I would also like to point out that there are missing spaces in lines 84, 86 and 91.
Questions for authors
-
Would it be possible to provide an overview of how to better frame this work in existing literature, either with extra experiments or with a different framing of the work and the "Related Work" section?
-
It is not clear why the performance is better on art from Western museums. In particular, it is not clear whether there might be some data leakage between the images in this benchmark and the training sets of the models. Could the authors clarify whether those images were available online and whether they might have entered the training sets of the models? And by doing this analysis, is there a chance that the different performance is not "simply" model bias but purely memorization from the training sets?