acceptodds
Under review as a conference paper at ICLR 2027

MuseFACET: Benchmarking Trustworthy and Controllable Image-to-Text Generation for Musuem Collection Interpretation

Abstract

Museum collection interpretation represents a challenging task for Multimodal Large Language Models (MLLMs), requiring accurate object identification and knowledge-grounded, context-aware generation beyond literal visual descriptions. However, existing efforts on open-ended text generation for artworks and museum objects remain limited in dataset scale and diversity or in evaluation protocols that rely largely on reference-based similarity metrics. To address these limitations, we introduce MuseFACET, a benchmark comprising 5,000 museum objects from 9 museums. MuseFACET presents two hierarchical tasks: Identification task for predicting object attributes from images and Interpretation task for generating coherent, context-aware descriptions. We further propose an evaluation framework emphasizing trustworthiness and controllability through identification accuracy and five interpretation criteria: Faithfulness, Completeness, Conciseness, Depth, and Acuity. Evaluation of 14 proprietary and open-source MLLMs shows that the highest accuracy on the Identification task reaches only 31.2%. For Interpretation, models exhibit persistent limitations in factual grounding and object-specific insight under the image-only setting, while providing metadata substantially improves performance. Model performance also varies across cultural regions, revealing uneven capabilities in cultural interpretation. MuseFACET provides a systematic benchmark for museum collection interpretation and a concrete evaluation protocol for trustworthy and controllable open-ended image-to-text generation.

Primary area:
datasets and benchmarks
Keywords:
Image-to-Text Generation, Musuem Collection Interpretation, Benchmark Dataset, Visual Understanding, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kTZtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MuseFACET, a benchmark for evaluating the capability of MLLM models to understand and interpret different objects and different characteristics in images belonging to museum collections. The authors introduce two tasks with relative evaluation frameworks, and evalaute 14 different models. The results highlight common struggles among the models, including different performance on images from different geographical sources.

Critical strengths

The proposed benchmark is collected in a thorough and sound way. It is particularly interesting that the authors include museums from different regions of the world. They also include human evaluation, which has been proposed with an iterative protocol.

Critical weaknesses

While reading the beginning of the paper, I was genuinely confused about the usage of the word "collection". I was expecting that the tasks of this benchmark would indeed include understanding of these image collections from a set point of view. There are indeed papers and tasks emerging on this topic, but it seems that the authors are not aware of such literature. I believe that the framing of the paper, as it is, is confusing, because the proposed tasks are actually performed on single images and it's just the aggregation of the results that then supports the usage of the word "collection".

I recommend the authors to get more familiar with existing literature on image-to-text generation for collections of images. Just to mention a few:

[1] Riccio, P., Galati, F., Schweighofer, K., Garcia, N., & Oliver, N. M. (2026, March). Imageset2text: Describing sets of images through text. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 11, pp. 8731-8739).

[2] Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., ... & Yeung-Levy, S. (2024, June). Describing differences in image sets with natural language. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24199-24208). IEEE.

[3] Deng, B., Peng, S., Genova, K., Wetzstein, G., Snavely, N., Guibas, L., & Funkhouser, T. (2025, October). Visual chronicles: Using multimodal llms to analyze massive collections of images. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 12769-12778). IEEE.

[4] Liu, M., Zhong, Z., Li, J., Franchi, G., Roy, S., & Ricci, E. (2026). Organizing unstructured image collections using natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8907-8918).

[5] Van Noord, N. (2023, October). Prototype-based dataset comparison. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 1944-1954). IEEE.

The first 4 papers in particular also propose new tasks in the context of understanding of image collections, and they all rely on MLLMs. [1] also proposes a dataset of artistic collections and has the task of providing interpretation of such collections. It would be interesting if such collection-specific models/frameworks could be somehow included into the evaluation of the benchmark. If unfeasible, it would still be relevant to contextualize this paper better within existing literature around these topics. Existing literature might also consider more art-specific tasks already proposed in the literature, considering for example

[6] Ramos, P., Gonthier, N., Khan, S., Nakashima, Y., & Garcia, N. (2025, February). No annotations for object detection in art through stable diffusion. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 6228-6237). IEEE.

or also, on a similar intent, the following paper also analyzes the understanding of art history by MLLMs

[7] Strafforello, O., Soydaner, D., Willems, M., Maerten, A. S., & De Winter, S. (2025, September). Have large vision-language models mastered art history?. In International Conference on Image Analysis and Processing (pp. 524-544). Cham: Springer Nature Switzerland.

And finally a very recent paper (makes sense that the authors were not aware of this one) is also art-specific and discusses the different interpretations that artistic images could have, could still be a nice addition/inspiration.

[8] Schaerf, L., Purificato, A., Riccio, P., Silvestri, F., & Garcia, N. (2026, September). Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-relational Representations. In European Conference on Computer Vision (pp. 273-290). Cham: Springer Nature Switzerland.

Overall, the confusing usage of the world "collection" and the lack of relevant literature in this paper makes me suggest that the authors should really make a further effort to ground their paper better in existing literature. Not only considering MLLMs but maybe also some of the models already proposed for image collection understanding and possibly adapting existing recently proposed tasks would really make a big different in how this paper positions itself and its contributions.

I would also like to point out that there are missing spaces in lines 84, 86 and 91.

Questions for authors
  • Would it be possible to provide an overview of how to better frame this work in existing literature, either with extra experiments or with a different framing of the work and the "Related Work" section?

  • It is not clear why the performance is better on art from Western museums. In particular, it is not clear whether there might be some data leakage between the images in this benchmark and the training sets of the models. Could the authors clarify whether those images were available online and whether they might have entered the training sets of the models? And by doing this analysis, is there a chance that the different performance is not "simply" model bias but purely memorization from the training sets?

Reviewer_gnH4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MuseFACET evaluates museum-object identification and open-ended interpretation using 5,000 objects from nine museums. The Identification task predicts title, date, materials, and country from an image. The Interpretation task generates descriptions under image-only and metadata-guided conditions, evaluated along five dimensions: Faithfulness, Completeness, Conciseness, Depth, and Acuity.

The paper evaluates 14 models and analyzes the effects of metadata and web search, as well as performance differences across regions. It also assesses agreement between automatic and human evaluation using 200 responses.

Critical strengths

1. The work of organizing heterogeneous museum records into a common evaluation resource is thorough.

Sections 3.1–3.2 describe schema alignment, metadata normalization, diversity-oriented sampling, human review, and error correction. The initial audit flagged issues in 18.8% of records, and ambiguous cases were reviewed further or excluded. The resulting dataset is a useful resource for comparing objects from multiple museums under a common protocol.

2. The multidimensional evaluation reveals distinctions that a single score can miss.

In Table 5, web search improves Completeness and Depth in all eight model–setting pairs, but improves Faithfulness in only three. The finding that richer interpretation does not necessarily imply greater factual faithfulness is important.

Table 6 reports inter-annotator agreement of α = 0.833 for the total score and a Spearman correlation of ρ = 0.758 between automatic scores and mean human ratings. Providing evidence of how well automatic evaluation aligns with human judgment is another strength.

The audited multi-institutional dataset and multidimensional evaluation are the main reasons for my weak acceptance, particularly the distinction between descriptive richness and factual faithfulness.

Critical weaknesses

1. The relationship between Identification accuracy and trustworthiness is not sufficiently disentangled.

Identifying production dates or geographic origins depends on both visual information and knowledge about the object. The current score does not clearly distinguish visual-recognition failures, missing knowledge, and appropriate abstention.

Appendix G permits a response of “not sure” when identification is difficult, but Appendix I.1 assigns it zero, as it does an incorrect answer. This measures identification success but does not distinguish confident errors from appropriate abstention. The relationship between each metric and the paper's emphasis on trustworthiness needs a clearer definition.

2. The extent to which human validation supports dimension-level comparisons and the full model set is unclear.

The human validation in Appendix M covers Interpretation responses from seven proprietary models. Evidence for assigning the same level of reliability to Identification or open-source outputs is limited.

Table 35 reports correlations across individual responses, including 0.546 for Conciseness and 0.531 for Depth. These correlations do not directly establish agreement on between-model rankings or the paired effects of adding metadata or web search. The scope of the available human judgments and the comparisons they directly support should be distinguished from conclusions based only on automatic scores.

The date-score discrepancy raised in Q3 also needs clarification, particularly whether it is localized or affects the main comparisons. I would reconsider my recommendation if this discrepancy or the limitations of human validation materially change the main comparative conclusions.

Questions for authors

Q1. How should visual identification, object knowledge, and abstention be distinguished when interpreting Identification results? Please explain how the frequency and treatment of “not sure” affect the main model comparisons and conclusions about trustworthiness.

Q2. Appendix M.1 describes a stratified sample from the main experiments, where web search is disabled. Please clarify whether any search-enabled outputs were included. For metadata or search comparisons with sufficient matched-object coverage in the existing sample, do human judgments support the direction of the reported dimension-specific effects? Please report the available paired sample sizes and distinguish directly validated comparisons from findings based only on automatic scores.

Q3. Appendix I.3 states that Date is evaluated deterministically, while Appendix J states that the two judges evaluate the same 1,000 outputs. However, the kimi-vl-A3B-thinking row in Table 30 reports Date scores of 44.4% and 14.6%. Please clarify the source of this discrepancy and its impact on other results.

Reviewer_d6xwOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

MuseFACET benchmarks museum-object identification and open-ended interpretation using 5,000 objects from nine museums. Identification predicts four metadata fields from an image; Interpretation compares image-only and verified-metadata-guided generation. A five-dimensional rubric evaluates description quality, with human validation on 200 responses. Experiments with 14 models show substantial identification errors and improved interpretation scores when reliable metadata is supplied. The main contribution is a curated resource and evaluation framework, rather than a new generation method.

Critical strengths

Substantive resource construction. The multi-museum resource involves schema alignment, metadata normalization, diversity-oriented sampling, and manual quality control. Pairing images with verified metadata and museum descriptions provides a useful basis for evaluating knowledge-supported interpretation beyond conventional captioning.

Meaningful human validation. The evaluation goes beyond reference-text similarity and includes an independent human check. Three annotators rate 200 responses, yielding total-score inter-annotator agreement of α=0.833 and human–LLM correlation of ρ=0.758. This supports the aggregate evaluation, although validation covers proprietary outputs and correlations are weaker for individual dimensions.

Critical weaknesses

Identification accuracy is not sufficiently separated from visual answerability and trustworthy uncertainty handling. A single image may not uniquely establish an object’s specific title, date, or provenance. Appendix G permits “not sure,” but Appendix I scores both abstention and incorrect predictions as zero. Consequently, the reported accuracy cannot distinguish appropriate uncertainty from unsupported assertions, and low scores may conflate model failures with insufficient input evidence. This materially limits the interpretation of Identification as evidence about trustworthiness, rather than simply catalog-attribute prediction.

The evidence for controllability is narrower than the framing suggests. The experiments assess a fixed requirement to produce a concise paragraph and compare access to metadata; they do not directly test responses to changing content or presentation targets. These results support knowledge-assisted description quality, but not broader control fidelity. Moreover, without a metadata-only comparison, the metadata-guided gains do not isolate the contribution of visual evidence. The framework’s value as a benchmark for controllable, visually grounded generation therefore remains insufficiently established.

Questions for authors

Can you quantify image-level answerability on a representative subset and distinguish appropriate abstention from incorrect assertions, for example through coverage and error rates among answered fields? This would help determine whether the Identification findings reflect model limitations or information unavailable in the input.

Is “controllability” intended only as compliance with fixed generation requirements? If so, please clarify the scope of the claims. If broader control is intended, can you provide direct evidence using varied content or presentation targets while checking factual fidelity?

Can you provide a small metadata-only comparison to establish whether metadata-guided Interpretation scores depend meaningfully on the image?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.