acceptodds
Under review as a conference paper at ICLR 2027

VDBench: A Benchmark for Visual Diachronic Analysis of Historical Depictions

Abstract

We introduce Visual Diachronic Analysis (VDA), a new computer vision challenge for studying how the visual context of a depicted subject or theme changes over time and how these changes relate to documented historical shifts in meaning. This raises the question to what extent such changes in meaning can be inferred computationally from historical depictions. To study such phenomena, we introduce VDBench, the first benchmark for VDA. In the absence of available and sufficiently annotated historical image collections, we propose an unconventional approach: we construct a historically grounded synthetic benchmark dataset. We compile a source-grounded collection of human-centred diachronic topics and design a controlled text-to-image pipeline to generate plausible synthetic historical depictions. Domain experts confirm the suitability and plausibility of the data. We specify several computer vision tasks, propose a suitable evaluation methodology, and provide baseline methods and implementations as reference for future work. The dataset and an interactive explorer are available at https://anonymousauthor287.github.io/projectpage/. All code will be released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
visual diachronic analysis, benchmark, synthetic data, text-to-image generation, vision-language models, temporal reasoning, visual context, semantic change

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UHfPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Visual Diachronic Analysis (VDA), a new computer vision task that aims to identify how the visual context and social meaning of subjects change over time. The authors further introduce VDBench, a benchmark which contains 41 historical topics, 201 historical periods, and 20,100 synthetically generated images. The authors propose two evaluation tasks: (1) visual context detection, which involves predicting the year, environment, activity, and caption of the generated images, and (2) topic and trajectory discovery, which aims to describe how a subject's representation changes over time. They find that reconstructing historical trajectories remains challenging, particularly when relying on predicted image metadata rather than ground-truth information.

Critical strengths
  1. I find the problem well-motivated and important for the community of digital humanities.
  2. It is apparent that the authors have put considerable effort and thought into the dataset construction, including LLM-based image generation and expert validation.
Critical weaknesses
  1. Use of synthetic images: My main concern is whether the benchmark actually evaluates the ability to discover historical changes from images. The authors first identify historical changes from existing sources and then instruct an image-generation model to represent the meaning of each period. While I understand the motivation for using synthetic data, I am not convinced that good performance on this benchmark would necessarily translate to real historical photographs. I think an evaluation on at least a small collection of real images would strengthen this contribution.

  2. Reliance on temporal information: Task 2 aims to identify how visual representations change over time. However, the results in app. h.5 show that sorting images by year and dividing them into periods can achieve better image-grouping results than the evaluated baselines. This raises the question of how much visual understanding is actually needed to perform the grouping task. I would have liked to see stronger evidence that the models identify changes in visual content rather than mainly relying on temporal information.

  3. Evaluation of historical meaning: The authors evaluate the generated descriptions using BERTScore and MoverScore, which measure similarity to reference texts rather than historical correctness. The analysis in app. h.3 shows that these metrics have limited ability to distinguish descriptions of different periods within the same topic. This makes it difficult to determine whether the models have correctly interpreted the historical changes. I think a human evaluation of the generated descriptions would be valuable here.

  4. Clarity and presentation: In general, I found the paper difficult to follow, especially when trying to understand the overall goal of the benchmark and its intended use. The paper introduces many concepts and technical details about dataset construction before clearly explaining what the models are expected to do.

Questions for authors
  1. Could the authors evaluate their methods on a small collection of real historical photographs to test whether the results generalise?
  2. Could the authors provide an evaluation where models must identify historical periods without access to year information?
1 comment
AuthorsOpenReview ↗

Response to Reviewer UHfP

We sincerely thank the reviewer for the thoughtful review and are glad that the problem is found well-motivated and important for the digital humanities, and that the effort in the dataset construction is recognised. The review led to new experiments, summarised below. All changes are highlighted in blue in the revised PDF.

W1/Q1. Real historical photographs. We fully agree. We had started such an evaluation before the submission, but it was not ready in time, and the revised paper now includes it (Sec. 5.2 "Proof-of-concept on real photographs", Sec. 6 "Results on real photographs", App. J). We collected 1,972 photographs of eight benchmark topics from the community archive Topothek and from Wikimedia Commons, kept only those whose metadata or description gives the date they were taken (not the upload date), and checked with CLIP and by hand that each shows its topic. A CSV file with the source link, topic, and period of every photograph is part of the anonymous dataset release (https://anonymousauthor287.github.io/projectpage/). Remarkably, the periods found in real photographs match those found in synthetic images of the same topics well. Compared with these synthetic images (Table 5):

  • The baselines work on real photographs. Grouping stays close to the synthetic results: ARI is 0.44 against 0.52 for Open-MLLM, and 0.49 against 0.63 for TrajEx.
  • The ranking holds. TrajEx groups better than Open-MLLM on both synthetic and real images.
  • The documented periods appear in real photographs, for example the camera's path from studio portraits to smartphones (Table 38). This indicates that VDBench captures meaningful diachronic shifts.

W2/Q2. Reliance on temporal information. Thank you for this observation and question, which we address in three ways.

The year-based rule. We added the rule proposed by the reviewer to the paper as a "naive baseline" that we refer to as "equal-count year binning" (ECYB-5) (Sec. 5.2 and App. H.5, "Equal-count year binning"). It scores well because 35 of the 41 topics have five periods and the number of images is balanced across periods. On topics with four or six periods, TrajEx reaches ARI 0.93 and 0.91, against 0.64 for the rule (Table 24). On three released subsets of our data with imbalanced periods, the rule drops from 0.95 to 0.69 on average, while TrajEx stays at 0.74 for all subsets (App. H.12, Table 25). To account for the reviewer's remark we have added this rule as a baseline for the grouping task.

How much visual understanding is actually needed to perform the grouping task This is a very valuable question of the reviewer. We tried to answer this question in our revision, see App. H.9 "Removing dates", Table 32). We removed every year from the image records, the prompts, and the captions. Grouping drops sharply (Open-MLLM ARI 0.56 to 0.21), but from image content alone Open-MLLM still groups better than chance, never groups a whole topic into only one period, and orders most periods correctly (Kendall's τ of 0.74). We can thus conclude that visual undertanding is essential for the grouping task and the task can be solved to a certain degree even without estimates for years.

Remark: Dating is itself visual. Without ground truth, the years are estimated from the visual content in Task 1. TrajEx uses the temporal projection of this work, which showed that vision-language models encode time in their visual representations. Following this method, the estimated temporal information is itself a product of visual understanding. Thus, using estimated years for the grouping task implicitly considers the visual content of the images.

W3. Evaluation of historical meaning. We agree. BERTScore and MoverScore measure similarity to reference texts, not historical correctness, which we discuss as a limitation in Sec. 7. Whether the historical changes are identified correctly can still be measured by comparing the grouping, period count, order, and ranges with the ground truth. A human evaluation of the descriptions would be valuable, and we plan this in future work. Furthermore, we currently run an experiment to evaluate how much historical knowledge comes from the model itself. Experiments are currently ongoing and we will add it to the appendix of an updated PDF as soon as it has finished.

W4. Clarity and presentation. Thank you for this feedback. Following it, the introduction now states what the benchmark is for and defines both tasks, what a method receives and what it must output, before the dataset construction (Sec. 1).

We hope our answers address the reviewer's concerns well and would be grateful if this were reflected in the final assessment.

Reviewer_tGUwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: Motivated by linguistic diachronic analysis, this paper proposes a new computer vision task named visual diachronic analysis to discover historical changes in visual context and meaning from images. Based on the definition of visual diachronic analysis , this paper further proposes two tasks: visual context detection and topic and trajectory discovery. Then, the VDBench benchmark is build to evaluate these two tasks conducted by various methods. VDBench contains 20,100 images across 41 topics and 201 periods.

Critical strengths
  1. The task of visual diachronic analysis is novel and valuable. It is a reasonable extntention from single language to multi-modal domain.
  2. This paper proposes two visual diachronic analysis tasks: visual context detection and topic and trajectory discovery. These two tasks demonstrate the practical value of visual diachronic analysis. Further, this paper builds the VDBench. The problem of data scarcity in the field of visual diachronic analysis was addressed through manual curation and image data synthesis.
  3. This paper provides a relatively complete description and introduction of the data construction pipeline, which facilitates reproduction and analysis
Critical weaknesses
  1. The evaluation baselines are quite not comprehensive. To demonstrate the value of both visual diachronic analysis tasks and benchmarks, more frontier LLM such as GPT 6 astra , opus 5.5, gemini4, kimi k3, GLM 5.3 are necessary to be evaluated.
  2. Some concerned about the dataset of the VDA bench. Why is the number of proposed topic 266 and is there any proof to support this? As for the quality of topic, trajectory and synthesized images are only partially sampled to verify and this make me suspect the quality of VDA bench. In the case of benchmarks where the data volume is not large, it is best to conduct a full manual inspection of all the data.
Questions for authors

Providing more baselines and high quality dataset can further increase the value of this paper.

1 comment
AuthorsOpenReview ↗

Response to Reviewer tGUw

We sincerely thank the reviewer for the positive assessment of the task, the benchmark design, and the documentation of the construction pipeline, and for the concrete suggestions, which we followed. All changes are highlighted in blue in the revised PDF.

W1. More frontier models. Thank you for this valuable suggestion, which we have taken up and added to the paper (Sec. 6, App. K). We evaluated seven frontier models, including all five you suggested: GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon, Kimi K3.1 Pro, and GLM 5.3 FlashX, plus GPT-6.1 Sol and GPT-6 Luna. Each model replaces Qwen3-VL in Open-MLLM's Task 2 pipeline and also receives every image. Since a full run on all topics would cost about USD 4,000, most of it for Task 1, which needs one call per image, we focused on Task 2 for the two topics where our open-weight baselines do best and the two where they do worst (Table 41, Fig. 12). We used synthetic images only: our real archival photographs show identifiable people, so for privacy reasons they are processed only on our own hardware and never sent to proprietary APIs. Running open-weight frontier models locally would avoid this restriction, but requires substantial compute. The results are as follows:

  • Baselines' best two topics. The frontier models perform very well and match our best baseline, TrajEx, which already groups these topics almost perfectly (ARI 0.88 to 1.00 against 0.99).
  • Baselines' worst two topics. All seven models group clearly better than both baselines (ARI 0.69 to 0.98 against 0.45 and 0.52), but most are still far from a perfect grouping, and their period and topic descriptions score only slightly higher than the baselines'. Even for frontier models, these topics remain far from solved.

W2. Number of topics and data quality. Thank you for raising this. The 266 topics are the candidates proposed by our LLM-assisted literature search, which we then filtered in several steps (Sec. 3, App. A.1, Table 6):

  • Source audit. We manually checked the proposed claims against their sources and kept only the 148 topics whose periods are supported by verified sources, with 706 period descriptions and 174 verified sources (App. A.3).
  • Expert review of every benchmark topic. Six domain experts from the humanities and social sciences (App. D, Table 11) reviewed the trajectories. All 41 benchmark topics were approved by a majority of their reviewers, topics with a tied vote were left out, and the experts corrected period boundaries and descriptions where needed. They rated the period descriptions 4.02 out of 5 on average (Sec. 4, Table 2, App. D.2).
  • Verification of every image. All 20,100 images are checked after generation: CLIP ranks the topic, the requested environment (out of 669 labels), and the activity (out of 177), and every image that does not achieve top ranks is regenerated (Sec. 3, App. B.3). In addition, experts judged 422 of 501 randomly chosen images (84%) plausible, and where two experts judged the same image, they agreed in 72% of the cases (Table 2).
  • Validity on real photographs. The documented periods also appear in 1,972 real archival photographs of eight topics, for example the camera's path from studio portraits to smartphones, which shows that VDBench captures meaningful diachronic shifts (Sec. 6, App. J, Table 38).

We agree that a full manual inspection of all images would be ideal. For this release, we therefore combined automatic checks of every image with expert sampling, and we plan to extend the expert review in future releases.

We hope our answers address the reviewer's concerns well and would be grateful if this were reflected in the final assessment.

Reviewer_1PHsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Visual Diachronic Analysis (VDA), which asks whether AI can identify how the context and social meaning of the same subject change across images from different historical periods. For example, women wearing trousers might initially represent a challenge to tradition, later serve practical needs during wartime, and eventually become everyday attire. Because real historical images lack comprehensive annotations, the authors use LLM-assisted literature searches, followed by manual verification and expert review, to establish historical periods for each topic. They then generate images based on descriptions of these periods. The resulting benchmark, VDBench, contains 20,100 synthetic images covering 41 topics and 201 periods. The evaluation consists of two tasks:

  • Task 1: Understanding individual images. Predict the year, environment, and activity, and describe the image content.
  • Task 2: Identifying changes across an image collection over time. Given images belonging to the same topic and information from Task 1, identify the topic, divide the collection into periods, and explain the historical meaning of each period.

The authors evaluate two baseline methods: Open-MLLM uses Qwen3-VL directly to perform the tasks, while TrajEx uses separate components for year estimation, temporal segmentation, and text generation. They also analyze how different inputs and evaluation metrics affect the results.

Critical strengths
  1. The research question is meaningful and has value across fields. Studying how the context and meaning of the same subject change over time is an interesting direction beyond recognizing objects in images. It could be useful for digital humanities and historical image research.

  2. The dataset construction includes source verification and expert review. The authors check whether the sources support the historical claims, and experts review the topics, periods, and descriptions, with opportunities to revise the period boundaries. This makes the data more credible than relying only on LLM-generated historical accounts.

Critical weaknesses
  1. The synthetic images may look historically plausible, but this does not show that the benchmark can measure the ability to analyze real historical images.
  2. Simple year-based rules can already solve much of the grouping problem in Task 2.
  3. Historical meaning, a key part of the paper’s contribution, is not yet evaluated reliably.
  4. The annotations mainly describe what the images were intended to show, without enough checks of what they actually show.
  5. Expert review still has limited coverage and agreement.

Overall, as a benchmark, not enough evidence in the score faithfully reflecting the ability to be evaluated.

Questions for authors
  1. Could the authors provide a small set of real historical images with independent annotations to check whether model performance and rankings are consistent across synthetic and real data?

  2. Since a simple year-based rule with five fixed groups already outperforms the main baselines on grouping, how would the authors adjust the evaluation to ensure that high scores require understanding changes in visual context, rather than mainly relying on years and regularities in sample counts?

  3. How much of the generated historical descriptions comes from the input images, and how much comes from the model’s existing historical knowledge? Could the authors add a comparison that generates descriptions using only the topic and years, without visual context information?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1PHs

We sincerely thank the reviewer for the careful, constructive review and are glad that the research question and the source-verified construction are found valuable. Your questions led to new analyses, summarised below. All changes are highlighted in blue in the revised PDF.

W1 / Q1. Real historic images. We fully agree. We had been working on an evaluation with real photographs before the submission, but it was not ready in time, and the revised paper now includes it (Sec. 5.2, Sec. 6, App. J). We collected 1,972 photographs for eight benchmark topics from the community archive Topothek and from Wikimedia Commons. We kept only photographs whose metadata or description gives the date they were taken (not the upload date) and checked with CLIP and by hand that each is relevant to its topic. A CSV file with the source link of every photograph and its topic and period is already part of the anonymous dataset release (https://anonymousauthor287.github.io/projectpage/). We compare with synthetic images of the same topics and periods in Table 5. Remarkably, the periods found in real images match those found in synthetic ones well, which shows that VDBench captures meaningful diachronic topics and meaning shifts. See Sec. 6 and App. J for details. In short:

  • The baselines work on real photographs. Grouping stays close to the synthetic results: ARI is 0.44 against 0.52 for Open-MLLM, and 0.49 against 0.63 for TrajEx. The period and topic texts score within 0.02 BERTScore of the synthetic ones.
  • The ranking holds. TrajEx groups better than Open-MLLM on both synthetic and real images. Dating real photographs is harder, with a year error two to three years higher, and here the two baselines are close (7.4 and 8.2 years).
  • The documented periods appear in real photographs, for example the camera's path from studio portraits to smartphones (Table 38).

W2 / Q2. Year-based rules. Thank you for this suggestion, which we found valuable and have added to the paper as a new baseline that we call equal-count year binning (ECYB-5): it sorts a topic's images by year and cuts them into five equal groups (Sec. 5.2, 6, App. H.5). The idea is sound, and the rule scores well on VDBench because 35 of the 41 topics have five periods and the number of images is balanced across periods. On imbalanced versions of VDBench and on topics with other numbers of periods, the rule falls behind TrajEx. On topics with 4 or 6 periods, TrajEx reaches ARI 0.93 and 0.91, while the rule reaches 0.64 (Table 24). We agree that the balanced nature of VDBench introduces a bias that can be exploited by such a rule. As soon as we extend VDBench and add more real archive photographs, the imbalance will automatically increase and the performance of the simple rule will drop further.

To assess the sensitivity of the benchmark to such a rule-based approach, we release three subsets that keep 70–100% or 40–100% of each period's images at random, or a share that grows from 70% to 100% over time (App. H.12). For these subsets, the performance of the rule drops from 0.95 to 0.69 on average, while TrajEx stays at 0.74 (Table 25).

W3 / Q3. Historical meaning and model knowledge. Thank you for this question. We agree that BERTScore and MoverScore measure agreement with reference texts rather than historical correctness, a limitation we discuss in Sec. 7. In VDA, a method never receives the topic name or the years as input: it starts from the images alone, estimates each image's year and visual context in Task 1, and derives the topic and periods (including descriptions) from the images in Task 2.

Generating descriptions from the topic name and years alone would show how much historical information comes from the model itself. The experiment is running but won’t finish in time, we plan to add the results to the camera-ready version.

W4. Intended versus actual content. All 20,100 images are checked after generation: We run a comprehensive CLIP-based verification on each image. Thereby we rank the topic, the requested environment (out of 669 labels) and the activity (out of 177) and regenerate each image which does not achieve top ranks (Sec. 3, App. B.3). In addition, experts judged 422 of 501 randomly chosen images (84%) plausible (Table 2).

W5. Expert coverage and agreement. Six domain experts from different social science disciplines (App. D, Table 11), the trajectories. All 41 benchmark topics were approved by a majority of the reviewers, and topics with a tied vote were left out. The experts rated the period descriptions 4.02 out of 5 on average and corrected period boundaries where needed (Sec. 4, Table 2, App. D.2). Where two experts judged the same image, they agreed in 71% of the cases. We strive to extend the coverage of expert reviews in future extensions of VDBench.

We hope that our answers and additions well address the reviewer's concerns, and we would be grateful if they were reflected in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.