- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Visual Diachronic Analysis (VDA), a new computer vision task that aims to identify how the visual context and social meaning of subjects change over time. The authors further introduce VDBench, a benchmark which contains 41 historical topics, 201 historical periods, and 20,100 synthetically generated images. The authors propose two evaluation tasks: (1) visual context detection, which involves predicting the year, environment, activity, and caption of the generated images, and (2) topic and trajectory discovery, which aims to describe how a subject's representation changes over time. They find that reconstructing historical trajectories remains challenging, particularly when relying on predicted image metadata rather than ground-truth information.
Critical strengths
- I find the problem well-motivated and important for the community of digital humanities.
- It is apparent that the authors have put considerable effort and thought into the dataset construction, including LLM-based image generation and expert validation.
Critical weaknesses
-
Use of synthetic images: My main concern is whether the benchmark actually evaluates the ability to discover historical changes from images. The authors first identify historical changes from existing sources and then instruct an image-generation model to represent the meaning of each period. While I understand the motivation for using synthetic data, I am not convinced that good performance on this benchmark would necessarily translate to real historical photographs. I think an evaluation on at least a small collection of real images would strengthen this contribution.
-
Reliance on temporal information: Task 2 aims to identify how visual representations change over time. However, the results in app. h.5 show that sorting images by year and dividing them into periods can achieve better image-grouping results than the evaluated baselines. This raises the question of how much visual understanding is actually needed to perform the grouping task. I would have liked to see stronger evidence that the models identify changes in visual content rather than mainly relying on temporal information.
-
Evaluation of historical meaning: The authors evaluate the generated descriptions using BERTScore and MoverScore, which measure similarity to reference texts rather than historical correctness. The analysis in app. h.3 shows that these metrics have limited ability to distinguish descriptions of different periods within the same topic. This makes it difficult to determine whether the models have correctly interpreted the historical changes. I think a human evaluation of the generated descriptions would be valuable here.
-
Clarity and presentation: In general, I found the paper difficult to follow, especially when trying to understand the overall goal of the benchmark and its intended use. The paper introduces many concepts and technical details about dataset construction before clearly explaining what the models are expected to do.
Questions for authors
- Could the authors evaluate their methods on a small collection of real historical photographs to test whether the results generalise?
- Could the authors provide an evaluation where models must identify historical periods without access to year information?
Response to Reviewer UHfP
We sincerely thank the reviewer for the thoughtful review and are glad that the problem is found well-motivated and important for the digital humanities, and that the effort in the dataset construction is recognised. The review led to new experiments, summarised below. All changes are highlighted in blue in the revised PDF.
W1/Q1. Real historical photographs. We fully agree. We had started such an evaluation before the submission, but it was not ready in time, and the revised paper now includes it (Sec. 5.2 "Proof-of-concept on real photographs", Sec. 6 "Results on real photographs", App. J). We collected 1,972 photographs of eight benchmark topics from the community archive Topothek and from Wikimedia Commons, kept only those whose metadata or description gives the date they were taken (not the upload date), and checked with CLIP and by hand that each shows its topic. A CSV file with the source link, topic, and period of every photograph is part of the anonymous dataset release (https://anonymousauthor287.github.io/projectpage/). Remarkably, the periods found in real photographs match those found in synthetic images of the same topics well. Compared with these synthetic images (Table 5):
W2/Q2. Reliance on temporal information. Thank you for this observation and question, which we address in three ways.
The year-based rule. We added the rule proposed by the reviewer to the paper as a "naive baseline" that we refer to as "equal-count year binning" (ECYB-5) (Sec. 5.2 and App. H.5, "Equal-count year binning"). It scores well because 35 of the 41 topics have five periods and the number of images is balanced across periods. On topics with four or six periods, TrajEx reaches ARI 0.93 and 0.91, against 0.64 for the rule (Table 24). On three released subsets of our data with imbalanced periods, the rule drops from 0.95 to 0.69 on average, while TrajEx stays at 0.74 for all subsets (App. H.12, Table 25). To account for the reviewer's remark we have added this rule as a baseline for the grouping task.
How much visual understanding is actually needed to perform the grouping task This is a very valuable question of the reviewer. We tried to answer this question in our revision, see App. H.9 "Removing dates", Table 32). We removed every year from the image records, the prompts, and the captions. Grouping drops sharply (Open-MLLM ARI 0.56 to 0.21), but from image content alone Open-MLLM still groups better than chance, never groups a whole topic into only one period, and orders most periods correctly (Kendall's τ of 0.74). We can thus conclude that visual undertanding is essential for the grouping task and the task can be solved to a certain degree even without estimates for years.
Remark: Dating is itself visual. Without ground truth, the years are estimated from the visual content in Task 1. TrajEx uses the temporal projection of this work, which showed that vision-language models encode time in their visual representations. Following this method, the estimated temporal information is itself a product of visual understanding. Thus, using estimated years for the grouping task implicitly considers the visual content of the images.
W3. Evaluation of historical meaning. We agree. BERTScore and MoverScore measure similarity to reference texts, not historical correctness, which we discuss as a limitation in Sec. 7. Whether the historical changes are identified correctly can still be measured by comparing the grouping, period count, order, and ranges with the ground truth. A human evaluation of the descriptions would be valuable, and we plan this in future work. Furthermore, we currently run an experiment to evaluate how much historical knowledge comes from the model itself. Experiments are currently ongoing and we will add it to the appendix of an updated PDF as soon as it has finished.
W4. Clarity and presentation. Thank you for this feedback. Following it, the introduction now states what the benchmark is for and defines both tasks, what a method receives and what it must output, before the dataset construction (Sec. 1).
We hope our answers address the reviewer's concerns well and would be grateful if this were reflected in the final assessment.