- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates why VLMs fail at reading analogue clocks. The authors evaluate four open-weight models (Qwen3-VL-8B, Qwen3.5-9B, Molmo2-8B, and Llama-3.2-11B-Vision) and GPT-5.5. The paper's main claim is that the bottleneck in clock reading is not the presentation of visual geometry because hand angles are predictable with linear probing. The main problem is the conversion of that geometric information into a correct numerical readout.
Critical strengths
The aim to perform a prehensive diagnostic framework is very important for the community. The paper takes the first effort to do such study. Some conclusions are beneficial for future works.
Critical weaknesses
Poor writing and presentation
-
The paper suffers from substantial issues in writing, organization, and presentation, making it difficult to follow the experimental designs, how to understand the results and draw the key findings.
-
Important explanation, claims and supporting analyses are frequently deferred to the appendix rather than being established in the main paper. These issues go beyond writing style: they substantially undermine the accessibility of the work and make it difficult to assess the significance and validity of its findings. For example, section 3 introduces four datasets/experiment settings that are key the study. But it's quite hard for me to understand what they are. Representative examples and important experimental details are deferred to the ~40 pages appendix, making it difficult to understand the subsequent analyses without repeatedly consulting supplementary material.
-
Section 4.2 is quite hard to follow. Especially, table 2 is not easy to comprehend. The authors should make the paper more readable for audiences.
-
Several arguments lack clear logical transitions. For example, the introduction states the research question as "what clock-relevant information current VLMs already represent, and at which stage between that representation and the answer the reading fails," and then argues that "answering this question requires a setting in which the relevant geometry can be specified precisely and varied independently of appearance. Clocks meet both requirements." Because the question is already defined in terms of clocks, justifying clocks as the setting is circular.
-
Insufficient evidence for some claims. For example, in Section 5, the authors conclude that poor clock reading cannot be explained by a loss of geometric information. However, the reported probing errors correspond to approximately 6–9 minute marks, suggesting that geometric representations may still be too imprecise for accurate clock reading.
My overall view
In general, I feel the direction of this work is good. They authors should delivery it better for audiences to understand and make the study more rigorous so that all the claims and conclusions are better support by the experiment results. The current form of this paper is quite under the ICLR standard.
Questions for authors
N/A
We thank you for the detailed feedback on our submission. In fact, receiving feedback on writing is especially valuable, as it helps us follow the principle of “progress before perfection” in improving how we communicate our findings.
Following this feedback, we have already implemented the following changes to the writing and presentation and uploaded a revised version (also making use of the one extra page allowed by the guidelines):
We hope you have a chance to review the changes in the revised version and that they help communicate our contributions more clearly. We will continue improving the presentation and writing in the final version, if accepted.
We have mostly addressed the last point about geometric information in our response to Reviewer 1, Point 1. Just to reiterate, we agree that the angular decoding errors in the base model remain substantial, and that insufficient geometric precision may contribute to incorrect readings. However, some evidence from the further analyses suggests that geometric precision alone does not fully explain the failures. In particular, gains in geometric decodability do not consistently translate into gains in reading accuracy (Figures 5 and 25), and even substantially more precise needle-angle decodability for other instruments does not translate into comparably high numerical reading accuracy (Figure 30). To investigate this question further, we also added a sample-wise analysis (Figure 26), comparing angular probe errors with the corresponding reading errors for each clock hand. We find generally weak and inconsistent correlations, meaning that more precisely decodable hand geometry does not necessarily lead to a more accurate reading, nor does a more accurate reading necessarily imply more precisely decodable hand geometry.