acceptodds
Under review as a conference paper at ICLR 2027

Reading Between the Hands: Diagnosing Visual Measurement in Vision-Language Models

Abstract

Vision-language models can solve increasingly complex multimodal tasks yet remain unreliable at reading analogue dial instruments. Clocks provide a familiar and highly controlled testbed for studying how fine-grained visual geometry is transformed into numerical readings. Rather than optimising benchmark performance, we ask *when* and *where* models fail at clock reading. Across four VLMs from the Qwen, Molmo and Llama families with distinct multimodal architectures, we combine controlled visual perturbations, equivalent text-based representations of hand geometry, test-time reasoning, component-wise supervised fine-tuning, and layer-wise probes of hour-, minute-, and second-hand geometry. Taken together, these analyses highlight three important aspects of the problem. First, clock reading does not behave like a simple perceive-then-compute pipeline: models can perform substantially better from images than when ground-truth hand positions are supplied explicitly in text, while additional reasoning can recover the text-based mapping but degrade image-based reading. Second, hand geometry is nevertheless linearly recoverable across the visual and language pathways, showing that the availability of geometric information does not guarantee its successful downstream use. Third, there is no universal repair point: the most effective adaptation target differs across architectures, and large improvements in clock reading rarely transfer to eight related dial instruments. Together, these results show that improved task performance, accessible geometric representations, and transferable measurement ability are distinct properties. We therefore characterise clock reading as an architecture-dependent problem of coordinating geometric perception with numerical readout, rather than as a single missing perceptual or reasoning capability.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, visual reasoning, analogue clocks, dial instruments, representation analysis, linear probing, supervised fine-tuning, cross-task generalisation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_qNvyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates why VLMs fail at reading analogue clocks. The authors evaluate four open-weight models (Qwen3-VL-8B, Qwen3.5-9B, Molmo2-8B, and Llama-3.2-11B-Vision) and GPT-5.5. The paper's main claim is that the bottleneck in clock reading is not the presentation of visual geometry because hand angles are predictable with linear probing. The main problem is the conversion of that geometric information into a correct numerical readout.

Critical strengths

The aim to perform a prehensive diagnostic framework is very important for the community. The paper takes the first effort to do such study. Some conclusions are beneficial for future works.

Critical weaknesses

Poor writing and presentation

  1. The paper suffers from substantial issues in writing, organization, and presentation, making it difficult to follow the experimental designs, how to understand the results and draw the key findings.

  2. Important explanation, claims and supporting analyses are frequently deferred to the appendix rather than being established in the main paper. These issues go beyond writing style: they substantially undermine the accessibility of the work and make it difficult to assess the significance and validity of its findings. For example, section 3 introduces four datasets/experiment settings that are key the study. But it's quite hard for me to understand what they are. Representative examples and important experimental details are deferred to the ~40 pages appendix, making it difficult to understand the subsequent analyses without repeatedly consulting supplementary material.

  3. Section 4.2 is quite hard to follow. Especially, table 2 is not easy to comprehend. The authors should make the paper more readable for audiences.

  4. Several arguments lack clear logical transitions. For example, the introduction states the research question as "what clock-relevant information current VLMs already represent, and at which stage between that representation and the answer the reading fails," and then argues that "answering this question requires a setting in which the relevant geometry can be specified precisely and varied independently of appearance. Clocks meet both requirements." Because the question is already defined in terms of clocks, justifying clocks as the setting is circular.

  5. Insufficient evidence for some claims. For example, in Section 5, the authors conclude that poor clock reading cannot be explained by a loss of geometric information. However, the reported probing errors correspond to approximately 6–9 minute marks, suggesting that geometric representations may still be too imprecise for accurate clock reading.

My overall view

In general, I feel the direction of this work is good. They authors should delivery it better for audiences to understand and make the study more rigorous so that all the claims and conclusions are better support by the experiment results. The current form of this paper is quite under the ICLR standard.

Questions for authors

N/A

1 comment

We thank you for the detailed feedback on our submission. In fact, receiving feedback on writing is especially valuable, as it helps us follow the principle of “progress before perfection” in improving how we communicate our findings.

W1 - W4: Poor writing and presentation

Following this feedback, we have already implemented the following changes to the writing and presentation and uploaded a revised version (also making use of the one extra page allowed by the guidelines):

  • We added a summarising table of the datasets in Section 3 to make it easier to understand them at a glance and see what our newly introduced datasets contribute.
  • We rewrote Section 4.2 and Table 2 to make the motivation, methodology, and interpretation of the results easier to follow.
  • We revised the main research question in the introduction and made the motivation for using clocks as our experimental setting non-circular.

We hope you have a chance to review the changes in the revised version and that they help communicate our contributions more clearly. We will continue improving the presentation and writing in the final version, if accepted.


W5: Imprecise geometric representations not acknowledged as a source of clock reading errors

We have mostly addressed the last point about geometric information in our response to Reviewer 1, Point 1. Just to reiterate, we agree that the angular decoding errors in the base model remain substantial, and that insufficient geometric precision may contribute to incorrect readings. However, some evidence from the further analyses suggests that geometric precision alone does not fully explain the failures. In particular, gains in geometric decodability do not consistently translate into gains in reading accuracy (Figures 5 and 25), and even substantially more precise needle-angle decodability for other instruments does not translate into comparably high numerical reading accuracy (Figure 30). To investigate this question further, we also added a sample-wise analysis (Figure 26), comparing angular probe errors with the corresponding reading errors for each clock hand. We find generally weak and inconsistent correlations, meaning that more precisely decodable hand geometry does not necessarily lead to a more accurate reading, nor does a more accurate reading necessarily imply more precisely decodable hand geometry.

Reviewer_qnacOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Reading Between the Hands investigates why vision-language models (VLMs) struggle to convert fine-grained visual geometry into numerical readings. It uses analogue clocks as a controlled setting in which hand geometry can be specified precisely and varied independently of appearance. Prior work shows that adaptation can improve clock reading, but leaves open what geometric information models encode, how that information influences answers, and which architectural components support learning.

The experiments are thoughtfully designed, but the principal conclusions largely reinforce familiar distinctions between information accessibility, causal use, and task performance. The paper’s additional explanatory contribution remains limited: it characterizes architecture-dependent outcomes without establishing what predicts them or identifying the computation responsible for failure.

Critical strengths

The experimental design provides a thoughtful, multifaceted diagnosis of a concrete visual reasoning failure. Clock reading is a useful controlled setting because hand geometry can be specified precisely while appearance and presentation factors are varied independently. The paper combines matched visual controls, layer-wise geometric probes, hand-specific activation patching in Qwen3-VL, component-specific fine-tuning across four architectures, and transfer tests on other instruments (Sections 4–7). These experiments address complementary questions: which visual changes affect behaviour, where geometry is linearly accessible, which activation states causally influence answers, which parameter updates improve performance, and whether those improvements generalise. Their combination helps distinguish representational accessibility, causal use, and adaptation effects, avoiding reliance on aggregate accuracy or probing alone. The cross-instrument evaluation also tests the scope of the learned capability rather than assuming that improved clock reading constitutes general visual measurement. Although the main causal evidence remains specific to clock reading, the diagnostic workflow offers a useful template for investigating other fine-grained visual reasoning tasks and motivates testable hypotheses about geometry-to-readout conversion.

Critical weaknesses

The diagnostic findings have limited demonstrated predictive or transferable value for model design. Component-specific fine-tuning identifies different effective update targets across architectures, but these targets are established through empirical comparison rather than predicted from the preceding diagnosis. Moreover, the paper finds that gains in geometric decodability do not reliably predict accuracy gains, leaving the connection between its representation analyses and effective interventions unresolved (Section 6.1; Figures 4–5). The study does not demonstrate that its diagnostics can select adaptation targets on an unseen architecture or reduce search cost. Substantial clock-reading improvements also rarely transfer to other dial instruments, so the proposed geometry-to-readout bottleneck has yet to yield a validated intervention for broader visual measurement (Section 7). These observations offer useful descriptive and mechanistic evidence, but the step from diagnosis to predictive, transferable design principles remains insufficiently established. A general improvement algorithm is not required for a valuable diagnostic contribution; however, the present evidence leaves its practical guidance largely dependent on model-specific experimentation.

Questions for authors

What scientific or practical value does the mechanistic diagnosis provide beyond documenting architecture-dependent adaptation outcomes? The absence of an explanation for a model's behaviour does not by itself establish the value of obtaining that explanation, and effective model improvement need not require mechanistic understanding. Could the authors clarify which falsifiable predictions or actionable decisions their analyses enable? For example, can diagnostics performed before fine-tuning predict effective adaptation targets on a held-out model and reduce search cost relative to empirical component selection, accounting for the cost of diagnosis? Alternatively, can the proposed geometry-to-readout bottleneck motivate an intervention that improves cross-instrument generalisation? The finding that geometric decodability gains do not reliably predict accuracy gains makes the connection between diagnosis and intervention particularly important. If the intended contribution is primarily scientific understanding, could the authors specify which competing explanations their evidence distinguishes, and which claims remain hypotheses requiring further causal tests? A general improvement algorithm is not necessary for a valuable diagnostic study, but its explanatory or decision-making benefit should be established beyond post-hoc descriptions of which components work best for each model.

1 comment

First of all, we would like to thank you for the thoughtful feedback and for precisely understanding the motivation of our work and the questions that previous studies have left unanswered.

W. Predictability of the adaptation targets

We really think that predicting the most effective component to fine-tune is a very interesting research question. We initially hypothesised that geometric information decodability could be a predictor of the most effective adaptation target. However, our experiments revealed something different: the component where geometric information is least decodable is not necessarily the component whose fine-tuning gives the largest improvement in downstream performance. We agree that predicting these adaptation targets remains an important open question (including whether they can be predicted in general), and following this discussion, we will highlight this distinction more explicitly in the final version.


Q. Practical value

Beyond the question of predicting effective adaptation targets, our causal analysis also gives rise to a concrete, testable hypothesis. We observed that the causal influence of hand information at the final prompt token depends on which answer component is generated next (Figure 4). This suggests that changing the order of hour, minute, and second generation should also change where the corresponding hand information becomes causally influential. While this remains to be tested, it is one example of how our diagnosis can motivate predictions about the internal mechanisms of visual reading.

We are also very glad that you pointed out that a general improvement algorithm is not a requirement for a diagnostic contribution like ours. We would be happy if our findings could motivate further research into the connection between representational diagnosis and effective adaptation.

Reviewer_rkLROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates clock-reading failures in vision-language models through controlled experiments, probing, and fine-tuning, and evaluates whether improvements transfer to other dial-reading tasks.

Critical strengths

The paper provides a systematic evaluation of clock reading across multiple vision-language models, supported by controlled datasets and complementary analyses.

Critical weaknesses

The main findings largely reflect standard expectations in machine learning, leaving their additional scientific contribution unclear.

The authors report that clock-specific fine-tuning improves clock reading, but this is an expected benefit of task-specific supervision. Their observation that optimal tuning components differ across models is likewise unsurprising given architectural differences. Limited transfer to other instruments is consistent with the familiar principle that training on one task does not guarantee improvement on another. Similarly, finding decodable geometric information in pretrained visual representations is not, by itself, a substantial conceptual advance.

The paper therefore primarily quantifies familiar patterns. To establish a clearer contribution, the authors should explain how these findings refine existing understanding, rule out plausible competing explanations, or enable new predictions.

Questions for authors

See Weaknesses.

3 comments

We thank you for the feedback and for raising points that challenge us to position our work and results more critically against the current knowledge and literature. In this light:

W1. Architecture-dependent fine-tuning: surprising or not?

Prior work (Fu et al. [1]) concludes that language-model fine-tuning produces the largest improvements across the vision-centric tasks they investigated. In contrast to this, our results show that the language decoder is not universally the most effective adaptation target (for example, Molmo2 benefits most from vision-only adaptation, while Qwen3.5 achieves substantial improvements through projector-only fine-tuning (Table 20)), and that adding more trainable components does not necessarily improve performance.


W2. Limited transfer to other instruments: expected or not?

Initially, when approaching this problem, we had multiple hypotheses about what could make clock reading difficult. One of them was that models might struggle with identifying pointer positions relative to a scale. If this were the main problem, we could expect that by teaching the model clock reading, it would learn this operation and that the ability could transfer to other instruments.

However, our empirical results show that such broad transfer does not happen, even though we are dealing with the same family of dial instruments. In fact, prior work (Sachdeva et al. [2]; Che et al. [3]) also highlights that transfer between visual perception capabilities is an empirical question, rather than something we should simply dismiss as unlikely.


W3. Decodable geometry in pretrained representations: should it be present or not?

We agree with your point that simply showing that geometric information about the clock hands is present in visual representations is not, by itself, novel. Before getting to the point that our goal was not just to show its presence, we would like to highlight that prior work (Tong et al. [4]; Li et al. [5]) has shown that not all the necessary visual information is always preserved as models undergo multiple stages of training and multimodal processing. Apart from showing that this information is present, we also show how it evolves (getting lost or, in contrast, sharpened) across VLM components, how fine-tuning changes its accessibility, and whether the model causally uses it to generate the answer. In particular, the hour/minute/second distinction in our activation-patching results adds task-specific detail to earlier work on visual-information flow.

We have also revised the manuscript to make these distinctions clearer. In particular, we strengthened the comparison with prior work on component-specific fine-tuning in Section 6 and clarified our motivation for testing transfer between dial instruments in Section 7.1. We will continue improving the discussion of these implications in the final version.


References

[1] Fu, S., Bonnen, T., Guillory, D., & Darrell, T. (2025). Hidden in Plain Sight: VLMs Overlook Their Visual Representations. Second Conference on Language Modeling (COLM). OpenReview.

[2] Sachdeva, B., Uppal, K., Java, A., & Balasubramanian, V. N. (2025). Understanding Task Transfer in Vision-Language Models. UniReps: 3rd Edition of the Workshop on Unifying Representations in Neural Models. OpenReview.

[3] Che, L., Xue, Z., Quan, Y., Liu, B., Shi, Z., Hurst, M., Feldman, J., Tang, R., Krishna, R., & Pavlovic, V. (2026). Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Models. arXiv:2603.18523. arXiv.

[4] Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024). Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. arXiv:2401.06209. arXiv.

[5] Li, W., Tang, R., Li, C., Zhang, C., Vulić, I., & Søgaard, A. (2025). Lost in Embeddings: Information Loss in Vision-Language Models. September 2025. Microsoft Research.

Thank you for the clarifications and revisions. However, I do not think they fully address my concerns.

W1: My concern is that models pretrained on different data with different training procedures can benefit differently from fine-tuning strategies on a particular downstream task (e.g., clock reading) is an expected outcome in deep learning, and similar empirical findings have long been documented in numerous VLM studies. For example, Frozen [1] and Flamingo [2] reported better performance when the language model remained frozen, while Zhao et al. [3, Table 5] found that vision-only adaptation outperformed language-only adaptation and that updating both components reduced performance. Fu et al. [4], meanwhile, reported what worked best within their evaluated settings—not that language-model tuning must be optimal everywhere. Although the authors provide adaptation experiments on specific models and report empirical findings, the underlying observation is consistent with what we would already expect.

[1] Tsimpoukelli et al. **Multimodal Few-Shot Learning with Frozen Language Models. NeurIPS 2021. See §4.1–4.2, Tables 1–2.

[2] Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022. See §3.3, Table 3, row viii.

[3] Zhao et al. Distilling Vision-Language Models on Millions of Videos. CVPR 2024. See §5.4, Table 5.

[4] Fu et al. Hidden in Plain Sight: VLMs Overlook Their Visual Representations. COLM 2025. See §4.3, Figure 7.

W2: My concern is that these experiments largely illustrate a familiar observation: similarities between tasks do not guarantee that training on one will improve the other. Tasks that appear similar to humans can still differ in their input distributions, so training on one may not help much with the other. Although the authors now provide more specific experiments in the clock-reading setting, motivated by the possibility that shared pointer-localization skills might transfer to other instruments, observing that this transfer is limited does not, by itself, offer a new insight. The result still amounts to the familiar point that training on one task does not necessarily improve performance on another.

W3: I agree that these analyses tell us more than just whether geometric information is present. This partly addresses my concern. These analyses do reveal some new patterns. However, it remains unclear what these patterns mean and how they could help us understand or improve model performance in practice.

Dear Reviewer rkLR,

Thank you very much for following up, especially now that we still have a chance to clarify further what we see as the contribution of our work.

Without intending to further emphasise the surprisingness of some results (W1 and W2), we will try to give a broader view that combines multiple findings from our work, rather than concentrating on the atomic results.


W1. Architecture-specific adaptation targets

Thank you for pointing to more works and results than initially discussed. In light of these, leaving aside the question of the “novelty” of architecture-dependent, task-specific adaptation targets, our work also tried to see whether this could be connected to the components where geometric information is least available.

As already discussed in our response to Reviewer qnac (W1), we have a negative answer to this hypothesis: the component where hand geometry is least decodable is not necessarily the one whose fine-tuning gives the largest accuracy improvement. We also observe that fine-tuning only the vision encoder can sharpen geometric representations downstream in the frozen language decoder.

So, our contribution here is also to investigate how the effective adaptation targets relate to the geometric information encoded and transformed across the model, which, to our knowledge, was not investigated in the cited prior work (and this was also our motivation for using clocks as a testbed for this, as already pointed out by Reviewer qNvy).


W2. Limited transfer to other instruments

We are also very glad that you noticed the additional insights in our results in your follow-up.

When we obtained the negative result that learning the shared operation of identifying pointer positions relative to a scale through clock reading did not transfer to other dial instruments, we investigated possible explanations for this. For example, our probing results show that needle angles and scale information can remain decodable for other instruments after clock-specific fine-tuning, even when numerical reading accuracy is low or degraded.

Following Reviewer RWnu's suggestions, we additionally checked whether clock fine-tuning caused degenerate clock-formatted predictions, but found that such answers were very rare (Appendix H.2). Our broader spatial evaluations also do not indicate widespread catastrophic forgetting.

Taken together, these individual findings suggest that the lack of transfer is not simply explained by the absence of geometric information or a collapse into clock-formatted answers. They support our interpretation that the conversion learned through clock fine-tuning does not necessarily generalise to instruments with different scales and numerical mappings.


W3. Revealed new patterns and their practical value

Again, we are very glad that you found our prior clarification helpful.

Reiterating and adding to the point about practical value clarified in our response to Reviewer qnac, we think the value of this diagnosis is precisely in understanding how the different stages of visual reading relate to one another, and which explanations for the failures are supported by the experiments. For example, the fact that geometric decodability does not reliably predict either reading accuracy or the best adaptation target tells us that improving or locating decodable geometry alone is not sufficient. We think this is an important distinction when trying to understand where and how to improve VLMs.

Beyond the question of predictability of the effective adaptation targets, our causal analysis also has practical value for further investigation. The analysis suggests that changing the order of numerical reading generation should also change which hand's information becomes causally influential at the final prompt token.

We agree that our work does not provide a clear method or framework for further training VLMs, but as also noted before, the contribution of diagnostic work such as ours is to provide empirical findings and an informational basis for further grounded work.

We hope this alternative view of the findings is helpful, and we will work on highlighting the synergy between the atomic results to present a more coherent picture in the final version.

Reviewer_RWnuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Through controlled datasets, layer-wise representation probing, causal interventions, and component-specific fine-tuning, the authors reveal that the primary bottleneck for vision-language models in clock reading lies in translating geometric information into accurate numerical readings; furthermore, cross-instrument transfer experiments demonstrate that fine-tuning on clock reading primarily enhances clock-specific capabilities rather than generalizable visual measurement skills.

Critical strengths

A major strength of this work is how thoroughly the authors build their diagnostic case. By combining controlled setups, layer-wise probing, activation patching, and targeted fine-tuning, they clearly separate whether the model merely encodes geometric information from whether it actually uses it.

Critical weaknesses

The paper attributes the main bottleneck to 'converting geometric information into numerical readings,' yet linear decodability alone does not guarantee that this information is sufficiently precise or effectively utilized by the model. Given that the base model still exhibits substantial angular errors, the current experiments fall short of cleanly disentangling insufficient perceptual precision from a failure in the conversion mechanism, which somewhat undercuts the certainty of the paper's central claim.

Questions for authors

Since the base model still shows relatively large angular decoding errors for the clock hands, what concrete evidence separates "insufficient geometric precision" from a "failure in geometry-to-value conversion"? Clarifying this directly impacts whether your core bottleneck hypothesis holds up.

The causal activation patching experiments are conducted primarily on Qwen3-VL. Do you have consistent evidence across other architectures? If not, which mechanistic claims should be explicitly scoped to Qwen3-VL?

Could the failure in cross-instrument transfer simply stem from output format bias or catastrophic forgetting induced by clock fine-tuning? Did you run any controls for these factors to back up the claim that the model is learning a "clock-specific conversion"?

4 comments

We thank you for the careful feedback and for highlighting these points where our conclusions can be substantially sharpened. We address the raised points below.

W1 & Q1. Geometric information decodability precision vs. conversion into the correct reading

We agree that geometric information decodability in the base model was not that precise either. This can be a possible source of incorrect final readings, alongside the step of converting this geometric information into numerical readings. Our interpretation of highlighting the conversion step is based on the following evidence:

  • We would like to clarify that by geometric information we mean the visual geometric information about the positions of the hands, which comes encoded from the vision encoder. As we can see from the trajectory of this information through the model in Figure 3, the geometric information is already present in the base model and maintains its relative decodability after entering the language decoder, improving further towards the final layers. After fine-tuning specifically for the clock-reading task, we see that the visual representations in the vision encoder layers do not gain substantially sharper decodability. So, qualitatively, we are getting roughly the same decodability from the vision encoder, while what happens in the language decoder is that it "learns" to sharpen its own representations on top of the visual ones. You can see this improvement in decodability especially in the final three-quarters of the language decoder layers. So, to get better at the visual task, the language decoder learns to "sharpen" the geometric information coming from the vision encoder, even when only the vision encoder is fine-tuned.

  • In Figures 5 and 25 we can see that even if we gain a lot in decodability, these gains are not directly converted into clock-reading accuracy. So, what matters is not only how much more decodable the geometric information becomes, but also whether this gain happens in the right place and whether the information is actually used (for example, in the final layers of the language decoder, where it can contribute to generating the correct tokens for precise numerical readings). In the submitted version, we compared decodability gains with accuracy gains. We complement these results by comparing decodability errors with absolute reading errors for each component we examine. As shown in the newly added Figure 26, the correlations between these two errors are generally weak and inconsistent across the hour, minute, and second hands, both before and after fine-tuning (Pearson's rr ranging from −0.261-0.261 to 0.2040.204). This means that having more precisely decodable geometric information for a particular sample does not necessarily lead to a more accurate numerical reading.

  • For the other non-clock dial instruments we investigated (ammeter and pressure gauge), we observed much more precise geometric decodability. However, this much higher precision was never translated into comparably high numerical reading accuracy for these instruments (Figure 28).


Q2. Causal analysis scope

Due to the high computational requirements of this step, yes, the causal analysis was conducted only for Qwen3-VL. We already make this distinction explicit in the abstract:

“In Qwen3-VL, activation patching shows...”

and in Section 5.1. We will further clarify this limitation wherever necessary in the revised manuscript.

Q3. Is this really an output-format issue or catastrophic forgetting with other instruments?

This is a very good question, and thank you for raising it.

To address the possibility of the fine-tuned model giving degenerate answers for all the instruments, we conducted a manual inspection of instrument predictions and did not observe a collapse into clock-formatted answers (either in their format or even in the units). The fact that some instruments, under certain setups and models, also show small improvements (for example, the compass in Table 5) argues against this explanation as well.

To further support the argument that the degradation is predominantly numerical error, not systematic degeneration into clock formatting, we also examined the absolute errors and counted predictions following clock format (H:MM/H:MM:SS) after clock fine-tuning (Qwen3-VL on MeasureBench synthetic set).

InstrumentBase MAE ± SDFine-tuned MAE ± SDClock-formatted answers after SFT
Ammeter9.151 ± 17.40840.002 ± 81.1660
Hygrometer34.760 ± 36.87032.189 ± 34.7480
Pressure gauge77.447 ± 125.750129.149 ± 251.3722/120 (1.7%)
Voltmeter15.420 ± 18.48030.363 ± 19.6440
Weighing scale0.751 ± 1.32126.584 ± 142.4201/60 (1.7%)

As shown, numerical errors increase for most instruments, while clock-formatted answers remain very rare. Further inspection also showed that the numerical predictions remain varied rather than converging to a small set of repeated values.

To share concrete examples of these failures, we enriched Appendix H.2 with representative images where the base model predicts the correct reading but the clock-fine-tuned model does not. We also explicitly address this concern in Section 7.1 of the revised manuscript.

Moreover, to investigate further on catastrophic forgetting, in Section 7.2 we conducted evaluations on broader spatial-reasoning benchmarks. As shown in Table 6, we did not see any symptoms of widespread catastrophic forgetting.

Together with our probing results showing that needle-angle and scale information remain decodable, they support our interpretation that clock fine-tuning does not necessarily teach a conversion procedure that generalises to other dial instruments. We therefore highlight further mechanistic analysis of checkpoints fine-tuned specifically for other instruments as an important direction for future work, to better understand how geometric information is converted into numerical readings.

Thank you for the detailed response and for following up on my questions with additional analyses.

Geometric information and numerical readings. I appreciate the acknowledgment that imprecision in the base model’s representation of hand positions may itself contribute to reading errors. I found the per-example analysis in Figure 26 especially helpful because it directly tests whether the examples with larger probe errors also tend to have larger reading errors. The weak and inconsistent correlations across clock hands, together with the relatively precise angle estimates but poor numerical readings for the ammeter and pressure gauge, support the narrower conclusion that probe precision alone does not explain reading accuracy. How the model uses the available geometric information to produce a numerical answer remains an open mechanistic question.

Scope of the causal analysis. Thank you for confirming that activation patching was performed only on Qwen3-VL. This resolves my question about the scope of the causal evidence. Making that qualification explicit wherever the patching results are discussed will help readers distinguish the findings for Qwen3-VL from the probing results across models.

Transfer to other instruments. Reporting both the absolute reading errors and the frequency of clock-formatted answers addresses my formatting concern directly. For example, the ammeter’s mean absolute error increased from 9.15 to 40.00 after fine-tuning, even though none of its predictions used clock formatting. This makes a systematic switch to clock-formatted answers an unlikely explanation for the observed degradation. The broader spatial-reasoning evaluations also help narrow the interpretation by providing evidence against widespread forgetting, while leaving open which instrument-specific capabilities were affected.

Overall, the additional evidence and the clearer account of what the analyses can establish have largely resolved my concerns.

Dear Reviewer RWnu,

Thank you very much for motivating us to conduct additional analyses and for taking the time to look through the revised version!

We are very glad that these helped address your concerns.

We also agree that understanding how the available geometric information is used to produce numerical readings remains an important open mechanistic question, and we hope our work can motivate further investigation in this direction.

Comments on the paper

2 comments

We thank all 4 reviewers for their feedback, especially the comments on writing and presentation. These helped us revisit how we communicate our findings and revise several parts of the manuscript to better convey our contributions.

To summarise the main changes in the revised version:

  • Introduction: We sharpened the main research question and clarified our motivation for using clocks as a controlled testbed for investigating visual measurement in VLMs.

  • Datasets (Section 3): We added a summarising table to help readers understand the existing and newly introduced datasets at a glance, what our datasets contribute, and where they are used in the experiments.

  • Controlled evaluation (Section 4.2): We rewrote this section and revised the presentation of the results to make the motivation, statistical analysis, and interpretation of model-specific sensitivities easier to follow. We also brought representative visual examples into the main text.

  • Fine-tuning and generalisation (Sections 6 and 7.1): We revisited these sections to position our findings more explicitly against prior work and to clarify why we initially expected clock-reading fine-tuning to transfer to other dial instruments.

  • Cross-instrument transfer (Appendix H.2): We added analyses of numerical reading errors, checked for degenerate clock-formatted answers, and included representative visual examples. We also discuss these findings in Section 7.1.

  • Geometric precision (Appendix E.5): We added a sample-wise analysis comparing angular probe errors with clock-reading errors, to further investigate whether imprecise geometric representations can explain the failures.

Following our discussion with Reviewer qnac, we also clarified some of the further scientific questions that our diagnosis raises, particularly whether effective adaptation targets can be predicted and how the order of answer generation may affect the causal use of geometric information.

To make it easier for the reviewers and area chair to locate the revisions discussed in our responses, we have also highlighted the revised and newly added parts of the manuscript in soft green.

We hope these revisions and additional analyses help make our contributions clearer and address the main concerns raised during the review.

As the discussion period comes to an end, we would like to summarise our discussions with the reviewers and highlight the points we have addressed:

1. Discussion with Reviewer RWnu:

Following this discussion, we added two analyses. The first investigates whether imprecise geometric information in the base model can account for numerical reading errors, showing that geometric decodability precision alone does not fully explain these failures and that how the model uses such information also matters.

The second investigates whether the failure to transfer to other instruments could be explained by catastrophic forgetting or degenerate clock-formatted answers. Our results provide evidence against widespread forgetting and systematic degeneration into clock-formatted answers, supporting our hypothesis that the conversion learned through clock fine-tuning does not necessarily transfer to other dial instruments.

We are glad that the reviewer confirmed that their concerns were largely addressed.


2. Discussion with Reviewer rkLR:

We further positioned our work in light of prior research. The important point is that the atomic results in our paper should not be interpreted separately. We believe their scientific contribution lies in the synergy between analyses of visual representations, their usage, and task-specific adaptation and transfer.

We tried to clarify this broader view in our final response, highlighting how the individual findings relate to one another and contribute to a larger understanding of visual reading failures.


3. Discussion with Reviewer qnac:

The reviewer raised questions about the predictability of effective adaptation targets. As shared in our response, our initial hypothesis that the most effective targets would be the components where geometric information is least decodable was not confirmed.

Another important point raised was the practical value of our findings (similar to Reviewer rkLR). We would like to reiterate the reviewer's point that a general improvement algorithm is not a requirement for a diagnostic contribution.

We think our work provides important insights and empirical findings that can motivate further mechanistic investigations of visual reading and the discovery of conversion mechanisms.

We hope our response has addressed these concerns, although we have not yet received confirmation from the reviewer.


4. Discussion with Reviewer qNvy:

The reviewer raised concerns about the presentation and writing of certain sections and results. We revised these parts, along with other parts of the manuscript, and highlighted the changes in the revised version.

Another concern about the role of geometric precision in reading errors was also addressed in our response to Reviewer RWnu, who acknowledged the additional evidence.

We hope that the revised presentation and the further analyses help address Reviewer qNvy's concerns, although we have not yet received confirmation.


Thank you again to all the reviewers and the Area Chair for following up. We hope that these clarifications and revisions help communicate the contribution of our work to understanding the internal mechanisms of visual reading in VLMs more clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.