- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper proposes a new metric for longitudinal benchmarks. It tests model consistency on how quantities evolve in benchmarks such as tumor growth or satellite imagery over time.
Critical strengths
The new consistency metric and evaluation is important, informative about the performance of models, and well motivated.
Critical weaknesses
The presentation of the paper is quite weak: the syntax is bad in multiple locations, making some paragraphs hard to read. The quality of the figures is low: either in the way the information is presented or in the choice of what to present. Figure 1 is also weak; it is an amalgamation of every piece of information in the paper without a real structure and feels like unrelated numbers and equations (a main figure like that is often the first thing a reader goes to a here it doesn't bring any understanding to the reader).
The vocabulary of the paper is also not presented early on, so we are faced with confusing concepts. For example, the concept “distribution-free certificate over a cohort” is only weakly linked to the objects the benchmark uses (cohort is never explicitly linked to a series of images from the same patient or earth location).
We are sometimes faced with a “wall of numbers” paragraph such as line 398. This makes it difficult to understand what the message is.
Questions for authors
-
Is the cocycle loss in Equation 6 a contribution of the paper, or is it taken from prior work? It is not motivated or referenced in Section 3.2. Why this form, and how was the variance penalty with λ = 5 chosen?
-
Do the findings hold for larger models, including API ones? Most open-weight models fail to beat the no-change predictor, so it would be informative to know whether the trend of bigger models doing better continues.
3 comments
Thank you for the answer and for the effort that went into the revision.
Presentation My concern about the presentation is still present; the paper's prose and overall writing are still a weak point for me (at the level of individual sentence syntax and at the level of the organisation of the paper). Additionally, the new figures and tables, while better, are still of low quality. For example, the captions are unclear or just catchphrases; the use of bold in tables is confusing and inconsistent.
Cocycle loss For the cocycle metric and its use as a loss in the probe, I appreciate the clarification.
Overall, the paper's contribution is promising. The label-free consistency check is useful and shows real limitations of VLMs for important tasks such as cancer progression tracking. As I understood it, the novelty doesn't lie in the loss or the metric used in the benchmark, but instead in the analysis performed on the models (including the interpretability experiments). I am not an expert on interpretability techniques, but I believe that probing is not sufficient to prove that the hidden direction actually expresses the desired variable. Moreover, the repair by projection doesn't seem to be able to make the model express this "encoded knowledge" beyond direction.
Given all that, I decided to maintain my current score.
2nd response to reviewer bxmU
We thank the reviewer for the careful second reading. We agree that the contribution lies in the analysis of the models, so we strengthened that part with new experiments and revised the presentation as asked. New text is purple in the revised PDF.
1. Probing alone does not show that the direction expresses change We agree, and we now intervene on the hidden state (Sec. 3.2, 4.2, new Fig. 3, App. B.8, Tables 18-19).
- Where change becomes readable. Probes after every fourth layer of Qwen3-VL-8B: the supervised probe reads change at ρ 0.36-0.42 up to layer 16 and 0.51-0.52 from layer 20 on; the label-free probe works only from layer 20 on (|ρ| ≤ 0.19 before, 0.42-0.48 after). The answer reaches 0.14.
- Steering. At layer 20 we add α·σ·d to the hidden state, where d is the direction along which states covary with measured change (fit on 28 training subjects) and σ its spread, on 240 held-out intervals from 55 subjects. The final-layer probe reading moves in proportion to α, from -3.83 to +2.95 log-ratio units at α = ∓48σ. Four random directions of the same norm, on the same step grid, move it at most 36% as far, with inconsistent signs. A second readout (the final-layer change direction) also singles out d (0.17 vs at most 0.07 per 10σ). The direction is carried through the remaining 16 layers; it is not an incidental correlate.
- How much reaches the answer. The answer moves the same way and grows with α (-0.18 to +0.34), but per step only 7.6% as far as the probe reading (95% CI 7.2-8.0%): the interventional form of "encoded but not reported".
- Second model, reported as found. On InternVL3.5-8B the answer does not follow d at all (largest shift 0.05). But random directions move that model's final-layer readings as much as d does, so there the test shows only that the answer ignores the direction, not that it is carried specifically. The paper scopes the claim to Qwen3-VL-8B accordingly.
2. The repair does not make the model express the encoded knowledge beyond direction
- Why projection cannot. Consistency is invariant to scale, so no target built from the constraints alone can set magnitude. Proposition 5 now states this.
- Readout distillation (new; Sec. 3.3, 4.3, Table 3, App. B.9, Table 20). We fine-tune the model to give, as its answer, the reading of a linear probe on its own final-layer state. The probe uses the labels of the 28 training subjects and is cross-fitted, so no target contains its own label. Recipe and held-out set are unchanged (1,716 intervals, 59 unseen subjects, 3 seeds).
- Qwen3-VL-8B: error 0.943, 17.5% below the "no change" predictor (paired CI [-0.270, -0.130]), the best of all fine-tuned runs, with the highest correlation (ρ 0.585). Projection fine-tuning: 1.120.
- Training directly on the same labels overshoots (sd 1.70) and is worse than "no change" (1.227). Same labels, different target.
- The distilled answers are slightly better than the probe reading they learned from (0.943 vs 0.988; CI [-0.073, -0.016]).
- Qwen3-VL-4B: 0.956, 16.4% below "no change". Labels for only half of the training subjects: 0.988, 13.6% below (CI [-0.215, -0.096]).
- So the model can express the encoded change, magnitude included; what it lacks is the readout. This repair needs some labels; a label-free head fit on 28 subjects was too weak to serve as a teacher.
3. Presentation
- Captions. Every caption opens with a bold title naming what is shown, then gives the data and what each axis, column, colour and mark means. Slogan-like panel titles are gone, e.g. "(c) Bias, not blindness" is now "(c) Sign accuracy under known deformation".
- Bold. One rule in every table, stated in its caption: bold marks the best fine-tuned run per column (only Table 3); shaded rows mark our methods; ✓/✗ mark the yes/no test the caption names. Bold no longer marks our own row or column.
- Organisation and prose. Section 4 opens with Data, Models and Conditions paragraphs, and each RQ subsection opens with its main result and ends with a takeaway. We rewrote Section 4 and the method paragraphs as connected sentences instead of lists of numbers. Supporting material moved to the appendix (datasets, per-model test-time results, full fine-tuning table), so the body has three tables and three figures, each cited where used.
Table and figure numbers changed; the guide at the start of the appendix lists every table and figure with its location.
Response to reviewer bxmU:
We thank the reviewer and rewrote the paper for clarity. Changed text is in blue in the revised PDF.
Presentation
Q1. Is the loss in Eq. 6 a contribution? Motivation and λ.
Section 3.2 now gives this motivation and the references.
Q2. Larger models and API models. We added the three largest open-weight models that fit our hardware: Qwen3-VL-30B-A3B, InternVL3.5-38B and Qwen2.5-VL-72B (AWQ). They ran under the protocol of the nine-model panel; unless stated otherwise, results use the panel (Sec. 4.1, App. B.7, Table 17, Fig. 4). Bigger does help, but only where changes are large.
API models were not run. The answer audit and test-time projection need only generated outputs, so both apply to API models unchanged. Representation probes need hidden states, and our fine-tuning experiments need weight access. We restricted this study to open-weight models for reproducibility (App. B.7).
We also added a second glioma cohort, a prompt without time language and lesion-masking controls, requested by the other reviewers; they strengthen the main finding (Sec. 4.2).