acceptodds
Under review as a conference paper at ICLR 2027

Encoded but Not Reported: Label-Free Consistency Audits of Longitudinal Vision-Language Models

Abstract

Vision-language models are asked how a disease changes between scans, but benchmarks score each question alone. We audit them without labels using a property of true change: change over an interval is the sum of changes over its parts. Answers obeying this cocycle rule form the gradient subspace of combinatorial Hodge theory, so a model's distance to it lower-bounds its error. Clipped at a pre-declared threshold, this bound gives a distribution-free cohort certificate. We also probe frozen representations and repair answers by projection onto this subspace. Experiments on brain MRI and satellite imagery show that most open-weight models fail to beat a no-change predictor. On a glioma cohort with large changes, representations across model families encode change that answers do not report, and label-free probes recover most of it. Fine-tuning on projected answers corrects their direction but not their magnitude. Code, results and model outputs are available at https://anonymous.4open.science/r/VLM-3j-ICLR27-725E/.

Primary area:
datasets and benchmarks
Keywords:
Longitudinal reasoning, Vision-language models, Label-free evaluation, Cocycle consistency, Disease progression estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bxmUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes a new metric for longitudinal benchmarks. It tests model consistency on how quantities evolve in benchmarks such as tumor growth or satellite imagery over time.

Critical strengths

The new consistency metric and evaluation is important, informative about the performance of models, and well motivated.

Critical weaknesses

The presentation of the paper is quite weak: the syntax is bad in multiple locations, making some paragraphs hard to read. The quality of the figures is low: either in the way the information is presented or in the choice of what to present. Figure 1 is also weak; it is an amalgamation of every piece of information in the paper without a real structure and feels like unrelated numbers and equations (a main figure like that is often the first thing a reader goes to a here it doesn't bring any understanding to the reader).

The vocabulary of the paper is also not presented early on, so we are faced with confusing concepts. For example, the concept “distribution-free certificate over a cohort” is only weakly linked to the objects the benchmark uses (cohort is never explicitly linked to a series of images from the same patient or earth location).

We are sometimes faced with a “wall of numbers” paragraph such as line 398. This makes it difficult to understand what the message is.

Questions for authors
  1. Is the cocycle loss in Equation 6 a contribution of the paper, or is it taken from prior work? It is not motivated or referenced in Section 3.2. Why this form, and how was the variance penalty with λ = 5 chosen?

  2. Do the findings hold for larger models, including API ones? Most open-weight models fail to beat the no-change predictor, so it would be informative to know whether the trend of bigger models doing better continues.

3 comments
AuthorsOpenReview ↗

Response to reviewer bxmU:

We thank the reviewer and rewrote the paper for clarity. Changed text is in blue in the revised PDF.

Presentation

  • Figure 1 is new. We removed the old diagram. The new figure tells one story on real data:
    • (a) One LUMIERE subject whose answers cannot all be true. Qwen3-VL-8B answers +120% and +40% for two consecutive intervals, which imply +208%, yet −65% for the whole interval. It also gives a reversed interval the same sign, and calls a scan compared with itself +45% larger. Measured changes are in grey.
    • (b) For the same subject, the inconsistency (0.59) needs no labels and lies below the error (0.87), so it gives a floor.
    • (c) The main result over 8 models: answers against probes with and without labels.
  • Figures 2 and 3 are redrawn at print size, with one validated palette and direct labels. Figure 2b now also shows the neutral-prompt control; Figure 3 places each masking result under its condition. Captions are short; details are in the text.
  • Vocabulary up front. Section 2 now opens with short definitions of subject, visit, interval, triangle, cohort and answer vector. "Cohort" means the subjects of one dataset (e.g. all LUMIERE patients), and the introduction says so where the certificate first appears. Section 3.1 states exactly which subjects and views the certificate covers. Appendix B.1 certifies each cohort separately.
  • Walls of numbers. Section 4 is rewritten. Each paragraph opens with its message, and lists of numbers moved into tables. Old line 398 is now two short paragraphs.
  • Small tables. Table 17 is now two full-size panels (pooled audit; error per cohort), with the nine-model panel and the larger models in separate blocks. Tables 8, 10 and 18 were split or reset the same way, so they are no longer shrunk to fit.
  • Plain English throughout. We rewrote every paragraph of the main text and the appendix in short, plain sentences: one idea per sentence, terms defined where they first appear, numbers given one at a time with a pointer to their table. The mean sentence length in the appendix fell from about 24 to 15 words. Facts, numbers and claims are unchanged; blue marks changes in content only.
  • A guide to the appendix. The appendix now opens with a contents list (each appendix, its page, and the main-text sections that use it) and a map of all tables and figures: what each shows, where it sits, and which section cites it. To keep the focus on the contributions, the appendix is now 26 pages; material that no claim in the main text depends on (extra tables and figures, notation, further proofs, prompts, transcripts, error log, compute) moved to a separate supplementary PDF.

Q1. Is the loss in Eq. 6 a contribution? Motivation and λ.

  • It is ours. Its first term is Definition 1's cocycle written as a squared penalty. This is the form cycle-consistency losses take for correspondence over time (Wang et al., CVPR 2019; Dwibedi et al., CVPR 2019).
  • Why the second term. The zero map satisfies every constraint and says nothing. The second term keeps prediction variance above one, much as VICReg's variance term prevents collapse (Bardes et al., ICLR 2022).
  • How λ was chosen. We fixed λ = 5 before any run. In the appendix sweep, the tested weights λ = 1, 5 and 20 all give a linear head 78–89% of the supervised correlation, while λ = 0 gives 5–17%.

Section 3.2 now gives this motivation and the references.

Q2. Larger models and API models. We added the three largest open-weight models that fit our hardware: Qwen3-VL-30B-A3B, InternVL3.5-38B and Qwen2.5-VL-72B (AWQ). They ran under the protocol of the nine-model panel; unless stated otherwise, results use the panel (Sec. 4.1, App. B.7, Table 17, Fig. 4). Bigger does help, but only where changes are large.

  • On LUMIERE, size tracks correlation with true change (Spearman +0.86 across the 12 models). There, Qwen2.5-VL-72B, InternVL3.5-38B and Qwen3-VL-32B are 6.0%, 12.7% and 9.5% below the no-change predictor (InternVL3.5-38B: ρ 0.54). Qwen2.5-VL-72B almost never calls identical scans progression (1.0%, against 87% for its 7B sibling).
  • On the three small-change cohorts, no model of any size is resolved below the no-change predictor. Pooled error therefore follows size only weakly (Spearman −0.55, p = 0.06).
  • Larger models narrow the readout gap but do not close it. InternVL3.5-38B, the best answerer on LUMIERE (ρ 0.504), is still read better by its probe (0.622; gap CI [+0.05, +0.23]).

API models were not run. The answer audit and test-time projection need only generated outputs, so both apply to API models unchanged. Representation probes need hidden states, and our fine-tuning experiments need weight access. We restricted this study to open-weight models for reproducibility (App. B.7).

We also added a second glioma cohort, a prompt without time language and lesion-masking controls, requested by the other reviewers; they strengthen the main finding (Sec. 4.2).

Thank you for the answer and for the effort that went into the revision.

Presentation My concern about the presentation is still present; the paper's prose and overall writing are still a weak point for me (at the level of individual sentence syntax and at the level of the organisation of the paper). Additionally, the new figures and tables, while better, are still of low quality. For example, the captions are unclear or just catchphrases; the use of bold in tables is confusing and inconsistent.

Cocycle loss For the cocycle metric and its use as a loss in the probe, I appreciate the clarification.

Overall, the paper's contribution is promising. The label-free consistency check is useful and shows real limitations of VLMs for important tasks such as cancer progression tracking. As I understood it, the novelty doesn't lie in the loss or the metric used in the benchmark, but instead in the analysis performed on the models (including the interpretability experiments). I am not an expert on interpretability techniques, but I believe that probing is not sufficient to prove that the hidden direction actually expresses the desired variable. Moreover, the repair by projection doesn't seem to be able to make the model express this "encoded knowledge" beyond direction.

Given all that, I decided to maintain my current score.

AuthorsOpenReview ↗

2nd response to reviewer bxmU

We thank the reviewer for the careful second reading. We agree that the contribution lies in the analysis of the models, so we strengthened that part with new experiments and revised the presentation as asked. New text is purple in the revised PDF.

1. Probing alone does not show that the direction expresses change We agree, and we now intervene on the hidden state (Sec. 3.2, 4.2, new Fig. 3, App. B.8, Tables 18-19).

  • Where change becomes readable. Probes after every fourth layer of Qwen3-VL-8B: the supervised probe reads change at ρ 0.36-0.42 up to layer 16 and 0.51-0.52 from layer 20 on; the label-free probe works only from layer 20 on (|ρ| ≤ 0.19 before, 0.42-0.48 after). The answer reaches 0.14.
  • Steering. At layer 20 we add α·σ·d to the hidden state, where d is the direction along which states covary with measured change (fit on 28 training subjects) and σ its spread, on 240 held-out intervals from 55 subjects. The final-layer probe reading moves in proportion to α, from -3.83 to +2.95 log-ratio units at α = ∓48σ. Four random directions of the same norm, on the same step grid, move it at most 36% as far, with inconsistent signs. A second readout (the final-layer change direction) also singles out d (0.17 vs at most 0.07 per 10σ). The direction is carried through the remaining 16 layers; it is not an incidental correlate.
  • How much reaches the answer. The answer moves the same way and grows with α (-0.18 to +0.34), but per step only 7.6% as far as the probe reading (95% CI 7.2-8.0%): the interventional form of "encoded but not reported".
  • Second model, reported as found. On InternVL3.5-8B the answer does not follow d at all (largest shift 0.05). But random directions move that model's final-layer readings as much as d does, so there the test shows only that the answer ignores the direction, not that it is carried specifically. The paper scopes the claim to Qwen3-VL-8B accordingly.

2. The repair does not make the model express the encoded knowledge beyond direction

  • Why projection cannot. Consistency is invariant to scale, so no target built from the constraints alone can set magnitude. Proposition 5 now states this.
  • Readout distillation (new; Sec. 3.3, 4.3, Table 3, App. B.9, Table 20). We fine-tune the model to give, as its answer, the reading of a linear probe on its own final-layer state. The probe uses the labels of the 28 training subjects and is cross-fitted, so no target contains its own label. Recipe and held-out set are unchanged (1,716 intervals, 59 unseen subjects, 3 seeds).
    • Qwen3-VL-8B: error 0.943, 17.5% below the "no change" predictor (paired CI [-0.270, -0.130]), the best of all fine-tuned runs, with the highest correlation (ρ 0.585). Projection fine-tuning: 1.120.
    • Training directly on the same labels overshoots (sd 1.70) and is worse than "no change" (1.227). Same labels, different target.
    • The distilled answers are slightly better than the probe reading they learned from (0.943 vs 0.988; CI [-0.073, -0.016]).
    • Qwen3-VL-4B: 0.956, 16.4% below "no change". Labels for only half of the training subjects: 0.988, 13.6% below (CI [-0.215, -0.096]).
  • So the model can express the encoded change, magnitude included; what it lacks is the readout. This repair needs some labels; a label-free head fit on 28 subjects was too weak to serve as a teacher.

3. Presentation

  • Captions. Every caption opens with a bold title naming what is shown, then gives the data and what each axis, column, colour and mark means. Slogan-like panel titles are gone, e.g. "(c) Bias, not blindness" is now "(c) Sign accuracy under known deformation".
  • Bold. One rule in every table, stated in its caption: bold marks the best fine-tuned run per column (only Table 3); shaded rows mark our methods; ✓/✗ mark the yes/no test the caption names. Bold no longer marks our own row or column.
  • Organisation and prose. Section 4 opens with Data, Models and Conditions paragraphs, and each RQ subsection opens with its main result and ends with a takeaway. We rewrote Section 4 and the method paragraphs as connected sentences instead of lists of numbers. Supporting material moved to the appendix (datasets, per-model test-time results, full fine-tuning table), so the body has three tables and three figures, each cited where used.

Table and figure numbers changed; the guide at the start of the appendix lists every table and figure with its location.

Reviewer_LiZTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates the reliability of VLMs in estimating longitudinal changes from medical images. Towards this the paper introduce a label-free consistency framework to identify inconsistent predictions, check if frozen representation contain information missing from the final answers, and improve the predictions through projection and finetuning.

Critical strengths
  • The reliability of VLMs, such as their tendency to generate answers that are not properly grounded in visual evidence, has been an ongoing challenge of the field. Looking at this problem from a longitudinal perspective is important, and this has direct clinical relevance (e.g., disease progression analysis).

  • I also find the connection between longitudinal change estimation and Hodge decomposition interesting. The formulation of consistency (Defn 1) through identity, antisymmetry, and cocycle constraints is novel and allows evaluating model predictions without relying on the GT annotations.

  • The provided experiments evaluate different VLM families and also include variety of controls that allows more in-depth analysis. Overall this paper has comprehensive experimental analysis.

Critical weaknesses
  1. My main question/concern is if the label-free consistency objective genuinely recover disease progression or simply captures other signals that are correlated with longitudinal change. Model relying on shortcuts is well established in medical imaging so ensuring that label-free probes can actually capture the disease related feature is crucial. The paper does acknowledge this concern (in Sec 4.2).

  2. Also the results seems to suggest that cocycle doesn't contribute much as the antisymmetry alone improves significantly (Table 5). How can we properly establish the importance of this cocycle formulation in general scenario?

  3. Also, the gains in estimating the magnitude of disease progression seems limited. The errors values are not moving much (e.g., 1.153 vs. 1.143 vs. 1.120). Are these differences statistically significant? If the proposed method can't accurately estimate how much a disease has progressed, the benefits is not well established.

  4. How generalizable the overall representation-level findings are? It could be my lack of familiarity about these datasets but the best results seems to be for one dataset (LUMIERE) and similar result doesn't seem to generalize for other datasets (also acknowledged by authors in Sec 4.2). So how can we reliably establish the presented work is generalizable?

Questions for authors

Address the questions in weakness section.

2 comments
AuthorsOpenReview ↗

Response to reviewer LiZT:

We thank the reviewer. We ran new experiments for every point. Changes are in blue in the revised PDF, and the new analyses are in Appendix B.

W1. Disease signal or shortcuts? We added three tests (Sec. 4.2, App. B.4).

  1. Lesion masking. In both scans of a pair we filled the union of both visits' lesion masks (dilated, oedema included) with the same constant. The masked region then has the same shape and value in both images and carries no change. The Qwen3-VL-8B probe falls from 0.505 to 0.131 [−0.029, +0.278], no better than the shuffled-image control. A box around the lesion alone keeps 0.470. With only the tumour core masked, the visible oedema keeps 0.305. The label-free heads collapse too (cocycle head 0.449 → 0.042). The result holds for all 8 readout models (masked probe 0.09–0.21, none resolved above shuffled; box 0.37–0.51; Table 12) and on MU-Glioma-Post (0.459 → 0.098; box 0.410; Table 11). So the readout comes from the lesion and its surroundings, not from the rest of the slice. With only the core masked, the oedema keeps 0.20–0.33 on LUMIERE but 0.155 on MU. Figure 3 (App. B.4) shows the hidden regions and every model's result per condition.
  2. Nuisance covariates. We partialled out interval length, visit order and timing, intensity statistics and brain area outside the lesion, and the clinical course (never shown to the model). The probe keeps 73–88% of its correlation for all 8 models, and every 95% CI excludes zero (Qwen3-VL-8B: 0.507 → 0.429 [0.215, 0.621]). It also holds on post-operative pairs alone. The covariates by themselves predict change at ρ 0.09 (ridge) and 0.31 (boosted trees), and neither CI excludes zero (Table 10).
  3. Second cohort. On MU-Glioma-Post the image-free and shuffled controls again stay within 0.05 of zero for every model.

We still say that a linear probe shows accessible information, not the model's mechanism.

W2. Importance of the cocycle. We now state plainly where the cocycle matters and where it does not.

  • Training: it adds nothing we can resolve. An antisymmetry target that cannot tie at zero matches the projection in correlation, matched-rate sign, AUROC and error (Δρ +0.003 [−0.018, +0.022], ΔAUROC +0.010 [−0.002, +0.020]); raw sign differs only through zero answers, and the cocycle residual is lower. Most of the earlier "+20.7 sign points" came from tied medians that made the antisymmetry control answer 0% on 41% of items: at matched no-change rates the difference is +0.5 [−0.3, +1.4], though a small AUROC edge over that tied control remains (+0.025 [+0.011, +0.037]). Training on forward questions only makes answers coherent but does not fix their direction.
  • Test time: it helps where antisymmetry cannot be used. On top of antisymmetry it adds a small but resolved 0.4%. Existing longitudinal benchmarks ask forward intervals only, so antisymmetry cannot even be checked there. In that setting, projecting forward answers lowers greedy error in 36 of 36 model–cohort pairs: 1.5% pooled, up to 12% on SpaceNet-7 (Table 7).
  • The audit relies on it. The cocycle gives the label-free bound on forward-only data. A head trained only on the cocycle recovers 88% of the supervised readout without labels.

Section 4.3, Takeaway 3 and the contributions now say this.

W3. Are the magnitude differences significant? Paired subject bootstrap, 59 held-out subjects (Sec. 4.3, Table 13):

  • base − trivial: +0.010 [−0.024, +0.046], not significant.
  • cocycle − trivial: −0.023 [−0.029, −0.017], significant but 2.0%.
  • cocycle − base: −0.033 [−0.067, −0.002]; Holm-adjusted p = 0.07, so not significant after correction.
  • scale-matched − trivial: −0.073 [−0.093, −0.053], 6.4%.
  • On MU-Glioma-Post the transferred adapters are 1.6% (cocycle) and 4.8% (scale-matched) below "no change".

So the answer is yes, significant but small. We do not claim magnitude estimation; the abstract, Sec. 4.3 and the conclusion say so. The paper's claims are the label-free audit, the readout diagnosis, and the repair of direction and coherence.

W4. Generalisation beyond LUMIERE. We added MU-Glioma-Post, a second post-treatment glioma cohort from another centre: 152 subjects, median |Δ| 0.72. The image rendering and prompt are identical to LUMIERE's.

  • The probe–answer gap is resolved for all 8 models (Table 3). Example: Qwen3-VL-8B 0.462 vs 0.111, gap +0.349 [+0.187, +0.490].
  • Within that cohort, the gap grows with the size of change (+0.02, +0.30, +0.38 across three bands).
  • The LUMIERE-trained repair transfers without retraining (ρ 0.115 → 0.457).
  • The audit and certificate behave as on LUMIERE.
  • The finding also survives a prompt without time language (App. B.3).

On the three small-change cohorts no gap is resolved, and the abstract and Sec. 4.2 state this scope.

I thank the authors for their rebuttal and for conducting additional experiments to address my concerns. I appreciate the effort put into strengthening the experimental evaluation and clarifying the contributions of the paper.

However, I still have some reservations about the main findings.

First, while the additional lesion-masking experiments are useful, I am still not fully convinced that they establish the ability to capture clinically meaningful disease progression. Lesion-related information is not necessarily equivalent to accurate disease progression. It does not necessarily establish that the probe captures clinically correct longitudinal changes.

Second, I appreciate the authors' acknowledgement that the cocycle formulation provides limited additional benefits over antisymmetry during training. While its role in label-free auditing is interesting, the new results do not clearly establish that cocycle consistency leads to substantially better disease progression estimation.

Third, I thank the authors for providing the statistical significance analysis. However, the results further highlight that the proposed approach does not substantially improve the estimation of actual progression magnitude. So its not clear how much the proposed framework contributes to clinically actual useful longitudinal assessment.

Overall, I believe this is an interesting work addressing an important problem, and the additional experiments have certainly strengthened the paper. However, considering the level of methodological contribution and empirical rigor expected at ICLR, I do not feel that my main concerns have been sufficiently resolved to justify increasing my score. Therefore, I have decided to maintain my original rating.

Reviewer_2jyyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a label-free audit of longitudinal change estimation based on identity, antisymmetry, and temporal composition. It represents consistent predictions as differences of per-visit potentials and uses the distance to this subspace to lower-bound prediction error. Frozen-representation probes assess whether change information is accessible but absent from generated answers, while projection provides consistent predictions and self-training targets. Experiments identify substantial inconsistencies and a representation–answer gap on a glioma cohort.

Critical strengths
  1. A principled and useful auditing formulation. The paper distinguishes consistency from accuracy and derives a valid error lower bound under an explicit condition: ground truth belongs to the consistent subspace. The cohort certificate clearly states its sampling and fixed-threshold assumptions.

  2. The fine-tuning experiments control for answer formatting, shrinkage, sample aggregation, and antisymmetry, with subject-level uncertainty estimates. The authors explicitly report that antisymmetry explains most improvements, magnitude estimation remains weak, and the readout finding does not generalize uniformly across cohorts.

Critical weaknesses
  1. The example prompts state that the second examination occurred after the first, while Appendix D describes reversal controls using the same direction-blind prompt. The handling of chronological wording for reversed and identity pairs therefore needs clarification. A neutral condition—“compare image A with image B,” without asserting acquisition order—would help determine how much inconsistency comes from visual comparison versus a prompt-induced progression prior. The existing interval-withheld condition is relevant, but its effect on the headline identity, antisymmetry, and readout results is not established. This would not invalidate the observed failures under the current prompts; it would strengthen their interpretation.

  2. The paper appropriately reports that antisymmetry explains 92–96% of the fine-tuning correlation gain. On Qwen3-VL-8B, full projection improves correlation from 0.493 to 0.520 over antisymmetry-only training, while sign accuracy rises from 43.8% to 64.5%. However, the frequency of exact-zero answers simultaneously falls from 41.0% to 8.4%. Thus, the larger sign-accuracy improvement combines directional discrimination with a substantial change in zero-answer behavior. The scale-matched projection experiment further demonstrates that output scale affects performance. A scale-matched antisymmetry baseline, sensitivity to numeric output precision, and evaluation at matched no-change rates would better isolate what temporal composition contributes beyond antisymmetry and calibration.

  3. The representation–answer gap is established across several model families, but principally on LUMIERE. No overall gap is statistically resolved on the other three cohorts, and the synthetic magnitude intervention is inconclusive. Fine-tuning is demonstrated on two sizes of one model family and one cohort. The authors acknowledge these limitations, but they constrain how broadly the conclusions can be applied. The mathematical contribution is also primarily an application of established projection machinery, rather than a new decomposition; the manuscript explicitly identifies its projection as HodgeRank. I therefore view the paper’s main contribution as the auditing perspective and controlled empirical diagnosis. Replication on another cohort with substantial changes would materially strengthen that contribution.

Questions for authors
  1. What exactly is included in the cohort certificate? Please explain how the 238 subjects used for certification are selected from the datasets in Table 1, how multiple views are combined within subjects, and which cohort distribution the pooled certificate represents.

  2. Does the sign-accuracy advantage persist at matched no-change rates? This would clarify whether the cocycle primarily improves discrimination or reduces zero predictions.

1 comment
AuthorsOpenReview ↗

Response to reviewer 2jyy:

We thank the reviewer for the careful reading. We ran every requested analysis. Changes are in blue in the revised PDF, and the new analyses are in Appendix B.

W1. Chronological wording; neutral prompt. You are right. Every pair prompt says "first/second timepoint" and "acquired Δt after the first". So a reversed probe states a false order by design, and an identity probe states six months. Section 4 now says this. We ran your control on all 9 models, all 4 cohorts, every interval and identity probe: "Image A/Image B", no time language (App. B.3, Fig. 2b).

  • The wording matters: error −0.025 [−0.029, −0.021], antisymmetry violations −24%, ρ with true change +0.067 (paired subject bootstrap).
  • The failures remain. On items both prompts parsed, only 4 of 9 models beat "no change" (2 of 9 originally; Table 8 now marks them). Identical scans are still called progression on 42.8% of probes (51.5% before), with shifts in both directions (Qwen3-VL-2B 99.7→60.0%, 4B 40.8→96.7%). 85% of nonzero neutral answers say Image B is larger.
  • Readout: features and answers recomputed under the neutral prompt for 4 models. The probes barely move, the answers improve, and the gap shrinks by 0.098 on average. It stays resolved for all 4 (Qwen3-VL-8B +0.250 [+0.025, +0.418]; Table 9).

So part of the inconsistency is prompt-induced, but the main failures and the readout gap remain without time language.

W2. Cocycle beyond antisymmetry and calibration. We agree, and we corrected the claim.

  • Zero answers. At matched no-change rates, the +20.7-point sign advantage over antisymmetry-only training falls to +0.5 [−0.3, +1.4]. Median pooling makes the two directions' medians tie, so 22% of the antisymmetry targets read 0%. This is most of the gap, not all: AUROC stays higher (+0.025 [+0.011, +0.037]), and with one-decimal targets on both sides +1.6 [+0.7, +2.5] matched-sign points remain.
  • Tie-free control (new, mean pooling, one decimal). It matches the projection in ρ (+0.003 [−0.018, +0.022]), AUROC (+0.010 [−0.002, +0.020]), error and matched-rate sign. Raw sign differs by +3.0 [+1.5, +4.6], only through its 5.7% zero answers (0.3% for the projection); the cocycle residual is lower.
  • Scale-matched antisymmetry (new). ρ 0.497, error 1.097, against 0.537 and 1.070 for the scale-matched projection. It inherits the tied zeros (33%).
  • Precision. One-decimal targets remove the rounding zeros but not the ties. Table 5 now also reports AUROC.
  • Training without reversed questions. Forward-only projection makes answers coherent but does not fix direction (ρ 0.163).
  • Test time. The cocycle adds a small but resolved 0.4% on top of antisymmetry (CI excludes 0). With forward intervals only, as in existing benchmarks, it lowers greedy error in 36 of 36 model–cohort pairs (Table 7).

Section 4.3, Table 5, Takeaway 3 and the contributions now say: in training the gain is antisymmetry; the cocycle adds coherence, not a gain in ρ, AUROC or error we can resolve.

W3. Second large-change cohort; scope. We added MU-Glioma-Post (TCIA, CC BY 4.0, another centre; 152 subjects, 538 visits, median |Δ| 0.72 vs 0.96). The image rendering and prompt are identical to LUMIERE's.

  • The probe–answer gap is resolved for all 8 readout models (7 of 8 on LUMIERE), now including Qwen3-VL-32B. Example: Qwen3-VL-8B 0.462 vs 0.111, gap +0.349 [+0.187, +0.490]. Text-only and shuffled controls stay within 0.05 of zero (Table 3).
  • Within the cohort the gap grows with the size of change (+0.02 / +0.30 / +0.38).
  • Only 1 of 9 models beats "no change", by 1.1%. Every certified floor lies below the error.
  • The LUMIERE-trained adapters transfer to all 152 subjects without retraining (ρ 0.115 → 0.457). Training on MU's own samples gives the same ρ (App. B.6).

We agree the projection is HodgeRank's. The introduction says so, and frames the contribution as the audit and the controlled diagnosis.

Q1. What the certificate covers. A subject enters if it has ≥3 visits. Each view gives one answer vector, and the subject's bound is the mean of its per-view bounds, which is at most its mean per-view error. Subjects are the sampling unit. The 238 subjects are 77 LUMIERE, 75 VS-MC-RC, 56 OASIS-2 and 30 SpaceNet-7, so the pooled certificate covers a mixture weighted by subject counts (Sec. 3.1). Table 6 certifies each cohort separately. The floor is the KL bound, fixed in advance; taking the larger of KL and empirical Bernstein is valid only at δ/2 each, and that joint floor is lower in all 45 cells (App. C.4). Every floor is compared with the error averaged the same way, within subject then over subjects, and lies below it.

Q2. Matched no-change rates. Mostly. At matched rates the sign advantage falls to +0.5 [−0.3, +1.4], but a small AUROC edge over the tied control remains (W2). Against the tie-free control, ρ, matched-rate sign, AUROC and error differences are unresolved; that is the claim the paper now makes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.