- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that the the internal state of the model may contain anomaly evidence that the model’s own scores for the answer letters fail to expose. Thus, they propose to train a small head that maps the internal states, together with the text of each answer option, to the ground-truth labels. When the new MCQ comes in, the answer will be based on 1) the VLM's score on the choice-tokens and 2) how the trained head predicts from the internal states.
Experiments show that this beats LORA on MMAD dataset, and the paper claims that the proposed method is more stable than LoRA in AnomalyCoT.
Critical strengths
The idea that the internal states may contain answer-relevant information that the model’s letter scores do not expose.
If it beats LoRA, it might inspire some more research in this line.
Critical weaknesses
- The paper lacks the evidence that the internal states actually contained evidence that the model’s letter scores fail to expose (their Contribution 1), though this is the central claim of this paper. As shown in Table 5 in Appendix~A, the probe-only method (the trained head that maps the internal state and the option texts to the ground-truth labels) achieves almost equally good performance when using the layer-0 and the last layer. If there is information in the internal states, I would expect the trained head to perform bad on the layer-0 (hardly any internal states yet), and to gradually perform well on the deeper, internal layers (a lot of internal state information).
To me, it looks like the trained head finds a short cut, since the layer-0 state carries essentially no image information, its accuracy there must come mostly from the option texts, i.e. learning which answer wordings tend to be correct.
- For the main results in Table 1 and Table 2:
- Table 1: On the held-out column, the mean gap over LoRA comes largely from a single backbone (Pixtral); for the other six VLMs the gaps are much smaller.
- Table 2: on AnomalyCoT, LORA and the proposed method is really a tie to me. Also, the statement that the proposed method is more stable also heavily depends on LoRA's crash on Pixtral.
- Both Table 1 & 2: the absolute margin between the baseline and the proposed method is not so informative to me, as the proposed method has access to the ground-truth label yet the baseline models don't.
So the evidence looks quite weak.
Questions for authors
When splitting the datasets for training/testing, is the splitting based on images, or questions? Is it possible that some images in the test set have already been seen in the training phase, but only the questions are new?
6 comments
Follow-up to Reviewer DiB6: where your concerns stand, and any remaining questions
Thank you again for your review. Your main concern, that the head may be learning from the option texts, led us to add a direct test, and the revised paper is built on what that test and the other controls support. Since our response of Nov 13 the remaining checks have finished and the PDF has been revised, so we summarize below what the revised paper claims and where your points stand. We would be grateful to know whether anything remains open.
What the revised paper claims, under equal supervision. OCP-R is a small correction added to the frozen option logits. With the same head and the same labels, adding the correction gives higher held-out accuracy than using the head as a replacement readout on all nine AnomalyCoT backbones (+6.54 points, exact Wilcoxon p = 0.0039). On the seven MMAD backbones it turns 4–8% of the questions from frozen-correct to wrong, against 13–18% for the replacement readout. It is above the last-four-layer LoRA baseline overall on all seven MMAD backbones (six intervals exclude zero, Table 20), and without modifying the backbone it obtains 80% of the all-layer LoRA's overall gain on Qwen2.5-VL (59% held-out).
Where your points stand.
- Evidence for hidden information (W1). The direct test is in Tables 5 and 8: a correction that reads only the option texts reproduces 86–96% of the gain on the seven MMAD backbones. The paper therefore presents OCP-R as supervised adaptation of the readout, and the abstract and the contribution list no longer argue from hidden visual evidence.
- Tables 1 and 2, Pixtral, stability (W2, W3). The paper no longer claims a held-out or stability advantage over LoRA, and it states that LoRA on all layers is more accurate at an equal parameter count. The Pixtral LoRA entry came from a run that did not train.
- Access to labels (W4). The comparisons above are between methods trained on the same labels.
- Split (your question). The split is by query image, and no evaluation image occurs in training on either benchmark.
On "learning which answer wordings tend to be correct". The gain comes largely from the option texts, and the paper says so. It is not a memory of which strings were correct. On the 834 evaluation questions whose answer string was never an answer in training, OCP-R still gains 8.59 points [5.73, 11.57]. In a manual rewrite audit (new Appendix H), the gain remains when the correct option is reworded: +7.56 points [1.61, 13.62] on average over seven backbones, against +8.80 with the original wording. A reworded distractor costs OCP-R 5.49 points on Qwen2.5-VL, which the Limitations state.
Presentation. The abstract and the introduction are rewritten to this reading, and Appendix G lists the changes from the submission in ten items.
If any concern is not yet addressed, or if the revised paper is still hard to follow in some place, we would be grateful to hear where, and we will answer before the discussion period ends. Thank you for your time and for the care of your review.
Thank the authors for the additional experiments and the careful revision. My first concern (W1) has been fully addressed but I still have some concerns for W2.
Specifically, I appreciate the corrected LoRA comparisons: OCP-R achieves higher overall MMAD accuracy than last-four-layer LoRA on all seven backbones. But the held-out results are less conclusive, and the matched-parameter comparison on Qwen2.5-VL favors all-layer LoRA, so I agree with the more specific conclusion in the revision, which is however also narrower. The new experiments give me a clearer understanding of where OCP-R helps, and I’m raising my rating from 1 to 2.
Response to Reviewer DiB6
We thank the reviewer for pointing out that improved accuracy does not establish recovery of hidden visual information. The new controls support a narrower interpretation.
W1. Shortcut through option texts and the central diagnosis. On Qwen2.5-VL/MMAD, a learned option-only correction added to the original image-conditioned frozen logits reaches 81.07%, versus 72.56% frozen and 82.38% for OCP-R. It reproduces 87% of the overall gain. Update (Nov 14). Across all seven MMAD backbones, this share is 86–96%. A simple option-string frequency prior reaches 78.63%. These results substantiate the concern that benchmark option regularities account for much of the improvement.
We also corrected “layer 0”: it is the first decoder block's output, not the embedding layer. This correction does not resolve your concern. We therefore used a learned constant query to construct an option-only correction.
On AnomalyCoT, option-only corrections reproduce 26–52% of the overall gain across nine backbones. That result alone does not identify a visual contribution. Update (Nov 16). On all nine backbones, now tested with an image-blind correction, question and option text reproduce most of the remaining overall improvement; full OCP-R is significantly lower overall on four (Qwen2.5-VL, Qwen2-VL, Pixtral, Idefics3), and no overall or held-out interval establishes an accuracy benefit from giving the correction access to images. The original images are still used by the frozen logits in every comparison.
We therefore withdraw the general claim that these accuracy gains demonstrate recovery of hidden visual anomaly evidence. The supported contribution is supervised adaptation of the option-level readout while retaining the frozen logits. The positive image-access result on MMAD's held-out source (+2.13 [0.71, 3.55]) is reported with that specific scope.
W2. Table 1 and Pixtral. Your concern about the held-out comparison is borne out by the fixed-input re-evaluation. Pixtral's held-out gap over rank-4 LoRA decreases from +11.58 to +2.72 points, and only one of seven backbones has a held-out interval excluding zero. We withdraw the general held-out superiority claim.
The overall comparison remains positive on all seven backbones, with intervals excluding zero on six. This table uses the original LoRA checkpoints and generate-and-parse evaluation. A separate Qwen2.5-VL experiment matches the option-level loss and forced-prefix protocol: OCP-R exceeds the tested last-four-layer rank-8 LoRA by +3.85 [2.29, 5.47] overall, while the held-out difference is inconclusive. Update (Nov 14). The all-layer, learning-rate-selected comparison is complete and reverses the order. Rank-1 LoRA on all 28 layers (0.43M parameters against OCP-R's 0.47M, same loss and scoring, three seeds) reaches 84.89 / 81.63, above OCP-R by 2.51 [1.10, 3.89] overall and 4.63 [1.85, 7.58] held-out. OCP-R's overall advantage over LoRA is therefore specific to the last-four-layer placement, and we claim no accuracy advantage at a matched parameter count. Larger-budget adaptation is also more accurate.
W3. Table 2 and stability. We agree and withdraw the stability claim against LoRA. The AnomalyCoT baseline is LoRA with a 30-token soft prompt, as clarified in General Response 2/2. Update (Nov 14). The Pixtral entry came from a run that did not train: its archived training loss is flat, and its 2,500- and 5,000-question runs give identical answer counts in all 13 subsets. The cause, an overflow in the joint gradient clipping of LoRA and soft prompt, occurs on Pixtral only. With the LoRA update restored (the soft prompt stays at its initial value), three seeds reach 52.24 ± 7.60 / 58.70 ± 1.07, so the entry does not show a LoRA failure on Pixtral. The evidence for retaining frozen logits comes from comparison with Vanilla OCP: across nine AnomalyCoT backbones and three seeds, OCP-R has higher held-out means on all nine, with a mean difference of +6.54 points (exact Wilcoxon p = 0.0039). This result does not establish superiority over LoRA.
W4. Access to labels. We agree that a supervised method's gain over an unadapted frozen readout cannot by itself establish a better learning method or the source of the information it uses. We treat that difference as supervised adaptation gain. Comparisons with LoRA, Vanilla OCP, option-only, and image-blind corrections use the same training labels within each experiment. The narrower claims above rest on those supervised comparisons.
Question. Split. The split is by query image, not by question. Every question about an evaluation image is assigned to evaluation, and checks by image path, MD5, and near-duplicate search found no train–evaluation image overlap on MMAD. No evaluation image appears in the AnomalyCoT training set either. In-domain categories can overlap with training, while the designated held-out sources are excluded from training.