acceptodds
Under review as a conference paper at ICLR 2027

OCP-R: Stabilizing Option-Level Readout for VLM Anomaly Detection

Abstract

VLM anomaly-detection benchmarks such as MMAD and AnomalyCoT test in- dustrial inspection with multiple-choice questions, matching the closed defect vo- cabularies of industrial quality assurance, so a frozen VLM must turn its hidden state into answer-option scores, a step we call the option-level readout. In this paper, we show this step is unreliable: frozen VLMs often hold anomaly evidence that their option logits do not expose. Two measurements fix the repair. Weak frozen readouts spend most of their MCQ supervision on answer-format rather than content tokens, so what is missing is content, not capacity; and a replace- ment readout discards decisions the frozen head already scores correctly, so the correction must add to the frozen logits rather than replace them. We propose Option-Conditional Residual Readout (OCP-R), a small trained head that scores each candidate option against the frozen state and adds the result to the frozen logits, zero at initialization so training departs from the frozen readout only where supervision demands. Across nine general-purpose and four anomaly-specialized VLMs, OCP-R raises mean MMAD accuracy from 62.17% to 78.27% while pre- serving 93% of the frozen model’s correct predictions, and on AnomalyCoT it is the only method with no catastrophic held-out drop on any backbone, against one for internal-weight adaptation and three for a replacement readout.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, industrial anomaly detection, multiple-choice evaluation, probing, readout stabilization, residual learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DiB6OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that the the internal state of the model may contain anomaly evidence that the model’s own scores for the answer letters fail to expose. Thus, they propose to train a small head that maps the internal states, together with the text of each answer option, to the ground-truth labels. When the new MCQ comes in, the answer will be based on 1) the VLM's score on the choice-tokens and 2) how the trained head predicts from the internal states.

Experiments show that this beats LORA on MMAD dataset, and the paper claims that the proposed method is more stable than LoRA in AnomalyCoT.

Critical strengths

The idea that the internal states may contain answer-relevant information that the model’s letter scores do not expose.

If it beats LoRA, it might inspire some more research in this line.

Critical weaknesses
  1. The paper lacks the evidence that the internal states actually contained evidence that the model’s letter scores fail to expose (their Contribution 1), though this is the central claim of this paper. As shown in Table 5 in Appendix~A, the probe-only method (the trained head that maps the internal state and the option texts to the ground-truth labels) achieves almost equally good performance when using the layer-0 and the last layer. If there is information in the internal states, I would expect the trained head to perform bad on the layer-0 (hardly any internal states yet), and to gradually perform well on the deeper, internal layers (a lot of internal state information).

To me, it looks like the trained head finds a short cut, since the layer-0 state carries essentially no image information, its accuracy there must come mostly from the option texts, i.e. learning which answer wordings tend to be correct.

  1. For the main results in Table 1 and Table 2:
  • Table 1: On the held-out column, the mean gap over LoRA comes largely from a single backbone (Pixtral); for the other six VLMs the gaps are much smaller.
  • Table 2: on AnomalyCoT, LORA and the proposed method is really a tie to me. Also, the statement that the proposed method is more stable also heavily depends on LoRA's crash on Pixtral.
  • Both Table 1 & 2: the absolute margin between the baseline and the proposed method is not so informative to me, as the proposed method has access to the ground-truth label yet the baseline models don't.

So the evidence looks quite weak.

Questions for authors

When splitting the datasets for training/testing, is the splitting based on images, or questions? Is it possible that some images in the test set have already been seen in the training phase, but only the questions are new?

6 comments
AuthorsOpenReview ↗

Response to Reviewer DiB6

We thank the reviewer for pointing out that improved accuracy does not establish recovery of hidden visual information. The new controls support a narrower interpretation.

W1. Shortcut through option texts and the central diagnosis. On Qwen2.5-VL/MMAD, a learned option-only correction added to the original image-conditioned frozen logits reaches 81.07%, versus 72.56% frozen and 82.38% for OCP-R. It reproduces 87% of the overall gain. Update (Nov 14). Across all seven MMAD backbones, this share is 86–96%. A simple option-string frequency prior reaches 78.63%. These results substantiate the concern that benchmark option regularities account for much of the improvement.

We also corrected “layer 0”: it is the first decoder block's output, not the embedding layer. This correction does not resolve your concern. We therefore used a learned constant query to construct an option-only correction.

On AnomalyCoT, option-only corrections reproduce 26–52% of the overall gain across nine backbones. That result alone does not identify a visual contribution. Update (Nov 16). On all nine backbones, now tested with an image-blind correction, question and option text reproduce most of the remaining overall improvement; full OCP-R is significantly lower overall on four (Qwen2.5-VL, Qwen2-VL, Pixtral, Idefics3), and no overall or held-out interval establishes an accuracy benefit from giving the correction access to images. The original images are still used by the frozen logits in every comparison.

We therefore withdraw the general claim that these accuracy gains demonstrate recovery of hidden visual anomaly evidence. The supported contribution is supervised adaptation of the option-level readout while retaining the frozen logits. The positive image-access result on MMAD's held-out source (+2.13 [0.71, 3.55]) is reported with that specific scope.

W2. Table 1 and Pixtral. Your concern about the held-out comparison is borne out by the fixed-input re-evaluation. Pixtral's held-out gap over rank-4 LoRA decreases from +11.58 to +2.72 points, and only one of seven backbones has a held-out interval excluding zero. We withdraw the general held-out superiority claim.

The overall comparison remains positive on all seven backbones, with intervals excluding zero on six. This table uses the original LoRA checkpoints and generate-and-parse evaluation. A separate Qwen2.5-VL experiment matches the option-level loss and forced-prefix protocol: OCP-R exceeds the tested last-four-layer rank-8 LoRA by +3.85 [2.29, 5.47] overall, while the held-out difference is inconclusive. Update (Nov 14). The all-layer, learning-rate-selected comparison is complete and reverses the order. Rank-1 LoRA on all 28 layers (0.43M parameters against OCP-R's 0.47M, same loss and scoring, three seeds) reaches 84.89 / 81.63, above OCP-R by 2.51 [1.10, 3.89] overall and 4.63 [1.85, 7.58] held-out. OCP-R's overall advantage over LoRA is therefore specific to the last-four-layer placement, and we claim no accuracy advantage at a matched parameter count. Larger-budget adaptation is also more accurate.

W3. Table 2 and stability. We agree and withdraw the stability claim against LoRA. The AnomalyCoT baseline is LoRA with a 30-token soft prompt, as clarified in General Response 2/2. Update (Nov 14). The Pixtral entry came from a run that did not train: its archived training loss is flat, and its 2,500- and 5,000-question runs give identical answer counts in all 13 subsets. The cause, an overflow in the joint gradient clipping of LoRA and soft prompt, occurs on Pixtral only. With the LoRA update restored (the soft prompt stays at its initial value), three seeds reach 52.24 ± 7.60 / 58.70 ± 1.07, so the entry does not show a LoRA failure on Pixtral. The evidence for retaining frozen logits comes from comparison with Vanilla OCP: across nine AnomalyCoT backbones and three seeds, OCP-R has higher held-out means on all nine, with a mean difference of +6.54 points (exact Wilcoxon p = 0.0039). This result does not establish superiority over LoRA.

W4. Access to labels. We agree that a supervised method's gain over an unadapted frozen readout cannot by itself establish a better learning method or the source of the information it uses. We treat that difference as supervised adaptation gain. Comparisons with LoRA, Vanilla OCP, option-only, and image-blind corrections use the same training labels within each experiment. The narrower claims above rest on those supervised comparisons.

Question. Split. The split is by query image, not by question. Every question about an evaluation image is assigned to evaluation, and checks by image path, MD5, and near-duplicate search found no train–evaluation image overlap on MMAD. No evaluation image appears in the AnomalyCoT training set either. In-domain categories can overlap with training, while the designated held-out sources are excluded from training.

AuthorsOpenReview ↗

Follow-up to Reviewer DiB6: where your concerns stand, and any remaining questions

Thank you again for your review. Your main concern, that the head may be learning from the option texts, led us to add a direct test, and the revised paper is built on what that test and the other controls support. Since our response of Nov 13 the remaining checks have finished and the PDF has been revised, so we summarize below what the revised paper claims and where your points stand. We would be grateful to know whether anything remains open.

What the revised paper claims, under equal supervision. OCP-R is a small correction added to the frozen option logits. With the same head and the same labels, adding the correction gives higher held-out accuracy than using the head as a replacement readout on all nine AnomalyCoT backbones (+6.54 points, exact Wilcoxon p = 0.0039). On the seven MMAD backbones it turns 4–8% of the questions from frozen-correct to wrong, against 13–18% for the replacement readout. It is above the last-four-layer LoRA baseline overall on all seven MMAD backbones (six intervals exclude zero, Table 20), and without modifying the backbone it obtains 80% of the all-layer LoRA's overall gain on Qwen2.5-VL (59% held-out).

Where your points stand.

  • Evidence for hidden information (W1). The direct test is in Tables 5 and 8: a correction that reads only the option texts reproduces 86–96% of the gain on the seven MMAD backbones. The paper therefore presents OCP-R as supervised adaptation of the readout, and the abstract and the contribution list no longer argue from hidden visual evidence.
  • Tables 1 and 2, Pixtral, stability (W2, W3). The paper no longer claims a held-out or stability advantage over LoRA, and it states that LoRA on all layers is more accurate at an equal parameter count. The Pixtral LoRA entry came from a run that did not train.
  • Access to labels (W4). The comparisons above are between methods trained on the same labels.
  • Split (your question). The split is by query image, and no evaluation image occurs in training on either benchmark.

On "learning which answer wordings tend to be correct". The gain comes largely from the option texts, and the paper says so. It is not a memory of which strings were correct. On the 834 evaluation questions whose answer string was never an answer in training, OCP-R still gains 8.59 points [5.73, 11.57]. In a manual rewrite audit (new Appendix H), the gain remains when the correct option is reworded: +7.56 points [1.61, 13.62] on average over seven backbones, against +8.80 with the original wording. A reworded distractor costs OCP-R 5.49 points on Qwen2.5-VL, which the Limitations state.

Presentation. The abstract and the introduction are rewritten to this reading, and Appendix G lists the changes from the submission in ten items.

If any concern is not yet addressed, or if the revised paper is still hard to follow in some place, we would be grateful to hear where, and we will answer before the discussion period ends. Thank you for your time and for the care of your review.

Thank the authors for the additional experiments and the careful revision. My first concern (W1) has been fully addressed but I still have some concerns for W2.

Specifically, I appreciate the corrected LoRA comparisons: OCP-R achieves higher overall MMAD accuracy than last-four-layer LoRA on all seven backbones. But the held-out results are less conclusive, and the matched-parameter comparison on Qwen2.5-VL favors all-layer LoRA, so I agree with the more specific conclusion in the revision, which is however also narrower. The new experiments give me a clearer understanding of where OCP-R helps, and I’m raising my rating from 1 to 2.

Reviewer_Bu6yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Option-Conditional Residual Readout (OCP-R) for multiple-choice VLM anomaly detection. It freezes the backbone and LM head and trains a small head that adds option-text-conditioned corrections to the original logits. The correction is initialized to zero. Experiments on MMAD and AnomalyCoT show substantial accuracy improvements and preservation of many baseline-correct predictions. The strongest evidence supports a useful lightweight adaptation method and residual anchoring over replacement readouts. A general cross-backbone stability advantage over LoRA is less conclusive, particularly when a failed LoRA run is excluded.

Critical strengths
  1. Lightweight, effective adaptation. OCP-R leaves the backbone and LM head unchanged. Table 1 shows that mean MMAD accuracy across seven general-purpose VLMs increases from 62.17% to 78.27%, with higher held-out accuracy than LoRA on six of seven backbones. The mean gains over LoRA are 6.34 percentage points overall and 2.93 percentage points held-out. Comparisons with similarly sized LoRA adapters and the three-seed Qwen2.5-VL experiment in Table 16 further support its practical value, without establishing robustness across seeds for every backbone.

  2. Evidence for residual anchoring. The prediction-transition analysis provides useful evidence for retaining frozen logits rather than replacing them. On Qwen2.5-VL/MMAD, Table 4 shows that OCP-R substantially outperforms Vanilla OCP, while the transition analysis in Table 13 reports preservation of 93.3% of baseline-correct predictions. Preservation is not guaranteed.

Critical weaknesses
  1. Incomplete isolation of the incremental visual contribution. The option-permutation, distractor-replacement, and image-ablation controls constrain several shortcuts. The authors also acknowledge that high layer-0 accuracy reflects option-text priors (Appendix A) and that meaning-preserving rewrites of correct options were not tested (Appendix B). However, the image ablations in Tables 11–12 evaluate the combined frozen-plus-residual predictor, rather than isolating the visual dependence of the improvement added by the residual. Recovery of frozen visual evidence and supervised benchmark adaptation can occur together; the unresolved issue is how much of the additional gain requires image-specific information rather than benchmark-specific phrasing. This limits the strength of the mechanistic interpretation, not the validity of the reported accuracy gains.

  2. Fragile stability advantage over LoRA. The comparison is sensitive to the failed Pixtral run, as the authors explicitly report in Appendix C. Excluding Pixtral from all methods in Table 2, LoRA has higher mean held-out accuracy than OCP-R (66.80% versus 65.54%) and a slightly better worst-case change (−2.83 versus −3.24 pp); neither method has a catastrophic drop under the stated threshold of Δheld-out < −5 pp. OCP-R retains higher overall accuracy (75.47% versus 73.27%) and a narrower change range. However, that range also increases with large positive improvements, so a narrower range alone does not establish better downside protection. The authors acknowledge that the Pixtral failure may be an optimization artifact; the remaining evidence does not yet establish a broader stability advantage over LoRA.

The residual anchor has stronger support against replacement readouts than the general stability claim against LoRA. On AnomalyCoT, its mean gain over Vanilla OCP is 0.29 pp overall and 3.51 pp held-out; even without Pixtral, Vanilla OCP has three catastrophic drops out of eight backbones, versus none for OCP-R. Together with the large Qwen2.5-VL/MMAD gains, this supports residual anchoring in the reported settings. The cross-backbone stability comparisons remain descriptive: the reported paired-permutation test gives p = 0.25 for spread against each comparator across all nine backbones, while Fisher's p = 0.21 concerns catastrophic drops against Vanilla OCP, not LoRA. These tests do not establish a stability advantage at conventional significance levels, but neither do they establish the absence of an effect.

Questions for authors
  1. Could you report the frozen baseline and OCP-R side by side under the same image-ablation conditions, so that changes in the incremental gain can be assessed rather than only changes in total OCP-R accuracy? The existing controls support visual dependence of the combined predictor, but a matched comparison would clarify the visual contribution to the residual's improvement. A limited, controlled query-image intervention would also be informative; the current shuffle changes the reference image.

  2. Given the acknowledged wording limitation, could you evaluate a small, manually audited subset with meaning-preserving rewrites of the correct option, keeping the image and answer position fixed and controlling for option length? Please compare the frozen baseline and OCP-R on both the original and rewritten versions. As a separate clarification, do any option sets admit multiple semantically defensible answers, and how is uniqueness determined? Lexical overlap alone does not establish such ambiguity; I have not verified a multiple-correct-answer case.

  3. Given the Pixtral sensitivity already discussed in Appendix C, what supports better downside protection than LoRA beyond a narrower change range? A limited optimization-sensitivity check of the failed run, or a more tightly scoped interpretation, would help. Table 16 already provides a three-seed Qwen2.5-VL/MMAD comparison; additional seeds on selected AnomalyCoT backbones would be useful supporting evidence, rather than requiring a full new cross-backbone study.

11 comments
AuthorsOpenReview ↗

Response to Reviewer Bu6y

We thank the reviewer for requesting matched image interventions. We compared the frozen readout and OCP-R on identical inputs and measured the accuracy increment.

Update (Nov 16). Later results: the image-blind control now covers nine backbones and the Pixtral check is complete (General Response 3/3); "the increment persists" holds on Qwen2.5-VL and is weaker on Qwen2-VL (our follow-up 1/2).

W1/Q1. Visual dependence of the increment. OCP-R was trained on the original image inputs; the images are changed only at evaluation. The results below report Qwen2.5-VL/MMAD overall accuracy, with OCP-R averaged over three seeds. Increment intervals are 95% paired cluster-bootstrap CIs over query images. Differences are computed before rounding.

Evaluation inputFrozenOCP-RIncrement [95% CI]
Original images72.5682.38+9.82 [7.81, 11.89]
Query image blacked out57.9273.26+15.35 [12.52, 18.13]
Both images blacked out49.1361.61+12.47 [10.26, 14.56]

The increment persists when visual evidence is removed, although absolute accuracy falls. This confirms your concern that the earlier ablations tested the combined predictor and did not isolate visual dependence of its improvement. The tested rank-4 LoRA checkpoint also retains an increment when both images are blacked out (+8.46, versus +5.97 with original images); this observation is not specific to OCP-R.

We separately swapped only the query image for 141 same-category normal/defective pairs, giving 282 bidirectional interventions while holding the question, option order, and reference image fixed. The frozen margin moves toward the new correct answer on 73.0% of inputs; the correction alone does so on 46.9%, averaged over three seeds. This intervention does not support interpreting the correction as an image-responsive anomaly detector. On the separate 386-question binary anomaly-detection subset, full OCP-R minus frozen AUROC is +0.0104 [−0.0079, 0.0281] for seed 42, also inconclusive.

The independently trained image-blind controls provide a complementary test without relying only on corrupted images. On MMAD, image access adds +2.13 [0.71, 3.55] on the held-out source but no established overall benefit. On AnomalyCoT:

OCP-R minus image-blind, in percentage points (95% CI):

BackboneOverall differenceHeld-out difference
Qwen2.5-VL−1.48 [−2.65, −0.31]−1.75 [−4.12, 0.57]
Qwen2-VL−1.28 [−2.56, −0.07]+1.62 [−1.71, 4.74]
LLaVA-1.5+0.61 [−0.63, 1.89]+3.10 [−0.97, 7.22]

Here, the frozen logits always retain the original images. We withdraw the general hidden-visual-evidence interpretation and limit the positive image-access finding to the measured MMAD held-out comparison.

Q2. Meaning-preserving rewrites and answer uniqueness. We prepared a 200-question set with two conditions: rewriting the correct option and rewriting one distractor as a control. Images and answer positions remain fixed, and the token-length change is restricted to at most two. The rewrites were drafted with an LLM, and the manual semantic audit is still pending. We therefore do not treat the preliminary scores as evidence of robustness to meaning-preserving rewrites. Wording sensitivity remains an unresolved limitation.

Evaluation follows the benchmark's designated labels. We have not established that every option set has a unique semantically defensible answer. The planned audit checks both preservation of meaning and whether multiple answers remain defensible; we cannot yet report an audited ambiguity count.

Q3. Downside protection. We withdraw the stability claim against LoRA. The AnomalyCoT baseline includes a 30-token soft prompt. In the Pixtral rerun, overflow in the soft-prompt gradient norm caused clipping to zero the updates. We are still checking whether the submitted run had the same failure. For the comparison with Vanilla OCP, we have three seeds on all nine AnomalyCoT backbones. The table reports held-out accuracy changes, in percentage points, from the forced-prefix frozen baseline, using each backbone's three-seed mean.

Held-out statisticOCP-RVanilla OCP
Mean change (pp)+1.47−5.07
Worst change (pp)−3.51−13.77
Backbones with a negative change2/98/9
Backbones with a change below −5 points0/95/9

OCP-R exceeds Vanilla OCP on all nine held-out means (+6.54 points; two-sided exact Wilcoxon p = 0.0039), including all eight backbones after excluding Pixtral. The overall difference is only +0.48 points (p = 0.43). The worst-case entries above are descriptive minima across backbone-level means, not across individual seed runs. These results support the held-out comparison with replacement readouts, without establishing guaranteed preservation or a stability advantage over LoRA.

Thank you for the detailed responses to each point of my review. After reading the responses, there are a few things I would still like to check.

W1/Q1.

  • Since the increment remains even when the images are removed, you withdrew the general interpretation about visual evidence. However, the paper states that each component is already known and that the contribution is the finding that the diagnosis points to this combination. Now that the core of the diagnosis has been withdrawn, could you explain what the main contribution of this paper is?

Q2.

  • You mentioned that the manual semantic audit for the meaning-preserving rewrite experiment is not finished yet. But since W1/Q1 showed that the increment remains even without images, this experiment is important for telling whether the correction reads the meaning of the options or is fitted to the wording of the benchmark. Once the audit is done, could you share the results together with the answer-uniqueness check?

Q3.

  • You mentioned that the AnomalyCoT baseline includes a 30-token soft prompt. However, the LoRA setup described in Section 4.1 and Appendix E does not include a soft prompt. If it was included, the LoRA in Table 2 differs from the configuration described in the paper (r = 4, about 0.246M parameters). Could you explain which runs included the soft prompt?
AuthorsOpenReview ↗

Response to Reviewer Bu6y’s follow-up (1/2): Main contribution

Thank you for reading our responses so closely and for taking the time to follow up. Each of your three questions points at something we had left unclear, and we are glad to have the chance to answer them. This part answers W1/Q1; the next part answers Q2 and Q3.

W1/Q1. The main contribution after the withdrawal. This is a fair question, and we should have answered it before you had to ask. The paper no longer claims a diagnosis. It also no longer claims an accuracy advantage over LoRA beyond one placement: rank-1 LoRA on all 28 layers, at OCP-R's parameter count, exceeds OCP-R by 2.51 [1.10, 3.89] overall and 4.63 [1.85, 7.58] held-out on Qwen2.5-VL/MMAD (General Response 3/3). What remains is a controlled measurement of what adapting the option-level readout gains on these two MCQ anomaly benchmarks and what it does not gain, with OCP-R as the instrument. Three results carry it. Update (Nov 16). The sentence that describes what remains as "a controlled measurement" with OCP-R "as the instrument" was wrong, and we correct it in our reply of Nov 16 below: the main contribution of the paper, OCP-R and its comparison with the replacement readout, has not changed. The three results that follow stand.

  1. Where the supervised gain comes from. On MMAD, a correction that reads only the option texts reproduces 86–96% of OCP-R's overall gain on seven backbones. It also reproduces 69% of the all-layer LoRA's gain on Qwen2.5-VL (an analysis made after seeing that result), so this share describes the benchmark more than our head. On AnomalyCoT the option texts explain 26–52% of the gain, and the rest comes from the question text: on the eight backbones tested, no interval for OCP-R minus an image-blind correction lies above zero, overall or held-out. Update (Nov 16). The control now covers all nine backbones, with the same result (General Response 3/3).

  2. Keeping the frozen logits. With the head and the training labels fixed, adding the correction to the frozen logits gives higher held-out accuracy than using the head as a replacement readout on all nine AnomalyCoT backbones (+6.54 points, exact Wilcoxon p = 0.0039). On seven MMAD backbones the replacement readout turns 13–18% of all questions from frozen-correct to wrong, against 4–8% for OCP-R.

  3. What a readout-level correction does not reach. On the anomaly-detection questions, where one Yes/No option set holds the option prior fixed, OCP-R changes AUROC by +0.010 to +0.038 on seven backbones and every interval includes zero. The all-layer LoRA raises AUROC from 0.788 to 0.832 on Qwen2.5-VL in the three-seed mean (seed 42: paired-bootstrap difference +0.041 [0.011, 0.075]).

As a method, OCP-R is what these limits leave: a 0.47M-parameter head that does not modify the backbone and obtains 80% of the all-layer LoRA's overall gain over the frozen readout on Qwen2.5-VL/MMAD (59% held-out). We do not present it as a new component or as a recovery of visual evidence. The revised abstract and contribution list state the contribution this way. Update (Nov 16). More exactly: the revised PDF proposes OCP-R as its method in the abstract and in the contribution list.

A correction to our first response. We wrote that the increment persists when the images are removed. That statement rested on Qwen2.5-VL alone. On Qwen2-VL the increment shrinks from +9.56 [7.35, 11.81] to +3.94 [1.11, 6.68] when both images are blanked, and its held-out interval then includes zero (+0.69 [−3.99, 5.14]). In the query-image swap on Qwen2-VL, the correction moves toward the new answer on 76.7% of inputs (frozen margin: 77.3%), and its shifts follow those of the frozen margin (r = 0.86), while AUROC changes by +0.014 [−0.006, 0.033]. Whether the correction responds to the image therefore depends on the backbone. On neither backbone does discrimination improve significantly, so the withdrawal of the visual-evidence interpretation stands.

Reviewer_UpceOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The method is simple and cheap. The backbone is untouched, and heads can be swapped per task on a shared model. That is a real deployment advantage.
  • The MMAD gains are large: mean accuracy over seven VLMs goes from 62.17 (frozen) and 71.93 (LoRA) to 78.27. The paired transition analysis and McNemar test on Qwen2.5-VL are convincing for that backbone.
  • The authors are upfront about limitations, including weak significance of the stability result, the failed Pixtral LoRA run, and poor cross-benchmark transfer. I appreciated this.
Critical strengths

The method is simple and cheap. The backbone is untouched, and heads can be swapped per task on a shared model. That is a real deployment advantage.

The MMAD gains are large: mean accuracy over seven VLMs goes from 62.17 (frozen) and 71.93 (LoRA) to 78.27. The paired transition analysis and McNemar test on Qwen2.5-VL are convincing for that backbone.

The authors are upfront about limitations, including weak significance of the stability result, the failed Pixtral LoRA run, and poor cross-benchmark transfer. I appreciated this.

Critical weaknesses
  1. The main diagnosis is not separated from option-text priors. In Table 5, the option-conditioned probe already reaches 69.14% at layer 0, before any decoder block, against 72.40% for the final frozen readout. With no images at all, OCP-R still gets 61.44% (Table 11). So a large share of what the head "recovers" may be regularities in how the benchmark phrases its options, learned from 5,000 in-benchmark training questions, rather than visual evidence hidden in the state. The Text Only control does not settle this, because it also removes the images from the frozen logits. The experiment I would like to see is a correction trained on layer-0 state (or on option text alone), added to the normal final-layer frozen logits under the same supervision, and compared with full OCP-R.

  2. The AnomalyCoT baseline looks broken by parsing. The zero-shot baseline gets 39.28% overall but 62.08% held-out. OCP-R gains +35.5 points overall but only +3.4 held-out. That pattern points to format or parsing failures in-domain. The parse-vs-prefix check was only run for Qwen2.5-VL on MMAD. Please report a forced-prefix frozen baseline on AnomalyCoT for all nine models.

  3. The comparison with LoRA is not as clean as presented. First, LoRA is also zero-initialized, so the "residual anchor" argument separates OCP-R from Vanilla OCP, not from LoRA. Second, identity init contributes about 0.3 points (Table 4, Kaiming W_k: 82.27 vs. 82.59), so what matters is adding to the frozen logits. A post-hoc sum of frozen and Vanilla OCP logits with one tuned weight is a missing baseline. Third, LoRA is limited to q/k/v in the last four layers, with no learning-rate tuning, and one run fails outright. Even so, rank-32 LoRA is within 0.32 points of OCP-R on MMAD held-out (75.72 vs. 76.04).

  4. The stability claim is weak. The paper's own tests give p = 0.25 (permutation) and p = 0.21 (Fisher). Once the failed Pixtral run is dropped, LoRA has no catastrophic drops and a higher mean held-out accuracy than OCP-R (66.80 vs. 65.54). The metric is also odd: using the range of held-out changes penalizes a method for large gains on some backbones (LoRA's range includes +14.98 on InternVL3). Worst-case change would be a fairer measure.

  5. The train/test split needs to be spelled out. GoodsAD is held out at the source level, but for the other sources it is unclear whether training and test questions can share images or objects. Please state which results are question-, image- or source-disjoint.

  6. Section 3's geometry is mostly a restatement of the model. By construction, delta_ij depends only on the option texts, so "same option set, same rotation" is expected. A replacement normal being near 90 degrees from u_ij is also what one gets by default in d = 3584. Combined with the fact that each component is known (residual logit adapters, zero-init, logit-level option debiasing), the novelty rests on the diagnosis, which is the part I find least supported (point 1).

  7. Minor: the Qwen2.5-VL result appears as 81.62, 82.59, 82.27, 82.92 and others across pipelines. One table of record, with a clear mapping to the claims it supports, would make the paper much easier to check.

Questions for authors
  1. How much of the gain survives against frozen logits plus a matched prior-only correction (layer-0 state or option text only)?
  2. What does the forced-prefix frozen baseline score on AnomalyCoT across all nine models?
  3. How does a post-hoc ensemble of frozen and Vanilla OCP logits compare with OCP-R?
  4. With LoRA on more layers and a per-model learning rate, does the stability comparison still hold?
  5. Are MMAD train and test questions disjoint at the image and object level?
  6. How does accuracy scale with the amount of training data (e.g., 100 / 500 / 5,000)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Upce (1/2)

We thank the reviewer for proposing the matched prior-only control. We report that control and the forced-prefix baselines for all nine AnomalyCoT backbones below.

W1/Q1. How much survives a matched prior-only correction? On Qwen2.5-VL/MMAD, OCP-R exceeds the option-only correction by +1.30 [0.49, 2.13] points overall, about 13% of its total gain over the frozen readout. The held-out difference is +1.01 [−0.30, 2.34]. The option-only control uses the same supervision and is added to the same image-conditioned frozen logits. We use a learned constant query because the submitted “layer 0” is actually the first decoder block's output and therefore does not remove query information.

A count prior using each option string's training-set log-odds of correctness, with its weight chosen on validation data, reaches 78.63% overall versus 81.07% for the learned option-only correction. Thus, even this simple prior reproduces much of the gain. The state-only residual without a separate option-text branch reaches 72.90%, close to the frozen 72.56%; we do not interpret this as proof of hidden visual evidence. Update (Nov 14). Across all seven MMAD backbones, the option-only correction reproduces 86–96% of the overall gain.

On AnomalyCoT, the option-only correction reproduces 26–52% of the overall gain across nine backbones. OCP-R exceeds it overall on all nine and on held-out sources on all nine in point estimates, with held-out intervals excluding zero on seven. Adding the question while withholding images from the correction reproduces most of the remaining overall improvement. Update (Nov 16). This image-blind control now covers all nine backbones. No interval for OCP-R minus image-blind lies above zero, overall or held-out; overall, image-blind is significantly higher on four (Qwen2.5-VL, Qwen2-VL, Pixtral, Idefics3).

W2/Q2. Forced-prefix frozen baselines on all nine AnomalyCoT backbones. All entries below use the same 1,482 questions. Accuracy and parse-failure entries are overall / held-out (%). Unparseable generated answers retain the original fallback to option A.

G&P denotes generate-and-parse.

BackboneG&P accuracyUnparseable outputsForced-prefix accuracy
Qwen2.5-VL44.94 / 64.370.00 / 0.0045.41 / 65.99
Qwen2-VL46.22 / 71.660.00 / 0.0046.49 / 71.26
LLaVA-1.531.78 / 56.280.07 / 0.0030.77 / 55.06
LLaVA-NeXT31.31 / 51.4212.01 / 6.0732.32 / 60.73
InternVL-339.88 / 59.921.62 / 0.0046.36 / 64.78
InternVL-2.541.57 / 68.420.20 / 0.4041.77 / 68.42
Pixtral34.48 / 57.892.02 / 0.8137.85 / 61.94
Idefics335.63 / 62.750.07 / 0.0034.62 / 63.16
LLaVA-OneVision46.63 / 68.020.00 / 0.0045.82 / 69.23
Mean39.16 / 62.301.78 / 0.8140.16 / 64.51

The gap between in-domain and held-out performance remains under forced-prefix evaluation, and the large overall gains from supervised adaptation remain. Protocol differences also include changes in answer generation: for example, InternVL-3's held-out score changes despite zero parse failures. We therefore do not attribute every generate-and-parse versus prefix difference to parsing. Your concern did identify substantial baseline underestimation for two MMAD models, which we correct in General Response 2/2.

W3/Q3. Residual anchoring and the post-hoc ensemble. We agree that LoRA also starts from the frozen model. On Qwen2.5-VL/MMAD, 4.01% of all questions change from frozen-correct to wrong under OCP-R, versus 4.07% under rank-4 LoRA and 16.16% under Vanilla OCP. These are the seed-42 transition counts; LoRA uses the original generate-and-parse protocol. We withdraw the claim that backbone tuning necessarily displaces correct frozen decisions. The comparison supports anchoring relative to a replacement readout.

We also evaluated frozen logits plus weighted Vanilla OCP logits. With 4,500 training questions and 500 validation questions used to select the weight, the ensemble reaches 80.37 / 74.01, versus 81.87 / 77.05 for OCP-R trained on the same 4,500 questions; both are three-seed means. The three-seed mean differences favor OCP-R, but the available paired intervals cover only seed 42 and include zero. Update (Nov 14). On seven MMAD backbones (seed-42 heads), OCP-R is significantly above the ensemble on three (LLaVA-1.5, InternVL-3, LLaVA-NeXT; +2.55 to +3.42 overall, +5.27 to +7.35 held-out) and indistinguishable from it on the other four. On three AnomalyCoT backbones, their overall means differ by at most 1.01 points and held-out differences have mixed signs. We present zero initialization as a default that exactly preserves the frozen logits before training, without claiming that it is the main source of the final accuracy gain.

AuthorsOpenReview ↗

Response to Reviewer Upce (2/2)

W3/Q4. LoRA objective, budget, and optimization. We corrected the submitted loss description and trained rank-4 and rank-8 LoRA with OCP-R's option-level objective. These runs adapt q/k/v in the last four layers for one epoch at 2e−4; OCP-R uses the five-epoch cached-head recipe described in General Response 2/2. All methods below use the same inputs and forced-prefix evaluation.

MethodTrainable parametersOverall / held-out
OCP-Rapproximately 473K82.38 / 77.00
LoRA, rank 4, option-level loss245,76077.66 / 74.76
LoRA, rank 8, option-level loss491,52078.52 / 76.68
LoRA, rank 1, all layers, option-level loss (update)430,08084.89 / 81.63

OCP-R's overall differences from the two last-four-layer rows are +4.72 [3.09, 6.40] and +3.85 [2.29, 5.47], respectively. Both held-out intervals include zero. Update (Nov 16). These two rows are single checkpoints; with a selected learning rate and three seeds, OCP-R is 4.07 [2.61, 5.57] above rank-4 LoRA overall (held-out +1.60 [−1.09, 4.22], inconclusive).

Update (Nov 14). The all-layer, matched-budget comparison is complete and reverses the order. Rank-1 q/k/v LoRA on all 28 layers (same loss and scoring; learning rate 5e−4, selected among 1e−4, 2e−4, and 5e−4 on 500 held-back training questions; three seeds) reaches 84.89 ± 0.28 / 81.63 ± 0.32, above OCP-R by 2.51 [1.10, 3.89] overall and 4.63 [1.85, 7.58] held-out. The sweep run at 2e−4, the rate of the last-four-layer rows, also exceeds OCP-R (84.00 / 80.03). OCP-R's accuracy advantage over LoRA is therefore specific to the last-four-layer placement; at the same parameter count, all-layer LoRA is more accurate.

The preserved larger-budget runs are more accurate: on Qwen2.5-VL/MMAD, all-linear rank-64 LoRA reaches 86.37 / 80.67 and full fine-tuning reaches 85.12 / 80.46. These historical runs used unfixed reference draws, so they are not part of the new fixed-input comparison.

W4. Stability. We withdraw the stability claim against LoRA. The AnomalyCoT baseline includes a 30-token soft prompt. In the Pixtral rerun, the soft-prompt gradient norm overflowed and clipping reduced all updates to zero. Update (Nov 14). The check is complete: the submitted Pixtral run did not train. Its archived training loss is flat (2.61–2.65), and its 2,500- and 5,000-question runs give identical answer counts in all 13 subsets. The overflow occurs on Pixtral only. With parameters whose gradient norm overflows left out of the update, the LoRA weights train while the soft prompt stays at its initial value: three seeds reach 52.24 ± 7.60 / 58.70 ± 1.07 (LM loss, generate-and-parse, selected learning rate). The three-seed comparison with Vanilla OCP gives higher held-out means for OCP-R on all nine AnomalyCoT backbones (+6.54 points; exact Wilcoxon p = 0.0039). This tests paired held-out accuracies across backbones, not seed variance or worst-case significance.

W5/Q5. Image and object separation. MMAD is split by query image. All questions about an evaluation image are assigned to evaluation, and the training pool excludes those images. Checks by path, MD5, and near-duplicate search found no train–evaluation query-image overlap. In-domain evaluation categories are represented in training; GoodsAD is held out by source and its categories are absent from training. Our verified guarantee is image disjointness. We have not established physical object-instance disjointness across different photographs and do not claim it.

W6. Geometry. We agree that the angles follow from the parametrization. We no longer use them as empirical evidence for visual recovery or as an independent novelty claim. Equation (4) describes how the correction depends on the candidate set; the new controls test which inputs reproduce the observed gains.

Minor/Q6. Table of record and scaling. General Response 2/2 distinguishes the pipelines behind the different Qwen2.5-VL values. For scaling, the Table 1 recipe with 1,250 optimizer steps gives:

Training questionsOCP-R: overall / held-outOption-only: overall / held-out
10063.79 / 61.9367.97 / 66.08
50074.26 / 70.7174.17 / 70.66
5,00081.07 / 75.5681.62 / 77.00

The 100- and 500-question rows average three sampling seeds; the 5,000-question row is a single run. Holding steps fixed gives 50 epochs at 100 questions. A separate five-epoch run on 100 questions reaches 76.36 / 73.86 for OCP-R and 75.33 / 73.27 for option-only. The low-data result depends on training duration; we do not infer a universal minimum label requirement or a consistent OCP-R advantage over option-only from this experiment. Update (Nov 14). The baseline LoRA (rank 4, last four layers) on one 500-question draw with the same step count reaches 75.33 / 73.64, so at this step count OCP-R (74.26 / 70.71) has no advantage either.

AuthorsOpenReview ↗

Follow-up to Reviewer Upce: where your requests stand, and any remaining questions

Thank you again for your detailed review. The controls and baselines you asked for are now part of the paper, and we are grateful for them. Since our responses of Nov 13 the remaining checks have finished and the PDF has been revised, so we summarize below what the revised paper claims and where each of your requests stands. We would be grateful to know whether anything remains open.

What the revised paper claims. OCP-R is a simple, cheap correction that leaves the backbone untouched and can be swapped per task, the strengths your review listed. Its gain over the frozen readout holds against the corrected baseline: +10.73 points on average over seven MMAD backbones (67.50% to 78.23%). The residual form is supported against both alternatives you named. OCP-R has higher held-out accuracy than the replacement readout on all nine AnomalyCoT backbones (+6.54 points, exact Wilcoxon p = 0.0039), and it is significantly above the post-hoc sum of frozen and replacement logits on three of seven MMAD backbones and level with it on the other four (Table 14).

Where your requests stand.

  • Matched prior-only control (W1/Q1). It is in Tables 5 and 8: option texts alone reproduce 86–96% of the gain on the seven MMAD backbones. The paper therefore presents OCP-R as supervised adaptation of the readout and no longer argues from hidden visual evidence.
  • Forced-prefix frozen baselines (W2/Q2). They are reported for all nine AnomalyCoT models (Tables 2 and 26). The gap between in-domain and held-out accuracy remains under forced-prefix scoring, so it is not a parsing artifact.
  • LoRA on more layers with a selected learning rate (W3/Q4). With a selected learning rate and three seeds, OCP-R is 4.07 points [2.61, 5.57] above LoRA on the last four layers (Table 21). Rank-1 LoRA on all layers is more accurate at the same parameter count, which the paper states; OCP-R obtains 80% of that LoRA's overall gain on Qwen2.5-VL without modifying the backbone.
  • Stability, split, geometry, table of record (W4–W6, minor). The paper no longer claims a stability advantage over LoRA. The split is by query image, with no evaluation image in training (Section 4.1). The geometry is no longer used as evidence. Table 29 maps the Qwen2.5-VL numbers of the submission to their pipelines.

New since our response. A manual rewrite audit, requested by Reviewer Bu6y and related to your point about benchmark phrasing, is in the new Appendix H. On audited items the gain over the frozen readout remains when options are reworded: averaged over seven backbones it is +8.80 points [3.73, 13.96] with the original wording, +7.56 [1.61, 13.62] with the correct option reworded, and +6.05 [0.91, 11.30] with a distractor reworded. On Qwen2.5-VL OCP-R stays 11 to 15 points above the replacement readout in every condition, and a reworded distractor costs it 5.49 points, which the Limitations state.

If any concern is not yet addressed, or if a result above raises a new question, we would be glad to answer before the discussion period ends. Thank you for the time and care you have given this paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (1/2): What the new evidence supports

We thank the reviewers for their comments. We added matched option-only and image-blind controls and repeated the comparison with replacement readouts across nine backbones and three seeds. These experiments support a lightweight supervised readout adapter that retains the frozen logits. They do not support the general claim that the accuracy gains demonstrate recovery of visual evidence hidden in the frozen state.

Matched input controls. Following Reviewer Upce's suggestion, we trained corrections with different inputs and added each to the same frozen logits computed with the original images. The option-only correction uses a learned constant query; the image-blind correction uses the question and options with image tokens removed. For the option-only, image-blind, and full corrections, supervision and frozen logits are held fixed while the correction's input changes. The complete predictor still receives images through the frozen logits.

The results below report overall / held-out accuracy (%), averaged over seeds 42, 1, and 7. Intervals are 95% paired cluster-bootstrap CIs over query images, using per-question correctness averaged across seeds; they quantify image-sampling uncertainty conditional on these runs. Differences are computed before rounding.

MMAD, Qwen2.5-VL (overall / held-out):

Correction inputOverall / held-out
No correction: frozen readout72.56 / 70.45
State only, without a separate option-text branch72.90 / 70.93
Option texts only81.07 / 75.99
Question and option texts, without images81.69 / 74.87
Image, question, and option texts: OCP-R82.38 / 77.00

On MMAD, the option-only correction reproduces 87% of the overall gain. Update (Nov 14). Across all seven MMAD backbones this share is 86–96%. A training-set option-string frequency prior reaches 78.63%, reproducing about 71% of the option-only gain. Giving the correction access to images adds +0.69 [−0.15, 1.50] overall and +2.13 [0.71, 3.55] on the held-out source relative to the image-blind correction. The latter is evidence for a benefit in this specific held-out setting; the overall interval includes zero.

AnomalyCoT (overall / held-out):

MethodQwen2.5-VLQwen2-VLLLaVA-1.5
Frozen readout45.41 / 65.9946.49 / 71.2630.77 / 55.06
Option-only correction56.23 / 64.5155.76 / 63.0250.02 / 54.66
Image-blind correction76.61 / 68.2977.98 / 66.1374.25 / 58.30
OCP-R75.12 / 66.5376.70 / 67.7574.85 / 61.40

Across all nine AnomalyCoT backbones, the option-only correction reproduces 26–52% of the overall gain. However, the image-blind control on the three backbones above reproduces most of the improvement beyond option-only conditioning. OCP-R minus image-blind is −1.48 [−2.65, −0.31], −1.28 [−2.56, −0.07], and +0.61 [−0.63, 1.89] overall, respectively. All three held-out intervals include zero. These comparisons do not establish an accuracy benefit from image access by the correction on these AnomalyCoT backbones. Update (Nov 16). The control now covers all nine backbones: no interval for OCP-R minus image-blind lies above zero, overall or held-out, and overall the image-blind correction is significantly higher on four (General Response 3/3).

Accuracy and residual anchoring. On AnomalyCoT, OCP-R has higher held-out accuracy than the replacement readout, Vanilla OCP, on all nine backbones when averaged over three seeds: mean difference +6.54 points, two-sided exact Wilcoxon signed-rank p = 0.0039. On Qwen2.5-VL/MMAD, OCP-R also exceeds the tested last-four-layer rank-8 LoRA baseline at a similar parameter count, with the same option-level objective and forced-prefix evaluation: +3.85 [2.29, 5.47] overall. Its held-out difference, +0.32 [−2.65, 3.23], is inconclusive. Update (Nov 14). With LoRA on all 28 layers at the same parameter count (rank 1, 0.43M parameters, selected learning rate, three seeds), the order reverses: LoRA reaches 84.89 / 81.63, above OCP-R by 2.51 [1.10, 3.89] overall and 4.63 [1.85, 7.58] held-out (General Response 3/3). OCP-R's accuracy advantage over LoRA is specific to the last-four-layer placement.

We therefore retain the contribution of a small, exchangeable readout correction on a frozen backbone and the empirical comparison with replacement readouts. We withdraw the general claims of hidden visual evidence recovery, held-out superiority over LoRA, and greater stability than LoRA. We also make no general superiority claim over larger or more extensively optimized adaptation.

AuthorsOpenReview ↗

General Response (2/2): Corrections and reporting

Checking the evaluation code and retained logs exposed errors in our protocol description and reported results. We apologize for these errors. All table numbers in these responses refer to the submitted manuscript.

  1. LoRA configuration, objective, and evaluation. The submitted LoRA used a language-modeling loss and generate-and-parse evaluation, although Section 4.1 described option-level cross-entropy and forced-prefix evaluation. On AnomalyCoT, the baseline labeled “LoRA” also includes a 30-token soft prompt, which was omitted from the manuscript description. We now distinguish these settings and report additional LoRA runs with the same option-level objective as OCP-R. The fixed-input, seven-backbone Table 1 comparison retains generate-and-parse for the original LoRA checkpoints; the separate Qwen2.5-VL comparison matches both the objective and forced-prefix evaluation.

  2. Reference-image sampling. MMAD reference images were selected using Python's per-process string hash, so different runs could see different reference images. We re-evaluated Table 1 with one fixed reference draw. OCP-R remains more accurate overall than rank-4 LoRA on all seven backbones, with paired intervals excluding zero on six. The held-out difference is positive on five and excludes zero on only one. We withdraw the general held-out advantage over LoRA.

  3. Frozen baselines. In the MMAD re-evaluation, 19.7% of InternVL-3 and 66.2% of LLaVA-NeXT generated answers cannot be parsed. Against forced-prefix frozen readouts, OCP-R's mean gain across seven backbones is +10.73 / +6.05 points overall / held-out, replacing the submitted +16.10 / +11.99. The corrected mean overall accuracies are 67.50% for the frozen readout and 78.23% for OCP-R.

  4. Pixtral: MMAD discrepancy and AnomalyCoT training failure. On MMAD, frozen and LoRA accuracies are 6–9 points higher in the re-evaluation than in the submission. The cause remains unresolved; we do not attribute the whole difference to reference sampling. The held-out OCP-R–LoRA gap is now +2.72 points, replacing +11.58. Separately, in the Pixtral AnomalyCoT rerun, the soft-prompt gradient norm overflowed, causing gradient clipping to reduce all updates to zero. Update (Nov 14). The submitted run had the same failure and did not train (General Response 3/3).

  5. Untraceable entries. We could not match several submitted entries to preserved logs: the Kaiming row and the no-bias held-out value in Table 4, shuffled options in Table 11, rank 32 in Table 17, and entries in Tables 13–14. We withdraw these unsupported entries, report documented reruns where available, and remove the rank-32 result rather than reconstructing it.

  6. Labels and parameter counts. “Layer 0” in Table 5 is the output of the first decoder block, not the embedding layer. The LoRA column in Table 9 is rank 4. The rank-32 parameter count is 1,966,080.

  7. Forgetting interpretation. The reruns do not support the claim that OCP-R mainly forgets attribute questions. We withdraw that statement, including its safety interpretation. On the binary anomaly-detection subset, no new false negatives were observed for OCP-R across the three tested seeds; this is a finite-sample observation, not a preservation guarantee.

Pipelines and reported values. The fixed-input Table 1 re-evaluation uses the stored Table 1 head (one epoch, batch 4, post-normalization state), giving Qwen2.5-VL 81.29 / 76.20. The cached-control experiment uses a different recipe (five epochs, batch 64, pre-normalization state) and gives 82.38 / 77.00 over three seeds. The 82.27 value in Tables 8–9 comes from training on all five MMAD sources and does not have a held-out source. These values refer to different configurations and must not be interchanged. Our table of record maps claims to their configuration, checkpoint or run, evaluation protocol, and seed count.

AnomalyCoT evaluation samples. The new generate-and-parse and forced-prefix comparison uses the same 1,482 questions: 1,235 in-domain and 247 held-out. Of the 1,500 sampled questions, 18 had option lists that our evaluation code could not parse and were excluded from both methods. These input-parsing exclusions are distinct from failures to parse a model's generated answer. The nine-backbone anchor comparison uses three seeds for OCP-R and Vanilla OCP. The submitted cross-backbone LoRA + soft prompt results use a single seed and are reported separately.

Update (Nov 14). The revised manuscript is uploaded. The check of the submitted Pixtral AnomalyCoT run and the learning-rate-selected LoRA comparison are complete (General Response 3/3); the manual rewrite audit remains in progress.

AuthorsOpenReview ↗

General Response (3/3): Completed checks and revised manuscript

This comment reports the checks that General Response 2/2 listed as in progress and announces the revised manuscript. Passages of our earlier responses that changed today are marked “Update (Nov 14)”; later changes are marked “Update (Nov 16)”.

1. The submitted Pixtral LoRA run on AnomalyCoT did not train. Its archived training loss stays between 2.61 and 2.65 from step 500 to the end, and the runs of that recipe on 2,500 and 5,000 questions give identical answer counts in all 13 subsets (0–2 of 13 on the six other backbones with both runs). The cause is the 30-token soft prompt that this baseline trains jointly with LoRA. On Pixtral its gradient is so large that the joint norm overflows in single precision, the clipping factor becomes zero, and no parameter is updated. A four-step check on all nine backbones finds the overflow on Pixtral only. With parameters whose gradient norm overflows left out of the update, the LoRA weights train while the soft prompt stays at its initial value: three seeds at a learning rate selected on held-back questions reach 52.24 ± 7.60 / 58.70 ± 1.07 (frozen 34.67 / 57.89; LM loss, generate-and-parse, on the 1,500 sampled questions). The Table 2 entry (33.42 / 52.63) is therefore an untrained model. All nine AnomalyCoT LoRA entries include the soft prompt; no MMAD LoRA run does.

2. At the same parameter count, LoRA on all layers is more accurate than OCP-R. Rank-1 q/k/v LoRA on all 28 layers has 430,080 parameters (OCP-R: about 473K). With OCP-R's objective, forced-prefix scoring, and the learning rate selected among 1e−4, 2e−4, and 5e−4 on 500 held-back training questions, three seeds reach 84.89 ± 0.28 / 81.63 ± 0.32 on Qwen2.5-VL/MMAD. OCP-R (82.38 / 77.00) is lower by 2.51 [1.10, 3.89] overall and 4.63 [1.85, 7.58] held-out, and all three sweep runs exceed it. Our statement that OCP-R exceeds LoRA at a similar parameter count holds for the last-four-layer placement only. With 500 training questions at equal steps, the baseline LoRA reaches 75.33 / 73.64 against 74.26 / 70.71 for OCP-R, so OCP-R has no advantage in that comparison either.

3. Image-blind control on all nine AnomalyCoT backbones (Update (Nov 16): Pixtral added). OCP-R minus the image-blind correction, in points [95% CI]:

BackboneOverallHeld-out
Qwen2.5-VL−1.48 [−2.65, −0.31]−1.75 [−4.12, 0.57]
Qwen2-VL−1.28 [−2.56, −0.07]+1.62 [−1.71, 4.74]
LLaVA-1.5+0.61 [−0.63, 1.89]+3.10 [−0.97, 7.22]
LLaVA-NeXT−1.15 [−2.72, 0.38]+0.67 [−3.28, 4.44]
InternVL3−0.90 [−2.18, 0.38]−2.16 [−5.81, 1.27]
InternVL2.5+0.76 [−0.54, 2.07]−0.67 [−3.52, 2.03]
Pixtral−2.56 [−4.00, −1.19]+0.13 [−2.83, 3.18]
Idefics3−2.59 [−4.09, −1.08]−2.70 [−6.35, 0.63]
LLaVA-OneVision−0.81 [−2.27, 0.67]+1.62 [−1.84, 5.48]

No interval lies above zero, and overall the image-blind correction is significantly higher on four backbones. It exceeds the option-only correction by 19–28 points overall on all nine, so on AnomalyCoT the gain beyond the option texts comes from the question text.

4. Other completed analyses. On seven MMAD backbones the option-only correction reproduces 86–96% of OCP-R's overall gain. The post-hoc sum of frozen and Vanilla OCP logits is indistinguishable from OCP-R on four of them and 2.55–3.42 points below it overall on three.

5. Rewrite audit. Update (Nov 17). Done for 150 of 200 items; see "Rewrite audit results" under Reviewer Bu6y. Update (Nov 16). The other three checks listed here on Nov 14 are complete and in the updated PDF: Pixtral's image-blind control (item 3) and the two LoRA runs below. On Qwen2.5-VL/MMAD, with the learning rate selected on held-back questions and three seeds, OCP-R is 4.07 [2.61, 5.57] above the last-four-layer rank-4 LoRA overall (held-out +1.60 [−1.09, 4.22], inconclusive). Without the soft prompt (one run per backbone at the baseline's learning rate), LoRA is below OCP-R overall on eight AnomalyCoT backbones and above it on LLaVA-OneVision (+3.98 [2.23, 5.69]); six of nine held-out intervals include zero (Appendix C). On LLaVA-1.5, LLaVA-NeXT, and Pixtral it stays below 45% overall, so we draw no claim about LoRA from these single runs; item 1's Pixtral runs use a selected learning rate and three seeds.

6. Revised manuscript. The uploaded revision removes the visual-recovery diagnosis from the abstract and the contribution list, states that all-layer LoRA is more accurate at an equal parameter count, adds the matched controls (Table 5), and corrects the LoRA description (Section 4.1, Table 2, Appendices C and E). Changes are not color-marked. Appendix G lists them in ten items and maps the Qwen2.5-VL numbers of the submission to their pipelines. Table numbers in our earlier responses refer to the submission: Tables 1–4 keep their numbers, and Tables 5, 8, 9, 11–12 (merged), 13, 14, and 17 are now Tables 6, 12, 13, 15, 17, 18, and 21.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.