acceptodds
Under review as a conference paper at ICLR 2027

RELATIONAL STRUCTURE RECOVERY FOR STRICT SCORE-ONLY VLM ADAPTATION

Abstract

Deployment interfaces for vision-language models may expose only a fixed tensor of prompt-level class scores, withholding images, embeddings, gradients, parameter updates, labels, and additional queries. Existing score-only methods mainly aggregate or reweight these scores. We ask instead whether the tensor contains a relational representation that supports target-conditioned adaptation. Prompt-class scores are repeated text-induced measurements: on non-degenerate rows, normalized class contrasts induce a shared geometry and form a maximal invariant to sample-wise offsets and positive scales, while prompt-dependent deviations provide complementary information after the shared response is accounted for. Prompt-Class Residual Interaction (PCRI) instantiates this shared–conditional decomposition by constructing a Score-space Core, extracting Core-conditioned prompt innovation, and using the joint representation only to correct the frozen semantic score field through target prototypes. With one configuration selected on a disjoint CIFAR-100 development setting and frozen across all evaluation domains, PCRI improves matched CARPRT by +2.378 percentage points on average across 11 datasets and three VLM backbones, with gains in 32/33 settings under identical cached scores. Matched controls attribute most Core-only gain to class-contrast standardization and isolate a further +0.306-point benefit from Core conditioning; support stress tests show that limited or unrepresentative target support can make the correction unreliable. PCRI uses no hidden representations or additional VLM calls after score extraction.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Score-Only Adaptation, Vision-Language Models / VLMs, Relational Structure Recovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SzFSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies adaptation when only a fixed tensor of prompt-level class scores is available, without access to images, embeddings, model updates, labels, or additional VLM queries. Paper proposes PCRI which constructs a shared representation from normalized class contrasts, groups prompt-specific deviations, and subtracts their ridge prediction from the shared representation. It uses the combined representation to construct prototypes from confident base pseudo-labels, then fuses prototype scores with the frozen base scores.

Critical strengths
  1. This paper concerns a practical and meaningful setup where deployment interface exposes only prompt-level scores, and feature-based adaption is not feasible.

  2. The empirical design is deliberate, and clearly isolate effectiveness of different designs. under matched comparisons, PCRI improves on 32 out of 33 settings.

Critical weaknesses
  1. The practical gain from distinctive component is modest. For example, standardized score prototypes already improves 1.884 while PCRI only improves 2.378 with dataset-level core normalization contributes only 0.003.

  2. Global fusion scale can reintroduce a nuisance removed by the representation. Eq.12 uses the pooled standard deviation of the base scores. With MPE aggregation, adding a sample-specific constant to every prompt score preserves base decisions, but will change this standard deviation. As a result, the correction strength and final predictions can change despite unchanged class contrasts.

Questions for authors
  1. Why eq.12 includes variation in sample-wise logit means? Can you report an offset stress test and compare the current scale with std(B-row_mean(B))? This would establish whether the identified sensitivity matters on the benchmarks.
1 comment
AuthorsOpenReview ↗

Response to Reviewer SzFS

Thank you for the constructive comments. We address the scale sensitivity and component value separately, with matched GPU experiments.

1. Response to Question 1 and Weakness 2 — Fusion-scale sensitivity.

Eq. (12) matches the global dispersion of the structural and base score matrices, allowing alpha to control correction strength across their numerical scales. Pooled Std(B) includes variation in sample-wise logit means, which leaves base class rankings unchanged. To examine this sensitivity, we implemented your suggested Std(B - row_mean(B)) as a row-centered scaling variant and clarified both formulations beside Eq. (12).

Across all 33 settings, we tested six offset levels k = {0, 0.25, 0.5, 1, 2, 4}, including zero, with three fixed seeds. Gaussian vectors were centered, normalized to unit population variance, and multiplied by k Std(B - row_mean(B)). Both rules received identical offsets. Representations, support and prototypes were fixed to isolate fusion-scale sensitivity; base predictions remained unchanged. The new GPU zero-offset gain is +2.382 pp, supplementing the submitted +2.378 pp result.

With injected offsets, at k = 4 the original rule's gain fell from +2.382 to +2.208 pp (-0.174 pp), with 7.078% of predictions changed relative to its own zero-offset output. Row-centered scaling retained +2.311 pp with no prediction changes at any tested level.

Without injected offsets, row-centered scaling gave +2.311 versus +2.382 pp, a difference of -0.071 pp (95% CI [-0.169, +0.011]; 8 wins, 3 ties, 22 losses). Row-centered scaling kept predictions unchanged across the tested offsets; the zero-offset accuracy comparison remained statistically inconclusive. Natural benchmarks may themselves contain offset variation.

We retain Eq. (12) for the submitted primary results and report row-centered scaling as an optional variant. Appendix D.6 provides the protocol and stability/accuracy figure. The invariance result concerns fixed-state fusion, without refitting the upstream aggregator.

2. Response to Weakness 1: Incremental component gains and the normalization control.

Incremental value beyond standardized-score prototypes. The submitted +1.844 pp result already implements PCRI’s shared-Core relational route: standardized-score prototypes with semantically anchored fusion. The component comparisons therefore assess both this shared-Core route and the additional predictive value of conditional prompt information.

To separate the value of the whole prompt branch from conditioning itself, we compare Full against Core-only and same-dimensional Direct-H. Across 33 matched GPU settings, Full exceeds Core-only by +0.531 pp (95% CI [+0.296, +0.787]; 29/33 improvements). This gain measures the additional benefit of the complete prompt branch after Core correction. Full also exceeds same-dimensional Direct-H by +0.316 pp ([+0.169, +0.460]; 31/33 improvements), supporting a contribution from conditioning beyond merely adding prompt information.

These are average increments: on Oxford-IIIT Pets, the Core and prompt-branch gains are +1.117/+0.763 pp with ViT-B/16 and +1.254/+0.818 pp with RN50. The incremental value is therefore not uniformly small. Across the full evaluation, the mean Full–Core gain remains positive on every backbone and ranges from +0.445 to +0.584 pp when any one dataset is omitted. The complete 33-setting breakdown, including negative increments, is provided in Appendix D.5.

The +0.003 pp normalization increment. This increment isolates the auxiliary upstream BlockNorm within the shared-Core pipeline. We test its practical relevance with a simpler normalization control. Removing upstream BlockNorm from the comparator while retaining the prototype head’s normalization yields One-BN Core; Full still exceeds it by +0.540 pp ([+0.298, +0.801]). Full also exceeds raw-score cosine prototypes by +0.725 pp ([+0.381, +1.074]). Thus, its incremental benefit persists against both a normalization-simplified Core and a stronger raw-score alternative.

In three resident-score GPU cases, Full takes 1.25–2.95× One-BN Core latency, while conditioning adds approximately 4–5% over Direct-H, without further VLM queries. These results support the relational correction and conditional prompt information as the substantive contributions, with an explicit accuracy–latency trade-off. Component controls and timing details are provided in Appendices D.5 and G.2, respectively.

Component intervals use 100,000 paired dataset-cluster bootstrap draws over 11 datasets, retaining all three backbones; they are pointwise and unadjusted. GPU results supplement the submitted results.

Reviewer_NMY2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies TTA for a strict score-only VLM interface scenario. It proposes PCRI, which views the score tensor as a collection of repeated measurements. It forms a normalized representation in score space and the resulting structure is used to adjust the original predictions through prototypes. The method is evaluated on 11 datasets with strong improvement.

Critical strengths

[+]: The paper studies a clear and relevant score-only setting.

[+]: The experiments use matched score tensors, which makes the main comparison convincing.

[+]: Failure cases and limitations are reported clearly and honestly. The discussion of score-space geometry is generally careful.

Critical weaknesses

[-]: Most of the gain comes from the relatively simple cross-class standardization step. The additional gain from core-conditioned prompt information is modest. Do you really need all of the components?

[-]: The scope of this study is somewhat narrow. What exactly does the 'score-only' scenario refer to? Does it have practical application scenarios? The authors are requested to provide specific examples for clarification.

[-]: Many designs are really heuristic. The hash-based prompt grouping is heuristic, and random partitions perform similarly.

[-]: The evaluation is limited to image classification and transductive target pools. I suggest the authors provide results on general cv tasks like detection or segmentation.

[-]: This paper contains many jargons and excessively long sentences, which makes it really difficult to follow. For example, line 083 (We formulate strict score-only adaptation as relational recovery from repeated prompt-class measurements and characterize normalized class contrasts as a maximal invariant under sample-wise positive-affine nuisance..). This manuscript should be carefully revised.

Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer NMY2

Thank you for the detailed feedback.

1. Response to W1: Components and cost.

To assess component value and complexity, we distinguish shared-Core correction from the prompt branch's added value. The +1.844 pp result measures our shared-Core route: standardized-score prototypes with semantically anchored fusion; z-score alone preserves class rankings.

To test the prompt branch beyond Core, and conditioning beyond added prompt features, we ran matched GPU controls. Across 33 settings, Full exceeds Core by +0.531 pp (95% CI [+0.296,+0.787]; 29/33 improvements). This measures the prompt branch's additional benefit after shared-Core correction. Full exceeds same-dimensional Direct-H by +0.316 pp ([+0.169,+0.460]; 31/33), isolating conditioning from simply adding prompt features.

These averages hide variation across settings. On Oxford-IIIT Pets, Core/prompt increments are +1.117/+0.763 pp (ViT-B/16) and +1.254/+0.818 pp (RN50). Full–Core means are positive on all three backbones and remain +0.445 to +0.584 pp after omitting any dataset. D.5 lists all 33 settings, including negatives. Pointwise, unadjusted CIs use 100,000 paired dataset-cluster draws (11 datasets, three backbones retained). GPU results supplement the submission.

To quantify cost, we timed three resident-score GPU cases: Full takes 1.25–2.95× One-BN Core latency (EuroSAT: 58.3/19.7 ms for 8,100 samples). Conditioning adds 4–5% over Direct-H, with no extra VLM calls (G.2). Core offers lower latency; Full adds measurable accuracy.

2. Response to W2: Applications.

Strict score-only describes adaptation AFTER initial inference. PCRI receives a fixed image-by-prompt-by-class tensor, prompt templates and alignment metadata, without images, embeddings, parameters, gradients, labels or further VLM queries.

For land-cover classification, a satellite-imagery company can run a frozen VLM internally and export agreed prompt–class scores and sample identifiers to an external partner. Images and the model stay internal; the partner applies PCRI and returns corrected predictions.

For product classification, a retailer can cache individual prompt–class scores from a hosted CLIP service, then refine predictions locally without further paid calls. Roboflow's comparison endpoint illustrates per-text similarity output; the restriction comes from the workflow/budget, not the platform lacking other endpoints.

These illustrative workflows require prompt-level scores, templates and alignment metadata, enabling downstream adaptation from cached scores without renewed model access.

3. Response to W3: Hash grouping.

Group averaging compresses prompt responses for conditioning. We use deterministic SHA-256 grouping to make the prompt partition easy for readers to reproduce. Each exact UTF-8 template maps to one of four groups independently of list order (with aligned score axes), without training or saved random-generator state. All prompts are retained; grouping neither selects data nor uses target scores or evaluation labels. Hashing is an implementation choice, not semantic discovery or a separate innovation.

To check whether gains depend on this particular hash partition, we tested five fixed random partitions, matching group counts/sizes, base scores and parameters across 33 settings. Each improves over Core by +0.436 to +0.514 pp, with its paired dataset-cluster 95% CI above zero. Averaging seeds within each setting gives +0.473 pp [+0.261,+0.719]. All five are reported without evaluation-label retuning (E.4). Gains survive alternative partitions; W1 tests conditioning separately.

4. Response to W4: Segmentation and held-out queries.

To test cross-task transfer, we evaluated complete segmentation maps (Section 5.6/Table 4). Matched heads share region scores, masks and decoding. With classification defaults and no segmentation-label tuning, PCRI gains +1.978/+0.661/+1.302 mIoU over CARPRT on VOC21/PC59/PC60 and also exceeds Core-only.

Validation-selected configurations improve over CARPRT in all seven settings (exploratory validation results; I.7). They score 76.326 vs CorrCLIP's 74.768 on VOC21 and 45.400 vs GLA-CLIP's 37.667 on Object81. These system comparisons differ in resources, readouts and tuning budgets.

To test dependence on queries during state fitting, five-fold query-excluded evaluation retains 97.6% of the classification gain (subsection 5.5) and yields +1.848 mIoU on VOC21 (I.3). Support-fitted states thus transfer to held-out queries within the same target dataset. Detection remains untested.

5. Response to W5: Clarity.

To improve readability, we split original line 083, explaining relations before invariance. Red-marked edits in the Abstract, Introduction and Methods put operations before terminology and clarify claim boundaries/grouping inputs. Formal propositions and assumptions are unchanged.

AuthorsOpenReview ↗

Supplementary response to Reviewer NMY2: component value and segmentation transfer

Thank you again for your feedback. We supplement our previous response with two component controls, a new segmentation analysis linking task transfer to prompt-branch value, and a concrete readability example.

Response to W1: Additional evidence on component value

PCRI combines shared relations from aggregated scores with complementary prompt information. To assess whether this combined design retains an advantage over stronger simplified alternatives, we highlight two matched controls from Appendix D.5 that were not detailed in our first response.

Across 33 matched GPU settings, Full PCRI exceeds One-BN Core by an average of +0.540 pp (95% CI [+0.298,+0.801]) and raw-score cosine prototypes by +0.725 pp (95% CI [+0.381,+1.074]). One-BN Core removes the auxiliary upstream normalization while retaining normalization in the prototype head. The raw-score control uses cosine prototypes without class-contrast standardization. These results show that Full PCRI's accuracy advantage persists against both a simplified Core pipeline and a raw-score prototype alternative.

Response to W4: Segmentation transfer and prompt-branch value

To test whether the prompt branch also adds value in semantic segmentation, we reanalysed the previously explored validation grid using five-fold label-held-out selection with three fixed seeds. Each head selects its fusion coefficient on four folds and is evaluated on the remaining fold. Core-only, Direct-H and Full PCRI share the same ten candidates, with other classification parameters fixed. They use identical cached scores, masks and decoding; cached states are fitted using the full unlabeled target pool.

The table reports mIoU for the designated primary seed, pooling pixel counts across all held-out folds. Full PCRI exceeds both matched CARPRT and Core-only in all seven configurations. Differences are computed before rounding.

SettingCARPRTCore-onlyFull PCRIPCRI - CARPRT
VOC2173.57674.61576.084+2.508
ADE15027.65427.76028.108+0.454
PC5948.58148.90649.167+0.586
PC6044.01644.91045.534+1.518
VOC2088.10888.32088.346+0.239
Stuff17132.00032.06132.320+0.320
Object8143.03543.03543.490+0.455

Full PCRI exceeds Core-only in all seven configurations under each of the three fixed fold assignments. On VOC21, it also exceeds Direct-H by +0.907 mIoU under the primary protocol. These matched comparisons extend the evidence for the prompt branch's incremental value to semantic segmentation.

The seven configurations cover four image collections, with paired label conventions sharing images and folds. Appendix I.8 (Tables S33-S35) reports this analysis, paired intervals and fold stability. The full-grid analysis (10/30/90 candidates for Core/Direct-H/PCRI), all three seeds and fold/candidate records are retained in the reproducibility package.

Under this primary protocol, the complete PCRI segmentation system exceeds locally reproduced ProxyCLIP (ECCV 2024), PEARL and GLA-CLIP (CVPR 2026) in all seven configurations, and native CorrCLIP (ICCV 2025) in five. On VOC21/PC60, it reaches 76.084/45.534 mIoU versus CorrCLIP's 74.768/44.226. These system comparisons use method-specific resources, readouts and parameter-selection protocols; the matched controls above hold scores, masks and decoding fixed.

Response to W5: A concrete example of the revised exposition

We revised multiple dense passages across the Abstract, Introduction and Methods, with the wording changes marked in red in the revised manuscript. As one example, the Introduction now explains the complete method through its operations:

"PCRI first constructs a shared representation from normalized class scores, then extracts prompt variation that is less linearly predictable from that representation. It combines both views to build target prototypes and correct the frozen base predictions."

This red-marked passage makes the roles of the Core, prompt branch and prediction correction explicit before the formal method description.

We would appreciate your assessment of whether these additional controls address your concerns about component value and task scope.

Reviewer_H26QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates strict score-only test-time adaptation for vision-language models as a relational recovery problem from the cached prompt-class score tensor. The proposed method, PCRI, first quotients out a sample-wise positive-affine nuisance from aggregated scores to obtain a "Score-space Core" of shared class-contrast geometry, then conditions prompt-derived residuals on this Core via ridge residualization, and finally uses this joint representation only to compute a dispersion-matched, semantically anchored correction to the frozen base logits. The paper reports improvements over matched CARPRT across 11 datasets and 3 backbones. Mechanism ablations isolate the class-contrast quotient as the dominant source of gain, and stress tests delineate an explicit support-coverage operating boundary.

Critical strengths
  1. The paper restricts adaptation to a single materialized prompt-class tensor with no embeddings, gradients, parameters, or further VLM queries, and then proves what cached scores cannot guarantee. This makes the claim about a precisely defined observable, rather than an implicitly feature-level adapter dressed up as "black-box."
  2. The staged ablation cleanly separates raw-score prototypes (+0.364 pp), cross-class standardization (+1.844 pp), Direct-H (+2.072 pp), and full PCRI (+2.378 pp), with the crucial control that Full Core minus U0 is only +0.003 pp. This shows the authors identified that their performance-critical transformation is the quotient, not the downstream normalization.
Critical weaknesses
  1. The 33-setting result is PCRI versus matched CARPRT only. The multi-aggregator evidence covers MPE, WPE, and CARPRT on just ten non-ImageNet CLIP ViT-B/16 datasets, and the same frozen hyperparameters are used without retuning. This limits the claim that PCRI "complements the starting predictor" to three aggregators, all of which are prompt-weighting rules over the same score tensor. There is no evidence found in the manuscript for generality to aggregators that operate on different score statistics (e.g., marginalization or entropy-based rules).
Questions for authors
  1. For the multi-aggregator results in Table 3, were the MPE and WPE base scores produced with their own upstream hyperparameters, or were any shared with CARPRT? Please state precisely what "same frozen hyperparameters" covers for the base predictor versus the PCRI correction.
  2. The sign test and dataset-clustered bootstrap both treat the 11 datasets as exchangeable units. Have you quantified cross-dataset correlation of per-setting gains, e.g., via a hierarchical or mixed-effects model, to assess the effective number of independent units underlying the CI [+1.554, +3.196]?
  3. The coverage-aware attenuation in Sec. F.3 reduces the worst-case degradation from −31.25 pp to −1.25 pp on development data. How about reporting this as a canonical, opt-in variant with the development-vs-evaluation separation made explicit, so that readers can assess the method's robustness directly rather than via an appendix-only extension?
1 comment
AuthorsOpenReview ↗

Response to Reviewer H26Q

Thank you for the thoughtful review. We have expanded the comparisons, examined statistical dependence, and formalized the optional attenuation variant.

1. Response to Weakness 1 and Question 1 — Aggregators and parameter sharing.

To test PCRI across different base-score constructions, we expanded the multi-aggregator evaluation to all 11 datasets and three backbones (33 settings), including ImageNet. The two new rules average per-prompt softmax probabilities, or weight raw scores using sample-specific negative-entropy softmax weights.

Mean gains over each rule's own base (pp; paired 95% CI) are MPE +2.926 [2.094, 3.755], mean-max weighting +3.064 [2.048, 4.190], probability averaging +1.039 [0.686, 1.410], and entropy weighting +2.789 [1.980, 3.604]. Each improves in 33/33 settings. CARPRT+PCRI retains the highest mean final accuracy, 64.704% versus its base's 62.322% (+2.382 pp; 32/33). This GPU replay supplements the submitted +2.378 pp.

No upstream weights are shared across aggregators. MPE uses uniform weights. The local rule previously labeled WPE averages each prompt's maximum class score over target samples and softmax-normalizes those statistics across prompts. We now name it mean-max weighting: it does not reproduce the full bias-corrected WPE/ZPE method; the literature values in Table 1 retain their attribution. CARPRT retains its own cached class-specific weights. The probability/entropy rules compute their own statistics. Temperatures are fixed at 1 without evaluation-label tuning.

The shared classification configuration is downstream PCRI: alpha=0.50, beta=0.75, G=4, ridge=30 and epsilon=1e-6, unchanged from the submission. Each base/PCRI pair uses identical aggregator-specific scores, and each PCRI state is fitted independently from those scores. We do not copy CARPRT's target state or weights to other rules. Table 3 and Appendix E.1 report the expanded results and protocol; the Figure S6 caption distinguishes its original ten-dataset scope and local comparator name.

2. Response to Question 2 — Cross-dataset dependence.

The original bootstrap keeps each dataset's three backbones together, but does not model dependence between datasets. We added analyses of the original 33 submitted gains, preserving their +2.378 pp mean.

A hierarchical model includes fixed backbone effects and nested task-family/dataset random effects. With six task families, 10,000 parametric bootstrap refits give an approximate 95% bootstrap-t interval of [+1.511, +3.282] pp; a five-family sensitivity grouping gives [+1.277, +3.442] pp. Estimated within-family correlations are 0 (a boundary estimate) and 0.097, giving model-based effective dataset counts of 11.0 and 9.08. Their broad profile-support ranges, [4.31, 11.0] and [3.93, 11.0], preclude treating these point estimates as proof of independence.

A separate all-pairs Gaussian equicorrelation sensitivity analysis yields [0.044, 4.711] pp at assumed rho=0.30 and [-0.440, 5.196] pp at rho=0.40. These assumptions differ from fitted within-family correlations. Both hierarchical intervals remain positive, while the sensitivity analysis shows that robustness to arbitrary dependence is not established. Section 5.1 summarizes the analysis; Appendix B.7 reports the assumptions and results, with Figure S9 visualizing how the uncertainty intervals vary across dependence models and assumptions.

3. Response to Question 3 — Main-text opt-in attenuation and independent validation.

Following your suggestion, Section 4.3 now defines coverage-aware attenuation as an opt-in variant, and Section 5.4 reports its independent evaluation. It uses alpha_eff=0.50*rho_pseudo^3, where rho_pseudo=K_pseudo/min(C,N_support); all other PCRI operations remain unchanged.

We tested the frozen rule across all 33 settings, using the class-restriction procedure in Appendix F.1 and five fixed seeds. Both variants use identical scores and batches, without retuning on evaluation labels. At the pre-specified 5%, 10% and 25% class-coverage levels, mean gain over matched CARPRT improves from -2.400 to -1.036 pp, a paired benefit of +1.364 pp (95% CI [+0.752, +2.014]). Negative-transfer frequency falls from 68.5% to 33.7%; 29/33 setting means improve. This extends the evidence beyond development data.

Severe failures remain (worst observed gains -45.000 and -33.867 pp), and full coverage costs 0.056 pp. Accordingly, attenuation is a risk-reduction option, while fixed-alpha PCRI remains default. Appendix F.3 separates the development result, this class-restriction test, and the 10%-sample-support test (+0.239 pp benefit; +0.271 pp on outside-support queries). New aggregator/attenuation intervals use 100,000 dataset-cluster bootstrap draws, retaining backbones and averaging repeated conditions within settings; they are pointwise and unadjusted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.