acceptodds
Under review as a conference paper at ICLR 2027

Gains Are Gains? What drives accuracy gains in transductive CLIP methods?

Abstract

Transductive CLIP methods leverage unlabeled test batches to improve recognition, with performance gains typically summarized by aggregate classification accuracy. Yet the same accuracy gain can arise for very different reasons: a method may reduce errors caused by classes that are absent from the batch, improve discrimination among the classes that are present, or do both. We introduce a decomposition that separates these two effects and complement it with controlled interventions that vary the unlabeled context while holding key aspects of the prediction problem fixed. Across multiple fine-grained recognition datasets and CLIP variants, the resulting analysis reveals substantial differences that aggregate accuracy obscures. Methods with similar final performance can behave very differently. Moreover, a considerable fraction of the gain can remain even when the adapted scoring function is replaced by frozen CLIP scores while preserving the class set induced by the method. The controlled interventions further show that the balance between these effects depends strongly on the composition of the unlabeled context. Together, these findings show that final accuracy alone incompletely characterizes transductive adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
CLIP, vision-language models, transductive inference, test-time adaptation, fine-grained recognition, zero-shot classification

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Zx6AOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Transductive CLIP uses a batch of unlabeled test images jointly to improve their classifications. It uses relationships across the test batch such as clusters of similar images or evidence that certain classes are common, to refine predictions. The paper claims that higher accuracy in transductive CLIP can hide worse discrimination between the classes actually present. A method may improve by suppressing classes absent from the test batch of images.

The paper separates these effects using the batch's true support. It measures ordinary accuracy over the full vocabulary, and accuracy after restricting the same fitted scores to the true support. The results show that MOON’s 10.32% overall gain combines worse discrimination with a larger reduction in absent-class obstruction. Another finding is that similar final accuracies can reflect opposing behaviors. MOON gains from reducing absent-class obstruction, while Adapt improves discrimination but introduces more obstruction. Experiments are done on CUB-200, FGVC-Aircraft, Oxford-IIIT Pets, Flowers-102, and Food-101.

Critical strengths
  1. This paper has carefully controlled context experiments to show that a transductive method can improve overall accuracy while becoming worse at distinguishing the classes actually present.
Critical weaknesses

The paper misses some citations to related work:

  1. Ren et al. Delving into the Openness of CLIP, Findings of ACL 2023 https://aclanthology.org/2023.findings-acl.610.pdf It measures accuracy on fixed target-class images while adding non-target classes to the candidate vocabulary. The authors show that competing class descriptions can substantially degrade recognition even though the underlying image–text scores remain unchanged.

  2. Zhou et al. Bayesian Test-Time Adaptation for Vision-Language Models, CVPR 2025 https://arxiv.org/pdf/2503.09248v1 This overlaps with the broader claim that adaptation can improve classification through changes to class plausibility, independently of improving the image–class matching model.

Questions for authors

Could you elaborate on the unique contributions of your work compared to existing work (see below):

  1. Ren et al. Delving into the Openness of CLIP, Findings of ACL 2023 https://aclanthology.org/2023.findings-acl.610.pdf It measures accuracy on fixed target-class images while adding non-target classes to the candidate vocabulary. The authors show that competing class descriptions can substantially degrade recognition even though the underlying image–text scores remain unchanged.

  2. Zhou et al. Bayesian Test-Time Adaptation for Vision-Language Models, CVPR 2025 https://arxiv.org/pdf/2503.09248v1 This overlaps with the broader claim that adaptation can improve classification through changes to class plausibility, independently of improving the image–class matching model.

1 comment

We thank the reviewer for the suggested references. We will add them to the related-work discussion, while clarifying that neither addresses the central research question of our paper.

1. Missing references

1a. Ren et al. (ACL 2023).

Ren et al. examine how expanding CLIP's candidate vocabulary affects recognition accuracy. Their results demonstrate that adding competing classes can hurt predictions.

However, our paper is not about whether candidate classes can hurt CLIP accuracy. We study what accounts for the accuracy gains achieved by existing transductive CLIP methods after adaptation. We quantify whether those gains arise from improved discrimination among represented classes or reduced obstruction from absent classes, using an exact decomposition of the accuracy change.

Ren et al. do not analyze this transductive adaptation problem or the resulting gain decomposition.

1b. Zhou et al. (CVPR 2025).

Zhou et al. develop a Bayesian test-time adaptation method that incorporates class-associated prior information. Its relevance is that class-related information can influence adapted predictions.

However, using class priors in an adaptation algorithm is not the same as measuring the sources of its observed accuracy gains. Our analysis is independent of the adaptation mechanism and determines how the final predictions change in terms of within-support discrimination and absent-class obstruction. Zhou et al. do not provide this evaluation.

Our contribution is distinct from 1a and 1b: we introduce an exact decomposition of transductive CLIP accuracy gains, a support-only control that isolates the contribution of the predicted class set, and controlled experiments establishing how these contributions vary across methods, semantic contexts, and support sparsity.

For example, we show that MOON's +10.32 pp accuracy gain consists of −3.09 pp in within-support discrimination and +13.41 pp in obstruction relief—behavior that neither cited study investigates.

We will include the references for completeness, but they do not diminish the novelty of the research question, diagnostic framework, or empirical findings presented in our paper.

2. Additional evidence of practical significance.

We extended our experiments to full-vocabulary support (S = C), where absent-class obstruction relief is exactly zero and all accuracy changes reflect within-support discrimination. The results are reported in the table in our response to Reviewer r9xL, Section 2. These results demonstrate that methods benefiting from sparse support may underperform frozen CLIP when all classes are represented, highlighting the decomposition's practical value for understanding deployment conditions and guiding method selection.

Reviewer_uYBtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines what accuracy gains from transductive CLIP methods consist of. Each method is fitted on the full vocabulary; its unchanged scores are then evaluated over the classes actually present in the batch. This yields an exact identity: full-vocabulary gain = Δrank (change in within-support accuracy) + ΔR (reduction in absent-class obstruction). Predicted-support precision/recall and a support-only (SO) control, which uses frozen CLIP scores over the method's predicted class set, complement the decomposition. Experiments cover five fine-grained datasets, three ViT-B CLIP checkpoints and eight adaptation variants alongside frozen CLIP. The reported results show that full-vocabulary gains can coexist with worse within-support accuracy, that SO can retain much of the net gain, and that methods with similar full accuracy can have opposite decompositions. Changing the classes or images around fixed queries also changes performance; in the main fixed-class experiment, more target-like context improves MOON/StatA's target-query accuracy. The support-size experiments show that sparse batches favour MOON and StatA in the evaluated settings.

Critical strengths

1. A simple, correct and useful evaluation framework. The true-support mask is applied only at evaluation time, the identity is exact (Appendix A), and the SO control uses predicted rather than true support. MOON's +10.32-point full-vocabulary gain coexists with a 3.09-point within-support decline, while SO retains much of the gain (70.89% versus 72.04% full accuracy). This provides a concrete reason to report more than aggregate accuracy.

2. Informative paired controls and broad evaluation. Methods share representations, text heads and episodes within each protocol, and adapters are reset per episode. Holding queries fixed across contexts makes their changes in performance interpretable. The sparsity grid varies support size at fixed NN and repeats this comparison at N=32/64/96N = 32/64/96 under balanced and skewed class counts. Bootstrap intervals, paired contrasts and three checkpoints are reported, and unfavorable results, including loss of gains at broader support and the limited LAION follow-up, are retained.

Critical weaknesses

1. Finding 3 needs a narrower interpretation and clearer experimental provenance.

  • Table 4 reports a high-minus-low improvement for MOON and StatA on the eight target-class queries of each episode. Making competing-class images more target-like could shift the adapter toward predicting the target, improving those queries while harming competitors. This would be consistent with Table 4, but would not by itself establish improved discrimination across all represented classes beyond the target-query metric.
  • Table 60 does not confirm the effect in its two-dataset LAION setting: StatA −0.56  [−3.17,+1.94]-0.56\;[-3.17, +1.94], MOON +1.44  [−0.37,+3.43]+1.44\;[-0.37, +3.43]. Both intervals include zero, and the different dataset subset limits direct comparison with the main experiment.
  • Section 5.2.2 says the near-context queries are reused, but Table 59 reports frozen accuracy of 61.96%, rather than 61.14% in Tables 3/23, and describes its inputs as "development pools". These values require reconciliation if the queries, frozen classifier and averaging weights are unchanged. Appendix B.1 specifies official test splits for evaluation, but B.3 does not clarify the image-level source of the 108 development episodes or their overlap with the evaluation pools.

2. Numerical inconsistencies and uneven tuning limit the method-level conclusions.

  • Several printed TransCLIP rows violate the exact identity beyond rounding tolerance: in Tables 1/5, 36.20−61.72=−25.5236.20 - 61.72 = -25.52, whereas −5.52−20.35=−25.87-5.52 - 20.35 = -25.87; Table 17 gives −26.14-26.14 versus −26.36-26.36. Smaller residuals occur in Tables 7 and 11, despite Appendix B.4's statement that the identity was checked before rendering. These discrepancies concern the reported values, not the algebraic identity. Smaller reporting issues include the missing minus sign for ADAPT's K=4K = 4 change (−1.43-1.43 in Table 61) and the claim that StatA worsens in the low condition: its interval, −0.54  [−2.12,+1.07]-0.54\;[-2.12, +1.07], does not establish deterioration.
  • The opposing response patterns in Section 5.2.1 coincide with unequal tuning opportunities. StatA, MOON and ADAPT settings were selected from 12 candidates on 108 development episodes; MOON's selected graph weight is 0, unlike its native setting. ZLaP, GDA-CLIP and TransCLIP, the methods showing the opposite pattern, retain released settings. Some of this contrast may reflect configuration choices rather than stable properties of the method families.
Questions for authors
  1. Can you regenerate the affected TransCLIP rows in Tables 1/5, 7, 11 and 17 from the same final scores, episode sets and averaging weights, documenting any numerical replays or exclusions, and explain the residuals? Do the corrections change any conclusion?
  2. For Finding 3, please report any evidence from fixed queries spanning all represented classes, together with target-prediction rates on competing classes. All-batch results would be useful descriptively, but the context images change across conditions. Alternatively, please explicitly restrict the claim to target-query accuracy.
  3. Which pools and query sets produced Tables 4, 59 and 60? What explains the frozen-baseline difference between Tables 3/23 (61.14%) and Table 59 (61.96%)? Please clarify the image-level relationship between these evaluation pools and the 108 development episodes used for tuning.
  4. Does Finding 2's opposing response persist under comparable tuning or existing sensitivity checks? Otherwise, please limit the conclusion to the tested configurations. Please also distinguish requested from realized NN and KK in the native regimes and report their dataset-level values, particularly for Pets in the nominal K=50–100K = 50\text{–}100 regime.
6 comments

Response to Reviewer uYBt — Part 1/2

We sincerely thank the reviewer for the thorough and constructive assessment. We investigated all concerns, corrected the numerical issue, and conducted additional experiments. The central findings remain supported.

1. TransCLIP numerical inconsistencies.

We identified the cause: class-assignment masses underflowed to zero during TransCLIP updates, producing non-finite predictions. The original results excluded these episodes, while the frozen baseline included them, creating the reported inconsistencies. We repaired the numerical computation and reran all 45,000 episodes with 100% successful coverage.

For OpenAI ViT-B/32, N = 64, K = 1–4, the corrected results satisfy:

Δfull=−25.72=−5.42−20.30=Δrank+ΔR.\Delta_{\mathrm{full}}=-25.72=-5.42-20.30=\Delta_{\mathrm{rank}}+\Delta_R.

The identity holds before rounding across all corrected rows. The maximum change in TransCLIP accuracy is 0.20 pp, with no substantive conclusion changed.

2. Finding 3 and target-prediction bias.

We conducted the requested all-class fixed-probe experiment on official test images. In each episode, eight images of the designated target class and one image from each of the seven competing classes stayed fixed across low and high contexts; only the remaining 49 images changed.

For OpenAI ViT-B/32 MOON, accuracy on the fixed target-class images increased by +4.36 pp [3.19, 5.59]. Accuracy on the fixed competing-class images changed by +0.35 pp [−0.62, 1.34], while predictions of the target class on those competing images increased by +2.96 pp [2.25, 3.75]. Full-vocabulary accuracy averaged equally across the eight classes increased by +0.85 pp [0.02, 1.71], but within-support macro accuracy changed by −1.05 pp [−1.94, −0.19].

These results support a context effect on the fixed target-class images, not a comparable improvement in discrimination across all represented classes. The increased target-prediction rate is consistent with the reviewer's proposed explanation, though it does not establish it as the sole cause. We have narrowed Finding 3 accordingly.

3. Evaluation pools, query reuse, and tuning overlap.

The 61.96% versus 61.14% frozen-baseline difference comes from different image pools, not inconsistent predictions. In the submitted version, Table 4 and Appendix Tables 59–60 used development-pool images, whereas Tables 3/23 used official-test images. Within each experiment, the same target images were held fixed across context conditions; the two experiments used different target images.

We have updated Table 4 to report the official-test results, using the same target queries as Tables 3/23, while retaining the development-pool results in the appendix as exploratory analyses. The updated Table 4 uses official-test images with no overlap with the 108 tuning episodes.

Importantly, the official-test results preserve the central finding: MOON's high–low within-support contrast is +5.70 pp [4.20, 7.27], while StatA's corresponding contrast is +0.65 pp [−1.15, 2.56]. These results support the observation that changing context images can improve recognition on unchanged target queries, particularly for MOON.

Response to Reviewer uYBt — Part 2/2

4. Finding 2: Hyperparameter sensitivity and realized episode sizes.

We tested whether Finding 2's opposing obstruction-relief responses persist under different hyperparameter settings. On OpenAI ViT-B/32, the near-minus-far obstruction-relief contrast is positive for TransCLIP (+14.82 pp [12.48, 17.16]), ZLaP (+6.50 pp [4.06, 8.97]), and GDA-CLIP (+6.38 pp [3.70, 9.01]), indicating greater obstruction relief in near contexts. MOON and StatA show the opposite response.

To test its sensitivity to hyperparameters, we reran MOON, StatA, and ADAPT using two configurations: (i) the hyperparameters selected using our 108 development episodes and used in the original experiments, and (ii) the hyperparameters from the methods' published implementations. Both configurations were evaluated on the same 540 paired official-test target classes.

MethodOur tuned configuration (pp)Published configuration (pp)
MOON−10.62 [−12.56, −8.78]−12.11 [−14.08, −10.20]
StatA−7.66 [−9.38, −5.92]−4.55 [−6.28, −2.84]
ADAPT−7.04 [−8.71, −5.48]−11.72 [−14.16, −9.34]

All values are near-minus-far obstruction-relief contrasts with 95% confidence intervals. Negative values indicate greater relief in far contexts. The opposing obstruction-relief response therefore persists for MOON and StatA under both tested configurations. ADAPT's overall accuracy contrast changes sign between configurations, although its obstruction-relief contrast remains negative, further illustrating why the decomposition is informative.

We follow the native episodic sampling protocol and regime notation of MOON (ICML 2026). As in the released implementation, N and K denote requested values; realized values may differ depending on dataset availability. For example, Pets contains only 37 classes, so its nominal K = 50–100 regime realizes K = 37. Similarly, realized N can be smaller when the selected classes contain insufficient distinct images. We audited the episode manifests and report dataset-level realized N and K for all six regimes in the appendix. All compared methods use identical episode manifests.

5. Additional evidence: practical significance of the decomposition.

We further extended the sparsity analysis to the case where the represented support covers the entire class vocabulary, while separately controlling the number of images per class. This provides a particularly informative boundary condition: when S = C, absent-class obstruction relief is exactly zero, so any remaining accuracy gain or loss comes entirely from within-support discrimination.

The four-setting results are reported in the table in our response to Reviewer r9xL, Section 2 (Broader implications and design principles).

These results expose a practically important distinction: MOON and StatA can benefit from sparse support yet underperform frozen CLIP when every class is represented. Increasing the number of images per class reduces these losses, while ADAPT shows a different response, with larger gains under broader support in these experiments.

This illustrates the value of our framework beyond aggregate accuracy: it identifies which sources of improvement are available under different batch conditions and helps inform method selection and future method design. A method's gains under sparse support should not automatically be assumed to transfer to dense-support deployment.

Thank you for the revised manuscript and the additional controls. The revisions clarify why the original experimental presentation overstated the evidence for Finding 3.

First, the original Section 5.2.2 said it reused the near-context queries. In fact, original Table 4 used a different development pool, and revised Appendix F discloses overlap with setting-selection episodes for CUB and Pets. This made a development result read like a result on the same official-test queries. The difference matters: when Table 4 was rerun on the official test split, StatA's high-minus-low contrast changed from +3.41 pp [1.68, 5.26] to +0.65 pp [−1.15, 2.56]. MOON's +5.70 pp contrast on OpenAI ViT-B/32 remains supported, but the LAION ViT-B/32 estimate is +0.76 pp [−0.22, 1.75] (Table 62). Please state the split and its separation from tuning alongside the setting-selection protocol in Appendix B.3.

Second, the original Finding 3 claimed improved discrimination among the represented classes, but Table 4 scored only the eight fixed queries from the target class. Table 64 adds one fixed probe from each of the seven competing classes. For MOON on OpenAI ViT-B/32:

  • Target accuracy rises by 4.36 pp.
  • Full-vocabulary macro accuracy rises by 0.85 pp.
  • Within-support macro accuracy falls by 1.05 pp.
  • Competing-class probes are also predicted as the target 2.96 pp more often.

This is consistent with a shift toward the target; it does not establish improved discrimination across the represented classes. The Introduction and Section 5.2.2 heading should reflect the narrower result.

As the response explains, the original TransCLIP comparison averaged over different sets of episodes: episodes with failed runs were omitted for TransCLIP and retained for frozen CLIP, despite the stated shared-episode protocol. The revised rows reconcile within rounding, which addresses my numerical objection. Appendix B should document the original failure counts and which runs the reported 45,000 reruns cover. Table 6 resolves the episode-size question. Table 65 supports the direction of the obstruction-relief contrast for MOON and StatA under both tested settings on OpenAI ViT-B/32; the robustness statement should specify that scope.

The decomposition and support-only control are useful, and the full-support experiments strengthen the case for reporting both components. My remaining concern is the strength of the empirical contribution: the revised evidence supports a conditional target-query effect and setting-specific comparisons, while the original broader discrimination claim is unsupported.

Reviewer_r9xLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an evaluation framework for transductive CLIP methods that decomposes their accuracy gains into two components: improved within-support discrimination Δrank\Delta_{rank} and reduced absent-class obstruction ΔR\Delta_R. The authors empirically evaluate several existing methods and characterize their gains through this decomposition. They also demonstrate context-dependent behavior by varying the semantic similarity of the surrounding classes and the confusability of the images representing those classes.

Critical strengths

The paper goes beyond aggregate accuracy by decomposing transductive gains into improved within-support discrimination and reduced absent-class obstruction. Together with the support-only control, this framework reveals that similar overall gains can arise from substantially different behaviors. This offers a more informative and interpretable basis for evaluating and comparing existing methods.

The paper also provides a well-controlled experimental setting. It systematically varies surrounding class identities, the images representing those classes, and support sparsity.These experiments reveal informative and sometimes counterintuitive findings, such as more confusable context images improving discrimination for certain methods, demonstrating why batch composition deserves explicit consideration in transductive evaluation.

Critical weaknesses

The findings are mainly empirical, with limited explanation of the underlying mechanisms. Several findings might deserve further explanation: why different methods exhibit opposing trade-offs between within-support discrimination and absent-class obstruction (Table 2); why more confusable context images improve discrimination for MOON and StatA but not ADAPT (Table 4); and why MOON and StatA are substantially more sensitive to support sparsity than ADAPT (Figure 2). Although the decomposition identifies which components account for the observed accuracy changes, it does not reveal the underlying mechanism: which algorithmic components from these selected methods produce these behaviors? The authors acknowledge this distinction, but further ablations or mechanistic analyses would increase the value of the findings.

As a result, the significance of the findings also remains unclear. Although the decomposition reveals how methods differ in their contributions from within-support discrimination and reduced absent-class obstruction, the paper provides limited guidance on how these observations could advance the field or inform the design of new methods. It would be useful if the paper could distill the findings into some broader insights or actionable design principles. For example, how would these insights guide the reduction of absent-class obstruction without sacrificing within-support discrimination?

Questions for authors
  1. Could the authors explain the mechanisms underlying the observed patterns, such as how the findings relate to the selected methods’ algorithmic components?
  2. What broader implications do these findings have for transductive learning? Can the decomposition yield actionable principles for designing or improving methods?
3 comments

We sincerely thank the reviewer for the constructive assessment and for recognizing the value of our evaluation framework and controlled experiments.

1. On mechanistic explanations.

As explicitly stated in Section 6, our measurements are diagnostic rather than mechanistic: they identify where accuracy changes occur, not which internal algorithmic operations produce them. Establishing the latter would require method-specific ablations and causal interventions across the evaluated algorithms, constituting a separate research investigation.

Our contribution is to provide a common framework for identifying and quantifying performance changes that aggregate accuracy conceals, and to establish how these changes depend on batch composition. By identifying which components change and under what conditions, our framework provides a concrete basis for formulating and testing the mechanistic questions the reviewer raises.

2. Broader implications and design principles.

Our findings have two practical implications: guiding method development and informing method selection under different deployment conditions.

First, the decomposition exposes two distinct objectives: improving within-support discrimination and reducing absent-class obstruction. This allows future mechanistic studies to investigate which algorithmic components affect each objective, and provides measurable targets for designing methods that improve one or both without concealing harmful trade-offs.

Second, our experiments show that the effectiveness of adaptation depends strongly on the batch's class composition. At fixed N = 64, increasing represented classes from K = 4 to K = 32 changes MOON's gain from +10.38 to −2.15 pp, and StatA's from +7.16 to −1.87 pp.

To investigate this further, we extended support coverage to the full vocabulary and separately controlled the number of images per class. On OpenAI ViT-B/32, the five-dataset mean accuracy gains over frozen CLIP are:

Support coverageImages/classMOONStatAADAPT
40%4+0.32−0.61+0.12
40%10+3.16+2.45+0.26
100%4−3.41−3.34+0.62
100%10−0.57−0.66+1.75

At full support, absent-class obstruction relief is exactly zero, exposing the methods' within-support ranking performance. MOON and StatA, which can provide substantial gains under sparse support, fall below frozen CLIP at full support in these settings. Increasing images per class reduces these losses. ADAPT exhibits a different pattern, with larger gains as support coverage and images per class increase in this experiment.

These results show why no single adaptation method should be assumed optimal across deployment conditions. Support coverage and available evidence per class can substantially change which methods are beneficial. They motivate evaluating and selecting methods with these conditions in mind, and developing adaptation strategies that respond to estimated support coverage and class-level evidence.

Thank you for clarifying the scope and demonstrating a potential use case for the proposed metrics. The response addresses most of my concerns. I consider this paper a solid contribution, even though I retain some reservations about its broader significance. I will raise my score accordingly.

We thank the reviewer for considering our response and additional experiments, and for updating their assessment. We appreciate the constructive feedback.

Reviewer_cr6gOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Summary:

This paper analyzes existing transductive CLIP methods by splitting their accuracy gains on an unlabeled test batch into two parts, accuracy among the classes actually present in the batch, and error reduction from filtering out absent classes. Across five fine-grained datasets and different batch setups, the authors show that methods like MOON and StatA mainly improve overall accuracy just by removing absent classes, while actually hurting discrimination among the present classes

Weaknesses and Strenghts

Generally the paper requires substantial writing imporovement, it looks like the paper have not been written for an human audience.

L45-46 → Papers includes lots of such a claims that are not supported by a reference. these are required for the justifications of the assumptions.

L48 → From my understanding of the paper, shouldn’t this be What actually caused the improvement or something similar?

Have the authors tried their method on some well established open-set classification problems?

L124, Not clear, needs re-writting

The scope of experiments are limited and requires further investigations,, e.g., does it transfer to some other multi-modal models like embeddinggemma ?

What the authors mean by Fitted method in L134? Fit to what. we didn’t had the labels right for the batch B right? Maybe its better to move some definition below this line above so it is understandable?

I understand that the authors have used the 1K batch samples per dataset, how does changing this in comparison to the size of teh dataset and batch will effect the outcome? as teh sizes of the datasets that the authors have used, e.g., CUB200 vs Food101 varies about 5 times .

The paper is mainly analysis of the existing transfudctive methods yet there is not clear definition/distinction between the the underlying working of these methods and why do they think that its correlate with their observations (especially Finding 2-4)

Critical strengths

Please see the paper Summary

Critical weaknesses

Please see the paper Summary

Questions for authors

Please see the paper Summary

2 comments

We thank the reviewer for their feedback and the opportunity to clarify several aspects of our work. We believe some concerns arise from differences between the reviewer's interpretation and the paper's stated setting, objectives, and experimental protocol. We address these points individually below, particularly those relevant to the assessment of technical soundness.

1. EmbeddingGemma. The reviewer suggests evaluating multimodal models such as EmbeddingGemma. However, the original EmbeddingGemma is text-only, while EmbeddingGemma 2 was released on October 6, 2026, eleven days after the ICLR submission deadline. Our study explicitly investigates transductive methods within the CLIP model family which is used very frequently in transductive setting.

2. Open-set classification. We propose an evaluation framework, not a new classification method. Our setting assumes a known vocabulary C and batch support S ⊆ C. Conventional open-set recognition considers unknown classes outside C. This request therefore concerns a different problem formulation, not a missing validation of our framework.

3. "What caused the improvement?" The reviewer suggests replacing "What actually improved?" with "What actually caused the improvement?" This changes the research question. Our framework measures where accuracy changes occur, not the internal causal mechanisms producing them, as explicitly stated in the introduction and Section 6.

4. Fitting without labels. Fitting does not imply supervised learning. Section 3 and Appendix B.1 explicitly establish that adaptation uses unlabeled batches; labels are used only for evaluation.

5. Dataset and batch sizes. The reviewer refers to "1K batch samples per dataset," whereas our protocol uses 1,000 sampled episodes per dataset. Batch size and support cardinality are also systematically varied in the controlled experiments and appendix. Dataset-size sensitivity is a separate question; the concern does not establish a flaw in the reported comparisons.

6. Specific comments on Lines 45–46 and 124.

Lines 45–46: The reviewer requests references for the observation that aggregate accuracy can obscure how transductive methods exploit batch-level information. This is precisely the research question our paper investigates. The proposed decomposition formalizes this ambiguity, and our experiments provide evidence for it.

Line 124: This passage defines the standard frozen CLIP baseline: s₀(xᵢ, c) denotes the image–class score, and predictions are obtained by maximizing that score over the vocabulary C. The index 0 simply denotes the frozen baseline. The reviewer does not identify which aspect of this definition is unclear.

7. Presentation and soundness. The claim that the paper "looks like it has not been written for a human audience" is a sweeping characterization unsupported by concrete examples beyond isolated clarity requests. Despite assigning the lowest soundness rating, the review identifies no mathematical error, invalid experimental protocol, or concrete technical flaw undermining the contribution. Several objections rely on incorrect premises; others request investigations outside the stated research question.

We would kindly appreciate knowing whether our response addresses your initial concerns. We remain happy to provide any further clarifications if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.