acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Text Semantics in Supervised Unpaired Multimodal Representation Learning

Abstract

Supervised *Unpaired Multimodal Representation Learning* (UMRL) improves visual classification using text representations, requiring neither *prior cross-modal alignment* nor *sample-level correspondence*. Under such conditions, what drives the performance gains remains unclear, particularly the role of text semantics. In this paper, we revisit supervised UMRL and identify visual representation adaptation and text-derived classifier initialization as the main contributors, while increasing the number of text representations and introducing a text classification loss provide only limited benefits. Interestingly, shuffling the correspondence between text representations and class labels has little effect on the text-derived classifier initialization. Motivated by this, we introduce a semantics-agnostic representation synthesis strategy for classifier initialization. Extensive experiments show that semantics-agnostic initialization performs comparably to its text-derived counterpart. It also eliminates the need for an additional text encoder and reduces representation construction time and storage overhead. Our observations suggest that text semantics may not be the primary driver of the performance gains in supervised UMRL. The codes are available at https://anonymous.4open.science/r/RethinkUMRL.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unpaired Multimodal Representation Learning, Few-Shot Learning, Unimodal enhancement, Supervised Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AXNVOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits supervised Unpaired Multimodal Representation Learning (UMRL) and investigates which components are responsible for the performance improvements previously attributed to auxiliary text representations. The authors decompose the supervised UMRL pipeline into four factors: the number of text representations per class, the auxiliary text classification loss, the learnable visual projector, and text-derived classifier initialization. Their experiments suggest that increasing the number of text representations and using the text classification loss have limited effects, whereas the visual projector and classifier initialization contribute more substantially to performance. The paper then studies whether the benefit of text-derived initialization actually depends on semantic correspondence. The authors randomly permute the assignment between text representations and class labels and observe little degradation. Motivated by this observation, they propose a semantics-agnostic initialization scheme in which each class representation is synthesized as vc=mq+ 1−mrcv_c=\sqrt m q+\ \sqrt{1-m}r_c where qq is shared across classes and rcr_c is sampled from the unit sphere in the subspace orthogonal to qq. The parameter m controls the expected pairwise cosine similarity. The resulting representations are used to initialize the classifier without relying on a text encoder. Experiments across multiple few-shot image classification datasets, ImageNet-1K, several DINO/DINOv2 backbones, and a full fine-tuning setting show that the proposed initialization can often approach the performance of text-derived initialization while avoiding the computational and storage cost of OpenLLaMA-3B-v2. Based on these observations, the authors argue that text semantics may not be the main source of the performance gains observed in supervised UMRL.

Critical strengths

1.The paper asks an important diagnostic question and performs a broad empirical decomposition.A major strength of the paper is that it revisits a potentially under-examined assumption in supervised UMRL: whether the observed benefit of auxiliary text actually comes from semantic transfer. Rather than treating the original multimodal pipeline as a monolithic method, the paper decomposes it into several components and separately studies the effects of the text loss, number of text representations, visual projection, and classifier initialization. This diagnostic perspective is useful because it challenges a natural but not necessarily justified interpretation of previous UMRL results. 2.The empirical evaluation is relatively extensive and has practical implications.The evaluation covers multiple datasets, several few-shot regimes, ImageNet-1K, multiple DINO/DINOv2 backbones, and a full fine-tuning setting. The practical observation that a text encoder may be unnecessary in this particular supervised pipeline is potentially useful, especially because the proposed alternative removes the OpenLLaMA-3B-v2 storage and text-extraction overhead.

Critical weaknesses
  1. The experiments do not causally isolate “text semantics”; the current results can also be explained by trainable re-optimization and additional downstream capacity.The main conceptual weakness is that the experiments do not cleanly isolate semantic information from optimization and architectural effects. The paper interprets the weak effect of class-label permutation and the success of semantics-agnostic initialization as evidence that text semantics are not a primary driver of supervised UMRL gains. However, in the evaluated setting, both the classifier and the learnable visual projector remain trainable after initialization. Therefore, destroying the initial semantic correspondence does not prevent supervised training from subsequently relearning the class assignment. The permutation experiment establishes that the initial association between a particular text vector and a particular classifier row is not essential after optimization, but this is weaker than establishing that semantic information is unimportant. This concern is particularly important because the proposed synthetic representations are used only as classifier initialization. Once supervised optimization starts, the class weights are free to move away from their initial values. A much cleaner semantic intervention would freeze the classifier initialized from text/synthetic prototypes and train only the visual mapping, or alternatively compare zero-step and early-training performance before the initialization is overwritten. A geometry-preserving counterfactual, such as preserving the complete Gram matrix of text prototypes while destroying semantic correspondence, would also provide a much stronger test. Relatedly, the claimed contribution of the visual projector is not fully isolated either. The comparison with and without gvg_v simultaneously changes the number of trainable parameters and the depth/capacity of the downstream visual classifier. Thus, Figure 4 demonstrates that adding a trainable projection module improves classification, but it does not establish that this improvement is specifically related to UMRL or cross-modal transfer. A matched-capacity image-only MLP/projector baseline is necessary. The paper analyzes the four components largely sequentially rather than through a factorial design. After finding that the text loss has little effect, the authors remove it and continue the remaining analysis under the modified pipeline. This does not rule out interactions between LtL_t, gvg_v, WtW_t, and the number of text examples. Therefore, the statement that the projector and initialization are the “main contributors” is stronger than what the current one-factor-at-a-time ablations strictly establish.

  2. The semantics-agnostic baseline is insufficiently controlled, and the evidence for “comparable performance” is weaker than claimed.The second major concern is that the proposed semantics-agnostic initialization is not compared against sufficiently well-matched initialization controls. Equation (2) explicitly constructs unit-norm vectors, whereas the paper does not clearly specify whether the Gaussian, uniform, Laplace, or standard random classifier baselines are normalized to have the same row norm, logit scale, covariance, and spectrum as the proposed initialization or the text-derived prototypes. This matters because classifier initialization scale alone can substantially alter few-shot optimization. This issue is especially visible in the sensitivity experiment. Whenm=1, Equation (2) gives vc= qv_c=\ q for every class, meaning that all classifier rows are exactly identical at initialization. Yet the paper reports relatively weak sensitivity to m. If an initialization with initially identical class prototypes performs similarly to one with distinct prototypes after training, this suggests that the relevant benefit may primarily be optimization-scale or normalization rather than the particular inter-class geometry introduced by the synthesis method. This possibility is not investigated. Stronger controls are therefore needed, including row-normalized Gaussian initialization, random orthogonal prototypes, simplex-ETF/hyperspherical prototypes, and ideally a baseline matched to the norm and Gram-spectrum of the text-derived prototypes. This omission is particularly notable because the related-work section itself discusses hyperspherical prototypes and simplex ETF classifiers. There is also an asymmetry in hyperparameter tuning. The proposed method introduces m, and Table 4 states that m∈0,0.3,0.6,0.9,1.0m\in{0,0.3,0.6,0.9,1.0} is selected using validation performance. This gives the semantics-agnostic method an additional dataset-dependent model-selection degree of freedom. Results with a single globally fixed m, selected independently of the target dataset and shot setting, would provide a cleaner comparison. Finally, the phrase “comparable performance” is somewhat stronger than the results warrant. Only three seeds are used and no equivalence or non-inferiority test is provided. Moreover, the additional-backbone experiments contain multiple nontrivial degradations. For example, on DINOv2 ViT-L/14 the semantics-agnostic initialization is lower by 1.58 points on SUN397 in the 1-shot setting, 1.34 points on Stanford Cars in the 2-shot setting, and 1.55 points on FGVC-Aircraft in the 4-shot setting. Similar drops appear for DINO ViT-B/8 and ViT-B/16. These are not isolated numerical noise and should be quantified rather than summarized only as “generally comparable.”

Questions for authors

Q1 Please clarify the exact initialization and normalization procedures for all classifier baselines. Are the rows of the text-derived classifier, Gaussian/Uniform/Laplace baselines, and standard random initialization normalized to the same norm? Are their logit scales matched? What distribution and variance are used for the random initialization in Figure 5? These details are important because the proposed synthetic representations are explicitly unit norm. Q2 Could the authors provide comparisons against normalized Gaussian random vectors, random orthogonal prototypes, and simplex-ETF/hyperspherical prototypes under the same optimization and hyperparameter budget? In particular, a Gram-matched baseline that preserves the pairwise geometry of the text prototypes while destroying their semantic correspondence would provide a substantially cleaner test of whether semantic content, rather than initialization geometry, matters. Q3 Does the semantic conclusion survive when supervised training cannot overwrite the initialization?Could the authors evaluate the correct, shuffled, and semantics-agnostic initializations when the classifier is frozen and only the visual projector is optimized, or report zero-step/early-training performance and the evolution of classifier weights? The current classifier is trainable, so a shuffled semantic assignment can simply be corrected during supervised optimization. Similarly, please compare gvg_v against a matched-capacity image-only projection head to determine whether its gains are specific to UMRL rather than additional downstream capacity.

8 comments
AuthorsOpenReview ↗

Response to AXNV — Part 1/14: Summary of Our Response

Summary of Our Response

We sincerely appreciate your detailed and constructive feedback. Our empirical study aims to better understand the factors influencing visual learning with unpaired text, particularly the roles of classifier initialization and textual semantics. Following your suggestions, we have conducted additional controlled experiments and statistical analyses to examine alternative explanations and further clarify our findings. Those additions provide further empirical evidence while remaining focused on our original research questions. We summarize several key points below.

  1. We provide additional statistical analyses across four visual backbones, including mean accuracy differences and 95% confidence intervals. The results show that semantics-agnostic initialization (SAI) achieves overall average performance close to semantic initialization (SI), although differences remain across individual settings and statistical equivalence is not established (A1).

  2. A central question in our study is how the geometric structure of classifier initialization affects visual learning. While text-derived prototypes encode semantic relationships, effective initialization does not necessarily require text-derived prototypes. Our experiments further show that different aspects of initialization geometry can affect downstream performance, with some differences persisting after supervised optimization (A2–A4, A6–A7).

  3. Motivated by variations in data distributions and class relationships, we propose SAI, which uses mm to control the expected inter-class similarity of synthetic representations. Even with a globally fixed m=0.9m=0.9, SAI achieves overall average performance close to SI (A5, A6).

  4. Our additional experiments show that model capacity alone cannot explain the observed performance differences, while auxiliary text classification loss provides no clear overall average benefit under the evaluated settings (A8–A9).

We will incorporate the corresponding results and analyses into the revision. We provide detailed responses to each comment below.

AuthorsOpenReview ↗

Response to AXNV — Part 2/14: W1: Random Seeds and Statistical Comparisons

W1. The claim of “comparable performance” is somewhat strong given only three random seeds and the absence of equivalence or non-inferiority tests. SAI also exhibits notable performance drops on individual tasks (e.g., 1.58 pp on SUN397 1-shot, 1.34 pp on Stanford Cars 2-shot, and 1.55 pp on FGVC-Aircraft 4-shot with DINOv2 ViT-L/14), with similar cases for DINO ViT-B/8 and ViT-B/16. These differences should be quantified rather than summarized as “generally comparable.”

A1. We appreciate your comment. First, we clarify the random-seed protocol in our original manuscript and provide additional quantitative analyses. We use three model-initialization seeds to control random network parameter initialization and, depending on the experimental setting, representation-sampling seeds to control text selection, random prototype sampling, or synthetic representation generation. Under a 3×3 configuration, each model seed is combined with three representation-sampling seeds, yielding nine runs per condition. Standard random initialization (RI) uses only the three model-initialization seeds. In the four-backbone comparisons below, SAI follows the 3×3 configuration, whereas SI uses a fixed text-sampling seed, yielding three runs per condition.

Second, to quantify performance differences more rigorously, we first average the results across runs for each experimental condition and calculate the corresponding signed accuracy differences and their absolute values. We then separately average the signed differences and absolute differences across experimental conditions within each dataset, using the eight datasets as statistical units to compute the overall mean signed accuracy difference, mean absolute accuracy difference, and their two-sided 95% Student's tt confidence intervals (CIs).

Using this protocol, we further analyze SI and SAI across four visual backbones (DINOv2 ViT-B/14, DINOv2 ViT-L/14, DINO ViT-B/16, and DINO ViT-B/8), covering eight datasets under 1/2/4-shot settings. The overall mean signed accuracy differences (SAI − SI; 95% CIs) are −0.361 [−0.776, +0.053], −0.514 [−0.877, −0.150], −0.363 [−0.967, +0.242], and −0.252 [−0.993, +0.490] percentage points (pp), respectively. Only ViT-L/14 shows a statistically significant average advantage for SI. The corresponding mean absolute accuracy differences (95% CIs) are 0.554 [0.276, 0.832], 0.682 [0.441, 0.923], 0.878 [0.477, 1.280], and 0.847 [0.418, 1.276] pp, respectively.

Taken together, those results provide a more precise context for our use of "generally comparable," which describes the overall performance trend across the evaluated settings rather than statistical equivalence or non-inferiority. We will clarify this distinction and incorporate the additional quantitative analyses into the revision.

AuthorsOpenReview ↗

Response to AXNV — Part 3/14: W2: Permutation Control and Frozen Classifier

W2. Since both the classifier and visual projector remain trainable, supervised optimization may relearn the disrupted semantic correspondence. Thus, the permutation experiment does not necessarily establish that text semantics are unimportant. Freezing the classifier or examining zero-step/early-training performance and classifier weight evolution would provide a cleaner test.

A2. We appreciate your insightful suggestion. Our study does not simply seek to determine whether textual semantics are important; rather, it investigates the geometric structure of class representations and its effects on visual learning. Textual semantics are reflected in both individual class prototypes and their inter-class similarity relationships, which collectively define the geometry of class representations. To investigate the role of class-label correspondence, we use random-permutation initialization (RSI), which randomly permutes text-derived class prototypes before classifier initialization. This permutation preserves the pairwise geometry of the prototype set but changes the correspondence between prototypes and class labels, thereby altering the label-indexed geometry. Thus, RSI examines the effects of class-label correspondence and its associated geometry on visual learning, rather than simply removing textual semantics.

More broadly, our study does not assume that effective classifier initialization must rely on geometry derived from real textual semantics. Through SAI, we explore whether synthetic class representations can provide effective initialization geometry without relying on real text prototypes.

To further investigate these factors, we compare SI, SAI, and RSI under two classifier settings: with the classifier WW either frozen or trainable, while keeping the visual projector trainable in both settings. For RSI, we report the average results over nine random permutations. The original permutation experiment is presented in Figure 6 of the manuscript. Together, these comparisons examine the effects of representation sources and class-label correspondence under different classifier-training settings. All experiments use the same image classification loss and cover eight datasets under 1/2/4-shot settings. Tables 1 and 2 report the results with classifier WW frozen and trainable, respectively.

Table 1. Comparison of different initialization methods with the classifier WW frozen.

MethodDTD 1-shotDTD 2-shotDTD 4-shotUCF101 1-shotUCF101 2-shotUCF101 4-shotOxford Pets 1-shotOxford Pets 2-shotOxford Pets 4-shotOxford Flowers 1-shotOxford Flowers 2-shotOxford Flowers 4-shotStanford Cars 1-shotStanford Cars 2-shotStanford Cars 4-shotSUN397 1-shotSUN397 2-shotSUN397 4-shotCaltech101 1-shotCaltech101 2-shotCaltech101 4-shotFGVC Aircraft 1-shotFGVC Aircraft 2-shotFGVC Aircraft 4-shot
SI40.20±0.3949.22±0.5059.94±0.1648.71±0.5361.37±0.8468.06±0.2470.21±0.3773.25±1.1986.96±0.2889.48±0.2795.20±0.2698.46±0.1115.61±0.1424.76±0.3836.49±0.2741.66±0.1950.42±0.4258.25±0.1482.48±0.5190.45±0.2793.85±0.2015.21±0.1823.25±0.1830.20±0.26
SAI41.23±0.6650.00±0.3761.07±0.1149.71±0.7562.87±0.0770.27±0.3773.41±1.3377.34±0.2088.33±0.3693.51±0.1897.15±0.2599.03±0.1013.98±0.4324.27±0.7438.99±0.1238.86±0.4449.83±0.0859.06±0.1583.38±0.1391.04±0.2395.28±0.0314.07±0.0923.11±0.2832.41±0.13
RSI39.57±0.1848.11±0.2859.15±0.0945.84±0.2258.25±0.3466.02±0.2566.12±0.3973.37±0.1385.31±0.0987.62±0.1994.63±0.0597.94±0.089.92±0.3618.48±0.0929.83±0.1331.91±0.0742.52±0.0652.34±0.0480.84±0.1389.46±0.0393.84±0.0512.64±0.1919.60±0.1527.63±0.04
Reviewer_rtZPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether the performance gains in supervised UMRL for image classification depend on text semantics. The experiments show that removing the text classification loss or randomly reassigning text representations to different classes has little effect on performance. Synthetic vectors also achieve performance comparable to text-derived classifier initialization.

Critical strengths
  • The paper offers a useful analysis of whether text semantics are necessary for the observed gains in supervised UMRL.

  • The findings are supported across multiple datasets, visual backbones, and training settings.

Critical weaknesses
  • The generality of the findings remains insufficiently established. It remains unclear whether the conclusions hold without strong pretrained visual encoders or with different text encoders.

  • The text classification loss is evaluated only after reducing text representations to one per class. Whether removing this loss also has little impact when multiple text representations are used remains unclear.

Questions for authors
  • The findings may depend on the quality of the pretrained visual representations. Although several DINO backbones are evaluated, it remains unclear whether the conclusions hold without strong pretrained visual encoders.

  • The experiments primarily use OpenLLaMA-3B-v2 as the text encoder. It remains unclear whether the same findings hold when using other text encoders.

  • The text classification loss is evaluated only after reducing the number of text representations to one per class. It remains unclear whether its limited effect on performance also holds when multiple text representations are used.

  • The analysis is limited to supervised image classification, where text is primarily used for classifier initialization. It remains unclear whether the findings extend to other unpaired multimodal learning settings involving explicit cross-modal alignment or prediction.

14 comments
AuthorsOpenReview ↗

Response to rtZP — Part 1/10: Summary of Our Response

Summary of Our Response

We sincerely appreciate your thoughtful comments and constructive suggestions. In response, we have conducted additional experiments to further examine the scope of our empirical findings across different visual and text encoders, experimental settings, and multimodal learning tasks. We summarize our responses below.

  1. We extend the experiments to ImageNet-1K pretrained ResNet18/50. The results show that semantics-agnostic initialization (SAI) achieves average performance close to semantic initialization (SI), while the benefits of visual projection and text-derived initialization may vary across visual encoders (A1).

  2. We replace OpenLLaMA-3B-v2 with RoBERTa-base. The results show that SAI achieves average performance close to SI, while the additional text classification loss provides no reliable overall improvement. The statistical evidence for the benefit of text-derived initialization may differ across text encoders (A2).

  3. We evaluate the text classification loss with 1, 5, 10, 20, and 30 text representations per class. The results show no reliable overall performance improvement across the evaluated settings (A3).

  4. We show that our semantics-agnostic synthetic representations can also facilitate category-level image-text retrieval without instance-level image-text pairs, achieving consistent improvements across both datasets (A4).

We will incorporate the corresponding results and analyses into the revision. We provide detailed responses to each comment below.

AuthorsOpenReview ↗

Response to rtZP — Part 2/10: Statistical Analysis and Q1 Experimental Overview

Q1. The findings may depend on the quality of the pretrained visual representations. Although several DINO backbones are evaluated, it remains unclear whether the conclusions hold without strong pretrained visual encoders.

A1. We appreciate your thoughtful suggestion. Our empirical study builds upon the supervised UMRL setting established in [1], and our primary experiments follow its default visual encoder configuration. We further extend the analysis to additional DINOv2 backbones. Following your suggestion, we conduct additional experiments using ImageNet-1K supervised pretrained ResNet18/50 to examine whether our main empirical observations remain consistent across different pretrained visual backbones.

We also evaluate a randomly initialized ResNet50 with a frozen visual encoder under the 1-shot setting. It achieves accuracies of only 3.63±0.44% and 0.78±0.22% on DTD and SUN397, respectively, indicating limited classification capability under those settings. We focus our additional experiments on pretrained ResNet18/50 across eight datasets and 1/2/4-shot settings. For each comparison, we report the overall mean accuracy difference, mean absolute difference, and their corresponding 95% confidence intervals, using the eight datasets as statistical units after aggregating results within each dataset.

[1] Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models. ICLR 2026.

AuthorsOpenReview ↗

Response to rtZP — Part 3/10: Q1: Text Classification Loss with ResNet18/50

A1.1. Effect of Text Classification Loss with Varying Numbers of Text Representations per Class

We evaluate the effect of the additional text classification loss using k∈1,5,10,20,30k\in\\{1,5,10,20,30\\} text representations per class across eight datasets under 1/2/4-shot settings, yielding 120 experimental conditions. Tables 1 and 2 report the mean accuracy difference Δ=Aw/Lt−Aw/oLt\Delta=A_{\mathrm{w/}\mathcal L_t}-A_{\mathrm{w/o}\mathcal L_t}.

Table 1. Effect of Text Classification Loss with Varying Numbers of Text Representations per Class (ResNet18).

kk / shotDTD·1DTD·2DTD·4UCF101·1UCF101·2UCF101·4Oxford Pets·1Oxford Pets·2Oxford Pets·4Oxford Flowers·1Oxford Flowers·2Oxford Flowers·4Stanford Cars·1Stanford Cars·2Stanford Cars·4SUN397·1SUN397·2SUN397·4Caltech101·1Caltech101·2Caltech101·4FGVC Aircraft·1FGVC Aircraft·2FGVC Aircraft·4
1+0.32−0.02−0.27−0.08+0.20−0.01−0.21−0.18+0.04−0.69−0.04+0.08−0.09+0.26+0.18+0.26−0.170.00+0.09+0.08+0.03+0.02+0.01+0.11
5+0.75−0.76−0.48−0.17−0.73−0.25+0.50+0.05−0.20−1.84−0.28−0.26+0.49+0.70+0.54+0.48−0.60−0.08+0.49−0.02−0.11+0.05+0.16−0.11
10+1.27−1.01+0.31+0.32−1.08−0.47+0.29+0.56−0.24−1.49−0.59−0.54+0.42+0.74+0.62+1.05−0.58−0.03−0.20−0.05−0.23−0.04+0.07−0.23
20+0.74−0.79+0.07+0.16−1.30−0.31+0.17+0.84−0.23−1.72−1.08−0.47+1.10+0.83+0.76+1.16−0.36−0.08−0.25−0.34−0.32+0.030.00−0.26
30+1.22−0.24+0.55−0.13−0.95−0.81−2.28+0.59−0.16−1.67−0.72−0.53+0.87+0.68+0.67+1.28+0.02+0.08−0.46−0.61−0.77−0.03−0.02−0.79

Table 2. Effect of Text Classification Loss with Varying Numbers of Text Representations per Class (ResNet50).

kk / shotDTD·1DTD·2DTD·4UCF101·1UCF101·2UCF101·4Oxford Pets·1Oxford Pets·2Oxford Pets·4Oxford Flowers·1Oxford Flowers·2Oxford Flowers·4Stanford Cars·1Stanford Cars·2Stanford Cars·4SUN397·1SUN397·2SUN397·4Caltech101·1Caltech101·2Caltech101·4FGVC Aircraft·1FGVC Aircraft·2FGVC Aircraft·4
1+0.68−0.06+0.01+0.38−0.06+0.05+0.33−0.01+0.01−0.07−0.31+0.34+0.74+0.16+0.30+0.06−0.20−0.01+0.13−0.01+0.09+0.03+0.14−0.06
5+0.27−0.94+0.65−1.67+0.76+0.11−0.71−2.14−0.30+0.99+0.30+1.23+1.88+1.09+0.98+0.03+0.52−0.09−0.06−0.25+0.20+0.30+0.43+0.79
10+1.22−1.28+0.32−0.79+1.52+0.18+0.10−2.96−0.29+1.65−0.55+0.50+1.69+0.86+1.31+0.97+0.74+0.14−0.08−0.50+0.19+0.08+0.04+1.08
20+1.44−0.96+0.65−0.42+0.83−0.08+0.93−3.88−0.23+1.30−0.42+0.44+1.73+1.05+1.42+0.45+1.14+0.29+0.38−0.11+0.11+0.32−0.15+1.04
30+0.83−1.17−0.22+0.97+0.73−0.67+0.96−3.76−0.23−0.93−0.08+0.19+2.05+0.77+1.28+0.61+0.71+0.48+0.13+0.04+0.36−0.11−0.06+1.08

The overall mean accuracy differences are −0.073 pp (95% CI: [−0.410, +0.264] pp) for ResNet18 and +0.204 pp (95% CI: [−0.247, +0.655] pp) for ResNet50. Neither difference is statistically significant at the 0.05 level. The corresponding mean absolute differences are 0.462 pp (95% CI: [0.290, 0.634] pp) and 0.651 pp (95% CI: [0.364, 0.939] pp), respectively. Overall, while the additional text classification loss provides improvements in some configurations, it does not demonstrate a reliable overall improvement across the evaluated datasets and few-shot settings. Those findings are consistent with our original observation and extend the analysis to ImageNet-supervised pretrained ResNet18/50. We will include those additional results in the revision.

Reviewer_iFy8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the sources of performance improvements in supervised UMRL, such as the number of text representations per class, the text classification loss, the learnable visual projector, and text-derived classifier initialization. The authors find that the visual projector and classifier initialization contribute substantially, while correct text-to-class correspondence has a limited effect. Based on these observations, the paper proposes an initialization method that does not depend on text semantics. It achieves performance close to text-based initialization across multiple datasets and visual backbones while removing the need for a text encoder.

Critical strengths
  1. The research question is valuable. The paper examines whether improvements from multimodal learning actually come from text semantics. This helps clarify the role of the auxiliary modality and encourages the development of more appropriate baselines.

  2. The method is simple and practically useful. In many settings, synthetic initialization achieves accuracy close to text-based initialization while reducing the computation and model dependencies needed to obtain text representations.

Critical weaknesses
  1. Synthetic initialization loses much of the gain from text-based initialization on some tasks. On Cars with 1 shot, it improves over random initialization by only 0.39 percentage points, compared with 1.56 for text-based initialization. Aircraft with 2 shots shows a similar gap, which deserves more discussion.

  2. The normalization of the random baselines in Table 2 is not specified. The comparison with the proposed unit-length vectors needs clarification before drawing conclusions about the source of the performance gap.

  3. The projector gains reported in Figures 1 and 4 do not match. For SUN397 with 1 shot, they are 4.18 and 3.24 percentage points, respectively. This discrepancy needs to be resolved to support the component analysis.

Questions for authors
  1. Why does synthetic initialization retain only a small portion of the gains from text-based initialization in the Cars and Aircraft settings discussed above? Does this suggest that the conclusion should be qualified by task?

  2. Were the random baselines in Table 2 normalized in the same way as the proposed vectors? If not, does the performance gap remain when their lengths are matched?

  3. What settings correspond to the SUN397 results in Figures 1, 4, and 5? How should the contribution of the visual projector be calculated?

3 comments

We sincerely appreciate your thoughtful review and provide detailed responses to your comments below.

W1. SAI achieves smaller gains over random initialization than SI on certain tasks, such as Cars (1-shot: +0.39 vs. +1.56 pp) and Aircraft (2-shot). How should these task-specific results be reflected in the scope of the paper's conclusions?

A1. We appreciate your careful observation. Given differences in data distributions and the amount of available labeled supervision, the relative gains of SAI and SI over RI may vary across tasks, as you noted. Our study aims to investigate whether synthetic class representations can provide effective classifier initialization without relying on text-derived prototypes.

To assess their overall performance, we analyze all 24 settings across eight datasets and 1/2/4-shot configurations. Both SI and SAI use all combinations of three representation-sampling seeds and three model-initialization seeds, resulting in nine runs per method per setting. The SI and SAI accuracies used in this analysis are each averaged over these nine runs. We then compute the signed SAI–SI accuracy differences and their absolute values for each setting, and separately average these two quantities across shot settings within each dataset. The eight datasets serve as statistical units for calculating two-sided 95% Student's tt confidence intervals.

The resulting estimates show that the overall mean accuracy difference (SAI − SI) is +0.075 percentage points (pp) (95% CI: [−0.543, +0.694] pp), with a mean absolute difference of 0.551 pp (95% CI: [0.145, 0.958] pp). These results indicate relatively close overall average performance between SAI and SI.

We will clarify the scope of our observations in the revision: SAI achieves overall average performance close to SI under the evaluated settings, while its gains over RI vary across tasks and can be smaller than those of SI.

W2. Table 2 does not clarify whether the random baselines were normalized to match the proposed unit-length vectors. Does the performance gap persist under consistent normalization?

A2. Thank you for your helpful suggestion. All three random baselines (Gaussian, Uniform, and Laplace) in Table 2, as well as Semantic Init., use row-wise L2 normalization at initialization to satisfy the same unit-length constraint as our Semantics-Agnostic Init. method. Therefore, the performance differences reported in Table 2 are observed under consistent normalization conditions. We will clarify this detail in the revision.

W3. For SUN397 (1-shot), the projector gains differ between Figures 1 and 4 (4.18 vs. 3.24 percentage points). Could the authors clarify the experimental settings in Figures 1, 4, and 5, and how the projector contribution is calculated?

A3. We appreciate your careful observation. Figures 1, 4, and 5 examine visual projection and classifier initialization under different experimental configurations. We clarify the settings and corresponding performance gains below.

(1) Figures 1 and 5: Figure 1 presents a stepwise ablation based on the original supervised UMRL framework. Since the visual representations are 384-dimensional and the text representations from OpenLLaMA-3B-v2 are 3200-dimensional, a 384→3200 visual projector is used to enable both modalities to share a 3200→C classifier, where C denotes the number of classes. For SUN397 (1-shot):

  • 34.15% → 38.33% (+4.18 pp): Introducing the 384→3200 projector together with a randomly initialized 3200→C classifier, compared with the baseline without a projector and with a 384→C classifier. This gain reflects the combined architectural change rather than the isolated contribution of the projector.
  • 38.33% → 41.07% (+2.74 pp): Replacing random classifier initialization with text-derived initialization under the same projector and classifier architecture. This comparison corresponds to Figure 5.

(2) Figure 4: To examine the effect of the visual projector gvg_v while controlling for classifier dimensionality, we use a 384→384 projector, keeping the classifier architecture (384→C) and its initial weights identical across the two settings. Both classifiers remain trainable under the same training protocol. Introducing gvg_v improves accuracy from 34.15% to 37.39%, yielding +3.24 pp. This comparison controls classifier dimensionality, although the projector introduces additional trainable parameters.

In summary, the +4.18 pp and +3.24 pp gains reflect different experimental configurations and should not be interpreted as identical measures of the projector's contribution. To avoid ambiguity, we will revise the labels and caption of Figure 1 to explicitly indicate that the +4.18 pp gain includes changes in both visual projection and classifier dimensionality. We will also clarify the corresponding experimental settings and gain calculations in the revision.

Thank you for the clarification. I remain unconvinced by the strength and consistency of SAI’s benefits and will maintain my original score.

Thank you for your further consideration. Our study aims to investigate whether effective classifier initialization can be achieved without relying on text-derived prototypes, rather than to establish SAI as consistently superior to SI. Also, extensive additional experiments presented in our responses to the other reviewers provide further support for our central observation:

  • With a globally fixed m=0.9m=0.9 across eight datasets and 1/2/4-shot settings, the mean accuracy difference (SAI − SI) is −0.146 percentage points (pp) (95% CI: [−0.684, +0.392] pp), with a mean absolute difference of 0.548 pp. Under this setting, SAI also achieves higher mean accuracy than Gaussian-L2, Random Orthogonal, and Simplex-ETF across all 24 dataset–shot settings.
  • Separate experiments across different visual backbones and text encoders further support this observation under a broader range of experimental settings. The mean SAI–SI differences are −0.444 pp with ResNet18, −0.149 pp with ResNet50, and +0.007 pp with RoBERTa-base. Additional experiments with DINO backbones show a similar overall trend, although SI exhibits a statistically significant average advantage with ViT-L/14. Our further logit-scale analysis provides additional insights into the factors influencing initialization performance.

Taken together, those extensive controlled experiments provide consistent empirical evidence that semantics-agnostic initialization can achieve overall average classification performance close to text-derived initialization across the evaluated settings. We believe those findings strengthen the empirical contributions of our study and will clarify their scope in the revision. We sincerely appreciate your feedback and would be happy to provide further clarification if you have any remaining concerns or suggestions. We hope the additional evidence will encourage you to reconsider your assessment.

Comments on the paper

11 comments
AuthorsOpenReview ↗

Response to AXNV — Part 9/14: W6: Geometric Initialization Baselines

W6. Additional comparisons with row-normalized Gaussian, random orthogonal, and simplex-ETF/hyperspherical prototypes are suggested, ideally including controls matched to the norm and Gram-spectrum of text-derived prototypes, under the same optimization and hyperparameter budget.

A6. Thank you for the reviewer's suggestion. We add three geometric initialization baselines: Gaussian-ℓ2\ell_2, Random Orthogonal, and Simplex-ETF—and compare them with SI and SAI under the same optimization protocol and hyperparameter search budget. Specifically, Gaussian-ℓ2\ell_2 is constructed by applying row-wise ℓ2\ell_2 normalization to standard Gaussian random vectors; Random Orthogonal uses unit-norm class prototypes that are mutually orthogonal; and Simplex-ETF uses unit-norm class prototypes with pairwise inner products of −1/(C−1)-1/(C-1), where CC is the number of classes. All three geometric baselines, SI, and SAI have unit classifier row norms at initialization. For SAI, we fix m=0.9m=0.9 across all datasets and shot settings. Furthermore, we introduce an additional GI baseline, whose construction and experimental results are detailed in A3.

Table 6. Comparison of different classifier initialization methods.

MethodDTD 1-shotDTD 2-shotDTD 4-shotUCF101 1-shotUCF101 2-shotUCF101 4-shotOxford Pets 1-shotOxford Pets 2-shotOxford Pets 4-shotOxford Flowers 1-shotOxford Flowers 2-shotOxford Flowers 4-shotStanford Cars 1-shotStanford Cars 2-shotStanford Cars 4-shotSUN397 1-shotSUN397 2-shotSUN397 4-shotCaltech101 1-shotCaltech101 2-shotCaltech101 4-shotFGVC Aircraft 1-shotFGVC Aircraft 2-shotFGVC Aircraft 4-shot
SI41.21±0.1550.63±0.3561.21±0.2850.30±0.2062.57±0.0869.62±0.4073.08±0.2576.32±0.4988.74±0.3393.26±0.1496.82±0.2399.01±0.0615.33±0.2225.90±0.5740.92±0.2141.07±0.2351.50±0.2160.09±0.0883.34±0.8691.40±0.6595.07±0.2315.04±0.2423.77±0.2633.31±0.15
Gaussian-ℓ2\ell_240.02±1.4449.20±0.7659.56±0.5447.05±0.8959.73±1.5568.44±0.5368.34±1.2874.20±0.7887.11±0.8989.58±1.5197.13±0.6798.50±0.2513.58±0.3424.37±0.4338.53±0.3735.74±1.0346.78±0.2756.35±0.8282.15±0.8489.37±0.9194.36±0.0814.24±0.2122.38±1.0631.94±0.28
Random Orthogonal40.24±2.1347.98±3.0059.88±0.4845.21±0.9460.10±0.5268.26±0.7966.61±2.5174.68±2.1187.90±0.0887.94±1.0596.45±0.9598.81±0.1914.01±0.1423.97±0.4537.71±0.8335.32±1.2247.26±0.2857.23±0.1681.43±0.5389.01±2.1694.62±0.1913.68±0.4222.24±0.6532.20±0.74
Simplex-ETF40.20±2.1547.94±3.0259.54±0.5745.12±0.9960.10±0.5068.20±0.7467.69±1.5274.63±2.1487.82±0.0787.90±1.0696.47±0.9498.81±0.1913.99±0.1323.79±0.6937.42±0.4835.33±1.2447.26±0.2857.12±0.3081.39±0.5588.98±2.2194.65±0.1813.67±0.4222.14±0.6832.21±0.80
SAI (m=0.9)41.62±0.3950.24±0.2661.34±0.0349.97±0.3462.20±0.3369.77±0.4274.03±0.4077.55±0.3289.13±0.1993.52±0.2897.14±0.1298.98±0.1614.14±0.4125.14±0.0940.63±0.2039.25±0.1550.21±0.0959.48±0.0583.71±0.1491.53±0.1995.30±0.1114.51±0.0923.05±0.1233.55±0.54
AuthorsOpenReview ↗

Response to AXNV — Part 10/14: W6: Geometric Baseline Comparisons and Analysis

Based on Table 6 we define the accuracy difference as Δ=ASAI−Abaseline\Delta=A_{\mathrm{SAI}}-A_{\mathrm{baseline}}. Compared with Gaussian-ℓ2\ell_2, Random Orthogonal, and Simplex-ETF, SAI achieves overall mean accuracy differences of +1.973 pp (95% CI: [+1.103, +2.842] pp), +2.218 pp (95% CI: [+1.430, +3.007] pp), and +2.234 pp (95% CI: [+1.531, +2.938] pp), respectively, with all confidence intervals above zero. SAI also achieves higher mean accuracy than all three baselines across all 24 dataset-shot settings. Since all differences are positive, the mean absolute differences equal the corresponding mean differences. Compared with SI, SAI achieves an overall mean accuracy difference of −0.146 pp (95% CI: [−0.684, +0.392] pp) and a mean absolute difference of 0.548 pp (95% CI: [0.240, 0.855] pp), indicating relatively close overall mean performance under the evaluated settings.

Overall, those results show that matching initial classifier row norms does not guarantee identical visual classification performance across initialization methods. As data distributions may vary across datasets, SAI enables tunable initialization by synthesizing semantics-agnostic class representations with expected inter-class similarity controlled by mm. Notably, we fix m=0.9m=0.9 across all datasets and shot settings without additional tuning. Under those conditions, SAI achieves higher mean accuracy than all three geometric baselines across all 24 settings, with overall mean performance close to SI. Those findings further support our central observation: effective classifier initialization does not necessarily require genuine textual semantics, as semantics-agnostic synthetic representations can also provide effective initialization structures. We will include those experiments and analyses in the revision.

AuthorsOpenReview ↗

Response to AXNV — Part 11/14: W7: Identical Class Prototypes at m = 1

W7. When m=1m=1, Equation (2) produces identical class prototypes, yet performance shows relatively weak sensitivity to mm. This raises the possibility that initialization scale or normalization, rather than the proposed inter-class geometry, may explain the results.

A7. Thank you for your careful observation. When m=1m=1, Equation (2) indeed produces identical class prototypes. Those prototypes are used only to initialize the trainable classifier; the resulting class weights remain trainable during subsequent training. The classifier WW and visual projector are jointly optimized using the image classification loss. Under cross-entropy loss, ground-truth labels generate class-dependent gradients, allowing initially identical class weights to gradually diverge. Thus, the competitive performance at m=1m=1 suggests that subsequent supervised optimization can learn a discriminative classifier even without initial class distinctions.

To examine whether initialization norm alone explains the results, we added unit-norm baselines, including Gaussian-ℓ2\ell_2, Random Orthogonal, and Simplex-ETF, and evaluated them under the same optimization protocol and hyperparameter search budget (see A6). Despite having identical class-weight norms, these baselines achieve different levels of final performance, indicating that weight-norm constraints alone cannot fully explain the observed differences. Together, those results suggest that final visual classification performance depends on both initialization structure and subsequent supervised optimization. We will clarify those findings and their scope in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.