- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper revisits supervised Unpaired Multimodal Representation Learning (UMRL) and investigates which components are responsible for the performance improvements previously attributed to auxiliary text representations. The authors decompose the supervised UMRL pipeline into four factors: the number of text representations per class, the auxiliary text classification loss, the learnable visual projector, and text-derived classifier initialization. Their experiments suggest that increasing the number of text representations and using the text classification loss have limited effects, whereas the visual projector and classifier initialization contribute more substantially to performance. The paper then studies whether the benefit of text-derived initialization actually depends on semantic correspondence. The authors randomly permute the assignment between text representations and class labels and observe little degradation. Motivated by this observation, they propose a semantics-agnostic initialization scheme in which each class representation is synthesized as where is shared across classes and is sampled from the unit sphere in the subspace orthogonal to . The parameter m controls the expected pairwise cosine similarity. The resulting representations are used to initialize the classifier without relying on a text encoder. Experiments across multiple few-shot image classification datasets, ImageNet-1K, several DINO/DINOv2 backbones, and a full fine-tuning setting show that the proposed initialization can often approach the performance of text-derived initialization while avoiding the computational and storage cost of OpenLLaMA-3B-v2. Based on these observations, the authors argue that text semantics may not be the main source of the performance gains observed in supervised UMRL.
Critical strengths
1.The paper asks an important diagnostic question and performs a broad empirical decomposition.A major strength of the paper is that it revisits a potentially under-examined assumption in supervised UMRL: whether the observed benefit of auxiliary text actually comes from semantic transfer. Rather than treating the original multimodal pipeline as a monolithic method, the paper decomposes it into several components and separately studies the effects of the text loss, number of text representations, visual projection, and classifier initialization. This diagnostic perspective is useful because it challenges a natural but not necessarily justified interpretation of previous UMRL results. 2.The empirical evaluation is relatively extensive and has practical implications.The evaluation covers multiple datasets, several few-shot regimes, ImageNet-1K, multiple DINO/DINOv2 backbones, and a full fine-tuning setting. The practical observation that a text encoder may be unnecessary in this particular supervised pipeline is potentially useful, especially because the proposed alternative removes the OpenLLaMA-3B-v2 storage and text-extraction overhead.
Critical weaknesses
-
The experiments do not causally isolate “text semantics”; the current results can also be explained by trainable re-optimization and additional downstream capacity.The main conceptual weakness is that the experiments do not cleanly isolate semantic information from optimization and architectural effects. The paper interprets the weak effect of class-label permutation and the success of semantics-agnostic initialization as evidence that text semantics are not a primary driver of supervised UMRL gains. However, in the evaluated setting, both the classifier and the learnable visual projector remain trainable after initialization. Therefore, destroying the initial semantic correspondence does not prevent supervised training from subsequently relearning the class assignment. The permutation experiment establishes that the initial association between a particular text vector and a particular classifier row is not essential after optimization, but this is weaker than establishing that semantic information is unimportant. This concern is particularly important because the proposed synthetic representations are used only as classifier initialization. Once supervised optimization starts, the class weights are free to move away from their initial values. A much cleaner semantic intervention would freeze the classifier initialized from text/synthetic prototypes and train only the visual mapping, or alternatively compare zero-step and early-training performance before the initialization is overwritten. A geometry-preserving counterfactual, such as preserving the complete Gram matrix of text prototypes while destroying semantic correspondence, would also provide a much stronger test. Relatedly, the claimed contribution of the visual projector is not fully isolated either. The comparison with and without simultaneously changes the number of trainable parameters and the depth/capacity of the downstream visual classifier. Thus, Figure 4 demonstrates that adding a trainable projection module improves classification, but it does not establish that this improvement is specifically related to UMRL or cross-modal transfer. A matched-capacity image-only MLP/projector baseline is necessary. The paper analyzes the four components largely sequentially rather than through a factorial design. After finding that the text loss has little effect, the authors remove it and continue the remaining analysis under the modified pipeline. This does not rule out interactions between , , , and the number of text examples. Therefore, the statement that the projector and initialization are the “main contributors” is stronger than what the current one-factor-at-a-time ablations strictly establish.
-
The semantics-agnostic baseline is insufficiently controlled, and the evidence for “comparable performance” is weaker than claimed.The second major concern is that the proposed semantics-agnostic initialization is not compared against sufficiently well-matched initialization controls. Equation (2) explicitly constructs unit-norm vectors, whereas the paper does not clearly specify whether the Gaussian, uniform, Laplace, or standard random classifier baselines are normalized to have the same row norm, logit scale, covariance, and spectrum as the proposed initialization or the text-derived prototypes. This matters because classifier initialization scale alone can substantially alter few-shot optimization. This issue is especially visible in the sensitivity experiment. Whenm=1, Equation (2) gives for every class, meaning that all classifier rows are exactly identical at initialization. Yet the paper reports relatively weak sensitivity to m. If an initialization with initially identical class prototypes performs similarly to one with distinct prototypes after training, this suggests that the relevant benefit may primarily be optimization-scale or normalization rather than the particular inter-class geometry introduced by the synthesis method. This possibility is not investigated. Stronger controls are therefore needed, including row-normalized Gaussian initialization, random orthogonal prototypes, simplex-ETF/hyperspherical prototypes, and ideally a baseline matched to the norm and Gram-spectrum of the text-derived prototypes. This omission is particularly notable because the related-work section itself discusses hyperspherical prototypes and simplex ETF classifiers. There is also an asymmetry in hyperparameter tuning. The proposed method introduces m, and Table 4 states that is selected using validation performance. This gives the semantics-agnostic method an additional dataset-dependent model-selection degree of freedom. Results with a single globally fixed m, selected independently of the target dataset and shot setting, would provide a cleaner comparison. Finally, the phrase “comparable performance” is somewhat stronger than the results warrant. Only three seeds are used and no equivalence or non-inferiority test is provided. Moreover, the additional-backbone experiments contain multiple nontrivial degradations. For example, on DINOv2 ViT-L/14 the semantics-agnostic initialization is lower by 1.58 points on SUN397 in the 1-shot setting, 1.34 points on Stanford Cars in the 2-shot setting, and 1.55 points on FGVC-Aircraft in the 4-shot setting. Similar drops appear for DINO ViT-B/8 and ViT-B/16. These are not isolated numerical noise and should be quantified rather than summarized only as “generally comparable.”
Questions for authors
Q1 Please clarify the exact initialization and normalization procedures for all classifier baselines. Are the rows of the text-derived classifier, Gaussian/Uniform/Laplace baselines, and standard random initialization normalized to the same norm? Are their logit scales matched? What distribution and variance are used for the random initialization in Figure 5? These details are important because the proposed synthetic representations are explicitly unit norm. Q2 Could the authors provide comparisons against normalized Gaussian random vectors, random orthogonal prototypes, and simplex-ETF/hyperspherical prototypes under the same optimization and hyperparameter budget? In particular, a Gram-matched baseline that preserves the pairwise geometry of the text prototypes while destroying their semantic correspondence would provide a substantially cleaner test of whether semantic content, rather than initialization geometry, matters. Q3 Does the semantic conclusion survive when supervised training cannot overwrite the initialization?Could the authors evaluate the correct, shuffled, and semantics-agnostic initializations when the classifier is frozen and only the visual projector is optimized, or report zero-step/early-training performance and the evolution of classifier weights? The current classifier is trainable, so a shuffled semantic assignment can simply be corrected during supervised optimization. Similarly, please compare against a matched-capacity image-only projection head to determine whether its gains are specific to UMRL rather than additional downstream capacity.
8 comments
Response to AXNV — Part 2/14: W1: Random Seeds and Statistical Comparisons
W1. The claim of “comparable performance” is somewhat strong given only three random seeds and the absence of equivalence or non-inferiority tests. SAI also exhibits notable performance drops on individual tasks (e.g., 1.58 pp on SUN397 1-shot, 1.34 pp on Stanford Cars 2-shot, and 1.55 pp on FGVC-Aircraft 4-shot with DINOv2 ViT-L/14), with similar cases for DINO ViT-B/8 and ViT-B/16. These differences should be quantified rather than summarized as “generally comparable.”
A1. We appreciate your comment. First, we clarify the random-seed protocol in our original manuscript and provide additional quantitative analyses. We use three model-initialization seeds to control random network parameter initialization and, depending on the experimental setting, representation-sampling seeds to control text selection, random prototype sampling, or synthetic representation generation. Under a 3×3 configuration, each model seed is combined with three representation-sampling seeds, yielding nine runs per condition. Standard random initialization (RI) uses only the three model-initialization seeds. In the four-backbone comparisons below, SAI follows the 3×3 configuration, whereas SI uses a fixed text-sampling seed, yielding three runs per condition.
Second, to quantify performance differences more rigorously, we first average the results across runs for each experimental condition and calculate the corresponding signed accuracy differences and their absolute values. We then separately average the signed differences and absolute differences across experimental conditions within each dataset, using the eight datasets as statistical units to compute the overall mean signed accuracy difference, mean absolute accuracy difference, and their two-sided 95% Student's confidence intervals (CIs).
Using this protocol, we further analyze SI and SAI across four visual backbones (DINOv2 ViT-B/14, DINOv2 ViT-L/14, DINO ViT-B/16, and DINO ViT-B/8), covering eight datasets under 1/2/4-shot settings. The overall mean signed accuracy differences (SAI − SI; 95% CIs) are −0.361 [−0.776, +0.053], −0.514 [−0.877, −0.150], −0.363 [−0.967, +0.242], and −0.252 [−0.993, +0.490] percentage points (pp), respectively. Only ViT-L/14 shows a statistically significant average advantage for SI. The corresponding mean absolute accuracy differences (95% CIs) are 0.554 [0.276, 0.832], 0.682 [0.441, 0.923], 0.878 [0.477, 1.280], and 0.847 [0.418, 1.276] pp, respectively.
Taken together, those results provide a more precise context for our use of "generally comparable," which describes the overall performance trend across the evaluated settings rather than statistical equivalence or non-inferiority. We will clarify this distinction and incorporate the additional quantitative analyses into the revision.
Response to AXNV — Part 3/14: W2: Permutation Control and Frozen Classifier
W2. Since both the classifier and visual projector remain trainable, supervised optimization may relearn the disrupted semantic correspondence. Thus, the permutation experiment does not necessarily establish that text semantics are unimportant. Freezing the classifier or examining zero-step/early-training performance and classifier weight evolution would provide a cleaner test.
A2. We appreciate your insightful suggestion. Our study does not simply seek to determine whether textual semantics are important; rather, it investigates the geometric structure of class representations and its effects on visual learning. Textual semantics are reflected in both individual class prototypes and their inter-class similarity relationships, which collectively define the geometry of class representations. To investigate the role of class-label correspondence, we use random-permutation initialization (RSI), which randomly permutes text-derived class prototypes before classifier initialization. This permutation preserves the pairwise geometry of the prototype set but changes the correspondence between prototypes and class labels, thereby altering the label-indexed geometry. Thus, RSI examines the effects of class-label correspondence and its associated geometry on visual learning, rather than simply removing textual semantics.
More broadly, our study does not assume that effective classifier initialization must rely on geometry derived from real textual semantics. Through SAI, we explore whether synthetic class representations can provide effective initialization geometry without relying on real text prototypes.
To further investigate these factors, we compare SI, SAI, and RSI under two classifier settings: with the classifier either frozen or trainable, while keeping the visual projector trainable in both settings. For RSI, we report the average results over nine random permutations. The original permutation experiment is presented in Figure 6 of the manuscript. Together, these comparisons examine the effects of representation sources and class-label correspondence under different classifier-training settings. All experiments use the same image classification loss and cover eight datasets under 1/2/4-shot settings. Tables 1 and 2 report the results with classifier frozen and trainable, respectively.
Table 1. Comparison of different initialization methods with the classifier frozen.
| Method | DTD 1-shot | DTD 2-shot | DTD 4-shot | UCF101 1-shot | UCF101 2-shot | UCF101 4-shot | Oxford Pets 1-shot | Oxford Pets 2-shot | Oxford Pets 4-shot | Oxford Flowers 1-shot | Oxford Flowers 2-shot | Oxford Flowers 4-shot | Stanford Cars 1-shot | Stanford Cars 2-shot | Stanford Cars 4-shot | SUN397 1-shot | SUN397 2-shot | SUN397 4-shot | Caltech101 1-shot | Caltech101 2-shot | Caltech101 4-shot | FGVC Aircraft 1-shot | FGVC Aircraft 2-shot | FGVC Aircraft 4-shot |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SI | 40.20±0.39 | 49.22±0.50 | 59.94±0.16 | 48.71±0.53 | 61.37±0.84 | 68.06±0.24 | 70.21±0.37 | 73.25±1.19 | 86.96±0.28 | 89.48±0.27 | 95.20±0.26 | 98.46±0.11 | 15.61±0.14 | 24.76±0.38 | 36.49±0.27 | 41.66±0.19 | 50.42±0.42 | 58.25±0.14 | 82.48±0.51 | 90.45±0.27 | 93.85±0.20 | 15.21±0.18 | 23.25±0.18 | 30.20±0.26 |
| SAI | 41.23±0.66 | 50.00±0.37 | 61.07±0.11 | 49.71±0.75 | 62.87±0.07 | 70.27±0.37 | 73.41±1.33 | 77.34±0.20 | 88.33±0.36 | 93.51±0.18 | 97.15±0.25 | 99.03±0.10 | 13.98±0.43 | 24.27±0.74 | 38.99±0.12 | 38.86±0.44 | 49.83±0.08 | 59.06±0.15 | 83.38±0.13 | 91.04±0.23 | 95.28±0.03 | 14.07±0.09 | 23.11±0.28 | 32.41±0.13 |
| RSI | 39.57±0.18 | 48.11±0.28 | 59.15±0.09 | 45.84±0.22 | 58.25±0.34 | 66.02±0.25 | 66.12±0.39 | 73.37±0.13 | 85.31±0.09 | 87.62±0.19 | 94.63±0.05 | 97.94±0.08 | 9.92±0.36 | 18.48±0.09 | 29.83±0.13 | 31.91±0.07 | 42.52±0.06 | 52.34±0.04 | 80.84±0.13 | 89.46±0.03 | 93.84±0.05 | 12.64±0.19 | 19.60±0.15 | 27.63±0.04 |
Response to AXNV — Part 1/14: Summary of Our Response
Summary of Our Response
We sincerely appreciate your detailed and constructive feedback. Our empirical study aims to better understand the factors influencing visual learning with unpaired text, particularly the roles of classifier initialization and textual semantics. Following your suggestions, we have conducted additional controlled experiments and statistical analyses to examine alternative explanations and further clarify our findings. Those additions provide further empirical evidence while remaining focused on our original research questions. We summarize several key points below.
We provide additional statistical analyses across four visual backbones, including mean accuracy differences and 95% confidence intervals. The results show that semantics-agnostic initialization (SAI) achieves overall average performance close to semantic initialization (SI), although differences remain across individual settings and statistical equivalence is not established (
A1).A central question in our study is how the geometric structure of classifier initialization affects visual learning. While text-derived prototypes encode semantic relationships, effective initialization does not necessarily require text-derived prototypes. Our experiments further show that different aspects of initialization geometry can affect downstream performance, with some differences persisting after supervised optimization (
A2–A4, A6–A7).Motivated by variations in data distributions and class relationships, we propose SAI, which uses m to control the expected inter-class similarity of synthetic representations. Even with a globally fixed m=0.9, SAI achieves overall average performance close to SI (
A5, A6).Our additional experiments show that model capacity alone cannot explain the observed performance differences, while auxiliary text classification loss provides no clear overall average benefit under the evaluated settings (
A8–A9).We will incorporate the corresponding results and analyses into the revision. We provide detailed responses to each comment below.