acceptodds
Withdrawn from ICLR 2027

When Language Gains Are Not Semantic: Rethinking Attribution in Language-Supervised Representation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language is increasingly used as auxiliary supervision for representation learning, yet improvements over single-task baselines are often interpreted as evidence that semantic information in language improves the learned representation. This inference is underdetermined by the usual comparison: introducing language supervision simultaneously changes the training objective, optimization dynamics, target geometry, and instance-level supervision. We study this problem as one of experimental attribution and introduce a controlled protocol that separates three distinct questions: whether an auxiliary intervention improves representation learning, whether correct input–target correspondence contributes to the gain, and whether structured targets provide information beyond generic instance-level supervision. Our protocol complements the standard sensor-only baseline with correspondence-destroying derangements and fixed unstructured instance targets, providing explicit controls for competing explanations of an observed improvement. We instantiate the framework in radar representation learning, where language and camera information are available during training but retrieval remains radar-only. On Boreas, correct captions improve Recall@1 by 3.18 points over radar-only training (95% CI [1.34, 5.02]). However, deranged captions and random instance targets also improve performance by 2.75 and 2.17 points, respectively. Correct captions exceed these controls by only +0.43 points (95% CI [−1.66, 2.52]) and +1.00 points ([−0.33, 2.33]), leaving both attribution effects unresolved. The same qualitative attribution result holds for richer camera-VLM descriptions, diagnostic-selected targets, and a reverse-direction replication on MulRan; direct frozen-camera supervision likewise fails to establish stable transfer. We further introduce a pre-training target audit based on cross-revisit stability and between-place collision. Increasing caption entropy from 2.91 to 10.19 bits reduces cross-revisit stability from 0.729 to 0.121, revealing a trade-off between descriptive richness and task-relevant invariance. Our results expose a broader methodological distinction in representation learning: a baseline improvement establishes that a training intervention helped, but does not identify which information within that intervention caused the gain. Semantic attribution therefore requires controls that separately test correspondence and target structure.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
language-supervised representation learning, semantic attribution, multimodal representation learning, auxiliary supervision, negative controls, cross-modal learning, learning with privileged information, radar representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_sAZUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

WHEN LANGUAGE GAINS ARE NOT SEMANTIC: RETHINKING ATTRIBUTION IN LANGUAGE-SUPERVISED REPRESENTATION LEARNING investigates whether performance gains from auxiliary language supervision come from the language alignment itself, or from other, typically uncontrolled, variables. For this, the authors analyze the impact of supervision with deranged captions and random instance targets, whose results are not statistically distinguishable from training with ordinary correct captions.

Critical strengths

S1 - The original research question is valid and often underexplored. The argument that adding language supervision introduces additional modifications beyond the semantic content of the captions warrants the need to investigate the true benefit of language alone.

S2 - The decomposition into intervention / correspondence / structure is a clean way to isolate the benefits of the language content itself. Furthermore, the control design is sensible, with the use of shuffled captions (with complete derangement) and random targets being consistent with the experimental setup proposed.

Critical weaknesses

W1 - A lot of important information is either omitted, under-explained, or deferred only to the appendix.

  1. The task of radar place recognition is only briefly defined mathematically, without much contextual explanation, making it harder for readers from different areas to follow the paper. The Boreas dataset is mentioned several times, but is never cited or properly introduced. Important details of radar preprocessing are also only given in the appendix. All of these make it harder for new readers to situate themselves on the task and context.

  2. L_place is defined only as a 'batch-hard triplet', without a clear description, explanation, citation or formulation. Hyperparameters and batch construction are identified in the appendix, which is not an issue, but the main text gives next to no information on the main training objective used. The functional form of L_aux is never actually defined; in Section 5.1 there's mention of InfoNCE, which I assume is the objective used, but that needs to be defined and made explicit earlier in the text. Furthermore, the projection head g is never described. These make it harder to understand the method being used to train the networks.

  3. Although the paper describes the general attributes used to construct captions, no concrete examples or detailed construction rules are given. The vocabularies are also never fully specified. For instance, what are the '27 meanings', 'all eight axes' and 'near + far' descriptors mentioned in Table 1? Without these, it's hard to properly interpret the obtained results.

W2 - Evidence doesn't fully support the claims, and the paper would benefit from a larger analysis of its results.

  1. Confidence intervals are extremely wide for the small gains observed. A tighter confidence interval (with more seeds, more evaluation routes, etc.) would make the results more meaningful.

  2. The batch size of 16 is very small for InfoNCE setups, which typically benefit from larger batch sizes. There are also no ablation studies to look at the effects of batch size and other hyperparameters (such as the small margin size of 0.2 used in the triplet objective).

  3. The intervention does not produce a statistically significant gain on MulRan, unlike on Boreas. Consequently, the replication does not reproduce the more interesting finding that significant auxiliary gains can persist without evidence of a benefit from correct language correspondence.

  4. The paper names several possible mechanisms that could lead to performance gains apart from language itself, but none of them are tested. Furthermore, an analysis of the model's latent space would help interpret the results. The triplet loss, for instance, could alone have left embeddings concentrated in a small latent region, and the auxiliary InfoNCE loss, which considers multiple negatives and uses mismatched or random targets as assigned positives in the control arms, could have served as a regularizer to spread representations a bit more, thus facilitating retrieval. This, though, cannot be concluded from the given evidence alone.

Questions for authors

Q1 - What does a caption look like? Please provide examples and the full vocabulary.

Q2 - How does the latent space differ between radar only and the different auxiliary arms?

Q3 - Does a regularizer-only baseline reproduce the gains obtained from L_aux?

Q4 - How does this behave under different hyperparameters and training times? Are the same behaviors and gains from L_aux maintained?

Reviewer_tyEeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether the improvements obtained by adding language supervision to representation learning with radar data are due to the semantics of the text or to other effects of training with auxiliary terms. To distinguish between these possibilities, the authors compare correct textual descriptions with reassigned ones and random targets. They evaluate these on the Boreas and MulRan datasets. Furthermore, the manuscript proposes to examine the descriptions before training using stability scores between visits to the same place and coincidences between different places.

Critical strengths
  1. The research question is relevant, as adding language supervision changes several training components, so an improvement over the baseline is not enough to attribute it to the semantics of the descriptions.

  2. The Boreas experiments show a concrete example of this problem and show why is important to include additional controls, as the reassigned descriptions and random targets also show performance improvement. The comparisons with paired seeds and confidence intervals also strengthen the case.

Critical weaknesses
  1. Language supervision, which is the main element of the research, is insufficiently described and validated. I think this is the main issue I have with the paper. For instance, for radar descriptions, the rules, thresholds, categories, and templates required to construct them are missing. For the ones that are generated from images, the full prompts, the allowed values, and image-description examples are missing. The appendices do not resolve these questions either. Furthermore, the inspection of 12 images offers little insight, as there is no explanation of how they were selected, who reviewed them, or what criteria were used. Without these details, it's hard to determine what semantic information is being tested and to distinguish whether the results reflect the absence of a measurable benefit due to language semantics or an experimental design that is inadequate for the task.

  2. The evidence is not enough to support the overall message of the paper. The experiments cover one approach per dataset, a single family of encoders, and a shared training configuration. MulRan does not replicate the core result from Boreas, as there is no demonstrable improvement from the auxiliary intervention, and the intervals are not narrow. These results are not sufficient to conclude that the effect depends on the dataset, especially since other evaluation conditions also vary. Furthermore, presenting the audit as a necessary condition is not justified. Overall, the conclusions go beyond what the experiments allow to establish.

  3. The novelty also needs to be evaluated, particularly in light of some work on representation learning that was not discussed, specifically Santurkar et al.’s “Is a Caption Worth a Thousand Images?”, which examines the impact of language-based supervision. Fang et al.’s “Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)” is also relevant here. While I do not deny that the radar-based case is worthy of study, I believe this work’s actual novelty is insufficient.

In addition to these limitations, there is an issue regarding clarity. From section 4 onwards, it becomes increasingly difficult to reconstruct the experiments, as information concerning the data and descriptions is scattered.

While the research question is relevant, I believe the work requires a substantial revision of its documentation, the validation of the supervision process, and the scope of its conclusions.

Questions for authors
  1. Can you provide details on how you construct the radar and camera descriptions, including rules, thresholds, categories, templates, prompts, etc? This information is necessary to evaluate what semantic content is being tested.

  2. How were the 12 images for manual inspection selected, and what criteria were used to consider the labels as acceptable? Did you examine pairs of visits to the same location to understand the causes of instability?

  3. How do you handle repeated descriptions in the loss function? With nine possible descriptions and batches of 16 observations, it's quite possible to have them. Do examples that share a description count as additional positives or as negatives?

  4. Why do you interpret the MulRan results as evidence that the effect depends on the dataset, when its confidence interval allows for an improvement similar to the one in Boreas?

Reviewer_bAYHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem of how to tell whether a gain from auxiliary language supervision from the meaning of the language itself. The authors argue that beating a single task baseline is not enough, because there are more variables added when adding a language, i.e., a loss, a projection head and instance level training signal. It proposes a decomposition method that compares correct language targets with two negative controls: captions shuffled (with other instances caption), and a fixed random target vector per sample. The authors perform experiments on radar place recognition task on Boreas and MulRan dataset, using radar derived captions, camera-VLM captions and fronzen camera features as training only targets. The results show that on Boreaas correct captions improve recall@1 over radar only training, while the shuffled and random controls also recover most of the gain. On MulRan, no auxiliary target helps.

Critical strengths

The paper studies an important question for the real contribution of language supervision in representation learning as auxiliary task. It breaks down the variables introduced by the auxiliary task and validates each of them with controlled experiments. It shows that a gain that would normally be reported as evidence for language supervision largely survives when captions are shuffled across samples or replaced by random targets. The two controls breakdown are cheap and can apply to any auxiliary target setup, which makes the paper a useful methodological check for the community.

Critical weaknesses
  1. The paper's writing need to be improved and currently it is hard to follow for readers who are not familiar with radar place recognition. The task, the datasets and the training targets are only briefly introduced before the results. The main datasets are not described, and the main paper use terms like traversal and keyframe without explanation. the task setup is one sentence, and the metric is given only as a formula without much explanation. The main claims of the paper is about language supervised representation learning in general, I would expect the reader won't be familiar with the task and need to be able to follow. I would strongly suggest the authors to polish the writing.

  2. The core idea of the paper (the decomposition) is not new, but the evidence provided from the paper is too narrow to change the how we understand language supervision in representation learning. The controls behind the decomposition, shuffled targets and random targets are well established, so the paper's value depends on showing that they change conclusion in practice. The study, however, covers a single task, two datasets, one model backbone (ResNet18), a small sentence encoder and a single auxiliary loss weight. The main captions are also computed from the radar scan itself and take only nine values. As a researcher or practitioner in representation learning, I am not sure what to take away beyond this specific setup. I think a larger setup for tasks, training parameters, backbone models etc. will be needed to show significance of the conclusion.

Questions for authors
  1. The training of the model is always with the fixed auxiliary weight (lambda = 0.1). Have you done experiments with different parameters setup and see how conclusion changes? These model training will be sensitive to training hyper parameters, would like to see how it change things.

  2. I am curious whether the protocol can detect a real language effect at this sample size. Do you have results for a positive control in the same pipeline, for example captions built from GPS place clusters, which carry correct place information by construction? I

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.