- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
WHEN LANGUAGE GAINS ARE NOT SEMANTIC: RETHINKING ATTRIBUTION IN LANGUAGE-SUPERVISED REPRESENTATION LEARNING investigates whether performance gains from auxiliary language supervision come from the language alignment itself, or from other, typically uncontrolled, variables. For this, the authors analyze the impact of supervision with deranged captions and random instance targets, whose results are not statistically distinguishable from training with ordinary correct captions.
Critical strengths
S1 - The original research question is valid and often underexplored. The argument that adding language supervision introduces additional modifications beyond the semantic content of the captions warrants the need to investigate the true benefit of language alone.
S2 - The decomposition into intervention / correspondence / structure is a clean way to isolate the benefits of the language content itself. Furthermore, the control design is sensible, with the use of shuffled captions (with complete derangement) and random targets being consistent with the experimental setup proposed.
Critical weaknesses
W1 - A lot of important information is either omitted, under-explained, or deferred only to the appendix.
-
The task of radar place recognition is only briefly defined mathematically, without much contextual explanation, making it harder for readers from different areas to follow the paper. The Boreas dataset is mentioned several times, but is never cited or properly introduced. Important details of radar preprocessing are also only given in the appendix. All of these make it harder for new readers to situate themselves on the task and context.
-
L_place is defined only as a 'batch-hard triplet', without a clear description, explanation, citation or formulation. Hyperparameters and batch construction are identified in the appendix, which is not an issue, but the main text gives next to no information on the main training objective used. The functional form of L_aux is never actually defined; in Section 5.1 there's mention of InfoNCE, which I assume is the objective used, but that needs to be defined and made explicit earlier in the text. Furthermore, the projection head g is never described. These make it harder to understand the method being used to train the networks.
-
Although the paper describes the general attributes used to construct captions, no concrete examples or detailed construction rules are given. The vocabularies are also never fully specified. For instance, what are the '27 meanings', 'all eight axes' and 'near + far' descriptors mentioned in Table 1? Without these, it's hard to properly interpret the obtained results.
W2 - Evidence doesn't fully support the claims, and the paper would benefit from a larger analysis of its results.
-
Confidence intervals are extremely wide for the small gains observed. A tighter confidence interval (with more seeds, more evaluation routes, etc.) would make the results more meaningful.
-
The batch size of 16 is very small for InfoNCE setups, which typically benefit from larger batch sizes. There are also no ablation studies to look at the effects of batch size and other hyperparameters (such as the small margin size of 0.2 used in the triplet objective).
-
The intervention does not produce a statistically significant gain on MulRan, unlike on Boreas. Consequently, the replication does not reproduce the more interesting finding that significant auxiliary gains can persist without evidence of a benefit from correct language correspondence.
-
The paper names several possible mechanisms that could lead to performance gains apart from language itself, but none of them are tested. Furthermore, an analysis of the model's latent space would help interpret the results. The triplet loss, for instance, could alone have left embeddings concentrated in a small latent region, and the auxiliary InfoNCE loss, which considers multiple negatives and uses mismatched or random targets as assigned positives in the control arms, could have served as a regularizer to spread representations a bit more, thus facilitating retrieval. This, though, cannot be concluded from the given evidence alone.
Questions for authors
Q1 - What does a caption look like? Please provide examples and the full vocabulary.
Q2 - How does the latent space differ between radar only and the different auxiliary arms?
Q3 - Does a regularizer-only baseline reproduce the gains obtained from L_aux?
Q4 - How does this behave under different hyperparameters and training times? Are the same behaviors and gains from L_aux maintained?