- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether commonly used collapse diagnostics, including effective rank, alignment, uniformity, and Wasserstein-based uniformity, which can establish that a learned representation captures task-relevant information rather than merely avoiding representational collapse. The authors construct a cross-lingual word-alignment testbed in which orthographic overlap provides a potential shortcut. By applying a substitution cipher to one side of translation pairs, they remove shared spelling while keeping the alignment supervision fixed. They compare contrastive learning, VICReg, and a SimSiam-style objective, with additional experiments on pretrained encoders and sentence-level alignment.
Critical strengths
- I think the motivation of this paper is solid. The paper identifies a genuine conceptual limitation in representation learning: a representation may be well distributed across dimensions while encoding nuisance features instead of the desired semantics. This distinction is important because effective rank and related metrics are frequently used not only for monitoring collapse but also for selecting hyperparameters.
- I like the some part of the experiment design: The substitution-cipher experiment is a compelling aspect of the methodology. Because the cipher preserves word identity, frequency, and co-occurrence while disrupting shared characters, it offers a useful way to test whether alignment depends on orthographic overlap. And the experiments go beyond effective rank by considering alignment, uniformity, and a Wasserstein-based measure. The comparison among contrastive learning, VICReg, and SimSiam also illustrates that apparently similar representational statistics can accompany substantially different retrieval behavior.
Critical weaknesses
-
I think there are some misclaim in the paper: the central conclusion that collapse diagnostics certify representational spread rather than task-relevant content is intuitively persuasive. However, the empirical evidence is concentrated on cross-lingual word alignment, where orthographic overlap is a particularly identifiable shortcut. This setting does not establish that the same failure occurs across representation-learning tasks more generally. In particular, the paper's own zero-shot natural-language-inference experiments report that the tested objectives transfer similarly, including rank-selected VICReg, despite their differences in bilingual lexicon induction.
-
Evaluation scope and experimental scale limit the strength of the evidence: The main testbed uses a 16-million-parameter byte encoder, and the reported seed counts are relatively small: three seeds for the principal scripture experiments and two seeds for several pretrained-encoder experiments. The paper does include TED2020, ByT5, XLM-R, and Tatoeba experiments, but the evidence is uneven across settings, and several conclusions rely on small seed counts or particular training budgets.
Questions for authors
see weakness part
We thank the reviewer for the review of our work and assessment of the generality and experimental coverage. We have added downstream classification, fresh-seed replication, and longer-budget experiments.
Point 1. The evidence does not establish generality beyond retrieval, and XNLI shows similar transfer.
Our added sentence-level XNLI experiment extends the evidence beyond retrieval to downstream cross-lingual classification. Across teacher-momentum settings, ciphered XNLI accuracy differs consistently despite high effective rank and similar uniformity on the sentence-retrieval probe.
The reviewer is correct that our original word-vector NLI experiment shows similar transfer across the tested objectives despite their different retrieval performance. That finding demonstrates that retrieval loss need not imply an absence of useful semantic information. The new experiment uses pretrained sentence representations and a different classifier, and shows that downstream classification can nevertheless differ across checkpoints with similar retrieval-probe diagnostics. We revised the abstract, contributions, Discussion, and Conclusion to distinguish these complementary findings and renamed the Discussion paragraph “Retrieval and downstream transfer.”
We evaluate all 24 checkpoints from the controlled ByT5 teacher-momentum experiment. We train an English NLI classifier over frozen sentence representations and evaluate cross-lingual XNLI transfer, selecting classifier checkpoints using English development accuracy. Each encoder checkpoint is evaluated with three classifier seeds, whose results are averaged before comparing the three encoder-training seeds.
The cipher introduces an input distribution shift relative to pretraining for all momentum settings. Our primary comparison pairs settings within the cipher condition and estimates the effect of target updating under that shift. The table reports mean XNLI accuracy on German, Spanish, and French.
The frozen-teacher advantage under the cipher is 3.87 percentage points, with per-encoder-seed differences of +5.30, +2.91, and +3.39. The corresponding plain-condition endpoint differences are below 0.5 points in every seed.
Point 2. Model scale, small seed counts, and uneven training budgets limit the evidence.
The added experiments strengthen replication and assess budget sensitivity within the evaluated model scales. The small custom encoder enables controlled comparisons across objectives, configurations, and training conditions, complemented by evaluations on standard pretrained ByT5 and XLM-R encoders. The additions comprise 213 training runs and continuations across selection replication, budget extensions, and controlled teacher experiments.
The new standard-encoder comparisons use three encoder-training seeds, while some existing pretrained-encoder comparisons in the appendix remain at two seeds. We now report seed counts and budgets explicitly for each table and distinguish the newly replicated results from those earlier comparisons. The additions strengthen replication and budget coverage within the evaluated architectures.
We appreciate your constructive comments and hope that our responses and additions address your concerns.