acceptodds
Under review as a conference paper at ICLR 2027

What Do Collapse Diagnostics Certify? Testing Alignment Objectives for Orthographic Shortcuts

Abstract

Objectives that align representations without negatives are judged by label-free diagnostics of collapse, effective rank and uniformity, which also choose their hyperparameters. We ask what these diagnostics certify about what a representation has learned. In cross-lingual alignment from parallel text a model can match related languages by shared spelling rather than meaning; a substitution cipher on one side of every such pair removes the shared letters with the supervision held byte-identical, and the fraction of a model's score kept under the cipher is how much of its alignment was not spelling. The diagnostics certify spread, not content. (1) At the weights their effective rank selects, the two anti-collapse devices, VICReg and SimSiam, reach contrast's rank or beyond and keep 0.16 and 0.29 of their score against contrast's 0.65. (2) Tuning brings both within reach of contrast while the rank stays flat, and the spectrum every diagnostic reads does not move under the cipher. (3) Selecting by these diagnostics favours a shortcut solution, and on a standard pretrained encoder a run that learned nothing. The objectives that learn the lexicon share a target that cannot drift toward the shortcut, the batch in contrast or a frozen teacher in distillation: attraction to a frozen teacher learns the lexicon as contrast does, without negatives. Retrieval of the training pairs, which needs no labels, tracks held-out accuracy at ρ = 0.96 where rank selects the shortcut. Anti-collapse devices should be selected by a task-shaped criterion and tested with the shortcut removed, not certified by spread alone.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, non-contrastive learning, contrastive learning, shortcut learning, representation collapse, effective rank, uniformity, model selection, knowledge distillation, cross-lingual alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ggi9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether commonly used collapse diagnostics, including effective rank, alignment, uniformity, and Wasserstein-based uniformity, which can establish that a learned representation captures task-relevant information rather than merely avoiding representational collapse. The authors construct a cross-lingual word-alignment testbed in which orthographic overlap provides a potential shortcut. By applying a substitution cipher to one side of translation pairs, they remove shared spelling while keeping the alignment supervision fixed. They compare contrastive learning, VICReg, and a SimSiam-style objective, with additional experiments on pretrained encoders and sentence-level alignment.

Critical strengths
  1. I think the motivation of this paper is solid. The paper identifies a genuine conceptual limitation in representation learning: a representation may be well distributed across dimensions while encoding nuisance features instead of the desired semantics. This distinction is important because effective rank and related metrics are frequently used not only for monitoring collapse but also for selecting hyperparameters.
  2. I like the some part of the experiment design: The substitution-cipher experiment is a compelling aspect of the methodology. Because the cipher preserves word identity, frequency, and co-occurrence while disrupting shared characters, it offers a useful way to test whether alignment depends on orthographic overlap. And the experiments go beyond effective rank by considering alignment, uniformity, and a Wasserstein-based measure. The comparison among contrastive learning, VICReg, and SimSiam also illustrates that apparently similar representational statistics can accompany substantially different retrieval behavior.
Critical weaknesses
  1. I think there are some misclaim in the paper: the central conclusion that collapse diagnostics certify representational spread rather than task-relevant content is intuitively persuasive. However, the empirical evidence is concentrated on cross-lingual word alignment, where orthographic overlap is a particularly identifiable shortcut. This setting does not establish that the same failure occurs across representation-learning tasks more generally. In particular, the paper's own zero-shot natural-language-inference experiments report that the tested objectives transfer similarly, including rank-selected VICReg, despite their differences in bilingual lexicon induction.

  2. Evaluation scope and experimental scale limit the strength of the evidence: The main testbed uses a 16-million-parameter byte encoder, and the reported seed counts are relatively small: three seeds for the principal scripture experiments and two seeds for several pretrained-encoder experiments. The paper does include TED2020, ByT5, XLM-R, and Tatoeba experiments, but the evidence is uneven across settings, and several conclusions rely on small seed counts or particular training budgets.

Questions for authors

see weakness part

1 comment

We thank the reviewer for the review of our work and assessment of the generality and experimental coverage. We have added downstream classification, fresh-seed replication, and longer-budget experiments.

Point 1. The evidence does not establish generality beyond retrieval, and XNLI shows similar transfer.

Our added sentence-level XNLI experiment extends the evidence beyond retrieval to downstream cross-lingual classification. Across teacher-momentum settings, ciphered XNLI accuracy differs consistently despite high effective rank and similar uniformity on the sentence-retrieval probe.

The reviewer is correct that our original word-vector NLI experiment shows similar transfer across the tested objectives despite their different retrieval performance. That finding demonstrates that retrieval loss need not imply an absence of useful semantic information. The new experiment uses pretrained sentence representations and a different classifier, and shows that downstream classification can nevertheless differ across checkpoints with similar retrieval-probe diagnostics. We revised the abstract, contributions, Discussion, and Conclusion to distinguish these complementary findings and renamed the Discussion paragraph “Retrieval and downstream transfer.”

We evaluate all 24 checkpoints from the controlled ByT5 teacher-momentum experiment. We train an English NLI classifier over frozen sentence representations and evaluate cross-lingual XNLI transfer, selecting classifier checkpoints using English development accuracy. Each encoder checkpoint is evaluated with three classifier seeds, whose results are averaged before comparing the three encoder-training seeds.

Teacher momentumPlain XNLICiphered XNLI
1.0 — frozen56.13%45.55%
0.99955.79%44.93%
0.9956.07%43.28%
0.956.18%41.69%

The cipher introduces an input distribution shift relative to pretraining for all momentum settings. Our primary comparison pairs settings within the cipher condition and estimates the effect of target updating under that shift. The table reports mean XNLI accuracy on German, Spanish, and French.

The frozen-teacher advantage under the cipher is 3.87 percentage points, with per-encoder-seed differences of +5.30, +2.91, and +3.39. The corresponding plain-condition endpoint differences are below 0.5 points in every seed.

Point 2. Model scale, small seed counts, and uneven training budgets limit the evidence.

The added experiments strengthen replication and assess budget sensitivity within the evaluated model scales. The small custom encoder enables controlled comparisons across objectives, configurations, and training conditions, complemented by evaluations on standard pretrained ByT5 and XLM-R encoders. The additions comprise 213 training runs and continuations across selection replication, budget extensions, and controlled teacher experiments.

  • To assess selection stability, the full grid was replicated at 3 fresh seeds (123 runs). Training-link retrieval has mean held-out BLI regret of 0.41 points versus 4.36 for effective rank, with lower regret in every seed. This replicates the submitted selection advantage on the previously explored grid.
  • To assess budget sensitivity, 30 continuations extend the principal configurations from 8,000 to 24,000 updates. Mean kept fractions change from 0.65 to 0.66 for contrast, 0.29 to 0.36 for rank-selected SimSiam, and 0.16 to 0.24 for rank-selected VICReg. At both budgets, every evaluated seed of the rank-selected configurations has a kept fraction at most 0.37, while every contrast seed is at least 0.58. The mean absolute SimSiam gaps do not narrow.
  • For standard-encoder evidence, the teacher-momentum and checkpoint-branching experiments comprise 60 runs, with 3 encoder seeds per comparison. The downstream XNLI extension evaluates all 24 teacher-momentum checkpoints and reports classifier-seed variability separately.

The new standard-encoder comparisons use three encoder-training seeds, while some existing pretrained-encoder comparisons in the appendix remain at two seeds. We now report seed counts and budgets explicitly for each table and distinguish the newly replicated results from those earlier comparisons. The additions strengthen replication and budget coverage within the evaluated architectures.

We appreciate your constructive comments and hope that our responses and additions address your concerns.

Reviewer_LFmiOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether commonly used collapse diagnostics, such as effective rank and uniformity, can reliably assess what alignment objectives have learned. Using cross-lingual word alignment as a testbed, the authors introduce a cipher-based intervention that removes shared spelling between languages while preserving translation supervision. Experiments with contrastive learning, VICReg, and SimSiam show that representations with healthy collapse diagnostics may still rely heavily on orthographic shortcuts. The paper further explores training-link retrieval as an alternative model selection criterion and examines the role of fixed targets in learning cross-lingual representations.

Critical strengths
  1. The paper presents a well-designed experimental analysis. The cipher-based intervention provides a useful way to examine reliance on orthographic shortcuts while keeping translation supervision unchanged. The comparisons across different alignment objectives, hyperparameter settings, and encoders make the empirical observations reasonably convincing.

  2. The paper highlights a meaningful limitation of existing representation diagnostics. The results show that effective rank and uniformity can appear healthy even when the learned representations rely heavily on shortcut features. This is a useful reminder that avoiding representation collapse does not necessarily imply learning task-relevant representations.

Critical weaknesses
  1. The generality of the findings remains somewhat limited. The analysis primarily focuses on cross-lingual alignment and nearest-neighbor retrieval, where orthographic similarity provides a natural shortcut. Interestingly, the paper reports that the same differences do not appear in its zero-shot NLI evaluation (Appendix C.3). This raises the question of whether the observed limitations of collapse diagnostics are specific to retrieval-oriented tasks or have broader implications for representation learning. A clearer discussion of this distinction would strengthen the paper.

  2. The explanation for why certain alignment objectives avoid shortcuts is not fully convincing. Section 7 suggests that objectives with a target that cannot drift toward the shortcut are more likely to learn the underlying lexicon. However, the evidence does not clearly isolate this mechanism from other differences between the objectives. The paper also reports that some SimSiam configurations can achieve high robustness under the cipher, while frozen-teacher distillation is less effective on XLM-R. These observations suggest that the relationship between target stability and shortcut learning may be more nuanced than the current explanation.

Questions for authors
  1. How can we distinguish the loss of orthographic shortcuts from the difficulty of adapting to altered input distributions, particularly for pretrained encoders? Could the observed performance degradation partly reflect differences in how the alignment objectives handle this distribution shift?

  2. Section 7 suggests that a target that cannot drift toward the shortcut is important for learning the underlying lexicon. However, some SimSiam configurations also achieve high robustness under the cipher. How should we reconcile these observations? What evidence supports target stability as the key factor rather than other differences between the objectives?

2 comments

We thank the reviewer for their review of our work and for their clarifying questions. Their questions prompted new controlled experiments that provide evidence of an effect of teacher updating on both retrieval and downstream classification. We used these results to sharpen the scope of the diagnostic claim and the target-stability explanation.

W1. Generality and the NLI results

We conducted a new controlled XNLI experiment which extends our evidence beyond nearest-neighbor retrieval to downstream cross-lingual classification. Under the cipher, teacher updating produces consistent classification differences despite high effective rank and similar uniformity on the sentence-retrieval probe. The original word-vector NLI result shows that useful semantic transfer can persist despite substantial retrieval loss. We revised the abstract and Discussion to reflect both findings and their respective settings.

We evaluate XNLI on all 24 checkpoints from the controlled ByT5 teacher-momentum experiment.

Teacher momentumPlain XNLICiphered XNLI
1.0 — frozen56.13%45.55%
0.99955.79%44.93%
0.9956.07%43.28%
0.956.18%41.69%

These are mean accuracies on German, Spanish, and French—the Latin-script languages. Ciphered accuracy decreases monotonically as momentum decreases in every encoder seed. The endpoint advantage for the frozen teacher averages 3.87 percentage points across three encoder seeds, each averaging three classifier seeds. Classifier checkpoints are selected using English development accuracy. The same checkpoints retain high effective rank and similar uniformity on the sentence-retrieval probe despite these downstream differences.

This extension was chosen after examining retrieval, with its evaluation protocol fixed before XNLI scoring. It uses pretrained ByT5 sentence representations and a different classifier from the original word-vector NLI experiment, so the two findings are compatible. We reported both findings with their respective settings. Together they support task-specific evaluation of cross-lingual representations because healthy collapse diagnostics alone do not establish retrieval quality, and similar readings on a retrieval probe can coexist with downstream classification differences. The frequency and mechanisms of these failures across other domains remain open.

W2/Q2. What supports target stability?

We acknowledge that our original explanation needs qualification. A fixed target is not necessary for robust SimSiam and in the paired comparison, the robust and fragile moving-target configurations differ only in the predictor learning-rate multiplier, which sets how quickly the prediction head learns relative to the encoder. The faster predictor retains 0.92–0.94 under the cipher, compared with 0.58–0.68 for the slower predictor.

We therefore test teacher updating while controlling this recipe difference by starting every arm from the same less-robust student and the same informative teacher from the robust run of that seed and condition. Predictor, loss, batches, and schedule are shared. Only the speed at which the teacher follows the student varies, with lower momentum producing faster updating.

The new comparison comprises 24 runs across 3 encoder seeds on ByT5. Retrieval columns below use the Latin-script languages, and all entries are means across encoder seeds.

Teacher momentumPlain retrieval (%)Ciphered retrieval (%)Ciphered training-pair retrieval (%)Ciphered effective rank
1.0 — frozen93.8587.1292.22729
0.99994.3281.8388.08750
0.9994.4068.8976.59746
0.994.0559.6066.62739
Original SimSiam with its own moving target (reference)94.0158.3066.06735

Ciphered retrieval decreases at every momentum step in every encoder seed, while plain retrieval stays similar and rank remains high. Training-pair retrieval does reflect the loss. The fastest-updating arm approaches the original SimSiam reference with its own moving target. This establishes an effect of updating an initially informative teacher under the cipher.

Freezing alone is not sufficient either, since freezing weaker intermediate targets can hurt performance. The original appendix momentum sweep remains a negative result because its contrast-initialized distillation setup did not produce the proposed failure without visible collapse.

We revised the explanation to state that preserving an informative target helps in this setting, whereas freezing alone is not a general prescription. The experiment isolates teacher updating and the weaker XLM-R distillation results remain a boundary of the evidence.

Q1. Shortcut removal versus adaptation difficulty

The main results use a custom byte encoder trained from random initialization, so they do not depend on a mismatch with a pretraining distribution. The query-level analysis further shows that most of the loss occurs where shared spelling was available.

The supervised Procrustes control retains its Latin-script BLI score of 21.78% under the cipher. This demonstrates that the task remains solvable under the preserved supervision.

We partition held-out queries according to whether any gold translation shares a character bigram with the query on the original strings, then follow the same queries under ciphering. The first three numeric columns report kept fractions at the original budget and the final column shows the change after continuing training. All results average the three original seeds.

ConfigurationKept fraction for all queriesKept fraction for overlap queriesKept fraction for zero-overlap queriesKept fraction for all queries, 8k→24k
Contrast0.650.530.900.65→0.66
Rank-selected VICReg0.160.080.770.16→0.24
Tuned VICReg0.560.400.900.56→0.61
Rank-selected SimSiam0.290.120.880.29→0.36
Tuned SimSiam0.340.230.670.34→0.41

The exploratory overlap split contains 36% of held-out queries but accounts for 83–103% of net retrieval loss across configurations and seeds. Values above 100% occur when zero-overlap queries improve. This locates most of the loss on queries where shared spelling was available.

At the original checkpoints, plain and ciphered retrieval on Latin-script training links with no shared character bigram differs by at most 2.85 points at every seed for 4 of the 5 configurations. For tuned SimSiam, the gap is larger and also loses training fit on untouched languages (29%→22% averaged across seeds), indicating a broader fitting difficulty.

The ordering of kept fractions is unchanged after tripling the budget. For both SimSiam configurations, the mean absolute plain–ciphered gap does not narrow because plain performance also improves. The overlap split was defined retrospectively.

For pretrained encoders, the ByT5 word-level comparison shows initialization-dependent sensitivity. Tuned VICReg retains 0.74 from random initialization and 0.44 from pretrained weights, while contrast retains 0.77 and 0.69, respectively. These comparisons are consistent with adaptation effects that differ between objectives. The controlled teacher-momentum experiment holds the input shift and initial student and teacher fixed across arms, isolating an effect of teacher updating under that shift.

The cipher preserves translation supervision, type identity (up to a bijection), frequencies, and co-occurrence structure. The kept fraction measures retrieval retained under this intervention. We stated these invariants and the interpretation together.

We appreciate your constructive comments and hope that our responses and additions address your concerns.

Reviewer_5uGLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether commonly used collapse diagnostics, including effective rank, uniformity, and alignment, reliably indicate what cross-lingual representations have learned. The authors compare contrastive learning, VICReg, and SimSiam-style objectives using a controlled byte-level testbed. To investigate dependence on orthographic overlap, they train matched configurations on either the original parallel corpus or a version with a fixed substitution cipher applied to the non-English side of Latin-script language pairs. The intervention preserves word identities up to a bijection, frequencies, co-occurrences, and supervision links. The main finding is that configurations with apparently healthy collapse diagnostics can perform poorly on bilingual lexicon induction and lose substantial retrieval accuracy under the cipher, while their representation spectra remain similar. The study extends the comparison to a second corpus, external dictionaries, pretrained ByT5 and XLM-R encoders, sentence retrieval, and frozen-teacher distillation. The authors also investigate model selection, finding that retrieval of training links predicts held-out retrieval performance better than effective rank within the evaluated configurations. Overall, the paper argues for task-specific evaluation and shortcut interventions alongside collapse diagnostics.

Critical strengths
  1. The intervention provides a useful and interpretable test of orthographic dependence. Applying a bijective substitution while retaining the same supervision links is a strong experimental design for studying sensitivity to shared spelling. The controls make the intervention particularly informative: on scripture, the supervised Procrustes baseline retains its Latin-script score of 21.78%, while the character-similarity baseline falls from 9.17% to 0.09%. Meanwhile, the rank-selected VICReg and SimSiam configurations retain only 0.16 and 0.29 of their Latin-script retrieval scores, compared with 0.65 for contrast, despite reaching effective ranks at or above contrast’s. These results clearly demonstrate that avoiding dimensional collapse is insufficient to guarantee successful nearest-neighbour alignment.

  2. The paper triangulates its central observation through extensive additional experiments and offers a practical selection alternative. The analysis includes two corpora, external MUSE dictionaries, multiple seeds for the main comparisons, longer training budgets, removal of the sentence-level contrastive term, and experiments with standard encoders and published projection heads. These checks help establish that the basic diagnostic failure is not confined to one custom architecture or one automatically induced evaluation lexicon. The training-link retrieval criterion also gives the study practical value: across the 41 testbed configurations, it correlates with held-out BLI at ρ=0.96\rho=0.96, compared with 0.84 for effective rank, and incurs substantially lower selection regret. The explicit reporting of counterexamples and limitations further improves the paper’s usefulness.

Critical weaknesses
  1. The proposed spectral explanation incorrectly groups together diagnostics with different mathematical dependencies. Section 8 states that effective rank, uniformity, Wasserstein uniformity, and alignment are functions of the representation spectrum; similar claims appear around Figure 5. However, under Equation 4, alignment depends on the identities of the paired examples: changing the target assignments can change alignment without changing either marginal spectrum. Standard uniformity depends on the full distribution of pairwise distances and is not generally determined by the covariance spectrum either. This distinction is consistent with the original definitions of alignment and uniformity in Wang and Isola (2020). Consequently, an unchanged singular-value spectrum does not establish that all these diagnostics must remain unchanged. This matters because spectral blindness is presented as a unifying explanation for the findings. The paper should distinguish empirical failures of particular diagnostics from conclusions that follow mathematically for spectrum-only measures.

  2. The interpretation of the kept fraction as separating spelling from learned meaning is stronger than the evidence establishes. The ratio compares independently trained models, so it measures sensitivity to the intervention, including changes in optimization difficulty and convergence, rather than directly decomposing an individual model’s knowledge. The appendix provides evidence that these effects matter: ciphering the Latin-script languages can substantially reduce SimSiam’s performance on untouched languages, and longer training considerably improves retention for several configurations. Furthermore, Appendix C.3 shows that the rank-selected VICReg configuration retains 0.94 of its above-chance XNLI margin despite retaining only 0.16 of its BLI score. Thus, a large nearest-neighbour retrieval loss does not establish a corresponding absence of transferable semantic information. Although Section 8 acknowledges these limitations, the abstract’s spelling-versus-meaning framing and statements that some configurations learned cognates alone remain too categorical. This affects how broadly the central empirical result can be interpreted.

Questions for authors
  1. Which conclusions actually follow from spectral invariance? Could you clarify the claims in Section 8 and Figure 5 that alignment and standard uniformity are functions of the representation spectrum? Under Equation 4, these quantities appear to require information beyond the singular values. Which parts of the argument would remain empirical observations after separating spectrum-only diagnostics from pair-dependent and distribution-dependent diagnostics?

  2. How much of the cipher-induced loss reflects optimization difficulty? Do the principal differences in kept fraction persist when plain and ciphered runs reach comparable training-link retrieval or demonstrable convergence? Appendix F.6 shows substantial recovery with additional training, while Appendix E.1 reports large losses on untouched languages. Evidence separating these effects would affect my confidence in interpreting the ratio specifically as orthographic dependence.

  3. What supports the fixed-target explanation beyond comparisons across different recipes? Appendix F.5 reports that the controlled target-momentum sweep does not produce the predicted regime of silent shortcut learning: moving targets either preserve retrieval or collapse visibly. Table 16 also identifies a SimSiam configuration retaining 0.96 without a frozen teacher. How should these findings qualify the claim that successful objectives share a target that cannot drift toward the shortcut? Is this intended as a descriptive hypothesis or an experimentally established mechanism?

  4. How stable is the proposed model-selection advantage under independent evaluation? Table 10 evaluates the 41 configurations at a single seed, and Appendix D.1 constructs the development/test split after the initial whole-test selection. Are there results confirming the training-link criterion’s selection advantage across independent training seeds or a prospectively defined evaluation split? Such evidence would strengthen the practical recommendation and help distinguish a reliable selection rule from a strong correlation within the explored configuration set.

2 comments

We thank the reviewer for the assessment of our work and for their clarifying questions. Their questions prompted new experiments that strengthened the empirical findings on diagnostic failure and model selection while clarifying the scope of the cipher intervention and the target-stability explanation.

Q1. Which conclusions actually follow from spectral invariance?

We acknowledge that standard uniformity and gold-pair alignment are not spectrum-only measures. The central empirical result remains. For rank-selected VICReg, effective rank changes from 45.61 to 45.58 while Latin-script BLI falls from 7.93% to 1.23%; for rank-selected SimSiam, from 41.35 to 40.73 while BLI falls from 7.04% to 2.06%.

Mathematically, identical normalized singular values imply identical effective rank, so rank cannot distinguish representations that share a spectrum but differ in retrieval; the near-stable rank above is the empirical observation. Section 3 already defines standard uniformity through pairwise distances, alignment through the gold pairs, and Gaussian Wasserstein uniformity through covariance eigenvalues under the stated normalization conventions.

As a separate, broader limitation, we constructed two representation configurations with identical spectra and values for all four diagnostics but retrieval accuracy of 50% and 0%, so these aggregate values do not uniquely determine retrieval.

Under ciphering, standard uniformity improves at every seed for tuned VICReg even as retrieval falls, and worsens for the other four configurations. Gold-pair alignment (held-out translation pairs, not a selection criterion) worsens at every seed for four configurations and is mixed for rank-selected SimSiam, while training-link alignment worsens for all five. These observations support diagnostic-specific conclusions, not universal invariance.

The model-selection result (Q4) is independent of the spectral explanation because it evaluates diagnostics on plain-trained models, with no ciphered counterpart.

We added a plain-versus-ciphered table covering all four diagnostics for the five audited configurations and all three seeds, and corrected the spectral grouping wherever it appears (abstract, results, Discussion, appendix, and Figure 5 caption).

Q2. How much of the cipher-induced loss reflects optimization difficulty?

After tripling the training budget, the ordering of kept fractions is unchanged and the rank-selected configurations still retain substantially less retrieval than contrast. The training-fit comparison shows where fit is comparable and where an optimization explanation remains possible.

We continued all 30 original plain and ciphered checkpoints of the randomly initialized byte encoder from 8k to 24k updates under the original learning-rate schedule (now Section 5.3; Table 22, Appendix G.2).

Results use the same 2,300 frozen held-out Latin-script queries on the paper’s strict evaluation support. Scores are means across three seeds; kept fraction is the mean of the per-seed ciphered/plain ratios.

ConfigurationPlain BLI (%), 8k→24kCiphered BLI (%), 8k→24kKept fraction, 8k→24k
Contrast14.29→14.299.28→9.430.65→0.66
Rank-selected VICReg7.93→8.291.23→1.970.16→0.24
Tuned VICReg11.35→10.976.30→6.640.56→0.61
Rank-selected SimSiam7.04→8.972.06→3.220.29→0.36
Tuned SimSiam12.71→14.234.36→5.770.34→0.41

Additional training improves ciphered SimSiam accuracy at every seed, but the mean absolute deficit does not narrow for either configuration because plain performance also improves. Tuned VICReg’s kept fraction rises partly because plain performance declines. In the ciphered SimSiam continuations, retrieval on overlap training links rises by 8.3–10.3 points, held-out overlap retrieval by 1.0–3.3.

At the original checkpoints, plain and ciphered retrieval on Latin-script training links with no shared character bigram differs by at most 2.85 points at every seed for 4 of the 5 configurations. Tuned SimSiam has a larger gap and also loses training fit on untouched languages (29%→22% averaged across seeds), indicating a broader fitting difficulty.

An exploratory analysis groups held-out queries by whether they share a character bigram with any correct translation before ciphering. Queries with this spelling overlap constitute 36% of the evaluation set but account for 83–103% of the net retrieval loss (above 100% when queries without overlap improve).

These results do not establish convergence or matched fit on the overlap links most affected by the intervention, since removing an easy spelling cue can itself make those links harder to learn.

We therefore interpret the kept fraction as retrieval retained under the intervention. The abstract now carries the Discussion’s point that substantial retrieval loss can coexist with useful semantic transfer, and categorical wording such as “learned the cognates alone” is replaced.

Q3. What supports the fixed-target explanation beyond comparisons across recipes?

The submitted fixed-target explanation was a descriptive hypothesis based on comparisons across recipes. Our new controlled experiment establishes that preserving an informative teacher improves cipher robustness in this setting. Within each condition and seed, all arms share the same initial student and informative teacher, predictor, loss, batch sequence, and schedule, with only teacher momentum varying. Lower momentum means the teacher follows the student more quickly.

The comparison comprises 24 runs across 3 encoder seeds on ByT5. Retrieval columns below use the Latin-script languages, and all entries are means across encoder seeds.

Teacher momentumPlain retrieval (%)Ciphered retrieval (%)Ciphered training-pair retrieval (%)Ciphered effective rank
1.0 — frozen93.8587.1292.22729
0.99994.3281.8388.08750
0.9994.4068.8976.59746
0.994.0559.6066.62739
Original SimSiam with its own moving target (reference)94.0158.3066.06735

Ciphered retrieval decreases at every momentum step in every encoder seed, while plain retrieval stays similar and effective rank remains high. Training-pair retrieval does track the loss. The fastest-updating arm approaches the original SimSiam reference.

A secondary sentence-level XNLI probe also shows consequences beyond retrieval. After averaging three classifier seeds per checkpoint, ciphered accuracy on the Latin-script languages decreases monotonically as momentum decreases within every encoder seed, with a 3.87-point average advantage for the frozen teacher over momentum 0.9. This extension was chosen after examining retrieval, with its evaluation protocol fixed before XNLI scoring.

A fixed target is not necessary, as moving-target SimSiam with the faster predictor retains 0.937, 0.921, and 0.936 across the three seeds, consistent with the robust configuration in Table 16. Freezing alone is also not sufficient, as our new 36-run checkpoint-branching experiment shows that freezing the student’s own intermediate target can substantially worsen ciphered retrieval. Appendix F.5 remains a negative result because its momentum sweep did not produce the proposed failure without visible collapse.

We revised Section 7 and the contribution statement to state that preserving an informative teacher helps in this setting, whereas freezing alone is not a general prescription. The controlled experiment establishes an effect of teacher updating under the cipher, but it does not uniquely identify drift toward spelling shortcuts as the mechanism.

Q4. How stable is the selection advantage under independent evaluation?

Training-link retrieval selects with lower held-out BLI regret than effective rank and standard uniformity in every fresh seed. We replicated the complete 41-configuration grid at three predefined fresh training seeds (123 runs; now Section 6.1, Table 23, Appendix G.3). The grid and selection rules were fixed before these runs. Each selector chooses within a seed without using test scores, and collapsed or poorly performing configurations remain in the primary candidate set.

The primary statistic is the paired difference in held-out BLI selection regret between effective rank and training-link retrieval. Regret is the best test score within that seed’s grid minus the selected configuration’s score.

CriterionSpearman ρ with strict held-out BLI, per seedMean regret on strict BLI (points)Mean regret on MUSE (points)Mean strict-BLI regret within VICReg (points)
Training-link retrieval0.97, 0.96, 0.940.410.640.55
Effective rank0.81, 0.82, 0.844.3610.653.51
Standard uniformity0.57, 0.59, 0.634.3610.653.51
Gaussian Wasserstein uniformity0.79, 0.78, 0.810.030.003.51

On strict BLI, the paired regret improvements are 2.78, 4.26, and 4.82 points, giving a mean of 3.95 points with a between-seed sample SD of 1.05. This replicates the submitted single-seed selection advantage over effective rank across fresh training seeds on the fixed, previously explored grid and evaluation sets, including MUSE.

The comparison depends on the criterion and the model family. Rank and standard uniformity select the same VICReg setting in every fresh seed, incurring mean pooled strict-BLI regret of 4.36 points. Within SimSiam, training-link retrieval and rank select the same configuration. Consistent with Section 6.1, Gaussian Wasserstein selection beats training-link retrieval in one seed and ties in two over the pooled grid. Within VICReg, however, it selects the same setting as rank in every seed, with mean regret of 3.51 points against 0.55 for training-link retrieval.

We appreciate your constructive comments and hope that our responses and additions address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.