- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies cross-lingual sharing in transformers using a toy single-layer transformer trained on multilingual modular addition task, where several disjoint token vocabularies (languages) all encode the same task. The authors find that standard-trained toy models transfer across languages and share circuits (the sets of weights and activation directions that carry out this computation), yet use largely unshared representations. They propose warm-up, an initial training phase with labels aggregated across all languages, which induces shared representations and significantly improves generalisation.
Critical strengths
- The experimental setting is clean and well-motivated. The task isolates cross-lingual transfer well and is generally easy to understand.
- The proposed warm-up algorithm is simple and empirically interpretable, showing promising results on the task.
Critical weaknesses
- The paper covers only a toy LLM scenario, but it would be interesting to see how this generalises to actual transofrmers (at least the circuit sharing part)
Questions for authors
General comment: The notation in the equations is a bit inconsistent. The paper introduces the symbol n_(ℓ) in the sentence after the equation, but defines it formally only a paragraph later, and the equation itself never uses n, which makes it a bit cnfusing when reading. Using the already present notation "a(l) and b(l) represent the numbers in language l" and then introducing the general notation ("which we generalise to n(l) in the following text") or moving the definition up would be much easier to follow
2 comments
Thank you for the response and addressing the notation confusion. I would be interested to see in how the ideas transfer to LLMs in future! I would like to keep my score and remain supportive of acceptance.
Thank you for your comments and for reading our paper.
We have modified the notation as you suggested to make Section 3 easier to follow.
(Whether our findings hold in LLMs was the most common question across reviews, so we've answered it once and included the same response for each reviewer who raised it.)
We're interested in a couple of ways to potentially extend the results to LLMs, which we now discuss in the paper in Section 8.
One is to estimate an analogue of the aggregated labels for LLMs to try to make training faster and yield more interpretable models, in the sense of encouraging similar representations for concepts that use similar circuits. We have several ideas for an analogue of warm-up for LLMs. Here is one such idea: at each token in the training corpus, have a cheap model decide whether a new word has just started. If so, pick a random real-world language ℓ, and give an existing LLM the full context of the current training sample with preamble instructions to predict what the next word would be if the next word was instead in language ℓ. Then, assign loss using the first token of the translated next word instead of the true next word. (You could also translate to several random languages and assign loss using the mean over the languages of the first token.) Do this procedure for some amount of epochs, and then swap to usual training as in our method.
As a first step toward this, we have tested a version of this in the toy model, where each warm-up label is the correct sum in one randomly chosen language, instead of the aggregated label over all languages. This is the kind of supervision a translation-based warm-up would provide in an LLM. We find that this random-language label regime performs nearly as well as warm-up (60/67 runs grok with random-language labels vs 62/67 runs in warm-up, see Table 5). Because random-language labels work nearly as well as aggregated labels, that would suggest a translation-based warm-up for LLMs is worth trying. It would also let us study questions that the toy model cannot answer, such as whether pretrained LLMs already share as much representation across languages as would help their minority languages. We are interested in pursuing this translation-based warm-up for LLMs in future work, as well as other ideas for extending warm-up to LLMs.
We are also interested in extending our mechanistic analysis to LLMs and developing methods to understand if LLMs can also share circuits without sharing representations: the distinction between shared circuits and shared representations in our toy models is a good demonstration that many common tools used for mechanistic interpretation (like cosine similarity for example), can only detect shared representations and not shared circuits. A cosine similarity of zero on an actual LLM, for example, would not be enough to conclude with certainty that circuits aren't shared.
Another idea is to try to extend the language ℓ subspace method used for the ablations in the toy models to LLMs to examine cross-lingual circuit sharing in larger models, possibly using MMMLU.