- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3
Read review
Paper summary
The paper shows that language models trained on identical content across different pseudo-languages (created by duplicating English data), each language having its own disjoint vocabulary (no shared tokens between languages), fail to learn shared representations of underlying “concepts”. As a result, these models suffer from learning slowdown and model capacity cost compared to an optimal setting in which each concept is learned jointly across all pseudo-languages.
Critical strengths
-
The controlled experimental design allows for easily interpretable results (unlike standard multilingual training, where the underlying data mapping is only partially known).
-
In my opinion, the most interesting findings come from the experiments with paired data, which show that even a perfectly learned mapping from vocabulary A to vocabulary B does not necessarily mean shared representations.
Critical weaknesses
-
The implicit hypotheses are not clearly motivated in the paper, leaving it to the reader to understand why such model behavior should be expected. Given that the paper does not use the full nine-page limit, there would be room for further justification for the underlying hypotheses.
-
The paper often describes experimental findings but does not investigate them further, leaving some findings shallow. Therefore, the paper has somewhat limited novelty. For example, the paired data experiments show interesting behavior but do not study or discuss it further. E.g. may such behavior be caused by models with c < 8 having enough capacity to learn separate representations for each compartment, while increasing the number of compartments may create competition for limited model capacity, pushing the model to learn shared representations.
Questions for authors
None
2 comments
I want to thank the authors for the response. These changes make the paper easier to follow so I have updated my presentation score accordingly. However, in my opinion the paper is still mostly descriptive, leaving it unclear how the findings will generalize to other settings, what are their implications to downstream behavior, and most importantly, what are the root causes of compartmentalization and how to overcome these limitations. Therefore, I will keep my original rating.
We thank the reviewer for suggestions that have helped materially improve our work, in particular for their interest in making hypotheses clearer.
We will denote new results since their initial review with "★", and rewritings with "Δ".
Weakness 1: implicit hypotheses
Δ We have provided further justification within the body of the paper, filling the original page limit. In particular, we stated hypotheses more explicitly:
Weakness 2: findings not investigated further (capacity explanation)
We view our results as consistent with a capacity-based hypothesis. In particular, weight decay creates capacity pressure, and lowers the transition threshold we observe from c=8 to c=6 at tr=0.75. ★ As an additional result, we analyze alignment of the same token embedding across multiple vocabularies, and observe that it rises at tr=0.75 for every c∈{4,6,8}, but stays near 0 for c=2. We suspect that there may be insufficient capacity pressure to discourage a redundant solution for c=2, although we note that we have been unable to induce sharing at c=2 for every variation of weight decay, learning rate, and translation ratio we have tried.
We believe the explicitly stated hypotheses address your first weakness, and the capacity-cost experiment, the learning-rate and weight-decay results, and the embedding-alignment analysis address your second. We welcome your insight on whether these analyses address your question about capacity. If this response doesV address your concerns, we would be grateful if you would consider raising your score.