- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the author challenge the common practice of constructing the embedding matrix with fully learnable practice and instead propose to use a deterministic binary encoding. The author would like to study whether the transformer LM can still achieve a reasonable capacity with only deterministic binary encoding. To answer this question, they pretrain 1.6B scale models with normal learnable embedding and deterministic binary encoding respectively. The empirical results show that the while the model trained with binary encoding lags far behind the standard model, it still has some reasonable capacity.
Critical strengths
The paper is clearly written and the tone is rigorous.
Critical weaknesses
The main weakness is I don't get any insight after reading the paper. The research question below
We investigate whether this token-specific parameterization is required for substantial language-modeling capability, or whether a shared Transformer can learn from fixed token identities
is not very valuable because I think majority of the people in the comunity would guess with high confidence the answer is "a Transformer surely can learn from fixed binary embedding albeit reduced perforamnce". And this is exactly what the empirical results in the paper showed. This work doesn't bring new "information" or new "insight".
Questions for authors
My question is whether I missed some subtle insight from the paper. I really think the reserach question itself is more or less a "known" result.
3 comments
Quantitative consequences of the specified restriction
The revision now includes five configurations at an identical budget of 48,444,211,200 prediction targets, including Learned and canonical Binary16 controls.
This extends the result beyond the qualitative expectation that fixed inputs can learn. It measures two distinct responses to the restriction: fixed row normalization partially narrows the observed gap on selected tasks, while adding one standard Transformer block does not broadly recover it. For example, LAMBADA accuracy is 43.35% for Learned, 40.07% for Binary16, 39.72% for Ext, and 41.88% for Norm.
The scientific object is the specified intervention: minimal deterministic binary identity coordinates, a parameter-free lift, and no independently trainable input rows or dedicated input generator. Its value is in establishing the capability retained under that restriction and measuring the consequences of concrete alternatives.
A performance cost is not a collapse of capability. The original 100B fixed-code results, the matched-budget controls, and the smaller repeated runs jointly characterize a working model family rather than only a prediction that some learning should be possible. The revised discussion makes its experimental and academic value explicit.
Thanks for your in-depth clarification. Following your reply, I made another pass to the work and I think I have missed some points of the work in my initial assessment. In particular, the value of parameter-free embedding goes beyond beating the performance of the baseline, but rather provide another approach for deployment condition where memory is constrained, such as on smart phone. I also appreciate your complementary experiments and they provide a clearer picture of the technique and its implications. But I have to admit that I don't have broad experience in this field so I can not provide you a meaning improvement suggestion (I lowered my confidence from 3 to 2)
I decide to raise my score from 2 to 3.
The contribution is the specified architectural intervention and its measured consequences
Thank you for making the concern explicit. We would like to clarify a distinction between the qualitative prediction quoted in the review and the contribution of the paper.
The paper is not an experiment on whether an arbitrary fixed input can produce any learning. It is an empirical study of an extreme, precisely specified restriction on the lexical interface. For 49,152 vocabulary items, we supply identity through 16 predetermined binary coordinates, repeat them to backbone width without trainable interface parameters, and introduce no dedicated learned embedding generator. We then measure what a large decoder-only model learns under this restriction.
The prediction that “a Transformer can learn from fixed binary embeddings, albeit with reduced performance” does not determine the extent or task profile of the retained capability. In the reported experiment, canonical codes support 52.40% HellaSwag normalized accuracy, 70.51% PIQA accuracy, and 42.75% LAMBADA accuracy. An invertible recoding also supports substantial capability. These measurements turn a qualitative expectation into a concrete characterization of a particular architectural constraint.
A component ablation does not derive its scientific value solely from outperforming the unrestricted control. Here, the learned-input model measures the utility of independently adjustable input rows. The fixed-code models establish how much language-modeling capability can be constructed without them. The observed gap and the retained capability are both results of the intervention.
The construction also makes a fundamental architectural distinction executable: token identification can be predetermined, while the learned use of token identity is implemented by shared downstream computation. This is not a claim that representations cease to be learned, nor that frozen embeddings have no precedents. The specific contribution is the minimal binary specification, fixed lift, absence of a dedicated input generator, and empirical characterization of the resulting model family at the studied scale.
Its methodological value is already instantiated. The released models provide an immutable and fully inspectable input boundary: token coordinates cannot move during training, while the rest of the system learns. This supports controlled investigation of input geometry, scale, and parameter allocation, and supplies a clear academic example separating categorical identity from learned lexical coordinates.
In response to the reviews, we have extended the experimental record with a normalized-input configuration and a deeper-backbone configuration, each evaluated at exactly 48,444,211,200 prediction targets, together with complementary three-seed evidence in a smaller regime. The revision distinguishes the observations supported by these experiments from effects that require additional matched controls.
We have therefore revised the presentation to foreground the work as a precisely specified architectural intervention, its quantitative characterization, and an inspectable experimental model family, rather than allowing its contribution to be reduced to the generic statement that fixed inputs can learn.