acceptodds
Under review as a conference paper at ICLR 2027

Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale

Abstract

A trainable input embedding table assigns each vocabulary item an independently adjustable vector. We investigate whether this token-specific parameterization is required for substantial language-modeling capability, or whether a shared Transformer can learn from fixed token identities. We compare three decoder-only language models trained from scratch with the same tokenizer, contextual backbone, untied output-head architecture, and training recipe, with a target budget of 100 billion prediction tokens per model. Their input interfaces are a learned table, canonical 16-bit token-ID codes, and one fixed invertible recoding over GF(2). The fixed codes are repeated to model width without an additional trainable input projection. Both fixed-code models acquire substantial capabilities: canonical codes achieve 52.40% HellaSwag normalized accuracy, 70.51% PIQA accuracy, and 42.75% LAMBADA accuracy. The fixed interfaces remove 100.7 million trainable parameters, yielding 1.711B-parameter models, but parameter reduction is not the central result. These single-run experiments distinguish architectural necessity from empirical utility: independently trainable token-specific input vectors are not required for the observed capabilities. A fixed identity interface also provides a controlled setting for studying representation learning downstream of an immutable input, without establishing where particular capabilities are localized.

Primary area:
generative models
Keywords:
LLM, fixed 16-bit token-ID codes

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eNv4OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the author challenge the common practice of constructing the embedding matrix with fully learnable practice and instead propose to use a deterministic binary encoding. The author would like to study whether the transformer LM can still achieve a reasonable capacity with only deterministic binary encoding. To answer this question, they pretrain 1.6B scale models with normal learnable embedding and deterministic binary encoding respectively. The empirical results show that the while the model trained with binary encoding lags far behind the standard model, it still has some reasonable capacity.

Critical strengths

The paper is clearly written and the tone is rigorous.

Critical weaknesses

The main weakness is I don't get any insight after reading the paper. The research question below

We investigate whether this token-specific parameterization is required for substantial language-modeling capability, or whether a shared Transformer can learn from fixed token identities

is not very valuable because I think majority of the people in the comunity would guess with high confidence the answer is "a Transformer surely can learn from fixed binary embedding albeit reduced perforamnce". And this is exactly what the empirical results in the paper showed. This work doesn't bring new "information" or new "insight".

Questions for authors

My question is whether I missed some subtle insight from the paper. I really think the reserach question itself is more or less a "known" result.

3 comments
AuthorsOpenReview ↗

The contribution is the specified architectural intervention and its measured consequences

Thank you for making the concern explicit. We would like to clarify a distinction between the qualitative prediction quoted in the review and the contribution of the paper.

The paper is not an experiment on whether an arbitrary fixed input can produce any learning. It is an empirical study of an extreme, precisely specified restriction on the lexical interface. For 49,152 vocabulary items, we supply identity through 16 predetermined binary coordinates, repeat them to backbone width without trainable interface parameters, and introduce no dedicated learned embedding generator. We then measure what a large decoder-only model learns under this restriction.

The prediction that “a Transformer can learn from fixed binary embeddings, albeit with reduced performance” does not determine the extent or task profile of the retained capability. In the reported experiment, canonical codes support 52.40% HellaSwag normalized accuracy, 70.51% PIQA accuracy, and 42.75% LAMBADA accuracy. An invertible recoding also supports substantial capability. These measurements turn a qualitative expectation into a concrete characterization of a particular architectural constraint.

A component ablation does not derive its scientific value solely from outperforming the unrestricted control. Here, the learned-input model measures the utility of independently adjustable input rows. The fixed-code models establish how much language-modeling capability can be constructed without them. The observed gap and the retained capability are both results of the intervention.

The construction also makes a fundamental architectural distinction executable: token identification can be predetermined, while the learned use of token identity is implemented by shared downstream computation. This is not a claim that representations cease to be learned, nor that frozen embeddings have no precedents. The specific contribution is the minimal binary specification, fixed lift, absence of a dedicated input generator, and empirical characterization of the resulting model family at the studied scale.

Its methodological value is already instantiated. The released models provide an immutable and fully inspectable input boundary: token coordinates cannot move during training, while the rest of the system learns. This supports controlled investigation of input geometry, scale, and parameter allocation, and supplies a clear academic example separating categorical identity from learned lexical coordinates.

In response to the reviews, we have extended the experimental record with a normalized-input configuration and a deeper-backbone configuration, each evaluated at exactly 48,444,211,200 prediction targets, together with complementary three-seed evidence in a smaller regime. The revision distinguishes the observations supported by these experiments from effects that require additional matched controls.

We have therefore revised the presentation to foreground the work as a precisely specified architectural intervention, its quantitative characterization, and an inspectable experimental model family, rather than allowing its contribution to be reduced to the generic statement that fixed inputs can learn.

AuthorsOpenReview ↗

Quantitative consequences of the specified restriction

The revision now includes five configurations at an identical budget of 48,444,211,200 prediction targets, including Learned and canonical Binary16 controls.

This extends the result beyond the qualitative expectation that fixed inputs can learn. It measures two distinct responses to the restriction: fixed row normalization partially narrows the observed gap on selected tasks, while adding one standard Transformer block does not broadly recover it. For example, LAMBADA accuracy is 43.35% for Learned, 40.07% for Binary16, 39.72% for Ext, and 41.88% for Norm.

The scientific object is the specified intervention: minimal deterministic binary identity coordinates, a parameter-free lift, and no independently trainable input rows or dedicated input generator. Its value is in establishing the capability retained under that restriction and measuring the consequences of concrete alternatives.

A performance cost is not a collapse of capability. The original 100B fixed-code results, the matched-budget controls, and the smaller repeated runs jointly characterize a working model family rather than only a prediction that some learning should be possible. The revised discussion makes its experimental and academic value explicit.

Thanks for your in-depth clarification. Following your reply, I made another pass to the work and I think I have missed some points of the work in my initial assessment. In particular, the value of parameter-free embedding goes beyond beating the performance of the baseline, but rather provide another approach for deployment condition where memory is constrained, such as on smart phone. I also appreciate your complementary experiments and they provide a clearer picture of the technique and its implications. But I have to admit that I don't have broad experience in this field so I can not provide you a meaning improvement suggestion (I lowered my confidence from 3 to 2)

I decide to raise my score from 2 to 3.

Reviewer_U8h7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper tests whether language models require independently trainable token specific input embeddings. It compares three models trained from scratch with the same Transformer architecture and training recipe. First, a learned-embedding baseline, fixed 16-bit binary token-ID codes repeated to the model width, and an invertible recoding of those binary codes. The fixed-code models contain approximately 1.71B trainable parameters, compared with 1.81B for the baseline, with a target training budget of 100B prediction tokens per model. Standard language-model evaluations demonstrate that both fixed interfaces support substantial capability, although they underperform the learned baseline on several benchmarks. The main contribution is an empirical demonstration of feasibility at this scale, rather than performance parity or a demonstrated efficiency improvement.

Critical strengths

The authors show that training models of this size with fixed minimal binary identities and evaluating them across multiple standard benchmarks provides stronger evidence than training convergence or selected generation examples alone. The learned-input control makes the associated quality cost explicit.

Critical weaknesses

As the related-work discussion acknowledges, shared, generated, and frozen input representations already challenge the necessity of independently trainable token vectors. The larger-scale binary-code experiment is useful, but the paper does not establish a broader empirical relationship, explain the mechanism, or demonstrate a practical advantage. The proposed use of fixed interfaces for studying representation formation remains prospective.

Another issue is that the intervention done changes trainability, input dimension, geometry, scale, and parameter count simultaneously. Missing comparisons with norm-matched fixed codes, frozen random embeddings, and trainable low-dimensional embeddings leave the source of the degradation unclear. For example, canonical binary codes reduce HellaSwag accuracy from 57.79 to 52.40 and increase WikiText perplexity from 16.50 to 18.58, but the experiments do not explain whether these gaps reflect the fixed interface itself or avoidable choices in its implementation and optimization.

Questions for authors
  1. Do you have controlled results, potentially at a smaller scale, separating input trainability from dimension and scale. for example, norm-matched fixed codes vs trainable 16-dimensional embeddings? this can strengthen the contribution beyond demonstrating feasibility.

  2. What are the exact optimizer-step and processed-token counts of each evaluated checkpoint? The paper specifies target budgets and emphasizes using checkpoint counters, but does not report those values.

3 comments
AuthorsOpenReview ↗

Input-scale control, exact snapshot counters, and interpretation

Thank you for identifying input scale, dimensionality, and trainability as separate factors. We have added a concrete scale-control configuration and revised the interpretation accordingly.

New fixed input-norm experiment.
Binary16-Norm retains the original 24-layer backbone and normalizes each nonzero canonical code row to RMS 0.02 before training. This matches the initial expected squared-norm scale of the learned input vectors. The operation is fixed and parameter-free; it does not center the codes, add a learned projection, or change their supports. Token zero retains its zero code.

This distinction matters in the supplied architecture because pre-attention RMSNorm does not remove the raw input's contribution to the residual stream. The control therefore addresses a concrete implementation and optimization factor raised in the review.

At exactly 48,444,211,200 prediction targets, the normalized model obtains 49.90% HellaSwag normalized accuracy, 70.29% PIQA accuracy, 41.88% LAMBADA accuracy, and 20.36 WikiText word perplexity.

We also trained a 25-layer model with the original unnormalized input. At the same budget, it obtains 49.25%, 70.08%, 39.72%, and 20.71 on those metrics. The smaller normalized model has better point estimates on several tasks, but this pair differs in both normalization and depth. We therefore report a configuration-level result rather than attributing the difference exclusively to normalization.

Controls separating trainability and dimension.
The new experiment addresses input normalization, but it does not complete the factorial comparison. Frozen random continuous codes and trainable 16-dimensional token rows remain distinct controls. We now explain their roles explicitly rather than treating all alternatives to a learned full-width table as interchangeable.

Smaller-scale replication.
We have incorporated an approximately 0.5B-scale series with three training seeds per interface. At 17.093B tokens per seed, learned and canonical-code models obtain mean validation perplexities of 2.44 and 2.36. This supports repeated successful learning in that regime; it does not isolate all factors of the input intervention or estimate large-scale benchmark variance.

Exact checkpoint counters.
For both newly evaluated models, the counters are:

  • optimizer updates: 184,800;
  • processed prediction targets: 48,444,211,200.

For the three original evaluated releases, the records used in the manuscript currently provide the approximately 100B run-level budget but not the exact checkpoint counters. We have made that distinction explicit in the checkpoint-provenance table rather than deriving exact values from launch arguments.

Contribution and practical scope.
We have revised the paper to foreground the experimentally instantiated interface rather than an efficiency claim: exact token identities are predetermined, their binary width is minimal, and the lift has no learned parameters. The learned backbone then constructs useful representations under that explicit restriction. The resulting model family offers a stable and inspectable boundary for representation studies, while the experiments quantify capability rather than claim an established systems advantage.

AuthorsOpenReview ↗

Direct norm control and exact training-budget records

The revised evaluation now includes Learned and unnormalized Binary16 controls at the same 48,444,211,200-target budget as Norm and Ext. All five models have completed exactly 184,800 updates.

This provides the requested direct input-scale comparison at unchanged backbone depth and trainable parameter count. Relative to Binary16, Norm raises LAMBADA accuracy from 40.07% to 41.88% and lowers perplexity from 22.32 to 19.90. HellaSwag normalized accuracy changes from 49.47% to 49.90%, and WikiText perplexity from 20.48 to 20.36. The complete results also show tasks that decline, so the conclusion is task-dependent partial recovery, not a uniform correction.

The parameter-reallocation control produces a different pattern: one additional standard block does not broadly recover Learned performance. The two explanations are therefore examined through separate matched-backbone or fixed-input comparisons, rather than the earlier Norm–Ext contrast alone.

The exact original final-run counts are now documented as well: 381,470 updates and 100,000,071,680 prediction targets for Learned, Binary16, and GF2, from terminal training logs. Intermediate counts are recorded in checkpoint metadata.

These additions turn input scale and this concrete parameter allocation into measured parts of the study. The core contribution remains the explicit minimal identity interface and its empirical characterization.

he new normalization control helps explain some of the performance gap. My remaining concern is that the paper tells us little about what is distinctive about this particular interface. A norm-matched frozen Gaussian baseline would show whether the binary codes offer any advantage over a straightforward frozen embedding table, or whether the 16-bit restriction comes with an additional quality cost. That limits the contribution in its current form so I keep my score.

Reviewer_PtUEOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a decoder-only language model needs a trainable embedding vector per token at scale. The paper replaces the embedding table of a 1.7 billion parameter (1.7B) model with a binary token ID repeated to hidden size with no trainable projection in between. A second variant passes the bits through a fixed, random, invertible map. Both setups and baseline keep the output head trainable. The authors claim that both fixed-code models achieve substantial capabilities, suggesting that a learned embedding layer is beneficial but not essential for a language model to acquire downstream capabilities. However, the paper frames this as necessity-versus-utility finding, and not parity, since the baseline with learnable embeddings remains better on most evaluations.

Critical strengths
  1. The method is clean and simple, and its description is easy to follow. The three-way comparison is cleanly controlled by using the same tokenizer, backbone, data and recipe.
  2. The paper reports its findings in an honest and transparent manner, does not overclaim, lists limitations accordingly, and provides the code and 1.7B-class weights.
Critical weaknesses
  1. The role of a trainable input embedding table has already been studied. Kocmi and Bojar [1] demonstrated that RNN language models with fixed random embeddings can still learn, but at the cost of increased perplexity. Hoffer et al. [2] replaced the embedding and classifier of a WikiText-2 LSTM with a fixed transform and obtained 81.2 ppl (fixed) vs. 74.1 ppl learned. Welch et al. [3] found that freezing random input embeddings results in only a modest increase in perplexity, whereas freezing output embeddings is harmful. Bogoychev [4] trained a Transformer LM with frozen, tied random embeddings and saw the perplexity score triple. Although the paper claims it is 'not the first,' it should be positioned against these results since its actual contributions (scale, untied output head, and deterministic minimal codes) are exactly what differs from them. Stating this more prominently would improve the manuscript.

  2. One seed is used per interface, so the 5-point HellaSwag/LAMBADA gap is not quantifiable. The authors change fixedness, input rank, input norm, and param count all at once, and acknowledge that the gap 'cannot be attributed to a single mechanism'. Untangling the contributions with clean ablations is listed as future work.

  3. There is no param-matched baseline, and Sec 6–7 extensively describe what the paper and experiments do not show.

Overall, I think this line of work has merit. In my opinion, the manuscript needs to be better positioned against prior and existing work. It also needs more scientific depth, including an analysis of why the gap arises, as well as statistical robustness in the form of seeds and ablations that distinguish fixedness from dimensionality, norm, and parameter count. Currently, it is mainly one observation. However, with more depth and precise positioning in literature, it could become a good paper, and I would be glad to read it again. My score reflects the manuscript as submitted. I am open to revisiting it if the authors can address the points above substantively.


[1] Tom Kocmi and Ondřej Bojar. 2017. "An Exploration of Word Embedding Initialization in Deep-Learning Tasks." In Proceedings of ICON-2017.

[2] Elad Hoffer, Itay Hubara, and Daniel Soudry. 2018. "Fix Your Classifier: The Marginal Value of Training the Last Weight Layer." ICLR 2018.

[3] Charles Welch, Rada Mihalcea, and Jonathan K. Kummerfeld, 2020. "Improving Low Compute Language Modeling with In-Domain Embedding Initialization." EMNLP 2020.

[4] Nikolay Bogoychev. 2021. "Not All Parameters Are Born Equal: Attention is mostly what you need." Fourth BlackboxNLP Workshop

Questions for authors
  1. Seed variance: With one run per interface, what is the spread of the HellaSwag/LAMBADA numbers from run to run? Even two extra seeds per interface would suffice. Would all 100 billion tokens be required for a seed study?

  2. Would learning a 16-to-2048 projection on top of the fixed codes recover the learned-table result? This would separate fixedness from dimensionality. What is your intuition here?

  3. Have you considered conducting an ablation study by reallocating the saved embedding parameters to the transformer backbone? This would allow you to distinguish the impact of fixed inputs from having fewer parameters. As already mentioned above, it may not be necessary to use all 100 billion tokens to gain meaningful insights here.

4 comments
AuthorsOpenReview ↗

Positioning, replication, and parameter-reallocation experiments

Thank you for the specific suggestions and for identifying the relevant frozen-embedding literature. We have used these points to sharpen both the positioning and the experimental account.

Positioning against prior work.
We have expanded the related-work discussion to include Kocmi and Bojar, Hoffer et al., Welch et al., and Bogoychev. We agree that successful learning with frozen inputs is not itself a new general observation.

The distinction we now foreground is the combination of minimal deterministic binary identity codes, a fixed lift, no independently trainable input rows, and no dedicated learned embedding generator before a standard decoder backbone, together with the measured downstream capability at the studied scale. We also make input/output treatment explicit: the output head remains trainable and untied, which distinguishes our intervention from experiments that freeze tied vocabulary parameters.

Training-seed variability.
We do not yet have additional independent training runs of the large-scale configurations and therefore cannot report the run-to-run spread of their HellaSwag or LAMBADA results. We have added a complementary smaller-scale series with three training seeds per interface:

InterfaceTokens per seedMean validation PPLRelative PPL seed range
Learned input17.093B2.444.8%
Canonical Binary1617.093B2.364.6%
Affine Binary1616.276B2.394.5%

These experiments support repeated successful learning under the fixed-input restriction. They use a different tokenizer, architecture, and training distribution, so we do not present them as estimates of large-model benchmark variance.

Full 100B runs are not intrinsically necessary for a seed study: independently trained models evaluated at a common shorter budget would estimate variability at that budget. Intermediate checkpoints along one trajectory, however, are not independent seeds.

Learning a 16-to-2048 projection.
This is a useful proposed control. Such a projection would allow the model to adapt the input geometry and its contribution to the residual stream while retaining shared low-rank structure. It would not be equivalent to independently trainable token rows. We have clarified this distinction in the research directions; we do not yet have results for this variant.

Reallocating input parameters to the backbone.
We implemented this suggestion using one additional unchanged Transformer block. The resulting 25-block model contains 1,778,489,344 trainable parameters versus 1,812,039,680 for the learned control, a conservative approximate match within 1.85%.

The removed input table corresponds to roughly one and a half standard blocks. We selected one extra block rather than changing FFN widths or introducing a specially sized module, thereby preserving the original block structure.

This run has been evaluated at exactly 48,444,211,200 prediction targets. We report it as an intermediate parameter-reallocation configuration, not yet as evidence that extra depth recovers the learned-input result: the original evaluated learned checkpoint has a larger training budget.

We have also added a fixed input-norm configuration at the same intermediate budget. Together, these additions extend the empirical characterization while keeping the remaining distinctions - training variability, exact budget matching, and input geometry - explicit.

Thank you for your response and for uploading the revised manuscript. Having read the revision, I appreciate the extended related work section. The paper is now better positioned within the existing line of frozen-input work.

Thank you also for the two new runs. Adding an additional transformer block seems a reasonable way to reallocate the removed parameters. However, both runs are evaluated at the 48B. As you state in Section 5.2, there is no Learned or Binary16 control with the same budget. Do you have any checkpoints of the original runs close to 48B that could be evaluated? Without these, the reported numbers cannot yet demonstrate whether extra depth or input normalization would close the gap to the Learned table. Interestingly, Norm is slightly ahead of Ext downstream despite having fewer parameters. This suggests that input scale may matter more than parameter count. However, this remains uncertain as long as the two runs differ in both depth and normalization.

The three-seed series shows that fixed codes can be reliably trained in a smaller setup. However, as it uses a different tokenizer, architecture and data, and the difference between the interfaces lies within the seed range, it does not reveal how much of the 5 point Hellaswag and LAMBADA gap (1.7B) is due to variation between runs. As you mentioned in your response, a seed study does not require a budget of 100B. I therefore wonder why the seeds were not run in the 1.7B setup with a smaller token budget. The learned projection on top of the fixed codes also remains untested.

Overall, the revision improves the positioning, but I still have concerns about the experimental depth.

AuthorsOpenReview ↗

Budget-matched Learned and Binary16 controls are now included

Thank you for the concrete suggestion to evaluate the original intermediate checkpoints. These evaluations are now complete. All five configurations are compared at exactly 48,444,211,200 prediction targets and 184,800 updates under the same evaluation configuration.

The canonical 24-layer control allows the two interventions to be assessed separately:

  • Normalization, with depth and parameter count unchanged: LAMBADA accuracy is 40.07% for Binary16 and 41.88% for Norm; perplexity is 22.32 and 19.90. Learned obtains 43.35% accuracy. Normalization therefore narrows the observed accuracy gap, although its effects across the suite are mixed.
  • Additional standard depth, with the input rule unchanged: HellaSwag normalized accuracy is 49.47% for Binary16 and 49.25% for Ext; LAMBADA accuracy is 40.07% and 39.72%. This allocation does not provide broad recovery at the measured budget.

These findings are now presented as the outcomes of the requested comparisons, rather than inferred from Norm versus Ext alone.

On compute allocation, we prioritized these structural controls at an operating point with informative downstream capability. This was a resource-allocation decision, not a claim that seed studies require 100B tokens. The smaller three-seed experiment assesses repeated fixed-code learning in its own regime; the large-scale comparisons remain checkpoint-level observations. The learned projection is a separate proposed experiment.

The paper now distinguishes three empirical questions directly: capability under fixed identities, the response to fixed row normalization, and the response to reallocating capacity into a standard backbone block.

Thank you for evaluating the intermediate checkpoints. I have also read your reply in the general thread and the updated manuscript. The comparison with an equal budget is more informative than before. It shows that the extra block does not close the gap to the Learned table, so the difference in parameter count alone is unlikely to explain it. Normalization helps with LAMBADA, but the effect is similar in size to the difference between Binary16 and GF2, so it's difficult to judge based on a single run. Without seeds at this scale and the learned projection, the reason for the gap remains unclear to me.

I understand that compute was limited during the rebuttal period. As a suggestion for future work, when using a learned projection, the code assignment determines which tokens share projection weights and therefore gradients. To test how much this matters, I would recommend comparing a few different code assignments. I have raised my scores.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Additional experiments and clarification of the contribution

We thank the reviewers for identifying three issues that deserve clearer treatment: positioning relative to frozen and generated embeddings, the effects of input scale and parameter allocation, and the distinction between evaluation uncertainty and training-seed variability.

We have revised the manuscript and added the following evidence.

1. Two additional large-scale runs, trained from scratch.

  • Binary16-Ext: the original fixed canonical input interface with one additional standard Transformer block. It has 25 blocks and 1,778,489,344 trainable parameters, approximately 1.85% fewer than the untied learned-input control. Width, attention structure, and FFN width are unchanged.
  • Binary16-Norm: the original 24-block backbone with fixed canonical codes normalized row-wise to RMS 0.02 for nonzero rows, matching the initial expected squared-norm scale of the learned input vectors. The zero code is preserved. No input parameters or learned projection are added.

Both evaluated snapshots have exactly 184,800 optimizer updates and 48,444,211,200 processed prediction targets. They are intermediate checkpoints of runs using the original 100B-target schedule, not completed 100B experiments.

MetricBinary16-ExtBinary16-Norm
HellaSwag normalized accuracy49.25±0.5049.25\pm0.5049.90±0.5049.90\pm0.50
ARC-Easy normalized accuracy60.19±1.0060.19\pm1.0061.41±1.0061.41\pm1.00
ARC-Challenge normalized accuracy32.42±1.3732.42\pm1.3734.22±1.3934.22\pm1.39
PIQA accuracy70.08±1.0770.08\pm1.0770.29±1.0770.29\pm1.07
LAMBADA accuracy39.72±0.6839.72\pm0.6841.88±0.6941.88\pm0.69
LAMBADA perplexity21.53±0.7821.53\pm0.7819.90±0.7119.90\pm0.71
WikiText word perplexity20.7120.7120.3620.36

Accuracy entries are percentages, and the errors are evaluation standard errors. The revised appendix reports the complete suite, including tasks on which the ordering reverses.

These results characterize two additional configurations at an identical token budget. Their direct comparison changes both depth and normalization; it therefore does not isolate either effect. We also do not treat comparison with the original approximately 100B checkpoints as an equal-budget test.

2. Complementary three-seed evidence at smaller scale.

We have incorporated an earlier approximately 0.5B-scale experiment with three training seeds per interface. Learned and canonical-code models each process 17.093B tokens per seed and obtain mean held-out perplexities of 2.44 and 2.36, respectively. An affine-code variant obtains 2.39 after 16.276B tokens per seed.

This series documents repeated successful fixed-code learning in a separate regime. We distinguish it explicitly from replication of the large-scale benchmark results and do not interpret seed ranges as confidence intervals.

3. More precise positioning.

The revised paper centers on a predetermined token-identity interface: minimal fixed binary coordinates, a parameter-free lift to backbone width, and no dedicated learned input generator. Its contribution is an empirical characterization of this restrictive interface and an inspectable model family for separating token identification from learned input coordinates. The related-work discussion now explicitly includes earlier frozen-input and component-freezing studies.

Dear Reviewer,

Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.

Thanks,

Area Chair

Thank you for your general response and for running the two additional tests. My comments can be found in my reply to your individual answer. In short, while I think the new configurations are useful additions, they cannot yet be interpreted without budget-matched Learned and Binary16 controls.

AuthorsOpenReview ↗

Five-model budget-matched evaluation and exact endpoint counts

We have completed the intermediate-budget controls requested in the discussion.

Learned, Binary16, GF2, Binary16-Ext, and Binary16-Norm are now evaluated at exactly 184,800 updates and 48,444,211,200 prediction targets each, using the same Harness configuration.

MetricLearnedBinary16GF2ExtNorm
HellaSwag acc_norm54.2449.4748.3949.2549.90
ARC-Easy acc_norm63.0560.4860.6560.1961.41
PIQA acc71.6069.5369.2170.0870.29
LAMBADA accuracy43.3540.0738.5039.7241.88
LAMBADA perplexity15.9422.3223.4521.5319.90
WikiText word perplexity18.4820.4821.2820.7120.36

Accuracy entries are percentages. The revised appendix contains the complete suite and evaluation standard errors.

The common-budget comparison gives two specific findings:

  • Fixed row normalization gives task-dependent partial recovery. At unchanged backbone depth and parameter count, LAMBADA accuracy increases from 40.07% to 41.88%, narrowing the observed gap to Learned from 3.28 to 1.47 percentage points.
  • Reallocation to one additional standard block produces a different response: it does not broadly recover the learned-input result at this budget. The full table reports improvements and declines for both modifications.

The exact original endpoint counts are also now documented: 381,470 updates and 100,000,071,680 prediction targets for each of the three original runs, according to their terminal training logs. The original 100B evaluations remain in the paper.

The revision now combines a five-model common-budget comparison, the original endpoint evaluations, and the complementary smaller three-seed series. It characterizes both the substantial capability retained under minimal predetermined token identities and the measured consequences of normalization and parameter allocation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.