acceptodds
Under review as a conference paper at ICLR 2027

Designing Better Transformers: Using Amorphous Network Proxies to Engineer the Unified Highway Transformer

Abstract

Standard transformer architectures define information flow through rigid, serialized sequences of discrete operators. In deep or very deep implementations, this leads to PreNorm Dilution, or Latent Amnesia, in which deep layers become blind to early layer signals in the residual stream, stalling convergence. Existing solutions to PreNorm Dilution are computationally expensive or add engineering complications. This work utilizes unconstrained Amorphous Neural Network (AmNN) clouds as diagnostic proxies allowing analysis of information flow that emerges free from the constraints of discrete operators. These paths empirically confirmed the demand for historical latent context in deeper layers. Modeling this flow back into a discrete operator transformer resulted in the creation of the Unified Highway Transformer (UHT). The UHT introduces an parallel memory bus () that builds a history of latent context from the residual stream and makes that history available to deep layers. To counter initialization instability during training, a Dynamic Variance Annealer initially scales residual projections entering the bus, phasing out scaling after initial epochs. The UHT was evaluated against a constrained 120-layer, 9.7-million total trainable parameter architectural stress test () on WikiText-103. A gradient tracker revealed dynamic, high-variance, gradient norm activation at deep levels in the UHT, illustrating that the UHT restored the representational capacity of the deep layers. In contrast, a 120-layer Pre-LN control showed gradient stagnation in deep layers. The UHT reduced terminal perplexity in the tested configuration (118 for the Pre-LN to 66 for the UHT). An ablation UHT removing all layer normalization operations achieved a similar perplexity of 68. A 180-layer test of the UHT with layer normalization operations showed similar performance. The UHT provides a scalable, parameter-efficient architecture for extreme-depth modeling that solves gradient stagnation in deep networks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
transformer architecture, extreme-depth transformers, deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_MNXBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript studies representation collapse and gradient stagnation in deep transformers (this is called "PreNorm Dilution" or "Latent Amnesia"). The authors propose using "Amorphous Neural Network (AmNN) clouds" as a diagnostic workbench to observe emergent cross-layer information flow. This is an unconstrained non-discrete network proxy from a 2026 patent application.

Based on heuristic interpretations of this diagnostic, the authors introduce the "Unified Highway Transformer" or UHT, which incorporates an O(1) parallel memory bus (ZmemRBLdmem) updated recurrently and injected into deeper layers via gated projections, as well as a decaying "Dynamic Variance Annealer" to stabilize early training. The architecture is tested on WikiText-103 using an extreme non-standard configuration: 120 layers (and 180 layers) with hidden dimension dmodel=64 (9.7M parameters). UHT achieves a perplexity of 66 versus 118 for a vanilla Pre-LN baseline.

Critical strengths

The paper looks at an important structural property of deep residual architectures (i.e., the tendency of deep layers to under-utilize capacity due to residual stream dilution) and proposes a parameter-efficient O(1) bus mechanism.

Critical weaknesses

W1. The architecture (L=120, d=64) looks like a concocted bottleneck. In a network where the embedding layer accounts for 20% of the entire model parameter budget, depth-wise signal dilution is an artifact of the extreme aspect ratio. While d=768 is briefly mentioned, the experiment is conducted on an unspecified "small dataset" without quantitative evaluation, test perplexity curves, or rigorous baselines.

W2. Degradation of signal and identity mapping in deep Pre-LN transformers has been analyzed extensively in prior literature. The proposed solution of an exponential moving average or parallel linear recurrent highway bus is conceptually very similar to classic Highway Networks, Recurrent Highway Networks, and existing dual-stream architectures (e.g., ResiDual, DeepNorm). DeepNorm, ReZero, and ResiDual already achieve non-vanishing gradients and stable convergence at depths of 100 to 1000 layers without requiring an auxiliary memory bus. Absent a comparison with those methods, the empirical claims sound ungrounded.

W3. While core ideas are understandable, the discussion heavily relies on non-standard terminology ("Latent Amnesia", "Amorphous Network clouds", "Dynamic Variance Annealer"). The transition from the amorphous proxy to the specific formulation of the UHT memory bus is qualitative and hand-wavy rather than mathematically derived.

Questions for authors
  1. How does UHT perform when directly compared against DeepNorm, ReZero, and ResiDual on a standard architecture (e.g., L=24 & dmodel=768 or L=36 & dmodel=1024) on WikiText-103 or OpenWebText under similar training compute?
  2. Could you provide the explicit mathematical optimization formulation for the "Amorphous Network cloud"? Precisely how do the amorphous proxy dictate the parameterization for Zmem rather than an attention-based or non-linear recurrent bus?
Reviewer_CBMYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper provides an interesting analysis by using "amorphous neural network" (AmNN) to measure which components of the residual stream are preferred by the computational blocks across the layers of the Pre-LN transformer. Authors find that Attention blocks upweight the upstream states of the residual stream, over their own layer's residual stream states. Building on this observation, authors propose a "Unified Highway Transformer" (UHT), that essentially adds a secondary unnormalized memory stream, which writes into and reads from the residual stream after every computational block. Through experiments on the singular WikiText-103 dataset, and comparing against the singular Pre-LN backbone as a baseline, authors find that the proposed UHT transformer consistently improves training convergence upto 180 layers and ~1B parameters.

Critical strengths

While the paper indeed is lacking some experimental rigor (as I point out below), I think its nevertheless important to state that the initial analysis is interesting:

  1. Interesting Analysis on an important problem: The pathology of the "Curse of Depth" [1] is a well-known important problem. And the core analysis in the paper of replacing layer-wise MLP/Attention Blocks with a unified data distribution, allowing measurement of the importance of residual state contributions across layers, is an interesting analysis and motivation, and not something I've seen popularly used.

  2. Preliminary results are interesting, though insufficiently supported: The depth scaling experiments of upto 180 layers appear interesting, given the large improvements over Pre-LN. However, this really must be shown on larger datasets than WikiText-103, on standard open-source training pipelines (like OLMo) and compared to stronger/fairer baselines like mHC [3], Depth-muP [2]. There are also other rigor concerns (see Weaknesses).

As a summary: The analysis is interesting; the results, which can only be termed preliminary, appear promising. However, given the caveats above (and as detailed in the weaknesses below), the significance of these results cannot be evaluated. I hope the feedback below is useful to the authors in guiding revisions of their work.

References:

[1]: Sun, W., Song, X., Li, P., Yin, L., Zheng, Y., & Liu, S. (2026). The curse of depth in large language models. Advances in Neural Information Processing Systems, 38, 163104-163136.

[2]: Yang, G., Yu, D., Zhu, C., & Hayou, S. (2024, May). Tensor programs VI: Feature learning in infinite depth neural networks. In International Conference on Learning Representations (Vol. 2024, pp. 55099-55150).

[3]: Xie, Z., Wei, Y., Cao, H., Zhao, C., Deng, C., Li, J., ... & Liang, W. (2025). mhc: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880.

Critical weaknesses

Lacking Experimental Support & Rigor:

  • Missing Key Baselines, Lacking Experimental Fairness: Every comparison is against the vanilla Pre-LN backbone, and authors do not evaluate against any of the depth-scaling baselines cited (such as DeepNorm) nor are any stronger baselines like DepthmuP [2] (for depth scaling analysis) or LayerNorm Scaling [4] compared against. However, the key concern for me, is that the proposed UHT is a multi-stream design, and therefore any fair evaluation requres other multi-stream baselines such as Hyper-Connections/mHC [3]. Without these in particular, it is difficult to evaluate the validity of the claims around UHT.

    • Authors also do not report any variances (multi-seed or otherwise) or test-set perplexities, so even the small gains in the results cannot be assessed fairly.
    • Another key concern is the reliability of the training pipeline used, which the paper provides little to no information about. Authors are encouraged to support their experimental results using standard open-source training pipelines (like OLMo).
  • Reliable data scales (tokens/param): All experiments in the paper are carried out on the singular WikiText-103, which has a total of ~0.1B training tokens. While this is entirely unreported in the paper, this indicates all models trained in the paper are quite under-trained (at around 10 tokens/param or lower, it seems). This makes validity of the comparsions, even to Pre-LN, quite questionable/unsupported.

  • Depth Evaluations: The depth scaling evaluations appear to be on particularly deep and narrow backbones (particularly the 180-layer scale). The concern therefore is that the proposed UHT's gains may be biased to very narrow backbones, and this therefore require controlling for, and the width:depth ratios should therefore be clearly specified in the benchmarking. See [1]. It's also unclear to me if depth is infact helping UHT? I note the shallow model perplexity (Test 2 I believe) is comparable to or exceeding the 120 layer result. Width:depth ratios and parameter sizes are confounding here, and both are unreported in the analysis. The depth scaling analysis should ideally show both consistent gains with depth scaling and compare favorably to a shallow baseline, to better support the claim of improved depth scaling.

There are some other, less load-bearing concerns, on the completeness/support of the "amorphous neural network" (AmNN) based analysis, which I'll refrain from emphasizing here, since the above weaknesses are quite significant themselves, and need prior clarification/correction.

Overall, while there are significant concerns over the experimental validity and rigor in the work above, my hope is that the feedback above is nevertheless helpful to the authors in future revisions of their work. I'm happy to engage in any discussions/clarifications the authors may require on the above concerns.

References:

[1]: Wang, K., Huang, Y., Kang, J., González-Martínez, D., Liu, W., & Liu, S. (2026). DepthBench: Measuring How Residual Connections Enable More Computational Depth. arXiv preprint arXiv:2609.32534.

[2]: Yang, G., Yu, D., Zhu, C., & Hayou, S. (2024, May). Tensor programs VI: Feature learning in infinite depth neural networks. In International Conference on Learning Representations (Vol. 2024, pp. 55099-55150).

[3]: Xie, Z., Wei, Y., Cao, H., Zhao, C., Deng, C., Li, J., ... & Liang, W. (2025). mhc: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880.

[4]: Sun, W., Song, X., Li, P., Yin, L., Zheng, Y., & Liu, S. (2026). The curse of depth in large language models. Advances in Neural Information Processing Systems, 38, 163104-163136.

Questions for authors

Authors are requested to kindly address the above concerns (see weaknesses) on: Experimental baselines, scale of the data used in training, and the bias towards deeper + narrow networks and comparisons to shallow baselines in the depth-scaling evaluations.

Reviewer_qWNeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to improve on the transformer architecture, in particular the problem that the information is lost for deep layers despite residual connections that should facilitate flow. Improving transformers is certainly a valuable research direction. But the paper falls short in presentation, evaluation and novelty. Already in the first sentence they speak of discrete operators, though NNs are known to be continuous...The paper introduces already in the abstract multiple opaque terms like "amorphous neural network clouds" without relating them to known concepts. It uses O-notation where it is totally unclear why you would even do that or what something different from O(1) would be. There are things like DenseNet etc. that introduces more residual layers for better gradient flow. The paper fails to evaluate against them and generally the related work does not give evaluate against and discusses prior work (like DenseNet but also works that focus on residual streams for transformer). The evaluation does not meet typical standards of a scientific conference(you need multiple datasets, an ablation study, nice tables, etc). I recommend that the authors fully revise their work and engage more with it, as the overall direction of the paper might prove fruitful.

Critical strengths

see above

Critical weaknesses

see above

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.