acceptodds
Under review as a conference paper at ICLR 2027

A new physical architecture that general intelligence needs

Abstract

Transformer-type models face high computational overhead due to quadratic complexity when handling ultra-long contexts. Existing sparse attention methods lose global dependency and perform poorly on long-text inference tasks. This paper proposes automatic route attention, a new physical architecture that does not require large computing power to obtain attention, compressing computational load while obtaining global information, reduces inference complexity from quadratic growth with sequence length to linear growth with sequence length. Under the current Transformer architecture, the supported context length is strictly limited by computing power. The current common order of magnitude is 1 million. The new physical architecture in this paper increase context length by four orders of magnitude under current computing and storage conditions. This architecture can help realize artificial general intelligence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model architecture, large language models, Continuous learning, Long-term memory, Artificial General Intelligence, AGI

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_WfnYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper mentions the continual learning problem and the limited context problem for current LLMs. To resolve this issue, this paper proposes a new architecture, namely Switcher, which uses new LoRA adapters to continually update the LLM. In general, this paper is a very short technical note, and does not give sufficient details or supporting evidence.

Critical strengths
  1. Continual learning and extending LLMs' context are both critical problems for the current development of LLMs. This paper studies some important problems.
Critical weaknesses
  1. The writing quality of this paper is terrible. It includes countless typos and upper/lower case letter problems. Here I just name a few: a. Between lines 045-046, "... Switcher,It makes", letter case problem b. Between lines 034-045, "... Adaptation (LoRA)Hu et al. (2021)", wrong reference format

I believe this paper was never carefully proofread. I am also not sure whether the figures in this paper are vector graphics.

  1. The logic of this paper is a disaster. I believe the logical flow of this paper is highly vague.

In the introduction, the authors first review the long context problem and the continual learning problem in the first three paragraphs. However, these paragraphs did not go into anything in depth that could motivate the main contribution of this paper.

Claims like "this architecture may help to achieve artificial general intelligence" are also too general and vague.

  1. This paper is not written in a way that makes any sense. Its technical content is almost unreadable and does not include any appropriate equations.

  2. This paper does not give any empirical results to support the empirical soundness of its proposed architecture. This is a very important issue.

Questions for authors

Please refer to weaknesses 1-4.

Reviewer_WY7YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is not complete. It only talks about the generic idea, and doesn't provide any experimentation results. It talks about the idea of having an adapter that decides between using knowledge embedded in the model or knowledge embedded in the new parameters learned from the context, in an architecture called Switcher. But the details of the architecture, how does the adapter work, and how training of this new knowledge parameters is performed are missing from the paper.

Critical strengths

None

Critical weaknesses

Incomplete

Questions for authors

Any details on how this architecture is trained or what results it gets?

Reviewer_oqkUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an architecture called Switcher to address continual learning and long-context inference in large language models. The proposed approach represents knowledge through discrete connections between word elements based on co-occurrence and distinguishes known from unknown information to guide selective parameter updates. The paper claims that this approach can extend the effective context length by four orders of magnitude while reducing inference complexity from quadratic to linear in the effective context length.

Critical strengths

The paper addresses two important challenges for large language models: continual knowledge acquisition without catastrophic forgetting and efficient use of extremely long contexts.

Critical weaknesses

The proposed method is insufficiently specified. Core concepts—including word-element connections, the knowledge boundary map, and the Switcher mechanism—lack precise definitions. The paper provides neither mathematical formulations nor pseudocode explaining how these components operate and interact. Consequently, the proposed architecture is difficult to assess or reproduce. The central claims lack experimental support. The paper contains no experimental evaluation or results section. In particular, the claimed improvements in effective context length, inference complexity, and continual learning are not supported by empirical evidence. The novelty and relationship to existing work are insufficiently established. The paper does not adequately discuss relevant approaches, particularly mixture-of-experts models and parameter-efficient continual learning. The limited bibliography and missing citations for several approaches mentioned in the text further hinder assessment of the contribution.

Questions for authors

Could the authors verify and correct the publication year of the second reference? What experiments support the claimed improvements in context length, inference efficiency, and resistance to catastrophic forgetting? Could the authors provide a formal specification or pseudocode for Switcher, including how the knowledge boundary is determined and how parameters are selectively updated?

Comments on the paper

1 comment

Thank you for your time and review.

We have carefully replied to all questions, the open discussion is about to close , up to now all concerns are answered, we will answer any concerns further.

For the contribution of this paper, we emphasize that: Continuous learning is still not solved up to our best knowledge, it is an important obstacle to general intelligence, this paper shows a direction for this problem, this is the contribution point if people do not know how to solve.

We will appreciate you reconsider this point, it is looking forward to your recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.