acceptodds
Withdrawn from ICLR 2027

CONTINUUMFORMER: PERSISTENT SOFT STRUCTURAL MEMBERSHIP FOR INCREMENTAL SEQUENCE AND DEPTH COMPUTATION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformers repeatedly construct input‑dependent relationships through attention, routing, and layer aggregation, yet these structural associations are usually treated as transient. We introduce ContinuumFormer, a Transformer architecture built around persistent soft structural membership: token‑to‑sequence‑structure and layer‑to‑depth‑structure associations are maintained as a persistent state through sparse, overlapping assignments. The state induces a weighted structural topology that parameterizes sparse structural reconstruction, and is updated through candidate retrieval, change detection, hysteresis, local correction, and an explicit update budget. We formalize this mechanism as a state‑transition system and derive conditional properties for topology construction, membership‑to‑topology stability, fixed‑state local sensitivity, and budgeted structural maintenance. We evaluate the architecture on a 150M‑parameter language model using matched variants, controlled ablations, and long‑context validation. The experiments show improved language‑modeling performance over the matched Vanilla baseline across the evaluated standard context lengths, while axis‑wise ablations reveal asymmetric behavior between sequence‑ and depth‑side organization. An additional extended‑context evaluation further probes robustness of the full persistent‑state configuration beyond the standard context range. The study is designed as controlled architectural validation rather than compute‑saturated language‑model pretraining. Overall, the results provide controlled empirical evidence that persistent structural membership can serve as an additional organizational abstraction for Transformer computation.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Transformers, Structural Membership, Long-Context Modeling, Incremental Computation, Neural Architecture, Sparse Computation, Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_gunKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ContinuumFormer (CF) which keeps a "persistent soft structural membership", manifested as "token‑to‑sequence‑structure and layer‑to‑depth‑structure". Main contribution, as authors claim, is "persistent membership state whose induced weighted relations parameterize structural reconstruction across both sequence and depth axes."

Experiments to confirm the hypothesis include training models at 150M and 3B scale, with scaling study done only at 150M model parameter size. The axes studied are: model component ablations and context length.

However, overall, the paper feels weak to me, for primarily two reasons:

  1. Experiment section is weak and quite confusing honestly - not only are the perplexity numbers REALLY weird (for example why does the perplexity of 32k get better than 16k so drastically), and what is happening to the 102K ppl? That seems like a actively-handicapped baseline, or major bugs.
  2. The importance of a "persistent soft structural membership" is not really explained - it seems that it offers SOME benefit in language modeling at small scales, but that is a fairly underwhelming argument to me.

Separately, this draft has a LOT of formatting issues

Critical strengths

The paper does a decent number of component and axis ablations. The authors are explicit about the limitations of the current experiments. The depth-side results in particular seem worth investigating further.

Critical weaknesses

I am confused by reported perplexities:

  1. Why does perplexity drop so much from 16k to 32k?
  2. It seems the benefit of CF drops severely from 8k/16k to 32k
  3. The 102k study is a different setup altogether?
  4. These are all extrapolation perplexities right? So again, why is 32k suddenly better?

Experiment setup also seems a bit weird: Opting for NoPE seems to make some sense in theory but I think its practically bad? I would like to see RoPE or other transformer numbers too.

I am confused by some results: Do I read correctly that Depth-structure only outperforms CF? Why do we use the token structure?

Questions for authors

(this section is paraphrased based on weakness)

  1. Why do the PPL values behave so strangely across context length, especially the sharp improvement at 32K?
  2. What explains the 60K → 371 PPL difference at 102K? How reproducible is this result?
  3. Do the gains remain with a standard RoPE-based Transformer?
  4. Do the gains remain at larger model/data scales with a matched baseline?
  5. Given that Depth-only is consistently better than Full-CF, what evidence is there that the sequence-side mechanism is actually useful?
  6. How does Depth-only compare against DenseFormer / Attention Residuals / Mixture-of-Depths under a matched setup?

Additional questions:

  1. Please give more details on dataset construction - how did you split data, how big are the val splits etc
  2. Can you measure this on any standard downstream benchmarks, rather than just long context validation?
Reviewer_symVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ContinuumFormer, which maintains token-to-sequence and layer-to-depth structure associations as a persistent state across updates. The main idea is to retrieve candidate structures, detect changes and correct only a budgeted subset of rows with hysteresis instead of recomputing structure associations from scratch. The paper presents four propositions on the induced topology. Empirical evaluations on a 150M model show 41-49% perplexity reduction over a vanilla Transformer at 4K-32K context, and 99% reduction under 102,400 token validation.

Critical strengths
  1. The idea of treating structural associations as a persistent state that is selectively updated, rather than recomputing them at every step is interesting.

  2. The sequence and depth axes are formulated as instances of one membership operator with a shared maintenance lifecycle in Section 2.3, which is a clean abstraction.

  3. The variants are compared under a matched setting (same base model, token budget and optimizer) in Section 4.2, and structural diagnostics such as update ratio and correction counts are reported together with perplexity in Table 3 and 5.

Critical weaknesses
  1. The perplexity results are not reliable enough to support the main claim. Table 1 shows that PPL at 32K is lower than at 4K for all methods, and the vanilla model has 916 validation PPL at 4K. Also, the depth router use the mean pooled hidden state of each layer (Appendix E.2), but it is not specified whether this pooling is causal. If it averages over the whole sequence, future tokens could affect the predictions.

  2. Ablations doesn't support the dual axis design. full-CF is worse than depth-only at all four validation lengths (Table 1), and worse than persistent (439) and soft membership (441) in training PPL (Table 3). Sequence-only (469) is even worse than vanilla baseline (464). Since the sequence axis is half of the proposed design, its contribution is not supported by experiments. The ablations also use a single seed, mentioned in Section 4.2.

  3. Propositions 1-4 are basic algebraic facts; MM⊤MM^\top is PSD, triangle inequality, Lipschitz composition, counting, and it also does not explain anything about the model behavior, and the actual forward computation is only described in Appendix E.2.

Questions for authors
  1. Is the mean pooling for the depth router causal? Why is the 32K perplexity lower than 4K for all methods?

  2. What does the sequence axis add over depth-only, given that full-CF is worse than Depth-only at all validation lengths?

Reviewer_HpjvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes persistent soft memberships over sequence and depth, maintained through selective updates and used for post-stack hidden-state reconstruction. It evaluates the approach on a small language model and provides basic algebraic properties of the induced membership topology.

Critical strengths

The axis-wise ablations expose important limitations of the proposed design, including the superiority of Depth-only over the full model.

Critical weaknesses
  1. The experiments do not establish the value of the central mechanism. Depth-only outperforms Full-CF at every standard validation length in Table 1. Table 3 shows little training-PPL improvement from adding persistence, followed by degradation with local correction and budget control. Sequence-only exhibits extreme validation failure, including within the training context length (Table 8), without an adequate diagnosis. Together with the absence of matched depth-aggregation baselines, these results leave the benefit of the proposed persistent dual-axis mechanism unsubstantiated.

  2. The theoretical and computational contributions are insufficient. Propositions 1–3 establish elementary Gram-matrix and norm properties, while Proposition 4 largely restates the imposed update budget. They do not analyze the evolving persistent state, accumulated approximation error, or effects on predictions. The implementation retains the full Transformer computation and adds post-stack reconstruction, with lower measured throughput. Thus, neither the analysis nor the systems results establish a substantive advantage for the proposed incremental computation.

Questions for authors
  1. Does the depth router pool hidden states across the entire current sequence? If so, how are predictions at earlier positions prevented from depending on later tokens through the depth-routing weights and reconstruction? Please clarify the exact causal computation used during both training and validation.

  2. Does the reported 102,400-token evaluation use a single context of that length or multiple shorter segments? Please specify the attention window, segmentation, state reuse/reset rules, and scored positions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.