- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes ContinuumFormer (CF) which keeps a "persistent soft structural membership", manifested as "token‑to‑sequence‑structure and layer‑to‑depth‑structure". Main contribution, as authors claim, is "persistent membership state whose induced weighted relations parameterize structural reconstruction across both sequence and depth axes."
Experiments to confirm the hypothesis include training models at 150M and 3B scale, with scaling study done only at 150M model parameter size. The axes studied are: model component ablations and context length.
However, overall, the paper feels weak to me, for primarily two reasons:
- Experiment section is weak and quite confusing honestly - not only are the perplexity numbers REALLY weird (for example why does the perplexity of 32k get better than 16k so drastically), and what is happening to the 102K ppl? That seems like a actively-handicapped baseline, or major bugs.
- The importance of a "persistent soft structural membership" is not really explained - it seems that it offers SOME benefit in language modeling at small scales, but that is a fairly underwhelming argument to me.
Separately, this draft has a LOT of formatting issues
Critical strengths
The paper does a decent number of component and axis ablations. The authors are explicit about the limitations of the current experiments. The depth-side results in particular seem worth investigating further.
Critical weaknesses
I am confused by reported perplexities:
- Why does perplexity drop so much from 16k to 32k?
- It seems the benefit of CF drops severely from 8k/16k to 32k
- The 102k study is a different setup altogether?
- These are all extrapolation perplexities right? So again, why is 32k suddenly better?
Experiment setup also seems a bit weird: Opting for NoPE seems to make some sense in theory but I think its practically bad? I would like to see RoPE or other transformer numbers too.
I am confused by some results: Do I read correctly that Depth-structure only outperforms CF? Why do we use the token structure?
Questions for authors
(this section is paraphrased based on weakness)
- Why do the PPL values behave so strangely across context length, especially the sharp improvement at 32K?
- What explains the 60K → 371 PPL difference at 102K? How reproducible is this result?
- Do the gains remain with a standard RoPE-based Transformer?
- Do the gains remain at larger model/data scales with a matched baseline?
- Given that Depth-only is consistently better than Full-CF, what evidence is there that the sequence-side mechanism is actually useful?
- How does Depth-only compare against DenseFormer / Attention Residuals / Mixture-of-Depths under a matched setup?
Additional questions:
- Please give more details on dataset construction - how did you split data, how big are the val splits etc
- Can you measure this on any standard downstream benchmarks, rather than just long context validation?