- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Continuous Language Bridge (CLB), a two-pass language modeling framework that refines next-token predictions before emission. The first pass produces a probability-weighted embedding of candidate tokens, which is interpolated with the current observed token’s embedding. A shared second pass processes this bridge state using first-pass historical keys and values, and a learned gate combines the two predictive distributions. The construction preserves autoregressive likelihood and the baseline’s persistent KV-cache footprint. The paper evaluates dense and MoE variants on LM1B, OpenWebText, and external corpora, reporting perplexity improvements over AR baselines.
Critical strengths
S1. Memory-efficient clean-KV reuse. CLB reuses first-pass historical keys and values while keeping second-pass states transient, maintaining the same persistent KV-cache footprint as the corresponding AR backbone. This offers a potentially useful trade-off between predictive quality and memory usage.
S2. Controlled comparisons on LM1B. AR, Loop, and CLB share initial backbone parameters, training batches, and cumulative token budgets. CLB achieves lower test perplexity in this setting, providing preliminary evidence of effectiveness, although the gains are modest and lack multi-seed validation.
S3. Detailed methodological description and transparent reporting. The paper specifies the bridge construction, attention mask, training objective, and probability mixture. The appendices also document negative results and limitations involving training budgets, initialization, and evaluation protocols.
Critical weaknesses
W1. Insufficient positioning and comparison against closely related work. The manuscript does not adequately establish CLB’s contribution relative to existing looped and predictive-feedback language models. Hyperloop Transformers explores loop-level residual-stream mixing (Zeitoun et al., 2026), while PonderLM already combines probability-weighted token embeddings, top- approximation, residual input updates, and repeated computation within a next-token decision (Zeng et al., 2026a). Soft Thinking, which the manuscript cites, provides another probability-weighted feedback mechanism, although its training-free reasoning setting differs from CLB’s pretraining setting (Zhang et al., 2025). Other relevant approaches include learned depth allocation in Ouro, per-token latent computation in PonderLM-2, and selective iteration in TaH (Zhu et al., 2025; Zeng et al., 2026b; Fu et al., 2026). The clean-KV design should also be distinguished from the KV-sharing variant of Mixture-of-Recursions (Bae et al., 2025b), while LoopFormer and LoopMoE provide relevant precedents for time- and step-conditioned looping and iteration-conditioned sparse computation (Jeddi et al., 2026; Chen et al., 2026). These approaches are not equivalent to CLB, but the manuscript needs a clearer account of its architectural distinctions and representative matched-budget comparisons beyond AR and a plain tied-layer Loop.
W2. The experimental presentation and controls do not clearly establish the source or robustness of the reported gains. Three issues are particularly important:
- Training regimes differ substantially across settings. On LM1B, the 10B-token budget comes from sampling with replacement from a 0.894B-token corpus, yielding cumulative exposure approximately 11.2 times the corpus size (Appendix E.3). On OWT, the approximately 134M and 812M models receive around 21B and 9.18B tokens, respectively—approximately 157 versus 11 tokens per parameter (Appendices B.2 and C.1–C.2). These choices require clearer justification and limit conclusions about performance across data regimes and model scales.
- The main presentation does not adequately foreground the mixed Loop comparisons. Loop outperforms CLB in every OWT comparison in Table S4, including 22.5529 versus 22.7734 PPL at 70K updates on the same 4,330 validation chunks. No matched 160K Loop result accompanies the main OWT endpoint. The 0.47% single-seed improvement on LM1B therefore does not establish a consistent advantage over repeated computation.
- The final architecture lacks component-level attribution. CLB combines predictive feedback, interpolation, coordinate conditioning, gradient detachment, auxiliary supervision, and probability mixing. The historical controls do not isolate their contributions; the mixture-of-softmax experiment is explicitly not a gate-only ablation of the final model (Appendix D.3).
W3. Inconsistent numerical precision in the appendices. The appendix tables report comparable metrics with inconsistent numbers of decimal places, making the presentation less uniform and comparisons harder to follow. The authors should standardize the displayed precision for each metric across tables while retaining unrounded values for calculating differences and relative improvements.
References
- Zeitoun, A., Torroba-Hennigen, L., and Kim, Y. (2026). Hyperloop Transformers. COLM 2026.
- Zeng, B., Song, S., Huang, S., Wang, Y., Li, H., He, Z., Wang, X., Li, Z., and Lin, Z. (2026a). PonderLM: Pretraining Language Models to Ponder in Continuous Space. ICLR 2026.
- Zhang, Z., He, X., Yan, W., Shen, A., Zhao, C., Wang, S., Shen, Y., and Wang, X. E. (2025). Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space. NeurIPS 2025.
- Zhu, R.-J., et al. (2025). Scaling Latent Reasoning via Looped Language Models. arXiv preprint arXiv:2510.25741.
- Zeng, B., Li, H., Song, S., Wang, Y., Wang, Z., He, Z., Wang, X., and Lin, Z. (2026b). PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space. ICML 2026.
- Fu, T., You, Y., Chen, Z., Dai, G., Yang, H., and Wang, Y. (2026). Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning. ICML 2026.
- Bae, S., Kim, Y., Bayat, R., Kim, S., Ha, J., Schuster, T., Fisch, A., Harutyunyan, H., Ji, Z., Courville, A., and Yun, S.-Y. (2025). Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. ICML 2025.
- Jeddi, A., Ciccone, M., and Taati, B. (2026). LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. ICLR 2026.
- Chen, W., Li, T., Huang, W., Yin, Y., Shang, L., and Qin, C. (2026). LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling. arXiv preprint arXiv:2606.04438.
Questions for authors
Q1. Table 1 reports analytical KV-cache footprints, but not measured inference performance. Can the authors compare CLB and Loop in terms of end-to-end decoding latency and throughput under matched hardware and inference settings? This would clarify whether CLB’s smaller persistent KV cache translates into a practical advantage despite its worse OWT perplexity.
Q2. Regarding W1, can the authors provide matched-budget comparisons covering predictive feedback, loop-level mixing, and cache reuse—for example, PonderLM, Hyperloop Transformers, PonderLM-2, and the KV-sharing variant of Mixture-of-Recursions? Using the same training corpus, token budget, and comparable inference compute would help establish the incremental value of CLB’s bridge, clean-KV reuse, and probability mixing.
Q3. Regarding W2.1, an the authors train AR, Loop, and CLB on a larger corpus with substantially less repetition, using identical training batches, token budgets, and learning-rate schedules? A controlled experiment at one representative scale, would help establish whether the LM1B gains persist beyond the heavily repeated-data setting.
Q4. Regarding W2.3, can the authors provide controlled ablations of the final CLB architecture, particularly removing predictive feedback while retaining the second pass and mixing head, replacing learned mixing with fixed averaging or second-pass-only output, and replacing interpolation with residual addition? These controls would help distinguish the contribution of the bridge from extra computation and probability mixing.
3 comments
Response to Reviewer bjvJ- Part II
4. Measured inference and precision (Q1; W3)
For nominal 700M, the archived end-to-end generation records yield 56.9906 tokens/s for AR and 29.0620 for CLB, or 17.5468 and 34.4092 ms per emitted token. The protocol generates four continuations of 1,024 tokens per model in FP32 with TF32 disabled; elapsed time includes sampling, logging, and cache rebuilds. We do not have matched Loop timing, so we cannot claim that its larger persistent cache translates into lower CLB latency. We will separate persistent cache storage, peak memory, FLOPs, and elapsed time in the discussion.
We will standardize PPL to four decimals and reported percentage changes to four decimals in the detailed response tables, retaining full-precision values for calculations. The stronger empirical claim is currently limited to the reported likelihood improvements at a fixed backbone/cache footprint; matched-budget superiority and component-level causal attribution remain open.
References
Thank you for the detailed response. The additional 170K OWT result strengthens the evidence that CLB improves perplexity over AR, from 21.5820 to 21.1077 at the same update count. My main concerns remain unresolved.
Regarding W1, W2.2, Q1, and Q2, the architectural distinctions are clearer, but the rebuttal adds no matched-budget comparisons with related methods. On OWT, CLB has higher perplexity than the simpler Loop baseline in every reported head-to-head comparison (Table S4). The new 170K result includes no Loop baseline. I would expect CLB and Loop to have roughly comparable decoding latency given their comparable number of layer evaluations per token. The reported wall-clock measurements cover AR and CLB, leaving that expectation unverified and CLB’s latency advantage over Loop unestablished.
Regarding W2.1, given the planned removal of the LM1B-based claim, I focus on the OWT evidence. Assuming the extended runs used the reported batch size and sequence length, the 134M CLB model reached approximately 167 tokens per parameter (TPP) at 170K updates. The longer 812M runs reported in the authors’ replies to other reviewers bring CLB from roughly 11 TPP to 32–33 TPP. The ordering between CLB and AR reverses across nearby checkpoints under the authors' approximate compute-matching protocol. Separately, the 2.1B MoE transfer checkpoint discussed in the reply to Reviewer v1T8 received about 4.35 tokens per total parameter during its 70K MoE stage after dense initialization. This stage-specific estimate is not directly comparable to dense-model TPP. The additional results cover longer training budgets, but they still leave the OWT evidence insufficient to establish an advantage over Loop.
Regarding W2.3 and Q4, the auxiliary-loss sweep partially addresses my concern. Removing first-pass supervision substantially worsens first-pass perplexity, while the final mixture perplexity changes much less. The sweep retains predictive feedback, interpolation, and learned probability mixing across all configurations, so it leaves their individual contributions unresolved. The token-level analysis reported in the replies to other reviewers further shows that most of the cross-entropy improvement occurs where the top prediction stays the same. This describes where the improvement occurs, while its source remains unclear.
I appreciate the narrower claims and the decision to report the unfavorable Loop comparisons in the main text. The added evidence strengthens the comparison with AR at matched update counts, but it leaves the quality and latency comparisons with Loop unresolved. I therefore maintain my original score.
Response to Reviewer bjvJ- Part I
Thank you for the precise related-work suggestions and for recognizing the clean-KV design and the controlled LM1B comparison. We address positioning, empirical attribution, and presentation separately.
1. Relationship to predictive-feedback and looped models (W1; Q2)
We agree that probability-weighted embeddings and within-token repeated computation are not new in isolation. PonderLM already feeds probability-weighted token embeddings back through residual input updates, with a top-K approximation. Hyperloop uses hyper-connections across loop-level residual streams. PonderLM-2 uses hidden-state latent thoughts before predicting the next token, and Mixture-of-Recursions includes a first-recursion KV-sharing variant. We will cite and compare these mechanisms directly rather than imply that feedback or KV reuse alone establishes novelty. [1–4]
CLB's particular design combines an interpolation between the observed predecessor embedding and its soft predicted continuation, a shared second pass attending to clean historical KV plus its own current-position state, and a learned probability mixture for the same target. Its coordinate selects an input blend, rather than a loop depth or the time/step-size trajectory used by LoopFormer. [5] These are architectural distinctions; they are not evidence of superiority over those methods. No completed matched-budget PonderLM, Hyperloop, PonderLM-2, or MoR comparison is available in our verified records. Ouro, Think-at-Hard, and LoopMoE also belong in a broader discussion of adaptive and sparse iterative computation; we will expand that discussion without implying that our plain Loop control represents all of them.
2. Data regimes and the unfavorable Loop results (W2; Q3)
You are right that the different data regimes limit cross-scale conclusions. The LM1B 10B budget measures cumulative exposure to a roughly 0.894B-token corpus, approximately 11.2 repetitions. The OWT budgets likewise imply very different tokens-per-parameter ratios across scales. We will label unique corpus size, cumulative exposure, effective updates, and actual parameter counts separately; the nominal 700M backbone has approximately 812M parameters. These experiments were not a controlled scaling-law study, and we will not interpret them as establishing a trend across data regimes.
We will move the OWT Loop comparison into the main text: at 70K updates on 4,330 chunks, PPL is 22.5529 for Loop and 22.7734 for CLB. The LM1B test result, 27.6247 versus 27.4940, is a small single-seed improvement. We do not have the requested jointly controlled, substantially less-repeated-corpus AR/Loop/CLB experiment. The later OWT 130M same-update comparison does provide AR/CLB PPLs of 21.5820/21.1077 at 170K, but it lacks the matched Loop arm and is not a substitute for Q3.
3. Component attribution (W2; Q4)
We located a follow-up auxiliary-loss sweep of the shared-12-layer implementation. At 20K updates and the same fixed evaluation coordinate t=0.6, the following records separate the first-pass and mixture outputs. Beta denotes the coefficient of first-pass auxiliary CE:
The beta=1.0 row is a historical reference, not a new replicate. This limited, single-seed follow-up shows that the auxiliary objective affects the balance between the two outputs: removing it greatly weakens the first pass without producing the best mixture. It does not resolve the requested feedback-free, fixed-averaging, second-pass-only-training, or residual-addition controls. A post-hoc eight-sequence held-out gate diagnostic is also available, but it is too small and differs from a retrained mixing-head ablation; we do not use it as a substitute. Later MASK/LoRA/feature variants change several components and are likewise not clean ablations of the submitted model.