acceptodds
Under review as a conference paper at ICLR 2027

Continuous Language Bridge

Abstract

Model-native self-correction motivates making a tentative prediction available for reassessment before emission. The design challenge is to expose that prediction without committing an unobserved token to the context. We introduce Continuous Language Bridge (CLB), which connects the observed predecessor to a soft predicted continuation in embedding space. A shared second pass reassesses the same next-token target at one selected bridge state, and a categorical mixture produces the final prediction. The framework retains exact autoregressive likelihood. We validate the CLB across different benchmark datasets and also from dense to sparse MOE structures at different scales. Specifically, CLB beats the naive AR and looped model by 1.27% and 0.47% on LM1B dataset, respectively. Across seven external datasets, CLB reduces perplexity relative to AR by an average of 8.79%, 5.79%, and 13.76% at the 130M, 700M, and 2.1B scales, respectively, with the MoE transfer result measured at 70K updates. The project code will be open-sourced.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language generation model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bjvJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Continuous Language Bridge (CLB), a two-pass language modeling framework that refines next-token predictions before emission. The first pass produces a probability-weighted embedding of candidate tokens, which is interpolated with the current observed token’s embedding. A shared second pass processes this bridge state using first-pass historical keys and values, and a learned gate combines the two predictive distributions. The construction preserves autoregressive likelihood and the baseline’s persistent KV-cache footprint. The paper evaluates dense and MoE variants on LM1B, OpenWebText, and external corpora, reporting perplexity improvements over AR baselines.

Critical strengths

S1. Memory-efficient clean-KV reuse. CLB reuses first-pass historical keys and values while keeping second-pass states transient, maintaining the same persistent KV-cache footprint as the corresponding AR backbone. This offers a potentially useful trade-off between predictive quality and memory usage.

S2. Controlled comparisons on LM1B. AR, Loop, and CLB share initial backbone parameters, training batches, and cumulative token budgets. CLB achieves lower test perplexity in this setting, providing preliminary evidence of effectiveness, although the gains are modest and lack multi-seed validation.

S3. Detailed methodological description and transparent reporting. The paper specifies the bridge construction, attention mask, training objective, and probability mixture. The appendices also document negative results and limitations involving training budgets, initialization, and evaluation protocols.

Critical weaknesses

W1. Insufficient positioning and comparison against closely related work. The manuscript does not adequately establish CLB’s contribution relative to existing looped and predictive-feedback language models. Hyperloop Transformers explores loop-level residual-stream mixing (Zeitoun et al., 2026), while PonderLM already combines probability-weighted token embeddings, top-KK approximation, residual input updates, and repeated computation within a next-token decision (Zeng et al., 2026a). Soft Thinking, which the manuscript cites, provides another probability-weighted feedback mechanism, although its training-free reasoning setting differs from CLB’s pretraining setting (Zhang et al., 2025). Other relevant approaches include learned depth allocation in Ouro, per-token latent computation in PonderLM-2, and selective iteration in TaH (Zhu et al., 2025; Zeng et al., 2026b; Fu et al., 2026). The clean-KV design should also be distinguished from the KV-sharing variant of Mixture-of-Recursions (Bae et al., 2025b), while LoopFormer and LoopMoE provide relevant precedents for time- and step-conditioned looping and iteration-conditioned sparse computation (Jeddi et al., 2026; Chen et al., 2026). These approaches are not equivalent to CLB, but the manuscript needs a clearer account of its architectural distinctions and representative matched-budget comparisons beyond AR and a plain tied-layer Loop.

W2. The experimental presentation and controls do not clearly establish the source or robustness of the reported gains. Three issues are particularly important:

  • Training regimes differ substantially across settings. On LM1B, the 10B-token budget comes from sampling with replacement from a 0.894B-token corpus, yielding cumulative exposure approximately 11.2 times the corpus size (Appendix E.3). On OWT, the approximately 134M and 812M models receive around 21B and 9.18B tokens, respectively—approximately 157 versus 11 tokens per parameter (Appendices B.2 and C.1–C.2). These choices require clearer justification and limit conclusions about performance across data regimes and model scales.
  • The main presentation does not adequately foreground the mixed Loop comparisons. Loop outperforms CLB in every OWT comparison in Table S4, including 22.5529 versus 22.7734 PPL at 70K updates on the same 4,330 validation chunks. No matched 160K Loop result accompanies the main OWT endpoint. The 0.47% single-seed improvement on LM1B therefore does not establish a consistent advantage over repeated computation.
  • The final architecture lacks component-level attribution. CLB combines predictive feedback, interpolation, coordinate conditioning, gradient detachment, auxiliary supervision, and probability mixing. The historical controls do not isolate their contributions; the mixture-of-softmax experiment is explicitly not a gate-only ablation of the final model (Appendix D.3).

W3. Inconsistent numerical precision in the appendices. The appendix tables report comparable metrics with inconsistent numbers of decimal places, making the presentation less uniform and comparisons harder to follow. The authors should standardize the displayed precision for each metric across tables while retaining unrounded values for calculating differences and relative improvements.

References

  • Zeitoun, A., Torroba-Hennigen, L., and Kim, Y. (2026). Hyperloop Transformers. COLM 2026.
  • Zeng, B., Song, S., Huang, S., Wang, Y., Li, H., He, Z., Wang, X., Li, Z., and Lin, Z. (2026a). PonderLM: Pretraining Language Models to Ponder in Continuous Space. ICLR 2026.
  • Zhang, Z., He, X., Yan, W., Shen, A., Zhao, C., Wang, S., Shen, Y., and Wang, X. E. (2025). Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space. NeurIPS 2025.
  • Zhu, R.-J., et al. (2025). Scaling Latent Reasoning via Looped Language Models. arXiv preprint arXiv:2510.25741.
  • Zeng, B., Li, H., Song, S., Wang, Y., Wang, Z., He, Z., Wang, X., and Lin, Z. (2026b). PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space. ICML 2026.
  • Fu, T., You, Y., Chen, Z., Dai, G., Yang, H., and Wang, Y. (2026). Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning. ICML 2026.
  • Bae, S., Kim, Y., Bayat, R., Kim, S., Ha, J., Schuster, T., Fisch, A., Harutyunyan, H., Ji, Z., Courville, A., and Yun, S.-Y. (2025). Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. ICML 2025.
  • Jeddi, A., Ciccone, M., and Taati, B. (2026). LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. ICLR 2026.
  • Chen, W., Li, T., Huang, W., Yin, Y., Shang, L., and Qin, C. (2026). LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling. arXiv preprint arXiv:2606.04438.
Questions for authors

Q1. Table 1 reports analytical KV-cache footprints, but not measured inference performance. Can the authors compare CLB and Loop in terms of end-to-end decoding latency and throughput under matched hardware and inference settings? This would clarify whether CLB’s smaller persistent KV cache translates into a practical advantage despite its worse OWT perplexity.

Q2. Regarding W1, can the authors provide matched-budget comparisons covering predictive feedback, loop-level mixing, and cache reuse—for example, PonderLM, Hyperloop Transformers, PonderLM-2, and the KV-sharing variant of Mixture-of-Recursions? Using the same training corpus, token budget, and comparable inference compute would help establish the incremental value of CLB’s bridge, clean-KV reuse, and probability mixing.

Q3. Regarding W2.1, an the authors train AR, Loop, and CLB on a larger corpus with substantially less repetition, using identical training batches, token budgets, and learning-rate schedules? A controlled experiment at one representative scale, would help establish whether the LM1B gains persist beyond the heavily repeated-data setting.

Q4. Regarding W2.3, can the authors provide controlled ablations of the final CLB architecture, particularly removing predictive feedback while retaining the second pass and mixing head, replacing learned mixing with fixed averaging or second-pass-only output, and replacing interpolation with residual addition? These controls would help distinguish the contribution of the bridge from extra computation and probability mixing.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bjvJ- Part I

Thank you for the precise related-work suggestions and for recognizing the clean-KV design and the controlled LM1B comparison. We address positioning, empirical attribution, and presentation separately.

1. Relationship to predictive-feedback and looped models (W1; Q2)

We agree that probability-weighted embeddings and within-token repeated computation are not new in isolation. PonderLM already feeds probability-weighted token embeddings back through residual input updates, with a top-K approximation. Hyperloop uses hyper-connections across loop-level residual streams. PonderLM-2 uses hidden-state latent thoughts before predicting the next token, and Mixture-of-Recursions includes a first-recursion KV-sharing variant. We will cite and compare these mechanisms directly rather than imply that feedback or KV reuse alone establishes novelty. [1–4]

CLB's particular design combines an interpolation between the observed predecessor embedding and its soft predicted continuation, a shared second pass attending to clean historical KV plus its own current-position state, and a learned probability mixture for the same target. Its coordinate selects an input blend, rather than a loop depth or the time/step-size trajectory used by LoopFormer. [5] These are architectural distinctions; they are not evidence of superiority over those methods. No completed matched-budget PonderLM, Hyperloop, PonderLM-2, or MoR comparison is available in our verified records. Ouro, Think-at-Hard, and LoopMoE also belong in a broader discussion of adaptive and sparse iterative computation; we will expand that discussion without implying that our plain Loop control represents all of them.

2. Data regimes and the unfavorable Loop results (W2; Q3)

You are right that the different data regimes limit cross-scale conclusions. The LM1B 10B budget measures cumulative exposure to a roughly 0.894B-token corpus, approximately 11.2 repetitions. The OWT budgets likewise imply very different tokens-per-parameter ratios across scales. We will label unique corpus size, cumulative exposure, effective updates, and actual parameter counts separately; the nominal 700M backbone has approximately 812M parameters. These experiments were not a controlled scaling-law study, and we will not interpret them as establishing a trend across data regimes.

We will move the OWT Loop comparison into the main text: at 70K updates on 4,330 chunks, PPL is 22.5529 for Loop and 22.7734 for CLB. The LM1B test result, 27.6247 versus 27.4940, is a small single-seed improvement. We do not have the requested jointly controlled, substantially less-repeated-corpus AR/Loop/CLB experiment. The later OWT 130M same-update comparison does provide AR/CLB PPLs of 21.5820/21.1077 at 170K, but it lacks the matched Loop arm and is not a substitute for Q3.

3. Component attribution (W2; Q4)

We located a follow-up auxiliary-loss sweep of the shared-12-layer implementation. At 20K updates and the same fixed evaluation coordinate t=0.6, the following records separate the first-pass and mixture outputs. Beta denotes the coefficient of first-pass auxiliary CE:

BetaFirst-pass PPLMixture PPL
0.040.142627.2100
0.229.117626.9330
0.328.634926.9705
0.528.307027.0600
1.027.963927.0934

The beta=1.0 row is a historical reference, not a new replicate. This limited, single-seed follow-up shows that the auxiliary objective affects the balance between the two outputs: removing it greatly weakens the first pass without producing the best mixture. It does not resolve the requested feedback-free, fixed-averaging, second-pass-only-training, or residual-addition controls. A post-hoc eight-sequence held-out gate diagnostic is also available, but it is too small and differs from a retrained mixing-head ablation; we do not use it as a substitute. Later MASK/LoRA/feature variants change several components and are likewise not clean ablations of the submitted model.

AuthorsOpenReview ↗

Response to Reviewer bjvJ- Part II

4. Measured inference and precision (Q1; W3)

For nominal 700M, the archived end-to-end generation records yield 56.9906 tokens/s for AR and 29.0620 for CLB, or 17.5468 and 34.4092 ms per emitted token. The protocol generates four continuations of 1,024 tokens per model in FP32 with TF32 disabled; elapsed time includes sampling, logging, and cache rebuilds. We do not have matched Loop timing, so we cannot claim that its larger persistent cache translates into lower CLB latency. We will separate persistent cache storage, peak memory, FLOPs, and elapsed time in the discussion.

We will standardize PPL to four decimals and reported percentage changes to four decimals in the detailed response tables, retaining full-precision values for calculations. The stronger empirical claim is currently limited to the reported likelihood improvements at a fixed backbone/cache footprint; matched-budget superiority and component-level causal attribution remain open.

References

  1. PonderLM
  2. Hyperloop Transformers
  3. PonderLM-2
  4. Mixture-of-Recursions
  5. LoopFormer

Thank you for the detailed response. The additional 170K OWT result strengthens the evidence that CLB improves perplexity over AR, from 21.5820 to 21.1077 at the same update count. My main concerns remain unresolved.

Regarding W1, W2.2, Q1, and Q2, the architectural distinctions are clearer, but the rebuttal adds no matched-budget comparisons with related methods. On OWT, CLB has higher perplexity than the simpler Loop baseline in every reported head-to-head comparison (Table S4). The new 170K result includes no Loop baseline. I would expect CLB and Loop to have roughly comparable decoding latency given their comparable number of layer evaluations per token. The reported wall-clock measurements cover AR and CLB, leaving that expectation unverified and CLB’s latency advantage over Loop unestablished.

Regarding W2.1, given the planned removal of the LM1B-based claim, I focus on the OWT evidence. Assuming the extended runs used the reported batch size and sequence length, the 134M CLB model reached approximately 167 tokens per parameter (TPP) at 170K updates. The longer 812M runs reported in the authors’ replies to other reviewers bring CLB from roughly 11 TPP to 32–33 TPP. The ordering between CLB and AR reverses across nearby checkpoints under the authors' approximate compute-matching protocol. Separately, the 2.1B MoE transfer checkpoint discussed in the reply to Reviewer v1T8 received about 4.35 tokens per total parameter during its 70K MoE stage after dense initialization. This stage-specific estimate is not directly comparable to dense-model TPP. The additional results cover longer training budgets, but they still leave the OWT evidence insufficient to establish an advantage over Loop.

Regarding W2.3 and Q4, the auxiliary-loss sweep partially addresses my concern. Removing first-pass supervision substantially worsens first-pass perplexity, while the final mixture perplexity changes much less. The sweep retains predictive feedback, interpolation, and learned probability mixing across all configurations, so it leaves their individual contributions unresolved. The token-level analysis reported in the replies to other reviewers further shows that most of the cross-entropy improvement occurs where the top prediction stays the same. This describes where the improvement occurs, while its source remains unclear.

I appreciate the narrower claims and the decision to report the unfavorable Loop comparisons in the main text. The added evidence strengthens the comparison with AR at matched update counts, but it leaves the quality and latency comparisons with Loop unresolved. I therefore maintain my original score.

Reviewer_uiVvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Continuous Language Bridge (CLB) to improve next-token prediction by enabling autoregressive language models to reconsider their initial predictions before token emission. CLB constructs a continuous embedding bridge between the observed context and the initial prediction, performs a second forward pass for prediction refinement, and combines the two predictions through a learned gate. Experiments across different model scales and architectures show consistent perplexity reductions over autoregressive baselines, along with modest improvements on downstream tasks.

Critical strengths
  1. An interesting direction for prediction-conditioned feedback. The paper explores incorporating the model's initial prediction into a second forward pass to refine the same next-token distribution before token emission. This provides an interesting perspective on incorporating additional computation into autoregressive language modeling.
  2. Compatibility with autoregressive modeling. CLB preserves exact autoregressive likelihood while reusing the clean-context KV cache. The two forward passes share Transformer parameters, avoiding the need for a separate verification network or additional persistent KV storage.
  3. Evaluation across multiple scales and architectures. The authors evaluate CLB on both dense and MoE architectures, covering different model scales and multiple external datasets. The reported results show consistent perplexity improvements over the corresponding autoregressive baselines.
Critical weaknesses
  1. Insufficient connection between the motivation and the proposed method. The paper motivates CLB through model-native self-correction and continuous verification. However, the proposed second forward pass does not explicitly assess the correctness of the initial prediction; instead, it processes a transformed representation derived from the first-pass prediction and produces another distribution for the same target. Since the soft predicted embedding is a deterministic transformation of the first-pass output, it does not introduce additional information beyond the observed context.
  2. Insufficient analysis of the sources of performance improvements. Although additional computation can improve predictive performance, the current evidence does not clearly establish whether the gains arise from meaningful prediction refinement, additional computation, or the complementary predictions combined by the mixture gate. Although Appendix H.4 and H.5 reports first-pass versus mixture comparisons, these results do not sufficiently isolate the contribution of the second-pass prediction or demonstrate its effectiveness in correcting initial predictions.
  3. Limited performance gains relative to the additional computational overhead. CLB requires an additional Transformer forward pass for each generated token, introducing substantial computation compared with standard autoregressive decoding. On LM1B, CLB reduces perplexity from 27.85 to 27.49 compared with AR, but only from 27.62 to 27.49 compared with the Loop baseline. Furthermore, the average accuracy improvements on five downstream multiple-choice benchmarks are only 0.61 and 0.83 percentage points at the 130M and 700M scales, respectively. While these results demonstrate improvements in predictive quality, the paper does not provide compute-matched comparisons or sufficient inference-efficiency measurements to establish whether these gains justify the additional computational cost.
  4. Limited evidence for reproducibility. The authors explicitly acknowledge limitations in reproducibility, stating that "Full reproduction additionally requires anonymized code, configurations, data indices, checkpoint metadata, and original logs; their incomplete availability is identified in the appendices."
Questions for authors

How does CLB perform under matched computational budgets? Given the additional Transformer forward pass required for each token, could the authors report training FLOPs, inference FLOPs, decoding latency, and throughput? More importantly, could the authors provide FLOPs-matched comparisons between CLB and baselines to evaluate whether the reported performance gains persist under equivalent computational budgets?

1 comment
AuthorsOpenReview ↗

Response to Reviewer uiVv

Thank you for recognizing the prediction-feedback direction and its compatibility with autoregressive modeling. Your questions motivated a more explicit separation of predictive quality, mechanism, and computational cost.

1. Motivation and what is actually refined

We agree that the soft draft is a deterministic function of the observed prefix and adds no new external information. Its role is computational: it exposes the model's current predictive representation as an input to another application of the shared backbone. We will describe this as prediction-conditioned refinement. In particular, the second pass is not an explicit correctness test, and lower mixture loss does not imply that its standalone distribution is uniformly better.

A paired token analysis of a follow-up 130M model at 170K updates makes this distinction concrete. On 2,046,000 targets, the final mixture changes 13,312 wrong top-1 predictions to correct ones, but also changes 11,824 correct predictions to wrong ones. The net gain is +0.0727 percentage points over its own first pass. Moreover, 88.5130% of the net CE improvement comes from positions whose top-1 token does not change. This supports improvement in probability assignment with modest net correction. Because it does not remove either the feedback input or the mixing head, it is a descriptive mechanism diagnostic, not a causal ablation.

2. Matched updates versus matched computation

A separate same-backend, same-update follow-up comparison at 130M uses 170K updates, the same 2,000 OWT validation sequences, and FP32 weights with BF16 autocast. The results are:

ReadoutPPLTop-1 accuracy
AR21.582041.7700%
CLB first pass21.718141.8412%
CLB mixture21.107741.9243%

The final CLB output improves PPL by 2.1977% and accuracy by 0.1543 percentage points over AR. Its first-pass PPL is slightly worse than AR, so we agree that first-to-mixture improvement alone overstates the net benefit. These are follow-up NPU measurements, separate from the CUDA correction counts above. Equal updates also do not mean equal FLOPs.

For the stronger compute question, the existing extended-training comparison uses a two-AR-updates-per-CLB-update approximation. At nominal 700M, CLB/AR PPL is 15.1178/15.1392 at 200K/400K updates, but 15.1268/15.0172 at 205K/410K; the sign changes. The 130M 220K/440K comparison favors AR, 20.8296/20.5446. These records do not establish a robust compute advantage. CLB has two backbone passes per prediction, but that alone does not supply exact training/inference FLOP counts because the attention implementation, draft, heads, and backward paths matter. We do not currently have audited FLOP totals or an exact FLOPs-matched experiment.

3. Inference measurements and the memory claim

The saved nominal-700M generation experiment gives 56.9906 tokens/s for AR and 29.0620 for CLB, corresponding to 17.5468 and 34.4092 ms per emitted token. These are aggregate end-to-end times for four prompts of 1,024 generated tokens each, using FP32, TF32 disabled, temperature 0.8, and top-p 0.95. They include sampling, logging, and context rollover/cache rebuilds. They should not be read as isolated decoding latency or a throughput benchmark over varying batch sizes; no matched Loop timing is available. The architectural advantage we can claim is retaining one persistent clean-context KV cache despite the extra pass. It does not imply faster generation or unchanged peak working memory.

We will put the mixed Loop results alongside this trade-off: CLB improves on Loop by 0.1307 PPL in the single-seed LM1B test, while Loop is better on the recorded OWT comparisons. The present evidence supports a likelihood–persistent-cache trade-off, not a universal advantage under an equivalent compute budget.

4. Reproducibility and revisions

The submitted supplementary archive contains evaluation records, protocol descriptions, sample/checkpoint identifiers, and table-to-artifact mappings; the additional tables here are recomputed from saved records. These support numerical checking but are not a complete runnable reproduction package. We will make that distinction explicit. The remaining requirements include executable training/evaluation code and environments, full data-index and resume provenance, and multi-seed validation. We will also foreground the unfavorable controls and qualify the modest downstream gains. We appreciate the request for stronger controls and do not present uncompleted experiments as resolving it.

Reviewer_v1T8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper makes a second pass through a decoder LM prior to each token being emitted. The input to that second pass is a interpolation at coordinate t between the current token's embedding and the probability weighted mean embedding of the top-K candidates from the first pass. The second pass attends to the clean KV cache of earlier positions and a gate derived from the first pass hidden state combines the two next token distributions (Eq 4). The experiments cover LM1B at 130M params and 10B tokens (vs AR, a tied layer Loop and MDLM), OWT at 130M and nominal 700M and a 2.1B MoE upcycled from the 700M models. CLB beats AR by 1.27% and Loop by 0.47% on LM1B, beats AR by roughly 1.7-2% in-domain on OWT and beats AR by 5.79%-13.76% on average across seven external corpora. A mixture risk bound (Theorem 1) is provided as theoretical backing.

Critical strengths
  1. The LM1B comparison is set up properly. All arms see the same global batch at each update index (t=0.6 is fixed in advance) and the test blocks are disjoint from the validation blocks used for the coordinate scans. It has the correct control too: a tied layer Loop which uses the same amount of extra compute with no feedback.

  2. The appendix is remarkably candid. It explicitly calls out unverified training budgets, mismatched hardware, a fixed coordinate MoE run which completely imploded at 40K and a number of results which were actually against the method.

  3. Re-using the clean KV cache is a good way of keeping the persistent KV memory equal to AR's (half of Loop's). The second pass also beats its own first pass in all 144 model and corpus readouts in H.4.

Critical weaknesses
  1. The primary concern I have is that the improvements appear to be due to additional computing power rather than to feedback. While CLB runs the backbone twice per token, AR does so only once, and Loop is the only baseline that matches this approach. On LM1B, Loop by itself achieves approximately 63% of the improvement that CLB obtains over AR (the score goes from 27.85 to 27.62, compared to 27.49 for CLB), which amounts to a 0.13 PPL reduction on a single seed. On OWT the situation is reversed. Table S4 shows that the 130M version of Loop outperforms CLB in all four comparisons (22.38 versus 22.61 at 70K, a difference of about 1%), and this is never mentioned in the main text. There is no baseline that has been matched in terms of computing power at either 700M or 2.1B. The introduction presents the work in the context of scaling laws, so the comparison I would like to see is AR against another model using the same training or inference FLOPs, and this comparison is not included in the paper.

  2. Beyond LM1B the comparisons are not controlled, as the appendix makes clear. It is impossible to verify the budget of the 130M AR checkpoint from its resume metadata (H.2). The 700M AR and CLB runs used different GPUs and different versions of PyTorch, and the CLB run was resumed during training (C.2). The MoE-AR and MoE-CLB models were derived from different dense checkpoints. Each configuration uses only one seed. Appendix C.2 states that these runs provide "a scale-specific likelihood comparison, not a fully isolated architectural or equal-compute effect".

  3. The average results across different corpora are very fragile. Within the domain, CLB is 1.7 to 2% better, but the average figures from outside the domain rise to 13.76%. With a model size of 2.1B, about three quarters of the total reduction come from LAMBADA, PubMed and Arxiv (each contributing 22 to 28%). It is precisely in these corpora that MoE-AR at 70K performs worse than the 700M dense AR it originally came from (for example, PubMed goes from 40.32 to 49.41, as shown in Tables S14 and S15). Therefore, a large part of the 13.76% figure for MoE-AR is due to its degradation when used out of domain. The 130M AR baseline is also unusual since Loop does worse than it when evaluated in-domain (22.38 compared to 21.64) yet still outperforms it by about 5% on both PubMed and Arxiv. My best guess is that averaging two distributions has the effect of smoothing the predictions, which proves beneficial in cases of distribution shift, and the differences in checkpoint budgets account for the rest. Neither of these explanations is the mechanism that the paper suggests. The models trained on LM1B show the same trend: on the external datasets without a separator, the range CLB is worse than AR across all eight collections while Loop beats both of them (as indicated in Table S16).

  4. I have no idea what contribution the paper itself makes. As shown in Table S13, when training is carried out at t = 0.1 (mostly draft), at t = 0.6, and at t = 0.9 (mostly observed token), the OWT PPL values obtained are 22.61, 22.50 and 22.58 respectively. Since the difference is less than half a percent on a single seed, the amount of draft involved in the bridge is of little importance. The missing ablation refers to a trained second pass that sees only the observed token embedding, using the same clean-KV mask and gate; it is not the same thing to evaluate a model that has been trained over a range at t = 1. Furthermore, Table S19 demonstrates that part of the within-checkpoint improvement simply compensates for the weaker first pass. At 130M, the first pass is 0.013 nats worse than independent AR, so the net gain over AR is only about half of the within-checkpoint gain.

The ground-truth injection control (Fig. 3d) doesn't help here. Training with the target in the input and evaluating without it is a train/test mismatch, and the 56 to 1160 PPL results show that and little else. The Fig. 3 caption says these controls "do not establish a general limitation", while Section 2 says they show GT-based blending "actually cannot help the language generation". The caption is right.

  1. The theory is general. Theorem 1 states that incorporating a second fixed predictor will reduce log loss if, on average, it is beneficial for some gate inputs and the learned gate is within ε of the best gate, which has been capped at 1/2. The theory contains no reference to the bridge, and Appendix A.1 mentions that "it also applies to other feedback predictors", meaning that it would equally apply to Loop with a gate. The gate condition is assumed rather than proven, and D, S, and ε are never estimated. Since 'exact autoregressive likelihood' is valid for any normalized readout of the prefix, I wouldn't regard it as a significant contribution. As for a smaller point, a probability floor of 1e-12 is used with mixture readouts, and B.1 states that the frequency with which it activates wasn't measured. The only effect it can have is to lower the loss as reported by CLB, so it's worth investigating.

  2. The efficiency and the claims made afterwards are exaggerated. While Section 4.2 uses the fact that the KV footprint is the same as proof of the model's 'resource-saving' features, CLB manages to double the number of FLOPs per token and, since the second pass has to wait for the distribution from the first pass, the per-token decoding latency should also increase by about a factor of two. There are no figures given regarding throughput or wall clock time. The improvements in the multiple-choice tests (+0.61 and +0.83 points) are minor considering that the tasks involve between 1,172 and 10,042 examples with ARC-C being close to chance, and at 130M the gain becomes –0.08 points when assessed using only continuation scoring (J.1).

Questions for authors
  1. How do you reconcile the OWT Loop result in Table S4 with LM1B, and why isn't it in the main text?

  2. On the external corpora, what are the first-pass and mixture PPL separately, and how does the average gate weight Πi,2\Pi_{i,2} compare with OWT? Separately, why is MoE-AR at 70K worse than its own 700M initialization on LAMBADA, PubMed and Arxiv?

  3. Did you ever train a variant whose second pass gets only the observed token embedding, with no draft?

  4. What are decode throughput and per-token latency for AR, Loop and CLB at 700M?

3 comments
AuthorsOpenReview ↗

Response to Reviewer v1T8 — Part 1

Thank you for the detailed assessment and for recognizing the controlled LM1B protocol and clean-KV design. We agree that the extra computation, the contribution of the predictive input, and the final probability mixture need to be distinguished. Our follow-up records allow us to sharpen that distinction.

1. Loop comparisons and computational budgets (W1; Q1)

The Loop comparison is mixed. On LM1B, the final test PPLs are 27.8477 (AR), 27.6247 (Loop), and 27.4940 (CLB). At 70K OWT updates on all 4,330 validation chunks, Loop is better: 22.5529 versus 22.7734 for CLB. We will present both outcomes in the main text. We do not have evidence identifying why the ordering reverses across these settings, or a matched 160K OWT Loop endpoint; we will not describe CLB as consistently superior to repeated computation.

We also examined later checkpoints using the approximation that one CLB update costs two AR updates. At nominal 700M, CLB/AR PPLs are 15.1874/15.2550 at 190K/380K updates, 15.1178/15.1392 at 200K/400K, and 15.1268/15.0172 at 205K/410K. Thus, the ordering changes across adjacent checkpoints. At 130M, the follow-up 220K/440K pair gives 20.8296/20.5446, favoring AR. These are approximate step-budget comparisons, not measured FLOPs matches: draft construction, attention, output heads, hardware/resume differences, and validation-selected coordinates remain. They do not establish a general compute advantage.

Regarding W2, we will distinguish the controlled LM1B comparison from the historical OWT results. The submitted 130M AR cumulative budget is not fully reconstructable, the nominal-700M runs differ in hardware/software and resume history, and the MoE arms use different initial dense checkpoints. The later diagnostics do not retroactively remove these limitations or provide multi-seed validation.

2. What changes in the prediction? (W3–W4)

We agree that the submitted within-checkpoint result needs precise interpretation: the 144 favorable readouts compare the final mixture with its own first pass; they do not show that the second-pass distribution alone is better. The second component is trained through the mixture likelihood and can specialize as a complementary predictor.

A follow-up 130M checkpoint at 170K updates provides a direct diagnostic on 2,046,000 OWT target positions, using t=0.55 selected previously by PPL. In the CUDA evaluation, the mixture corrects 13,312 initially incorrect top-1 predictions and reverses 11,824 correct predictions, a net gain of 1,488 positions (+0.0727 percentage points). Top-1 is unchanged on 95.5080% of positions, which account for 88.5130% of the net CE improvement. Mean second-pass weight is 8.0498%. These observations support a distributional improvement with modest net token correction; they do not establish an explicit correctness detector or isolate feedback from mixing. They are descriptive measurements of a follow-up checkpoint, not an independent-seed replication.

A separate same-backend NPU evaluation at 170K updates gives AR PPL/accuracy of 21.5820/41.7700% and CLB mixture PPL/accuracy of 21.1077/41.9243%. The same-update gains are 2.1977% in PPL and 0.1543 percentage points in accuracy. This comparison still gives CLB more computation. We keep these NPU counts separate from the CUDA token decomposition because BF16 backend differences slightly change top-1 counts.

3. Missing feedback-free control (W4; Q3)

We do not have a completed, verified training run that removes only the draft while retaining the submitted model's observed-token input, clean-KV mask, conditioning, and mixing head. Evaluating a range-trained checkpoint at t=1 is not that control. Our later MASK, feature-feedback, and LoRA variants change other components and therefore cannot fill this gap. We will explicitly distinguish the observed benefit of the full model from a causal attribution to predictive feedback alone.

AuthorsOpenReview ↗

Response to Reviewer v1T8 — Part 2: transfer, efficiency, and numerical checks

4. External-corpus readouts and MoE degradation (W3; Q2)

The table below separates independent AR, CLB's first pass, and its final mixture for the 70K MoE transfer checkpoint, at the prespecified t=0.6. The accompanying evidence tables give the corresponding 130M and 700M results, including OWT. These comparisons match evaluation inputs, not training initialization or compute.

CorpusAR PPLCLB firstCLB mixture
PTB70.164771.594464.2651
WikiText23.467222.572920.8514
LM1B48.306048.724248.1104
LAMBADA60.011350.966646.1411
AG News42.614744.040641.1233
PubMed49.410444.153435.8034
Arxiv48.090044.948237.4018

You are correct that the MoE-AR comparator deteriorates relative to its dense AR initialization on LAMBADA, PubMed, and Arxiv. The respective dense-AR to MoE-AR PPLs are 47.2346 to 60.0113, 40.3193 to 49.4104, and 42.7219 to 48.0900. We have not isolated the cause. The two MoE arms also start from different dense checkpoints, so the 13.7623% mean transfer reduction cannot be attributed solely to the CLB architecture. As a sensitivity analysis, excluding these three corpora reduces the mean relative improvement to 5.8648% over the remaining four. We will report this alongside the full per-corpus results and retain the unfavorable LM1B-trained transfer results.

The archived final-checkpoint transfer records contain first-pass CE and mixture CE, but do not provide the per-corpus mean gate weights needed for the requested OWT-versus-external comparison. The 8.0498% OWT weight in Part 1 concerns a different checkpoint and cannot substitute for those measurements. Distribution smoothing/calibration is a plausible explanation consistent with the token analysis, rather than an alternative that the present evidence rules out.

5. Measured generation cost (W6; Q4)

From the saved nominal-700M generation records, four prompts with 1,024 new tokens each take 71.8715 s for AR and 140.9400 s for CLB. This corresponds to 56.9906 versus 29.0620 tokens/s, or 17.5468 versus 34.4092 ms per emitted token. Both use the same generation protocol: FP32 with TF32 disabled, temperature 0.8, top-p 0.95, and context rollover at 1,023 observed tokens. These elapsed times include sampling, logging, and cache rebuilds; they are amortized end-to-end costs, not isolated decode-kernel latency. We do not have matched Loop timing or repeated timing trials. We will state the benefit specifically as an unchanged persistent KV cache, while making the added computation and observed slowdown explicit; this is not a claim about lower total peak memory or faster generation.

6. Probability floor, theory, and claim scope (W5–W6)

The 170K follow-up token diagnostic records zero activations of the 1e-12 mixture-probability floor across all 2,046,000 scored positions. Thus, floor activation does not explain the measured improvement for that checkpoint and evaluation. This does not certify the floor incidence of every manuscript checkpoint or external corpus; the corresponding full audit remains outstanding.

We will frame Theorem 1 as a conditional statement about mixture risk, applicable beyond CLB. It does not establish that CLB's learned gate satisfies the assumed risk comparison, and the relevant moments and gate error have not been estimated. Exact autoregressive factorization is a compatibility property of the prefix-only normalized readout. We will also remove the general claim that GT-based blending cannot help generation: the reported injection control demonstrates a failure under its particular training–evaluation mismatch. Finally, the small multiple-choice gains, including the unfavorable 130M continuation-only result, will be described as modest and scoring-dependent, without a general reasoning or statistical-significance claim.

Thank you for the rebuttal. The new evidence reinforces my core concerns rather than resolving them:

  • W1, Q1: Under the 2×2\times-update compute proxy, plain AR outperforms CLB at 130M (21.0251 vs. 21.4220 at 270K/135K; 20.5446 vs. 20.8296 at 440K/220K) and at 700M (15.0172 vs. 15.1268 at 410K/205K), while running nearly 2×2\times faster at decoding (56.99 vs. 29.06 tokens/s, with no Loop timing to show a practical KV-cache speedup). Meanwhile, Loop still beats CLB on OWT (22.5529 vs. 22.7734).
  • W3, W4: The 170K token-level audit shows nearly symmetric flips (13,312 wrong$\to$correct vs. 11,824 correct$\to$wrong, net +0.073 pp accuracy), with 88.51% of the CE gain coming from unchanged top-1 positions and an 8.05% mean gate weight. Combined with the missing no-draft ablation (Q3) and weaker first-pass external PPLs (Table A), this points to probability smoothing rather than meaningful prediction feedback.

Given the lack of a causal no-draft control and the unfavorable compute-matched and OWT Loop results, I will maintain my overall rating of 2.

Comments on the paper

3 comments
AuthorsOpenReview ↗

## Overall response

We thank the reviewers for identifying the central question: which benefits come from prediction-conditioned feedback, and which come from additional computation or probability mixing? Our contribution is a specific within-token architecture that combines an observed-to-predicted embedding bridge, a shared second pass using clean historical KV states, and a learned categorical readout. The evidence supports improved likelihood in the reported settings while retaining the AR backbone's persistent KV-cache size. We will distinguish this result from stronger claims about compute efficiency or reliable token-level self-correction.

The responses below add evidence from our existing follow-up experiments: a same-update 130M comparison; a paired token-level analysis of corrections and probability changes; extended-training comparisons using an explicitly approximate compute proxy; and measured end-to-end generation times. We also provide first-pass and mixture perplexities separately for all three OWT-trained scales. We identify follow-up checkpoints explicitly and retain the unfavorable comparisons. In particular, we will move the mixed Loop results into the main text and narrow the theoretical and efficiency claims to what the experiments establish.

AuthorsOpenReview ↗

Supporting numerical tables- Part I

These tables separate the submitted-checkpoint transfer evaluation from later diagnostic runs. Perplexities are shown to four decimals; all changes are calculated from unrounded records. A repeated evaluation across datasets is not an independent training seed.

A. 130M: independent AR, CLB first pass, and mixture

CorpusAR PPLCLB firstCLB mixtureReduction %
OWT21.641521.933021.28211.6609
PTB109.1370117.9495101.13417.3330
WikiText32.724833.375630.57576.5672
LM1B64.139864.385663.49721.0019
LAMBADA59.775560.127255.58157.0163
AG News77.305780.122272.94485.6411
PubMed58.008162.479947.273218.5059
Arxiv55.766062.641847.140815.4667

A. 700M: independent AR, CLB first pass, and mixture

CorpusAR PPLCLB firstCLB mixtureReduction %
OWT16.988017.083416.65681.9496
PTB79.768183.400573.55437.7899
WikiText25.128524.358923.00628.4458
LM1B52.013952.175651.64900.7015
LAMBADA47.234646.588344.61395.5482
AG News50.771152.907649.07813.3345
PubMed40.319342.274938.12475.4431
Arxiv42.721942.572038.75699.2809

A. 2.1B: independent AR, CLB first pass, and mixture

CorpusAR PPLCLB firstCLB mixtureReduction %
OWT15.154115.379514.97821.1606
PTB70.164771.594464.26518.4082
WikiText23.467222.572920.851411.1467
LM1B48.306048.724248.11040.4048
LAMBADA60.011350.966646.141123.1126
AG News42.614744.040641.12333.4996
PubMed49.410444.153435.803427.5388
Arxiv48.090044.948237.401822.2254

All entries above use the same scored inputs within each corpus/model pair, context length 1,024, observed prefix 1, and prespecified CLB t=0.6. First-pass PPL is exp(saved first-pass CE). The native mixture evaluator uses its recorded 1e-12 target-probability floor. The 130M AR archive records 80K but does not establish its full cumulative training exposure; it must not be relabeled as verified 160K. The nominal-700M CLB checkpoint records 35K after a resume and is labeled 70K effective updates in the manuscript protocol. The MoE transfer pair is 70K, distinct from the 60K in-domain table in the manuscript. These are checkpoint comparisons, not equal-training-budget causal estimates.

B. Approximate compute comparisons from extended training

Each row pairs a recorded CLB checkpoint with twice as many AR updates. This is a compute proxy only. Positive change means CLB PPL is worse. All rows use 2,000 validation sequences of length 1,024; CLB uses validation-selected coordinates. No interpolation is used. The full recovered October 19 comparison series is retained, including unfavorable rows.

ScaleCLB updatesAR updatesCLB PPLAR PPLChange %
130M13500027000021.422021.0251+1.8877
700M20000040000015.117815.1392-0.1412

A later 130M pair is CLB 220K: 20.8296 PPL and AR 440K: 20.5446 PPL, a +1.3874% change. This is a later NPU follow-up and has the same approximate-compute limitation. Neither this table nor the small nominal-700M differences demonstrates exact equal-FLOPs superiority.

C. Generation timing

ModelTokensElapsed sTokens/sAmortized ms/token
ar409671.871556.990617.5468
bridge4096140.940029.062034.4092

Aggregate over four prompts per model, 1,024 emitted tokens per prompt; FP32, TF32 disabled, temperature 0.8, top-p 0.95, EOS disabled, and maximum real context 1,023. CLB uses two passes, t=0.6, top-10 draft embeddings, and the learned probability mixture. These saved generation timings include sampling/logging and context rollover/cache reconstruction. They do not supply repeated timing uncertainty, isolated per-step latency, or loop throughput.

AuthorsOpenReview ↗

Supporting numerical tables- Part II

D. What the 170K token diagnostic establishes

QuantityMeasured result
Scored positions2,046,000
Wrong to correct13,312
Correct to wrong11,824
Net additional correct predictions1,488
Within-model accuracy improvement0.0727 percentage points
Unchanged top-1 positions95.5080%
Share of net CE gain at unchanged top-1 positions88.5130%
Mean second-pass gate weight8.0498%
Mixture probability-floor activations0

This is a later 130M CLB checkpoint at 170K updates, CUDA BF16-autocast evaluation, t=0.55. It is not the paper's 130M final checkpoint. The diagnostic compares its mixture with its own first pass. The independently trained AR checkpoint in the same CUDA archive is at 240K and is not update-matched; the update-matched 170K AR/CLB numbers in the replies come from a separate NPU evaluation. Floor incidence is established only for this diagnostic, not every headline corpus/checkpoint.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.