Continuous Language Bridge

Model-native self-correction motivates making a tentative prediction available for reassessment before emission. The design challenge is to expose that prediction without committing an unobserved token to the context. We introduce Continuous Language Bridge (CLB), which connects the observed predecessor to a soft predicted continuation in embedding space. A shared second pass reassesses the same next-token target at one selected bridge state, and a categorical mixture produces the final prediction. The framework retains exact autoregressive likelihood. We validate the CLB across different benchmark datasets and also from dense to sparse MOE structures at different scales. Specifically, CLB beats the naive AR and looped model by 1.27% and 0.47% on LM1B dataset, respectively. Across seven external datasets, CLB reduces perplexity relative to AR by an average of 8.79%, 5.79%, and 13.76% at the 130M, 700M, and 2.1B scales, respectively, with the MoE transfer result measured at 70K updates. The project code will be open-sourced.