- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Authors introduce ConvergeFlow. This method is a continuous language model that transforms noise into token embeddings without a separately trained decoder. It uses MSE training and a structured predictor, which the authors claim converges to valid token embeddings under specific assumptions. The paper also introduces sampling methods to control quality and diversity.
Critical strengths
Controlled evidence for MSE fine-tuning. The comparison starts from the same LangFlow checkpoint and includes a crossover experiment. This provides stronger evidence for the proposed objective than benchmark comparisons alone and is the most convincing empirical contribution.
Critical weaknesses
- Architectural novelty over LangFlow is not substantiated. LangFlow already adds the Gaussian log-likelihood to its neural logits (Appendix C.1, equation 44). Let denote the logits before this bias and define After warmup, the bias coefficient equals one, giving
Questions for authors
-
Parameterization and novelty. Do the authors agree that (15)–(16) are mathematically equivalent to LangFlow’s Gaussian-biased logits and embedding barycenter after warmup? If not, what specific constraint or implementation detail breaks this equivalence?
-
Validity of the convergence proof. How is the reduction in Appendix D.1, lines 1003–1004, justified when the auxiliary weights in (33) depend on the random initial state? In particular, what argument permits the fixed-weight density and Jacobian calculations in (40)–(41) to apply to the original process?
5 comments
Official Comment by Reviewer iq5y
Thank you for clarifying the connection to LangFlow, expanding the sampling section, and revising the proof in Appendix D.1.
But I think, my main concern, raised in Weakness 1, is not fully resolved. The theory and the training change still seem disconnected. Theorem 1 aims to establish convergence of the sampling trajectory for a parameterization shared with LangFlow. In practice, the main change to training is replacing CE with MSE. Since both losses give the same optimal embedding prediction (see FMLM [1, Proposition 3.2]), I see limited novelty in that change alone.
I understand that MSE may be easier to optimize in practice, which you mention in the paper, which I understand intuitively, but cannot understand why it should work or is there any theoretical explanation of that. Figure 3(b) is the strongest evidence for me in practice, although I am unsure how consistent the effect is across training seeds. Given the rebuttal deadline, I am not asking for new experiments.
My question is specifically about the proof. As I read it, Theorem 1 concerns convergence of the sampling trajectory for a fixed predictor. Is there a step in the proof that I probably have not noticed, but uses the fact that the predictor was trained with MSE, or would the same argument apply to a CE-trained predictor with the same parameterization? What is MSE loss gives in theory, this convergence guarantees or something else? If MSE loss is not connected with Theorem 1 then what is the connection between theoretical and practical sights of the paper?
If you can clarify the connection between Theorem 1 and the choice of MSE, I would be willing to raise my score.
[1] FMLM (https://arxiv.org/pdf/2602.16813)
Thank you for the follow-up. We clarify below the connection between Theorem 1 and MSE training, and the practical benefits of MSE.
Connection between Theorem 1 and the choice of MSE
The relationship is not that MSE training leads to the convergence established by Theorem 1, but that this convergence guarantee allows us to train the predictor with MSE alone when the embeddings are fixed.
For example, ELF also uses an MSE objective for continuous denoising. However, MSE alone is insufficient in their setting even with fixed embeddings, as there is no guarantee of convergence. Its training additionally includes a CE objective for decoding. In our setting, convergence to the fixed token embeddings justifies recovering tokens through nearest-neighbor decoding, without training an additional decoding mechanism.
In summary, Theorem 1 shows that the standard flow-matching MSE alone suffices to train the predictor with fixed, pretrained token embeddings.
Why MSE can work better in practice
We agree that CE and MSE can both recover the exact data predictor at their population optima, as discussed in FMLM. However, under limited model capacity, minimizing the two objectives may yield different predictors. Specifically, a network that can represent the exact conditional token posterior can also represent the exact conditional embedding mean. The converse need not hold, since different token distributions can yield the same embedding mean. Accurately predicting this mean therefore does not necessarily require recovering the full token posterior. Hence, as you mentioned, intuitively, directly supervising the continuous prediction required by flow matching may be easier to optimize in practice, which is again based on Theorem 1.
We have clarified this connection between Theorem 1 and the choice of MSE in the revised manuscript.
Thank you again for your efforts to review our paper. As the discussion period ends in one day, we would be grateful to know whether you have further concerns.
We thank the reviewer for carefully reading our paper and recognizing the strengths of our work. We address the reviewer's concerns below.
Our contributions built on LangFlow
We agree that our main experiments use the same model architecture and initialize from its CE-pretrained checkpoint, with the token embeddings kept fixed, as acknowledged in our paper. As discussed in the paper, optimizing the embeddings with MSE alone admits collapsed solutions. Building on this setup, our contributions are as follows:
∙ Theoretical contribution: To our knowledge, we provide the first convergence guarantee to valid token embeddings for this parameterization. Under the assumptions of Theorem~1, the continuous trajectory converges even when the learned data predictor is imperfect, without required additional projection step.
∙ Practical contribution: With the pretrained embeddings held fixed, we can continue training the data predictor using MSE alone. MSE directly supervises the continuous prediction required by flow matchings, whereas CE supervises token probabilities that are subsequently mapped to an embedding-weighted prediction. This direct supervision may improve conditional-mean estimation under finite model capacity, consistent with our empirical results. Therefore, our work provides theoretical guidance for designing continuous language models.
We have modified Sections 1, 4, and 5 in the revised manuscript to highlight the shared parameterization and dependence on CE-pretrained components, and to clarify the theoretical and practical contributions built upon them. In particular, Section 1 explains the motivation for direct MSE supervision, stating: "CE supervises conditional token probabilities, which are mapped to an estimate of the conditional embedding mean required by FMs. Direct MSE supervision of this continuous prediction may improve its accuracy under finite model capacity."
Clarify on convergence analysis
Thank you for pointing out this issue. We have added a covering argument to address the dependence of the auxiliary weights on the realized sampling trajectory. Specifically, we construct a finite ε-net for weight sequences on a time grid of spacing ε in [0,1]. Since this grid is independent of N, its cardinality is also independent of N, only depends on L, V, and ε. Assigning each sampling time tk to its nearest grid point maps these representatives to the sampling time steps. The resulting set has cardinality bounded independently of N and approximates every realized weight trajectory with error O(ε). For a representative f approximating the realized weight trajectory, the triangle inequality gives \begin{align} \min_{j\in[V]}|x_{t_N}-e_j| \le \min_{j\in[V]}|\widehat x_{t_N}(\widehat f)-e_j| + |x_{t_N}-\widehat x_{t_N}(\widehat f)|. \end{align} For fixed ε, the fixed-weight convergence result and a union bound over the ε-net show that the first term vanishes in probability as N→∞ and tN→1. Letting ε→0 then makes the approximation error vanish, yielding the desired convergence. We have included the detailed argument in the Appendix D.1 of the revised manuscript.
Reorganization of the background and sampling parts
Thank you for the valuable suggestion. With the additional space available in the revised manuscript, we have moved the motivation and implementation details of the guidance and iterative refinement methods to the main text and relocated some standard flow-matching derivations to the appendix.