- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposed FLUID-Omni, a commit-repair-write approach for joint text and audio/speech modeling and generation. The authors claim that compared with a traditional autoregressive multi-modal LLM such as Qwen-2.5-Omni-7B, FLUID-Omni better balances performance and speed.
Critical strengths
- The method makes intuitive sense, and the resulting model achieves reasonable performance.
- The paper is well-organized and generally easy to follow.
Critical weaknesses
- My biggest concern is that the core claim of the paper, "substantially improves generation efficiency while preserving omni-modal generation quality," which is stated in the abstract, the introduction, and the conclusion, isn't fully supported by the experiment results. While model generation quality seems decent in Table 1, I could not find a comparison in generation efficiency. Moreover, the model performance in Table 2 seems mediocre, and the Qwen-2.5-Omni-7B baseline isn't in the comparison.
- In Eq.8, the authors mentioned that the refining process consists of replacement, insertion, and deletion operations. However, how these operations are implemented is not clear to me. Do you run another forward prediction conditioned on the generated context, but without any mask tokens? Or something else?
- The authors define a criterion for determining prediction stability as the prediction not changing before/after refinement. However, I could not find how we guarantee the model prediction eventually converges, and what to do if it doesn't (e.g., the prediction exhibits a cyclic behavior instead of converging). Also, due to the randomness of the sampling process (I'm assuming a sampling process because the authors did not specify whether we should use greedy decoding or sampling), even if the prediction does not change after one refinement run, it might change with a second refinement step. Do we still consider this stable? That is, is it sufficient to ensure stability by only inspecting the edits in a single refinement pass, or should we observe the edits over multiple refinements?
- The authors proposed a three-stage training pipeline (they first claimed it to be two-stage but then mentioned stage 3), but didn't clearly explain the need for it. Why can't we go straight to stage 3 and jointly optimize the two objectives?
- To better understand the contribution of the refining process, a core difference between FLUID-Omni and existing diffusion LLMs, we need to compare with a baseline model based on Qwen-2.5-Omni-7B that is trained with the same data but without the refining stage. Ideally, we should have one baseline that removes inference-time repairing altogether, and another one that keeps inference-time repairing but doesn't train with this mechanism. However, I could not find this comparison.
- Minor -- typo in Figure 1: latency->atency.
Questions for authors
See weaknesses.