acceptodds
Withdrawn from ICLR 2027

FLUID-Omni: Parallelizing Streaming Omni Generation with Commit-Repair-Write Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive omni-modal models naturally support streaming generation because each newly produced token extends a stable causal prefix that can be immediately consumed. However, their sequential decoding process limits generation efficiency. Discrete diffusion decoding provides a promising alternative through parallel generation, yet directly applying it to omni-modal generation introduces a fundamental challenge: future predictions may become available before they are reliable enough for irreversible commitment. We identify this prediction–commitment mismatch as a key barrier to efficient parallel streaming generation. We introduce FLUID-Omni, a causal diffusion framework that bridges parallel generation and streaming reliability through a Commit–Repair–Write paradigm. Instead of treating all predicted future tokens as equally reliable, FLUID-Omni selectively commits stable predictions, locally repairs inconsistent regions while preserving reusable context, and continuously writes new predictions in parallel. To align training with streaming inference, we develop a multi-stage adaptation strategy that equips the model with causal parallel prediction and pipeline-consistent refinement capabilities. Furthermore, we introduce prior-calibrated codec decoding to mitigate biased confidence estimation in audio generation. Experiments across semantic, acoustic, and streaming evaluations demonstrate that FLUID-Omni substantially improves generation efficiency while preserving omni-modal generation quality. These results show that parallel diffusion can achieve reliable streaming generation when prediction and commitment are explicitly decoupled.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Omni-Modal Generation;Diffusion Decoding;Streaming Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Zk8pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed FLUID-Omni, a commit-repair-write approach for joint text and audio/speech modeling and generation. The authors claim that compared with a traditional autoregressive multi-modal LLM such as Qwen-2.5-Omni-7B, FLUID-Omni better balances performance and speed.

Critical strengths
  • The method makes intuitive sense, and the resulting model achieves reasonable performance.
  • The paper is well-organized and generally easy to follow.
Critical weaknesses
  • My biggest concern is that the core claim of the paper, "substantially improves generation efficiency while preserving omni-modal generation quality," which is stated in the abstract, the introduction, and the conclusion, isn't fully supported by the experiment results. While model generation quality seems decent in Table 1, I could not find a comparison in generation efficiency. Moreover, the model performance in Table 2 seems mediocre, and the Qwen-2.5-Omni-7B baseline isn't in the comparison.
  • In Eq.8, the authors mentioned that the refining process consists of replacement, insertion, and deletion operations. However, how these operations are implemented is not clear to me. Do you run another forward prediction conditioned on the generated context, but without any mask tokens? Or something else?
  • The authors define a criterion for determining prediction stability as the prediction not changing before/after refinement. However, I could not find how we guarantee the model prediction eventually converges, and what to do if it doesn't (e.g., the prediction exhibits a cyclic behavior instead of converging). Also, due to the randomness of the sampling process (I'm assuming a sampling process because the authors did not specify whether we should use greedy decoding or sampling), even if the prediction does not change after one refinement run, it might change with a second refinement step. Do we still consider this stable? That is, is it sufficient to ensure stability by only inspecting the edits in a single refinement pass, or should we observe the edits over multiple refinements?
  • The authors proposed a three-stage training pipeline (they first claimed it to be two-stage but then mentioned stage 3), but didn't clearly explain the need for it. Why can't we go straight to stage 3 and jointly optimize the two objectives?
  • To better understand the contribution of the refining process, a core difference between FLUID-Omni and existing diffusion LLMs, we need to compare with a baseline model based on Qwen-2.5-Omni-7B that is trained with the same data but without the refining stage. Ideally, we should have one baseline that removes inference-time repairing altogether, and another one that keeps inference-time repairing but doesn't train with this mechanism. However, I could not find this comparison.
  • Minor -- typo in Figure 1: latency->atency.
Questions for authors

See weaknesses.

Reviewer_Ng8qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a diffusion decoding conversion scheme for autoregressive language models. Autoregressive decoding cannot be parallized due to its sequential nature, so masked diffusion decoding scheme is presented as a more efficient alternative.

The naive parallel masked diffusion has the risk that the paralel generation might not be causal, and contain errors. The paper proposes a scheme called 'commit-repair-write' to improve the generation quality.

They compare in terms of accuracy with other autoregressive vision language models, and other diffusion vision language models. They seem to slightly improve over the baselines they compare. They also compare in terms of accuracy with few autoregressive audio language models.

Critical strengths
  • On vision language modeling benchmarks the proposed FLUID Omni seems provide an improvement in terms of accuracy over the baseline models the autors are comparing against, for the most part. (Table 1)

  • Even though some of the details are not very well explained in the paper of the proposed scheme (e.g. Text-token commitment rule is not stated), the method seems to be sensible for improving diffusion decoding, as shown by the performance numbers.

Critical weaknesses
  • My biggest issue is that the paper does not contain a latency comparison with the autoregressive and diffusion language models they are comparing against. Without this information it is difficult to assess the effectiveness of the approach. I would suggest adding comparisons in terms latency with some of the most competitive vision language models you have in Table 1, and also the most competitive audio language models you have in Table 2. You can include RTF comparisons for example.
Questions for authors
  • When you do the conversion to the diffusion decoding, I cannot find in the paper what you are actually fine tuning? Are you using a LORA adapter, or are you fine tuning the whole model for example?

  • So you initialize your own model, FLUID Omni from QWEN 2.5 Omni. Though you are not directly comparing with it to see how much performance improvement or drop you are observing after the conversion. Do you have these numbers?

  • Did you try your conversion method on text-only LLMs as well? I found it a bit difficult to understand what makes your proposed diffusion conversion scheme specific to multimodal LLMs.

Reviewer_y9niOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to speed up streaming omni-modal generation by replacing autoregressive decoding with discrete diffusion. Starting from Qwen2.5-Omni-7B, FLUID-Omni converts both text and speech generation into block-causal masked diffusion. Its Commit-Repair-Write decoding keeps a committed prefix, a repair window that is locally edited, and a write window predicted in parallel; only the longest stable prefix is committed. For audio, confidence is calibrated by subtracting the codec-token prior. The model is trained on about 253k self-distilled samples. Validation covers vision-language benchmarks, spoken question answering, and speech quality, plus ablations on prior calibration and decoding strategies.

Critical strengths
  1. Important problem. Streaming speech generation in omni models is slow because audio token streams are dense. Exploring parallel diffusion decoding for this setting is timely and practically relevant.
  2. Intuitive design. Decoupling prediction from commitment, repairing locally instead of remasking globally, and calibrating codec confidence by its prior are simple, sensible ideas for streaming diffusion decoding.
Critical weaknesses
  1. No efficiency evidence. The title and abstract center on efficiency, yet the paper reports no latency, throughput, tokens per step, first-packet latency, or RTF. The efficiency section reports only an accuracy aggregate.

  2. Missing base-model comparison. Neither table reports Qwen2.5-Omni-7B under autoregressive decoding. FLUID-Omni also trails Qwen3-Omni by about 16 points on WebQ and TriviaQA and on UTMOS; its higher average relies on two Chinese benchmarks where Qwen3-Omni scores anomalously low.

  3. Missing method details. It is unclear how insertions and deletions work in fixed-length blocks, how many repair iterations are run, and how the block size and thresholds are set. The streaming latency of the token-to-wave stage is not discussed.

  4. No acceleration baselines. The paper compares only against generic AR and diffusion models, not against other acceleration methods such as speculative decoding, multi-token prediction, accelerated block diffusion, or other parallel speech decoding approaches.

Questions for authors
  1. Speed-quality trade-off. Please report Qwen2.5-Omni-7B with autoregressive decoding under the same evaluation, together with first-packet latency, throughput, and RTF for both models.

  2. Repair implementation. How are insertions and deletions handled in fixed-length blocks, how many repair iterations are used, and what block size and commit thresholds are chosen?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.