acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

27 views

Abstract

We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and quantized images as discrete tokens or combine discrete language prediction with continuous image generation. The former introduces a visual quantization bottleneck. The latter requires modality-dependent objectives and sampling procedures. Fully continuous modeling avoids these trade-offs and enables a shared generative process, but remains underexplored for multimodal pretraining. Multimodal Flow introduces a unified continuous architecture that integrates multimodal continuous representations with a shared chunk-causal flow backbone. It organizes text blocks and images as ordered continuous hyperchunks, preserving textual token order and visual spatial structure. The backbone learns a single vector field over these hyperchunks through Flow Matching. Joint attention enables cross-modal interaction, while modality-specific feed-forward networks process each modality. The model predicts multiple target chunks in parallel during training and generates hyperchunks sequentially at inference. We instantiate MF-1 and pretrain it on multimodal data. Across 0.6B, 1.2B, and 1.6B scales, continued pretraining consistently improves multimodal modeling. With only 150B pretraining tokens, MF-1 achieves an average score of 82.8 across GenEval and DPG-Bench and 75.3 across VQAv2, MMBench, and POPE, remaining competitive with unified models trained on substantially more data. Under matched data, optimization, and parameter budgets, Multimodal Flow further outperforms representative hybrid and discrete models. These results establish continuous chunk-based embedding flow modeling as a new fully continuous paradigm for unified multimodal modeling. The code and model will be released to facilitate future research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Modeling, Fully Continuous Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DgcbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Multimodal Flow, a unified framework that models language and vision using Flow Matching in continuous embedding spaces. It introduces ordered hyperchunks and a shared chunk-causal Transformer to support both text and image generation under a single training objective. The authors train MF-1 up to 1.6B parameters and demonstrate competitive performance on image generation and visual understanding benchmarks. Scaling experiments, controlled comparisons with discrete and hybrid models, and ablation studies further support the proposed approach.

Critical strengths
  1. A promising direction for unified multimodal modeling. The paper explores a promising alternative to existing discrete and hybrid approaches by modeling both language and vision through a shared continuous generative process. Beyond the specific architecture, this work demonstrates the potential of fully continuous modeling as a new direction for unified vision-language models.

  2. Strong experimental validation. The authors train models up to 1.6B parameters and evaluate them across image generation and visual understanding tasks. The controlled comparisons with discrete and hybrid architectures under matched training budgets provide useful evidence for the effectiveness of the proposed framework.

Critical weaknesses

Limited insights into scaling and data efficiency. While Figure 4 studies the scaling behavior of MF-1, the comparison with discrete and hybrid modeling paradigms in Table 4 is limited to a single model scale and pretraining budget. It remains unclear whether the advantages of fully continuous modeling persist or grow with increasing model size and training data, and whether this approach offers better data efficiency. Given that the paper proposes a new modeling paradigm, a deeper understanding of these properties would be valuable.

Questions for authors
  1. Data efficiency. Table 4 compares different modeling paradigms at a fixed pretraining budget of 50B tokens. Could the authors provide performance comparisons at different training stages? This would help clarify whether Multimodal Flow is more data-efficient than the discrete and hybrid baselines.

  2. Scaling behavior. Figure 4 demonstrates scaling trends within MF-1, but it remains unclear whether its advantages over other modeling paradigms persist as model size increases. Do the authors have any evidence or preliminary observations about how these relative advantages change with model scale?

5 comments
AuthorsOpenReview ↗

Response to Question 1 (and Weakness)

We sincerely thank you for the encouraging review and thoughtful questions. We truly appreciate your recognition of fully continuous modeling as a promising new direction for unified multimodal models. We completely agree that understanding data efficiency and model scaling across paradigms is a central question for this line of research, and we are glad to share new experimental observations and thoughts on both aspects below.


Question 1 (and Weakness): Data Efficiency Across Pretraining Stages

Could the authors provide performance comparisons at different training stages to clarify whether Multimodal Flow is more data efficient than the discrete and hybrid baselines?

Tracking intermediate pretraining checkpoints is very helpful for understanding how data efficiency evolves across modeling paradigms.

Following your suggestion, we extended the controlled comparison of the three matched 1.6B architectures (Multimodal Flow, Transfusion-style, and Chameleon-style) from 50B up to 150B pretraining tokens (40K to 240K steps) under identical data mixtures and optimization schedules:

  • Pretraining GenEval (%):
Architecture (1.6B)40K (25B)80K (50B)120K (75B)160K (100B)200K (125B)240K (150B)
Multimodal Flow8.3420.6222.6023.0324.0025.01
Transfusion-style12.9514.2016.7217.3719.3420.07
Chameleon-style0.732.123.454.434.785.39
  • Pretraining CLIPScore (%):
Architecture (1.6B)40K (25B)80K (50B)120K (75B)160K (100B)200K (125B)240K (150B)
Multimodal Flow81.9282.4182.5982.8382.9582.97
Transfusion-style81.5482.1182.2082.3782.4682.34
Chameleon-style66.2871.2072.8773.9274.8775.17

These trajectories highlight two interesting patterns:

  1. Comparison with the fully discrete paradigm: Multimodal Flow leads Chameleon-style by a wide margin from the earliest checkpoint (40K) through 240K steps on both metrics, as modeling continuous semantic embeddings directly avoids spending early capacity on quantized visual codebooks.
  2. Comparison with the hybrid paradigm: At 40K steps, Transfusion-style warms up slightly faster on GenEval because standard AR text representations form quickly early in training. Once continuous language and vision representations align around 80K steps, Multimodal Flow shows stronger sample efficiency: its performance at 80K steps already exceeds Transfusion-style at 240K steps (150B tokens) on both GenEval and CLIPScore, and continues to improve steadily.
AuthorsOpenReview ↗

Response to Question 2 (and Weakness)

Question 2: Preliminary Observations at Larger Model Scale (3B)

Do the authors have any evidence or preliminary observations about how these relative advantages change with model scale?

We appreciate your interest in how paradigm differences behave at larger model scales. Exploring this question provides valuable guidance for scaling unified continuous models.

Due to compute limits during the rebuttal period, pretraining multiple 3B models from scratch for hundreds of billions of tokens was not feasible. To obtain preliminary evidence at a larger scale, we trained matched 3B versions of Multimodal Flow and Transfusion-style on the same SFT data under identical optimization settings:

Model (Matched 3B)POPE ↑MMBench ↑SEEDB ↑GQA ↑Avg ↑
Multimodal Flow (3B)87.2747.5948.8456.6860.10
Transfusion-style (3B)80.1832.9924.5856.2648.50

At the 3B scale, Multimodal Flow continues to show clear gains across all four VQA benchmarks (+11.60% higher average score). We hypothesize that optimizing a single continuous Flow Matching objective allows added backbone capacity to be shared more coherently across vision and language than jointly optimizing two heterogeneous objectives. While full pretraining at larger scales is still needed for definitive conclusions, both the 150B trajectories and this 3B comparison are encouraging, and we plan to further scale Multimodal Flow in both data and model size.


Thank you again for your insightful comments and constructive engagement with our work. We hope these observations address your questions, and we welcome any further discussion.

Dear Reviewer,

Thank you again for recognizing the promise of fully continuous multimodal modeling and for your thoughtful questions about data efficiency and scaling!

With less than two days remaining in the discussion period, we wanted to briefly follow up on our responses. Your questions motivated us to extend the matched learning curves to 150B pretraining tokens and to conduct a preliminary comparison of matched 3B models trained on the same SFT data.

We would welcome your thoughts on these additional results and would be happy to discuss any further questions or observations.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_u68gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to generate both text and images using a unified continuous flow matching. Text is represented by blocks of eight tokens, and images are represented by spatial chunks, and these are sequentially generated in the shared model. The extensive experiments demonstrate the effectiveness of the proposed method.

Critical strengths
  1. This paper is well-organized and easy to follow.
  2. The proposed paradigm employs a unified flow matching objective, making learning and inference in a clear unified multimodal modeling.
  3. The experiments are comprehensive.
Critical weaknesses
  1. The evaluation of the model on language is still not sufficient. More evaluation on the standard language benchmarks is necessary.
  2. One of my major concerns is that fixed eight-token chunking may split words or phrases across chunk boundaries, particularly in Chinese. For example, “你” and “好” in “你好” could fall into different chunks. Since each chunk is encoded independently, I am concerned that this separation may disrupt the semantic continuity of sentences and lead to worse text generation than autoregressive LLMs. Please provide a fair comparison with AR LLMs under matched model sizes, training data, and computational budgets, and examine how chunk size affects semantic coherence and generation quality.
  3. A concern is that MF-1 and the baselines are not directly comparable: MF-1 benefits from a pretrained T5 encoder and a separately trained decoder, while the baselines operate on token IDs alone. As a result, it remains unclear how much of the gain can be attributed to continuous flow matching rather than these extra components.
  4. It is necessary to include a training cost comparison such as FLOPs or GPU hours.
  5. It is necessary to include the text and image generation latency.
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1

We sincerely thank you for the positive assessment and thoughtful suggestions, and for recognizing the clarity of our paper, our unified Flow Matching formulation, and our comprehensive experiments. Below, we address each concern with new experiments and detailed analyses, which will be included in the revised manuscript.


Weakness 1: Evaluation on Standard Language Benchmarks

Thank you for this valuable suggestion. Adding standard pure language benchmarks provides a much more complete evaluation of MF-1's language modeling capabilities.

Since flow models output continuous embeddings rather than exact token log likelihoods, we follow the generative evaluation protocol of CoLa (Guo et al., 2026, NeurIPS 2026 Spotlight) by decoding latents into text answers before scoring across five language benchmarks. We compare MF-1 against recent continuous LMs (CoLa and Aurora-LM (Liang et al., 2026)) and our matched 1.6B Transfusion-style baseline (AR cross entropy for text, trained on the exact same data and token budget):

ModelText ParadigmMMLUOBQARACEARC-CSIQAAvg ↑
Multimodal Flow (Ours)Continuous Flow23.826.233.723.731.527.8
CoLaContinuous Diffusion19.624.224.220.626.823.1
Aurora-LMContinuous Diffusion22.227.830.621.230.226.4
Transfusion-style (Matched)Autoregressive21.919.824.821.719.021.4

Without language instruction tuning, base pretrained models have limited format adherence under generative evaluation, placing MMLU slightly below the 25.0% random baseline across all models (including AR at 21.9%). Under the exact same protocol, MF-1 achieves the highest overall average (27.8%), surpassing both continuous LMs and the matched AR baseline.

References

  • H. Guo, Q. Zhao, Y. Zhao, S. Nie, R. Zhu, Q. Guo, F. Wang, T. Yang, H. Zhao, G. Wei, et al. Continuous latent diffusion language model. Advances in Neural Information Processing Systems (NeurIPS), 2026 (Spotlight). arXiv:2605.06548.
  • J. Liang, Y. Liao, Y. Cao, et al. Aurora-LM: Autoencoding unified representation for continuous latent diffusion language modeling. arXiv preprint arXiv:2608.02602, 2026.
AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2: Fixed Chunking Across Word Boundaries, Fair AR Comparison, and Chunk Size Ablation

We appreciate your careful consideration of chunk boundary effects. Examining word splitting, boundary reconstruction, matched AR performance, and chunk size sensitivity helps thoroughly validate our hyperchunk design.

1. Tokenization and cross chunk context.
Subword tokenizers keep frequent words intact, and when a word spans two chunks, each subword remains a valid vocabulary unit. Although the frozen T5-small encoder encodes each chunk independently at the input interface, sentence level continuity is modeled by the chunk causal flow backbone: through joint chunk causal attention with MRoPE, every position in target chunk ckc_k attends directly to all token positions in preceding clean chunks c<kc_{<k}, giving the start of ckc_k full contextual access to the end of ck−1c_{k-1}.

2. Reconstruction accuracy within vs. across chunk boundaries.
We evaluated clean latent encoding and decoding reconstruction on English evaluation text over complete blocks of 8 tokens (excluding padded tail blocks). A multi subword word is counted as correct only when all constituent subwords are reconstructed identically:

Reconstruction MetricSamplesErrorsAccuracy (%) ↑
Single Token1,018,5602499.9976
Within Chunk Multi Subword Word82,894499.9952
Across Boundary Multi Subword Word17,462299.9885

Words split across chunk boundaries achieve 99.9885% whole word accuracy, virtually matching within chunk words and confirming that chunked encoding preserves lexical integrity.

3. Fair comparison with a matched AR model.
As shown in the Weakness 1 table above, Multimodal Flow (27.8% average) outperforms the matched Transfusion-style model using token level AR text modeling (21.4% average) across all five language benchmarks.

4. Ablation on chunk size (B=4,8,16B = 4, 8, 16).
Pretraining with B=4,8,B = 4, 8, and 1616 on identical data from 40K to 240K steps yields consistent captioning CLIPScore (%) and T2I DPG-Bench (%) performance:

  • CLIPScore (%) across training steps:
Chunk Size40K80K120K160K200K240K
B=4B = 481.9782.4782.7682.7882.8282.97
B=8B = 881.9282.4182.5982.8382.9582.97
B=16B = 1681.8482.4082.6382.7682.9182.99
  • DPG-Bench (%) across training steps:
Chunk Size40K80K120K160K200K240K
B=4B = 452.2861.9067.2868.3469.9470.69
B=8B = 850.7461.4366.4167.4469.2069.68
B=16B = 1649.8261.8466.2467.6269.0569.87

All three settings converge to nearly identical CLIPScore and very close DPG-Bench scores, showing that semantic coherence is robust to chunk size and that B=8B = 8 balances granularity and parallel speed well.

AuthorsOpenReview ↗

Response to Weakness 3, 4, 5

Weakness 3: Isolating Continuous Flow Matching from the Frozen T5 Encoder and Text Decoder

To isolate the contribution of the frozen T5-small encoder from unified continuous Flow Matching, we trained an additional controlled 1.6B baseline, Transfusion-T5, on the exact same 150B token budget by adding frozen T5-small features for text conditioning during image generation while retaining token ID AR modeling for text:

Model (150B Pretraining)GenEval (%) ↑CLIPScore (%) ↑
Multimodal Flow25.0182.97
Transfusion-style20.0782.34
Transfusion-T520.8582.27

Adding frozen T5-small conditioning to Transfusion-style brings only a minor +0.78% gain on GenEval and slightly lowers CLIPScore, whereas Multimodal Flow outperforms Transfusion-T5 by +4.16% on GenEval and +0.70% on CLIPScore, confirming that the gains come primarily from unified continuous Flow Matching.


Weakness 4: Training Cost Comparison (FLOPs and GPU Hours)

Thank you for this helpful suggestion. Including explicit training compute and hardware cost makes our paradigm comparison much more informative:

Model (1.6B, 50B Tokens)Backbone FLOPsH20 GPU Hours
Multimodal Flow3.523×10203.523 \times 10^{20}2,256
Transfusion-style3.523×10203.523 \times 10^{20}2,112
Chameleon-style3.523×10203.523 \times 10^{20}2,514

All three matched 1.6B backbones process the same packed token budget and therefore share identical backbone FLOPs, while minor GPU hour differences reflect frozen codec execution and attention mask patterns (+6.8% vs. Transfusion-style and -10.3% vs. Chameleon-style).


Weakness 5: Text and Image Generation Latency

Reporting generation latency is essential for understanding practical deployment efficiency. Below are measured latencies on an RTX 5090 GPU with FlashAttention and CUDA Graphs enabled:

Task & SettingOutputLatency ↓Throughput ↑
Text (B=8B=8, 16 SDE steps, CFG) [Default]512 tokens2,497 ms205 tok/s
Text (B=8B=8, 4 SDE steps, CFG)512 tokens728 ms703 tok/s
Text (B=16B=16, 4 SDE steps, CFG)512 tokens388 ms1,319 tok/s
Image (256 tokens, 64 ODE steps, CFG)224×224224 \times 224484.79 ms2.063 img/s

Because MF-1 generates all BB tokens of a chunk in parallel per step, using 4 SDE steps reaches 703 tok/s (faster than 389 tok/s for a matched 1.6B AR baseline under identical acceleration) while preserving VQA accuracy (62.5% vs. 62.6% average across five benchmarks). Overall, chunk parallel sampling makes inference efficiency a promising strength of Multimodal Flow for further exploration.

Thank you again for your valuable comments on Multimodal Flow! We hope our responses address your concerns, and we welcome any further discussion.

Dear Reviewer,

Thank you again for your positive assessment of our work and for your constructive suggestions, which have helped us strengthen the evaluation and analysis.

With less than two days remaining in the discussion period, we wanted to briefly follow up on our responses. Following your suggestions, we have added language benchmark results, chunk-boundary reconstruction and chunk-size analyses, a Transfusion-T5 control, and training cost and generation latency measurements.

We would welcome any further thoughts you may have on these additions. If any points would benefit from further clarification, we would be happy to elaborate.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_W8aAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Multimodal Flow, a model that handles both language and vision in fully continuous form. Text blocks of 8 tokens are encoded with T5, images with SigLIP2, and both become ordered continuous "hyperchunks." A 1.6B chunk-causal backbone then learns a Flow Matching vector field over these hyperchunks, using shared attention and modality-specific FFNs.

Understanding and generation tasks change only how the chunks are ordered, while the objective and sampling procedure stay the same. Trained from scratch on just 150B tokens, MF-1 scores 0.82 on GenEval and 83.44 on DPG-Bench. In controlled comparisons it outperforms both hybrid (Transfusion-style) and fully discrete (Chameleon-style) approaches.

Critical strengths

A genuinely unified formulation. Multimodal Flow uses a single training objective (the Flow Matching loss) and a single sampling procedure for both modalities and both directions, covering understanding as well as generation. Switching between unimodal modeling, image captioning/VQA, and text-to-image generation only changes how the hyperchunks are ordered, with no changes to the loss or the sampler.

Critical weaknesses

Weakness 1. Weak text evaluation. Text generation is judged only by GPT-2-large perplexity and captioning metrics, and there are no pure language benchmarks such as MMLU, so the model's language modeling can't be fairly compared with AR LMs. Its understanding scores (e.g., GQA 58.3, MMBench 67.2) also lag well behind LLM-initialized models such as Janus-Pro.

Weakness 2. Expensive text inference. In the VQA setting, each 8-token block needs 16 SDE steps, doubled by CFG, which comes to about 4 forward passes per token versus 1 for AR decoding. The fixed 8-token block size also isn't ablated, and variable-length outputs aren't discussed.

Weakness 3. Questions about fair comparison. The controlled comparison uses only 50B pretraining tokens, which is far from sufficient for a reliable comparison at the 1.6B scale. Modern unified models are typically pretrained on trillions of tokens, and discrete/autoregressive baselines are known to be especially data-hungry. Under such a limited budget, the Chameleon-style and Transfusion-style baselines are likely undertrained, so the comparison may favor the continuous formulation, and it is unclear whether the reported ranking would still hold at larger data scales. Separately, the GenEval score of 0.82 is measured after 5B tokens of downstream finetuning, and the paper doesn't clearly address possible overlap between that finetuning data and GenEval-style prompts.

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

Thank you for the constructive feedback and for recognizing our unified formulation across understanding and generation! Below we address each point with new experiments and clarifications, which will be added to the revised paper.


Weakness 1: Pure Language Evaluation & Comparison with Unified Models Initialized from Pretrained LLMs

Thank you for pointing this out. Evaluating pure language benchmarks and clarifying the comparison with LLM initialized models are essential for giving a complete picture of MF-1's capabilities.

1. Pure language benchmarks against continuous LMs and matched AR baselines.
Since flow models output continuous embeddings rather than exact token log likelihoods, we follow the generative evaluation protocol of CoLa (Guo et al., 2026, NeurIPS 2026 Spotlight) by decoding latents into text answers before scoring across five language benchmarks. We compare MF-1 against recent continuous LMs (CoLa and Aurora-LM (Liang et al., 2026)) and our matched 1.6B Transfusion-style baseline (AR cross entropy for text, trained on the exact same data and token budget):

ModelText ParadigmMMLU (%)OBQA (%)RACE (%)ARC-C (%)SIQA (%)Avg (%) ↑
Multimodal Flow (Ours)Continuous Flow23.826.233.723.731.527.8
CoLa (Guo et al., 2026)Continuous Diffusion19.624.224.220.626.823.1
Aurora-LM (Liang et al., 2026)Continuous Diffusion22.227.830.621.230.226.4
Transfusion-style (Matched)Autoregressive (AR)21.919.824.821.719.021.4

Because base pretrained models without language instruction tuning have limited instruction following ability under generative evaluation, MMLU accuracy falls slightly below the 25.0% random baseline across all evaluated models, including the AR baseline (21.9%). Under the same protocol, MF-1 achieves the best average score (27.8%), surpassing both continuous LMs and the matched AR baseline.

2. Contextualizing VQA performance against models initialized from pretrained LLMs (Janus-Pro).

The gap on certain VQA benchmarks reflects pretraining token scale rather than modeling paradigm. Janus-Pro (discrete) and JanusFlow (hybrid) initialize from DeepSeek-LLM pretrained on 1.1T to 2.0T tokens, whereas MF-1 is trained from scratch on only 150B multimodal tokens. Even with ~10x fewer tokens, MF-1 nearly matches Janus-Pro on POPE (86.1 vs. 86.2) and GQA (58.3 vs. 59.3), substantially outperforms models trained without LLM initialization such as Muddit (400B) and D-DiT (4.3T) (Table 3), and beats discrete and hybrid baselines under matched budgets from scratch (Table 4 and Weakness 3).

References

  • H. Guo, Q. Zhao, Y. Zhao, S. Nie, R. Zhu, Q. Guo, F. Wang, T. Yang, H. Zhao, G. Wei, et al. Continuous latent diffusion language model. Advances in Neural Information Processing Systems (NeurIPS), 2026 (Spotlight). arXiv:2605.06548.
  • J. Liang, Y. Liao, Y. Cao, et al. Aurora-LM: Autoencoding unified representation for continuous latent diffusion language modeling. arXiv preprint arXiv:2608.02602, 2026.
AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2: Text Inference Efficiency, Block Size Ablation, and Variable Length Generation

We appreciate this insightful comment. Examining practical decoding speed, chunk size sensitivity, and variable length generation is crucial for assessing the practicality of our framework, and will make our presentation much clearer.

1. Decoding speed & denoising steps.
Theoretical FLOPs per token do not map linearly to latency: AR decoding runs sequentially one token at a time and is memory bandwidth bound at low batch sizes, whereas MF-1 computes all BB chunk tokens and batched CFG branches in parallel per step, updating the KV cache once per chunk. Measured on an RTX 5090 GPU (with FlashAttention and CUDA Graphs enabled for all models):

SettingSeq. Steps / TokRel. FLOPsSpeed (tok/s) ↑vs. AR
AR Baseline (Matched 1.6B)1.001.0x3891.00x
MF-1 (B=8B=8, 16 SDE steps, CFG) [Default]2.004.0x2050.53x
MF-1 (B=8B=8, 4 SDE steps, CFG)0.501.0x7031.81x
MF-1 (B=16B=16, 4 SDE steps, CFG)0.250.5x1,3193.39x

Reducing VQA sampling from 16 to 4 SDE steps matches AR FLOPs (1.0x), reaches 1.81x the speed of AR (703 tok/s), and preserves VQA accuracy:

MF-1 SettingSpeed ↑MMBSEEDBGQAPOPEOK-VQAAvg ↑
B=8B=8, 16 SDE steps, CFG205 tok/s67.262.458.386.139.162.6
B=8B=8, 4 SDE steps, CFG703 tok/s67.162.558.086.138.762.5

Using B=16B=16 further boosts speed to 1,319 tok/s (3.39x AR) while keeping pretraining captioning CLIPScore (%) nearly unchanged:

Chunk Size40K80K120K160K200K240K
B=8B = 881.9282.4182.5982.8382.9582.97
B=16B = 1681.8482.4082.6382.7682.9182.99

2. Variable length outputs.
Sequences are not restricted to multiples of B=8B=8 (Appendix B.2). During training, target text ends with <EOS>. At inference, chunks are generated and decoded sequentially, and once the first <EOS> appears within a block, trailing tokens are truncated and generation stops immediately. This enables both single word VQA answers and longer descriptive outputs of arbitrary length.

Overall, these results show that chunk parallel decoding makes inference efficiency a promising strength of Multimodal Flow rather than a bottleneck, warranting further exploration of few step samplers and larger chunk sizes in future work.

AuthorsOpenReview ↗

### Response to Weakness 3

Weakness 3: Controlled Scaling to 150B Tokens & GenEval Finetuning Data Analysis

Thank you for raising these important questions on experimental fairness. Testing larger pretraining budgets and verifying the role of finetuning data significantly strengthen the rigor of our paradigm comparison.

1. Controlled comparison from 0 to 150B pretraining tokens.
We scaled all three matched 1.6B architectures from Table 4 up to 150B pretraining tokens (40K to 240K steps) under identical data and schedules:

  • Pretraining GenEval (%):
Architecture (1.6B)40K (25B)80K (50B)120K (75B)160K (100B)200K (125B)240K (150B)
Multimodal Flow8.3420.6222.6023.0324.0025.01
Transfusion-style12.9514.2016.7217.3719.3420.07
Chameleon-style0.732.123.454.434.785.39
  • Pretraining CLIPScore (%):
Architecture (1.6B)40K (25B)80K (50B)120K (75B)160K (100B)200K (125B)240K (150B)
Multimodal Flow81.9282.4182.5982.8382.9582.97
Transfusion-style81.5482.1182.2082.3782.4682.34
Chameleon-style66.2871.2072.8773.9274.8775.17

Multimodal Flow maintains a clear, consistent lead from 80K through 240K steps (150B tokens) on both metrics, confirming that Table 4 is not an artifact of baseline undertraining at 50B tokens.

2. GenEval prompt overlap and role of finetuning.
Checking our public T2I finetuning sets (BLIP3o-60k, LAION DALL-E 3, ShareGPT-4o-Image) against all 553 GenEval test prompts confirms zero exact match overlap. Moreover, training a randomly initialized 1.6B model solely on this finetuning mixture for a matched 155B token budget (Table 5) reaches only 52.7% on GenEval:

Initialization (155B Total Tokens)GenEval ↑DPG-Bench ↑SEEDB ↑MMB ↑OK-VQA ↑
Random Initialization (155B FT only)52.757.8131.636.022.9
Mixed Pretraining (150B PT + 5B FT)82.183.4462.467.239.1

This confirms that the SFT data is not tailored to GenEval, and that MF-1's final performance relies on capabilities learned during mixed pretraining together with SFT instruction alignment.


Thank you again for your valuable comments on Multimodal Flow! We hope our new experiments and clarifications resolve your concerns, and we welcome any further discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chairs and Reviewers,

We would like to express our sincere gratitude to the Area Chairs for coordinating the review process, and to Reviewers W8aA, u68g, and Dgcb for the considerable time, care, and expertise they devoted to evaluating our paper. Their insightful questions and constructive suggestions inspired a series of deeper controlled experiments and analyses during the rebuttal period, which have made our work much more thorough and complete.

We are deeply encouraged by the reviewers' shared positive recognition of our core contributions: exploring a promising new direction for unified multimodal modeling beyond discrete tokenization and hybrid paradigms (Dgcb); establishing a genuinely unified continuous formulation in which language and vision, across both understanding and generation, share a single Flow Matching objective and sampling procedure—switching tasks simply by reordering hyperchunks (W8aA, u68g, Dgcb); and presenting the framework clearly and transparently alongside comprehensive, carefully controlled empirical validation up to 1.6B parameters (u68g, Dgcb). Below is a concise synthesis of our new rebuttal findings addressing the main discussion themes (with full data tables provided in our individual responses):

  1. Language & Multimodal Understanding (Reviewers W8aA, u68g)
    Across five standard pure-language benchmarks (MMLU, OBQA, RACE, ARC-C, SIQA), MF-1 outperforms both recent continuous diffusion LMs and our strictly matched 1.6B autoregressive baseline (27.8% vs. 21.4% avg.). Moreover, VQA gaps relative to LLM-initialized models (e.g., Janus-Pro) stem from a ~10×10\times pretraining token difference (150B from scratch vs. 1T–2T pretrained LLM backbones) rather than continuous modeling; MF-1 nearly matches Janus-Pro on POPE and GQA while outperforming larger-budget from-scratch unified models.

  2. Chunked Text Formulation & Boundary Fidelity (Reviewers W8aA, u68g)
    Pretraining ablations across chunk sizes B∈{4,8,16}B \in \{4, 8, 16\} converge to virtually identical alignment and generation quality, confirming robustness to chunk granularity. Clean-latent reconstruction over 1M+ tokens achieves >99.98% exact whole-word accuracy even for multi-subword words split across chunk boundaries, while chunk-causal attention preserves full cross-chunk semantic context and <EOS> truncation naturally supports variable-length outputs.

  3. Training Cost & Inference Speed (Reviewers W8aA, u68g)
    All three matched 1.6B paradigms share identical training backbone FLOPs and comparable GPU hours. At inference, within-chunk parallelism amortizes KV-cache memory bandwidth: reducing text sampling to 4 SDE steps matches AR FLOPs while achieving 1.8×1.8\times–3.4×3.4\times faster wall-clock decoding than the matched 1.6B AR baseline with virtually unchanged VQA accuracy, alongside sub-0.5s image generation on an RTX 5090.

  4. Scaling, Data Efficiency & Architectural Attribution (Reviewers W8aA, u68g, Dgcb)

    • 150B & 3B Scaling: Extending our controlled 1.6B comparison to 150B pretraining tokens shows Multimodal Flow consistently leading from 50B onward—surpassing the 150B hybrid baseline using only 50B tokens—while a preliminary 3B matched SFT comparison retains a +11.6% average VQA lead.
    • Fair Attribution: Augmenting the hybrid baseline with frozen T5 conditioning (Transfusion-T5, 150B tokens) brings minimal change (+0.8% GenEval, still 4.2% behind Multimodal Flow), and checking all 553 GenEval prompts confirms zero SFT overlap, verifying that the gains come genuinely from unified continuous Flow Matching.

We have incorporated these supplementary experiments, ablations, and clarifications into our revised manuscript, with all updates highlighted in blue for easy reference. Thank you again to all Reviewers and Area Chairs for your thoughtful engagement and guidance in helping us strengthen this paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.