acceptodds
Under review as a conference paper at ICLR 2027

ModalVerse: Multimodal Autoregression in Parallel Inference

Abstract

Multimodal autoregressive models commonly serialize different modalities into a single sequence, coupling modality-local causal progression with cross-modal interaction. We introduce ModalVerse, a framework that decouples these two operations to enable parallel yet cooperative multimodal inference. ModalVerse organizes modality sequences into parallel lanes that share an autoregressive backbone while maintaining separate causal histories. Within each layer, modality-local updates proceed concurrently, followed by cross-lane attention that exchanges information across modalities. Moreover, to sustain interaction when output lengths differ, continuation tokens keep completed lanes participating in cross-modal communication after visible generation ends. The framework supports two complementary settings: parallel multimodal input processing, where distributed evidence is integrated into a shared prediction, and concurrent multimodal output generation, where heterogeneous outputs advance independently while remaining coordinated. On MUSIC-AVQA, balanced parallel input lanes achieve up to faster decoder prefill and reduce peak memory by up to while maintaining comparable question-answering accuracy. On adapted A-OKVQA, ModalVerse improves joint text–speech accuracy from with independent parallel generation to , closely matching with concatenated generation. These results show that multimodal autoregression need not serialize modalities into a single sequence: modality-specific computation can proceed in parallel while preserving effective cross-modal interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Omni-Modal Models, Parallel Multimodal Generation, Parallel Multimodal Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BFVNOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

ModalVerse puts modality sequences into parallel lanes that share an autoregressive backbone. Each lane attends to its own history and exchanges information with other lanes at the same step. Lanes that have finished their visible output continue computing to support longer outputs. The paper tests distributed audio-visual inputs on MUSIC-AVQA and concurrent text-label and spoken-answer generation on adapted A-OKVQA. It reports lower decoder-prefill cost with balanced input lanes and better joint output accuracy than independent parallel generation.

Critical strengths
  1. The three-lane model reaches 78.32% accuracy, compared with 79.69% for the parameter-matched concatenation baseline, while reducing prefill cost in the reported setup. Since accuracy is measured on all 5,096 test questions, this provides useful evidence that splitting a long multimodal context across lanes can retain most task performance. The length and batch-size sweep also helps show how the trade-off changes with input size.

  2. Moving from the separately trained Independent Parallel model to ModalVerse raises joint text–speech accuracy from 29.07% to 38.26% . This suggests that letting the lanes exchange information has practical value. The communication and continuation ablations add some support for that interpretation, even though they do not yet separate all of the design choices.

Critical weaknesses
  1. The timing protocol in Section 5.1 measures decoder-only prefill on the first one or four test examples, excludes media encoding, and uses SDPA math. The additional experiments cover more synthetic lengths and batches, but still use the same backend. The gain is clear under these conditions, but it remains uncertain how much would carry over to optimized kernels and a representative mix of real inputs. There are also no output-generation timings, time-to-first-speech results, or peak KV memory measurements. These seem particularly important here because completed lanes keep computing and accumulating KV entries.

  2. Both parallel models are trained for ten epochs and select epoch six, but Independent Parallel removes communication and continuation together and uses a different warmup: 160 rather than 320 steps. These differences make it difficult to determine how much each component contributes to the joint-accuracy gain. The inference-time switches show that the trained model uses these components, but do not answer what happens when each alternative is trained under comparable conditions. Native Concat's five epochs are not by themselves a concern, since the authors report convergence; a comparison that separates the proposed additions while controlling training is still missing.

Questions for authors
  1. Timings with an optimized attention backend on representative real inputs would help assess the practical inference benefit. Please include output-generation latency, time to first speech, and KV memory with continuation enabled.

  2. What joint accuracies do separately trained communication-only, continuation-only, and full models achieve under controlled schedules or matched computation? This would clarify which additions account for the cooperative gain.

  3. Is there an independent final evaluation after checkpoint selection on the 1,137 question validation set? The reported results appear to use that same set. How uncertain is the 0.42 shortfall from Native Concat when resampling original questions rather than their four correlated option orderings?

Reviewer_KC9XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose ModalVerse to process different modalities in parallel lanes. Each lane has its own causal history. Attention between lanes exchanges information after each decoder layer. Continuation tokens keep a completed output lane active while another lane continues generating.

On MUSIC-AVQA, the three-lane model reaches 78.32% answer accuracy. At batch size four, it reports 1.51 times faster decoder prefill and 34.46% lower peak memory than the parameter-matched concatenation baseline. On an adapted A-OKVQA task, joint text and speech accuracy is 38.26%. The corresponding figures are 29.07% for independent parallel generation and 38.68% for concatenated generation.

Critical strengths

The method addresses both parallel input processing and parallel output generation. The shared backbone, attention between lanes, and continuation tokens form a coherent design. This makes the contribution broader than an input-only prefill technique.

ModalVerse exceeds independent parallel generation by 9.19 percentage points. Its result is also within 0.42 percentage points of concatenated generation. This supports the value of communication between output lanes, subject to the comparison concern below.

Critical weaknesses

The reported speedup uses precomputed modality features and measures decoder prefill on only the first one or four test examples. It forces the PyTorch attention math backend. In the appendix, three lanes are slower than concatenation at 1,536 tokens: 89.70 versus 82.39 milliseconds. There is no measured output decoding latency. Yet continuation tokens keep the completed lane computing and adding cache entries during speech generation. FlashAttention-3 [1] targets efficient attention on Hopper hardware. MMInference [2] provides optimized multimodal prefill. Neither work is cited or evaluated as a relevant efficiency comparison. Thus, the reported 1.51 times speedup may depend strongly on the chosen backend. arxiv.org

In adapted A-OKVQA, text is an answer option label, and speech recites that option. The 4,548 evaluation instances come from four orderings of only 1,137 distinct questions. This does not test open-ended text and speech coordination. ModalVerse also uses 320 warmup steps, while the independent baseline uses 160. The 9.19 point difference therefore cannot be attributed solely to attention between lanes. Disabling that attention only at inference changes a model trained to use it; this is not a training-matched ablation. Qwen3-Omni [3], which reports streaming speech and first-packet latency, is a relevant omitted reference for framing the output claim.、

[1] Shah, J., et al. “FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision.” arXiv:2407.08608, 2024.

[2] Li, Y., et al. “MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention.” Proceedings of the 42nd International Conference on Machine Learning, 2025, pp. 34998–35020.

[3] Xu, J., et al. “Qwen3-Omni Technical Report.” arXiv:2509.17765, 2025.

Questions for authors

These are clarification requests that could change the assessment.

  1. What are the full input-to-output latency, peak memory, and time to first speech packet for ModalVerse and concatenation when both use the same optimized attention backend? How much compute is spent on continuation tokens after the text lane ends?

  2. Does the joint accuracy gain persist when ModalVerse and independent parallel generation use identical training schedules? What evidence shows that it persists for open-ended responses rather than option labels and recited option text?

Reviewer_xhDQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ModalVerse. It separates modality-local autoregressive computation from cross-modal interaction. Modality sequences are organized into parallel lanes that share a backbone while maintaining separate causal histories. Cross-lane attention exchanges current hidden states after each decoder block, and continuation tokens keep completed lanes participating in computation until the remaining outputs finish.

The framework supports both distributed multimodal input processing and concurrent multimodal output generation. On MUSIC-AVQA, a balanced three-lane layout reduces decoder prefill latency and peak memory with a modest accuracy decrease. On adapted A-OKVQA, ModalVerse improves joint text–speech accuracy over independent parallel generation and approaches the concatenated baseline.

Critical strengths
  1. This paper is well-motivated and proposes a coherent framework. Separating modality-local processing from cross-modal communication provides a reasonable way to accommodate different contexts and output lengths. Supporting both input aggregation and concurrent output generation within this framework is appealing.

  2. The proposed method gets a meaningful improvement over independent generation. In Table 2, joint text–speech accuracy increases from 29.07% for Independent Parallel to 38.26% for ModalVerse, close to Native Concat at 38.68%. This is useful evidence that communication can improve jointly correct multimodal outputs.

  3. The paper includes communication ablations, continuation-token evaluations at multiple checkpoints, and a controlled prefill sweep over context length and batch size. Keeping the sampled video frames unchanged across input layouts also strengthens the input-side comparison.

Critical weaknesses
  1. The incremental contribution beyond existing cross-sequence communication is not sufficiently isolated. The paper appropriately acknowledges Bridge (Dong et al., 2026) in Section 3.2. Bridge already exchanges current-step hidden states across parallel sequences through attention inserted after feed-forward blocks, with input normalization and residual connections. ModalVerse's distinctive elements therefore lie primarily in heterogeneous modality/role conditioning, distributed-input aggregation, lane balancing, and continued computation after output completion. However, the experiments do not sufficiently establish which extensions are responsible for the benefits beyond introducing communication itself. Modality/role embeddings are not ablated. Table 4 does isolate the inference-time effect of continuation and shows consistent improvements, including a 0.97-percentage-point increase in joint accuracy at epoch 6. What remains unclear is whether updating completed lanes is preferable to simpler alternatives, such as continuing to expose their terminal representations, and at what computational cost. Consequently, the substantial gain over Independent Parallel supports the usefulness of communication, but provides less direct evidence for the value of the paper-specific additions. A focused comparison against a minimally adapted communication baseline, or equivalent component-wise controls, would materially strengthen the novelty claim.

  2. The efficiency evidence is useful but limited in scope, particularly for output generation. Section 5.1 measures decoder-only prefill using precomputed audio/video features and the SDPA math backend. The real-input timing experiments use the first test example for batch size 1 and the first four examples for batch size 4. Appendix C provides a valuable synthetic scaling study, but does not establish performance across the distribution of real input lengths or with optimized attention implementations. Figure 4 also shows that the two-lane model is slower and more memory-intensive than Concat P-Match at batch size 4. The balanced three-lane layout produces the reported advantage. This is a useful finding rather than a contradiction, but it makes the conditions under which lane partitioning is beneficial important to characterize. On the output side, the paper reports accuracy but no time-to-first-audio, total generation latency, or throughput. Section 4.2 appropriately presents streaming as a potential benefit, so this is an evidence gap rather than a demonstrated false claim. Nevertheless, output-side measurements are important for assessing the practical value of the framework, especially because continuation tokens keep completed lanes executing the backbone and extending their KV caches.

  3. The output evaluation and experimental controls leave uncertainty about generality and robustness. The adapted A-OKVQA task in Section 5.2 requires a text option label and speech containing the corresponding answer. This is a sensible test of answer coordination, but the short text output provides limited evidence for sustained interaction between longer modality sequences. Likewise, the ASR-based exact-match metric in Appendix B.1 evaluates answer content but not establish broader speech-generation quality. These limitations constrain the demonstrated scope rather than invalidate the reported accuracy gains. The comparison also uses different training schedules: Native Concat converges after five epochs, the parallel methods select epoch six, and ModalVerse and Independent Parallel use 320 and 160 warmup steps, respectively. Different convergence behavior does not by itself make the comparison unfair, but an explanation of schedule selection or a controlled sensitivity check would help establish that the conclusions are not schedule-dependent. Finally, Tables 1 and 3 measure reliance on communication by disabling it in an already trained model. Table 2 appropriately includes a separately trained independent baseline, but the inference-time drops should not be interpreted as the achievable gap to a trained alternative. No variability estimates are reported for the smaller differences. Since the 4,548 evaluation instances come from four orderings of 1,137 questions, uncertainty estimates should account for dependence within each question.

Questions for authors
  1. Can the authors clarify the minimal changes needed beyond Bridge-style communication and provide focused controls for modality/role embeddings and continuation?
  2. How do the input-side gains change with representative real workloads and optimized attention backends? What are the output-side latency and memory costs, including continuation?
  3. How were the differing training schedules selected, and how robust are the principal results to this choice and to repeated evaluation or training runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.