- Rating
- 1
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
ModalVerse puts modality sequences into parallel lanes that share an autoregressive backbone. Each lane attends to its own history and exchanges information with other lanes at the same step. Lanes that have finished their visible output continue computing to support longer outputs. The paper tests distributed audio-visual inputs on MUSIC-AVQA and concurrent text-label and spoken-answer generation on adapted A-OKVQA. It reports lower decoder-prefill cost with balanced input lanes and better joint output accuracy than independent parallel generation.
Critical strengths
-
The three-lane model reaches 78.32% accuracy, compared with 79.69% for the parameter-matched concatenation baseline, while reducing prefill cost in the reported setup. Since accuracy is measured on all 5,096 test questions, this provides useful evidence that splitting a long multimodal context across lanes can retain most task performance. The length and batch-size sweep also helps show how the trade-off changes with input size.
-
Moving from the separately trained Independent Parallel model to ModalVerse raises joint text–speech accuracy from 29.07% to 38.26% . This suggests that letting the lanes exchange information has practical value. The communication and continuation ablations add some support for that interpretation, even though they do not yet separate all of the design choices.
Critical weaknesses
-
The timing protocol in Section 5.1 measures decoder-only prefill on the first one or four test examples, excludes media encoding, and uses SDPA math. The additional experiments cover more synthetic lengths and batches, but still use the same backend. The gain is clear under these conditions, but it remains uncertain how much would carry over to optimized kernels and a representative mix of real inputs. There are also no output-generation timings, time-to-first-speech results, or peak KV memory measurements. These seem particularly important here because completed lanes keep computing and accumulating KV entries.
-
Both parallel models are trained for ten epochs and select epoch six, but Independent Parallel removes communication and continuation together and uses a different warmup: 160 rather than 320 steps. These differences make it difficult to determine how much each component contributes to the joint-accuracy gain. The inference-time switches show that the trained model uses these components, but do not answer what happens when each alternative is trained under comparable conditions. Native Concat's five epochs are not by themselves a concern, since the authors report convergence; a comparison that separates the proposed additions while controlling training is still missing.
Questions for authors
-
Timings with an optimized attention backend on representative real inputs would help assess the practical inference benefit. Please include output-generation latency, time to first speech, and KV memory with continuation enabled.
-
What joint accuracies do separately trained communication-only, continuation-only, and full models achieve under controlled schedules or matched computation? This would clarify which additions account for the cooperative gain.
-
Is there an independent final evaluation after checkpoint selection on the 1,137 question validation set? The reported results appear to use that same set. How uncertain is the 0.42 shortfall from Native Concat when resampling original questions rather than their four correlated option orderings?