- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Omni-Instruct, a benchmark for instruction following in any-to-any multimodal systems. It distinguishes instruction understanding, output-modality routing, and compliance with fine-grained requirements. The benchmark comprises 1,800 instances derived from 600 base requests, each delivered through text, image, or audio instructions. Six models are evaluated using routing accuracy (RA), conditional compliance (CC), and end-to-end (E2E) scores. The results reveal substantial differences across instruction modalities and show why routing failures merit separate analysis.
Critical strengths
-
The paper addresses a relevant evaluation gap. Existing multimodal benchmarks often emphasize response quality or assume a fixed output channel; here, the requested output modality is itself part of the instruction.
-
Matching the underlying requests across text, image, and audio instruction carriers makes the comparisons more informative by controlling for differences in task content.
-
Reporting RA, CC, and E2E separately helps distinguish incorrect output modalities from failures to satisfy content constraints. This is more diagnostic than a single aggregate score.
-
The study goes beyond model rankings: it compares six systems, calibrates an automatic judge against human labels, reports uncertainty estimates, and examines several possible sources of instruction-modality gaps.
Critical weaknesses
-
Routing accuracy conflates several failure sources. A missing or invalid artifact counts as a routing failure, which is appropriate for end-to-end evaluation but does not establish whether the model misunderstood the instruction or its output interface and generation pipeline failed.
-
The human calibration in Appendix D covers text-delivered instructions only, despite the paper’s emphasis on image- and audio-delivered instructions. It also does not validate every subsequently revised rubric item. This limits confidence in cross-carrier compliance comparisons.
-
CC is conditioned on successful routing, so its value may reflect different subsets of tasks for different models. E2E addresses overall performance, but direct cross-model interpretations of CC remain vulnerable to selection effects.
-
Although instruction and context modalities can differ, the benchmark does not systematically cover interleaved instructions or multi-turn interaction. The probes in Section 5.5 are useful, but they do not fully quantify the contribution of perceptual errors to the remaining performance gap.
Questions for authors
-
Could the authors measure each model’s basic ability to produce a valid artifact in every target modality under short, explicit prompts? This would help separate generation/interface limitations from instruction-following failures.
-
Appendix D notes that judge calibration uses text-delivered instructions. Could the authors extend human validation to image and audio instruction carriers, including the revised rubric items, and report agreement by carrier?
-
The routing rule permits accompanying text for image and audio targets. How sensitive are the RA and E2E results to requiring an exact output-modality match?
-
Could the authors compare CC on a shared subset of correctly routed instances, or otherwise control for differences in instance difficulty across models?
-
Section 5.5 already studies cleaner carriers, localization hints, and instruction reporting. Could an oracle OCR/transcription control further estimate the contribution of perception errors?6. Will the authors release the benchmark items, rubrics, judge prompts, and model-specific inference adapters needed to reproduce the evaluation?
5 comments
4. (W4, Q5) Interaction scope and instruction-text control
Multi-turn and interleaved instructions. We agree that these are important evaluation settings, and Omni-Instruct does not currently cover multi-turn interaction or interleaved context. Our focus is on a complementary axis. Existing multi-image and interleaved benchmarks [1–3] typically interleave modalities within the context, while the instruction itself is given as text. In contrast, Omni-Instruct separates the instruction carrier from the context and varies their modalities independently. This allows us to evaluate how models respond to user queries composed of diverse modality combinations, including cases where the instruction itself is delivered through a non-text modality.
Section 5.5 also includes a simple form of interleaved instruction. While retaining the original image or audio instruction, we added a text cue indicating where the instruction is located, so that the directives are distributed across modalities (Appendix G.2). This cue yielded negligible E2E changes (−0.13 and +0.18 pp for image and audio instructions), suggesting that a textual directive alone does not help models act on an instruction delivered through another modality.
As this is a preliminary probe, extending Omni-Instruct to more diverse instruction scenarios is a valuable direction for future work. Examples include instructions whose content is split across modalities, conflicting directives, and multi-turn interactions.
Oracle OCR/transcription control for perception errors. We retained the original image/audio instruction and additionally supplied the oracle text. The study covers 72 requests in two non-text instruction carriers, yielding 144 outputs per model. The following table reports the original image/audio-instruction condition and the source-text-added condition:
| Model | RA: baseline → +text | CC: baseline → +text | E2E: baseline → +text | Δ E2E [95% CI], pp |
|---|---|---|---|---|
| AnyGPT | 38.19 → 52.78 | 40.61 → 36.50 | 15.51 → 19.27 | +3.75 [−0.54, +8.14] |
| MIO | 34.72 → 36.81 | 21.57 → 30.73 | 7.49 → 11.31 | +3.82 [+0.83, +7.06] |
| HCX-Omni | 37.50 → 48.61 | 41.42 → 57.45 | 15.53 → 27.93 | +12.39 [+5.44, +19.34] |
| ModaVerse | 34.03 → 68.75 | 21.93 → 39.76 | 7.46 → 27.33 | +19.87 [+14.32, +25.70] |
| AR-Omni | 25.00 → 29.86 | 12.13 → 14.65 | 3.03 → 4.37 | +1.34 [−0.78, +3.66] |
| NExT-GPT | 31.94 → 20.14 | 14.37 → 31.53 | 4.59 → 6.35 | +1.76 [−0.81, +4.69] |
Providing the instruction text alongside the original carrier improves E2E performance most for ModaVerse and HCX-Omni, providing an approximate indication of the contribution of perception errors. However, substantial errors remain even when the instruction is explicitly available in text, suggesting that difficulties extend beyond recovering the instruction from its original carrier.
[1] Liu et al. Holistic Evaluation for Interleaved Text-and-Image Generation. EMNLP 2024.
[2] Xia et al. MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models. ICLR 2025.
[3] Li et al. UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark. CVPR 2026.
5. (Q1) Basic artifact production under short, explicit prompts
As suggested, we tested each model using short, explicit 40 prompts to generate image/audio, such as: "Generate speech audio saying: Hello.", "Show me a famous landmark in NYC.", "Create a picture.", and "Can you generate an image of a red apple?".
The table compares these basic generation rates with routing accuracy on the corresponding subset of Omni-Instruct. Specifically, we measure RA on the text-to-image and text-to-audio tasks without context.
| Model | Basic generation (%) | Omni-Instruct (%) |
|---|---|---|
| AnyGPT | 100.0 | 66.0 |
| MIO | 52.5 | 29.0 |
| HCX-Omni | 85.0 | 65.0 |
| ModaVerse | 100.0 | 99.0 |
| AR-Omni | 45.0 | 3.0 |
| NExT-GPT | 50.0 | 4.0 |
The gap is particularly large for AR-Omni and NExT-GPT, whose basic generation rates reach 45.0% and 50.0%, respectively, but fall to benchmark routing accuracies of 3.0% and 4.0%. All six models can produce both image and audio artifacts in the basic controls, confirming that their benchmark routing failures cannot be explained solely by an absence of the corresponding generation capability.
Nevertheless, several models remained unreliable even under simple requests, indicating that generation or interface limitations also contribute. Benchmark routing accuracy was lower than basic generation success for every model, although the gap varied substantially. We therefore interpret RA as the evaluated system's ability to deliver the requested modality, with failures potentially arising from both generation/interface limitations and instruction following.
6. (Q3) Sensitivity to disallowing accompanying text
We recomputed RA and E2E under the suggested, stricter rule that disallows accompanying text for image/audio targets, after excluding media/control syntax and intermediate generation prompts.
| Model | RA: default → strict | E2E: default → strict |
|---|---|---|
| AnyGPT | 44.33 → 21.89 | 18.64 → 8.11 |
| MIO | 37.33 → 37.17 | 13.26 → 13.23 |
| HCX-Omni | 43.06 → 25.22 | 22.90 → 15.01 |
| ModaVerse | 52.44 → 52.44 | 14.37 → 14.37 |
| AR-Omni | 28.06 → 24.11 | 3.91 → 3.20 |
| NExT-GPT | 33.50 → 33.00 | 7.10 → 6.94 |
As seen in the table above, disallowing accompanying text substantially drops performance for some models, particularly AnyGPT and HCX-Omni. To understand these drops, we further examined the accompanying text in the outputs that passed default routing but failed the stricter rule. The table below summarizes selected text types and representative examples.
| Text type | # Cases | Models with observed cases | Examples |
|---|---|---|---|
| Acknowledgement / presentation | 211 | AnyGPT, HCX-Omni, AR-Omni, NExT-GPT | "Sure, I can help you with that."<br>"Sure, here is the image you requested." |
| Duplicated transcript | 161 | HCX-Omni | "Four dogs are running towards the camera in a grassy field." |
| Instruction repetition | 153 | AnyGPT, HCX-Omni, AR-Omni | "Create an image with a plant beside a chair on a bedroom floor." |
| Tool narration | 7 | HCX-Omni | "Using the t2i_model_generation tool" |
| Punctuation | 3 | MIO | "." |
These cases show that models use their final output fields differently, sometimes retaining media placeholders or transcripts alongside the requested artifact. Acknowledgements such as "Sure, here is the image you requested" also naturally accompany an artifact delivery. We therefore conclude that allowing accompanying text is more appropriate for measuring delivery of the requested modality, but report the stricter results separately in Appendix F.6.
7. (Q6) Benchmark and code release
We uploaded the requested benchmark items, final rubrics, evaluation code, model-specific inference adapters and settings, on https://anonymous-hf.com/a/g6o0a5rllup2/. The full judge prompt is also provided in Appendix C.2. We will further improve the documentation and user instructions to facilitate reproduction.
1. (W1) Observable sources of routing failure
We thank the reviewer for this insightful comment. We agree that routing failure can arise at different stages and does not by itself identify instruction misunderstanding. To distinguish observable failure types, we audited the saved model streams and artifacts for all 5,625 routing failures on image/audio-target requests. The table below further breaks down the observed failures; percentages are relative to each model's total number of failures.
As shown in the table, most failures involved a missing requested-media artifact: 5,415 of the 5,625 failed media requests (96.3%) returned text only or another modality without observed initiation of the requested media. In another 128 cases (2.3%), the requested media was observed but no valid artifact was delivered. AnyGPT and MIO produced 16 and 32 incomplete audio streams, respectively, resulting in decoding errors. AR-Omni produced 80 readable audio files only 25–50 ms long, failing the validity criterion.
Nevertheless, most failures produced the wrong output modality, failing to follow the user's explicit instruction about which modality to generate. They therefore constitute routing failures, even though the output records alone cannot establish whether the cause was misunderstanding or an execution failure.
2. (W2, Q2) Human agreement with multimodal instructions under the final rubric
We agree that validating the final rubrics on non-text instruction carriers would strengthen our paper. Following the comment, three human annotators additionally evaluated 120 outputs from audio- and image-delivered instructions (60 per carrier; 627 content atoms under the final rubrics).
Agreement is numerically higher for audio- and image-delivered instructions than for text-delivered instructions, supporting the applicability of our judge to non-text instruction carriers. We have added these results to Appendix D (Table A4).
3. (W3, Q4) Selection effects in conditional compliance
We thank the reviewer for highlighting this issue. We agree that Conditional Compliance is conditioned on successful routing and may therefore be computed on different subsets for different models. Following the reviewer's suggestion, we evaluated all six models on the same 133 instances that they all routed successfully. The table below compares these scores with overall CC and with CC on separately sampled inputs matched to the shared subset's task and instruction-modality composition.
Shared CC differs from overall CC. On separate inputs matched to its coarse task and instruction-carrier composition, Matched CC is within 0.8–2.3 percentage points of Shared CC for four models, with larger differences for MIO (3.57 points) and ModaVerse (7.30 points). Coarse matching does not control instance difficulty or make the successfully routed subsets identical.
These results highlight the importance of accounting for subtask and instruction-modality composition when comparing CC. Finer task-level breakdowns provide more informative comparisons, although differences in the successfully routed instances can remain. We add Appendix F.3 reporting these breakdowns together with routing-success counts.
In our framework, CC is the conditional component of the decomposition E2E = RA × CC, rather than an independent measure for ranking model capabilities. We therefore use RA and E2E for cross-model comparisons of modality delivery and overall compliance, respectively, while interpreting CC as a diagnostic within the evaluated task groups.