acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Any-to-Any Instruction Following with Omni-modal Instruction

Abstract

Any-to-any models are expected to follow user instructions and respond in multiple modalities. In this setting, instruction following is not limited to satisfying content or format constraints, but also requires understanding how the instruction is conveyed. To extensively assess this capability of any-to-any systems, we introduce a novel benchmark, Omni-Instruct, which evaluates instruction following in any-to-any models with various context, instruction and output modalities across text, image, and audio. Using our proposed framework, we extensively evaluate recent any-to-any models, decoupling the output-modality routing from instruction compliance. From our study, we discover that 1) routing failure is the dominant factor for most apparent performance drops, 2) even when correctly routed, models show distinct compliance failure patterns, and 3) models remain vulnerable to omnimodal instructions, struggling to understand and follow requests conveyed through images or speech.

Primary area:
datasets and benchmarks
Keywords:
Multimodal, Any-to-Any, Instruction following, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DPeiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Omni-Instruct, a benchmark for instruction following in any-to-any multimodal systems. It distinguishes instruction understanding, output-modality routing, and compliance with fine-grained requirements. The benchmark comprises 1,800 instances derived from 600 base requests, each delivered through text, image, or audio instructions. Six models are evaluated using routing accuracy (RA), conditional compliance (CC), and end-to-end (E2E) scores. The results reveal substantial differences across instruction modalities and show why routing failures merit separate analysis.

Critical strengths
  1. The paper addresses a relevant evaluation gap. Existing multimodal benchmarks often emphasize response quality or assume a fixed output channel; here, the requested output modality is itself part of the instruction.

  2. Matching the underlying requests across text, image, and audio instruction carriers makes the comparisons more informative by controlling for differences in task content.

  3. Reporting RA, CC, and E2E separately helps distinguish incorrect output modalities from failures to satisfy content constraints. This is more diagnostic than a single aggregate score.

  4. The study goes beyond model rankings: it compares six systems, calibrates an automatic judge against human labels, reports uncertainty estimates, and examines several possible sources of instruction-modality gaps.

Critical weaknesses
  1. Routing accuracy conflates several failure sources. A missing or invalid artifact counts as a routing failure, which is appropriate for end-to-end evaluation but does not establish whether the model misunderstood the instruction or its output interface and generation pipeline failed.

  2. The human calibration in Appendix D covers text-delivered instructions only, despite the paper’s emphasis on image- and audio-delivered instructions. It also does not validate every subsequently revised rubric item. This limits confidence in cross-carrier compliance comparisons.

  3. CC is conditioned on successful routing, so its value may reflect different subsets of tasks for different models. E2E addresses overall performance, but direct cross-model interpretations of CC remain vulnerable to selection effects.

  4. Although instruction and context modalities can differ, the benchmark does not systematically cover interleaved instructions or multi-turn interaction. The probes in Section 5.5 are useful, but they do not fully quantify the contribution of perceptual errors to the remaining performance gap.

Questions for authors
  1. Could the authors measure each model’s basic ability to produce a valid artifact in every target modality under short, explicit prompts? This would help separate generation/interface limitations from instruction-following failures.

  2. Appendix D notes that judge calibration uses text-delivered instructions. Could the authors extend human validation to image and audio instruction carriers, including the revised rubric items, and report agreement by carrier?

  3. The routing rule permits accompanying text for image and audio targets. How sensitive are the RA and E2E results to requiring an exact output-modality match?

  4. Could the authors compare CC on a shared subset of correctly routed instances, or otherwise control for differences in instance difficulty across models?

  5. Section 5.5 already studies cleaner carriers, localization hints, and instruction reporting. Could an oracle OCR/transcription control further estimate the contribution of perception errors?6. Will the authors release the benchmark items, rubrics, judge prompts, and model-specific inference adapters needed to reproduce the evaluation?

5 comments

1. (W1) Observable sources of routing failure

We thank the reviewer for this insightful comment. We agree that routing failure can arise at different stages and does not by itself identify instruction misunderstanding. To distinguish observable failure types, we audited the saved model streams and artifacts for all 5,625 routing failures on image/audio-target requests. The table below further breaks down the observed failures; percentages are relative to each model's total number of failures.

ModelFailed requestsText onlyEmpty outputWrong modality onlyExtra modalityInvalid requested media
AnyGPT603265 (43.9%)0 (0.0%)304 (50.4%)18 (3.0%)16 (2.7%)
MIO1056927 (87.8%)22 (2.1%)75 (7.1%)0 (0.0%)32 (3.0%)
HCX-Omni798678 (85.0%)0 (0.0%)120 (15.0%)0 (0.0%)0 (0.0%)
ModaVerse850808 (95.1%)0 (0.0%)42 (4.9%)0 (0.0%)0 (0.0%)
AR-Omni1128899 (79.7%)2 (0.2%)145 (12.9%)2 (0.2%)80 (7.1%)
NExT-GPT11901151 (96.7%)38 (3.2%)1 (0.1%)0 (0.0%)0 (0.0%)
Total56254728 (84.0%)62 (1.1%)687 (12.2%)20 (0.4%)128 (2.3%)

As shown in the table, most failures involved a missing requested-media artifact: 5,415 of the 5,625 failed media requests (96.3%) returned text only or another modality without observed initiation of the requested media. In another 128 cases (2.3%), the requested media was observed but no valid artifact was delivered. AnyGPT and MIO produced 16 and 32 incomplete audio streams, respectively, resulting in decoding errors. AR-Omni produced 80 readable audio files only 25–50 ms long, failing the validity criterion.

Nevertheless, most failures produced the wrong output modality, failing to follow the user's explicit instruction about which modality to generate. They therefore constitute routing failures, even though the output records alone cannot establish whether the cause was misunderstanding or an execution failure.

2. (W2, Q2) Human agreement with multimodal instructions under the final rubric

We agree that validating the final rubrics on non-text instruction carriers would strengthen our paper. Following the comment, three human annotators additionally evaluated 120 outputs from audio- and image-delivered instructions (60 per carrier; 627 content atoms under the final rubrics).

Instruction modalityOutputsRubric atomsAgreement ↑Cohen's κ ↑ICC(A,1) ↑
Text12057589.0%0.7660.849
Audio6031491.1%0.7630.835
Image6031393.3%0.8200.880

Agreement is numerically higher for audio- and image-delivered instructions than for text-delivered instructions, supporting the applicability of our judge to non-text instruction carriers. We have added these results to Appendix D (Table A4).

3. (W3, Q4) Selection effects in conditional compliance

We thank the reviewer for highlighting this issue. We agree that Conditional Compliance is conditioned on successful routing and may therefore be computed on different subsets for different models. Following the reviewer's suggestion, we evaluated all six models on the same 133 instances that they all routed successfully. The table below compares these scores with overall CC and with CC on separately sampled inputs matched to the shared subset's task and instruction-modality composition.

ModelOverall CC ↑Shared CC (133) ↑Matched CC ↑
HCX-Omni53.1971.6472.48
AnyGPT42.0442.1140.39
MIO35.5347.8644.29
ModaVerse27.3936.8229.52
NExT-GPT21.2032.5331.22
AR-Omni13.9411.1113.45

Shared CC differs from overall CC. On separate inputs matched to its coarse task and instruction-carrier composition, Matched CC is within 0.8–2.3 percentage points of Shared CC for four models, with larger differences for MIO (3.57 points) and ModaVerse (7.30 points). Coarse matching does not control instance difficulty or make the successfully routed subsets identical.

These results highlight the importance of accounting for subtask and instruction-modality composition when comparing CC. Finer task-level breakdowns provide more informative comparisons, although differences in the successfully routed instances can remain. We add Appendix F.3 reporting these breakdowns together with routing-success counts.

In our framework, CC is the conditional component of the decomposition E2E = RA × CC, rather than an independent measure for ranking model capabilities. We therefore use RA and E2E for cross-model comparisons of modality delivery and overall compliance, respectively, while interpreting CC as a diagnostic within the evaluated task groups.

4. (W4, Q5) Interaction scope and instruction-text control

Multi-turn and interleaved instructions. We agree that these are important evaluation settings, and Omni-Instruct does not currently cover multi-turn interaction or interleaved context. Our focus is on a complementary axis. Existing multi-image and interleaved benchmarks [1–3] typically interleave modalities within the context, while the instruction itself is given as text. In contrast, Omni-Instruct separates the instruction carrier from the context and varies their modalities independently. This allows us to evaluate how models respond to user queries composed of diverse modality combinations, including cases where the instruction itself is delivered through a non-text modality.

Section 5.5 also includes a simple form of interleaved instruction. While retaining the original image or audio instruction, we added a text cue indicating where the instruction is located, so that the directives are distributed across modalities (Appendix G.2). This cue yielded negligible E2E changes (−0.13 and +0.18 pp for image and audio instructions), suggesting that a textual directive alone does not help models act on an instruction delivered through another modality.

As this is a preliminary probe, extending Omni-Instruct to more diverse instruction scenarios is a valuable direction for future work. Examples include instructions whose content is split across modalities, conflicting directives, and multi-turn interactions.

Oracle OCR/transcription control for perception errors. We retained the original image/audio instruction and additionally supplied the oracle text. The study covers 72 requests in two non-text instruction carriers, yielding 144 outputs per model. The following table reports the original image/audio-instruction condition and the source-text-added condition:

ModelRA: baseline → +textCC: baseline → +textE2E: baseline → +textΔ E2E [95% CI], pp
AnyGPT38.19 → 52.7840.61 → 36.5015.51 → 19.27+3.75 [−0.54, +8.14]
MIO34.72 → 36.8121.57 → 30.737.49 → 11.31+3.82 [+0.83, +7.06]
HCX-Omni37.50 → 48.6141.42 → 57.4515.53 → 27.93+12.39 [+5.44, +19.34]
ModaVerse34.03 → 68.7521.93 → 39.767.46 → 27.33+19.87 [+14.32, +25.70]
AR-Omni25.00 → 29.8612.13 → 14.653.03 → 4.37+1.34 [−0.78, +3.66]
NExT-GPT31.94 → 20.1414.37 → 31.534.59 → 6.35+1.76 [−0.81, +4.69]

Providing the instruction text alongside the original carrier improves E2E performance most for ModaVerse and HCX-Omni, providing an approximate indication of the contribution of perception errors. However, substantial errors remain even when the instruction is explicitly available in text, suggesting that difficulties extend beyond recovering the instruction from its original carrier.

[1] Liu et al. Holistic Evaluation for Interleaved Text-and-Image Generation. EMNLP 2024.
[2] Xia et al. MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models. ICLR 2025.
[3] Li et al. UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark. CVPR 2026.

5. (Q1) Basic artifact production under short, explicit prompts

As suggested, we tested each model using short, explicit 40 prompts to generate image/audio, such as: "Generate speech audio saying: Hello.", "Show me a famous landmark in NYC.", "Create a picture.", and "Can you generate an image of a red apple?".

The table compares these basic generation rates with routing accuracy on the corresponding subset of Omni-Instruct. Specifically, we measure RA on the text-to-image and text-to-audio tasks without context.

ModelBasic generation (%)Omni-Instruct (%)
AnyGPT100.066.0
MIO52.529.0
HCX-Omni85.065.0
ModaVerse100.099.0
AR-Omni45.03.0
NExT-GPT50.04.0

The gap is particularly large for AR-Omni and NExT-GPT, whose basic generation rates reach 45.0% and 50.0%, respectively, but fall to benchmark routing accuracies of 3.0% and 4.0%. All six models can produce both image and audio artifacts in the basic controls, confirming that their benchmark routing failures cannot be explained solely by an absence of the corresponding generation capability.

Nevertheless, several models remained unreliable even under simple requests, indicating that generation or interface limitations also contribute. Benchmark routing accuracy was lower than basic generation success for every model, although the gap varied substantially. We therefore interpret RA as the evaluated system's ability to deliver the requested modality, with failures potentially arising from both generation/interface limitations and instruction following.

6. (Q3) Sensitivity to disallowing accompanying text

We recomputed RA and E2E under the suggested, stricter rule that disallows accompanying text for image/audio targets, after excluding media/control syntax and intermediate generation prompts.

ModelRA: default → strictE2E: default → strict
AnyGPT44.33 → 21.8918.64 → 8.11
MIO37.33 → 37.1713.26 → 13.23
HCX-Omni43.06 → 25.2222.90 → 15.01
ModaVerse52.44 → 52.4414.37 → 14.37
AR-Omni28.06 → 24.113.91 → 3.20
NExT-GPT33.50 → 33.007.10 → 6.94

As seen in the table above, disallowing accompanying text substantially drops performance for some models, particularly AnyGPT and HCX-Omni. To understand these drops, we further examined the accompanying text in the outputs that passed default routing but failed the stricter rule. The table below summarizes selected text types and representative examples.

Text type# CasesModels with observed casesExamples
Acknowledgement / presentation211AnyGPT, HCX-Omni, AR-Omni, NExT-GPT"Sure, I can help you with that."<br>"Sure, here is the image you requested."
Duplicated transcript161HCX-Omni"Four dogs are running towards the camera in a grassy field."
Instruction repetition153AnyGPT, HCX-Omni, AR-Omni"Create an image with a plant beside a chair on a bedroom floor."
Tool narration7HCX-Omni"Using the t2i_model_generation tool"
Punctuation3MIO"."

These cases show that models use their final output fields differently, sometimes retaining media placeholders or transcripts alongside the requested artifact. Acknowledgements such as "Sure, here is the image you requested" also naturally accompany an artifact delivery. We therefore conclude that allowing accompanying text is more appropriate for measuring delivery of the requested modality, but report the stricter results separately in Appendix F.6.

7. (Q6) Benchmark and code release

We uploaded the requested benchmark items, final rubrics, evaluation code, model-specific inference adapters and settings, on https://anonymous-hf.com/a/g6o0a5rllup2/. The full judge prompt is also provided in Appendix C.2. We will further improve the documentation and user instructions to facilitate reproduction.

Reviewer_sVoKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OMNI-INSTRUCT, a benchmark for evaluating instruction following in any-to-any multimodal models. The benchmark varies the modalities of the instructions, input context, and output to evaluate whether models can successfully complete tasks across different modality combinations. The results show that output modality routing is a major bottleneck, and models struggle more when instructions are provided through images or speech rather than text.

Critical strengths
  1. The problem formulation is valuable. Evaluating instruction following across different input and output modalities is an important direction.

  2. The benchmark construction is comprehensive, covering different modality combinations and tasks, with human verification.

Critical weaknesses
  1. Limited novelty. The benchmark mainly extends existing instruction following evaluations to different modalities. The main findings are somewhat expected and provide limited new insights.

  2. [minor] Unclear practical relevance. It is unclear how well the benchmark reflects real world applications. In particular, how common is it for users to provide instructions through images rather than text? Also, I think many multimodal systems can explicitly specify the output modality, making it unclear how important autonomous modality routing is in practice.

Questions for authors

I suggest that the authors provide more context on how current omni models handle output modality routing, both in terms of the underlying mechanisms and practical implementation. In particular, can routing failures be addressed through simple engineering solutions, such as explicitly specifying the target output modality? I'm not saying that having models infer the output modality is unimportant, but I would like to better understand the gap between these two approaches. It would also help to provide more real-world examples to explain why this problem matters in practice.

7 comments

(W1) Novelty

We thank the reviewer for this clarifying question. Agreeing that our description on contributions distinguished from prior work were not crystal clear, we summarize it as follows:

  1. Unlike existing benchmarks, our Omni-Instruct is the first one allowing the impact of how the instruction and context are conveyed in any-to-any setting. We explicitly distinguish instruction from context and convey the same request through text, image, and audio, holding the context, output target, and rubric fixed. This reveals whether the carrier changes how the any-to-any model acts on the request, and this can be evaluated by no other public benchmark, to the best of our knowledge.
  2. To the best of our knowledge, this is the first benchmark evaluating artifact delivery as an IF requirement. We explicitly evaluate whether the requested output modality is delivered, rather than assuming a preselected response channel. Given the observations that many models are not still capable of this trivial-looking routing yet, we claim that our Omni-Instruct indeed adds an important axis in the any-to-any benchmark.
  3. Our approach allows finer-grained analysis on the failures which have not been possible with existing benchmarks. By separating delivery failures (RA) from content failures (CC), Omni-Instruct explicitly distinguishes partial satisfaction from complete fulfillment, pinpointing the failed area. To the best of our knowledge, such an analysis has not been provided by other benchmarks.

(W1) Findings

We thank the reviewer for this constructive feedback. We list more interesting findings beyond the expected performance gap on non-text carriers as follows:

  1. Models describe or repeat non-text instructions instead of executing them. This occurs even without context. When context is present, models may instead copy or describe the source material without applying the requested operation (Sec. 5.4).
  2. Spoken instructions can trigger audio output even when another modality is requested. This occurs in affected models even without an audio context or target, showing that output selection is associated with the instruction carrier despite the stated requirement (Sec. 5.2; Fig. 6).
  3. Models can satisfy individual constraints without performing the requested task. Surface constraints such as length can pass for misdirected responses. Requiring all constraints to pass substantially reduces scores and widens model gaps. RA and CC also yield different rankings on their respective routed subsets (Secs. 5.1 and 5.3).
  4. Cleaner inputs and explicit instruction-location cues recover little of the performance gap. Simplifying image and audio carriers yields limited recovery, as does identifying where the instruction is. Even correctly reporting the requested action, output modality, and requirements does not guarantee execution (Sec. 5.5, Figs. 8–9). These findings go beyond lower non-text scores by identifying task substitution, unrequested output modalities, and persistent execution failures under the tested interventions, providing specific targets for model improvement.

(W2, Q1) [minor] Practical relevance and real-world examples

We clarify the practical motivation through documented uses of image and audio instructions, and examples where the required output varies with the request or must be selected by the system.

Multimodal Instruction. First, images and audio can often carry instructions in practice. Photoshop supports user annotation on an image to guide edits [1], and Gemini provides a similar annotation-based editing interface [2]. Photoshop also accepts spoken editing requests [3]. On wearable devices like AI glasses, verbal instructions are naturally provided through audio, often requesting edits on a visual scene. Image-carried instructions are also common in everyday use: users often submit a photo or screenshot of a homework or exam problem instead of retyping it, so the instruction itself lies in the image. MMMU-Pro [4] reflects this practice with a vision-only setting where questions are embedded in screenshots or photos.

Modality routing. Explicitly specifying the output modality via a UI picker or API parameter is not always possible. Screenless interfaces such as smart speakers and earbuds have no place for a picker; display-equipped glasses can both show and speak, so the output channel must be chosen per request. Interleaved outputs, such as a recipe with step-by-step photos, switch modality within a single response, which a per-request setting cannot express. Even in chat assistants, image generation is often triggered by a natural-language prompt rather than a dedicated mode [5].

Moreover, the output modality may not be known from the query and is instead decided during task execution. Visual Sketchpad [6] generates and inspects diagrams through code to solve geometry questions. Sensible Agent [7] chooses audio, visual, or combined presentation for proactive suggestions, e.g., visual in a museum and audio during grocery shopping. These workflows require the system to select intermediate and output modalities as part of the task.

[1] Adobe. Use Markup for AI image generation. Adobe Photoshop User Guide.
[2] Google. Gemini Drop: December 2025. Google Blog, Dec. 2025.
[3] Adobe. Image editing just got smarter with AI in Photoshop and Firefly. Adobe Blog, Mar. 2026.
[4] Yue et al. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL 2025.
[5] Microsoft. Using Image Generation in Microsoft Copilot. Microsoft Support.
[6] Hu et al. Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models. NeurIPS 2024.
[7] Lee et al. Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agents. UIST 2025. See also the Google Research blog post.

(Q1) Routing mechanisms and explicit output specification

Output modality routing in current omni models. Current omni models and assistants express the routing decision in one of three ways. (i) Media tokens (e.g., AnyGPT, MIO, AR-Omni, HCX-Omni): the vocabulary is extended with discrete image/audio codebook tokens, and the output modality is determined by which tokens are sampled during autoregressive decoding. Routing is thus a probabilistic outcome of next-token prediction conditioned on the whole input, and can be biased by it, e.g., spoken instructions favoring audio tokens (Sec. 5.2). (ii) Modality tags (e.g., NExT-GPT, ModaVerse): the model generates a special tag that dispatches an external decoder with a conditioning prompt. The tag is itself a generated token, so it can be omitted or mismatched, and the decoder is conditioned on the model's generated signal-token representations or textual prompt, rather than directly on the original request. (iii) Tool calls (e.g., the image-generation tool in OpenAI's Responses API): when a generation tool is made available, the model decides whether and how to invoke it by emitting a structured call.

In all cases, routing is a model output rather than a fixed rule, so none of these mechanisms guarantees the requested modality, and delivery further requires the generated sequence to be completed and decoded into a valid artifact. Further details, including how frontier assistants select output forms, are added in Appendix C.1.

Explicit specification as a simple solution. Examining the model implementations, there are some provide prompt-based controls for steering the output modality. We therefore tested explicit specification by appending "Provide text.", "Provide an image.", or "Provide audio." to each base request, conveyed in the same carrier as the instruction.

ModelRACCE2ECC on recovered cases
ModaVerse52.78→54.6332.06→28.9316.92→15.8033.93
AnyGPT43.52→48.6141.27→38.3117.96→18.6219.64
HCX-Omni43.98→44.4452.41→57.3023.05→25.4742.15
MIO37.50→29.6329.94→33.5611.23→9.9512.50
NExT-GPT32.41→32.8724.49→22.817.94→7.5015.87
AR-Omni26.39→31.4812.91→15.003.41→4.7213.42

As seen in the table above, explicit specification yields +0.85%P improvement on routing accuracy (RA) on average, and +0.26%P on end-to-end (E2E) evaluation. Among cases whose routing is recovered, CC remains limited. For example, 19.64% for AnyGPT and 42.15% for HCX-Omni. Explicit specification may modestly improve routing, but the generated content can still fail to satisfy the request.

Model-provided output controls. We additionally tested two controls exposed by the reference implementations, both requiring the user or caller to specify the target output modality externally. MIO's mode="voice" selects its original speech-only system prompt, while AnyGPT's to_modality selects target-specific decoding configurations.

Model / controlRACCE2E
AnyGPT / target-specific decoding config43.52→53.2441.27→33.5217.96→17.84
MIO / voice mode for audio targets37.50→50.0029.94→33.5911.23→16.80

MIO's voice mode improves both routing and E2E performance, while AnyGPT's target-specific decoding configurations improve RA by 9.72 percentage points without improving E2E. These results show that simple engineering interventions can help, but their effectiveness varies across models and interventions. Explicitly specifying the target modality is therefore insufficient on its own to ensure both delivery of the requested artifact and compliance with its content requirements, supporting the evaluation of routing and content compliance together even when output controls are available.

Reviewer_HVvJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Omni-Instruct to test instruction following across text, images, and audio. It separates choosing the requested output type from following the content requirements. Tests on 6 models show frequent failures at both stages, especially when instructions are given through images or speech.

Critical strengths

Separating output routing from content compliance gives a clearer view of model failures. The results show that models with better routing can still perform poorly on the content requirements.

Critical weaknesses
  1. The comparison with prior work is incomplete. Please include close benchmarks such as AlignAnything [R1], VoiceBench [R2], and InSerter [R3], etc, and explain more clearly what this Omni-Instruct benchmark adds.

  2. The human validation provides useful evidence, but covers only text-instructed cases with the original grading rules. A small human check using the final rules across all instruction types would strengthen the results. It would also help to show whether repeated judging changes scores or model rankings.

  3. Reproducing the results is difficult from the manuscript alone. I could not find the full judge prompt or an anonymous link to the final dataset and evaluation code. Please provide these materials, together with the judge settings and saved decisions, so the reported scores remain checkable if the API changes. Reporting the construction and evaluation costs would also help others assess the resources needed.

  4. The benchmark reuses material from widely used datasets, but possible overlap with the evaluated models’ training data is not examined. Although the instructions are newly written, familiarity with the source material may still affect scores and model comparisons. Please clarify the source splits and check whether the main findings hold on a small set of fresh material.

[R1] Ji, Jiaming, et al. "Align anything: Training all-modality models to follow instructions with language feedback." arXiv preprint arXiv:2412.15838 (2024). [R2] Chen, Yiming, et al. "Voicebench: Benchmarking llm-based voice assistants." Transactions of the Association for Computational Linguistics 14 (2026): 378-398. [R3] Wang, Dingdong, et al. "Inserter: Speech instruction following with unsupervised interleaved pre-training." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Questions for authors
  1. Can you report human agreement on image/audio instruction cases using the final grading rules?

2.Which source splits are used, and have you checked possible overlap with the evaluated models’ training data? Do the main findings hold on a small set of fresh material?

3 comments

1. (W1) Comparison with related benchmarks

We thank the reviewer for providing relevant benchmarks. We highlight distinguishing factors of our Omni-Instruct compared to these prior work as follows, and reflect it to Sec. 2.

[R1] AlignAnything (EvalAnything). EvalAnything evaluates understanding and generation separately. Understanding is tested as text QA over visual and auditory context, and generation with context-free text prompts (mostly caption-style). Within generation, instruction following is measured with the output modality predetermined, while modality selection is evaluated on a separate prompt set as agreement with human preference. The selection target is thus a preference distribution rather than a modality the user explicitly requested. Consequently, any-to-any tasks that require non-text outputs grounded in given material (e.g., image or speech editing), or instructions conveyed through images or speech, are not evaluated. OMNI-INSTRUCT targets these settings directly by crossing every context–output pairing with text, image, and speech instructions, and treats the explicitly requested output modality as a requirement. Each instance jointly requires understanding the instruction, routing to the requested modality, and satisfying context-grounded requirements, with routing (RA) and compliance (CC) scored on the same response.

[R2] VoiceBench. It also treats speech as a carrier of user instructions and evaluates whether models respond appropriately, including instruction-following tasks. Its real and synthetic spoken instructions additionally test robustness to speaker, environmental, and content variations, and it compares responses to text-form and speech-form instructions. However, its evaluation focuses on text-response quality, rather than whether a model delivers a requested image or audio artifact. Our Omni-Instruct extends the evaluation to image-carried instructions and requested text, image, and audio outputs, allowing us to examine how changes in the instruction carrier affect both artifact delivery and content compliance.

[R3] InSerter (SpeechInstructBench). This benchmark also directly evaluates instruction compliance beyond text input. The study contains comparison between speech-form and text-form inputs. However, the tasks are limited only to textual responses to spoken instructions, rather than crossing text, image, and audio instruction carriers with specified output modalities. Our Omni-Instruct adds this broader evaluation setting and separates successful delivery of the requested artifact from satisfaction of its content requirements.

Overall, these benchmarks address related aspects of instruction following, but differ from our work in both problem definition and evaluation coverage. We have clarified how our formulation incorporates output-modality routing into instruction following and evaluates it across instruction carriers, context modalities, and requested outputs in Sec. 2.

2. (W2, Q1) Human validation and repeated judging

We agree that validating the final rubrics on non-text instruction carriers would strengthen our paper. Following the comment, we let three human annotators additionally evaluate 120 outputs generated from audio- and image-delivered instructions, with 60 outputs per carrier. We take the majority vote among the annotators. The final content rubrics contain 627 atoms.

The result table below validates human-judge agreement extended to audio and image instructions.

Instruction modalityOutputsRubric atomsAgreement ↑Cohen's κ ↑ICC(A,1) ↑
Text12057589.0%0.7660.849
Audio6031491.1%0.7630.835
Image6031393.3%0.8200.880
Audio + Image12062792.2%0.7910.855

Agreement is numerically higher for audio- and image-delivered instructions than for text-delivered instructions, supporting the applicability of our judge to non-text instruction carriers. We have added these results to Appendix D (Table A4) of the revised manuscript.

Reliability of repeated judging. Following the suggestion, we evaluated the 108 outputs three times with fixed prompts, rubrics, media on Gemini 3.5 Flash at temperature 0. The panel covers all 36 instruction-context-output modality combinations, with three outputs per combination, across all 12 tasks, and all 6 evaluated models. Across 578 content-rubric atoms, 556 (96.19%) received identical labels in all three evaluations, with a three-run Fleiss' κ of 0.943.The model ordering was unchanged across all three runs.

3. (W3) Release Benchmark materials and cost

We uploaded the requested benchmark items, final rubrics, evaluation code, model-specific inference adapters and settings, and saved model outputs on https://anonymous-hf.com/a/g6o0a5rllup2/. The full judge prompt is also added in Appendix C.2. We will further improve the documentation and user instructions to facilitate reproduction. The instruction-image/audio generation costs approximately USD 36, including an estimated TTS component. A single judging pass over six models costs approximately USD 60, with an additional USD 44 for ablations.

4. (W4, Q2) Potential data overlap and Source splits

We appreciate the reviewer for raising this concern, and agree that newly written instructions alone may not rule out familiarity with the underlying context material. We have added a dedicated section in Appendix A.2 detailing the source splits we used and the publicly disclosed training data of all six evaluated models. In brief, we mostly used held-out test or evaluation partitions where available (validation partitions when test annotations are not released), but some sources provide no separate test split, and the evaluated models either do not disclose their training data or do not specify which splits were used. Thus, we cannot fully guarantee that our benchmark contexts are isolated from model training.

To examine whether this affects our findings, we additionally evaluated all six models on fresh requests with material released after the models' public releases (Wikimedia Commons photographs, DVIDS broadcast recordings), and compared them with the matching subtasks of the benchmark. The table below reports Routing Accuracy(RA) and End-to-End Score(E2E) on the original and fresh sets, along with the difference (Fresh − Original) and its 95% bootstrap interval.

ModelRA (orig → fresh)E2E (orig → fresh)Δ E2E [95% CI]
AnyGPT44.6 → 45.221.4 → 18.9−2.4 [−4.1, −0.7]
MIO32.9 → 33.213.1 → 14.0+0.9 [−0.6, +2.4]
HCX-Omni42.0 → 41.422.0 → 22.5+0.5 [−1.7, +2.6]
ModaVerse50.5 → 50.115.3 → 16.5+1.2 [−0.4, +2.7]
AR-Omni25.0 → 25.94.1 → 4.4+0.4 [−0.6, +1.2]
NExT-GPT28.0 → 27.45.6 → 6.3+0.7 [−0.2, +1.5]

From these results, performance on the fresh set closely matches the original: RA changes by less than 1 pp, and E2E intervals including zero for all models except AnyGPT, which is also small relative to the gaps between models, and the E2E ranking stays the same across the two sets.

The main findings, including the dominance of routing failure in E2E loss, the gap between correct routing and compliance, and sensitivity to the instruction carrier, persist on the fresh set. We added these detailed results to Appendix G.5. In summary, although some unverifiable overlap may exist, the results on fresh material remain similar to the original.

For reference, Reviewer HVvJ’s response to our rebuttal and our reply were posted under the thread of Reviewer sVoK. We thank Reviewer HVvJ again for the constructive review and for engaging with our responses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion and Revisions

We sincerely thank all three reviewers for their constructive feedback and thoughtful engagement during the discussion period. We are encouraged that the reviewers found the problem formulation valuable (sVoK, DPei), the decoupling of output-modality routing from content compliance diagnostic (HVvJ, DPei), and the benchmark construction comprehensive (sVoK). We are also glad that our responses helped address some of the reviewers’ concerns, and we are grateful to Reviewer sVoK for raising their score and to Reviewers HVvJ and DPei for maintaining their positive assessments. Below, we summarize the main clarifications, additional analyses, and manuscript revisions made during the discussion.

  • Contribution and practical relevance (HVvJ, sVoK): We clarified how Omni-Instruct differs from related benchmarks: it independently varies instruction carriers and jointly evaluates requested-modality delivery and content compliance. We also provided practical examples that motivate non-text instructions and output-modality selection (Section 2; responses to HVvJ W1 and sVoK W1–W2).
  • Evaluation reliability (HVvJ, DPei): We extended human validation to image- and audio-delivered instructions under the final rubrics, obtaining 92.2% human–judge agreement across 627 rubric atoms. Three repeated judging runs yielded 96.19% unanimous atom labels and unchanged model rankings (Appendix D, Table A4; responses to HVvJ W2 and DPei W2).
  • Routing failures and output controls (DPei, sVoK): We audited failed outputs and tested basic artifact generation, explicit output instructions, and model-provided output controls. These analyses distinguish observable delivery failures from their possible causes and show that improved routing does not consistently translate into improved end-to-end compliance (Appendix C.1; responses to DPei W1 and Q1, and sVoK Q1).
  • Conditional compliance and routing criteria (DPei): We compared CC on shared correctly routed instances and on composition-matched subsets, clarifying its role as a conditional diagnostic. We also reported the sensitivity of the results to disallowing accompanying text for image/audio outputs (Appendices F.3 and F.6; responses to DPei W3, Q3, and Q4).
  • Instruction perception and evaluation scope (DPei): We added an oracle instruction-text control. Providing the instruction text substantially improved performance for some models, while considerable errors remained. We also clarified that systematic multi-turn and interleaved-instruction evaluation remains outside the scope of the current benchmark (responses to DPei W4 and Q5).
  • Potential data overlap (HVvJ): We documented source splits and available training-data disclosures, and evaluated models on fresh context material. The main findings and the E2E model ordering persisted in this supplementary evaluation, although training-data overlap cannot be fully ruled out (Appendices A.2 and G.5; response to HVvJ W4 and Q2).
  • Reproducibility (HVvJ, DPei): We released benchmark items, final rubrics, evaluation code, model-specific inference adapters and settings, and saved model outputs. We also provided the full judge prompt and reported construction and evaluation costs (Appendix C.2; responses to HVvJ W3 and DPei Q6).

We will incorporate all clarifications and additional analyses from the discussion into the final version. We thank the reviewers again for their careful consideration, which has substantially improved our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.