- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes OmniVAL, a validated chain-of-thought distillation framework for native omni-modal models. It extends the DocVAL framework to multiple modalities by introducing modality-specific validation rules and cross-modal routing checks. The framework filters teacher-generated reasoning traces and iteratively fine-tunes a Nemotron 3 Nano Omni student using structured corrective feedback. Experiments across 16 benchmarks report improvements in grounding and multimodal understanding.
Critical strengths
Strong empirical performance. The paper reports substantial improvements across diverse benchmarks, particularly on grounding-intensive tasks. The extensive evaluation and ablation studies demonstrate the practical effectiveness of the proposed training pipeline.
Critical weaknesses
1. Limited methodological novelty.
The proposed framework appears to be primarily an engineering extension of existing validated CoT distillation methods, particularly DocVAL. More concerningly, Figure 1 appears almost identical to that of DocVAL, which is unacceptable for an ICLR submission and raises serious concerns about the paper's claimed methodological novelty. Its main components—teacher-generated reasoning traces, rule-based validation, modality-specific feedback, and iterative supervised fine-tuning—largely follow established techniques. The primary extensions involve adapting validation rules to additional modalities and introducing a cross-modal routing consistency check.
The paper does not sufficiently establish what new methodological insight is contributed beyond integrating existing techniques into a broader multimodal training pipeline.
2. Poor presentation quality.
The presentation requires substantial improvement. The writing is unnecessarily verbose and repetitive, making it difficult to identify the core technical contributions. Figures 1 and 2 are overly cluttered, with small text, inconsistent visual organization, and unclear connections between components. Some graphical elements appear potentially AI-assisted or automatically assembled, although their origin cannot be determined from appearance alone. Regardless of how they were produced, the figures lack the clarity and careful technical presentation expected of an ICLR submission.
The current presentation substantially reduces the paper's readability and should be comprehensively revised.
Questions for authors
1. Novelty. What is the fundamental methodological contribution of OmniVAL beyond extending DocVAL to additional modalities? Please clarify which components introduce genuinely new learning mechanisms rather than engineering adaptations of existing methods.
2. Figure similarity and attribution. Figure 1 of OmniVAL bears a striking resemblance to Figure 1 of DocVAL, not only in the overall pipeline but also in the arrangement of components. Although DocVAL is cited in the related work, the figure caption does not acknowledge any adaptation from the prior work. Could the authors clarify whether Figure 1 was adapted from DocVAL and provide appropriate attribution if applicable? This extensive structural and visual similarity also reinforces my concerns about the limited methodological novelty of the proposed framework.
3. True native omni models. The paper repeatedly emphasizes "true native omni models." However, the student backbone already possesses multimodal capabilities before OmniVAL training. Moreover, the term "native" can imply deeper architectural integration, as exemplified by encoder-free multimodal architectures such as Gemma 4 12B, whereas Nemotron 3 Nano Omni still employs separate vision and audio encoders. What precisely distinguishes the "true native omni" models considered in this paper from conventional multimodal architectures? More importantly, what specific contribution does OmniVAL make to native omni-modal modeling beyond fine-tuning an existing multimodal backbone? The authors should clarify the definition, its architectural implications, and its relevance to the proposed method, or reconsider the corresponding claims.
4. Distillation baselines. The experiments include several internal ablations but lack direct comparisons with closely related CoT and validated distillation approaches, particularly DocVAL and other established multimodal reasoning distillation methods, adapted to the same backbone and training budget. Such comparisons are important for establishing the incremental value of the proposed framework.
5. Presentation. I strongly encourage the authors to substantially rewrite the manuscript and redesign the main figures. The current presentation obscures the technical contributions and falls below the standard expected for an ICLR paper.
2 comments
Thank you for the additional experiments. The M4 ablations provide useful evidence of its effectiveness, but my concern about methodological novelty remains: the overall pipeline closely follows DocVAL, and M4 alone does not establish a sufficiently distinct methodology.
The manuscript also remains unrevised, and the figures and "true native omni" terminology have not been updated yet. My final score will depend on the revised manuscript.
We thank the reviewer and address Figure 1 first.
1. Figure 1: provenance and attribution
Figure 1 was adapted from DocVAL's Figure 1. Its teacher→validator→student→inference layout is inherited. DocVAL is cited in Related Work section, but the caption omitted adaptation credit. We apologize for this oversight. The similarity also obscures OmniVAL's contributions. We will credit DocVAL (Neogi et al., 2025) in the caption, update the reference to ICML 2026, and redraw Figure 1 to separate inherited stages from modality-routed traces, M4 cross-modal validation, and modality-native corrections. We do not claim the inherited stages as novel.
2. Methodological novelty: validating evidence choice
DocVAL established validated teacher CoT, filtering, pixel-level corrections, and iterative refinement for document-image grounding. OmniVAL does not claim a new optimizer or backbone. Its distinct supervision target is which evidence stream supports an answer, and whether the selected route, reasoning trace, grounding, and evidence agree. An omni model may answer from subtitles when speech is required: even a correct answer and precise localization may rely on the wrong source. Answer accuracy and within-stream localization alone cannot detect this failure.
M4 checks (i) predicted versus reference grounding modality, (ii) trace versus emitted-grounding modality, and (iii) consistency with evidence in that modality (§3.2/App. A.3). It enables routing corrections alongside pixel, time, cell, and speaker corrections. This extends the object of validation beyond answer–location agreement.
Table 6 isolates M4, retaining the rest of OmniVAL:
Gains are modest on DocVQA but larger on cross-stream and GUI grounding. Table 11 reports routing: audio-only 76.3→91.7; joint AV 73.6→88.2. In Table 22's interventions, M4 increases accuracy loss when relevant evidence is removed (32.4→36.8), while reducing the effect of irrelevant evidence removal (9.5→6.2), distractor flips (18.3→12.7), and hallucinations (22.1→16.5). These show observable evidence reliance, not CoT faithfulness. Principle: for multi-stream tasks, validate which source supports an answer, not just where evidence lies.
3. Meaning of “true native omni”
We agree the phrase may imply encoder-free design. Nemotron 3 Nano Omni uses modality-specific vision/audio encoders. Here native omni means one integrated inference model handling text, image, audio, and video without external OCR/ASR services, separate specialist inference pipelines, or validator calls at test time; it does not mean encoder-free. We will clarify the definition and temper “true native” in the title/abstract. OmniVAL changes training supervision, not the pretrained architecture.
4. New matched distillation controls
A complete same-backbone, data/compute-matched DocVAL adaptation remains desirable. Table 6's no-M4 row is not such a reproduction: it isolates routing validation within OmniVAL.
New measured controls, using the same omni backbone and D3∪D4 source pool with matched steps and approximately matched training FLOPs, provide additional points of comparison:
Versus matched validated-CoT SFT, OmniVAL gains +4.6 / +9.5 / +9.0 / +5.6 points, respectively. Rejection sampling also trails, especially on ScreenSpot-Pro and WorldSense. These test additional-exposure and sample-selection explanations without constituting a full DocVAL reproduction. Table 15 further fixes B2 examples/steps: full structured feedback improves ScreenSpot-Pro 65.0→73.4 and DailyOmni 80.0→86.0 over corrected targets alone. We will report the protocols and retain the direct-DocVAL-baseline limitation.
5. Presentation
We agree the presentation obscures the contribution. We will redesign Figures 1–2 with larger labels, fewer arrows, an M1–M5 legend, and separate training-only from tool-free inference; shorten teacher-choice discussion; and reorganize the Introduction around inherited workflow → cross-stream failure → M4 supervision → controlled evidence. We will accurately disclose figure-generation assistance, if any.
We hope these explanations and clarifications address the concerns the reviewer had, and kindly request the reviewer to reconsider the score. We're happy to provide further clarifications if needed. Thanks!