acceptodds
Under review as a conference paper at ICLR 2027

Open-OmniCoach: Evidence-Enhanced Trajectory Learning for Omni Video Agents

Abstract

Training capable omni video agents requires more than diverse question-answer pairs: agents must learn to reliably find, connect, and correct task-relevant evidence in omni-setting. However, we argue that existing data pipelines provide limited control over task-relevant evidence and weak supervision for these capabilities. In this work, we present Open OmniCoach, a new framework for evidence-seeking omni video agents. It contains a high-quality dataset, OmniCoach-240k, and a new agentic reinforcement learning (RL) method, Step-Level Evidence-Guided Policy Optimization (SEPO). For the former, OmniCoach-340k has both multi-span trajectory data and reflection trajectory data. We design a three-stage data pipeline by leveraging both fine-grained omni-evidence and the agent's own errors. Meanwhile, compared with other RL baselines, SEPO provides turn-level credit based on improvements in evidence coverage and temporal localization. Applying both trajectory data and the SEPO method, we obtain Open OmniAgent models. The resulting models outperform the baselines across multiple video and audio-visual understanding tasks by significant margins, demonstrating the effectiveness of the Open OmniCoach framework. In particular, Open OmniAgent-30B improves average accuracy over Qwen3-Omni by 6.3 percentage points across nine benchmarks, with gains of 12.2 and 13.3 points on LVOmniBench and LVBench, respectively. Code, datasets, and models will be available to the community.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal agents, video understanding, agentic reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RhjBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper constructs approximately 240K training trajectories through hierarchical audio-visual evidence modeling, multi-segment interaction, and teacher-guided error recovery, and further introduces SEPO based on evidence gain. The resulting 7B and 30B agents consistently outperform their respective base models across multiple audio-visual and video understanding benchmarks.

Critical strengths
  • The data pipeline focuses on cross-span evidence comparison and error recovery, which are both useful capabilities for audio-visual agents.
  • The experiments not only demonstrate the effectiveness of the proposed method but also provide insights into its performance across different data scales.
Critical weaknesses
  • The evidence scorer only matches intervals and modalities without verifying evidence understanding or including the initial overview. Evidence gains should be validated to ensure they reflect newly acquired information and actual answer improvement.

  • Reflection data includes both teacher corrections and successful follow-up trajectories, so the overall QA improvement cannot solely be attributed to the reflection ability itself, as it also involves an increase in data scale.

Questions for authors
  • The paper mentions using Gemini-3.6 as the teacher. How is the COT in this trajectory generated? Additionally, the same teacher is used for reflection, so how does it manage to correct itself when mistakes occur in the steps?
  • The effectiveness of SEPO compared to GSPO is not very evident. Could you report results from multiple checkpoints to ensure the findings are not cherry-picked, or provide additional evidence to demonstrate consistent effectiveness?
6 comments
AuthorsOpenReview ↗

Response to Reviewer RhjB (1/3)

Thank you for recognizing the importance of cross-span evidence comparison and error recovery for audio-visual agents. We also appreciate your positive assessment of our empirical results and data-scaling analyses. Below, we provide detailed responses to the issues you raise.

W1: Details About the Evidence Scorer

1. Role of the initial overview. All 8 rollouts of a query receive the same initial overview (0.5 FPS frames and full audio; Appendix D.1), while tool calls retrieve higher-resolution frames or synchronized clips for local details missed by this overview. Because the initial overview is shared across the rollout group, SEPO scores only the incremental local evidence retrieved by tool calls (Appendix E.2). When the overview alone suffices for a one-turn answer, the turn weight stays wi,1=1w_{i,1}=1 without penalty (Appendix E.3, Eq. (17)).

2. Outcome gating. The sign of the trajectory advantage AiA_i is set by the final answer reward, and the evidence score only rescales turn weights within [0.9,1.1][0.9, 1.1] without changing the sign of AiA_i (Eq. (6); Appendix E.3, Eq. (17)). Retrieving a relevant interval while misinterpreting its content and answering incorrectly therefore retains a non-positive advantage.

AuthorsOpenReview ↗

Response to Reviewer RhjB (2/3)

W2: About Reflection Supervision and Data Scale

Unlike simply adding more Stage-2 trajectories, reflection trajectories branch directly from the student's own failed rollout prefixes (similar to DAgger [ref1]; Section 2.3), teaching recovery on error states the student actually visits. Notably, 9.7K of our 17.4K reflection trajectories (spanning 3,926 queries) recover hard cases where all 8 student rollouts failed or standard Stage-2 teacher sampling yielded no accepted trajectory.

Table C: Controlled ablation on a 25% OmniCoach subset (62.0K trajectories; component counts rounded independently; Qwen3-Omni-30B-A3B) comparing reflection against equal successful trajectories.

25% Data Composition (62.0K Traj.)Multi-Span + Refl.Sup. Tokens / Ep.Video-MMEVideo-MME-v2OmniVideoBenchLVOmniBenchAvg.
Replace Refl. w/ Equal Success57.7K + 4.4K Succ.21.15M72.037.441.441.348.0
Refl. (1 turn after mistake)57.7K + 4.4K Refl.20.94M72.237.942.141.748.5
Refl. (2 turns after mistake)57.7K + 4.4K Refl.21.56M72.638.542.542.148.9
Refl. (3 turns after mistake)57.7K + 4.4K Refl.21.96M72.938.742.442.549.1
25% OmniCoach (Full Refl.)57.7K + 4.4K Refl.22.82M72.838.942.742.449.2

1. Outperforms equal successful-trajectory volume. With the total number of trajectories fixed, reflection supervision outperforms additional successful trajectories, especially on multi-step reasoning benchmarks.

2. Driven by corrective turns rather than follow-up trajectory length. With pre-mistake student turns masked (loss = 0), the first corrective turn alone already outperforms the equal-success baseline while using fewer tokens. Two or three corrective turns recover most of the full reflection gain; longer follow-up trajectories add little. We have added this ablation to Section 4.3.

[ref1] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011.

AuthorsOpenReview ↗

Response to Reviewer RhjB (3/3)

Q1: Details About Teacher CoT and Error Correction

In Stage-2, we do not replay Gemini's original CoT or expose reference answers or evidence annotations. Given only the question, the initial media overview, and the tool interface, gemini-3.6-flash generates its observation and think fields on the fly at each step to decide whether to call a tool or answer, and we keep only the rollouts whose final answers pass verification.

In Stage-3, the errors being corrected come from failed student rollouts rather than gemini-3.6-flash itself. An independent verifier first checks the student trajectory against the media to locate the first unsupported or contradicted step. Given the prior history, that unexecuted failed turn, and the verifier's note, gemini-3.6-flash writes a media-grounded correction and revised action on the fly, then continues the rollout until reaching a verified answer (the verifier's note is stripped before training). We have clarified this workflow in Section 2.3.

Q2: About Multi-Checkpoint and Multi-Seed Results

Table F: Evaluation across RL training checkpoints from the SFT checkpoint (GSPO → SEPO, Accuracy %).

RL ProgressVideo-MMEVideo-MME-v2OmniVideoBenchLVOmniBench4-Bench Avg.
50% of schedule75.1 → 75.543.0 → 43.846.6 → 47.547.8 → 47.753.1 → 53.6
75% of schedule75.4 → 76.043.3 → 44.146.9 → 47.647.7 → 48.153.3 → 54.0
100% of schedule75.6 → 75.943.0 → 44.446.6 → 47.947.8 → 48.053.3 → 54.1

Table D: Comparison of SEPO vs. GSPO across 3 random seeds (Qwen3-Omni-30B-A3B, Accuracy %, mean ± std).

MethodVideo-MMEVideo-MME-v2OmniVideoBenchLVOmniBenchAvg.
GSPO75.3 ± 0.2243.2 ± 0.3746.5 ± 0.4047.7 ± 0.4853.2 ± 0.27
SEPO75.9 ± 0.1944.4 ± 0.3047.9 ± 0.4548.1 ± 0.3554.1 ± 0.22

1. Gains across checkpoints and seeds. On the 4-benchmark average, SEPO leads GSPO at the 50%, 75%, and 100% checkpoints (+0.5, +0.6, +0.8). The advantage therefore does not depend on choosing a favorable checkpoint. 2. Process-level gains on multi-step reasoning. We audited all 1,000 evaluation trajectories on OmniVideoBench (3,577 human-annotated step-level evidence facts) using objective tool-log metrics and a blind observation audit (full protocol under Reviewer 82hi W2). Focused Hit ≤30s is the recall of annotated timestamps within ±2s by spans ≤30s; Error-to-Correct (+ Ans. Right) measures how often an early WRONG observation is revised to CORRECT in later turns (and answered right).

Table E: Process-level evidence acquisition and error recovery on OmniVideoBench (1,000 queries).

MethodAvg. CallsInvalid / No-Prog. Calls (%) ↓Avg. Span (s) ↓Focused Hit ≤30s (%) ↑Error-to-Correct (+ Ans. Right) (%) ↑
SFT3.98.4%43.3s58.2%9.0% (5.5%)
GSPO5.415.1%46.0s61.3%12.0% (5.0%)
SEPO5.513.2%42.1s62.9%12.6% (7.1%)

At comparable search depth to GSPO, SEPO retrieves tighter, more relevant evidence, makes fewer invalid or unproductive calls than GSPO , and more often recovers from early perceptual errors.

Reviewer_82hiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to train audio-visual agents to acquire relevant evidence and recover from intermediate mistakes. Open-OmniCoach combines a hierarchical evidence annotation and question-generation pipeline, multi-span tool-use trajectories, and teacher-corrected trajectories collected from student failures. It also introduces SEPO, which adjusts trajectory-level advantages using incremental evidence acquisition scores. The resulting training recipe is applied to Qwen2.5-Omni-7B and Qwen3-Omni-30B-A3B. The authors report improvements across audio-visual understanding, general video understanding, and temporal grounding benchmarks. The largest gains come from trajectory SFT; reflection supervision and reinforcement learning provide additional improvements.

Critical strengths
  1. The paper offers a useful approach to supervising the agent’s interaction process. The connection between evidence annotations, tool-use trajectories, and correction of student errors is well motivated. I particularly like the reflection construction: it resumes from a confirmed mistake in an actual student rollout, preserving the preceding context and teaching a correction before continuing. This makes the supervision more relevant to the failures an agent encounters than training exclusively on successful teacher demonstrations.
  2. The overall recipe produces substantial improvements across different evaluations. The 30B model improves over its reported backbone on all five audio-visual benchmarks, including gains of 9.5 points on OmniVideoBench and 12.2 points on LVOmniBench. Improvements on the 7B backbone and the grounding evaluations strengthen the empirical case. Table 4 is also useful: applying the same trajectory-generation procedure to different query sources provides evidence that the constructed queries contribute beyond the choice of training format.
Critical weaknesses
  1. The ablations leave the contributions of multi-span retrieval and reflection only partly resolved. In Table 5, adding multi-span trajectories also introduces agentic trajectory training, so the large improvement does not establish how much joint retrieval helps over sequential retrieval. Reflection supervision then adds 17K examples without a comparison against the same amount of additional successful-trajectory data. The combined recipe is convincing, but the evidence is weaker for the paper’s explanation of why these particular designs help. Matched comparisons would make the methodological contribution easier to assess.
  2. The evidence for SEPO is relatively limited. Its gains over GSPO are 0.5, 1.1, 1.0, and 0.3 points on the four benchmarks in Table 5. These results are encouraging, but no run-to-run variation is reported. More importantly, the evaluation mostly measures final-answer accuracy, whereas the motivation concerns better evidence acquisition and error correction. A focused analysis of retrieved evidence quality or recovery success would help establish whether SEPO improves the intended behavior. At present, I find the contribution of the training data better supported than that of the new optimization method.
Questions for authors
  1. Do you have a matched comparison between multi-span and sequential retrieval, keeping the query set, trajectory-generation procedure, and retrieval budget comparable? This would help distinguish the benefit of joint inspection from the broader benefit of trajectory training.
  2. Does reflection supervision outperform adding an equal number of successful trajectories? Is there evidence that it improves correction of held-out student failures, rather than only aggregate answer accuracy?
  3. How stable are the SEPO gains across training runs? Please also clarify the implemented update: Section 3.2 describes a trajectory-level importance ratio, while Appendix E.3 describes a turn-level ratio and additional process penalties. Were these choices shared by the GSPO baseline?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 82hi (1/3)

Thank you for recognizing the value of supervising the agent's interaction process and resuming reflection from confirmed student mistakes. We also appreciate your positive assessment of our empirical gains across backbones, tasks, and query sources. Below, we provide detailed responses to the issues you raise.

W1&Q1: About Multi-Span vs. Sequential Retrieval

To isolate multi-span retrieval from downstream trajectory training, we evaluate single-span sequential retrieval (Base) and multi-span joint retrieval (Ours) directly on the same off-the-shelf Gemini 3.6 backbone without any SFT or RL(In fact, this is exactly the experiment in the early stage of our exploration ). Across 1,000 stratified queries from 872 unique videos across 7 benchmarks (900 MCQs across 6 benchmarks + 100 TimeLens temporal grounding queries), both variants run 3 independent rollouts per query under identical maximum step and frame budget constraints.

Table A: MCQ comparison on the fixed Gemini 3.6 backbone (Base → Ours, %). avg@3: mean accuracy across 3 rollouts; maj@3: majority vote (≥2/3 correct); pass@3: oracle coverage (≥1/3 correct).

Benchmark (#Queries)avg@3 (Base → Ours)maj@3 (Base → Ours)pass@3 (Base → Ours)
OmniVideoBench (200)67.3 → 70.767.5 → 70.577.5 → 80.5
LVOmniBench (150)67.6 → 71.169.3 → 72.078.7 → 81.3
WorldSense (150)66.9 → 64.466.0 → 65.376.0 → 70.7
Daily-Omni (150)83.6 → 84.283.3 → 82.789.3 → 89.3
JointAVBench (150)61.1 → 64.961.3 → 66.070.7 → 74.7
Video-MME-v2 (100)60.0 → 66.758.0 → 66.074.0 → 79.0
Overall (900, query-weighted)68.1 → 70.668.1 → 70.777.9 → 79.3
Overall (macro-averaged)67.7 → 70.367.6 → 70.477.7 → 79.3

Table B: Temporal grounding on TimeLens (100 queries, fixed Gemini 3.6 backbone) and retrieval efficiency.

VariantmIoU[email protected][email protected][email protected]best@3 mIoUbest@3 [email protected]Avg. Rounds ↓
Sequential48.966.753.331.055.561.03.5
Multi-Span51.969.758.733.757.966.02.3

1. Cross-segment reasoning without training. With the backbone fixed, multi-span retrieval improves cross-segment reasoning and temporal grounding without additional training.

2. Higher synthesis ceiling and fewer rounds. Multi-span retrieval finds more valid answers with fewer retrieval rounds, providing more candidates for rejection sampling. The gains are not universal: on WorldSense, extra spans can distract from single-cue queries; Stage-2 filtering and SFT further sharpen span selection. We have added this comparison to Appendix D.2.

AuthorsOpenReview ↗

Response to Reviewer 82hi (2/3)

W1&Q2: About Reflection vs. Successful Trajectories

To test whether the gains of reflection supervision come from error-recovery supervision rather than extra trajectory volume, we ran a controlled ablation on a 25% OmniCoach subset (62.0K trajectories: 57.7K multi-span + 4.4K reflection) with Qwen3-Omni-30B-A3B across the four benchmarks in Table 5.

Table C: 25% OmniCoach subset (62.0K trajectories, Qwen3-Omni-30B-A3B) comparing reflection against equal successful trajectories and post-error turn truncation.

25% Data Composition (62.0K Traj.)Multi-Span + Refl.Sup. Tokens / Ep.Video-MMEVideo-MME-v2OmniVideoBenchLVOmniBenchAvg.
Replace Refl. w/ Equal Success57.7K + 4.4K Succ.21.15M72.037.441.441.348.0
Refl. (1 turn after mistake)57.7K + 4.4K Refl.20.94M72.237.942.141.748.5
Refl. (2 turns after mistake)57.7K + 4.4K Refl.21.56M72.638.542.542.148.9
Refl. (3 turns after mistake)57.7K + 4.4K Refl.21.96M72.938.742.442.549.1
25% OmniCoach (Full Refl.)57.7K + 4.4K Refl.22.82M72.838.942.742.449.2

1. Comparison against equal successful trajectories. Reflection supervision outperforms an equal number of additional successful trajectories, with the largest differences on multi-step reasoning benchmarks.

2. Signal from corrective turns vs. follow-up tokens. With pre-mistake student turns masked (loss = 0), supervising only the first corrective turn already outperforms the equal-success baseline while using fewer tokens. Two or three corrective turns recover most of the full reflection gain, with little benefit from longer follow-up trajectories. We have added this ablation to Section 4.3.

AuthorsOpenReview ↗

Response to Reviewer 82hi (3/3)

W2&Q3: Details About SEPO Stability and Process Behavior

1. Multi-seed stability (W2, Q3). Agentic SFT already improves over the base model by +8.5 to +11.7 points on Table 5 benchmarks. Starting from this strong SFT checkpoint, we evaluated GSPO and SEPO across 3 random seeds.

Table D: Comparison of SEPO vs. GSPO (Qwen3-Omni-30B-A3B, Accuracy %, mean ± std).

MethodVideo-MMEVideo-MME-v2OmniVideoBenchLVOmniBenchAvg.
GSPO75.3 ± 0.2243.2 ± 0.3746.5 ± 0.4047.7 ± 0.4853.2 ± 0.27
SEPO75.9 ± 0.1944.4 ± 0.3047.9 ± 0.4548.1 ± 0.3554.1 ± 0.22

SEPO achieves higher mean accuracy than GSPO on all four benchmarks, with non-overlapping mean ± std intervals on three (all except LVOmniBench). The margin is also smaller on Video-MME, where about 35% of queries require no tool calls, so SEPO reduces to GSPO for these one-turn answers.

2. Process-level evidence quality and error recovery (W2). We audited all 1,000 evaluation trajectories on OmniVideoBench (3,577 human-annotated step-level evidence facts) using two protocols:

  • Objective tool-log metrics: Avg. Calls, Invalid / No-Prog. Calls (malformed or duplicate calls), Avg. Span duration, and Focused Hit ≤30s (recall of annotated timestamps within ±2s by spans ≤30s).
  • Blind observation audit (gemini-3.6-flash): Given only the question, annotated facts, and per-turn JSON observation summaries , the judge labels each (turn, fact) pair as CORRECT, WRONG, or ABSENT. Error-to-Correct (+ Ans. Right) measures how often an early WRONG observation is revised to CORRECT in later turns (and answered right).

Table E: Process-level evidence acquisition and error recovery on OmniVideoBench (1,000 queries).

MethodAvg. CallsInvalid / No-Prog. Calls (%) ↓Avg. Span (s) ↓Focused Hit ≤30s (%) ↑Error-to-Correct (+ Ans. Right) (%) ↑
SFT3.98.4%43.3s58.2%9.0% (5.5%)
GSPO5.415.1%46.0s61.3%12.0% (5.0%)
SEPO5.513.2%42.1s62.9%12.6% (7.1%)

At comparable search depth to GSPO, SEPO retrieves tighter spans with higher focused recall, makes fewer invalid or unproductive calls than GSPO (13.2% vs. 15.1%), and more often corrects early perceptual errors to reach the right answer. The reduction in invalid/no-progress calls relative to GSPO also holds across LVOmniBench, Video-MME-v2, WorldSense, and LVBench. In training rollouts after a zero-gain first call (ΔUi,1=0\Delta U_{i,1} = 0), recovery rises from 28.9% to 34.4%.

3. Equivalence of Section 3.2 and Appendix E.3 & shared GSPO settings (Q3). Each rollout batch performs a single gradient update (one PPO epoch), so passing turns individually (Appendix E.3) yields the exact same token-level policy gradient as Eq. (7) in Section 3.2, with clipping bounds [0.8, 1.28] acting only as numerical safeguards. GSPO and SEPO share the exact same codebase, rewards, protocol checks, and KL weight, differing solely in λ=0\lambda = 0 vs. λ=0.1\lambda = 0.1.

We have added Tables D–E and aligned Section 3.2 with Appendix E.3 in our revised version.

Reviewer_CEBoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Open-OmniCoach, an evidence-augmented trajectory learning framework for omni video agents. The authors design a three-stage data pipeline, based on which the OmniCoach dataset is constructed. They then propose an RL method named SEPO, and finally validate their approach via experiments. The manuscript is clearly written with a reasonable method design and careful experimental setup. Nevertheless, it still suffers from inconsistent statements across the text and calculation errors in experimental comparisons.

Critical strengths

1 The paper is easy to follow. 2 The proposed model achieves the improvements comparing with the baselines.

Critical weaknesses
  1. Neither the dataset nor the code is publicly available, making it difficult to independently reproduce the results. I suggest releasing both to support further evaluation and comparison.

  2. Some figures are difficult to read. For example, the left side of Figure 1 is too small to read without zooming in considerably. In Figure 2, stages and steps are shown in the same style, making the hierarchy unclear. I suggest improving the layout and making the distinction between stages and steps more explicit.

  3. The reported dataset sizes are inconsistent. The abstract mentions both OmniCoach-240K and OmniCoach-340K, while the main text reports 230K + 17K = 247K. I also noticed what appears to be leftover draft content in Table 1, which gives the number of trajectories as 248.2K. Could the authors clarify these differences and use consistent numbers throughout the paper?

  4. The experiments mainly use relatively older models. It would be helpful to include results with more recent models, such as Qwen3.6 or Qwen3.8, to see whether the proposed augmentation method remains effective with newer model architectures.

  5. There seems to be a contradiction between the motivation and Table 1. Section 1 and Appendix A.2 state that previous methods (Xing 2026; Zeng 2026) only use successful trajectories and do not provide error-recovery supervision. However, Table 1 marks both Video-o3 and OmniAgent as supporting Error Recovery. Could the authors clarify this discrepancy?

Questions for authors

Please find the comments from weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CEBo

Thank you for recognizing the clarity of our presentation, the design of our three-stage framework, and the improvements over baselines. We also appreciate your careful check of our experimental setup and manuscript consistency. Below, we provide detailed responses to the issues you raise.

W1: About Code and Dataset Release

As noted in the abstract, we will publicly release the code, dataset, and model checkpoints upon acceptance.

W2: About Figure Readability and Layout

We have updated both figures in the revision. In Figure 1, we enlarged the left panel and increased the font sizes for legibility; in Figure 2, we redesigned the layout to separate the high-level stages from the fine-grained steps.

W3: About Dataset Counts and Score Calculation

The dataset counts differ due to rounding and whether Stage-1 QA pairs are included. The trajectory set totals 248,164 samples (230,726 multi-span + 17,438 reflection), reported as 248.2K in Table 1 and as OmniCoach-240K in rounded shorthand. In Sections 1–2 and 4.1, 247K sums the rounded components (230K + 17K), while 10K+ is a conservative lower bound for the 17.4K reflection trajectories. OmniCoach-340K in the abstract further includes 95.7K Stage-1 QA pairs (248.2K + 95.7K = 343.9K, rounded to 340K).

The two +6.3 gains correspond to different model scales and benchmark sets: the abstract reports the 30B model's mean gain over all 9 benchmarks with public Qwen3-Omni scores (5 in Table 2 + 4 in Table 3), whereas Table 2 reports the 7B model's mean gain over the 5 audio-visual benchmarks. The 0.1-point differences in Table 2's Δ\Delta rows come from subtracting unrounded scores before rounding to one decimal place. We have unified the naming and clarified these scopes across the abstract, Sections 1–2, Section 4.1, and Table 1.

W4: About Newer Omni Backbones

Because our pipeline uses audio-visual SFT and RL, it requires an open-weight backbone that accepts both audio and video. Qwen3.6 has no omni variant and cannot process audio, while Qwen3.8-Omni-Flash is a closed-source API without released weights. We therefore evaluate on Qwen2.5-Omni-7B and the latest open-weight omni backbone, Qwen3-Omni-30B-A3B, and obtain consistent gains across both architectures (Tables 2–3).

W5: About Error Recovery in Table 1

Table 1 marks whether a dataset contains any recovery step at all: Video-o3 includes 2K synthetic self-reflection samples, and OmniAgent keeps teacher trajectories that retry environment errors (e.g., invalid timestamps). By contrast, Section 1 refers specifically to recovering from failed student rollouts (Section 2.3), which neither prior work supports. We have updated Table 1 and its caption to clarify this distinction.

AuthorsOpenReview ↗

A quick follow-up before the discussion closes

Dear Reviewer CEBo,

We hope this message finds you well. With the deadline of discussion period is approaching, we wanted to make sure we have fully addressed your concerns. If you have any further points or feedback you would like us to consider, please let us know. Your insights are invaluable to us, and we are open to any further suggestions you might have to improve the paper.

Thank you for the time and effort you have dedicated to reviewing our paper.

AuthorsOpenReview ↗

Follow-up on the discussion

Dear Reviewer CEBo,

Thank you again for reviewing our paper. Before the discussion closes, could you let us know if any of your concerns remain after our response? If a numerical comparison still seems inconsistent, it would help to know which one so we can check it directly.

Best regards, The authors

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer CEBo,

As the discussion period is drawing to a close, we wanted to follow up on our rebuttal. If you have any remaining concerns, could you let us know while there is still time for us to respond?

Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Response

We sincerely thank the reviewers and the Area Chair for their time and constructive feedback. We have carefully studied all comments with detailed responses and additional experiments.

Our framework, Open-OmniCoach, has been recognized for its clarity and empirical design:

  • the clearly structured writing and experimental setup (CEBo, 82hi)
  • the well-motivated supervision of interaction processes, cross-span evidence comparison, and recovery from failed student rollouts (82hi, RhjB)
  • the consistent gains across 7B and 30B backbones, query sources, and data scales (CEBo, 82hi, RhjB)

The core contribution of Open-OmniCoach lies in turning passive audio-visual QA into verifiable, process-supervised evidence acquisition. Beyond the specific benchmarks, our work highlights three takeaways for training omni-modal agents:

  • Task synthesis is central to agent training. Organizing video content into a hierarchical evidence timeline and disclosing task skills progressively (Sec. 2.1) makes it straightforward to construct challenging queries and expand task types (Table 4).
  • Trajectory synthesis goes beyond rollout distillation. Rather than relying solely on rejection-sampled teacher rollouts, structured pipelines—such as multi-span harnesses and recovery branching from failed student rollouts (Stages 2–3)—provide stronger supervision than scaling successful trajectories alone.
  • SEPO applies beyond our dataset. Because SEPO only requires task-level evidence spans, existing QA datasets can obtain those spans from a single forward pass of a frontier model to provide step-level credit during student RL.

Our responses further clarify this design and strengthen its supporting evidence:

  1. Multi-span vs. sequential retrieval (82hi). Evaluated directly on the same off-the-shelf Gemini 3.6 backbone without trajectory training (1,000 stratified queries under identical budgets), multi-span retrieval lifts overall MCQ avg@3 from 68.1 to 70.6 and TimeLens [email protected] from 53.3 to 58.7, while cutting retrieval rounds from 3.5 to 2.3.
  2. Reflection supervision vs. data scale (82hi, RhjB). On a 25% subset (62.0K trajectories), replacing the 4.4K reflection trajectories with 4.4K additional Stage-2 successful trajectories lowers the four-benchmark average from 49.2 to 48.0 (-1.5 on Video-MME-v2, -1.3 on OmniVideoBench). Supervising just the first corrective turn after the mistake uses fewer tokens than this equal-success baseline (20.94M vs. 21.15M) yet already reaches 48.5, and 2–3 corrective turns capture 75%–92% of the full gain (48.9–49.1).
  3. SEPO stability and process behavior (82hi, RhjB). Across three random seeds, SEPO raises the four-benchmark mean over GSPO from 53.2 ± 0.27 to 54.1 ± 0.22 (with non-overlapping mean ± std intervals on Video-MME, Video-MME-v2, and OmniVideoBench), and across 50%/75%/100% RL checkpoints SEPO leads at every stage, with its 50% checkpoint (53.6) already exceeding GSPO's 75% peak (53.3). Trajectory audits on 1,000 OmniVideoBench queries further confirm fewer invalid/no-progress calls (13.2% vs. 15.1%), tighter retrieved spans (42.1s vs. 46.0s), higher focused recall (62.9% vs. 61.3%), and higher perceptual error-to-answer recovery (7.1% vs. 5.0%), with the shared initial overview excluded from scoring and advantage signs fixed by final outcome rewards.
  4. Backbones and numerical clarifications (CEBo). Audio-visual SFT and RL require an open-weight omni backbone (Qwen3.6 has no audio/omni variant and Qwen3.8-Omni-Flash is API-only), and the two +6.3 gains refer to 30B across nine benchmarks and 7B across five audio-visual benchmarks. We also reconcile all dataset and trajectory counts (248.2K trajectories = 230.7K multi-span + 17.4K reflection) and distinguish Table 1's broad Error Recovery label from student-rollout recovery in Section 1.

The individual replies also detail the blind observation-audit protocol, teacher CoT and verifier checks, and the equivalence of Section 3.2 and Appendix E.3 under single-step on-policy updates. All supplementary tables and planned figure/text updates (Figures 1–2, Table 1, Section 4.3, and Appendices) are provided in our responses and will be incorporated into the revised manuscript.

We hope our responses and supplementary experiments address the reviewers' concerns and assist the Area Chair's evaluation.

Best regards,

Authors of Submission 8951

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.