- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a framework MoHA to adapt models into different harness to improve their performance on long video understanding tasks. By analyzing execution trace, it select modules like memory, overview, verification, and some other configurations. Methods show performance gain in Video-MME benchmark.
Critical strengths
- The objective is clear: adapting the harness to the underlying models for specific downstream tasks.
- The model selection is appropriate, covering representative open-source Qwen models and a closed-source GPT model.
Critical weaknesses
1.Important baselines are missing, how does Qwen Code + Qwen3.5 and Codex+GPT5.6 perform on benchmark test cases? 2.The method mainly selects predefined modules and tunes FPS and input scale, and test on the validation set. Methodological novelty is limited, with few new insights. 3.Lack of in-depth analysis how/why some modules benefit certain models and some don’t. 4.Details on the configuration of the optimizer and judge models are missing.
Questions for authors
Please address the weakness mentioned above
Response to Reviewer Ad4x (1/2)
Thank you for recognizing the validation-gated design and for identifying the baseline and statistical issues affecting our results. Below, we separately address answer completion, evidence use, and comparison with direct observers.
W1 / Q1 — Invalid answers, corrected baselines, and the remaining gains.
A1: We report end-to-end accuracy within 16 planner calls, scoring invalid answers as incorrect. Qwen failures mainly exhaust this budget during search/observe calls or end without a parseable option; after the paper’s terminal-answer rescoring, unresolved outputs are:
On Video-Holmes, the final planner output of Qwen3.5-27B still contains an observe or search request in 1,117/1,837 trajectories (60.8%). At the 16-call limit, a new observe/search request leaves no remaining planner call to integrate further evidence and submit an answer. If the final output also lacks an extractable answer option, the episode is scored as invalid. Therefore, the high invalid-answer rate of 54.3% for Qwen3.5-27B on Video-Holmes is observed.
Random-guess baseline. Replacing only unresolved H0 outputs with uniformly random options gives these expected accuracies:
Random replacement reduces the gaps, but MoHA remains above adjusted H0 expectations.
Relaxed call budget. We completed 16-to-32-call continuations for all 526 originally invalid Qwen3.5-9B H0 trajectories on Video-MME-v1 Long: 57/526 (10.8%) produced valid answers, 34/526 (6.5%) were correct, and 469/526 (89.2%) remained invalid. Doubling the budget still fails to rescue most cases, suggesting limitations in the planner’s effective use of H0. The corresponding 27B continuations are running.
Q2 — Direct-answer Omni baselines.
A2: We evaluated Qwen2.5-Omni and Qwen3-Omni with one direct-answer request per question, using up to 128 uniformly sampled frames and 300 seconds of audio:
Qwen3-Omni matches 9B MoHA on Long and exceeds both Qwen planner stacks on Holmes. Video duration may partly explain this pattern:
The 300-second audio budget covers nearly all Holmes videos but only part of a typical Long video, making direct answering particularly competitive on Holmes.
Longer videos demand targeted observation and evidence integration across distant segments, motivating MoHA’s harness adaptation. Performance also depends jointly on the planner, observer, and harness: a strong observer cannot ensure accurate answers if the planner fails to invoke it appropriately or integrate its evidence. We will add these baselines and analysis to the revised manuscript.