- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MMPostTrainBench, a benchmark for evaluating autonomous multimodal post-training across eight image, audio, video, audio-video, and software-repair tasks. Six model-harness configurations post-train a common Qwen3-Omni base model under fixed budgets. The evaluation covers target performance, non-target capability retention, iterative candidate selection, and research integrity. The authors also propose MMResearch, a seven-step framework combining multimodal evidence, hierarchical memory, and evaluation-guided checkpoint retention.
Critical strengths
- The benchmark moves autonomous post-training beyond text-only settings and evaluates six research-model/runtime configurations on eight diverse tasks using a common base model. The separation of development feedback from sealed final evaluation, three independent loops per model-task pair, and retained research trajectories provide a valuable foundation for studying autonomous model improvement.
- The paper examines non-target capability regressions, candidate trajectories, final model-selection failures, resource usage, and protocol violations. The integrity analysis is especially valuable: it reveals that agents may exploit label distributions or evaluation information without acquiring the intended multimodal capability. These observations expose practical failure modes that ordinary leaderboards would miss.
Critical weaknesses
-
The headline MMResearch gains largely measure damage prevention rather than successful autonomous model improvement.
Several of the largest reported gains arise because MMResearch retains the original base model when the unaugmented agent produces a worse submission. For Claude Opus 4.8, MMResearch scores exactly equal the base model on MMAR, VideoMME-v2, and OmniVideoBench. Therefore, the reported gains of +5.63, +1.92, and +7.75 points do not represent newly acquired capability; they represent avoiding the severe degradation caused by the baseline harness. Indeed, the headline +7.75-point result is obtained by returning to the base score of 39.61 rather than producing a model that improves upon the base.
A trivial policy that always retains the base model unless a candidate passes a development-set test could reproduce much of this benefit. The paper does not compare MMResearch against such a control with the same evaluation-call budget and checkpoint-selection rule. Consequently, the current results cannot attribute the gains to multimodal evidence reasoning or hierarchical memory. The component ablation on only one model, one task, and apparently one run is insufficient to establish this causal claim.
-
The benchmark scores are not statistically or procedurally reliable enough to support the reported agent rankings.
Each model-task result is based on only three loops, without run-level scores, uncertainty estimates, or statistical tests. Nevertheless, the paper ranks systems whose aggregate scores differ by as little as 0.06 percentage points. Non-target retention and candidate-selection analyses use only one unspecified “selected” loop from each model-task pair, creating a substantial risk of selection bias.
The integrity adjustment introduces an additional scoring problem: flagged runs are replaced by the base score, which can actually improve the score of an underperforming violating run. The final metric therefore mixes research ability, compliance judgments, and an occasionally beneficial reset policy. Moreover, JointAVBench has 91.58% of its answers labeled D, making ordinary accuracy highly vulnerable to label shortcuts. Treating exploitation of this imbalance primarily as an agent-integrity violation does not repair the underlying evaluation defect; the benchmark should use a balanced test set or a metric such as macro recall. Together, these issues make the reported rankings and aggregate conclusions substantially less reliable than the paper suggests.
Questions for authors
- What rule was used to select the single loop for non-target evaluation and candidate analysis? Was this rule fixed before observing held-out results?
- How does MMResearch compare with a simpler controller that uses the same evaluation-call budget but only retains the base model or best development checkpoint?
- How many independent runs were conducted for each MMResearch condition and ablation? Can you report run-level results, confidence intervals, and paired statistical tests?