acceptodds
Withdrawn from ICLR 2027

MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autonomous research seeks sustained model improvements through iterative experimentation and feedback. LLM agents show promise in automating machine learning and language-model post-training, but their ability to sustain multimodal improvement remains unclear. We introduce MMPostTrainBench, a benchmark spanning eight tasks in image, audio, video, and joint audio-video understanding and image-grounded software repair. Agents operate from a common base model within fixed budgets, using development feedback before independent evaluation of their submitted models. Evaluation covers target and non-target model outcomes, iterative model improvement and selection, and research integrity. Across all eight tasks, 52.1% of model–task means fall below the base, and evaluated submissions also exhibit non-target regressions. Model performance does not consistently improve across research iterations, and agents do not reliably select the best evaluated candidate for submission; final submissions trail that candidate by up to 5.38 percentage points. Extending autonomous research from text-only to multimodal tasks introduces additional sources of error in perception, cross-modal alignment, and temporal grounding. The observed regressions and selection gaps highlight the need to balance targeted improvements with non-target capability preservation and to retain gains across research iterations. These requirements motivate MMResearch, a multimodal research framework that connects mediagrounded evidence to hypotheses and interventions, carries findings across rounds through hierarchical memory, and retains candidates using development evaluation. Added to existing code-agent runtimes, it improves submitted-model accuracy by up to 7.75 percentage points for Claude Opus 4.8 with Claude Code and 2.33 points for GPT-5.6-sol with Codex.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Autonomous Research Agents, Self-Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9XkDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MMPostTrainBench, a benchmark for evaluating autonomous multimodal post-training across eight image, audio, video, audio-video, and software-repair tasks. Six model-harness configurations post-train a common Qwen3-Omni base model under fixed budgets. The evaluation covers target performance, non-target capability retention, iterative candidate selection, and research integrity. The authors also propose MMResearch, a seven-step framework combining multimodal evidence, hierarchical memory, and evaluation-guided checkpoint retention.

Critical strengths
  1. The benchmark moves autonomous post-training beyond text-only settings and evaluates six research-model/runtime configurations on eight diverse tasks using a common base model. The separation of development feedback from sealed final evaluation, three independent loops per model-task pair, and retained research trajectories provide a valuable foundation for studying autonomous model improvement.
  2. The paper examines non-target capability regressions, candidate trajectories, final model-selection failures, resource usage, and protocol violations. The integrity analysis is especially valuable: it reveals that agents may exploit label distributions or evaluation information without acquiring the intended multimodal capability. These observations expose practical failure modes that ordinary leaderboards would miss.
Critical weaknesses
  1. The headline MMResearch gains largely measure damage prevention rather than successful autonomous model improvement.

    Several of the largest reported gains arise because MMResearch retains the original base model when the unaugmented agent produces a worse submission. For Claude Opus 4.8, MMResearch scores exactly equal the base model on MMAR, VideoMME-v2, and OmniVideoBench. Therefore, the reported gains of +5.63, +1.92, and +7.75 points do not represent newly acquired capability; they represent avoiding the severe degradation caused by the baseline harness. Indeed, the headline +7.75-point result is obtained by returning to the base score of 39.61 rather than producing a model that improves upon the base.

    A trivial policy that always retains the base model unless a candidate passes a development-set test could reproduce much of this benefit. The paper does not compare MMResearch against such a control with the same evaluation-call budget and checkpoint-selection rule. Consequently, the current results cannot attribute the gains to multimodal evidence reasoning or hierarchical memory. The component ablation on only one model, one task, and apparently one run is insufficient to establish this causal claim.

  2. The benchmark scores are not statistically or procedurally reliable enough to support the reported agent rankings.

    Each model-task result is based on only three loops, without run-level scores, uncertainty estimates, or statistical tests. Nevertheless, the paper ranks systems whose aggregate scores differ by as little as 0.06 percentage points. Non-target retention and candidate-selection analyses use only one unspecified “selected” loop from each model-task pair, creating a substantial risk of selection bias.

    The integrity adjustment introduces an additional scoring problem: flagged runs are replaced by the base score, which can actually improve the score of an underperforming violating run. The final metric therefore mixes research ability, compliance judgments, and an occasionally beneficial reset policy. Moreover, JointAVBench has 91.58% of its answers labeled D, making ordinary accuracy highly vulnerable to label shortcuts. Treating exploitation of this imbalance primarily as an agent-integrity violation does not repair the underlying evaluation defect; the benchmark should use a balanced test set or a metric such as macro recall. Together, these issues make the reported rankings and aggregate conclusions substantially less reliable than the paper suggests.

Questions for authors
  1. What rule was used to select the single loop for non-target evaluation and candidate analysis? Was this rule fixed before observing held-out results?
  2. How does MMResearch compare with a simpler controller that uses the same evaluation-call budget but only retains the base model or best development checkpoint?
  3. How many independent runs were conducted for each MMResearch condition and ablation? Can you report run-level results, confidence intervals, and paired statistical tests?
Reviewer_h9BFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a benchmark to measure autonomous agents' capabilities in automating multimodal post-training. The benchmark is composed of a suite of 8 tasks spanning different modalities, such as image, audio, video and joint audio-video understanding, as well as image-grounded software repair. In all these tasks, the agents start from a common base model and, within a fixed compute budget (8 GPUs for 24h), are allowed to train candidate models and, finally, to select one for their submission. The evaluation of such agents covers the performance of the submitted model, whether the development process gradually improves the candidates' performance, and whether the agent maintains research integrity. The results obtained show that such agents are not reliably able to improve performance on all the above-cited axes. To address this gap, the authors propose MMResearch, a framework added on top of existing coding-agent runtimes that improves their development process by leveraging multimodal evidence observation, which allows directing the research questions, and a hierarchical memory that tracks the current research status with previous experiments and results, all organized in a multi-step research loop. They show that adding this framework to two SOTA agents improves the accuracy of their submitted models with respect to the same agents without it.

Critical strengths
  • S1: Timely setting.
    Extending autonomous post-training research to multimodal models can be a useful contribution, especially in understanding failure modes of current frameworks.
  • S2: Research Integrity insights can be relevant for safety.
    All tested agents were flagged at least once on research integrity flaws, this insight can be an interesting finding that the safety community might benefit from.
Critical weaknesses
  • W1: MMResearch's gains mostly come from falling back to the base model.
    Combining results from Tables 2 and 5 we can see that in 4/8 setups the gains of MMResearch are only matching the base model performance, this doesn't seem to support the improvement claimed by the authors.
    AgentTaskw/o MMResearchw/ MMResearchBaseClaimed gain (vs w/o)Gain vs base
    Opus 4.8MMMU-Pro41.7243.7441.91+2.02+1.83
    Opus 4.8MMAR66.7772.40 (= base)72.40+5.630.00
    Opus 4.8VideoMME-v222.8924.81 (= base)24.81+1.920.00
    Opus 4.8OmniVideoBench31.8639.61 (= base)39.61+7.750.00
    GPT-5.6-solMMMU-Pro41.8844.2141.91+2.33+2.30
    GPT-5.6-solMMAR72.3172.9072.40+0.60+0.50
    GPT-5.6-solVideoMME-v224.3524.81 (= base)24.81+0.460.00
    GPT-5.6-solOmniVideoBench42.9942.6139.61−0.38+3.00
  • W2: MMResearch's effect on non-target capabilities is never measured.
    In the abstract the authors say "The observed regressions and selection gaps highlight the need to balance targeted improvements with non-target capability preservation and to retain gains across research iterations.". However, there's no evaluation of MMResearch on capability retention, it's only measured on the baselines (Table 3). It's unclear whether MMResearch addresses the issue stated in the abstract.
  • W3: Submission loss considers dev-set noise as agent failure. In the paper's example (Sec. 4.2, Tab. 4): Qwen's three MMMU-Pro candidates score 62/137, 62/137 and 59/137 on dev, and 45.65, 44.71 and 43.94 on test. Submitting the second is a logical choice given what information was available to the agent, and yet it's assigned a loss of Rrec=0.94R^{rec} = 0.94. I would suggest the authors to investigate this selection error on the mismatch between the performance of the best and submitted candidates on the dev set, since this is the only information available to the research agent.
  • W4: The writing of the paper is hard to parse, terms are often used before being introduced.
    To give some examples:
    • in the abstract "target and non-target model outcomes" can be interpreted as two models rather than two outcomes.
    • in the introduction "Harbor-compatible task packages", Harbor is never defined, cited or clarified.
Questions for authors
  • Q1: How do the authors plan to address W1?
  • Q2: How does MMResearch affect non-target capabilities?
  • Q3: Will the submission loss be "decomposed" in selection-error and dev-set noise?
  • Q3: Do the authors plan to substantially improve the writing of the paper? At the moment it's an active obstacle to understanding their contribution.
Reviewer_oUwSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MMPostTrainBench evaluates agents that post-train a shared Qwen3-Omni backbone across eight tasks covering image, audio, video, joint audio--video understanding and visual software repair. Agents use development feedback within a fixed research budget, and frozen submissions are evaluated separately. The benchmark measures target performance, non-target retention, candidate trajectories, submission selection and integrity. Across six model/runtime configurations, 52.1% of model--task means fall below the base. The authors also introduce MMResearch, combining media-grounded diagnosis, hierarchical memory and development-guided candidate retention. improving such capability.

Critical strengths
  1. The setting is worthwhile and important. Improving multimodal models is valuable for RSI experiments, but it is also challenging: agents must handle media processing, alignment, and perception errors in addition to writing training code. This makes the benchmark valuable.

  2. The evaluation is sensible and goes beyond final target scores. Non-target probes, candidate histories, and submission gaps expose capability loss and unstable experimentation that a conventional leaderboard would miss. Design choices such as separating execution from verification, freezing submissions, and conducting three independent loops per configuration–task pair are also sensible.

Critical weaknesses
  1. The main gains from MMResearch are not sufficiently disentangled between better research capabilities and avoiding the submission of degraded models. The largest improvement, +7.75 pp, raises the OmniVideoBench score from 31.86 to 39.61, which exactly matches the base model's score. In Table 5, four augmented scores also match the base model at the reported precision. Preserving existing capabilities is certainly valuable, but the authors should clarify how much of the improvement comes from multimodal diagnosis and hierarchical memory, and how much comes from more conservative checkpoint retention and fallback to the base model.

    Moreover, Opus 4.8 is the only model in the main table that performs substantially below the base, with a score of 40.58. A +7.75 pp improvement in this setting therefore seems less surprising.

  2. The authors should report variability across independent runs. The main table reports averages over three independent runs, but the largest overall average improvement is only 0.73 percentage points. This gain is relatively marginal, making variability across runs particularly important.

Questions for authors
  • The benchmark appears to be primarily a multimodal extension of PostTrainBench (Rank et al., 2026). What are the fundamental differences between the two?

  • The base scores are inconsistent. The paper states that base and submitted models are evaluated on “the same fixed held-out test instances” (Sec. 3.2), yet the base scores differ across tables: Table 2 reports 41.91 / 72.40 / 59.94 on MMMU-Pro / MMAR / JointAVBench, whereas the “matched” baselines in Table 9 are 41.72 / 72.90 / 60.06. These gaps (0.1–0.5 pp) are comparable to many reported gains and regressions. The authors should clarify the split used in each table and re-report the results under a consistent protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.