acceptodds
Withdrawn from ICLR 2027

Recursive Self-Improvement in Unified Multimodal Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified multimodal models (UMMs) understand and generate both text and images, which lets a model produce its own training data. Existing self-improvement in UMMs keeps supervision on the visual side, where image understanding judges image generation. We propose recursive cross-capability self-improvement (RSI), a training loop in which the text and visual abilities of a UMM supply training data for one another. In each round, the model generates images and reads them to find where it falls short. It then writes programs aimed at these shortcomings, and execution verifies every result against its specification. Verified renders train image generation, while labeled renders and the model's own correct programs train visual understanding and program writing. Program execution thus acts as a source of truth outside the model, so errors do not accumulate across rounds. We study RSI on charts and build BasicChartBench to evaluate open models early in training. On requests worded differently from training, four rounds of RSI raise the score from 45.7% to 60.2%, while continued training stays at 46.3%. Verified construction carries most of the gain, and targeting the model's failures adds 3.5%. Along the way, the share of verified programs rises from 48.9% to 95.2%, and the reader's accuracy on edited renders rises from 55.6% to 87.4%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Recursive Self-Improvement, Executable Supervision, Unified Multimodal Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gDp8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes recursive cross-capability self-improvement (RSI) for unified multimodal models. In each round, the model generates charts, diagnoses errors in its own outputs, writes programs to construct targeted training examples, and uses program execution to verify those examples. The authors also introduce BasicChartBench, a benchmark for evaluating chart generation by open models at an early training stage. Experiments with BAGEL-7B-MoT show that RSI improves performance over four rounds, while continued training on the original data yields little improvement. The results suggest that verified programmatic data accounts for most of the gain, with adaptive data selection providing an additional benefit.

Critical strengths
  1. The paper combines model-based error diagnosis with program execution as an external verification mechanism. The ablation comparing verified data with self-filtered generations provides evidence that reliable supervision is important for improving chart generation.

  2. The experiments include several relevant controls, including continued training, fixed data allocation, single-round diagnosis, and a frozen reader. The authors also compare results with an external judge and report a human audit of the reader’s judgments on native generations.

Critical weaknesses
  1. The contribution of recursive cross-capability improvement is not sufficiently isolated. The largest gains appear to come from training on verified programmatically generated charts. The additional benefit of targeting model weaknesses is more modest, while freezing the reader or using a single diagnosis remains competitive on L0. The paper should clarify what RSI contributes beyond verified synthetic-data training and adaptive data allocation.

  2. The diagnosis signal remains unreliable on the model’s native generations. In the human audit, the reader still accepts about one third of requirements that humans judge to be absent. The reported improvement in native-generation accuracy is also uncertain, with a 95% interval of [0.0, 7.9]. Although the authors report a strong correlation between reader and human violation rates, further evidence is needed that this ranking is sufficiently reliable to drive data allocation and produce consistent gains.

  3. The evaluation is limited in model, task, and scope. The study uses one model and focuses on chart generation over four rounds. It is unclear whether the proposed loop generalizes to other unified multimodal models or structured visual tasks. Moreover, RSI yields only a small improvement on StructT2IBench, while performance decreases on MMBench and MMMU, raising questions about the breadth of the gains and their effect on general capabilities.

  4. The benchmark may be closely coupled to the proposed data-generation and evaluation pipeline. BasicChartBench is built from the same specification framework and question-generation rules used for diagnosis and training. Although the paper states that no request shares content with training examples, it does not fully establish that the test set is independent in terms of templates, combinations of chart attributes, or evaluation structure. More analysis is needed to rule out gains from this structural similarity.

  5. The reporting contains inconsistencies that make key results difficult to interpret. The abstract reports a final score of 60.2%, whereas Figure 3 reports 59.0% for the pooled benchmark; Table 1 shows 60.2% on L0. The text also describes edited-render accuracy as increasing from 55.6% to 87.4%, while Table 5 lists 87.4% for the “Changed” subset and 98.9% for “All.” These may refer to different subsets or scoring measures, but the paper should state this clearly and align the abstract, main text, and tables. The figures and overall presentation would also benefit from clearer labeling and improved readability.

Questions for authors
  1. Please clarify the reported final scores: how do the abstract’s 60.2% and Figure 3’s 59.0% differ in evaluation scope? Likewise, please explain the relationship between the reported 87.4% edited-render accuracy and the “Changed” and “All” results in Table 5.

  2. What is the distinct contribution of updating the reader and rediagnosing the model across rounds, beyond training on verified program-generated data? Please quantify this contribution, especially on L0, where the single-diagnosis and frozen-reader controls are close to RSI.

  3. How do the authors ensure that BasicChartBench is independent of training beyond content-level overlap? Please discuss potential overlap in templates, attribute combinations, and question-generation structure, and clarify how the small gain on StructT2IBench relates to the benchmark results.

  4. Please reconcile the claims about generalization with the observed decreases on MMBench and MMMU, and clarify whether the method’s intended scope is chart generation or broader structured visual tasks.

Reviewer_twGmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Recursive Self-Improvement (RSI) for Unified Multimodal Models (UMMs) to improve image generation through iterative interactions among generation, visual understanding. During the training, the construction of training data will be adjusted depends on the errors in its generated charts. They also use an external executor to render and verify the resulting charts. What's more, they introduce BasicChartBench to evaluate the chart-generation ability of those weak models.

Critical strengths

This paper introduces BasicChartBench, which can evaluate the chart-generation ability of those weak models. It also can give an evaluation to those models in early training stage.

Critical weaknesses
  1. The experiment results shows that RSI mainly provides notable improvements on their BasicChartBench. But in other benchmark, e.g. the chart subset of BizGenEval and understanding part, still performs poor. This may indicate the method is not general or only good at the specific setting.

  2. The RSI may be overclaimed. From fig3, the Fixed Distribution baseline already gains a large improvement. And table1 also shows that the frozen reader can get a very competive result with full RSI. I'm curious about whether the simple data-synthesis method with this executor can already get a good performance. And how much performance is gained actually from the RSI. It will be helpful if you provide the experiment result that only use the executor-generated data to do the continued training.

Questions for authors

Please see the weakness.

Reviewer_GZf8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies recursive self-improvement (RSI) for chart generation. The model diagnoses its own images and writes programs to create training data within predefined capability dimensions, such as chart type, number format, and vocabulary. Verified renders train generation, while labeled renders and correct programs train reading and program writing. Verified targets drive most gains. Adapting data helps further, and repeated diagnosis matters more than updating the reader.

Critical strengths
  1. Executing and verifying programs supplies correct chart examples that the model cannot obtain simply by detecting its own mistakes. Verified renders outperform self-filtered model images under matched requests and generation-training steps. Adapting the training data also beats a fixed distribution, including on unseen content and wording.

  2. The paper clearly reports the reader's limits, supported by a blind human audit. Its large gains on edited renders transfer only weakly to the model's own images. The frozen-reader and single-diagnosis experiments also show that repeated diagnosis helps even without updating the reader.

Critical weaknesses
  1. Binding (matching chart marks to the right labels and values) remains difficult despite better text rendering. Gains on harder external benchmarks are small. The paper supports an early-learning result, but the practical gains for reliable chart generation are still limited.

  2. The gains across capabilities are uneven. Most of the program-writing gain comes in the first round. The reader improves on edited renders, but does not become better at catching real errors in generated charts in the human audit (Table 2). General understanding also declines on MMBench and MMMU. Repeated diagnosis helps generation, but the evidence that these capabilities keep improving each other across rounds is more limited.

Questions for authors
  1. Which capability dimensions advanced to a harder level through the reader-based mastery rule, and in which rounds? The paper reports the vocabulary change, but this uses the separate OCR gates. This would clarify whether the reader guided increases in difficulty, beyond allocating more data to current weaknesses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading. Rather than answer them point by point, we use this note to state what we believe the paper contributes, what we have learned since the submission, and where we see this direction going.

The question behind this work is whether the abilities of a unified multimodal model can teach one another. A model that both sees and draws can find its own mistakes, and a model that writes programs can construct the images it cannot yet draw. Together, these abilities form a loop in which the model decides what it needs to learn. Such a loop feeds on its own outputs, however, and errors made in one round are learned in the next. Our central design choice is to place the source of truth outside the model: every chart the model learns from in the loop comes from a program it wrote and is checked by execution against its specification. The model's own judgment decides what to build, never what is correct. This separation, between deciding what to learn and verifying what is learned, is the core idea of the paper, and we expect it to extend to other content that can be written as programs, such as diagrams, tables, and documents.

Reviewers twGm and gDp8 attributed most of the gain to verified construction rather than to the loop. We see verified construction as part of the loop, not an alternative to it. The programs are written by the model itself, so its ability to write code supplies correct training data for its ability to draw. This is one half of the loop. The diagnosis is the other half: it decides what to build, adds to verified construction mainly in the structure and binding of a chart, and, when repeated on the changing model, helps further on the harder external benchmarks.

We also want to correct two points in Reviewer twGm's review. First, the Executor does not synthesize data. It only runs and checks programs, and every program is written by the model itself, so "the simple data-synthesis method with this executor" is not a simpler alternative to RSI; it is RSI without its diagnosis. Second, this experiment is already in the paper. The fixed-distribution control, whose result the review cites in the same paragraph, trains on the same amount of executor-verified data with the same program writer, budget, and update, and differs from RSI only in ignoring the diagnosis. RSI leads it in every round on both levels of BasicChartBench, and in our follow-up experiments it also leads on StructT2IBench.

Since the submission, we have also changed how the loop updates the model. Understanding and generation are now trained jointly, so that the generator learns together with the parameters through which it reads its requests. With this change, the gains carry over to benchmarks built independently of our pipeline: over three rounds, the chart subset of StructT2IBench rises from 11.0% to 26.7%, and that of BizGenEval from 7.6% to 16.5%. We also rebuilt BasicChartBench with open-ended questions scored by an external judge that shares nothing with the model.

The loop has clear limits at this stage. Binding values to the right marks remains hard, the model reads its own generations less reliably than program renders, and knowledge-heavy understanding such as MMMU declines slightly. Because every chart the model builds is verified by execution, these limits affect what the model chooses to learn, not the correctness of what it learns.

We believe the size of the gains is set mainly by scale. Open-source unified models are still small compared with today's foundation models, open or proprietary, such as Kimi K3, GPT, Gemini, and Claude: Kimi K3 alone has 2.8T parameters, while the open-source unified model BAGEL has 14B. Even at this scale, the loop improves the model in every round. How much it can improve depends on the abilities that drive it: how well the model finds its own errors, and how well it writes programs to correct them. These are exactly the abilities that grow as base models become stronger, while execution does not depend on the model and keeps the data correct at any scale. We therefore expect the gains of the loop to grow with the size of the unified model, and testing this across scales is the most important next step of this work.

The question we set out to answer is not how much a 14B model can gain, but whether a unified model can keep supplying its own abilities with correct training data. We believe the paper shows that it can, and we offer it as a paradigm for unified models as they grow stronger.

We have decided to withdraw this submission and continue developing the work. We thank the reviewers again; their questions made the work clearer and stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.