- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes recursive cross-capability self-improvement (RSI) for unified multimodal models. In each round, the model generates charts, diagnoses errors in its own outputs, writes programs to construct targeted training examples, and uses program execution to verify those examples. The authors also introduce BasicChartBench, a benchmark for evaluating chart generation by open models at an early training stage. Experiments with BAGEL-7B-MoT show that RSI improves performance over four rounds, while continued training on the original data yields little improvement. The results suggest that verified programmatic data accounts for most of the gain, with adaptive data selection providing an additional benefit.
Critical strengths
-
The paper combines model-based error diagnosis with program execution as an external verification mechanism. The ablation comparing verified data with self-filtered generations provides evidence that reliable supervision is important for improving chart generation.
-
The experiments include several relevant controls, including continued training, fixed data allocation, single-round diagnosis, and a frozen reader. The authors also compare results with an external judge and report a human audit of the reader’s judgments on native generations.
Critical weaknesses
-
The contribution of recursive cross-capability improvement is not sufficiently isolated. The largest gains appear to come from training on verified programmatically generated charts. The additional benefit of targeting model weaknesses is more modest, while freezing the reader or using a single diagnosis remains competitive on L0. The paper should clarify what RSI contributes beyond verified synthetic-data training and adaptive data allocation.
-
The diagnosis signal remains unreliable on the model’s native generations. In the human audit, the reader still accepts about one third of requirements that humans judge to be absent. The reported improvement in native-generation accuracy is also uncertain, with a 95% interval of [0.0, 7.9]. Although the authors report a strong correlation between reader and human violation rates, further evidence is needed that this ranking is sufficiently reliable to drive data allocation and produce consistent gains.
-
The evaluation is limited in model, task, and scope. The study uses one model and focuses on chart generation over four rounds. It is unclear whether the proposed loop generalizes to other unified multimodal models or structured visual tasks. Moreover, RSI yields only a small improvement on StructT2IBench, while performance decreases on MMBench and MMMU, raising questions about the breadth of the gains and their effect on general capabilities.
-
The benchmark may be closely coupled to the proposed data-generation and evaluation pipeline. BasicChartBench is built from the same specification framework and question-generation rules used for diagnosis and training. Although the paper states that no request shares content with training examples, it does not fully establish that the test set is independent in terms of templates, combinations of chart attributes, or evaluation structure. More analysis is needed to rule out gains from this structural similarity.
-
The reporting contains inconsistencies that make key results difficult to interpret. The abstract reports a final score of 60.2%, whereas Figure 3 reports 59.0% for the pooled benchmark; Table 1 shows 60.2% on L0. The text also describes edited-render accuracy as increasing from 55.6% to 87.4%, while Table 5 lists 87.4% for the “Changed” subset and 98.9% for “All.” These may refer to different subsets or scoring measures, but the paper should state this clearly and align the abstract, main text, and tables. The figures and overall presentation would also benefit from clearer labeling and improved readability.
Questions for authors
-
Please clarify the reported final scores: how do the abstract’s 60.2% and Figure 3’s 59.0% differ in evaluation scope? Likewise, please explain the relationship between the reported 87.4% edited-render accuracy and the “Changed” and “All” results in Table 5.
-
What is the distinct contribution of updating the reader and rediagnosing the model across rounds, beyond training on verified program-generated data? Please quantify this contribution, especially on L0, where the single-diagnosis and frozen-reader controls are close to RSI.
-
How do the authors ensure that BasicChartBench is independent of training beyond content-level overlap? Please discuss potential overlap in templates, attribute combinations, and question-generation structure, and clarify how the small gain on StructT2IBench relates to the benchmark results.
-
Please reconcile the claims about generalization with the observed decreases on MMBench and MMMU, and clarify whether the method’s intended scope is chart generation or broader structured visual tasks.
General Response
We thank the reviewers for their careful reading. Rather than answer them point by point, we use this note to state what we believe the paper contributes, what we have learned since the submission, and where we see this direction going.
The question behind this work is whether the abilities of a unified multimodal model can teach one another. A model that both sees and draws can find its own mistakes, and a model that writes programs can construct the images it cannot yet draw. Together, these abilities form a loop in which the model decides what it needs to learn. Such a loop feeds on its own outputs, however, and errors made in one round are learned in the next. Our central design choice is to place the source of truth outside the model: every chart the model learns from in the loop comes from a program it wrote and is checked by execution against its specification. The model's own judgment decides what to build, never what is correct. This separation, between deciding what to learn and verifying what is learned, is the core idea of the paper, and we expect it to extend to other content that can be written as programs, such as diagrams, tables, and documents.
Reviewers twGm and gDp8 attributed most of the gain to verified construction rather than to the loop. We see verified construction as part of the loop, not an alternative to it. The programs are written by the model itself, so its ability to write code supplies correct training data for its ability to draw. This is one half of the loop. The diagnosis is the other half: it decides what to build, adds to verified construction mainly in the structure and binding of a chart, and, when repeated on the changing model, helps further on the harder external benchmarks.
We also want to correct two points in Reviewer twGm's review. First, the Executor does not synthesize data. It only runs and checks programs, and every program is written by the model itself, so "the simple data-synthesis method with this executor" is not a simpler alternative to RSI; it is RSI without its diagnosis. Second, this experiment is already in the paper. The fixed-distribution control, whose result the review cites in the same paragraph, trains on the same amount of executor-verified data with the same program writer, budget, and update, and differs from RSI only in ignoring the diagnosis. RSI leads it in every round on both levels of BasicChartBench, and in our follow-up experiments it also leads on StructT2IBench.
Since the submission, we have also changed how the loop updates the model. Understanding and generation are now trained jointly, so that the generator learns together with the parameters through which it reads its requests. With this change, the gains carry over to benchmarks built independently of our pipeline: over three rounds, the chart subset of StructT2IBench rises from 11.0% to 26.7%, and that of BizGenEval from 7.6% to 16.5%. We also rebuilt BasicChartBench with open-ended questions scored by an external judge that shares nothing with the model.
The loop has clear limits at this stage. Binding values to the right marks remains hard, the model reads its own generations less reliably than program renders, and knowledge-heavy understanding such as MMMU declines slightly. Because every chart the model builds is verified by execution, these limits affect what the model chooses to learn, not the correctness of what it learns.
We believe the size of the gains is set mainly by scale. Open-source unified models are still small compared with today's foundation models, open or proprietary, such as Kimi K3, GPT, Gemini, and Claude: Kimi K3 alone has 2.8T parameters, while the open-source unified model BAGEL has 14B. Even at this scale, the loop improves the model in every round. How much it can improve depends on the abilities that drive it: how well the model finds its own errors, and how well it writes programs to correct them. These are exactly the abilities that grow as base models become stronger, while execution does not depend on the model and keeps the data correct at any scale. We therefore expect the gains of the loop to grow with the size of the unified model, and testing this across scales is the most important next step of this work.
The question we set out to answer is not how much a 14B model can gain, but whether a unified model can keep supplying its own abilities with correct training data. We believe the paper shows that it can, and we offer it as a paradigm for unified models as they grow stronger.
We have decided to withdraw this submission and continue developing the work. We thank the reviewers again; their questions made the work clearer and stronger.