- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates quality assessment for multimodal code generation and introduces "PrismaForge." It employs both task-agnostic and task-specific rubrics to jointly evaluate code execution and the resulting visual output, subsequently utilizing these for data filtering and reinforcement learning (RL). Based on this, the authors developed "PrismaCoder-2M" and trained models with 9B and 35B parameters. These models outperform most open-source alternatives across various vision-based code generation tasks, with some approaching the performance of commercial models.
Critical strengths
- The paper proposes fine-grained, instruction-specific rubrics to jointly evaluate code and its corresponding visual output. By applying these rubrics to data filtering and model training, the authors effectively address the lack of a unified quality signal for heterogeneous visualization tasks.
- The construction of a ~2-million-sample dataset, combined with the training of models at different scales, yields consistent improvements across various tasks and metrics. Furthermore, experiments across different model backbones and detailed case studies demonstrate the framework's significant practical value.
Critical weaknesses
- While the paper successfully demonstrates that rubric-based filtering outperforms direct-score-based filtering—and that the inclusion of RL further enhances model performance—it remains difficult to disentangle the specific contributions of high-quality SFT data, additional RL training, and the rubric-based reward mechanism itself. Appendix D provides a limited comparison—restricted to Python plotting tasks—between the proposed rubric-based rewards and direct-score-based rewards; however, it lacks similar controlled comparisons for other representative tasks. Consequently, the evidence supporting the overall system's effectiveness is considerably stronger than the evidence for the unique advantages of the rubric design itself.
- The training data incorporates publicly available sources such as WebCode2M and UniSVG, which are also linked to the benchmarks used for evaluation. Although the authors state in Appendix A.3 that they employed instruction n-gram matching for data decontamination, it is not explicitly clarified whether they checked for identical or highly similar input images, rendering results, or duplicate samples across different data sources. In tasks such as "screenshot-to-code," text-based deduplication may fail to identify overlaps at the visual level. This does not necessarily imply that a data leak has occurred; however, given the context of large-scale public data re-synthesis, it is necessary to more explicitly rule out this possibility.
Questions for authors
- Beyond matching instruction n-grams, have you checked for visual near-duplicates—such as reference images, rendered outputs, or samples appearing across different data sources?
2 comments
Thank you for the detailed response and additional experiments. The controlled RL comparison across Python and HTML provides useful evidence for the effectiveness of rubric-based rewards. The additional image-level decontamination check also helps address my concerns about evaluation independence. I appreciate the authors' efforts to clarify and strengthen the empirical evidence.
Response to Reviewer 9KDH
Thank you for your insightful review! We appreciate your recognition of our unified quality control framework and our variety of data and model contributions. We would like to address your concerns and questions as follows:
Thank you for pointing out this concern.
We agree that the original Appendix D comparison covers a limited task domain. A full-scale ablation across all RL tasks would require substantial computational and API resources. To provide more direct evidence beyond the original setting, we conducted an additional controlled RL comparison and expanded the training pool from Python-only tasks to include HTML tasks.
Following the protocol in Appendix D, we trained two scoring variants from the same SFT checkpoint. The resulting models evaluated on both ChartMimic (Python) and ArtifactsBench (HTML) are shown below:
Note: ArtifactsBench was evaluated using Gemini 3.8 Flash because Gemini 2.5 Pro has been deprecated.
Under these matched conditions, rubric-based RL exceeds direct-scoring RL by 2.59 points on ChartMimic and 1.81 points on ArtifactsBench. It also improves on the SFT baseline on both benchmarks, whereas direct-scoring RL slightly decreases performance on ChartMimic. Thus, the new experiment more directly isolates the benefit of the reward formulation and extends the comparison beyond Python plotting alone.
These results complement the rubric-based filtering ablation in Section 6.1 and the broader model results in Section 6.2. Specifically, the filtering experiments support the contribution of SFT data curation, while this matched RL comparison supports the contribution of rubric-based rewards relative to direct scoring. Although a complete factorial ablation of every pipeline stage remains beyond the scope of the current experiments, the additional comparison clarifies the distinct sources of improvement.
For multimodal data, it is worth pointing out that a substantial portion of our training data is derived from long-established public datasets that have undergone decontamination during their original construction. Furthermore, our newly collected data originates from source pools distinct from those used to construct the evaluation benchmarks.
To directly rule out the possibility of visual overlapping, we performed an additional image-level decontamination check between the training and evaluation sets. Specifically, we computed 64-bit perceptual hashes for the training and evaluation images and compared their Hamming distances. We treated image pairs with a Hamming distance ≤ 4 as potential near-duplicates requiring further attention. Under this criterion, we identified no overlapping image pairs between our training and evaluation sets.
We thank the reviewer for the suggestion and we will incorporate these details into our final paper.