acceptodds
Under review as a conference paper at ICLR 2027

PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis

16 views

Abstract

Multimodal code intelligence connects the symbolic world of programs with the visual world their execution creates. Interest in this capability is burgeoning, as models are increasingly asked to produce charts, webpages, animations, and semi-structured artifacts. Recently, proprietary models have pushed far ahead, producing interactive artifacts even from a text description or a screenshot alone. Yet their data and training remain closed, and open progress is held back by the quantity and quality of available data and by the pipelines and methods for producing it. Quality is the harder of the two, since a sample must be correct as a program, well built, and above all visually right once rendered. No existing signal covers all three. Execution often returns just a pass or fail, code similarity does not map to the visual output, and task-specific verifiers do not generalize. To push open progress further, we start from the data side with PrismaForge, a hierarchical data pipeline centered on a synthesized rubric that decomposes these factors into checkable criteria. The same rubric serves as both gatekeeper and reward: it curates the supervised corpus, then drives reinforcement learning on visual-coding tasks. Leveraging PrismaForge for data synthesis and filtering, we build PrismaCoder-2M, the most diverse open dataset for multimodal code intelligence and the largest to date. On the model side, we train PrismaCoder-9B and PrismaCoder-35B on this data and release both. They lead the open state of the art by a wide margin and rival leading commercial systems. Beyond the headline benchmarks, the curated corpus lifts a backbone family outside our release, and its gains hold under a different judge. Code and data are available at this link.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, Code Generation, Code Data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9KDHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates quality assessment for multimodal code generation and introduces "PrismaForge." It employs both task-agnostic and task-specific rubrics to jointly evaluate code execution and the resulting visual output, subsequently utilizing these for data filtering and reinforcement learning (RL). Based on this, the authors developed "PrismaCoder-2M" and trained models with 9B and 35B parameters. These models outperform most open-source alternatives across various vision-based code generation tasks, with some approaching the performance of commercial models.

Critical strengths
  1. The paper proposes fine-grained, instruction-specific rubrics to jointly evaluate code and its corresponding visual output. By applying these rubrics to data filtering and model training, the authors effectively address the lack of a unified quality signal for heterogeneous visualization tasks.
  2. The construction of a ~2-million-sample dataset, combined with the training of models at different scales, yields consistent improvements across various tasks and metrics. Furthermore, experiments across different model backbones and detailed case studies demonstrate the framework's significant practical value.
Critical weaknesses
  1. While the paper successfully demonstrates that rubric-based filtering outperforms direct-score-based filtering—and that the inclusion of RL further enhances model performance—it remains difficult to disentangle the specific contributions of high-quality SFT data, additional RL training, and the rubric-based reward mechanism itself. Appendix D provides a limited comparison—restricted to Python plotting tasks—between the proposed rubric-based rewards and direct-score-based rewards; however, it lacks similar controlled comparisons for other representative tasks. Consequently, the evidence supporting the overall system's effectiveness is considerably stronger than the evidence for the unique advantages of the rubric design itself.
  2. The training data incorporates publicly available sources such as WebCode2M and UniSVG, which are also linked to the benchmarks used for evaluation. Although the authors state in Appendix A.3 that they employed instruction n-gram matching for data decontamination, it is not explicitly clarified whether they checked for identical or highly similar input images, rendering results, or duplicate samples across different data sources. In tasks such as "screenshot-to-code," text-based deduplication may fail to identify overlaps at the visual level. This does not necessarily imply that a data leak has occurred; however, given the context of large-scale public data re-synthesis, it is necessary to more explicitly rule out this possibility.
Questions for authors
  1. Beyond matching instruction n-grams, have you checked for visual near-duplicates—such as reference images, rendered outputs, or samples appearing across different data sources?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9KDH

Thank you for your insightful review! We appreciate your recognition of our unified quality control framework and our variety of data and model contributions. We would like to address your concerns and questions as follows:

W1: Appendix D provides a limited comparison—restricted to Python plotting tasks—between the proposed rubric-based rewards and direct-score-based rewards; however, it lacks similar controlled comparisons for other representative tasks.

Thank you for pointing out this concern.

We agree that the original Appendix D comparison covers a limited task domain. A full-scale ablation across all RL tasks would require substantial computational and API resources. To provide more direct evidence beyond the original setting, we conducted an additional controlled RL comparison and expanded the training pool from Python-only tasks to include HTML tasks.

Following the protocol in Appendix D, we trained two scoring variants from the same SFT checkpoint. The resulting models evaluated on both ChartMimic (Python) and ArtifactsBench (HTML) are shown below:

MethodChartMimic (Overall)ArtifactsBench (Overall)
PrismaCoder-9B-SFT77.1391528.04
+ RL (Direct Scoring)76.0428.99
+ RL (Rubric-based Scoring)78.6330.80

Note: ArtifactsBench was evaluated using Gemini 3.8 Flash because Gemini 2.5 Pro has been deprecated.

Under these matched conditions, rubric-based RL exceeds direct-scoring RL by 2.59 points on ChartMimic and 1.81 points on ArtifactsBench. It also improves on the SFT baseline on both benchmarks, whereas direct-scoring RL slightly decreases performance on ChartMimic. Thus, the new experiment more directly isolates the benefit of the reward formulation and extends the comparison beyond Python plotting alone.

These results complement the rubric-based filtering ablation in Section 6.1 and the broader model results in Section 6.2. Specifically, the filtering experiments support the contribution of SFT data curation, while this matched RL comparison supports the contribution of rubric-based rewards relative to direct scoring. Although a complete factorial ablation of every pipeline stage remains beyond the scope of the current experiments, the additional comparison clarifies the distinct sources of improvement.

W2 & Q1: Text-based deduplication may fail to identify overlaps at the visual level. Beyond matching instruction n-grams, have you checked for visual near-duplicates—such as reference images, rendered outputs, or samples appearing across different data sources?

For multimodal data, it is worth pointing out that a substantial portion of our training data is derived from long-established public datasets that have undergone decontamination during their original construction. Furthermore, our newly collected data originates from source pools distinct from those used to construct the evaluation benchmarks.

To directly rule out the possibility of visual overlapping, we performed an additional image-level decontamination check between the training and evaluation sets. Specifically, we computed 64-bit perceptual hashes for the training and evaluation images and compared their Hamming distances. We treated image pairs with a Hamming distance ≤\leq 4 as potential near-duplicates requiring further attention. Under this criterion, we identified no overlapping image pairs between our training and evaluation sets.

We thank the reviewer for the suggestion and we will incorporate these details into our final paper.

Thank you for the detailed response and additional experiments. The controlled RL comparison across Python and HTML provides useful evidence for the effectiveness of rubric-based rewards. The additional image-level decontamination check also helps address my concerns about evaluation independence. I appreciate the authors' efforts to clarify and strengthen the empirical evidence.

Reviewer_7AhMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PrismaForge, a data filtering pipeline that dynamically generates instruction-specific evaluation criteria for each training sample using a two-dimensional rubric covering visual and code quality, thereby providing richer supervision signals. Building on PrismaForge, the authors construct PrismaCoder-2M, currently the largest multimodal code dataset. They further use PrismaCoder-2M for SFT and the PrismaForge pipeline for RL to train two models with 9B and 35B parameters, respectively achieving superior performance over several open-source models across multiple benchmarks.

Critical strengths
  1. This paper introduces PrismaForge, which dynamically synthesizes code and visual rubrics to perform fine-grained evaluations of generated code and its rendered outputs, producing task-specific scoring criteria.
  2. Building on PrismaForge, this paper constructs and releases PrismaCoder-2M, the largest and most comprehensive open multimodal code dataset, providing a substantial data resource for future research in multimodal code generation.
  3. This paper trains and releases two models, PrismaCoder-9B and PrismaCoder-35B-A3B, through SFT and rubric-based RL, achieving SOTA performance among open-source models across multiple benchmarks.
Critical weaknesses
  1. The main contribution of this paper is the data construction and training framework named PrismaForge. However, the experiments primarily focus on comparing the performance of the final models against other open-source models, and comparing Rubric-based Filtering with Direct-judge Filtering. Given the differences in training data, backbone architectures, and training strategies across models. It is suggested to conduct controlled experiments using the same backbone, data scale, and training budget, while employing different pipelines to construct SFT datasets and generate RL reward signals, and then comparing the performance of the resulting models. This would provide a more comprehensive validation of the effectiveness of the proposed method.
  2. The data generation, rubric generation, and quality evaluation processes in this paper rely on powerful closed-source models, including Claude Opus 4.6, Gemini 3 Pro, and Gemini 3 Flash, respectively. However, the impact of replacing the generation and evaluation models within the pipeline on data filtering quality and downstream model performance has not been sufficiently investigated. It is suggested to further explore the effectiveness of PrismaForge when built upon models with varying capability levels, to assess the framework's sensitivity to the choice of underlying models.
  3. This paper primarily emphasizes the use of code and visual rubrics for quality evaluation and their reuse in both SFT data filtering and RL. However, Appendix D further reveals that an additional Holistic Score is incorporated into the reward formulation for certain tasks, accounting for approximately 30%–40% of the final reward. It would be beneficial to include more comprehensive ablation studies comparing Rubric Only, Holistic Only, and Rubric + Holistic, to better disentangle the contributions of each component and assess the effectiveness of the proposed rubric-based reward design.
  4. This paper draws on a large number of existing publicly available datasets as sources for data synthesis. Currently, duplicate detection primarily relies on long n-gram matching between training and test instructions. Given the multimodal nature of the code data considered in this work, relying solely on text-based matching may not be sufficient to identify semantically similar samples with different textual expressions, or samples that share identical or highly similar visual content.
Questions for authors
  1. This paper primarily describes the scoring mechanism based on code and visual rubrics, while Appendix D further introduces an additional Holistic Score, which accounts for 30%–40% of the final reward in certain tasks. Could the authors further clarify the motivation behind this design choice? Is the Holistic Score also used for SFT data filtering, or is it exclusively employed during RL?
  2. Appendix A.3 mentions that data decontamination is primarily performed through long n-gram matching between training and test instructions. Could the authors provide more details on the decontamination settings, such as the specific n-gram length and matching threshold used?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7AhM (1/3)

Thank you for your detailed and constructive review! We appreciate your recognition of PrismaForge and our contributions in data construction and model development. We would like to address your concerns and questions as follows:

W1: It would be better to conduct controlled comparisons using the same backbone, data scale, and training budget, while varying the SFT data construction and RL reward pipelines. This would provide a more comprehensive validation of the effectiveness of the proposed method.

Thank you for your suggestion. We agree that a fully controlled, end-to-end comparison would provide additional evidence of the effectiveness of PrismaForge. However, such experiments require substantial computational and API resources, particularly given the multiple stages involved in data construction, SFT, and RL training.

Nevertheless, we have conducted controlled experiments targeting the key components of PrismaForge to isolate and validate their individual contributions:

  • Data curation (Section 6.1): The effectiveness of rubric-based quality evaluation for data filtering and selection;
  • Effectiveness across model backbones (Section 6.2): The benefits of our framework are not limited to a particular architecture;
  • RL reward design (Appendix D): Rubrics-based reward formulation yields greater improvements than direct scoring on Python plotting tasks.

To strengthen the last point, we additionally expanded the RL training pool from Python-only tasks to Python and HTML tasks. Starting from the same SFT checkpoint and keeping the remaining training conditions fixed, we compared direct-scoring and rubric-based RL. Both resulting models were evaluated on ChartMimic (Python) and ArtifactsBench (HTML):

MethodChartMimic (Overall)ArtifactsBench (Overall)
PrismaCoder-9B-SFT77.1391528.04
+ RL (Direct Scoring)76.0428.99
+ RL (Rubric-based Scoring)78.6330.80

Note: ArtifactsBench was evaluated using Gemini 3.8 Flash because Gemini 2.5 Pro has been deprecated.

Compared with direct scoring, rubric-based rewards yield higher scores on both benchmarks. Moreover, direct-scoring RL hurts the model on ChartMimic performance, while rubric-based RL improves performance on both benchmarks. This provides evidence for the advantage of the reward formulation under controlled conditions beyond the original Python-only evaluation. Together, the component-level experiments support the central design choices of PrismaForge.

AuthorsOpenReview ↗

Response to Reviewer 7AhM (2/3)

W2: It would be better to evaluate PrismaForge with models of varying capabilities to assess its robustness on data generation and validation.

Thank you for the suggestion. Given the cost of retraining downstream models and the time limit of this discussion period, we conducted an additional data-selection sensitivity analysis to see the influence of backbone model changes. We examined three alternative configurations using Qwen3.5-35B-A3B:

  1. Direct judge samples with Qwen3.5-35B-A3B without rubrics;
  2. Replace the judge model LJ\mathcal{L}^J in PrismaForge with Qwen3.5-35B-A3B;
  3. Replace the rubric generation model LR\mathcal{L}^R and judge model LJ\mathcal{L}^J in PrismaForge with Qwen3.5-35B-A3B.

We start from the same seed pool and apply these three configurations for data filtering. We then measure the Jaccard similarity between the resulting selected datasets and the same dataset filtered by the original model configuration in our main experiments. The results are shown below:

Data-selection configurationJaccard similarity to original PrismaForge
Direct Judge (Qwen)0.31
PrismaForge (LJ\mathcal{L}^J = Qwen)0.82
PrismaForge (LR\mathcal{L}^R = Qwen, LJ\mathcal{L}^J = Qwen)0.69

Replacing only the judge retains substantial overlap with the original selected dataset (0.82). Replacing both the rubric generator and judge reduces this overlap to 0.69, but it remains considerably higher than the overlap obtained with direct judging (0.31). These findings suggest that rubric-guided selection is relatively stable under models with varying capabilities.

AuthorsOpenReview ↗

Response to Reviewer 7AhM (3/3)

W3 & Q1: Could the authors further clarify the motivation for incorporating the Holistic Score and whether it is used in SFT filtering or only in RL.

As discussed in Appendix D, our motivation for incorporating a holistic score stems from the unpredictable nature of some particular tasks considered in this work. These tasks are characterized by admitting multiple valid implementations and considerable flexibility in visual presentation. Consequently, some aspects of output quality can only be reliably assessed by examining the actual generated artifact in the context of the original instruction and reference image.

Rubrics generated solely from the instruction may not fully anticipate all possible errors in the generated outputs, while holistic scores alone are less consistent and comparable across samples. Our combined reward formulation is therefore designed to leverage the complementary strengths of both approaches on these particular tasks.

Regarding its use in different training stages, we employ holistic scoring only during RL training and not in SFT data filtering. This is because SFT data filtering primarily aims to identify and retain sufficiently high-quality training examples, while RL training benefits from more fine-grained reward signals that can distinguish subtle differences in output quality among generated responses. In our experiments, we found that rubric-based filtering alone was sufficient to achieve the desired data quality for SFT, whereas the combined reward formulation was adopted for RL to provide a more comprehensive quality signal.

W4 & Q2: Text-based decontamination may fail to detect visually overlapping samples in multimodal code datasets. Could the authors provide more details on the decontamination settings?

We clarify our text-based decontamination procedure and provide an additional image-level overlap analysis below.

For text decontamination, we use 5-gram matching with MinHash and Locality-Sensitive Hashing to efficiently identify candidate pairs with high textual similarity. Specifically, we compute Jaccard similarity over the corresponding n-gram sets and remove a training sample if its similarity to any evaluation sample exceeds 0.9.

For multimodal data, it is worth pointing out that a substantial portion of our training data is derived from long-established public datasets that have undergone decontamination during their original construction. Furthermore, our newly collected data originate from source pools distinct from those used to construct the evaluation benchmarks.

To directly address this concern, we performed an additional image-level decontamination check between the training and evaluation sets. Specifically, we computed 64-bit perceptual hashes for the training and evaluation images and compared their Hamming distances. We treated image pairs with a Hamming distance ≤\leq 4 as potential near-duplicates requiring further attention. Under this criterion, we identified no overlapping image pairs between our training and evaluation sets.

We thank the reviewer for this constructive suggestion and will incorporate these details into our final paper.

Reviewer_APaGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PrismaForge synthesizes instruction-specific code and visual rubrics for data filtering and reinforcement-learning rewards. The resulting PrismaCoder-2M corpus supports 9B and 35B-A3B multimodal coding models. Experiments cover chart, webpage, SVG, chemical, and other visual-programming tasks, with additional backbone and judge analyses.

Critical strengths
  • The broad task coverage and improvements over the corresponding base models in Tables 2–3 make the data and pipeline practically valuable.
  • The equal-sized, same-pool filtering comparison in §6.1 favors rubric-based selection. Together with the additional-backbone results in §6.2, this supports benefits beyond simply increasing data volume.
Critical weaknesses
  • Appendix D explicitly lacks a full-training comparison of rubric rewards against direct scoring; it instead reports a focused Python plotting comparison. Thus the evidence for rubric-based curation is stronger than the evidence for the reward design across all RL domains.
  • In Appendix H.2, humans and model judges use the same rubric in the rubric condition, on 100 Python/HTML samples. Higher agreement supports consistency within that scoring scheme, but does not by itself establish better alignment with independent human preferences across the full dataset.
Questions for authors
  1. In the focused RL comparison in Appendix D, were the prompt pool, starting checkpoint, rollout budget, and judge model held fixed?
  2. Does any existing evaluation compare the resulting artifacts using human preferences collected without showing the generated rubric?
1 comment
AuthorsOpenReview ↗

Response to Reviewer APaG

Thank you for your thoughtful review! We appreciate your recognition of the broad task coverage and the controlled filtering results. We would like to address your comments and questions as follows:

W1. Appendix D explicitly lacks a full-training comparison of rubric rewards against direct scoring.

Thank you for highlighting this limitation. As mentioned in Appendix D, due to the constraints of the resouces we have, it is hard to do a full training comparison across this two methods. Nevertheless, Fig.5 in Section 6.2 demonstrated consistent performance improvements from the RL stage across all evaluated benchmarks on both Gemma-4-31B and Qwen3-VL-30B-A3B. These results suggest that our rubric-based reward design is effective across different model backbones and evaluation domains.

To address this concern more directly, we conducted additional controlled RL experiments comparing rubric-based scoring against direct scoring. Importantly, we expanded the original Python-only training pool to include HTML tasks, allowing us to examine whether the benefits of rubric-based rewards generalize beyond a single domain. Both methods were trained under identical experimental settings, with the reward formulation being the only difference. The results are shown below:

MethodChartMimic (Overall)ArtifactsBench (Overall)
PrismaCoder-9B-SFT77.1391528.04
+ RL (Direct Scoring)76.0428.99
+ RL (Rubric-based Scoring)78.6330.80

Note: ArtifactsBench was evaluated using Gemini 3.8 Flash because Gemini 2.5 Pro has been deprecated.

Compared with direct scoring, rubric-based rewards yield higher scores on both benchmarks. Moreover, direct-scoring RL hurts the model on ChartMimic performance, while rubric-based RL improves performance on both benchmarks. These controlled results provide more direct evidence that the benefits of rubric-based rewards extend beyond the original Python-only comparison. Together with the broader RL improvements in Figure 5, they strengthen the empirical support for our reward design.

We will incorporate these additional experiments and the corresponding discussion into the final version of the paper.

Q1: In the focused RL comparison in Appendix D, were the prompt pool, starting checkpoint, rollout budget, and judge model held fixed?

Yes. In both the original comparison in Appendix D and the additional experiments described above, we controlled for all four factors: the prompt pool, starting checkpoint, rollout budget, and judge model were identical across the compared methods. The only experimental variable was the reward formulation.

W2 & Q2: The human study in Appendix H.2 does not by itself establish better alignment with independent human preferences across the full dataset. Does any existing evaluation compare the resulting artifacts using human preferences collected without showing the generated rubric?

Thank you for this important observation.

To address this concern, we conducted an additional human evaluation in which annotators assessed artifact quality independently, without access to the generated rubrics. Specifically, human annotators assigned holistic quality scores on a 1–5 scale based on their own judgments. We then compared these independent human scores with model-generated scores obtained using the rubric-based evaluation framework, following the evaluation setup in Appendix H.2. The statistical results are shown below:

Scoring comparisonKrippendorff's alphaKendall's tau_b
Human Direct vs. Model Direct0.300.28
Human Direct vs. Model Rubric0.650.57

The results demonstrate substantially stronger alignment between rubric-based model scores and independent human judgments than between direct model scores and human judgments.

The key idea here is that human evaluators can assess artifact quality holistically while implicitly considering multiple dimensions. In constrast, direct model scoring requires a judge to translate these multidimensional assessments into a single numerical score without explicit evaluation criteria, which can lead to inconsistent scoring across responses. Our rubric-based framework addresses this challenge by explicitly decomposing quality assessment into interpretable criteria, providing a more structured and consistent basis for assigning rewards.

We will include this additional evaluation and discussion in the final version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.