acceptodds
Under review as a conference paper at ICLR 2027

RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development

Abstract

Beyond basic tool use and coding skills, effectively incorporating visual references into agentic generation is equally essential for applying AI agents to real-world productivity tasks. We study this crucial capability through webpage development, a practical and readily evaluable setting in which existing benchmarks remain largely idealized: they typically focus on reconstructing visual targets or generating webpages from textual instructions alone, overlooking the more common workflow of developing from visual examples. Specifically, we introduce RefCraft, a benchmark comprising 300 realistic multimodal webpage-development tasks that combine natural-language instructions with screenshots from one to five reference webpages. Agents must selectively interpret and utilize these references while rendering, visually inspecting, and iteratively refining their outputs. We evaluate the resulting open-ended artifacts using independently verifiable keypoints and complementary programmatic and VLM judges. Across nine frontier (M)LLMs, the strongest system achieves an overall score of only 52.12 out of 100, revealing that visual-reference-conditioned agentic generation remains a substantial challenge for current models in practical webpage-development settings.

Primary area:
datasets and benchmarks
Keywords:
Visual Agents, Visual Reference, Web Development

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_G8tVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

RefCraft evaluates webpage generation from natural-language instructions and one to five visual references, where agents must selectively copy, adapt, or avoid reference properties. It contains 300 tasks with hidden requirement-level keypoints. A shared agent harness supports iterative rendering and revision, while programmatic checks and a VLM judge assess functional and perceptual requirements. The final score gates average keypoint completion by core functionality and reference following. Experiments compare nine frontier models and include reference ablations, human evaluation, judge sensitivity, and a small cross-harness study.

Critical strengths
  1. The task formulation captures a useful distinction from screenshot reconstruction: a successful solution must interpret which properties of which reference should carry over to a new artifact. The controlled reference-use specification and diverse task composition make this a meaningful benchmark setting.
  2. The evaluation design combines rendered evidence with executable checks instead of judging source-code claims alone. The 100-artifact expert study and additional judge/harness analyses provide substantive validation, and the component scores help distinguish reference-following failures from broken functionality.
Critical weaknesses
  1. The minimum-based score can amplify a low or noisy component, making the reported overall score sensitive to gating decisions. Although the paper provides useful agreement studies and alternative protocol comparisons, the holistic comparison changes both the judging procedure and aggregation. It does not isolate whether the proposed gate improves alignment with human assessment relative to simpler aggregation of the same component judgments.

  2. Functional evaluation depends on successfully eliciting behavior from unconstrained generated pages. Appendix B.5 limits the capture planner to eight states, four candidate controls per item, and 900 ms per transition. It remains unclear how often a low functionality score reflects an actual broken feature rather than the evaluator failing to reach or recognize a valid implementation. This matters because core functionality directly gates the total score.

Questions for authors
  1. On the existing expert-rated artifacts, could you compare the proposed minimum rule with the base score or a simple component average, keeping all item judgments fixed? Reporting agreement with expert task scores and paired uncertainty for closely ranked models would clarify the gate's benefit without requiring new generation runs.

  2. Could you manually replay a small sample of low-core-score or unresolved interaction cases and report how many are genuine implementation failures versus capture/measurement failures? Please also clarify how unavailable evidence affects C and the final score. A few representative failure cases would make the functional evaluation easier to trust.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer G8tV

Dear Reviewer G8tV, thank you for taking the time to review our paper and for your constructive suggestions. We hope the following responses address your concerns.

Q1: Can the minimum rule be compared with B or a simple component average against expert task scores, including paired uncertainty for closely ranked models?

A: Thank you for this constructive suggestion. We chose the minimum operator based on the intuition that, in these tasks, both functionality and reference following are critical to user-perceived quality. For example, a user would not give a moderate score if the main function is broken, even if the reference-use requirements are perfectly satisfied. The minimum operator can better reflect this intuition than averaging.

We conducted an additional experiment: we re-aggregated the saved component observations for the nine models in Table 1 across all 300 tasks, keeping both the artifacts and the observations fixed. We compared the minimum operator with B alone, the arithmetic mean, the geometric mean, and a fixed weighted mean with weights B:C:R = 2:1:1. Each rule was applied per task before averaging across tasks.

ModelMinimumB onlyArithmeticGeometricWeighted 2:1:1
GPT-6 Astra52.1275.6877.1766.8176.70
Qwen3.8-Max48.9573.8674.6562.6174.35
Grok 4.648.8372.9673.7861.7273.54
Kimi K347.5572.7473.5661.1673.28
Claude Fable 544.6172.0570.8957.2471.12
GLM-5.342.6071.2371.0056.1970.98
DeepSeek-V4-Pro37.0971.1666.2048.1667.37
Seed2.1 Pro32.0669.0061.6641.8963.51
Gemini 3.1 Pro19.0161.4552.6825.3754.95

The absolute scores vary, as expected when applying different aggregation functions, but the relative rankings of the models remain largely stable. To further examine alignment with human judgments, we extended the analysis of the correlation between task-level scores and mean expert scores in Section 5.3.3. As shown below, the minimum operator achieves the strongest alignment, supporting our intuition that peripheral requirements should not compensate for a broken primary function or an unmet reference-use requirement.

ModelMinimumB onlyArithmeticGeometricWeighted 2:1:1
GPT-6 Astra0.750.590.720.730.71

We also calculated the relative score gaps for two pairs of closely ranked models: Qwen3.8-Max versus Kimi K3 and Grok 4.6 versus Kimi K3. As shown below, the minimum operator yields greater relative score separation between these closely ranked models.

ComparisonMinimumB onlyArithmeticGeometric
Qwen3.8-Max − Kimi K32.94%1.54%1.48%2.36%
Grok 4.6 − Kimi K32.68%0.30%0.31%0.91%

In summary, we chose the minimum-based scoring protocol because, compared with the alternative aggregation methods tested, it aligns better with human judgments and yields greater relative score separation between closely ranked models.

Q2: How often does a low core score reflect a broken feature versus failure to reach or recognize a valid implementation? How does unavailable evidence affect C and the final score?

A: Thank you for this interesting and important question. To improve evaluation efficiency, we use a fixed time window for collecting page evidence, which we acknowledge can lead to missed evidence. We counted cases of missing evidence and found that the rate was no more than 3%, which is relatively low. Moreover, in our scoring protocol, rubrics with unavailable evidence are excluded from the scoring denominator, so they do not directly lower the final score.

To investigate further, we replayed 20 samples with low core scores and found that only one involved unavailable evidence; the other 19 exhibited actual missing core functionality. We provide a visualization of this evidence-collection failure in Figure 10 of the revised paper. The low score for this particular task was still attributable to missing core functionality. Given our treatment of unavailable evidence and its low occurrence rate, its impact on the scores is limited, so our evaluation remains reliable.

Thank you for comparing aggregation rules on the same artifacts and manually replaying low-scoring cases. These checks clarify why the minimum rule is useful and show that the low functionality scores in the audited cases reflect genuine defects. I will raise my soundness score from 2 to 3. Some uncertainty remains about the ordering of closely matched models, so I am keeping my overall rating

Dear Reviewer G8tV, thank you for your response and for increasing our soundness score. We are pleased that our responses have addressed most of your concerns.

Regarding uncertainty in the ordering of closely matched models, we acknowledge that this is an interesting question that we omitted from the main text, and we would like to provide a quick response. We selected a model pair with a large performance gap, GPT-6 Astra versus GLM-5.3, and a pair with a small gap, Qwen3.8-Max versus Kimi K3. We performed 10,000 paired bootstrap resamples over the same 300 tasks. The mean score differences and marginal 95% bootstrap intervals are shown below:

ComparisonScore difference95% interval
GPT-6 Astra − GLM-5.39.52[7.90, 10.98]
Qwen3.8-Max − Kimi K31.40[−0.17, 2.49]

The results show a clear performance difference between GPT-6 Astra and GLM-5.3. However, the 95% interval for Qwen3.8-Max versus Kimi K3 includes zero, so this analysis indeed does not provide sufficient evidence to confidently establish their relative ordering. We think this can be viewed from two perspectives:

  1. We admit that there is indeed room to improve our evaluation method to obtain more stable rankings across models.
  2. Some models may truly perform similarly on these tasks. In such cases, deliberately choosing a method to enforce a stable ordering with high confidence is difficult. This therefore seems to involve a trade-off in addressing this issue.

In any case, thank you again for prompting us to rethink this issue and helping us identify directions for future improvement. We will incorporate this experiment into our revision. We wish you a pleasant day!

Reviewer_uVyAOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RefCraft is 300 webpage development tasks where the agent gets a text instruction plus screenshots of one to five reference pages and must write a single self contained HTML file. Tasks are built by sampling a reference use specification over six visual dimensions and five requirement types (copy, adapt, invert, specify, ignore), verbalizing it into a request, then decomposing it into keypoints checked either programmatically or by a VLM judge. The task score is the minimum of a keypoint score, a core function score and a reference following score, and nine frontier models run in a shared OpenCode harness, topping out at 52.12.

Critical strengths
  1. The benchmark setting is the right one. Reconstructing a page from a screenshot is not what people actually do, whereas borrowing a palette from one site and a layout from another is, and the copy/adapt/invert/specify/ignore taxonomy makes that concrete enough to sample and verify. It also pays off, since the finding that adapt scores well below copy and invert is something no other formulation would surface.

  2. Validation goes well past what most benchmark papers bother with. Ten experts on 100 artifacts at weighted kappa 0.64, Spearman 0.89 against an external human preference leaderboard, three judges on a fixed subset, and a cross harness rerun. The screenshot only ablation is the most persuasive piece, since unresolved keypoints jump from 15 to 1,987 once browser measurements are removed.

Critical weaknesses
  1. The min aggregation is doing most of the work. Keypoint scores sit between 61 and 76 while overall scores span 19 to 52, so the leaderboard mostly reports whichever component collapsed, and there is no ablation over the aggregation rule itself. The same thing makes the reference ablation partly tautological, since removing references has to fail reference keypoints and R gates the total.

  2. The two dimensions carrying the headline failure are the two the evaluator judges least reliably. Interaction scores lowest of the six visual dimensions at 43.6 and overall visual character second lowest at 58.8, and Table 3 reports evaluator against expert agreement for exactly those categories at weighted kappa 0.55 and 0.47, against 0.72 for concrete visual attributes. That is moderate agreement at best, and the human annotators disagree with each other most in the same places, so the claim that agents cannot coordinate style across design choices or handle interactive state is weakest precisely where the paper states it most forcefully.

Questions for authors
  1. What do the scores and the ranking look like under a weighted mean or a geometric mean of B, C and R? Right now a reader cannot separate model behavior from the min operator.

  2. Please fix the Appendix A.6 spreads and say whether a 6.3 point harness swing is compatible with the stability claim, given Qwen3.8-Max, Grok 4.6 and Kimi K3 are 1.4 points apart.

  3. Does the full 300 task ranking move when the VLM judge changes, or only the 32 task subset? And was any model other than Gemini 3.1 Pro tried as profiler and query generator?

  4. Is Table 1 a single rollout per task? It is recommended to include an analysis of the model's stability across different rollouts.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer uVyA (1/2)

Dear Reviewer uVyA, thank you for your detailed review and positive assessment. We hope the following responses address your remaining concerns.

Q1: The minimum operator may dominate the leaderboard. What happens under weighted or geometric means?

A: Thank you for this insightful question. We chose the minimum operator based on the intuition that, in these tasks, both functionality and reference following are critical to user-perceived quality. For example, a user would not give a moderate score if the main function is broken, even if the reference-use requirements are perfectly satisfied. The minimum operator can better reflect this intuition than averaging.

We also conducted an additional ablation experiment: we re-aggregated the saved component observations for the nine models in Table 1 across all 300 tasks, keeping both the artifacts and the observations fixed. We compared the minimum operator with B alone, the arithmetic mean, the geometric mean, and a fixed weighted mean with weights B:C:R = 2:1:1. Each rule was applied per task before averaging across tasks.

ModelMinimumB onlyArithmeticGeometricWeighted 2:1:1
GPT-6 Astra52.1275.6877.1766.8176.70
Qwen3.8-Max48.9573.8674.6562.6174.35
Grok 4.648.8372.9673.7861.7273.54
Kimi K347.5572.7473.5661.1673.28
Claude Fable 544.6172.0570.8957.2471.12
GLM-5.342.6071.2371.0056.1970.98
DeepSeek-V4-Pro37.0971.1666.2048.1667.37
Seed2.1 Pro32.0669.0061.6641.8963.51
Gemini 3.1 Pro19.0161.4552.6825.3754.95

The absolute scores vary, as expected when applying different aggregation functions, but the relative rankings of the models remain largely stable. To further examine alignment with human judgments, we extended the analysis of the correlation between task-level scores and mean expert scores in Section 5.3.3. As shown below, the minimum operator achieves the strongest alignment, supporting our intuition that peripheral requirements should not compensate for a broken primary function or an unmet reference-use requirement.

ModelMinimumB onlyArithmeticGeometricWeighted 2:1:1
Pearson0.750.590.720.730.71

Q2: Gating by R makes the reference ablation partly tautological.

A: Thank you for this question. For the reference ablation experiment in Table 2, we conducted an additional experiment in which we replaced the minimum operator with the arithmetic mean. GPT-6 Astra's score still dropped substantially, from 77.2 to 53.6, indicating that minimum gating is not the main driver of the performance drop. However, we agree that this ablation setting is somewhat unusual, since references are themselves central to our benchmark. Our original intention in conducting this experiment was to demonstrate that our setting is meaningful: models cannot recover the reference-dependent requirements from the textual query and their priors alone, although this finding is somewhat intuitive. We will clarify the motivation and interpretation of this experiment in the revision and also consider whether to remove it.

Q3: Interaction and overall visual character score lowest, but also have the weakest evaluator–expert agreement.

A: Thank you for pointing this out. We agree that the relatively weak evaluator–expert agreement limits the strength of our conclusions about Interaction and Overall visual character, and we will temper our claims regarding these two dimensions. However, we would like to emphasize that low correlation does not directly indicate whether absolute scores are high or low. We analyzed the scores assigned by human experts and found that Interaction and overall visual character also received lower scores than the other dimensions. Given the relatively subjective nature of these two dimensions, scores vary across human and model evaluators. Nevertheless, the finding that these dimensions score lower than others, such as Visual attributes, appears reliable overall.

AuthorsOpenReview ↗

Author Response to Reviewer uVyA (2/2)

Q4: Please fix the Appendix A.6 spreads and explain whether a 6.3-point harness swing is compatible with the stability claim.

A: We apologize for the incorrect values reported in the table. The values for the Codex harness in Table 7 (now Table 10 in our revison) are outdated. The correct values are 52.0 and 49.3, so the differences of 2.8 and 1.1 reported in the main text are correct (with 1.1 resulting from rounding). We have updated the table in the revision. We acknowledge that, even after this correction, the variation is not negligible. However, our stability claim is primarily based on the relatively stable ranking between models with a clearer performance gap, namely GPT-6 Astra and Kimi K3. The true performance differences among Qwen3.8-Max, Grok 4.6, and Kimi K3 are indeed small, as also reflected in their pairwise comparisons in Table 4.

Q5: Does the full 300-task ranking move when the VLM judge changes, or only the 32-task subset? Was another model tried as profiler and query generator?

A: The experiment on stability across VLM judges in Appendix A.3 indeed used a fixed 32-task sample for a quick initial assessment. We additionally selected four models and evaluated them across all 300 tasks. The results below show that their ranking remains relatively stable when the VLM judge changes.

ModelGemini 3.1 ProGPT-6 AstraClaude Opus 5
GPT-6 Astra52.153.253.5
Kimi K347.647.349.2
GLM-5.342.641.442.7
DeepSeek-V4-Pro37.135.636.9

For the profiler and query generator, we use Gemini as the base model. However, their outputs serve only as initial drafts of the benchmark content, and we will invite experts to audit and revise them, as discussed in Section 3.4. Therefore, directly comparing different models in this setting is less informative. Nevertheless, we collected some queries and profiling results generated by different models, and our qualitative analysis suggests that they are all reasonably sound. An example can be found in Figure 9 in our revision.

Q6: Is Table 1 a single rollout per task? Please analyze stability across different rollouts.

A: Yes, Table 1 uses a single rollout per task. We additionally selected four models and conducted three rollouts per task for each model. The results are shown below:

ModelRun #1Run #2Run #3
GPT-6 Astra52.152.452.2
Kimi K347.647.447.6
GLM-5.342.642.342.7
DeepSeek-V4-Pro37.137.537.2

The results are relatively stable across rollouts, which supports our initial decision to use a single rollout for efficiency. Nevertheless, we acknowledge that multiple rollouts do provide more reliable estimates.

Thank you for the responses, which have addressed my main concerns, particularly the aggregation-rule ablation and the full-benchmark judge-swap experiment. I will keep my positive rating.

Dear Reviewer uVyA, thank you for your response and positive rating. We are glad to see that we have addressed your main concerns. We wish you a nice day!

Reviewer_inFQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RefCraft, a benchmark comprising 300 realistic multimodal webpage-development tasks. Each task combines natural-language instructions with screenshots from one to five reference webpages. The resulting open-ended artifacts are evaluated using independently verifiable keypoints and complementary programmatic and VLM judges. The paper evaluates multiple models, including Qwen3.8-Max and Kimi K3.

Critical strengths
  1. RefCraft allows models to follow user instructions and reference webpage screenshots through an “inspect-and-iterate” process, supporting a broader range of webpage-development scenarios. The resulting artifacts are assessed through a dual-track evaluation protocol.

  2. The paper benchmarks nine frontier (M)LLMs, including Qwen3.8-Max and Kimi K3, under a unified harness. The results reveal considerable room for improvement in existing models.

Critical weaknesses

The substantial discrepancy between holistic scores and those produced by the proposed evaluation protocol indicates that the results are highly sensitive to the scoring protocol. Since the comparison changes both assessment granularity and aggregation, the justification for the chosen scoring protocol appears insufficient.

Questions for authors

Table 7 implies cross-harness spreads of 3.8 points for GPT-6 Astra and 6.3 points for Kimi K3, rather than the stated 2.8 and 1.1. Are the reported figures correct? Please reconcile these discrepancies.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer inFQ

Dear Reviewer inFQ, thank you for taking the time to review our paper. We are pleased that you recognize the value of our benchmark setting and experimental results. We appreciate your focused questions about the scoring protocol and the cross-harness results, and we hope the following responses address your concerns.

Q1: The results seem sensitive to the scoring protocol, and the holistic comparison changes both assessment granularity and aggregation, so the justification for the chosen scoring protocol is insufficient.

A: Thank you for this question. We would first like to clarify the experiment in Appendix A.4 that you referred to, as we acknowledge that its description is somewhat confusing. In RefCraft, we use a diverse set of keypoints to evaluate an artifact. A naive alternative is to evaluate the output directly without decomposing the assessment into keypoints, which is exactly our holistic scoring setting. As stated in our original paper (now at line 869 in the revision), because this setting changes both assessment granularity and aggregation, this comparison alone cannot establish which protocol better captures artifact quality. This statement is consistent with your concern. The only conclusion we draw from this experiment is that holistic scoring is prone to overlooking certain dimensions, leading to an overestimation of artifact quality. Our evaluation protocol is not necessarily optimal, but it is more reliable and better able to identify concrete defects. We have clarified this in the revision.

Moreover, beyond the above experiment, we agree that ablations of different scoring protocols are important. We therefore conducted an additional experiment: we re-aggregated the saved component observations for the models in Table 1 on RefCraft, keeping both the artifacts and the observations fixed. We compared the minimum operator with B alone, the arithmetic mean, the geometric mean, and a fixed weighted mean with weights B:C:R = 2:1:1. Each rule was applied per task before averaging across tasks. The results are shown below:

ModelMinimumB onlyArithmeticGeometricWeighted 2:1:1
GPT-6 Astra52.1275.6877.1766.8176.70
Qwen3.8-Max48.9573.8674.6562.6174.35
Grok 4.648.8372.9673.7861.7273.54
Kimi K347.5572.7473.5661.1673.28
Claude Fable 544.6172.0570.8957.2471.12
GLM-5.342.6071.2371.0056.1970.98
DeepSeek-V4-Pro37.0971.1666.2048.1667.37
Seed2.1 Pro32.0669.0061.6641.8963.51
Gemini 3.1 Pro19.0161.4552.6825.3754.95

The absolute scores vary, as expected when applying different aggregation functions, but the relative rankings of the models remain largely stable. To further examine alignment with human judgments, we extended the analysis of the correlation between task-level scores and mean expert scores in Section 5.3.3. As shown below, the minimum operator achieves the strongest alignment, supporting our intuition that peripheral requirements should not compensate for a broken primary function or an unmet reference-use requirement.

ModelMinimumB onlyArithmeticGeometricWeighted 2:1:1
Pearson0.750.590.720.730.71

These results suggest that the keypoint-based evaluation protocol is relatively reliable and stable, and that our chosen aggregation method aligns well with human judgments. If we have misunderstood your concerns or you have further questions, please let us know. We would be happy to discuss them further.

Q2: There seems to be a mismatch between the results in Table 7 and the main text.

A: Thank you very much for catching this inconsistency. The values for the Codex harness in the original Table 7 (now Table 10 in the revision) were outdated. The correct values are 52.0 and 49.3, so the differences of 2.8 and 1.1 reported in the main text are correct (with 1.1 resulting from rounding). We have updated the table in the revision. Thank you again for bringing this to our attention.

Dear Reviewer inFQ,

We greatly appreciate the time you've invested in reviewing our submission. Having submitted our response, we want to know if it has addressed your concerns. As the end of the rebuttal phase approaches, we look forward to hearing from you for feedback or any further clarification that you might require. Thanks!

Thank you for the detailed response, which have addressed my main concerns. I have read the rebuttal and the other reviews.I will keep my rating.

Dear Reviewer inFQ,

Thank you for your response. We are glad that our rebuttal has addressed your main concerns, and we would be happy to discuss further if you have any remaining questions. Thank you again for your valuable time, and we wish you a good day.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comments to All Reviewers

We sincerely thank all reviewers for their time and constructive feedback. We are encouraged that the benchmark setting and contributions of our work have been recognized. In particular, Reviewer uVyA and Reviewer G8tV highlighted that our benchmark provides a meaningful and appropriate setting, while Reviewer BRCu noted that it reveals considerable room for improvement in frontier models. We also acknowledge the concerns that have been raised, especially regarding ablations of the evaluation protocol, and we try to address them in detail below. Moreover, we have also updated the paper PDF to incorporate all revisions for your consideration, with additions shown in green and modifications in red. If our responses adequately resolve your concerns, we would greatly appreciate your consideration in updating your score.

Area_Chair_GX8kOpenReview ↗

AC Note: Rebuttal Received, Please Engage in Discussion

Dear authors and reviewers,

Thank you to the authors for their detailed rebuttal and for the time and care they put into addressing the reviewers' comments. I appreciate the additional clarifications and results.

Reviewers: please read the authors' responses carefully and reply to them before the end of the discussion period. Even a short acknowledgment is valuable. It is especially helpful to say whether your concerns have been addressed and, if not, what remains unresolved. If the rebuttal changes your assessment, please update your score and review accordingly, whether up or down. If your score stays the same, a brief explanation of why helps me and the senior area chairs.

Authors: I will follow the discussion closely and take your rebuttal fully into account when forming my recommendation.

Thank you all for your time and effort.

Best regards, Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.