- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
RefCraft evaluates webpage generation from natural-language instructions and one to five visual references, where agents must selectively copy, adapt, or avoid reference properties. It contains 300 tasks with hidden requirement-level keypoints. A shared agent harness supports iterative rendering and revision, while programmatic checks and a VLM judge assess functional and perceptual requirements. The final score gates average keypoint completion by core functionality and reference following. Experiments compare nine frontier models and include reference ablations, human evaluation, judge sensitivity, and a small cross-harness study.
Critical strengths
- The task formulation captures a useful distinction from screenshot reconstruction: a successful solution must interpret which properties of which reference should carry over to a new artifact. The controlled reference-use specification and diverse task composition make this a meaningful benchmark setting.
- The evaluation design combines rendered evidence with executable checks instead of judging source-code claims alone. The 100-artifact expert study and additional judge/harness analyses provide substantive validation, and the component scores help distinguish reference-following failures from broken functionality.
Critical weaknesses
-
The minimum-based score can amplify a low or noisy component, making the reported overall score sensitive to gating decisions. Although the paper provides useful agreement studies and alternative protocol comparisons, the holistic comparison changes both the judging procedure and aggregation. It does not isolate whether the proposed gate improves alignment with human assessment relative to simpler aggregation of the same component judgments.
-
Functional evaluation depends on successfully eliciting behavior from unconstrained generated pages. Appendix B.5 limits the capture planner to eight states, four candidate controls per item, and 900 ms per transition. It remains unclear how often a low functionality score reflects an actual broken feature rather than the evaluator failing to reach or recognize a valid implementation. This matters because core functionality directly gates the total score.
Questions for authors
-
On the existing expert-rated artifacts, could you compare the proposed minimum rule with the base score or a simple component average, keeping all item judgments fixed? Reporting agreement with expert task scores and paired uncertainty for closely ranked models would clarify the gate's benefit without requiring new generation runs.
-
Could you manually replay a small sample of low-core-score or unresolved interaction cases and report how many are genuine implementation failures versus capture/measurement failures? Please also clarify how unavailable evidence affects C and the final score. A few representative failure cases would make the functional evaluation easier to trust.
5 comments
Thank you for comparing aggregation rules on the same artifacts and manually replaying low-scoring cases. These checks clarify why the minimum rule is useful and show that the low functionality scores in the audited cases reflect genuine defects. I will raise my soundness score from 2 to 3. Some uncertainty remains about the ordering of closely matched models, so I am keeping my overall rating
Dear Reviewer G8tV, thank you for your response and for increasing our soundness score. We are pleased that our responses have addressed most of your concerns.
Regarding uncertainty in the ordering of closely matched models, we acknowledge that this is an interesting question that we omitted from the main text, and we would like to provide a quick response. We selected a model pair with a large performance gap, GPT-6 Astra versus GLM-5.3, and a pair with a small gap, Qwen3.8-Max versus Kimi K3. We performed 10,000 paired bootstrap resamples over the same 300 tasks. The mean score differences and marginal 95% bootstrap intervals are shown below:
| Comparison | Score difference | 95% interval |
|---|---|---|
| GPT-6 Astra − GLM-5.3 | 9.52 | [7.90, 10.98] |
| Qwen3.8-Max − Kimi K3 | 1.40 | [−0.17, 2.49] |
The results show a clear performance difference between GPT-6 Astra and GLM-5.3. However, the 95% interval for Qwen3.8-Max versus Kimi K3 includes zero, so this analysis indeed does not provide sufficient evidence to confidently establish their relative ordering. We think this can be viewed from two perspectives:
- We admit that there is indeed room to improve our evaluation method to obtain more stable rankings across models.
- Some models may truly perform similarly on these tasks. In such cases, deliberately choosing a method to enforce a stable ordering with high confidence is difficult. This therefore seems to involve a trade-off in addressing this issue.
In any case, thank you again for prompting us to rethink this issue and helping us identify directions for future improvement. We will incorporate this experiment into our revision. We wish you a pleasant day!
Author Response to Reviewer G8tV
Dear Reviewer G8tV, thank you for taking the time to review our paper and for your constructive suggestions. We hope the following responses address your concerns.
Q1: Can the minimum rule be compared with B or a simple component average against expert task scores, including paired uncertainty for closely ranked models?
A: Thank you for this constructive suggestion. We chose the minimum operator based on the intuition that, in these tasks, both functionality and reference following are critical to user-perceived quality. For example, a user would not give a moderate score if the main function is broken, even if the reference-use requirements are perfectly satisfied. The minimum operator can better reflect this intuition than averaging.
We conducted an additional experiment: we re-aggregated the saved component observations for the nine models in Table 1 across all 300 tasks, keeping both the artifacts and the observations fixed. We compared the minimum operator with B alone, the arithmetic mean, the geometric mean, and a fixed weighted mean with weights B:C:R = 2:1:1. Each rule was applied per task before averaging across tasks.
The absolute scores vary, as expected when applying different aggregation functions, but the relative rankings of the models remain largely stable. To further examine alignment with human judgments, we extended the analysis of the correlation between task-level scores and mean expert scores in Section 5.3.3. As shown below, the minimum operator achieves the strongest alignment, supporting our intuition that peripheral requirements should not compensate for a broken primary function or an unmet reference-use requirement.
We also calculated the relative score gaps for two pairs of closely ranked models: Qwen3.8-Max versus Kimi K3 and Grok 4.6 versus Kimi K3. As shown below, the minimum operator yields greater relative score separation between these closely ranked models.
In summary, we chose the minimum-based scoring protocol because, compared with the alternative aggregation methods tested, it aligns better with human judgments and yields greater relative score separation between closely ranked models.
Q2: How often does a low core score reflect a broken feature versus failure to reach or recognize a valid implementation? How does unavailable evidence affect C and the final score?
A: Thank you for this interesting and important question. To improve evaluation efficiency, we use a fixed time window for collecting page evidence, which we acknowledge can lead to missed evidence. We counted cases of missing evidence and found that the rate was no more than 3%, which is relatively low. Moreover, in our scoring protocol, rubrics with unavailable evidence are excluded from the scoring denominator, so they do not directly lower the final score.
To investigate further, we replayed 20 samples with low core scores and found that only one involved unavailable evidence; the other 19 exhibited actual missing core functionality. We provide a visualization of this evidence-collection failure in Figure 10 of the revised paper. The low score for this particular task was still attributable to missing core functionality. Given our treatment of unavailable evidence and its low occurrence rate, its impact on the scores is limited, so our evaluation remains reliable.