acceptodds
Under review as a conference paper at ICLR 2027

VisExpress: Beyond Rendering—Benchmarking How LLM-Generated HTML Answers Convey Information

Abstract

As the code-generation capabilities of Large Language Models (LLMs) advance, applications powered by LLMs increasingly present answers as HTML pages integrating text, charts, media, and interactive components. When appropriately designed, such responses can make information easier to understand and navigate. However, existing benchmarks primarily focus on code correctness, component quality, or design fidelity, paying limited attention to how effectively a complete HTML answer communicates information. To address this issue, we introduce VisExpress, a benchmark for evaluating the visual expression quality of complete LLM-generated HTML answers. VisExpress comprises 786 queries across 16 domains, combining real-user queries with curated queries that broaden the coverage of presentation tasks. It includes component-necessity annotations and covers 12 component forms organized into four categories: structured text, charts, media, and interactive components. We further develop a multimodal evaluation framework consisting of 13 metrics organized into two complementary dimensions. Design Normativity assesses the basic visual quality of page elements and layouts, while Presentation Effectiveness assesses whether component selection and information organization help users find and understand information. Using multimodal evidence—including screenshots, individual images, and recordings of animations and interactions—the framework evaluates both missing necessary representations and redundant components. Experiments across 12 models show that the model with the highest Design Normativity score trails the best model in Presentation Effectiveness by 9.6 points. While most models produce orderly layouts, component selection and information organization remain key challenges. The framework shows positive alignment with human judgments and matches the aggregate model ranking observed in our human study.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Evaluation, Visual Information Presentation, HTML Answer Benchmark, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_2u6qOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VisExpress, a benchmark for evaluating the visual expression and information communication quality of LLM-generated HTML pages, comprising 786 queries across 16 domains. The authors propose a multimodal evaluation framework with 13 metrics organized into two dimensions, Design Normativity and Presentation Effectiveness, to assess page quality, component selection, and information organization. Through experiments on 12 models and human evaluations, the authors find that strong visual design does not necessarily translate into effective information communication, with component selection and information organization remaining key challenges for current models. The paper provides a comprehensive and well-structured description of its dataset composition and evaluation framework.

Critical strengths
  1. Practical Relevance. The paper addresses a meaningful research problem by moving beyond the visual aesthetics or code correctness of HTML pages to examine how effectively they organize and communicate information. This focus is more closely aligned with the practical needs of real-world generative UI applications.

  2. Comprehensive Evaluation Framework. The authors introduce two evaluation dimensions, Design Normativity and Presentation Effectiveness, covering not only component quality and layout but also component necessity, information organization, and redundant content. The paper also provides detailed analyses and human evaluations to validate the proposed framework.

  3. Dataset Scale and Diversity. The dataset covers a reasonably diverse range of queries, and the authors provide a detailed description of the dataset construction process, including human review. If publicly released, the dataset could serve as a valuable resource for future research.

Critical weaknesses
  1. Limited Novelty of the Evaluation Framework. The paper primarily integrates existing evaluation metrics and relies heavily on VLM-based assessment. It does not introduce substantial methodological innovations that clearly distinguish it from existing approaches. Therefore, whether the proposed evaluation framework should be considered a novel contribution remains questionable.

  2. Insufficient Fine-Grained Error Analysis and Visual Examples. Although the authors propose a comprehensive set of evaluation dimensions, the paper lacks detailed visual examples of specific error cases that would help illustrate the purpose and effectiveness of each metric. More qualitative analyses and comparisons of generated HTML pages would strengthen the justification for the proposed evaluation dimensions.

  3. Insufficient Related Work Discussion and Comparisons. The paper does not adequately discuss or compare against relevant prior works, such as WiserUI-Bench, making it difficult to clearly establish its advantages over existing benchmarks. Additionally, more visual examples from the dataset would help readers better understand the benchmark's design, functionality, and evaluation capabilities.

Questions for authors

Same as weakness

Reviewer_rTzVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VisExpress, a benchmark for evaluating LLM-generated HTML answers. It comprises 786 queries across 16 domains, component-necessity annotations, and an evaluation framework with 13 metrics grouped into two dimensions: Design Normativity and Presentation Effectiveness. Using this benchmark, the authors evaluate 12 models. They also conduct a human ranking study involving outputs from four models, with five annotators per query, to examine the alignment between automated evaluation scores and human judgments.

Critical strengths

The problem is timely and important, and the benchmark provides a useful evaluation framework. Evaluating HTML-based answers requires assessing more than visual quality or successful rendering. VisExpress addresses this need by assessing basic presentation quality, component selection, and information organization, providing a structured basis for evaluating and improving generated HTML answers.

Critical weaknesses
  1. The fine-grained evaluation needs stronger justification and empirical validation. The paper defines 13 metrics with scoring rubrics, but the choice and granularity of these metrics are not sufficiently justified. Although Section 4 outlines general evaluation principles, the connection between these principles and individual metrics should be made clearer, supported by theoretical foundations or empirical evidence. Moreover, Section 5.2 reports metric-level scores and illustrative examples but provides limited independent validation of whether the metrics capture their intended aspects of performance. For example, assessing representative outputs could help determine whether low scores correspond to the intended failure types and whether the reported model-specific strengths and weaknesses are reflected in the actual outputs.

  2. The human study provides limited validation of the proposed evaluation framework. First, ranking generated HTML answers captures user preferences but does not directly measure how effectively the answers support information understanding. A task-based study measuring accuracy and completion time would provide more direct evidence, while controlling for content correctness and task difficulty. Second, the reported agreement is moderate: mean human Kendall's W is 0.4596, and human–evaluator S1 agreement is 56.95%, compared with 62.15% human–human agreement. While these results suggest some alignment, further analysis is needed to understand the sources of disagreement and identify which judgments are reliable. In particular, examining cases where the evaluator disagrees with a strong human consensus could reveal limitations of the proposed metrics.

  3. The contribution of component-necessity annotations to evaluation is unclear. The dataset provides Required, Optional, and Inappropriate labels, but the evaluation appears to assess component appropriateness without explicitly using these annotations. While the labels help characterize dataset coverage, their added value to evaluation is not demonstrated. The authors should clarify their role and consider using them as independent references to validate component-necessity and omission judgments.

Questions for authors
  1. What theoretical or empirical evidence supports the selection, definitions, and granularity of the 13 metrics? Can you independently verify that low scores reflect the intended failure types and that the reported model-specific strengths and weaknesses are evident in the generated outputs?

  2. Beyond preference rankings, what evidence shows that higher scores correspond to better information acquisition or comprehension? A task-based study measuring accuracy and completion time would provide more direct evidence. Alternatively, further analysis of the existing human study, including agreement across task or component types and representative disagreement cases, could help clarify the reliability and limitations of the metrics.

  3. What evaluation benefits do the component-necessity annotations provide beyond characterizing dataset coverage? Can you validate their usefulness by comparing the evaluator's necessity and omission judgments with the annotations and analyzing any discrepancies?

  4. How consistent are the metric scores and component-appropriateness judgments across repeated evaluations of the same HTML answers? Given that most evaluators use a temperature of 1.0, a repeat-evaluation analysis would help establish their reliability.

  5. What motivates disabling reasoning for some models while using low reasoning effort for others? How sensitive are the findings, particularly those related to component selection and information organization, to these settings?

Reviewer_oRFYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VisExpress, a benchmark with 786 queries for evaluating the presentation of LLM-generated HTML answers. It uses 13 metrics to assess visual quality, component choices, and information organization. Experiments compare 12 models and examine agreement between the proposed scores and human rankings.

Critical strengths
  1. The paper addresses a relevant problem and evaluates complete HTML answers, considering both static content and the behavior of interactive components.

  2. The human study is substantial, with five annotators per query. Model identities are hidden and answer order is randomized, which helps reduce bias in the comparisons.

Critical weaknesses
  1. The paper does not show what the proposed evaluation adds. It introduces many judging criteria and hand-chosen weights, but provides no comparison showing that this design produces more useful or reliable assessments than a straightforward whole-answer judge. Even the full score has no clear advantage over Presentation Effectiveness alone.

  2. The claim about conveying information needs stronger support. The human study asks raters to compare readability, organization, and component usefulness. These judgments support agreement on presentation preferences. They do not establish whether higher-scoring answers help users understand the information or find what they need more easily.

Questions for authors

What evidence shows that the proposed scoring system captures communication effectiveness beyond general presentation preferences?

Reviewer_KAwBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VisExpress, a benchmark for evaluating how well a full LLM-generated HTML conveys information, rather than checking code correctness or fidelity to a reference page.

It contributes:

1: A dataset of 786 self-contained queries across 16 domains, drawn from domain-specific candidates and real-user queries, with 731 component-necessity labels over 8 annotation types, produced by Kimi-K2.5 and human spot-checked.

2: A 13-metric multimodal LLM-judge framework, with Gemini 3 Flash Preview, Gemini 2.5 Pro, and Qwen 3.5-397B serving as judges. The metrics fall under two axes: Design Normativity (element quality, layout regularity) and Presentation Effectiveness (expression coherence, visual expressiveness). The final score were calculated by multiplying the Design Normativity and Presentation Effectiveness.

3: An evaluation of 12 frontier LLMs, all run through the same generation-and-rendering pipeline at a 390×1000 mobile viewport, scored on full-page screenshots, cropped component images, extracted text, and video captures of animations and interactions.

4: A blinded human ranking study across four models: 770 matched queries, 5 annotators per query from a pool of 60, with an ablation on the aggregation method.

Critical strengths

VisExpress tackles a real and under-measured research question: whether a LLM-generated HTML answer actually conveys information. Its scoring is purely information-gain-based per component, with penalties for both omissions and redundancy, in contrast to code-correctness or visual-fidelity metrics used in prior benchmarks.

The reporting is unusually complete for a judge-based benchmark: a 786×5 human ranking study with a human–human reference, bootstrap CIs, a common-query robustness check, and an aggregation ablation.

Critical weaknesses

While I think the author put great effort and the paper is easy to read, the evidence that the automatic scores track human judgment is too thin to support the paper's claims.

1: Contribution 1 is overstated.

  • Table 10 lists no judge input containing the necessity annotations. So from my understanding, a judge must re-infer at runtime, producing model-dependent applicability denominators
  • The redundancy penalty is too weak with only 3 Inappropriate labels.
  • There is no dedicated metric for Maps and Audio cards.

2: The human agreement numbers lack a meaningful baseline, and the human study does not cover the leaderboard.

  • Agreement is compared only against random. Because humans strongly prefer Claude Fable 5.1 overall (1.95 vs. 2.57–2.88), a trivial baseline ranking every answer by its model's global mean score could reach similar agreement. The authors need to provide a meaningful baseline to show that the judges add per-answer signal.
  • The human study covers only 4 of 12 models, all well separated. For example, GPT-6 Astra, the model behind the headline D-vs-B finding, is not included.

3: Design Normativity (D) rewards models for skipping optional components.

  • Animation quality and Interaction quality only count toward D when a model generates those components (Equation 2). Since most annotations are Optional, so a model that skips them simply dodges the hardest metrics in D while taking no omission penalty under Presentation Effectiveness (B). I think this is why GPT-6 Astra's got the top D score (90.31) despite generating far fewer animations and images than Claude Fable 5.1.
  • Table 20 shows D alone is near chance against humans and B alone beats D·B/100, yet the product lifts GPT-6 Astra from 7th in B to 2nd overall.

4: The benchmark's scope is narrower than claimed.

The abstract, introduction and Table 1 present a general benchmark for "LLM-generated HTML answers". I think the claims need more qualifications since:

  • Queries and pages are Chinese-only, and rendering is a 390×1000 mobile viewport.
  • Models emit component declarations into a shared renderer rather than free-form HTML, and the renderer silently drops failed components, which conflates parse errors with deliberate omissions.
  • Layout Regularity largely measures the framework rather than the model.

5: Judge reliability is unquantified.

  • Most judges run a single call at T = 1.0 with no test-retest variance.
  • None of the 13 leaf metrics is validated against humans individually.
  • Gemini and Qwen judge their own model families without a self-preference check.
Questions for authors

1: Are the Required/Optional/Inappropriate annotations passed to any judge or used to fix metric applicability? -If not, why let T = 1.0 judges determine applicability dynamically per answer rather than anchoring it to the dataset's labels?

2: How are Maps and Audio cards evaluated?

3: What exactly do S1, S2, and Top-1 refer to for the model-prior baseline? Does VisExpress exceed it at the query level?

4: How were judges assigned to metrics (Table 13), and was that assignment tuned against the human rankings? I think the judge-to-metric assignment in Table 13 needs more justification.

5: What justifies keeping D·B/100 when it produces large leaderboard reversals? Can you report human preferences on a sample comparing a high-D/lower-B model against a lower-D/high-B model?

6: What language are the queries, pages, judge prompts and annotators in? How many of the 786 queries are real-user, curated, or synthesized? Can models emit arbitrary HTML/CSS, or only the fixed component declarations?

7: How often were declared components silently dropped by the renderer, per model?

8: How were the 60 ranking annotators and label reviewers recruited and compensated, and what is the consent basis for the real-user queries?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.