acceptodds
Withdrawn from ICLR 2027

HTMLPresentBench: How Well Do Agents Present Task Results Through HTML?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

AI agents can carry out complex tasks, but their usefulness also depends on how effectively the results are communicated to users. Long outputs and execution traces often make it difficult for users to extract key findings and assess the evidence supporting them. We study how to evaluate reports generated from completed agent trajectories. These reports should faithfully convey important outcomes and limitations, while making results and supporting evidence easy to understand and find. We introduce an assessment framework for trajectory-grounded reporting that evaluates reports along three axes: content, visual quality, and information access. By grounding evaluation in the underlying trajectory, task evidence, and readers’ information needs rather than a single reference presentation, it accommodates diverse report designs while providing explicit, inspectable criteria. To instantiate and validate this assessment, we introduce HTMLPresentBench, containing 60 completed task trajectories across 13 domains, and evaluate eleven report generators that turn the same fixed trajectories into user-facing HTML reports. Our evaluation reveals tradeoffs across content, visual quality, and information access. HTML improves information access on average over Markdown, but can reduce content or visual quality, while design-oriented skills further improve access in some settings at the cost of other dimensions. Human evaluation further provides evidence that the full assessment pipeline better reflects readers’ judgments than a simple LLM-as-a-Judge baseline. Because trajectory-specific references are constructed once and reused across candidate reports, the framework supports scalable, controlled comparison of models and reporting methods on shared executions, and can be applied to new trajectories and tasks.

Primary area:
generative models
Keywords:
LLM Agents, Agent Evaluation, Benchmarking, HTML Report Generation, Human–AI Communication

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cPsDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HTMLPresentBench to evaluate how agents communicate completed task results through HTML, separately from their ability to execute the original task. The benchmark contains 60 trajectories across 13 domains and compares 11 report generators under four generation methods, including Markdown and three HTML-based settings. The evaluation combines human-reviewed trajectory references, rendered-page evidence, and an LLM-based pipeline to assess Content, Visual Quality, and Information Access. The results suggest that richer presentation can improve access to information while reducing content or visual quality. A human study provides an initial comparison of the evaluator with a direct LLM-as-a-Judge baseline.

Critical strengths
  1. The paper addresses a useful and distinct evaluation problem. A completed agent task is not necessarily communicated accurately or accessibly to its user. Fixing the execution trajectory across report generators is a sensible way to isolate reporting quality from task-execution performance, and separating Content, Visual Quality, and Information Access makes the resulting tradeoffs interpretable.

  2. The evaluation uses concrete, evidence-linked checks rather than a single aesthetic judgment. Human-reviewed references are fixed before candidate evaluation, and the pipeline combines trajectory facts with source-level and desktop/mobile rendering evidence. The qualitative examples illustrate consequential failures, such as omitting qualifications about task outcomes, obscuring evidence, and producing unreadable layouts.

Critical weaknesses

1. The statistical reporting is not sufficiently auditable, and there is a consequential internal inconsistency.

Tables 1 and 3 describe the reported uncertainties as standard errors over scored tasks, but do not provide the actual number of scored tasks for each model, method, and axis or explain exclusions. If the scores are bounded in [0,100] and each estimate uses all 60 tasks, several mean/standard-error combinations are incompatible with the conventional standard error of a sample mean:

  • GPT-6-Astra / Direct HTML / Content: 79.2 ± 6.1, versus a standard-error upper bound of approximately 5.3 for n=60.
  • GLM-5.2 / Direct HTML / Content: 74.0 ± 6.0, versus an upper bound of approximately 5.7.
  • Grok-4.5 / HTML Anything / Content: 61.9 ± 6.6, versus an upper bound of approximately 6.3.
  • Gemini 3.8 Flash / Frontend Design / Visual: 66.5 ± 7.8, versus an upper bound of approximately 6.1.
  • Qwen 3.8 Max / Frontend Design / Visual: 67.3 ± 6.4, versus an upper bound of approximately 6.1.

The bound used here is SE <= sqrt(mu*(100-mu)/(n-1)). These discrepancies remain after allowing for displayed rounding. Smaller scored subsets or a different, mislabeled uncertainty statistic could explain these values, but neither explanation is documented. If subsets differ, the comparability of the reported averages also needs to be established.

Confidence-interval inconsistency (Table 2, p. 9): the table gives the Access improvement's 95% confidence interval as [0.5, 22.5], whereas the text on the same page gives [-0.5, 22.5] and explicitly states that it includes zero. The former excludes zero; the latter includes zero. This changes whether the interval supports a positive improvement and requires correction. The aggregate means, directional counts, and overall-score figure are otherwise largely arithmetically consistent with the reported tables; my concern is the uncertainty estimates and the missing underlying accounting.

2. The central scoring procedure and its validation are insufficiently specified to establish the reliability of the benchmark scores.

Appendix E.3 states that fixed formulas convert criterion-level judgments into numerical scores, but the supplied manuscript does not fully specify the numerical label mappings, subdimension weights, severity penalties, treatment of not-applicable cases, or distance-to-score rules. This prevents independent reconstruction of the main scores from the reported judgments. There is also an unresolved mismatch in Conciseness: Appendix B (p. 14) counts each occurrence aligned with important information and explicitly does not separately penalize repetition, while the C4 prompt (pp. 21-22) requires one-to-one matching and labels subsequent occurrences as repeated rather than matched. The paper needs to specify whether and how repeated units enter the numerator and reconcile the definition with the implementation.

The human validation does not resolve these concerns. Although 18 participants each evaluate 10 sets, the retained annotation counts and number of unique trajectories after exclusions are not reported. Content validation targets understandability rather than faithfulness and coverage, and the human comparison includes only the three HTML methods, not Markdown. It therefore does not directly establish that the full Content score is valid or that HTML improves human performance over Markdown. With only 60 trajectories and 3-5 examples per domain, the evidence currently supports a promising pilot evaluation rather than a sufficiently validated, reproducible benchmark. These issues motivate my weak-reject recommendation despite the useful task formulation.

Questions for authors

1. Sample sizes and standard errors. What is the actual sample size for every model/method/axis in Tables 1 and 3? Please provide the scored task IDs, exclusion/failure rules, and the exact uncertainty calculation. In particular, please explain these five reported mean/standard-error combinations:

  • GPT-6-Astra / Direct HTML / Content: 79.2 ± 6.1.
  • GLM-5.2 / Direct HTML / Content: 74.0 ± 6.0.
  • Grok-4.5 / HTML Anything / Content: 61.9 ± 6.6.
  • Gemini 3.8 Flash / Frontend Design / Visual: 66.5 ± 7.8.
  • Qwen 3.8 Max / Frontend Design / Visual: 67.3 ± 6.4.

As explained above, these are incompatible with the conventional standard error if all 60 tasks are used and scores lie in [0,100]. Are method comparisons computed on the same tasks?

2. Conflicting confidence intervals and human-evaluation counts. Which Access confidence interval in Table 2 is correct: [0.5, 22.5] or [-0.5, 22.5]? Please report the retained annotations, unique trajectories, and comparison counts for each human-evaluation metric, together with the aggregation and bootstrap procedure sufficient to reproduce the estimates. The reported 18 participants × 10 sets does not establish the final sample sizes after exclusions.

3. Exact scoring rules and repetition handling. Please provide the complete criterion-to-score mappings and aggregation formulas, including weights, severity penalties, not-applicable handling, and continuity-distance rules. In particular, how are repeated units scored in C4, and how does this agree with the formula on p. 14? A worked example from judgments to final scores would help.

4. Scope of human validation. Is there existing human validation of factual faithfulness/coverage or a direct human comparison between Markdown and HTML beyond the study described? If not, please clarify the scope of the conclusions supported by the current understandability and report-preference study.

Reviewer_NnqdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an evaluation framework for reports that an agent writes about the 60 completed task trajectories across 13 domains provided by the benchmark. Reports are scored on three separately reported axes: content (faithfulness, coverage, self-contained verifiability, conciseness), visual quality (text readability, visual consistency, object readability, plus a rendering gate), and information access (findability of, and continuity between, the answers to a reader-need tree). For each trajectory an agent drafts an atomic fact list, a coverage checklist and a reader-need tree, a human reviewer edits them, and they are frozen and reused across all reports for that task. Reports are rendered at desktop and mobile viewports, interactive controls are exercised, an HTML semantic map is built, and a single evaluator model (GPT-5.6-Luna) issues per-item judgments with evidence that a program converts to scores; some visual checks are computed deterministically.

Critical strengths
  • The problem is well chosen and precisely delimited: reporting from a completed trajectory, failures included, is a stage that research-report, slide and front-end benchmarks do not cover, and separating it from execution lets reporting quality be studied on fixed inputs.
  • The framework converts a vague judgment into inspectable units: references fixed before scoring and shared across reports, closed label sets with cited evidence, a second adversarial review for fabrication flags, and deterministic measurement where external standards exist; scoring defects against a page’s own rules rather than aesthetics is explicit and defensible.
  • The evaluation is broad (11 × 4 × 60), costs are reported throughout, the qualitative figures name concrete failure modes, and the findings are informative rather than confirmatory.
Critical weaknesses

W1. The benchmark measures reporting of a single trajectory, not synthesis of results from many. The unit is one trajectory → one report; content is anchored to one execution record and the need tree asks about that run. Aggregating across runs, finding trends, deciding what to show are out of scope, although the introduction’s motivation (extracting key findings from long outputs) and the Fig. 1 example, an analysis over market data, suggest that broader setting. The scope should be stated explicitly and the framing narrowed accordingly.

W2. The human-reviewed references are the framework's substitute for ground truth, and their construction is under-specified. Also, as the drafts come from the same model that later judges, the references are anchored to that model's view of what matters; the single sentence in Appendix D.4 noting that prose fluency was dropped for lacking stable judgments across reviewers indicates that agreement was examined, yet no agreement figures are reported for the retained criteria.

W3. The axes are normative rather than objective, and the reported scores cannot be reconstructed. The paper is clear that visual quality measures observable defects rather than appeal and that information access measures support for finding rather than reading behaviour; this is a reasonable design, but it means that a plain Markdown page scores near the ceiling by construction (97.7), so "HTML lowers visual quality" should be read as "models writing their own CSS introduce detectable defects". The label-to-score formulas, severity weights and partial-label treatment are not given, and no sub-dimension scores are reported.

Questions for authors

Q1. How much do the three axis scores vary when the same generator reports the same trajectory two or three times, and when the same report is scored repeatedly by the evaluator? How do these variances compare with the between-method differences in Table 1?

Q2. What formulas map the per-item judgments to the 0–100 axis scores, how are severities and partial labels weighted, and what are the sub-dimension scores (C1–C4, D1–D3, findability and continuity) for the generators in Table 1?

Reviewer_U57eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HTMLPresentBench evaluates how AI agents communicate completed task outcomes through HTML reports. It introduces a trajectory grounded assessment framework covering content fidelity, visual quality, and information access, alongside 60 trajectories across 13 domains. Experiments reveal trade-offs between accessible presentation, factual completeness, and readability, supported by targeted human comparisons.

Critical strengths
  1. The paper addresses an underexplored aspect of agent performance: how effectively agents communicate their execution results to human users.
  2. The work provides a relative comprehensive evaluation dimensions.
  3. The experiments reveal meaningful trade-offs between information accessibility, factual completeness, and visual quality. The qualitative examples further illustrate these failure modes.
Critical weaknesses
  1. The scale of benchmark is limited: The benchmark contains only 60 trajectories across 13 domains, leaving relatively few examples per domain. This limits the generalizability of the observations.
  2. Markdown benefits from a standardized renderer, whereas HTML quality depends on model-generated layouts and styling. This makes it difficult to separate the advantages of the presentation format from differences in implementation quality.
  3. Table 2 reports a positive confidence interval for the Information Access improvement, while the corresponding discussion reports an interval containing zero. This inconsistency should be clarified.
Questions for authors

See weakness above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.