- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HTMLPresentBench to evaluate how agents communicate completed task results through HTML, separately from their ability to execute the original task. The benchmark contains 60 trajectories across 13 domains and compares 11 report generators under four generation methods, including Markdown and three HTML-based settings. The evaluation combines human-reviewed trajectory references, rendered-page evidence, and an LLM-based pipeline to assess Content, Visual Quality, and Information Access. The results suggest that richer presentation can improve access to information while reducing content or visual quality. A human study provides an initial comparison of the evaluator with a direct LLM-as-a-Judge baseline.
Critical strengths
-
The paper addresses a useful and distinct evaluation problem. A completed agent task is not necessarily communicated accurately or accessibly to its user. Fixing the execution trajectory across report generators is a sensible way to isolate reporting quality from task-execution performance, and separating Content, Visual Quality, and Information Access makes the resulting tradeoffs interpretable.
-
The evaluation uses concrete, evidence-linked checks rather than a single aesthetic judgment. Human-reviewed references are fixed before candidate evaluation, and the pipeline combines trajectory facts with source-level and desktop/mobile rendering evidence. The qualitative examples illustrate consequential failures, such as omitting qualifications about task outcomes, obscuring evidence, and producing unreadable layouts.
Critical weaknesses
1. The statistical reporting is not sufficiently auditable, and there is a consequential internal inconsistency.
Tables 1 and 3 describe the reported uncertainties as standard errors over scored tasks, but do not provide the actual number of scored tasks for each model, method, and axis or explain exclusions. If the scores are bounded in [0,100] and each estimate uses all 60 tasks, several mean/standard-error combinations are incompatible with the conventional standard error of a sample mean:
- GPT-6-Astra / Direct HTML / Content: 79.2 ± 6.1, versus a standard-error upper bound of approximately 5.3 for n=60.
- GLM-5.2 / Direct HTML / Content: 74.0 ± 6.0, versus an upper bound of approximately 5.7.
- Grok-4.5 / HTML Anything / Content: 61.9 ± 6.6, versus an upper bound of approximately 6.3.
- Gemini 3.8 Flash / Frontend Design / Visual: 66.5 ± 7.8, versus an upper bound of approximately 6.1.
- Qwen 3.8 Max / Frontend Design / Visual: 67.3 ± 6.4, versus an upper bound of approximately 6.1.
The bound used here is SE <= sqrt(mu*(100-mu)/(n-1)). These discrepancies remain after allowing for displayed rounding. Smaller scored subsets or a different, mislabeled uncertainty statistic could explain these values, but neither explanation is documented. If subsets differ, the comparability of the reported averages also needs to be established.
Confidence-interval inconsistency (Table 2, p. 9): the table gives the Access improvement's 95% confidence interval as [0.5, 22.5], whereas the text on the same page gives [-0.5, 22.5] and explicitly states that it includes zero. The former excludes zero; the latter includes zero. This changes whether the interval supports a positive improvement and requires correction. The aggregate means, directional counts, and overall-score figure are otherwise largely arithmetically consistent with the reported tables; my concern is the uncertainty estimates and the missing underlying accounting.
2. The central scoring procedure and its validation are insufficiently specified to establish the reliability of the benchmark scores.
Appendix E.3 states that fixed formulas convert criterion-level judgments into numerical scores, but the supplied manuscript does not fully specify the numerical label mappings, subdimension weights, severity penalties, treatment of not-applicable cases, or distance-to-score rules. This prevents independent reconstruction of the main scores from the reported judgments. There is also an unresolved mismatch in Conciseness: Appendix B (p. 14) counts each occurrence aligned with important information and explicitly does not separately penalize repetition, while the C4 prompt (pp. 21-22) requires one-to-one matching and labels subsequent occurrences as repeated rather than matched. The paper needs to specify whether and how repeated units enter the numerator and reconcile the definition with the implementation.
The human validation does not resolve these concerns. Although 18 participants each evaluate 10 sets, the retained annotation counts and number of unique trajectories after exclusions are not reported. Content validation targets understandability rather than faithfulness and coverage, and the human comparison includes only the three HTML methods, not Markdown. It therefore does not directly establish that the full Content score is valid or that HTML improves human performance over Markdown. With only 60 trajectories and 3-5 examples per domain, the evidence currently supports a promising pilot evaluation rather than a sufficiently validated, reproducible benchmark. These issues motivate my weak-reject recommendation despite the useful task formulation.
Questions for authors
1. Sample sizes and standard errors. What is the actual sample size for every model/method/axis in Tables 1 and 3? Please provide the scored task IDs, exclusion/failure rules, and the exact uncertainty calculation. In particular, please explain these five reported mean/standard-error combinations:
- GPT-6-Astra / Direct HTML / Content: 79.2 ± 6.1.
- GLM-5.2 / Direct HTML / Content: 74.0 ± 6.0.
- Grok-4.5 / HTML Anything / Content: 61.9 ± 6.6.
- Gemini 3.8 Flash / Frontend Design / Visual: 66.5 ± 7.8.
- Qwen 3.8 Max / Frontend Design / Visual: 67.3 ± 6.4.
As explained above, these are incompatible with the conventional standard error if all 60 tasks are used and scores lie in [0,100]. Are method comparisons computed on the same tasks?
2. Conflicting confidence intervals and human-evaluation counts. Which Access confidence interval in Table 2 is correct: [0.5, 22.5] or [-0.5, 22.5]? Please report the retained annotations, unique trajectories, and comparison counts for each human-evaluation metric, together with the aggregation and bootstrap procedure sufficient to reproduce the estimates. The reported 18 participants × 10 sets does not establish the final sample sizes after exclusions.
3. Exact scoring rules and repetition handling. Please provide the complete criterion-to-score mappings and aggregation formulas, including weights, severity penalties, not-applicable handling, and continuity-distance rules. In particular, how are repeated units scored in C4, and how does this agree with the formula on p. 14? A worked example from judgments to final scores would help.
4. Scope of human validation. Is there existing human validation of factual faithfulness/coverage or a direct human comparison between Markdown and HTML beyond the study described? If not, please clarify the scope of the conclusions supported by the current understandability and report-preference study.