- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AmbiHTMLBench evaluates document-to-HTML reconstruction using 1,115 annotated pages, objective content and layout metrics, and a human-calibrated VLM preference proxy. The paper benchmarks ten multimodal systems and reports improvements from distillation-based fine-tuning of Qwen models using Gemini-generated HTML.
Critical strengths
- The task is practically relevant: faithful HTML reconstruction can support document editing, information localization, and interaction.
- The dataset curation and annotation effort are valuable, and separating objective fidelity from perceived visual quality provides complementary diagnostic information. The human study and internal validation further support the evaluation effort.
Critical weaknesses
- The preference proxy has an unresolved encoding bias. Its coefficients sum to approximately 0.8678, so unanimous ties yield preference scores of approximately 0.4339 for A and 0.5661 for B. Although acknowledged in Appendix B.3, this asymmetry leaves the robustness of the subjective ranking uncertain. Results from a symmetric estimator and canonical A/B assignment statistics are needed.
- Metric validity requires more direct evidence. Controlled perturbation experiments would help establish whether the metrics respond appropriately to missing text, reordered columns, collapsed layout blocks, incorrect table spans, and damaged formula structure. Robustness to visually equivalent HTML implementations should also be examined.
Questions for authors
- How do the subjective rankings change after correcting the encoding asymmetry?
- Can the authors provide controlled-perturbation results and finer-grained dataset statistics by document type, language, and text density?
2 comments
| Experiment | Strength | N | Text ED | TEDS | TEDS-S | Layout | Total |
|---|---|---|---|---|---|---|---|
| missing_text | 0 | 100 | 0.1349 | 0.6931 | 0.7180 | 0.8432 | 0.8332 |
| missing_text | 0.1 | 100 | 0.2106 | 0.6931 | 0.7180 | 0.8387 | 0.7983 |
| missing_text | 0.3 | 100 | 0.3590 | 0.6931 | 0.7180 | 0.8303 | 0.7301 |
| reordered_columns | 0 | 100 | 0.1564 | 0.6758 | 0.6984 | 0.8572 | 0.8276 |
| reordered_columns | 0.1 | 100 | 0.1571 | 0.6758 | 0.6984 | 0.8550 | 0.8264 |
| reordered_columns | 0.3 | 100 | 0.1600 | 0.6758 | 0.6984 | 0.8445 | 0.8206 |
| reordered_columns | 1.0 | 100 | 0.1570 | 0.6758 | 0.6984 | 0.8073 | 0.8056 |
| collapsed_blocks | 0 | 100 | 0.1501 | 0.7045 | 0.7304 | 0.8445 | 0.8286 |
| collapsed_blocks | 0.1 | 100 | 0.1505 | 0.7045 | 0.7304 | 0.8365 | 0.8250 |
| collapsed_blocks | 0.3 | 100 | 0.1525 | 0.7045 | 0.7304 | 0.8134 | 0.8140 |
| table_spans | 0 | 100 | 0.2211 | 0.6958 | 0.7128 | 0.8452 | 0.7969 |
| table_spans | 0.1 | 100 | 0.2211 | 0.6958 | 0.7128 | 0.8417 | 0.7953 |
| table_spans | 0.3 | 100 | 0.2211 | 0.6958 | 0.7128 | 0.8415 | 0.7953 |
| formula_structure | 0 | 30 | 0.2355 | 0.6838 | 0.7778 | 0.8339 | 0.7841 |
| formula_structure | 0.1 | 30 | 0.2438 | 0.6705 | 0.7778 | 0.8337 | 0.7787 |
| formula_structure | 0.3 | 30 | 0.2555 | 0.6705 | 0.7778 | 0.8326 | 0.7732 |
| equivalent_html | 0 | 100 | 0.1349 | 0.6931 | 0.7180 | 0.8432 | 0.8332 |
| equivalent_html | 1.0 | 100 | 0.1349 | 0.6931 | 0.7180 | 0.8432 | 0.8332 |
Thank you much for your comment, your comment is very practical and very helpful for our research.
[W1, Q1] Encoding Bias
Response:
Thank you for your comment. For asymmetry, our subjective evaluation is designed to reflect human preferences, and in human study, Among 159 valid repeated pairs from 27 participants, 40 received different overall judgments, which is asymmetric. Therefore, we fitted the five judge weights to human preferences by least squares without constraining their sum.
To test asymmetry robustness, flipping experiments are done, the results are shown below. The ranking of the models do indeed fluctuate locally, but most of the judgments remained unchanged, and the overall ranking is relatively stable.
We appreciate your concern about encoding asymmetry and examined two normalized alternatives: proportional rescaling and temperature-1 softmax.
On 1,872 comparisons covered by the 27 participants, we compared ten-model rankings. Both normalized alternatives have higher ranking Spearman point estimates and higher log loss. We then applied the three formulas to the same 50,000 newly collected judge decisions on 10,000 reconstructed comparisons and fitted Bradley–Terry rankings:
At a 0.5 preference threshold, the favored side changes for 624/10,000 comparisons with proportional weights (6.24%; page-bootstrap 95% CI: 5.77%–6.73%) and 794/10,000 with softmax (7.94%; 95% CI: 7.39%–8.50%).
[W2, Q2] Metric Validity
Response:
In our objective evaluation protocol, after accurate matching, the metrics are quite explicable, since they are calculated by the difference of strings. Moreover, as we use a headless chromium to confirm the elements' content and position, so that visually equivalent HTML will have the same score.
For subjective protocol, the judge models are fed with images rendered from HTML inferences, so visually equivalent HTML implementations are sure to have the same score. For other metrics, it is hard to decide how they affect the judging confidence, so we simply aligned the result with human experts.
Controlled-perturbation Results are given below, and the metrics are working just as we expected.