acceptodds
Under review as a conference paper at ICLR 2027

AmbiHTMLBench: A Multidimensional Benchmark for Image-to-HTML Reconstruction of Heterogeneous Documents

Abstract

Document parsing systems often produce Markdown, which captures linear text but offers limited support in many scenarios (e.g., for fine-grained spatial structure, table geometry, visual hierarchy, and rendering semantics). HTML provides a richer representation for heterogeneous, layout-sensitive documents while supporting editing, inspection, and reuse. However, document-to-HTML reconstruction lacks a dedicated benchmark for systematic evaluation. Thus, we introduce AmbiHTMLBench to evaluate models' ability to reconstruct real-world documents as renderable HTML that preserves their content, structure, and visual appearance. AmbiHTMLBench comprises 1,115 diverse pages with structured annotations and two complementary evaluation protocols: (1) The objective protocol aligns rendered blocks between reference and predicted outputs to measure table fidelity, page-level text error, and two-dimensional layout consistency; and (2) the reference-anchored subjective protocol evaluates perceived fidelity using LLM judges calibrated against human preferences. Using AmbiHTMLBench to evaluate ten multimodal systems, we find different systems lead under the two protocols (with the highest objective score as 0.8092). We further demonstrate that task-specific fine-tuning enables a compact expert model to perform effective document-to-HTML reconstruction with 4.41 weighted points on Qwen-9B base.

Primary area:
datasets and benchmarks
Keywords:
Document-to-HTML Reconstruction, Multimodal Document Understanding, Document Parsing, Layout Preservation, Editable Document Representation, Document Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9CVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AmbiHTMLBench evaluates document-to-HTML reconstruction using 1,115 annotated pages, objective content and layout metrics, and a human-calibrated VLM preference proxy. The paper benchmarks ten multimodal systems and reports improvements from distillation-based fine-tuning of Qwen models using Gemini-generated HTML.

Critical strengths
  • The task is practically relevant: faithful HTML reconstruction can support document editing, information localization, and interaction.
  • The dataset curation and annotation effort are valuable, and separating objective fidelity from perceived visual quality provides complementary diagnostic information. The human study and internal validation further support the evaluation effort.
Critical weaknesses
  1. The preference proxy has an unresolved encoding bias. Its coefficients sum to approximately 0.8678, so unanimous ties yield preference scores of approximately 0.4339 for A and 0.5661 for B. Although acknowledged in Appendix B.3, this asymmetry leaves the robustness of the subjective ranking uncertain. Results from a symmetric estimator and canonical A/B assignment statistics are needed.
  2. Metric validity requires more direct evidence. Controlled perturbation experiments would help establish whether the metrics respond appropriately to missing text, reordered columns, collapsed layout blocks, incorrect table spans, and damaged formula structure. Robustness to visually equivalent HTML implementations should also be examined.
Questions for authors
  • How do the subjective rankings change after correcting the encoding asymmetry?
  • Can the authors provide controlled-perturbation results and finer-grained dataset statistics by document type, language, and text density?
2 comments

Thank you much for your comment, your comment is very practical and very helpful for our research.

[W1, Q1] Encoding Bias

Response:

Thank you for your comment. For asymmetry, our subjective evaluation is designed to reflect human preferences, and in human study, Among 159 valid repeated pairs from 27 participants, 40 received different overall judgments, which is asymmetric. Therefore, we fitted the five judge weights to human preferences by least squares without constraining their sum.

To test asymmetry robustness, flipping experiments are done, the results are shown below. The ranking of the models do indeed fluctuate locally, but most of the judgments remained unchanged, and the overall ranking is relatively stable.

No flipRandom 50% flipFull flip
Flipped matches05,000 / 10,00010,000 / 10,000
Rank Spearman vs. original1.00000.93940.9030
Pairwise preference direction changes—616 / 10,000 (6.16%)1,261 / 10,000 (12.61%)
Rank 1gpt-5.6-solgpt-5.6-solgpt-5.6-sol
Rank 2gpt-5.6-terragemini-3.1-pro-previewgemini-3.1-pro-preview
Rank 3gemini-3.1-pro-previewgpt-5.6-terraKimi-K3
Rank 4qwen3.7-plusKimi-K3gpt-5.6-terra
Rank 5Kimi-K3claude-opus-5claude-opus-5
Rank 6claude-opus-5qwen3.7-plusqwen3.7-plus
Rank 7claude-sonnet-5claude-sonnet-5claude-sonnet-5
Rank 8GLM-5V-TurboGLM-5V-TurboGLM-5V-Turbo
Rank 9grok-4.5ERNIE-5.0ERNIE-5.0
Rank 10ERNIE-5.0grok-4.5grok-4.5

We appreciate your concern about encoding asymmetry and examined two normalized alternatives: proportional rescaling and temperature-1 softmax.

On 1,872 comparisons covered by the 27 participants, we compared ten-model rankings. Both normalized alternatives have higher ranking Spearman point estimates and higher log loss. We then applied the three formulas to the same 50,000 newly collected judge decisions on 10,000 reconstructed comparisons and fitted Bradley–Terry rankings:

ModelOriginalProportional [95% rank CI]Softmax [95% rank CI]
GPT-5.6-Sol11 [1, 1]1 [1, 1]
GPT-5.6-Terra23 [2, 4]2 [2, 3]
Gemini 3.1 Pro Preview32 [2, 3]3 [2, 3]
Qwen3.7-Plus46 [5, 6]6 [5, 6]
Kimi-K354 [3, 4]4 [4, 4]
Claude Opus 565 [5, 6]5 [5, 6]
Claude Sonnet 577 [7, 7]7 [7, 7]
GLM-5V-Turbo88 [8, 8]8 [8, 8]
Grok-4.5910 [10, 10]10 [10, 10]
ERNIE-5.0109 [9, 9]9 [9, 9]

At a 0.5 preference threshold, the favored side changes for 624/10,000 comparisons with proportional weights (6.24%; page-bootstrap 95% CI: 5.77%–6.73%) and 794/10,000 with softmax (7.94%; 95% CI: 7.39%–8.50%).

[W2, Q2] Metric Validity

Response:

In our objective evaluation protocol, after accurate matching, the metrics are quite explicable, since they are calculated by the difference of strings. Moreover, as we use a headless chromium to confirm the elements' content and position, so that visually equivalent HTML will have the same score.

For subjective protocol, the judge models are fed with images rendered from HTML inferences, so visually equivalent HTML implementations are sure to have the same score. For other metrics, it is hard to decide how they affect the judging confidence, so we simply aligned the result with human experts.

Controlled-perturbation Results are given below, and the metrics are working just as we expected.

ExperimentStrengthNText EDTEDSTEDS-SLayoutTotal
missing_text01000.13490.69310.71800.84320.8332
missing_text0.11000.21060.69310.71800.83870.7983
missing_text0.31000.35900.69310.71800.83030.7301
reordered_columns01000.15640.67580.69840.85720.8276
reordered_columns0.11000.15710.67580.69840.85500.8264
reordered_columns0.31000.16000.67580.69840.84450.8206
reordered_columns1.01000.15700.67580.69840.80730.8056
collapsed_blocks01000.15010.70450.73040.84450.8286
collapsed_blocks0.11000.15050.70450.73040.83650.8250
collapsed_blocks0.31000.15250.70450.73040.81340.8140
table_spans01000.22110.69580.71280.84520.7969
table_spans0.11000.22110.69580.71280.84170.7953
table_spans0.31000.22110.69580.71280.84150.7953
formula_structure0300.23550.68380.77780.83390.7841
formula_structure0.1300.24380.67050.77780.83370.7787
formula_structure0.3300.25550.67050.77780.83260.7732
equivalent_html01000.13490.69310.71800.84320.8332
equivalent_html1.01000.13490.69310.71800.84320.8332
Reviewer_SVyYOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AmbiHTMLBench, a benchmark for evaluating whether multimodal models can faithfully reconstruct real-world documents into renderable and editable HTML while preserving content and visual layout. It combines objective metrics with a human-calibrated subjective evaluation and shows that the models performing best on automatic metrics are not necessarily those preferred by human perception.

Critical strengths
  1. Comprehensive evaluation. AmbiHTMLBench evaluates reconstruction across text, table structure, 2D layout, and perceived visual fidelity.
  2. Diverse real-world documents. The benchmark includes 1115 pages spanning institutional PDFs, web-native documents, designed posters, etc. The annotation is from experts.
  3. Useful diagnostic insights. The objective and subjective leaderboards produce noticeably different rankings, showing that automatic reconstruction metrics and perceived fidelity capture different failure modes.
Critical weaknesses
  1. The novelty positioning is not sufficiently established. AmbiHTMLBench does not discuss several closely related works, including Image2Struct (Josselin et al.), Web2Code (Yun et al.), and BigDocs-Bench (Rodriguez et al., ICLR 2025). These works similarly reconstruct documents/images into HTML or other structured representations and evaluate fidelity after rendering. Thus, the current novelty claim appears somewhat overstated; the main contribution seems to be extending existing paradigms to heterogeneous real-world documents with more fine-grained, document-specific multidimensional evaluation.

  2. Despite being framed as document reconstruction, the benchmark operates at the individual-page level and does not evaluate cross-page document structure or relationships.

  3. The dataset construction may introduce parser-induced selection bias. Samples with unusable MinerU parsing results are removed before benchmark construction, and MinerU-derived layouts are further used for diversity filtering. This may systematically exclude challenging or unconventional layouts that existing parsers already struggle with.

Questions for authors
  1. How does AmbiHTMLBench substantially differ from closely related benchmarks such as Image2Struct, Web2Code, and BigDocs-Bench beyond extending them to heterogeneous documents?
  2. How do the authors justify that the proposed objective metrics faithfully measure document reconstruction quality, given that layout evaluation depends on text-based matching and that the semantic correctness of figures/charts is largely not evaluated?
Reviewer_HXk6OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces AmbiHTMLBench, a benchmark composed of document pages for evaluating how faithfully models can reconstruct real-world documents as renderable HTML. It pairs an objective block-alignment protocol with a subjective protocol based on human-calibrated VLM judges, and evaluates ten multimodal systems.

Critical strengths

I am not an expert in this area. Overall, the pipeline seems reasonable to me. Compared with other output formats, HTML does offer advantages in both popularity and expressiveness, which makes document-to-HTML reconstruction a worthwhile task to benchmark.

Critical weaknesses

I lack the domain expertise needed for a more specialized evaluation. I kindly ask the AC to give more weight to the opinions of reviewers with more expertise in this area.

The subjective scoring formula is asymmetric, which seems odd to me. Authors acknowledge this and fix a "canonical" A/B order during scoring, but they do not specify how that order is assigned. If some models systematically occupy one position more often, the leaderboard could be biased. The authors should specify a clearer evaluation protocol that removes the effect of this asymmetry.

Questions for authors

None

1 comment

Thank you for your comment. For asymmetry, our subjective evaluation is designed to reflect human preferences, and in human study, Among 159 valid repeated pairs from 27 participants, 40 received different overall judgments, which is asymmetric. Therefore, we fitted the five judge weights to human preferences by least squares without constraining their sum.

To test asymmetry robustness, flipping experiments are done, the results are shown below. The ranking of the models do indeed fluctuate locally, but most of the judgments remained unchanged, and the overall ranking is relatively stable.

No flipRandom 50% flipFull flip
Flipped matches05,000 / 10,00010,000 / 10,000
Rank Spearman vs. original1.00000.93940.9030
Pairwise preference direction changes—616 / 10,000 (6.16%)1,261 / 10,000 (12.61%)
Rank 1gpt-5.6-solgpt-5.6-solgpt-5.6-sol
Rank 2gpt-5.6-terragemini-3.1-pro-previewgemini-3.1-pro-preview
Rank 3gemini-3.1-pro-previewgpt-5.6-terraKimi-K3
Rank 4qwen3.7-plusKimi-K3gpt-5.6-terra
Rank 5Kimi-K3claude-opus-5claude-opus-5
Rank 6claude-opus-5qwen3.7-plusqwen3.7-plus
Rank 7claude-sonnet-5claude-sonnet-5claude-sonnet-5
Rank 8GLM-5V-TurboGLM-5V-TurboGLM-5V-Turbo
Rank 9grok-4.5ERNIE-5.0ERNIE-5.0
Rank 10ERNIE-5.0grok-4.5grok-4.5

We appreciate your concern about encoding asymmetry and examined two normalized alternatives: proportional rescaling and temperature-1 softmax.

wjprop=βj∑kβk,wjsoft=eβj∑keβk,p(A)=∑jwjxj,w_j^{\mathrm{prop}}=\frac{\beta_j}{\sum_k\beta_k},\quad w_j^{\mathrm{soft}}=\frac{e^{\beta_j}}{\sum_k e^{\beta_k}},\quad p(A)=\sum_j w_jx_j,

On 1,872 comparisons covered by the 27 participants, we compared ten-model rankings against the human reference and measured log loss against human soft labels:

WeightsRanking Spearman [95% CI]Mean log loss [95% CI]
Original0.9030 [0.8545, 0.9515]0.9514 [0.8776, 1.0268]
Proportional0.9758 [0.8909, 0.9879]1.2686 [1.1682, 1.3650]
Softmax0.9758 [0.9030, 0.9879]1.2434 [1.1435, 1.3405]

Both normalized alternatives have higher ranking Spearman point estimates and higher log loss. We then applied the three formulas to the same 50,000 newly collected judge decisions on 10,000 reconstructed comparisons and fitted Bradley–Terry rankings:

ModelOriginalProportional [95% rank CI]Softmax [95% rank CI]
GPT-5.6-Sol11 [1, 1]1 [1, 1]
GPT-5.6-Terra23 [2, 4]2 [2, 3]
Gemini 3.1 Pro Preview32 [2, 3]3 [2, 3]
Qwen3.7-Plus46 [5, 6]6 [5, 6]
Kimi-K354 [3, 4]4 [4, 4]
Claude Opus 565 [5, 6]5 [5, 6]
Claude Sonnet 577 [7, 7]7 [7, 7]
GLM-5V-Turbo88 [8, 8]8 [8, 8]
Grok-4.5910 [10, 10]10 [10, 10]
ERNIE-5.0109 [9, 9]9 [9, 9]

At a 0.5 preference threshold, the favored side changes for 624/10,000 comparisons with proportional weights (6.24%; page-bootstrap 95% CI: 5.77%–6.73%) and 794/10,000 with softmax (7.94%; 95% CI: 7.39%–8.50%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.