acceptodds
Under review as a conference paper at ICLR 2027

ImageParser: Hierarchical, Grounded Image Parsing with Small Vision–Language Models

Abstract

We investigate image perception as reconstruction in language: preserving the descriptive information needed to reconstruct an image up to semantic equivalence. ImageParser, a family of 0.8B, 2B, and 4B vision–language models, converts images into hierarchical, spatially grounded JSON parses. Each parse links global content and appearance to localized entities and parts, preserving visible text and explicit parent relationships without predefined categories or downstream questions. We train ImageParser on 6.1 million image–parse pairs produced by frontier perception models and iterative multimodal verification, averaging over 3,000 annotation tokens per image. On ImageParserBench, a human-reviewed benchmark of 1,975 images, ImageParser-4B leads all evaluated general-purpose VLMs, including Gemini 3.8 Flash and GPT-5.6-sol, across six parsing metrics. It improves caption and hierarchy F1 over their strongest external baselines by 20.0 and 11.8 percentage points, respectively. Even the 0.8B model surpasses the evaluated proprietary models in semantics, type, and hierarchy. Task-specific readouts reuse these parses for detection, OCR, and captioning; reconstruction and pixel-free QA assess information preservation and remaining limitations. These results show that compact VLMs learn reusable hierarchical image representations.

Primary area:
generative models
Keywords:
Image Caption, Scene Graph, Visual Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NGNVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ImageParser, a family of small vision-language models (0.8-4B) that generate hierarchical, spatially grounded JSON representations of images, including region descriptions, bounding boxes, OCR, relationships, color palettes, etc. The work develops ParseEngine, an automated annotation pipeline combining frontier LVLMs and specialist models to generate 6.1M image-parse pairs for supervised finetuning. It also proposes ImageParserBench, a human-reviewed benchmark of 1975 images for evaluating hierarchical image parsing. Experiments show improvements over general-purpose VLMs on ImageParserBench, with additional evaluations on object detection, OCR, captioning, image reconstruction, and pixel-free QA.

Critical strengths
  • Comprehensive annotation pipeline and useful benchmark. ParseEngine integrates VLMs, specialist models, and iterative refinement to generate detailed image annotations at scale. The human-reviewed ImageParserBench benchmark with over 63K annotated regions across diverse visual domains is also beneficial for the research community. While their individual components are not particularly novel, ParseEngine and ImageParserBench are useful contributions for hierarchical image parsing.
  • Interesting experiments on information preservation. The JSON-to-image reconstruction and pixel-free QA experiments demonstrate that the generated representations preserve substantial visual information, allowing downstream models to reconstruct images or answer questions w/o access to the original pixels. The use of 2 different image generators strengthens reconstruction results, while pixel-free QA experiments provide useful insights into both the capabilities and limitations of the representation.
Critical weaknesses
  • Limited modeling novelty beyond distillation and insufficient training data validation. ImageParser is trained by standard autoregressive SFT on ParseEngine annotations. While distilling a complex annotation pipeline into compact models is a practical contribution, the model family itself introduces little methodological novelty. The paper does not compare ImageParser results against ParseEngine results directly, making it unclear what is gained or lost through distillation beyond simpler inference, albeit at the cost of less flexibility to choose component models. Additionally, the 6.1M training annotations are generated automatically w/o quantitative evaluation of their quality or ablations w.r.t. contributions of individual refinement stages. This raise further questions w.r.t. the necessity of the model beyond distillation, compared to just using the data pipeline, since there is no difference in data quality due to lack of human validation.
  • Limited evidence of generalization to external benchmarks. The results outside of ImageParserBench are not as convincing. On LVIS detection, ImageParser performs reasonably well but does not surpass the strongest baselines, especially reasoning-based ones. On OCRBench v2, the strongest results may largely reflect only the specialist OCR supervision used by ParseEngine (since there is no particular OCR-based model contribution); plus, comparisons with competitive models such as Qwen3-VL (available in Table 3(a) and (c)) are missing. On CAPEval, ImageParser achieves highest precision but lower coverage than several baselines with a different evaluation protocol from reported methods. These results provide limited evidence that the proposed representation is consistently better across downstream tasks.
  • Unclear implications of autoregressive prediction for set-based image parsing. Image regions are naturally represented as a set with hierarchical relationships, but ImageParser predicts them sequentially in a fixed parent-before-child order. Although the paper shows experiments for global-first and layout-first decoding, these experiments only change the ordering of top-level fields, not the ordering of individual regions. It remains unclear whether the model is sensitive to arbitrary region ordering or whether errors accumulate over long sequences, particularly for images with many regions.
Questions for authors
  • How does ImageParser compare directly against ParseEngine on held-out images in terms of parsing quality and inference cost? What are the main benefits of distillation beyond reducing the number of models required at inference?
  • Could the authors provide a quantitative validation of the automatically generated training annotations and assess the contribution of ParseEngine's refinement stages?
  • Could the authors evaluate whether ImageParser's improvements hold on independently annotated benchmarks, and provide more comprehensive comparisons with recent VLMs on external tasks?
  • How sensitive is ImageParser to the ordering of regions in the output JSON? Have the authors considered randomizing sibling-region order during training or evaluating robustness to different region serializations?
3 comments

Thank you for the detailed rebuttal and additional experiments. I appreciate the direct comparison with ParseEngine, the additional external baselines, and the region-order ablation, which address several of my original concerns.

However, my main concern remains the limited novelty and significance of the work. While the data pipeline and benchmark are useful contributions, I am still not fully convinced that the paper offers sufficiently novel findings beyond distillation and structured supervision. Additionally, while the annotation-stage analysis is useful, it mainly shows changes in annotation quantity and detail rather than directly validating the correctness of the generated training annotations.

I will therefore maintain my original score. Thank you again for your time and effort in addressing my concerns.

AuthorsOpenReview ↗

Further Clarifications on Annotation Quality and Novelty

We sincerely thank the reviewer for the thoughtful follow-up and for acknowledging our additional experiments.

On annotation quality validation. We appreciate the opportunity to further clarify our data validation process. Beyond the annotation-stage statistics in W2, we would like to highlight the direct, reference-free quality evaluation already presented in General Response [2/2]. As shown in Table 2, an independent audit using GPT-6 Astra (which was completely decoupled from our annotation pipeline) confirms that ParseEngine generates highly accurate annotations: 11.3 correct facts per region, only 0.79 major errors per 100 claims, and 95.3% exact OCR. Furthermore, the downstream performance of ImageParser-4B also provides strong empirical validation: trained solely on these annotations, our 4B model surpasses frontier proprietary models—including GPT-5.6-Sol and Gemini-3.8-Flash—in zero-shot parsing (Table 1 in General Response [1/2]). Such gains would not be achievable if the underlying supervision were noisy or incomplete. Taken together, we hope these direct and downstream results alleviate your concerns regarding annotation quality.

On novelty. We understand the reviewer's perspective and kindly invite you to view our contribution as an integrated whole, as detailed in General Response [1/2]. Rather than standard distillation, our work brings together four tightly coupled components: (1) a novel structured representation, (2) a multi-stage annotation pipeline, (3) a new benchmark, and (4) empirical evidence that a single compact model can match complex multi-model pipelines. Collectively, these contributions advance the field beyond conventional distillation paradigms.

Thank you again for your time and constructive engagement. In light of these additional results and clarifications, we sincerely hope you will reconsider your assessment and consider raising your score.

We respectfully ask the reviewer to reconsider what constitutes novel findings in this work. Beyond the pipeline and benchmark, we highlight two findings that we believe are non-trivial and significant:

Finding 1: Compact models are sufficient for highly complex structured parsing. Prior to this work, it was unclear whether a single small model could jointly produce hierarchical region decomposition, bounding boxes, descriptions, OCR, color palettes, and parent–child relations in one pass, each of which traditionally required a dedicated specialist model. We demonstrate that even a 0.8B model can achieve this, and a 4B model matches a multi-call pipeline of frontier proprietary models. This is a surprising and practically important result, providing a solid foundation for next-generation image generation pipelines that require billion-scale high-quality annotation demanding.

Finding 2: Hierarchical parsing approaches semantic equivalence with the source image. We show that sufficiently detailed hierarchical parses can serve as a text-only proxy for the original image, enabling both faithful image reconstruction from parse alone and pixel-free visual QA (a new evaluation paradigm we propose). This finding has broad implications: if a structured text representation can substitute for raw pixels in downstream reasoning, it opens new directions for embodied AI, efficient visual reasoning, and language-centric vision systems. To our knowledge, no prior work has demonstrated this level of semantic equivalence from a single-pass structured parse.

We believe these findings go substantially beyond "distillation and structured supervision" and sincerely hope the reviewer can reconsider their assessment accordingly.

Reviewer_yzMmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ImageParser, a family of compact vision-language models that convert an image into a query-independent hierarchical JSON representation. The representation includes global descriptions, region-level bounding boxes and captions, OCR text, color palettes, and explicit parent-child relationships. The authors build ParseEngine to produce 6.1M image-JSON training pairs using frontier VLMs, OCR models, and deterministic processing, and train 0.8B, 2B, and 4B models with supervised fine-tuning. They also introduce ImageParserBench, a human-reviewed benchmark of 1,975 images with over 63K regions.

Critical strengths

Thank you for the detailed response. My main concerns have been largely addressed.

The reference-free evaluation with GPT-6-Astra adds useful evidence beyond agreement with ParseEngine references. The release plan (models, benchmark identifiers, evaluation code and prompts) is concrete. I also appreciate the clarification on Figures 2 and 5. Please state the source of each panel in the captions and update the AI Use Statement accordingly.

I find this direction worth pursuing. The quality of current text-to-image generation depends heavily on how fine-grained the text condition is. A compact parser that produces hierarchical, grounded descriptions covering text and color is directly useful for conditioning and for building data. The reconstruction results with both renderers support this. The revision and rebuttal reflect a substantial amount of work. Methodological novelty remains modest, but I consider the task formulation, data pipeline and benchmark a solid contribution. I raise my score to 3.

Critical weaknesses

Both the training targets and benchmark references are produced using ParseEngine, with the latter reviewed and refined by humans. This raises a concern that ImageParser may benefit from learning the conventions of the same pipeline used to create the evaluation targets. Human review helps, but it is still unclear how well the benchmark measures actual semantic completeness, rather than agreement with the annotation pipeline.

More broadly, there are already many papers that collect large amounts of data, fine-tune a model, and report benchmark gains. Given that the training here is essentially standard SFT, I find it difficult to see a sufficiently strong methodological contribution for a conference paper.

I also noticed that quite a few images in the paper look like they may have been generated using AI-based coding tools. The authors should clarify the sources of these images and distinguish original images from generated or reconstructed ones.

Questions for authors
  1. Since both the training targets and benchmark references are produced using ParseEngine, how do the authors ensure that ImageParserBench measures actual semantic correctness rather than agreement with the annotation pipeline?

  2. Which components will be released, including model weights, benchmark annotations, evaluation code, and the ParseEngine pipeline? Given the use of a large internal image collection and proprietary VLMs for annotation, how much of the data construction and evaluation can be independently reproduced?

Reviewer_gNNHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to convert images into structured language representations that are independent of downstream questions and encode spatial grounding and explicit parent–child relationships. The authors propose ImageParser, which uses a unified JSON format to represent global content and appearance, region locations and descriptions, visible text, colors, and hierarchical relationships. They use ParseEngine to construct approximately 6.1 million image–parse annotation pairs and train models with 0.8B, 2B, and 4B parameters. On the authors' ImageParserBench, ImageParser-4B outperforms the evaluated external general-purpose VLMs on all six parsing metrics. Its region-description F1 and hierarchy F1 exceed the corresponding strongest external baselines by approximately 20.0 and 11.8 percentage points, respectively. The benchmark contains 1,975 images, with 63,376 regions reported in the main text; reference annotations are generated with model assistance and subsequently reviewed by humans. Further experiments on detection, OCR, caption-content readout, image reconstruction, and question answering without access to the original image assess the representation’s reusability and information retention.

Critical strengths
  1. Strong parsing results with small models. In Table 1, ImageParser-4B achieves a region-description F1 of 0.5722, compared with 0.3719 for the corresponding strongest external baseline, and a hierarchy F1 of 0.4868, compared with 0.3685 for the corresponding baseline. These results support the feasibility of teaching compact models complex parsing tasks through dense structured supervision.
  2. A relatively comprehensive data construction pipeline. Figure 3 and Section 4.1 describe region discovery, OCR integration, structural validation, iterative correction, and color extraction, providing a concrete approach to training a unified parsing model.
  3. Downstream evaluations demonstrate both utility and limitations. Table 5 compares original-image and JSON inputs across five external question-answering benchmarks, holding the readout model fixed within each comparison. Original-image inputs perform better overall, but the JSON representations retain a substantial amount of information useful for answering questions.
Critical weaknesses
  1. The independent contribution of explicit parent–child relationships remains unclear. Table 2 examines description length, training schemes, and decoding order, but does not directly compare representations with and without parent–child references. The current JSON already uses flat-list serialization. I suggest removing explicit parent relationships while retaining fields such as region bounding boxes and descriptions, using the same backbone and data and matching the training budget as closely as possible, to distinguish the contribution of structural information from that of dense supervision.
  2. Generation budgets and failure rates for external baselines are insufficiently documented. Appendix D.3 describes the models, prompt structure, and output adaptation, but does not list maximum output lengths, image-input budgets, or reasoning settings, where applicable, for each model. Section 4.2 treats invalid predictions as empty predictions. Since the target JSON can be lengthy, truncation or formatting failures may affect the overall scores. There is currently no evidence that unfair settings have affected the rankings, but reporting these configurations and the rates of invalid outputs and truncation would help clarify the sources of the performance gaps.
  3. The scope of human review and differences in region partitioning require further clarification. Section 4.2 explicitly states that the test references are drafted by ParseEngine and undergo additional human review, rather than constituting exhaustive human ground truth. A shared annotation pipeline does not, by itself, establish training–test leakage. Nevertheless, multiple reasonable answers may exist for region granularity and parent–child assignments. I suggest clarifying whether human review covers missing regions, bounding boxes, and parent–child relationships, and independently categorizing a small sample of mismatches to distinguish genuine errors, omissions in the references, and reasonable alternative partitionings.
Questions for authors
  1. Is there a controlled comparison that retains the same region bounding boxes and descriptions but removes explicit parent–child references? How would this affect parsing, question answering, and reconstruction?
  2. Could you report the maximum output length, image-input budget, and reasoning settings, where applicable, for each baseline in Table 1, together with invalid-JSON and truncation rates?
  3. Which fields are covered by the human review of the test references, and what consistency criteria are used? Could you report the proportions of genuine errors, reference omissions, and reasonable partitioning differences in a small sample of mismatched predictions? Minor Issue Figure 3 on page 4 reports 63,760 regions, whereas the main text in Section 4.2 reports 63,376. These figures should be made consistent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.