- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ImageParser, a family of small vision-language models (0.8-4B) that generate hierarchical, spatially grounded JSON representations of images, including region descriptions, bounding boxes, OCR, relationships, color palettes, etc. The work develops ParseEngine, an automated annotation pipeline combining frontier LVLMs and specialist models to generate 6.1M image-parse pairs for supervised finetuning. It also proposes ImageParserBench, a human-reviewed benchmark of 1975 images for evaluating hierarchical image parsing. Experiments show improvements over general-purpose VLMs on ImageParserBench, with additional evaluations on object detection, OCR, captioning, image reconstruction, and pixel-free QA.
Critical strengths
- Comprehensive annotation pipeline and useful benchmark. ParseEngine integrates VLMs, specialist models, and iterative refinement to generate detailed image annotations at scale. The human-reviewed ImageParserBench benchmark with over 63K annotated regions across diverse visual domains is also beneficial for the research community. While their individual components are not particularly novel, ParseEngine and ImageParserBench are useful contributions for hierarchical image parsing.
- Interesting experiments on information preservation. The JSON-to-image reconstruction and pixel-free QA experiments demonstrate that the generated representations preserve substantial visual information, allowing downstream models to reconstruct images or answer questions w/o access to the original pixels. The use of 2 different image generators strengthens reconstruction results, while pixel-free QA experiments provide useful insights into both the capabilities and limitations of the representation.
Critical weaknesses
- Limited modeling novelty beyond distillation and insufficient training data validation. ImageParser is trained by standard autoregressive SFT on ParseEngine annotations. While distilling a complex annotation pipeline into compact models is a practical contribution, the model family itself introduces little methodological novelty. The paper does not compare ImageParser results against ParseEngine results directly, making it unclear what is gained or lost through distillation beyond simpler inference, albeit at the cost of less flexibility to choose component models. Additionally, the 6.1M training annotations are generated automatically w/o quantitative evaluation of their quality or ablations w.r.t. contributions of individual refinement stages. This raise further questions w.r.t. the necessity of the model beyond distillation, compared to just using the data pipeline, since there is no difference in data quality due to lack of human validation.
- Limited evidence of generalization to external benchmarks. The results outside of ImageParserBench are not as convincing. On LVIS detection, ImageParser performs reasonably well but does not surpass the strongest baselines, especially reasoning-based ones. On OCRBench v2, the strongest results may largely reflect only the specialist OCR supervision used by ParseEngine (since there is no particular OCR-based model contribution); plus, comparisons with competitive models such as Qwen3-VL (available in Table 3(a) and (c)) are missing. On CAPEval, ImageParser achieves highest precision but lower coverage than several baselines with a different evaluation protocol from reported methods. These results provide limited evidence that the proposed representation is consistently better across downstream tasks.
- Unclear implications of autoregressive prediction for set-based image parsing. Image regions are naturally represented as a set with hierarchical relationships, but ImageParser predicts them sequentially in a fixed parent-before-child order. Although the paper shows experiments for global-first and layout-first decoding, these experiments only change the ordering of top-level fields, not the ordering of individual regions. It remains unclear whether the model is sensitive to arbitrary region ordering or whether errors accumulate over long sequences, particularly for images with many regions.
Questions for authors
- How does ImageParser compare directly against ParseEngine on held-out images in terms of parsing quality and inference cost? What are the main benefits of distillation beyond reducing the number of models required at inference?
- Could the authors provide a quantitative validation of the automatically generated training annotations and assess the contribution of ParseEngine's refinement stages?
- Could the authors evaluate whether ImageParser's improvements hold on independently annotated benchmarks, and provide more comprehensive comparisons with recent VLMs on external tasks?
- How sensitive is ImageParser to the ordering of regions in the output JSON? Have the authors considered randomizing sibling-region order during training or evaluating robustness to different region serializations?
3 comments
Further Clarifications on Annotation Quality and Novelty
We sincerely thank the reviewer for the thoughtful follow-up and for acknowledging our additional experiments.
On annotation quality validation. We appreciate the opportunity to further clarify our data validation process. Beyond the annotation-stage statistics in W2, we would like to highlight the direct, reference-free quality evaluation already presented in General Response [2/2]. As shown in Table 2, an independent audit using GPT-6 Astra (which was completely decoupled from our annotation pipeline) confirms that ParseEngine generates highly accurate annotations: 11.3 correct facts per region, only 0.79 major errors per 100 claims, and 95.3% exact OCR. Furthermore, the downstream performance of ImageParser-4B also provides strong empirical validation: trained solely on these annotations, our 4B model surpasses frontier proprietary models—including GPT-5.6-Sol and Gemini-3.8-Flash—in zero-shot parsing (Table 1 in General Response [1/2]). Such gains would not be achievable if the underlying supervision were noisy or incomplete. Taken together, we hope these direct and downstream results alleviate your concerns regarding annotation quality.
On novelty. We understand the reviewer's perspective and kindly invite you to view our contribution as an integrated whole, as detailed in General Response [1/2]. Rather than standard distillation, our work brings together four tightly coupled components: (1) a novel structured representation, (2) a multi-stage annotation pipeline, (3) a new benchmark, and (4) empirical evidence that a single compact model can match complex multi-model pipelines. Collectively, these contributions advance the field beyond conventional distillation paradigms.
Thank you again for your time and constructive engagement. In light of these additional results and clarifications, we sincerely hope you will reconsider your assessment and consider raising your score.
We respectfully ask the reviewer to reconsider what constitutes novel findings in this work. Beyond the pipeline and benchmark, we highlight two findings that we believe are non-trivial and significant:
Finding 1: Compact models are sufficient for highly complex structured parsing. Prior to this work, it was unclear whether a single small model could jointly produce hierarchical region decomposition, bounding boxes, descriptions, OCR, color palettes, and parent–child relations in one pass, each of which traditionally required a dedicated specialist model. We demonstrate that even a 0.8B model can achieve this, and a 4B model matches a multi-call pipeline of frontier proprietary models. This is a surprising and practically important result, providing a solid foundation for next-generation image generation pipelines that require billion-scale high-quality annotation demanding.
Finding 2: Hierarchical parsing approaches semantic equivalence with the source image. We show that sufficiently detailed hierarchical parses can serve as a text-only proxy for the original image, enabling both faithful image reconstruction from parse alone and pixel-free visual QA (a new evaluation paradigm we propose). This finding has broad implications: if a structured text representation can substitute for raw pixels in downstream reasoning, it opens new directions for embodied AI, efficient visual reasoning, and language-centric vision systems. To our knowledge, no prior work has demonstrated this level of semantic equivalence from a single-pass structured parse.
We believe these findings go substantially beyond "distillation and structured supervision" and sincerely hope the reviewer can reconsider their assessment accordingly.
Thank you for the detailed rebuttal and additional experiments. I appreciate the direct comparison with ParseEngine, the additional external baselines, and the region-order ablation, which address several of my original concerns.
However, my main concern remains the limited novelty and significance of the work. While the data pipeline and benchmark are useful contributions, I am still not fully convinced that the paper offers sufficiently novel findings beyond distillation and structured supervision. Additionally, while the annotation-stage analysis is useful, it mainly shows changes in annotation quantity and detail rather than directly validating the correctness of the generated training annotations.
I will therefore maintain my original score. Thank you again for your time and effort in addressing my concerns.