acceptodds
Under review as a conference paper at ICLR 2027

SeeTheFlow: Structure-Guided Flowchart Parsing

Abstract

Flowchart parsing requires vision-language models (VLMs) to recover not only textual and visual content but also the underlying topology. VLMs performance degrades sharply as diagrams become larger and more densely connected. Existing approaches primarily rely on visual reasoning or additional VLM inference, while the benefit of explicitly supplying externally extracted structure remains less explored. We introduce **SeeTheFlow**, a VLM-training-free framework that extracts an explicit structural prior from flowchart images using deterministic computer vision and OCR, then conditions a VLM on this prior through a lightweight, cost-aware router that intervenes only on diagrams likely to benefit. Across six VLM backbones, SeeTheFlow improves connection weighted F1 by up to 30 percentage points on large, structurally complex diagrams and by 11.2 points on real-world flowcharts, while invoking a second VLM inference on only roughly one-third of inputs rather than applying structural guidance unconditionally. We further introduce a 1,200-instance benchmark with controlled structural complexity, isolating diagram topology from lexical and rendering variation. Our results show that explicit, selectively applied structural evidence can improve robustness to structural complexity while avoiding the cost of additional inference on every diagram. Our code and data are publicly available at https://anonymous.4open.science/r/SeeTheFlow_ICLR-0502.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Flowchart Parsing, Diagram Understanding, Graph Topology Recovery, Training-Free Framework, Cost-Aware Routing, Document AI, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KPSGOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SeeTheFlow, a training-free framework for flowchart parsing that combines CV/OCR derived structural priors with VLM inference. It introduces an adaptive SVM router to selectively apply structural guidance and an optional verification mechanism to refine predicted graphs. The authors also construct a benchmark of 1,200 flowcharts with varying structural complexity. Experiments show improvements in CW-F1, particularly on large and complex flowcharts, while reducing unnecessary VLM inference calls.

Critical strengths
  1. Addresses the challenge of recovering node connectivity and control-flow structures in complex flowcharts.
  2. Proposes a training-free framework combining CV/OCR derived structural priors with adaptive routing to reduce unnecessary VLM calls.
  3. Introduces a benchmark of 1,200 flowcharts with controlled structural complexity.
Critical weaknesses
  1. Table 1 only reports relative improvements without absolute baseline scores, making it difficult to assess performance degradation as diagram complexity increases.
  2. The effectiveness of the method varies substantially across VLMs. While Gemini-3.5 achieves a 22.8 pp improvement in CW-F1, Llama-4 and Qwen-3.5 experience drops of 14.0 and 7.8 pp, respectively. The proposed router is model-independent and cannot account for these differences. Moreover, the paper lacks a detailed analysis of why structural guidance degrades performance for certain models.
  3. Several inconsistencies appear in the reported results, including intervention rates in Figure 4, error proportions in Figure 3, and average VLM call counts in Table 1.
Questions for authors
  1. Table 1 reports only the CW-F1 improvements over the baseline, without providing the absolute baseline scores for the pooled and large subsets. Given the paper's central claim that image-only VLM performance degrades with increasing diagram complexity, could the authors report the absolute CW-F1 scores for both the baseline and SeeTheFlow across different diagram sizes? This would help clarify the extent of performance degradation and how effectively SeeTheFlow mitigates it.
  2. Figure 4 reports an intervention rate of 8% for small diagrams, whereas both its caption and the main text in Sections 6 and 7 report 4%. Could the authors clarify which value is correct and explain the discrepancy? Were the figure and the reported statistics generated from the same evaluation runs?
  3. In Figure 3 (right), the proportion of missing connections remains at 37% for both Baseline and SeeTheFlow (13.9 → 11.2 errors per diagram), whereas Section 6 states that this proportion decreases from 36.3% to 28.3%. Similarly, the proportions of extra connections reported in the figure (8% → 14%) differ from those in the text (19.5% → 30.3%). Could the authors clarify whether these statistics are computed using different evaluation subsets or definitions, and explain these discrepancies?
Reviewer_wfPiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the parsing and topology recovery of flow diagrams and proposes a training-free approach. The authors show that a multi-stage approach improves flowchart graph construction in 4 out of 6 VLMs, while degrading performance in the remaining 2 VLMs. The framework uses an image-based model to extract a graph-structured prior, followed by a cost-aware SVM router that decides whether to retain the image-only baseline prediction or invoke a second, structure-conditioned VLM call.

Critical strengths
  • Up to 20-30 % gain without VLM training by using a SVM based classifier is interesting
  • The paper is well structured and relatively easy to follow, although the overall presentation could certainly benefit from improvement.
Critical weaknesses
  • The analysis and discussion of the experimental results are rather limited and could be further improved.
  • The scope of the study is rather limited, as it focuses only on graph reconstruction. The study can be viewed as focusing primarily on the automated annotation of flowcharts. However, it remains unclear how these improved annotations contribute to more general downstream tasks, such as Visual Question Answering (VQA). I believe that demonstrating improvements on tasks such as VQA is essential to assess the broader impact of the proposed approach on more standard benchmarks, beyond visual annotation alone.
  • The paper would benefit from the inclusion of a human evaluation.
  • The overall presentation and writing quality of the paper need improvement
Questions for authors

Q: It is not clear why it is degrading for 2 out of 6 frameworks, it raises questions why OCR guidance hurts the performance in one and not the other? Q: How do you form the prompts for the iteration phase? Q: Do you have qualitative and quantitative analysis of the types of charts goes through structure-conditioned VLM calls?
Q: Why does abstract claim 6 VLM framework improvement, while results show 4 out of 6?

Reviewer_WHHkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SeeTheFlow combines a graph prior extracted by CV/OCR, a learned router, and optional structural verification to parse flowcharts without fine-tuning the VLM. The paper introduces a benchmark with controlled structural complexity and examines parsing accuracy, inference cost, label perturbations, and transfer to real diagrams. It reports improvements on complex flowcharts, with substantial variation across backbones and prior quality.

Critical strengths
  1. The explicit graph prior provides an inspectable intermediate representation, and selective intervention is motivated by the observation that inaccurate priors can damage otherwise correct predictions. The framework does not require task-specific VLM fine-tuning.

  2. The benchmark varies structural complexity and subgraph conditions within shared semantic domains. Label perturbations and error analysis provide useful evidence beyond aggregate accuracy, including failures of the proposed method.

Critical weaknesses
  1. The novelty is limited. The contribution is principally a system integration of CV/OCR extraction, heuristic repair, SVM routing, and rule-based verification.

  2. The contribution of recovered connectivity is not isolated. The prior also supplies OCR text, and CW-F1 uses label matching, so label perturbations cannot separate topology gains from OCR assistance.

  3. The shared router ignores backbone identity despite the regressions for Llama and Qwen. This limits its ability to select beneficial interventions across models.

  4. The verifier forbids edge deletion, although structural guidance increases spurious connections. It therefore cannot accept corrections that require removing those connections.

  5. The efficiency claim lacks consistent accounting. The inference stages in Sections 5.2 and 5.3 do not align with the call range in Table 1, while Section 7 leaves intervention rates and average calls unreconciled.

Questions for authors
  1. What benefit does the learned router provide over the simple size threshold mentioned in Section 5.2 under the same inference budget?

  2. With OCR text and detected node locations held fixed, what changes when the prior's recovered edges are removed?

  3. How does the shared router handle a diagram whose prior helps one backbone but harms another, and what explains the regressions in Table 2?

  4. What evidence justifies prohibiting edge deletion despite the increase in spurious connections reported in Figure 3?

  5. Do the accuracy, intervention rates, and call counts in Table 1 and Section 7 use the same evaluation population, and how are refinement calls included?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.