acceptodds
Under review as a conference paper at ICLR 2027

Tool-Augmented Multimodal Agents for Piping & Instrumentation Diagram Understanding

Abstract

Piping and instrumentation diagrams (P&IDs) are key engineering documents that specify equipment, instrumentation, and piping connectivity in complex industrial systems. Despite their central role in engineering, procurement, and construction workflows, P&IDs remain largely interpreted manually due to dense layouts, heterogeneous drafting standards, and project-specific conventions. Interpreting these diagrams requires not only symbol detection but also reliable reasoning about line connectivity, topology, and semantic attributes; capabilities that existing approaches struggle to achieve. We introduce PID-Agents, the first tool-augmented multimodal agent framework for structured P&ID understanding. Rather than relying solely on static visual recognition, PID-Agents enables vision-language agents to actively explore diagrams through a set of novel domain-specific drawing tools, including region-of-interest rendering, event-driven line tracing, and local connectivity probing. Through iterative tool use, the agent gathers visual evidence and reconstructs pipeline-level relations and attributes beyond object detection. To support specification-driven reasoning, we further integrate retrieval-augmented specification, grounding attribute inference and consistency checks in external engineering documents. The resulting system converts P&IDs (rasterized as vector pdf) into connectivity-centric structured representations consisting of entities, typed relations, and provenance, enabling reliable end-to-end interpretation of real industrial drawings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
piping and instrumentation diagrams, multimodal agents, tool-augmented agents, engineering drawing understanding, document understanding

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LxwSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the conversion of piping and instrumentation diagrams (P&IDs) into structured listings containing identifiers, connectivity, and engineering attributes. PID-AGENTS first uses OCR and symbol detection to locate pipeline identifiers and instrument tags, and retrieves project specification documents to prefill engineering attributes. It then processes individual pipelines or instruments using Gemini 3 Flash with tools for local image cropping, text search, line-segment localization, event-driven tracing, and connectivity probing to recover endpoints and associated attributes. The paper reports a cross-project evaluation from Project A to Project B, with a quantitatively annotated subset containing 212 pipeline connections and 622 instrument instances. The main contribution is a tool-augmented parsing workflow for real industrial drawings, together with empirical validation in this application.

Critical strengths
  1. The problem has practical value, and the task goes beyond symbol detection. The paper integrates pipeline endpoints, instrument-to-pipeline associations, and specification attributes into a unified output (Sections 2–3), which more directly addresses the needs of engineering listing workflows. Section 3.3 describes a reasonable division of labor: tools trace line segments, while semantic decisions at junction or endpoint events are informed by local observations. Two execution traces on OPEN100 drawings in Appendix D illustrate this workflow.
  2. Results on real industrial drawings provide positive evidence for the complete workflow. Table 1 includes component-level comparisons involving perception, RAG, and the full agent stage. For pipeline listing, specification retrieval primarily improves specification fields, while adding the agent and drawing tools increases From/To accuracy from 45.75%/41.51% with VLM + Perception + RAG to 63.21%/66.51%. The average field accuracy for instrument listing also increases from 48.96% for the best baseline reported in the table to 77.55%. These results support the overall value of the tool-augmented stage, although they do not isolate the contribution of each tool.
Critical weaknesses
  1. The incremental benefit of the specialized topology tools over general local inspection remains unclear. Section 3.3 describes the interaction between event-driven tracing and agent-based endpoint identification, and Table 1 supports the gains from the complete tool-augmented stage. However, the reported experiments do not separately compare an agent using only local crops and text search with the full agent that additionally uses line-segment localization, tracing, and connectivity probing. Consequently, it remains unclear how much of the improvement comes from clearer local visual inputs and additional inference computation, and how much comes from the topology tools emphasized in the paper. The implementation descriptions in Section 3.3 and Appendix B.1 also remain relatively high-level. For example, rules for handling line crossings and breaks, and criteria for filtering candidate contact points, are not specified in detail, limiting assessment of the tool designs and their operating conditions.
  2. The end-to-end evaluation of connectivity requires further specification. Section 4.1 (L365–371) explicitly evaluates From and To separately and uses partial matching for endpoints, but neither the main paper nor the appendix details the matching rules or clearly explains how missed identifiers, extra predicted records, and timed-out records are counted. The 88.57% in Table 1 is the mean accuracy across pipeline fields, including several specification fields with near-perfect scores; the From/To values of 63.21%/66.51% are accuracies for the two endpoint fields. These metrics do not directly provide the proportion of pipeline connections for which both endpoints are correct. Clarifying endpoint normalization and semantic-equivalence rules, reporting the accuracy of complete ordered endpoint pairs using existing predictions, and explaining the treatment of incomplete records would help assess the reliability of the resulting connectivity listings.
Questions for authors
  1. Could the authors compare an agent using only local crops and text search with the full agent that additionally uses line-segment localization, tracing, and connectivity probing, while keeping the OCR/detection outputs and VLM fixed and using comparable model-call budgets? Reporting From/To accuracy and the proportion of connections with both endpoints correct under clearly defined matching rules would help establish the incremental contribution of the specialized topology tools over general local inspection.
2 comments
AuthorsOpenReview ↗

Response to Reviewer LxwS (1/2)

We thank the reviewer for realizing the practical value of the task and its unified output, the reasonable division of labor between the tools and the semantic decisions, and the positive evidence on real industrial drawings. We address each concern below with new experiments and revisions, which are highlighted in blue in the revised paper.

Incremental Benefit of the Specialized Topology Tools over General Local Inspection (W1, Q1.1)

R1 We ran this comparison on a subset of the evaluation set with 64 pipelines and 193 instruments (revised Section 4.4, new Table 4). The OCR/detection outputs and the VLM are fixed, and only the tool list changes. All scores use the partial matching of Table 1, which R2 defines.

ToolsFromToBothLLM calls
Local crops and text search (ROI Rendering + OCR-Based Search)0.54690.62500.437520.8
+ Line-segment localization and tracing (Line Search + Event-Driven Line Tracing)0.73440.68750.593823.1
+ Connectivity probing (full agent)0.71880.67190.562525.3

LLM calls is the mean number of model calls per pipeline on this subset. The model-call budgets are comparable. Adding line-segment localization and tracing raises Both from 43.8% to 59.4% with only 2.3 more model calls per pipeline. The full agent reaches 56.3% with 4.5 more calls. The difference between the last two rows is two of the 64 pipelines. The agent calls Connectivity Probing only about 0.2 times per pipeline, and repeated runs show that this difference is within run-to-run variation (new Appendix B.4).

Additional inference computation does not explain the gain. Configurations without ROI Rendering use 37 to 85 model calls per pipeline, yet none of them exceeds 45.3% Both (Table 4). Local crops and text search alone give 43.8% Both, and the specialized topology tools add 12.5 to 15.6 points at a comparable model-call budget.

Rules for line crossings, breaks, and candidate contact points. The new Appendix B.4 specifies these rules.

  • Line map. The tracer runs on a line map built once per sheet. OCR text is erased so that labels neither break a line nor create a false one. Only horizontal and vertical runs are kept, and breaks of about 1 mm are closed.
  • Crossings and junctions. At a four-way crossing, the tracer continues straight without an event, since a pipeline usually runs straight across another line. At a T-junction, it stops and reports the available directions. It also stops when the line ends or reaches a large solid region such as a filled symbol, and it reports which of the two happened.
  • Candidate contact points. Connectivity Probing spreads from the stop point along the connected lines and closes small gaps on the way. Wherever it reaches a solid region such as an arrowhead or a filled symbol, it records a contact candidate. Candidates are not filtered, except that candidates within 5 mm of each other are merged. For each candidate, the tool returns the nearest OCR text or detected symbol and, for an arrowhead, the direction in which it points.
  • Decision. The agent then decides which candidate, if any, continues the pipeline or names its endpoint.
AuthorsOpenReview ↗

Response to Reviewer LxwS (2/2)

End-to-End Evaluation of Connectivity (W2, Q1.2)

R2 Using the existing predictions, the table below reports the accuracy of complete ordered endpoint pairs (Both), together with From and To, under strict matching and under the partial matching of Table 1 (new Appendix C, Table 8).

MethodFrom (strict)To (strict)Both (strict)From (partial)To (partial)Both (partial)
VLM baseline0.30190.18400.06600.46230.40090.2642
VLM + Perception0.29250.21230.08960.43870.41040.2453
VLM + Perception + RAG0.30660.21230.11790.45750.41510.2547
Rule-based Controller0.20750.30660.08490.30190.41040.1509
Pixel-Graph Search0.25940.28300.09430.49060.43400.2311
Rule-based Topology0.19340.22640.03770.31130.33490.1226
PID-Agents0.43870.44810.20750.63210.66510.4953

PID-Agents gets both endpoints right for 49.5% of the pipelines under partial matching and for 20.8% under strict matching. No other method exceeds 26.4% and 11.8%, respectively.

Endpoint normalization and semantic-equivalence rules. The ground truth for From and To comes from the project line list, which is compiled by hand. The list often names an endpoint differently from the label on the drawing. For example, it records the group name "P-101A/B" for a line that starts at pump P-101B alone. Both strings are first uppercased, and runs of whitespace are collapsed. A predicted endpoint is then correct if it equals the ground truth, if one string contains the other, if the two share an equipment tag after a group such as "P-101A/B" is expanded, or if both contain the header token "HDR". Strict matching instead requires the first predicted endpoint to equal the ground truth exactly after uppercasing and trimming spaces. Both counts ordered pairs. A pipeline counts only when the predicted From matches the ground-truth From and the predicted To matches the ground-truth To, so a swapped pair counts as wrong.

Incomplete records. The new Appendix C states three rules. A missed identifier, which is a ground-truth pipeline that a method does not list, counts as wrong in every field. An extra predicted record, which has no ground-truth row on its sheet, does not affect accuracy. A timed-out record is scored as wrong. In the PID-Agents runs of Table 1, every listed pipeline returned an answer. PID-Agents lists 209 of the 212 pipelines, and each of the three missing ones continues across two adjacent sheets.

Mean accuracy across pipeline fields. We agree that the 88.57% mixes near-perfect specification fields with the two endpoint fields. We therefore report From, To, and Both separately in the new Table 3 and Table 8, and our claims about connectivity now rest on these columns.

We hope that these results address your concerns, and we would be grateful if you would consider raising your score.

Reviewer_Wm5hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PID-AGENTS, a framework for extracting structured information from piping and instrumentation diagrams. It combines OCR, symbol detection, RAG, and a VLM agent equipped with tools for local image inspection and pipeline tracing. The goal is to recover pipeline connectivity and engineering attributes, not just detect individual symbols. Experiments on industrial P&IDs show improvements over VLM-based baselines, especially for connectivity-related fields and instrument listing.

Critical strengths
  1. P&ID understanding is a challenging task with practical value. The paper focuses on connectivity reconstruction, which is more useful for downstream applications than symbol detection alone.
  2. The use of drawing-specific tools is reasonable. Line tracing and connectivity probing give the VLM access to local geometric information that is hard to recover from a full-page image.
  3. The framework is fairly complete. It combines OCR, symbol detection, specification retrieval, and connectivity reconstruction into a single pipeline.
  4. The cross-project setting is a good choice. Training on Project A and testing on Project B meaningfully evaluates generalization across drawing styles.
  5. The instrument-listing results in Table 1 are quite good. The average accuracy increases from 0.4896 to 0.7755 compared with the perception-and-RAG baseline. The improvements in Fluid, Line Size, and Line Spec are particularly large.
Critical weaknesses
  1. The comparisons in Table 1 are limited to Gemini 3 Flash and its variants. Since the main contribution is connectivity reconstruction, it would be important to compare it with an existing graph-based or traditional line-tracing method. Otherwise, it is hard to tell whether the agent is actually better than a well-designed non-agent pipeline.
  2. Most of the improvement in pipeline listing comes from RAG rather than the agent. In Table 1, the average accuracy increases from 0.3501 to 0.8534 after adding RAG, but only reaches 0.8857 with PID-AGENTS. This is not necessarily a problem, but the agent's contribution should be stated more precisely.
  3. The From and To accuracies are only 0.6321 and 0.6651. These are better than the baselines, but still rather low for the main task. More importantly, the paper evaluates the two endpoints separately. It would be useful to know how often an entire pipeline connection is correct.
  4. Section 4.1 mentions approximately 3,500 pipeline instances, but only 212 verified From-To connections are used for quantitative evaluation. The paper does not clearly explain how it selected this subset. This raises questions about how representative the reported connectivity accuracy is.
  5. The contribution of each drawing tool is unclear. Section 3.3 introduces several tools, but Table 1 only compares the full agent with non-agent baselines. In particular, I would like to know the individual contributions of event-driven tracing and connectivity probing.
  6. The evaluation uses partial matching for pipeline endpoints. Since connectivity is the central task, the exact criteria should be made clear. It would also be useful to report strict exact-match accuracy.
  7. There is no runtime analysis for the agent stage. Each line may require multiple tool calls and VLM interactions, but the paper does not report how many calls are needed or how long it takes to process a sheet.
Questions for authors

The application is interesting, and the system clearly improves on direct VLM baselines. The use of drawing-specific tools is also reasonable. However, I am not fully convinced that the experiments isolate the proposed agent's contribution. My main concern is the lack of non-agent connectivity baselines and tool-level ablations. The relatively low endpoint accuracy and small connectivity evaluation set also make the results less convincing. I would be more positive if the authors addressed these points more clearly.

  1. Can the authors explain why no traditional line-tracing or graph-based connectivity method is included as a baseline?
  2. What percentage of pipelines have both From and To endpoints correctly identified?
  3. How were the 212 verified connections selected from the full dataset?
  4. How much does performance drop without event-driven tracing or connectivity probing?
  5. What are the exact partial-matching rules for endpoints?
  6. What is the average runtime and number of tool calls per pipeline?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Wm5h (1/3)

We thank the reviewer for recognizing the practical value of connectivity reconstruction, the reasonable use of drawing-specific tools, the completeness of the framework, the cross-project setting, and the strong instrument-listing results. We address each concern below with new experiments and revisions, which are highlighted in blue in the revised paper.

Comparison with Traditional Baselines (W1, Q1)

R1 The original submission compared only VLM variants, because no prior system produces the full listing with its specification attributes. We agree that the connectivity fields should still be compared with existing methods. We have therefore added three non-agent baselines (revised Section 4.4, new Table 3, all fields in Table 10). Two are existing methods, the pixel-graph search of Mani et al. (2020) and the rule-based topology reconstruction of Kim et al. (2022). The third is a rule-based controller that applies our own drawing tools under a fixed rule. All three use the same OCR and symbol detection outputs as PID-Agents and make no model calls.

MethodFromToBothPipe. Avg.Inst. Avg.
VLM + Perception + RAG0.45750.41510.25470.85340.4896
Rule-based Controller0.30190.41040.15090.27360.5163
Pixel-Graph Search (Mani et al., 2020)0.49060.43400.23110.28990.5482
Rule-based Topology (Kim et al., 2022)0.31130.33490.12260.26850.5180
PID-Agents0.63210.66510.49530.88570.7755

Both means that From and To are both correct. Pipe. Avg. and Inst. Avg. are the averages over the pipeline-listing and instrument-listing fields. The non-agent methods read no specification documents, so their pipeline averages are low, and the endpoint columns give the fair comparison. None of them recovers both endpoints for more than 23.1% of the pipelines, compared with 49.5% for PID-Agents (McNemar test, p < 0.001). They also link instruments to pipelines less accurately. Their Fluid, Line Size, and Line Spec accuracies are at most 0.59/0.52/0.59, while PID-Agents reaches 0.72/0.66/0.72 (Table 3). These baselines include pixel-graph traversal, rule-based topology reconstruction, and a non-agent pipeline built on our own tools, and none of them comes close to PID-Agents.

Contribution of RAG vs. the Agent (W2)

R2 We agree, and we now state the agent's contribution more precisely. RAG and the agent improve different fields (Table 1, Table 3).

  • RAG fills the specification fields. For example, it raises Rating from 0.00 to 0.99 and PWHT from 0.04 to 0.99. However, it moves From/To only from 0.44/0.41 to 0.46/0.42 and Both from 24.5% to 25.5%.
  • The agent recovers connectivity. It raises From/To from 0.46/0.42 to 0.63/0.67 and Both from 25.5% to 49.5%. It also raises the instrument fields that depend on the linked pipeline, namely Fluid, Line Size, and Line Spec, from 0.24/0.15/0.25 to 0.72/0.66/0.72. The instrument average rises from 0.49 to 0.78.

The pipeline average understates the agent's contribution. It covers 13 fields, and only From and To depend on the drawing topology. The agent's gains of 17.5 and 25.0 points on these two fields therefore move the average by only about 3 points. RAG alone could not improve these connectivity-related fields.

AuthorsOpenReview ↗

Response to Reviewer Wm5h (2/3)

Endpoint Accuracy and Entire Pipeline Connections (W3, W6, Q2, Q5)

R3 The table below reports how often From, To, and both endpoints are correct, under strict matching and under the partial matching of Table 1 (new Appendix C, Table 8).

MethodFrom (strict)To (strict)Both (strict)From (partial)To (partial)Both (partial)
VLM baseline0.30190.18400.06600.46230.40090.2642
VLM + Perception0.29250.21230.08960.43870.41040.2453
VLM + Perception + RAG0.30660.21230.11790.45750.41510.2547
Rule-based Controller0.20750.30660.08490.30190.41040.1509
Pixel-Graph Search0.25940.28300.09430.49060.43400.2311
Rule-based Topology0.19340.22640.03770.31130.33490.1226
PID-Agents0.43870.44810.20750.63210.66510.4953

Exact partial-matching rules. The ground truth for From and To comes from the project line list, which is compiled by hand. The list often names an endpoint differently from the label on the drawing. For example, it records the group name "P-101A/B" for a line that starts at pump P-101B alone. We therefore use partial matching so that such differences in notation are not counted as errors. Both strings are first uppercased, and runs of whitespace are collapsed. A predicted endpoint is then correct if it equals the ground truth, if one string contains the other, if the two share an equipment tag after a group such as "P-101A/B" is expanded, or if both contain the header token "HDR".

Strict exact-match accuracy. Strict matching requires the first predicted endpoint to equal the ground truth exactly after uppercasing and trimming spaces. Under strict matching, PID-Agents gets 43.9% of the From endpoints and 44.8% of the To endpoints right. No baseline exceeds 30.7% on either field.

Entire pipeline connections. PID-Agents gets both endpoints right for 49.5% of the pipelines under partial matching and for 20.8% under strict matching. No other method exceeds 26.4% and 11.8%, respectively. PID-Agents nearly doubles the both-endpoint accuracy of VLM + Perception + RAG (25.5% to 49.5%). It is significantly more accurate than every baseline under both criteria (McNemar test, p < 0.001 for partial and p < 0.05 for strict matching).

Why the endpoint fields remain the hardest. The specification fields follow from the line number and the project documents, so RAG fills them almost perfectly. An endpoint instead requires a chain of decisions on the drawing. The agent must find the line label, select the right segment, trace through junctions, crossings, and inline symbols, and name the terminal object. A mistake at any step makes the endpoint wrong, so errors accumulate along the chain (Table 12). The ground truth also comes from a hand-compiled line list that often names endpoints differently from the drawing. This lowers the scores of every method and explains the large gap between strict and partial matching. Despite these difficulties, PID-Agents is the most accurate method under both criteria. With Claude Opus 5.5 as the backbone, it reaches 1.00 in From and To accuracy on a 20-pipeline subset (new Appendix D), which suggests that endpoint accuracy will rise with stronger VLMs.

Selection of the Evaluation Subset (W4, Q3)

R4 The 212 connections were not selected individually. PID-Agents takes about 215 seconds per pipeline and 48.5 minutes per sheet, and every compared method and tool configuration needs its own runs. We therefore fixed a subset of evaluation sheets within our experimental budget, resulting in 212 pipelines, instead of running all of the approximately 3,500 pipelines, and annotated these sheets in detail. We used every pipeline on them. Here, we chose these sheets because they follow one connected process flow and cover diverse equipment types. They cover the Toluene Drum, Catalyst Make-up Drum, Chlorination Feed Drum, Chlorination Reactors #1 and #2, MCT/Toluene Separator, MCT Column, Hydrolysis Feed Pump Package, Hydrolysis Reactor, and MTBE Drum.

Their pipeline density is also typical. The median number of pipelines per sheet is 19 on these sheets and 17 over all sheets with ground truth in the dataset. None of these criteria depends on how hard the endpoints are to find, and every method is evaluated on the same pipelines.

The set is also large enough to separate the methods clearly. The 95% confidence interval for Both under partial matching is [42.9%, 56.2%] for PID-Agents and [20.9%, 32.7%] for the strongest baseline, the VLM baseline. The McNemar test gives p < 0.001 against every baseline. We agree that more sheets and projects would make the evaluation more representative, and our Limitations state that our evaluation covers two process areas of a single project.

AuthorsOpenReview ↗

Response to Reviewer Wm5h (3/3)

Individual Contributions of Event-Driven Tracing and Connectivity Probing (W5, Q4)

R5 We evaluated all ten combinations of the four drawing tools below on a subset of the evaluation set with 64 pipelines and 193 instruments, changing only the tool list (revised Section 4.4, new Table 4, all fields in Table 11).

ROIOCRTraceProbeFromToBothPipe. Avg.Inst. Avg.
✓0.48440.59380.37500.86060.8745‾\underline{0.8745}
✓0.39060.40620.26560.83890.7369
✓0.29690.29690.14060.82330.7444
✓0.21880.21880.09380.81130.7340
✓✓0.54690.62500.43750.86780.8756
✓✓0.51560.46880.37500.85340.7651
✓✓0.29690.29690.18750.82330.7651
✓✓✓0.73440.68750.59380.88700.8595
✓✓✓0.57810.53120.45310.86300.7749
✓✓✓✓0.7188‾\underline{0.7188}0.6719‾\underline{0.6719}0.5625‾\underline{0.5625}0.8846‾\underline{0.8846}0.8756

ROI, OCR, Trace, and Probe denote ROI Rendering, OCR-Based Search, Event-Driven Line Tracing, and Connectivity Probing. Line Search is available only when ROI, OCR, and Trace are all enabled. Bold marks the best result, and underline marks the second best.

Event-driven tracing. Without Event-Driven Line Tracing and Line Search, the agent keeps ROI Rendering and OCR-Based Search and recovers both endpoints for 43.8% of the pipelines. Adding the tracing tools raises Both to 59.4% and From/To from 0.55/0.63 to 0.73/0.69. Performance therefore drops by 15.6 points of Both without event-driven tracing. Tracing also helps without ROI Rendering, where adding it to OCR-Based Search raises Both from 26.6% to 37.5%.

Connectivity probing. Removing Connectivity Probing from the full agent changes Both by only two of the 64 pipelines (59.4% vs. 56.3%). The agent calls it about 0.2 times per pipeline, usually when a trace stops at a gap or an inline symbol. On the pipelines where the agent called it, we repeated the runs three times with and without the tool. Keeping the tool gave slightly more correct endpoint pairs on average, and the difference in Table 4 is within run-to-run variation (new Appendix B.4). Probing therefore acts as a fallback for difficult stops, such as the off-page connector in Figure 7 (Appendix G), rather than as a main source of the gain.

ROI rendering is the most effective tool. Removing it from the full agent lowers Both from 56.3% to 45.3% and the instrument average from 0.88 to 0.77.

Runtime Analysis (W7, Q6)

R6 The revision now adds a runtime analysis for the agent stage (new Section 4.5, Table 5).

UnitLLM callsTool callsCost (USD)Time
Pipeline (From/To)25.013.30.206215 s
Instrument (linked pipeline, I/O)23.37.40.218206 s
Sheet (full listing)2,63395324.5948.5 min

On average, a pipeline takes 215 seconds, 25.0 model calls, and 13.3 tool calls. The tool calls consist of 7.3 ROI renderings, 4.6 line traces, 1.3 text and line searches, and 0.2 connectivity probes. A sheet takes 48.5 minutes with about 10 parallel tasks. PID-Agents lists about 25 pipelines and 81 instruments per sheet and processes each of them as an independent task, so the time per sheet falls as more tasks run in parallel.

We believe that these results directly address the points identified as most important, namely the non-agent connectivity baselines, the tool-level ablations, the endpoint accuracy, and the evaluation set. We hope that they resolve your concerns, and we would be grateful if you would consider raising your score.

Reviewer_F6HvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PID-AGENTS, a tool-augmented agent framework for understanding piping and instrumentation diagrams. This framework combines several tools. The experiments show improvements over VLM-based baselines.

Critical strengths
  1. This paper targets at a practical problem. Understanding P&ID drawings requires more than object detection, and converting them into structured connectivity information could be usefuul for industrial application.

  2. The experiments show improvements over the VLM baselines, especially for connectivity-related fields on Gemini 3 Flash.

Critical weaknesses
  1. The framework mainly combines OCR, object detection, RAG, VLM agents, and drawing-specific tools, and it’s not clearly that different from traditional image processing or graph traversal methods. There’s also no comparison with existing line-detection, graph-reconstruction, or rule-based connectivity methods.

  2. The tool ablation is not detailed enough, so it’s hard to tell whether the gains come from the agent’s reasoning or simply from the specialized tools.

  3. The evaluation is relatively limited, only about 150 drawings, 212 annotated connections, and one cross-project setting. It mainly reports field-level accuracy and doesn’t clearly report full From–To connection accuracy, graph-level consistency, or errors caused by missed/incorrect detections.

  4. Runtime, number of tool calls, and computational cost are not well reported, so scalability to large industrial projects remains unclear.

  5. The experiments are conducted only on Gemini 3 Flash. It is unclear whether the gains hold with stronger models like GPT-6 Astra or Claude Opus 5.5. That means how these models do without these tools?

Questions for authors
  1. How does it compare with traditional line-tracing or graph-based connectivity methods? Could you add a non-agent baseline using the same OCR and symbol detection results?

  2. How well does it generalize to more industrial projects with different drawing conventions? Currently there is only one cross-project setting.

  3. On average, how many tool calls, how much time, and what computational cost are needed to process a drawing or a pipeline identifier? How does this compare with traditional non-agent approaches?

  4. Why are many Table 1 values identical to four decimals?

6 comments
AuthorsOpenReview ↗

Response to Reviewer F6Hv (1/4)

We thank the reviewer for acknowledging the practical value of converting P&ID drawings into structured connectivity information and the improvements on connectivity-related fields. We address each concern below with new experiments and revisions, which are highlighted in blue in the revised paper.

Difference from and Comparison with Traditional Methods (W1, Q1)

R1 The main difference lies in who makes the connectivity decisions. Traditional line-detection, graph-reconstruction, and rule-based connectivity methods resolve every junction and endpoint with fixed rules. Our drawing tools use low-level operations similar to traditional image processing and graph traversal. However, they stop at each junction, line end, and symbol and return what they observe. The VLM then decides how the pipeline continues and which object is its endpoint, using flow arrows, labels, equipment context, and the project legend. Retrieval over the project documents adds the specification attributes, which traditional methods do not produce.

To test whether this difference matters, we added three non-agent baselines that use the same OCR and symbol detection results (revised Section 4.4, new Table 3). Two are existing connectivity methods, the pixel-graph search of Mani et al. (2020) and the rule-based topology reconstruction of Kim et al. (2022). The third is a rule-based controller that applies our own drawing tools under a fixed rule. None of them makes model calls.

MethodFromToBothPipe. Avg.Inst. Avg.
VLM + Perception + RAG0.45750.41510.25470.85340.4896
Rule-based Controller0.30190.41040.15090.27360.5163
Pixel-Graph Search (Mani et al., 2020)0.49060.43400.23110.28990.5482
Rule-based Topology (Kim et al., 2022)0.31130.33490.12260.26850.5180
PID-Agents0.63210.66510.49530.88570.7755

Both means that From and To are both correct. Pipe. Avg. and Inst. Avg. are the averages over the pipeline-listing and instrument-listing fields. The non-agent methods read no specification documents, so their pipeline averages are low, and the endpoint columns give the fair comparison. None of the non-agent baselines recovers both endpoints for more than 23.1% of the pipelines, compared with 49.5% for PID-Agents (McNemar test, p < 0.001).

AuthorsOpenReview ↗

Response to Reviewer F6Hv (2/4)

Agent's Reasoning vs. Specialized Tools (W2)

R2 We ran two new analyses to separate the agent's reasoning from the specialized tools.

Same tools without the agent's reasoning. The Rule-based Controller in R1 uses exactly the same tools as PID-Agents, with a fixed rule in place of the VLM. It recovers both endpoints for only 15.1% of the pipelines, compared with 49.5% for PID-Agents. This is even below VLM + Perception + RAG (25.5%), which uses no drawing tools. The specialized tools alone do not produce the gains.

Detailed tool ablation. We evaluated ten combinations of the four drawing tools on a subset of the evaluation set with 64 pipelines and 193 instruments (revised Section 4.4, new Table 4, all fields in Table 11). The OCR and detection outputs and the VLM stay fixed, and only the tool list changes.

ROIOCRTraceProbeFromToBothPipe. Avg.Inst. Avg.
✓0.48440.59380.37500.86060.8745‾\underline{0.8745}
✓0.39060.40620.26560.83890.7369
✓0.29690.29690.14060.82330.7444
✓0.21880.21880.09380.81130.7340
✓✓0.54690.62500.43750.86780.8756
✓✓0.51560.46880.37500.85340.7651
✓✓0.29690.29690.18750.82330.7651
✓✓✓0.73440.68750.59380.88700.8595
✓✓✓0.57810.53120.45310.86300.7749
✓✓✓✓0.7188‾\underline{0.7188}0.6719‾\underline{0.6719}0.5625‾\underline{0.5625}0.8846‾\underline{0.8846}0.8756

ROI, OCR, Trace, and Probe denote ROI Rendering, OCR-Based Search, Event-Driven Line Tracing, and Connectivity Probing. Line Search is available only when ROI, OCR, and Trace are all enabled. Bold marks the best result, and underline marks the second best. The ablation supports three conclusions.

  • No single tool explains the gain. With one tool, the agent recovers both endpoints for only 9.4% to 37.5% of the pipelines.
  • The tools are complementary. ROI Rendering and OCR-Based Search together reach 43.8%, and adding Event-Driven Line Tracing with Line Search raises this to 59.4%.
  • Tracing helps most when the agent can inspect what the tracer reports. Without ROI Rendering, adding tracing to OCR-Based Search raises Both from 26.6% to 37.5%. With ROI Rendering, the agent can view each junction and endpoint that the tracer reports, and adding tracing with Line Search raises Both from 43.8% to 59.4%.

Together with the rule-based controller, these results show that the gains come from the agent's reasoning over complementary tools, not from any tool alone. Removing Connectivity Probing changes Both by only two of the 64 pipelines, because the agent calls it about 0.2 times per pipeline. Repeated runs show that this difference is within run-to-run variation (new Appendix B.4). For instrument listing, ROI Rendering alone already gives most of the gain. This is expected, because an instrument usually connects to a nearby pipeline, whereas a pipeline endpoint can lie far from its line label.

AuthorsOpenReview ↗

Response to Reviewer F6Hv (3/4)

Evaluation Scope, Full From-To Connection Accuracy, and Detection Errors (W3, Q2)

R3 We address each part of this concern below.

Evaluation scope. Our dataset contains approximately 150 P&ID sheets from a large petrochemical plant project. A full listing with PID-Agents takes about 48.5 minutes per sheet (Table 5), and every compared configuration needs its own runs. We therefore fixed a subset of evaluation sheets within our experimental budget and annotated them in detail. We evaluate every verified pipeline connection and instrument tag on these sheets, which gives 212 pipelines and 622 instruments (revised Section 4.1). The selected sheets follow one connected process flow and cover diverse equipment types, including drums, reactors, a separator, a column, and a pump package. Their pipeline density is also typical, with a median of 19 pipelines per sheet against 17 over all sheets with ground truth. Industrial P&IDs with listing-level ground truth are scarce, because annotation is expensive and often constrained by confidentiality. Although the set is modest, the gap between PID-Agents and every baseline on Both is statistically significant (McNemar test, p < 0.001).

Generalization to other projects (Q2). We agree that a single cross-project setting is a limitation, and Appendix H states it. Three points still support generalization. First, the OCR and symbol detection models are trained on Project A and tested on Project B, which differ substantially in plotting configurations, symbol styles, and scan artifacts. Second, the agent stage uses the VLM without fine-tuning and reads project-specific conventions from the project legend and specification documents at run time. Third, Appendix G shows two execution traces on open-source OPEN100 drawings, which come from a different source.

Full From-To connection accuracy. The table below reports how often both endpoints are correct, under strict and partial matching of Table 1 (new Appendix C, Table 8).

MethodBoth (strict)Both (partial)
VLM baseline0.06600.2642
VLM + Perception0.08960.2453
VLM + Perception + RAG0.11790.2547
Rule-based Controller0.08490.1509
Pixel-Graph Search0.09430.2311
Rule-based Topology0.03770.1226
PID-Agents0.20750.4953

PID-Agents is the most accurate method under both criteria. No other method exceeds 11.8% and 26.4%, respectively. Appendix C defines both criteria.

Errors caused by missed/incorrect detections. All methods that use perception share the Stage 1 outputs, so detection errors do not favor PID-Agents. Three of the 212 pipelines are missing from the pipeline list that Stage 1 produces, and each of them continues across two adjacent sheets. A missed pipeline counts as wrong in every field (new Appendix C). For instruments, VLM + Perception + RAG and PID-Agents recover almost the same tags drawn on the sheets (467 and 468 of 622). Table 2 and Table 7 report the OCR and symbol detection accuracy.

Graph-level consistency. We agree that the framework reasons at the item level and does not yet enforce graph-level consistency across related pipelines and instruments. The Limitations state this. Two of our metrics capture part of it. Both requires the two endpoints of a pipeline to be correct together. The instrument fields Fluid, Line Size, and Line Spec are correct only when an instrument is linked to the right pipeline.

Runtime, Tool Calls, Computational Cost, and Scalability (W4, Q3)

R4 The revision now reports the runtime, number of tool calls, and computational cost (new Section 4.5, Table 5).

UnitLLM callsTool callsCost (USD)Time
Pipeline (From/To)25.013.30.206215 s
Instrument (linked pipeline, I/O)23.37.40.218206 s
Sheet (full listing)2,63395324.5948.5 min

On average, one pipeline identifier takes 13.3 tool calls, 25.0 model calls, USD 0.206, and 215 seconds. A full sheet with about 25 pipelines and 81 instruments takes 48.5 minutes and USD 24.59 with about 10 parallel tasks. The instrument listing dominates the cost.

Scalability. Each pipeline identifier or instrument tag is an independent task with bounded retries and a timeout. The cost therefore grows linearly with the number of items. The time per sheet falls as more tasks run in parallel as they are also independent and orthogonal to the project size.

Comparison with traditional non-agent approaches. The non-agent methods in R1 make no model calls, so their computational cost is negligible. However, none of them recovers both endpoints for more than 23.1% of the pipelines, compared with 49.5% for PID-Agents, and they produce no specification attributes. Plus, PID-Agents stores the ROI coordinates and tool observations behind each field, which supports efficient human verification.

Reviewer_toFsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PID-AGENTS, a tool-augmented multimodal framework for converting rasterized piping and instrumentation diagrams (P&IDs) into structured pipeline and instrument listings. The system consists of two stages. First, OCR, symbol detection, and retrieval from project specifications are used to identify engineering entities and initialize listing attributes. Second, a vision-language agent uses drawing-specific tools, including ROI rendering, event-driven line tracing, and connectivity probing, to recover pipeline endpoints and instrument-to-pipeline relationships. On a cross-project evaluation over industrial P&IDs, the full system raises From/To accuracy from 0.46/0.42 to 0.63/0.67 and the instrument-listing average from 0.49 to 0.78, compared with a Gemini 3 Flash baseline augmented with perception and retrieval.

Critical strengths
  • The tool design reflects a sensible division of labor. The tracer returns discrete events such as junctions and endpoints instead of asking the VLM to read geometry from pixels. The model is used mainly where semantic judgment is needed. The option to store provenance for each inferred field is also a good choice for auditability.
Critical weaknesses
  • The paper does not show whether the improvement comes from the added tools or from the decisions made by the multimodal agent. Going from VLM + Perception + RAG to PID-AGENTS adds the drawing tools and agentic control at the same time. Table 1 shows that the specification fields are essentially unchanged between the two, so in pipeline listing the entire gain sits in the From/To fields, which is exactly where the tools act. In instrument listing, gains appear across all fields, including Tag Loc. (0.75 → 0.92), which may partly come from the symbol-anchored tag extraction in Stage 1 (Appendix B.1) rather than from the agent. Without a baseline that uses the same tools under a fixed rule-based controller (for example, trace from the line label and take the first equipment or connector hit), it is unclear how much the VLM’s choices actually contribute. Tool-level ablations would also help, such as removing tracing, probing, or ROI inspection.

  • The related-work discussion should map each tool to the closest prior methods, so that the real contribution of the tool design becomes clear. Event-driven tracing and connectivity probing are close to the pixel-graph traversal of Mani et al. (2020) and to the rule-based topology reconstruction of Kim et al. (2022). Kim et al. already outputs DEXPI-format P&IDs and reports high topology precision and recall, yet the paper mentions it in only one sentence. ROI rendering is essentially a zoom-in operation, related to active-perception agents for high-resolution images such as V* and DeepEyes. There is also recent MLLM-based P&ID work that is not discussed: Zhu et al. (2026) separates visual extraction from topology reasoning in a two-stage MLLM workflow, and arXiv:2409.00082 uses ReAct-style tool-calling agents for P&ID analysis. The “first tool-augmented multimodal agent” claim should be qualified in light of these.

  • The experimental analysis could ask more valuable questions about the system as a whole. First, endpoint accuracy does not establish that complete pipelines are understood. From and To are scored separately with partial matching. With 63.21% and 66.51% on 212 lines, the fraction with both endpoints correct could be anywhere between about 30% and 63%, so exact pair accuracy should be reported. Moreover, each sheet is processed independently, off-page connectors are treated as endpoints rather than being stitched across sheets, and no global consistency is enforced (Appendix E). If connections cannot be chained across drawings, claims such as “complete understanding” and “reliable end-to-end interpretation” should be narrowed. Finally, Section 4 is a little unbalanced: the setup is long, while Q3 mainly re-describes Table 1 without new experiments. This space would be better spent on the analyses above.

Questions for authors
  • Table 1 introduces drawing tools and agent control together. With identical Stage 1 outputs and drawing tools, how much does VLM control improve accuracy over a fixed rule-based controller?

  • Which technical advances distinguish your tracing and probing tools from prior P&ID topology-recovery methods? How does PID-AGENTS differ from prior multimodal agents, and what scope supports the “first tool-augmented multimodal agent” claim?

  • Section 4.1 evaluates From and To separately using partial matching. What fraction of the 212 evaluated pipelines has both endpoints exactly correct? Please define the partial-matching criteria and report exact endpoint-pair accuracy for PID-AGENTS and the strongest baseline.

3 comments
AuthorsOpenReview ↗

Response to Reviewer toFs (1/3)

We thank the reviewer for recognizing the sensible division of labor in our tool design, in which the tracer returns discrete events and the VLM makes the semantic judgments, and the choice to store provenance for auditability. We address each concern below with new experiments and revisions, which are highlighted in blue in the revised paper.

Improvement from the Tools vs. the Multimodal Agent's Decisions (W1, Q1)

R1 We have added the baseline that the reviewer suggested (revised Section 4.4, new Table 3). The Rule-based Controller uses the same Stage 1 outputs and the same drawing tools as PID-Agents, with a fixed rule in place of the VLM. It traces from the line label, takes the first equipment or connector hit on each side, and probes when a trace stops at a gap.

MethodFromToBoth
VLM + Perception + RAG (no drawing tools)0.45750.41510.2547
Rule-based Controller (drawing tools, fixed rule)0.30190.41040.1509
PID-Agents (drawing tools, VLM control)0.63210.66510.4953

Both means that From and To are both correct. With identical Stage 1 outputs and drawing tools, VLM control improves From, To, and Both by 33.0, 25.5, and 34.4 points over the fixed rule-based controller. The rule-based controller even falls below VLM + Perception + RAG, which uses no drawing tools. PID-Agents is significantly more accurate than both baselines on Both (McNemar test, p < 0.001). The tools alone therefore do not produce the gain. The gain appears when the VLM controls the tools. At a T-junction or an inline symbol, the right next step depends on flow arrows, labels, and equipment context. A fixed rule cannot weigh these cues, but the VLM can, as the execution traces in Appendix G show.

We also ran the tool-level ablation that the reviewer suggested. It covers ten combinations of the four drawing tools on a subset of the evaluation set with 64 pipelines and 193 instruments, and only the tool list changes (revised Section 4.4, new Table 4, all fields in Table 11).

ROIOCRTraceProbeFromToBothPipe. Avg.TagInst. Avg.
✓0.48440.59380.37500.86060.98450.8745‾\underline{0.8745}
✓0.39060.40620.26560.83890.89640.7369
✓0.29690.29690.14060.82330.89640.7444
✓0.21880.21880.09380.81130.89640.7340
✓✓0.54690.62500.43750.86780.98450.8756
✓✓0.51560.46880.37500.85340.90160.7651
✓✓0.29690.29690.18750.82330.90160.7651
✓✓✓0.73440.68750.59380.88700.9637‾\underline{0.9637}0.8595
✓✓✓0.57810.53120.45310.86300.89640.7749
✓✓✓✓0.7188‾\underline{0.7188}0.6719‾\underline{0.6719}0.5625‾\underline{0.5625}0.8846‾\underline{0.8846}0.98450.8756

ROI, OCR, Trace, and Probe denote ROI Rendering, OCR-Based Search, Event-Driven Line Tracing, and Connectivity Probing. Line Search is available only when ROI, OCR, and Trace are all enabled. Bold marks the best result, and underline marks the second best.

  • Removing ROI inspection lowers Both from 56.3% to 45.3% and the instrument average from 0.88 to 0.77. ROI Rendering is the most effective tool.
  • Removing tracing leaves ROI Rendering and OCR-Based Search, which reach 43.8%. Adding Event-Driven Line Tracing with Line Search raises Both to 59.4%.
  • Removing probing changes Both by only two of the 64 pipelines (59.4% vs. 56.3%). The agent calls Connectivity Probing about 0.2 times per pipeline. On the pipelines where it was called, repeated runs show that this difference is within run-to-run variation (new Appendix B.4).

Regarding Tag Loc. (the Tag column of Table 1, 0.75 → 0.92), the gain does not come from the symbol-anchored tag extraction in Stage 1. VLM + Perception + RAG and PID-Agents both receive the Stage 1 outputs, and they recover almost the same tags drawn on the sheets (467 and 468 of 622). The gain comes from 103 instruments that have no symbol of their own on the sheets but are defined in the project legend (revised Appendix B.1). Of these, 54 are high and low alarms, which the legend marks with an H or L next to an instrument bubble. The other 49 are implied by simplified assemblies that the legend defines, such as the flow element of a flow transmitter. PID-Agents zooms into the legend sheet with ROI Rendering, finds both kinds of definitions, and adds these instruments on every sheet. The tool ablation is consistent with this. Tag accuracy is 0.96 to 0.98 whenever ROI Rendering is enabled and 0.90 otherwise.

AuthorsOpenReview ↗

Response to Reviewer toFs (2/3)

Mapping Each Tool to Prior Methods and Qualifying the "First" Claim (W2, Q2, Q3)

R2 We thank the reviewer for this suggestion. The revised related work (Section 5) now relates each tool to the closest prior methods.

Event-driven tracing and connectivity probing (Q2). The low-level operations of these tools are similar to the pixel-graph traversal of Mani et al. (2020). The technical difference lies in what the tools return and who makes the decisions. Mani et al. (2020) and the rule-based topology reconstruction of Kim et al. (2022) resolve every junction and endpoint with fixed rules. Our tools instead stop at these points and return observations to the VLM (new Appendix B.4).

  • Event-Driven Line Tracing runs on a line map in which OCR text is erased, so labels neither break a line nor create a false one. It continues through corners and four-way crossings, but it stops at a T-junction, a line end, or a filled symbol. It returns each event with its coordinates and an ROI rendering of the path, and at a T-junction it also reports the available directions.
  • Connectivity Probing spreads from the stop point and returns every contact candidate without filtering. Each candidate comes with its nearest OCR text or detected symbol and, for an arrowhead, the direction in which it points.
  • The VLM then chooses the next branch and the endpoint from these observations.

The Rule-based Controller in R1 shows why this difference matters. It runs our own tools under a fixed rule and recovers both endpoints for only 15.1% of the pipelines. The new Table 3 also compares both prior methods with PID-Agents under the same perception outputs (revised Section 4.4).

MethodFromToBoth
Pixel-Graph Search (Mani et al., 2020)0.49060.43400.2311
Rule-based Topology (Kim et al., 2022)0.31130.33490.1226
PID-Agents0.63210.66510.4953

The revised Section 5 now describes Kim et al. (2022) in more detail, including its DEXPI-based output. Kim et al. (2022) report high topology precision and recall on five drawings after recognition errors were corrected by hand. In Table 3, all methods use the same perception outputs without manual correction, which is the setting of automated listing.

ROI rendering. We agree that ROI Rendering is essentially a zoom-in operation. The revised Section 5 relates it to active-perception agents for high-resolution images, V* (Wu & Xie, 2024) and DeepEyes (Zheng et al., 2026). We do not claim the zoom itself as a contribution. Our contribution is its combination with tracing and probing tools designed for P&IDs, and the ablation in R1 shows that this combination matters. ROI Rendering alone recovers both endpoints for 37.5% of the pipelines, while adding OCR-Based Search, Line Search, and Event-Driven Line Tracing raises this to 59.4%.

MLLM-based P&ID work and the "first" claim (Q3). Zhu et al. (2026) infer topology from extracted tags in a two-stage MLLM workflow. Sakhinana et al. (2024) use ReAct-style tool-calling agents that retrieve documents to answer questions. Neither work applies geometric tools to the drawing, and neither produces listings. General tool-using agents and active-perception agents also have no tools for tracing P&ID topology. We have qualified the claim in light of these studies. The revised abstract now states it for P&ID listing (Definition 1), which converts P&IDs into structured records of identifiers, connectivity, and engineering attributes. To the best of our knowledge, PID-Agents is the first tool-augmented multimodal agent framework for this task.

AuthorsOpenReview ↗

Response to Reviewer toFs (3/3)

Complete Pipeline Accuracy, Matching Rules, and Scope of Claims (W3, Q4)

R3 The table below reports performance with both endpoints correct, under strict matching and under the partial matching of Table 1 (new Appendix C, Table 8).

MethodBoth (strict)Both (partial)
VLM baseline0.06600.2642
VLM + Perception0.08960.2453
VLM + Perception + RAG0.11790.2547
Rule-based Controller0.08490.1509
Pixel-Graph Search0.09430.2311
Rule-based Topology0.03770.1226
PID-Agents0.20750.4953

PID-Agents gets both endpoints exactly right for 20.8% of the pipelines. The strongest baseline under strict matching, VLM + Perception + RAG, reaches 11.8%. Under partial matching, PID-Agents reaches 49.5%, and the strongest baseline, the VLM baseline, reaches 26.4%. This value lies within the 30% to 63% range that the reviewer derived and closer to its upper end. PID-Agents is significantly more accurate than every baseline under both criteria (McNemar test, p < 0.05 for strict and p < 0.001 for partial matching).

Partial-matching criteria. The ground truth for From and To comes from the project line list, which is compiled by hand. The list often names an endpoint differently from the label on the drawing. For example, it records the group name "P-101A/B" for a line that starts at pump P-101B alone. We therefore use partial matching so that such differences in notation are not counted as errors. Both strings are first uppercased, and runs of whitespace are collapsed. A predicted endpoint is then correct if it equals the ground truth, if one string contains the other, if the two share an equipment tag after a group such as "P-101A/B" is expanded, or if both contain the header token "HDR". Strict matching instead requires the first predicted endpoint to equal the ground truth exactly after uppercasing and trimming spaces. A pipeline counts under Both only when From and To each match their own ground-truth field.

Cross-sheet connectivity and scope of claims. We agree with the reviewer and our Limitations (Appendix H) state this explicitly. The introduction now describes the problem as P&ID listing instead of the complete understanding problem. We will also revise the last sentence of the abstract to "enabling end-to-end P&ID listing of real industrial drawings". We keep "end-to-end" only in the sense that the system turns rasterized P&IDs into listings with no manual step in between.

Balance of Section 4. We followed this suggestion. We shortened Q3 and added two more valuable questions. Q4 compares PID-Agents with non-agent methods, and Q5 measures how much each drawing tool contributes (revised Section 4.4). The new Section 4.5 reports the computational cost.

We hope that these results address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary to All Reviewers

We thank all reviewers for their thoughtful and constructive feedback. We appreciate the recognition of the practical value of the problem (F6Hv, Wm5h, LxwS), the division of labor between the drawing tools and the VLM (toFs, Wm5h, LxwS), the completeness of the framework (Wm5h), the provenance for auditability (toFs), the cross-project evaluation on real industrial drawings (Wm5h, LxwS), and the improvements on connectivity-related fields (F6Hv, Wm5h, LxwS).

The revised paper highlights all changes in blue\color{blue}{\textbf{blue}} and adds the following results and clarifications.

  1. Non-agent baselines that use the same OCR and symbol detection results. They include a rule-based controller with our drawing tools and two existing connectivity methods (toFs, F6Hv, Wm5h). See revised Section 4.4, new Table 3, and Table 10.
  2. Tool ablation over ten combinations of the four drawing tools, together with the detailed tool rules (toFs, F6Hv, Wm5h, LxwS). See revised Section 4.4, new Table 4, Table 11, and Appendix B.4.
  3. Accuracy of complete From-To connections under strict and partial matching, with the matching rules and the scoring of incomplete records (toFs, F6Hv, Wm5h, LxwS). See new Appendix C and Table 8.
  4. Runtime, number of tool calls, and computational cost (F6Hv, Wm5h). See new Section 4.5 and Table 5.
  5. Results with a stronger backbone, Claude Opus 5.5 (F6Hv). See new Appendix D and Table 9.
  6. Related work on P&ID topology recovery, active-perception agents, and VLM-based P&ID work, with the "first" claim qualified to P&ID listing (toFs). See revised Section 5 and the abstract. For the same reason, we changed the title from "…Diagram Understanding" to "…Diagram Listing".
  7. Source of the Tag gain, the evaluation subset, and the limitations (toFs, F6Hv, Wm5h). See revised Appendix B.1, Section 4.1, and Appendix H.

The new results separate the agent's decisions from its tools. All methods below use the same OCR and symbol detection results. Both means that From and To are both correct.

MethodOur drawing toolsVLM decisionsBoth (strict)Both (partial)
Pixel-Graph Search (Mani et al., 2020)0.09430.2311
Rule-based Topology (Kim et al., 2022)0.03770.1226
Rule-based Controller✓0.08490.1509
VLM + Perception + RAG✓0.11790.2547
PID-Agents✓✓0.20750.4953

Our drawing tools under a fixed rule recover both endpoints for only 15.1% of the pipelines, and the VLM without the tools reaches 25.5%. PID-Agents, in which the VLM controls the tools, reaches 49.5%. It is significantly more accurate than every baseline (McNemar test, p < 0.001 under partial matching). The tool ablation also shows the benefit of the topology tools over general local inspection. With only ROI Rendering and OCR-Based Search, the agent recovers both endpoints for 43.8% of the pipelines. Adding Line Search and Event-Driven Line Tracing raises this to 59.4% with only 2.3 more model calls per pipeline.

To the best of our knowledge, PID-Agents is the first tool-augmented multimodal agent framework for P&ID listing. P&ID listing converts P&IDs into structured listings that contain identifiers, connectivity, and engineering attributes. Existing connectivity methods resolve junctions and endpoints with fixed rules, and VLM-based P&ID work does not apply geometric tools to the drawing. In PID-Agents, the tools trace the drawing and stop at each junction and endpoint, and the VLM makes the decision there.

We believe that the rebuttal and the revised paper now communicate our contributions more clearly, and we hope that they address the remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.