Tool-Augmented Multimodal Agents for Piping & Instrumentation Diagram Understanding

Piping and instrumentation diagrams (P&IDs) are key engineering documents that specify equipment, instrumentation, and piping connectivity in complex industrial systems. Despite their central role in engineering, procurement, and construction workflows, P&IDs remain largely interpreted manually due to dense layouts, heterogeneous drafting standards, and project-specific conventions. Interpreting these diagrams requires not only symbol detection but also reliable reasoning about line connectivity, topology, and semantic attributes; capabilities that existing approaches struggle to achieve. We introduce PID-Agents, the first tool-augmented multimodal agent framework for structured P&ID understanding. Rather than relying solely on static visual recognition, PID-Agents enables vision-language agents to actively explore diagrams through a set of novel domain-specific drawing tools, including region-of-interest rendering, event-driven line tracing, and local connectivity probing. Through iterative tool use, the agent gathers visual evidence and reconstructs pipeline-level relations and attributes beyond object detection. To support specification-driven reasoning, we further integrate retrieval-augmented specification, grounding attribute inference and consistency checks in external engineering documents. The resulting system converts P&IDs (rasterized as vector pdf) into connectivity-centric structured representations consisting of entities, typed relations, and provenance, enabling reliable end-to-end interpretation of real industrial drawings.