Hierarchical Point-Image Fusion with Context Enhancement for Panoptic Symbol Spotting in CAD Drawings

Panoptic symbol spotting in computer-aided design (CAD) drawings jointly identifies countable object instances and assigns semantic labels to all graphical primitives, including those representing uncountable stuff. Reliable recognition depends on both the geometry of individual primitives and the spatial arrangement of surrounding strokes, with these cues encoded differently in vector and raster representations. We propose a point–image fusion framework that combines bidirectional backbone interaction with local context enhancement before primitive fusion, bringing complementary visual context overlooked by vector-only methods into primitive representation learning. Specifically, we introduce Hierarchical Point-Image Fusion (HPIF) to enable spatially aligned exchange of local geometric and visual cues, enriching point features with multiscale visual context from the image branch and image features with geometric information from vector primitives. To strengthen each primitive's representation with surrounding geometric and visual evidence, our Primitive-Oriented Context Enhancement (POCE) module combines radius-bounded attention over same-layer primitives with regular and shifted window attention on image features, further improving panoptic recognition. Extensive experiments demonstrate that our framework outperforms previous methods on FloorPlanCAD and ArchCAD-Public, achieving state-of-the-art panoptic quality (PQ) of 92.7 and 90.6, respectively. Code is available at http://anonymous.4open.science/r/code-BED2.