RT-ISTR: A Real‑Time Instance Segmentation Transformer Network for Microscopic Image

Real-time instance segmentation in microscopic imaging is hindered by the trade-off between pixel-wise accuracy and computational efficiency. Existing Transformer-based models deliver high precision but suffer from prohibitive parameter counts and latency, while CNN-based alternatives sacrifice boundary quality. We propose RT-ISTR, an end-to-end Transformer framework that unifies detection and segmentation with minimal overhead. Our lightweight Multi-scale Mask Enhancement Module fuses shallow spatial details from the backbone with deep semantic features via hierarchical upsampling and element-wise addition, producing rich prototype masks. The detection branch's query embeddings are reused to dynamically query these prototypes through dot-product, enabling pixel-level masks at negligible extra cost. On COCO, RT-ISTR-L achieves 44.6% \(AP^mask\) with 102 GFLOPs and 32M parameters, running at 64 FPS, outperforming YOLOv8, YOLO11, and YOLO26 in mask accuracy. RT-ISTR challenges the paradigm that high-accuracy segmentation demands heavy models, offering an efficient, deployable solution for real-time microscopic analysis and edge-based applications.