acceptodds
Under review as a conference paper at ICLR 2027

RT-ISTR: A Real‑Time Instance Segmentation Transformer Network for Microscopic Image

Abstract

Real-time instance segmentation in microscopic imaging is hindered by the trade-off between pixel-wise accuracy and computational efficiency. Existing Transformer-based models deliver high precision but suffer from prohibitive parameter counts and latency, while CNN-based alternatives sacrifice boundary quality. We propose RT-ISTR, an end-to-end Transformer framework that unifies detection and segmentation with minimal overhead. Our lightweight Multi-scale Mask Enhancement Module fuses shallow spatial details from the backbone with deep semantic features via hierarchical upsampling and element-wise addition, producing rich prototype masks. The detection branch's query embeddings are reused to dynamically query these prototypes through dot-product, enabling pixel-level masks at negligible extra cost. On COCO, RT-ISTR-L achieves 44.6% \(AP^mask\) with 102 GFLOPs and 32M parameters, running at 64 FPS, outperforming YOLOv8, YOLO11, and YOLO26 in mask accuracy. RT-ISTR challenges the paradigm that high-accuracy segmentation demands heavy models, offering an efficient, deployable solution for real-time microscopic analysis and edge-based applications.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Microscopic Cell Segmentation, Multi-scale Fusion, Prototype Mask, Real-Time Instance Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WntyOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

RT-ISTR extends the D-FINE detector to instance segmentation by introducing a lightweight Multi-scale Mask Enhancement module and reusing detection query embeddings for mask prediction. The module combines shallow backbone features with encoder features through upsampling and addition, and masks are generated through a dot product between query-derived embeddings and prototype masks. Experiments on COCO and a self-built Traditional Chinese Medicine microscopy dataset report competitive segmentation accuracy with relatively low computational cost. However, the evaluation does not convincingly establish the robustness of the gains, their generality to microscopy, or the claimed real-time deployment benefit.

Critical strengths

A straightforward and potentially useful engineering design. Reusing D-FINE’s detection queries and adding a lightweight mask-generation branch is a practical way to extend a strong detector to instance segmentation. Sharing the backbone, encoder, and decoder could make the approach attractive when detection and segmentation are both required.

Promising computational efficiency at some model scales. The reported results include configurations with competitive mask accuracy and substantially fewer FLOPs than relevant baselines. For example, RT-ISTR-M achieves 43.2 mask AP on COCO with 66.8 GFLOPs, compared with YOLO26-M’s 42.1 AP and 121.5 GFLOPs. These results suggest potential value, provided the comparisons and actual runtime benefits are validated.

Critical weaknesses

The microscopy evaluation is too limited and insufficiently documented. The central motivation concerns microscopic images with ambiguous boundaries, clutter, and overlapping instances, but the microscopy evidence comes exclusively from the self-built TCM dataset. COCO provides useful general-purpose evaluation but does not establish generalization across microscopy tasks or acquisition conditions.

Essential TCM dataset information is missing, including image and instance counts, category composition, annotation procedures, and train/validation/test split construction. It is therefore difficult to assess the dataset’s difficulty, the independence of the evaluation samples, or the reproducibility of the results. Furthermore, aggregate AP and selected visualizations do not directly substantiate the claimed improvements in boundary fidelity or separation of densely overlapping instances. Validation on an independent microscopy benchmark would materially strengthen the application-specific contribution.

The magnitude and reliability of the accuracy–speed improvement are not convincingly established. On TCM, RT-ISTR-L improves mask AP from YOLOv8-L’s 80.3 to 82.1, a gain of 1.8 percentage points. On COCO, its advantage over YOLO26-L is only 0.1 AP, while RT-ISTR-X achieves lower mask AP than YOLO26-X, 45.7 versus 47.0. Reduced FLOPs are a positive result, but these comparisons support an accuracy–compute trade-off more clearly than consistent accuracy superiority. Without repeated runs or uncertainty estimates, the robustness of the smaller gains remains unknown.

More importantly, the runtime reporting is inconsistent: the abstract and conclusion state 64 FPS for RT-ISTR-L, whereas Table 1 and Section 3.2.1 report 23 FPS. The inference protocol is insufficiently specified to establish a fair latency comparison, and no deployment results substantiate the claims about resource-constrained edge applications. Since real-time performance is a central contribution, this discrepancy directly affects the assessment of usefulness.

The ablation also compares MME with a shallow-feature baseline rather than a clearly defined, competitive multiscale mask head. This does not adequately isolate the benefit of the proposed fusion design from the general benefit of adding semantic features.

Questions for authors

Dataset and evaluation clarification: Please provide the TCM dataset size, categories, instance counts, annotation process, and split construction. Are images from the same specimen or acquisition session kept within one split? Were hyperparameters selected on a separate validation set?

Robustness of the gains: Do the reported improvements persist across independent training runs under matched training, pretraining, and input-resolution settings? In particular, how should the 1.8-point TCM improvement and 0.1-point COCO improvement be interpreted relative to run-to-run variability?

Runtime and practical usefulness: Please reconcile the 64 FPS and 23 FPS results. What hardware, batch size, numerical precision, inference implementation, and timing boundaries were used? A matched batch-one, end-to-end latency comparison on microscopic images would materially affect my assessment of the real-time contribution.

Ablation clarification: What exactly is retained in the baseline without MME, and why does its mask AP deteriorate so substantially? How does MME compare with a conventional multiscale mask head added to the same D-FINE backbone and decoder under a comparable computational budget?

Reviewer_XovUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds an instance segmentation branch to the D-FINE real-time detector. A Multi-scale Mask Enhancement (MME) module fuses the stride-4 backbone feature S2 with the encoder outputs P3–P5 using 1×1 convolutions, bilinear upsampling and element-wise addition to produce a stride-4 prototype map with d_p channels. The decoder's detection queries are passed through a single linear layer to obtain mask coefficients, and masks are produced by a dot product with the prototypes, in the manner of YOLACT. Training adds per-pixel focal and dice losses to the D-FINE objective. The method is evaluated at five scales (N–X) on COCO and on a self-built traditional Chinese medicine (TCM) microscopy dataset. RT-ISTR-L reports 44.6 mask AP at 102 GFLOPs / 32M parameters on COCO and 82.1 mask AP on TCM. Ablations, all on TCM, cover MME vs. S2-only.

Critical strengths
  1. The segmentation head adds little compute and the results hold up across scales. Adding a prototype map and one linear layer to D-FINE raises compute from 91 to 102 GFLOPs at the L scale, and the resulting mask AP tracks or slightly exceeds YOLO-seg at matched GFLOPs from N through L (Tables 1, 5, 6); the M-scale gap is the clearest case, 43.2 vs. 42.1 at roughly half the FLOPs of YOLO26-M. This is a credible operating point for a DETR-based real-time segmenter.
  2. The paper reports a complete N-to-X family rather than a single tuned configuration, and the TCM results are consistent with the COCO results: every variant exceeds the re-trained YOLO, Mask R-CNN and YOLACT baselines, with the largest gains at AP75, the metric most sensitive to boundary quality. The dataset is undocumented, which limits how far this can be interpreted, but the pattern across scales and metrics is coherent.
Critical weaknesses
  1. The reported numbers do not agree with each other, including the headline speed. The abstract and conclusion state 64 FPS for RT-ISTR-L; Table 1 and Sec. 3.2.1 state 23 FPS. D-FINE-L is listed at 125 FPS and 91 GFLOPs, so an extra 11 GFLOPs producing a 5× slowdown needs an explanation, or one of these figures is wrong. Further mismatches: Sec. 3.2.2 gives AP75mask = 92.2 while Table 2 gives 94.1; App. B.4 gives RT-ISTR-S 46.6 box / 39.5 mask / 42.4 AP75 while Table 5 gives 45.7 / 38.8 / 41.6; Sec. 3.3 says the S2-only variant reaches 34.9 mask AP, but the gap annotated in Fig. 3(a) (38.2 below 82.1) implies 43.9. The re-measured FLOPs for Cascade Mask R-CNN and HTC (1787 and 1735 GFLOPs) are several times what is normally reported for these models at 1333×800 and are inconsistent with the 207 GFLOPs listed for Mask R-CNN in the row above. Mask-RT-DETR-X is discussed in the text but appears in no table. Since the contribution is an accuracy/efficiency trade-off, not knowing which numbers to trust goes directly to soundness.

  2. The novelty is modest and the ablations do not test the design arguments the paper makes. The segmentation head is YOLACT's prototype-plus-coefficient scheme with a linear coefficient layer; MME is a semantic-FPN-style top-down sum. Attaching this to D-FINE is reasonable engineering, but it is presented as a new paradigm. Sec. 2.4 argues for two-stage progressive fusion over one-step upsampling, and Sec. 2.5 argues at length that reused detection queries suffice and that a linear projection is enough; none of this is ablated. The only architectural ablation is S2-only vs. full MME, on TCM only, with no COCO ablations at all. The 1.2 box AP drop from D-FINE-L (54.0) to RT-ISTR-L (52.8) is not discussed. At the L/X scales the comparison is also weaker than the abstract suggests: 44.6 vs. 44.5 for YOLO26-L is a tie, and RT-ISTR-X (45.7) is behind YOLO26-X (47.0). Larger RF-DETR-Seg variants are left out of Table 1 even though smaller ones appear in Table 5.

Questions for authors
  1. Which FPS for RT-ISTR-L is correct, 23 or 64? Please state hardware, precision, batch size and whether TensorRT was used, and report FPS for the YOLO baselines under the same protocol.

  2. Please reconcile the discrepancies listed above and explain how the ⋆ FLOPs for Cascade Mask R-CNN and HTC were measured (input size, tool).

  3. Is the "test split of COCO" val2017 or test-dev? Are the baseline numbers taken from the same split and the same input resolution as yours?

  4. Why does box AP fall by 1.2 relative to D-FINE-L? Is this due to the shared queries, the loss weighting, or a different schedule?

  5. TCM dataset: number of images, number of classes, instances per class, image resolution, train/val/test split, annotation protocol, and whether it will be released. Were the baselines trained with their default recipes and comparable epochs? YOLO26 scoring below YOLOv8 on TCM (Table 2) suggests the baseline tuning may be uneven.

Reviewer_XP7dOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets real-time instance segmentation of microscopic images, aiming to balance pixel accuracy and computational efficiency. Built on the D-FINE detection framework, the authors propose RT-ISTR, which consists of two main components: a Multi-scale Mask Enhancement (MME) module and query embeddings reused from the detection branch. The method is evaluated on COCO and a self-collected Chinese-medicine microscopic cell dataset.

Critical strengths

The paper has a clear problem formulation and an intuitive architecture, and achieves good results on the microscopic cell dataset.

Critical weaknesses

There are notable issues with novelty, consistency of the experimental results, and the support provided for the core claims.

Questions for authors
  • Limited novelty. The method mainly combines multi-scale feature fusion and reused detection queries to perform instance segmentation. The overall design is rather direct, and the substantive technical contribution over existing methods is not clearly articulated. In particular, the authors need to better clarify how MME differs from existing feature-fusion schemes, and what advantages reusing detection queries brings over existing mask-prediction mechanisms.

  • Inconsistencies in the reported experimental results. The abstract and conclusion claim that RT-ISTR-L reaches 64 FPS on COCO, while Table 1 reports 23 FPS. The authors should clarify the numbers and explain the discrepancy. In addition, Fig. 3(a) is inconsistent with the ablation results reported in the main text: the text says mask AP increases from 34.9 to 82.1, whereas the figure annotates a gain of 38.2 percentage points, which would imply a baseline of 43.9. The authors should double-check the relevant numbers.

  • The current data does not directly support the claim of higher real-world inference efficiency. According to Table 1, RT-ISTR-L adds computation and parameters over D-FINE-L, and its inference speed is noticeably lower. While the original D-FINE has no instance-segmentation capability, so the missing mask metrics are understandable, the authors do not adequately explain whether the two are evaluated under the same hardware and inference settings. The current results are therefore insufficient to support the claim that the additional computational overhead is negligible.

  • Ablation settings are unclear and cannot fully verify the effectiveness of the module design. Fig. 3(a) compares using only the shallow feature S2S_2 with using MME, which can preliminarily show that multi-scale feature fusion helps improve segmentation performance. However, the authors have not sufficiently verified the role of the hierarchical-fusion design and its key components, nor provided direct validation of the core design of reusing detection queries. The authors should clarify the ablation setup and add the necessary comparisons.

  • Dataset evaluation settings and performance claims need further clarification. The COCO results are described as coming from the test split, but the specific data version and evaluation split are not specified. In addition, RT-ISTR's lead on COCO is not always clear: RT-ISTR-L's mask AP is only slightly higher than YOLO26-L, while RT-ISTR-X's mask AP is lower than YOLO26-X. The results on the microscopic cell dataset in Table 2 are more favorable, but the dataset's scale, split, and annotation details are not adequately described, so the strong claims about the method's general advantage and its applicability to microscopic images are not sufficiently supported. The authors should provide a detailed explanation.

  • The current manuscript has several issues in writing and details. For example, citation formats are inconsistent in places, several key experimental settings are unclear, and the performance numbers in the abstract, main text, and figures need to be reconciled.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.