acceptodds
Under review as a conference paper at ICLR 2027

ProtoTwinMamba: Prototype-Guided Spatial Ordering and Temporal Modeling for Video Polyp Segmentation

Abstract

Video polyp segmentation remains challenging due to foreground–background ambiguity and inter-frame appearance variations. In recurrent spatial modeling, token order shapes how contextual information propagates, yet fixed grid traversals do not explicitly account for foreground and background relevance. Motivated by this observation, we propose ProtoTwinMamba, which uses region guidance not only to refine features but also to organize their spatial processing order. Specifically, a coarse prediction provides soft pooling weights for constructing frame-wise foreground and background prototypes. Prototype similarities are combined with the coarse prior to produce region-specific guidance maps, shared by masked linear attention for feature refinement and two independently parameterized Mamba branches for spatial scan ordering. Sorting tokens by their respective guidance scores groups high-relevance tokens together, forming foreground- and background-conditioned sequences without hard token partitioning. Each branch then restores the original feature grid before bidirectional temporal aggregation at fixed spatial locations. Moreover, a shallow detail-enhancement pathway preserves local detail, while a convolutional decoder fuses shallow and deep features to produce the final segmentation masks. Extensive experiments on SUN-SEG and CVC-612 demonstrate that ProtoTwinMamba achieves state-of-the-art performance on both benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Polyp Segmentation, State Space Models, Foreground–Background Modeling, Prototype Learning, Spatio temporal Modeling.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kPcNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets video polyp segmentation and argues that the token order of a recurrent spatial model should follow foreground/background relevance rather than the raster grid. ProtoTwinMamba turns a coarse mask into soft pooling weights, builds frame-wise foreground and background prototypes, and converts prototype similarity plus the coarse prior into two guidance maps; masked linear attention refines features with them and two separately parameterized Mamba branches sort the same tokens by them before scanning. Grids are restored before a bidirectional temporal scan, and a shallow detail path plus a convolutional decoder produce the masks. With PVTv2-B2 it reports the best listed scores on all four SUN-SEG subsets and beats a retrained STDDNet on a custom CVC-612 split.

Critical strengths
  1. It is cheaper than what it beats: 27.93 M parameters and 35.7 GFLOPs per frame versus 38.16 M / 75.17 for STDDNet and 82.73 M / 86.58 for SALI. Whole-model capacity therefore cannot explain the gain, which removes the first alternative explanation I would test.

  2. The ordering claim is tested the right way: FixOrd changes only the scan order and costs 2.6 and 2.2 Dice on the two unseen splits, with Figure 9 showing the permutations. The paper also states plainly what the mechanism does not do (no partition, grid restored, non-causal), so its claims are sized correctly.

Critical weaknesses
  1. Table 7 gives 0.749 Dice on Hard-Unseen for removing both Mamba branches while Table 5 gives 0.734 for removing only the background one — impossible for nested variants, and the same inversion appears on two other subsets. With a single run and no variance estimates the 0.761 of w/o FG versus 0.762 of the full model cannot be read at all, so the second contribution has no evidence.

  2. The tables stop at STDDNet (ICCV 2025) and omit CP-Mamba — the paper's own closest prior, which orders Mamba tokens by predicted category — along with every 2025-2026 VPS method cited in Section 1, and CVC-612 has one baseline. The headline +20.4 / +22.4 Dice is measured against SALI's PVTv2-B2 row (0.592 / 0.579), 6.4 / 5.7 points below its own B5 row, so the margin is an artefact of the configuration.

Questions for authors

Q1. Given Eq. 2 and the margin term in Eq. 19, is the background guidance roughly the complement of the foreground guidance, which would make the background branch a near-reverse scan? If so, what does the twin design add over a bidirectional scan — can you report a single-branch forward+backward variant?

Q2. Please reconcile Tables 5 and 7: w/o FG+BG scores 0.749 on Hard-Unseen, w/o BG alone 0.734, which cannot both hold if the variants are nested. With no SUN-SEG validation split, on what data were the five-frame clip length and the loss weights chosen, when all are optimal on the test subsets?

Q3. Why does the comparison stop at STDDNet, and why is CP-Mamba absent when Section 2.2 names it as the closest prior? Please add CP-Mamba and the strongest 2025-2026 method under your protocol, plus SALI's B5 row alongside the B2 row you report.

Q4. On CVC-612, the weighted F-measure rises from 0.407 to 0.540 while Dice moves from 0.716 to 0.726. How does a one-point Dice gain produce a 13-point F-measure gain — same script and frame set for both methods?

Q5. What are the parameter counts of each row in Tables 5-7? Since removing a branch deletes a full Mamba block and MLA head, a capacity-matched control is needed to show the deltas come from ordering rather than capacity.

3 comments

Thank you for your valuable comments. We have carefully addressed your concerns in our rebuttal and would greatly appreciate it if you could review our responses when convenient.

After considering the responses in the rebuttal and the other reviews, I retain my original rating of the paper.

Dear Reviewer,

Do you have any remaining questions or concerns? We would be happy to discuss them further with you.

Reviewer_4nNLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProtoTwinMamba for video polyp segmentation. Frame-wise foreground and background prototypes are derived from coarse predictions, and their similarities to spatial features are combined with coarse probabilities to generate guidance maps. These maps jointly guide feature refinement and token ordering in two independently parameterized Mamba branches. After spatial processing, tokens are restored to their original grid positions for bidirectional temporal aggregation. A shallow detail-enhancement pathway and a convolutional decoder integrate the resulting features. Experiments on SUN-SEG and CVC-612 evaluate segmentation performance, with ablations examining the contributions of ordering, temporal modeling, and prototype guidance.

Critical strengths

The motivation is clear, and the experiments broadly support the design choices introduced in the Introduction. The paper evaluates spatial ordering, feature refinement, temporal modeling, branch structure, and supervision. In Table 5, guided ordering improves Dice over fixed ordering by 2.6 and 2.2 percentage points on Easy-Unseen and Hard-Unseen, respectively; removing temporal modeling reduces Dice by 2.0 and 3.0 points. These results support the practical value of the proposed spatial-temporal design.

Critical weaknesses

The state-of-the-art claim exceeds the scope of the comparisons. Tables 1-2 omit recent methods discussed in the paper, including WavePolyp, HFSTI-Net, and DALA. WavePolyp, published at ICLR 2026, evaluates on both SUN-SEG and CVC-612 and is directly relevant. The strongest PVTv2-B2 variant is also not compared with STDDNet using the same backbone, despite the availability of official training logs and checkpoints. Moreover, with Res2Net-50, Easy-Seen Dice ties STDDNet at 0.905, while Hard-Seen Dice/IoU decrease from 0.881/0.810 to 0.873/0.803. CVC-612 includes only one baseline. These results demonstrate competitive performance under the reported settings but do not establish state-of-the-art performance across both benchmarks.

Questions for authors
  1. Why are recent cited methods such as WavePolyp and the PVTv2-B2 version of STDDNet excluded? The authors should provide comparisons under a consistent evaluation protocol or narrow the SOTA claim accordingly.

  2. Keeping all other model components and training settings unchanged, how do directly sorting by coarse prediction probabilities alone or prototype similarities alone compare with the mixed guidance? Why does Equation (2) use a fixed equal-weight mixture of the two signals? Appendix C.1 has already verified the overall effect of the prototype module; the above controls would further clarify the complementarity of the two signals in sorting and the benefits of the mixing design.

  3. In “w/o Proto Module,” what guidance is used for MLA, gating, and sorting? Does “No-L3” remove the entire level-3 processing pathway or only its temporal module?

  4. Please specify clip sampling and temporal spacing, how predictions from overlapping clips are merged, and whether metrics are aggregated by frame or sequence. Are the published baseline results evaluated consistently with these choices?

  5. The authors should provide comparisons with simpler temporal aggregation methods and report end-to-end latency and throughput under the actual inference protocol. In addition, does Uni-TM use strictly causal inputs?

Reviewer_ULEzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a method for Polyp segmentation involving a frame-wise prototype extraction that uses the coarse foreground/background segmentations to help improve the feature representations for certain layers. This is followed by a "twin mamba" branch that combines the refined spatial and temporal information across different frames of the input video. In order to better model the spatial features, the spatial branch uses a region-based guidance to order the sequence of the tokens processed by the Mamba module. The spaThe method is evaluated on two datasets SUN-SEG and CVC-612, and achieve state-of-the-art results when compared with competitor methods.

Critical strengths
  • This approach shows a non-trivial improvement over previous approaches on a relatively difficult task.
  • The approach its different components are well motivated
Critical weaknesses
  • The method's figures 2/3, are quite difficult to understand. It made understanding the overall method quite difficult and unclear.
  • Although the motivation is clear, the actual improvement caused by each of the components is not studied enough. The current ablation study just removes the component and shows a decrease in performance, but one question would be is if this decrease in performance is caused by the component or if it is just the removal of general model capacity/parameters.
Questions for authors
  • To better understand the contribution of the fg/bg prototype guidance, do we have an understanding of the importance of the coarse segmentation? It is mentioned that this can be a limitation in the appendix, but can we understand how much the defects in the coarse segmentation effect the overall quality. One simple test/ablation can be run would be to use a "ground-truth" segmentation to guide the prototype refinement. Also, one can continually degrade/erode the ground-truth to understand how the final model quality improves/changes.

Comments on the paper

1 comment

We sincerely thank the reviewer for the constructive comments and the efforts devoted to the review process. We greatly appreciate your positive evaluation and score for our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.