acceptodds
Withdrawn from ICLR 2027

UniConvSeg: Query-Free Convolutional Unified Segmentation in Images and Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified segmentation models commonly rely on Transformer object queries andrepeated global query-pixel interactions, raising the question of whether one efficient model can cover image, prompt, and video segmentation using convolution alone. We present UniConvSeg, a purely convolutional, query-free architecture that shares a ConvNeXt V2 feature pyramid and combines dense object-conditioned kernels, a convolutional panoptic readout, a prompt-conditioned maskdecoder, bounded convolutional memory, and dataset-conditioned vocabulary selection. Training proceeds through task-specific prompt/VOS and panoptic stagesfollowed by joint optimization over panoptic, prompted, VOS, and video-panopticdata with sampling probabilities (0.40, 0.20, 0.20, 0.20). With 122M parameters,UniConvSeg reaches 130 FPS for image-panoptic inference on an A800, 5.9x the 22 FPS of OMG-Seg under the same batch-1 end-to-end protocol. It achieves 52.7 COCO PQ, 43.2 instance AP, 48.3 VIPSeg VPQ, 53.2 YT-VIS AP, 88.7DAVIS J&F, and 68.1 promptable-segmentation mIoU. These results, togetherwith controlled readout, memory, training, and association ablations, show that purely convolutional task conditioning can provide a strong accuracy-efficiency tradeoff across four segmentation interfaces without a Transformer query decoder.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified Segmentation, Efficient Vision Models, Attention-Free Architecture, Dynamic Convolution, Panoptic Segmentation, Video Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6NN6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UniConvSeg, a query-free convolutional framework for unified image and video segmentation. It integrates dynamic convolution, prompt-conditioned decoding, and temporal memory within a shared backbone. Experiments demonstrate competitive segmentation performance and improved inference efficiency across multiple benchmarks.

Critical strengths
  1. Efficiency: The proposed framework achieves relatively high inference throughput while maintaining reasonable segmentation performance across multiple tasks.
  2. Experimental coverage: The paper evaluates the framework on several image and video segmentation benchmarks and includes ablation studies to examine the contributions of individual components.
Critical weaknesses
  1. Limited novelty: The framework mainly integrates existing techniques without introducing substantial methodological innovations.
  2. Insufficiently controlled comparisons: The efficiency comparison with OMG-Seg uses different backbones and model sizes, making it difficult to isolate the benefits of the query-free design.
Questions for authors

See Weaknesses above.

1 comment

The authors did not address my concerns promptly, so I will maintain my original score.

Reviewer_ijzUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents UniConvSeg, a query-free, purely convolutional framework that unifies multiple segmentation tasks in a single model, including image panoptic segmentation, promptable segmentation, video object segmentation, and video panoptic segmentation. The model shares a ConvNeXt V2 backbone and feature pyramid, while using task-specific convolutional readouts: dense object-conditioned kernels, convolutional panoptic readout, prompt-conditioned mask decoder, and bounded convolutional memory. The authors also introduce dataset-conditioned vocabulary selection to address label-space conflicts across datasets. Experiments show that UniConVSeg achieves comparable segmentation accuracy while improving efficiency with fewer parameters and higher inference speed.

Critical strengths
  1. The paper presents a well-motivated idea that replaces query-based unified segmentation with task-specific convolutional readouts, leading to an efficient architectural design and tackling multiple image and video segmentation tasks with a single model.
  2. The paper provides extensive comparisons with a broad range of task-specific and unified segmentation methods, and clearly summarizes their differences in task coverage and architecture.
Critical weaknesses
  1. Limited evidence for the benefits of the query-free unified representation

UniConvSeg replaces the shared object-query representation with task-specific convolutional readouts to improve efficiency. In query-based models, object queries provide a shared object-centric representation that can be reused across tasks during joint training. In contrast, UniConvSeg uses substantially different readout mechanisms for different segmentation tasks. It is therefore unclear how object-level information is shared across tasks and whether these task-specific representations enable meaningful cross-task complementary learning beyond sharing the backbone and feature pyramid.

This concern is also relevant to the results in Table 2. Despite the substantial efficiency gain, UniConvSeg achieves lower image PQ and video VPQ than OMG-Seg. Moreover, the two methods use different ConvNeXt backbones, making it difficult to isolate how much of the accuracy and efficiency difference comes from the proposed query-free architecture itself rather than backbone capacity or implementation differences. A controlled comparison using the same backbone, input resolution, and evaluation protocol, together with parameters and FPS, would better isolate the effect of the architectural design. Additional analysis of cross-task transfer would also help clarify whether the task-specific readouts provide complementary benefits in joint multi-task learning.

  1. Incomplete Comparison with the Most Relevant Unified Baselines

Although Table 2 includes a broad range of prior methods, the comparison is relatively sparse for the most relevant segmentation baselines. For example, SAM and SAM2 are applicable to promptable and video segmentation, but the table does not report all available results for these tasks. Similarly, several unified segmentation methods are compared using only a limited set of metrics.

This makes it difficult to understand how UniConvSeg compares with existing methods across different segmentation tasks. It would be helpful if the authors reported additional baseline results on tasks where direct comparison is possible and explained why the remaining entries cannot be evaluated. This would make the accuracy and inference-speed comparison with UniConvSeg clearer.

Questions for authors
  • Can the authors report segmentation performance on an unseen dataset? The current evaluation is mostly conducted on datasets included in the training setup; evaluating on unseen datasets would help rule out possible overfitting to the training datasets.
  • Does the dataset-conditioned vocabulary selection require the dataset identity to be known at inference time? If so, how would UniConvSeg handle inputs from an unknown dataset where the valid label space is not predefined?
Reviewer_i4hVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Segmentaion is foundamental task in computer vision with many forms. Yet a number of unified models have been presented, it is interesting and valuable to investigated a new unified segmentation models, i.e., UniConvSeg. Experiments show that it reaches competitive segmentation accuracy with high efficiency.

Critical strengths
  • A unified convolutional segmentation models for unifying image/video segmentaion tasks.
  • Competitive segmentation accuracy with high efficiency.
  • Dataset-conditioned selection for combining multiple datasets during training.
Critical weaknesses
  • Whether it is necessary to emphsize "Query-free"? This work emphasizes that UniConvSeg is query-free in contrast to existing query-based methods. If the authors want to emphasize this point, more discussions are required to suggested to provide for explaning that "Query-free" benefits accuracy, efficiency or memory, etc.
  • More discussions and comparison are suggested to provide. For example, the performance of frontier VLM models and SAM 3 in these segmentation tasks.
  • UniConvSeg generally cannot beat SoTAs in terms of segmentation accuracy but is computationally efficient. This makes UniConvSeg not very promising but only okay for the communicity.
  • The technical contributions of this work are also suggested to be enhanced and emphasized.
Questions for authors

See the weakness. I will raise my rating if the authors well address my concerns in the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.