- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes COOPER, a method for adapting class embeddings in frozen vision-language models. Unlike conventional approaches that jointly optimize a shared class embedding over multiple training examples, COOPER optimizes a separate embedding for each image–class pair and then combines these embeddings using weights based on their training IoU gains. The resulting embeddings are stored in a fixed, reusable class bank. The authors also introduce COOPER-CLS, which uses joint optimization with a regularization term that keeps the learned embeddings close to their original representations for image classification. The paper further analyzes the differences between local aggregation and joint optimization from a theoretical perspective. It also studies different embedding adaptation strategies through prompt-space analysis, segmentation experiments, cross-domain transfer, and image classification. The experiments cover 11 segmentation datasets with SAM3 and 9 classification backbones. COOPER improves upon the original SAM3 on all 11 segmentation datasets, with an average gain of 16.30 mIoU points. The problem studied in this paper is relevant, and the experimental analysis is fairly extensive. However, the main method is closely related to existing techniques, and the current results do not yet provide enough evidence that per-example optimization followed by aggregation offers a consistent advantage over well-designed joint optimization methods.
Critical strengths
S1. Rather than introducing another trainable module, the paper studies how different strategies for optimizing class embeddings affect generalization when the model weights are frozen. In particular, the authors observe that joint optimization can continue to reduce training loss without necessarily improving test performance. This is an interesting problem worth investigating.
S2. The experiments cover 11 segmentation datasets spanning natural images, remote sensing, and medical imaging, as well as several classification backbones from the CLIP, PE, and SigLIP families. The authors also evaluate cross-dataset transfer, image-to-video transfer, and changes in embedding direction and magnitude. These experiments provide useful evidence about the behavior of the learned embeddings.
S3. The theoretical analysis explains how local aggregation and joint optimization can lead to different solutions by examining the relationship between loss curvature and local optima. The authors also analyze how ineffective local updates can affect aggregation. The appendix includes additional controlled comparisons, ablation studies, and failure cases, which help clarify when the proposed approach works and where it falls short.
Critical weaknesses
W1. Limited novelty of the main method. The main claimed contribution is to optimize an embedding independently for each training example and then construct a fixed class bank through weighted aggregation. However, local parameter optimization, parameter averaging, class prototype aggregation, and performance-based weighting have all been studied in prior work. For example, FedFomo [1] uses performance improvements relative to a reference model to determine aggregation weights. ProDA [2] directly learns output prompt embeddings rather than input prompt embeddings. DP²FL [3] independently trains prompt parameters with a frozen foundation model and aggregates them using validation-based weights. Although these methods differ from COOPER in their tasks, optimization granularity, and the parameters being aggregated, the underlying ideas are closely related. The paper mainly discusses local training and aggregation through FedAvg [4] and Model Soups [5], but gives less attention to prior work on performance-weighted prompt aggregation and output embedding learning. Furthermore, the class bank itself is simply a fixed collection of learned embeddings, one for each class, and does not appear to introduce a new mechanism. I would therefore like the authors to clarify what is technically new about per-example embedding refinement beyond applying existing local optimization and weighted aggregation techniques to SAM3.
W2. The effectiveness of gain-weighted aggregation is not sufficiently supported. COOPER assigns aggregation weights based on the improvement in training IoU obtained by each independently optimized embedding. However, a larger improvement on the training example does not necessarily mean that the resulting embedding will generalize better to unseen images. Table 3 shows that most of the benefit from gain weighting comes from Kvasir-SEG (+37.53 pp). The improvements on other datasets are much smaller, and weighting even hurts performance in some cases. Moreover, the component ablation in Appendix E.4 shows that gain weighting does not outperform uniform averaging on CamVid, VOC, or LoveDA. These results raise questions about how generally useful the proposed weighting scheme is. The authors should explain why training IoU gain is used as the weighting criterion and whether it offers any clear advantage over alternatives such as final IoU, performance on separate validation examples, or other measures of embedding quality.
W3. The local–joint comparisons do not provide sufficient evidence for the claimed advantage of the proposed method. A central question of this paper is whether optimizing embeddings independently for each example and then aggregating them leads to better generalization than joint optimization. However, the methods compared in Table 2 do not use fully matched training budgets, candidate selection procedures, or optimization settings. Although Appendix E.1 provides more carefully controlled experiments across three random seeds, these experiments cover only three datasets: VOC, LoveDA, and CamVid. The results show that local aggregation performs slightly better on VOC and LoveDA, while joint optimization performs substantially better on CamVid. These findings suggest that local aggregation does not consistently outperform joint optimization. The current evidence is insufficient to establish that local aggregation offers a reliable advantage over well-tuned joint optimization across different domains and support set sizes. Furthermore, the paper does not provide a reliable criterion for deciding when local aggregation should be preferred over joint optimization.
Questions for authors
Q1. Could the authors clarify the key differences between COOPER and prior methods such as ProDA [2], FedFomo [1], and DP²FL [3], which have already explored output embedding learning or performance-weighted parameter aggregation? Beyond applying local optimization and weighted averaging to SAM3 class embeddings, is there a more specific technical contribution?
Q2. Could the authors report GPU-hours, wall-clock time, and final mIoU for COOPER and joint optimization under the same hardware and total compute budget? In particular, how do the computational cost and segmentation performance change as the number of support examples per class increases from 4 to 16, 32, 64, or more?
Q3. Could the authors report GPU-hours, wall-clock time, and final mIoU for COOPER and joint optimization under the same hardware and total compute budget? In particular, how do the computational cost and segmentation performance change as the number of support examples per class increases from 4 to 16, 32, 64, or more?
I would be willing to reconsider my rating if the authors can address these concerns, particularly those regarding the novelty of the proposed method.
[1] Zhang, M., Sapra, K., Fidler, S., Yeung, S., and Alvarez, J. M. Personalized Federated Learning with First Order Model Optimization. ICLR, 2021. https://arxiv.org/abs/2012.08565 [2] Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. Prompt Distribution Learning. CVPR, 2022. https://openaccess.thecvf.com/content/CVPR2022/html/Lu_Prompt_Distribution_Learning_CVPR_2022_paper.html [3] Chang, Y., Shi, X., Zhao, X., Chen, Z., and Ma, D. Dual Prompt Personalized Federated Learning in Foundation Models. Scientific Reports, 15, 28026, 2025. https://doi.org/10.1038/s41598-025-11864-4 [4] Chen, T., et al. SAM3-Adapter: Efficient Adaptation of Segment Anything 3 for Camouflage Object Segmentation, Shadow Detection, and Medical Image Segmentation. arXiv:2511.19425, 2025. https://arxiv.org/abs/2511.19425