acceptodds
Withdrawn from ICLR 2027

COOPER: CLASS-OUTPUT OPTIMIZATION VIA PER-EXAMPLE EMBEDDING REFINEMENT

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Methods that learn class embeddings for frozen vision–language models typically optimize them jointly across labeled examples. We compare this approach with local optimization followed by aggregation: fitting one embedding per example, then combining the fits within each class. COOPER builds a reusable class bank from these local refinements; its classification counterpart, COOPER-CLS, fits the bank jointly. Both freeze model weights, select banks using training data, and deploy fixed banks without test-time optimization. Our theory explains why the two approaches can produce different embeddings and when weighting can reduce bias from ineffective refinements. We certify that no convex combination of the sampled text-prompt embeddings reproduces the tested COOPER corrections. We evaluate COOPER on eleven segmentation datasets with SAM3 and COOPER-CLS across nine classification backbones. COOPER yields positive gains over SAM3's original embeddings on all eleven datasets, averaging +[mean gain] mIoU points, and its banks also transfer to other datasets and video. Although controlled comparisons give dataset-dependent local–joint rankings, COOPER outperforms the tested fixed-recipe joint-fitting baselines on average, including joint fitting initialized with COOPER embeddings. For cosine classification with CLIP, PE, and SigLIP, however, our controls favor joint fitting over local aggregation. Our work shows that the choice between local aggregation and joint fitting shapes how well learned class embeddings generalize.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision–language models, embedding optimization, parameter-efficient adaptation, local-to-class aggregation, open-vocabulary segmentation, few-shot classification.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pfWqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes COOPER, a method for adapting class embeddings in frozen vision-language models. Unlike conventional approaches that jointly optimize a shared class embedding over multiple training examples, COOPER optimizes a separate embedding for each image–class pair and then combines these embeddings using weights based on their training IoU gains. The resulting embeddings are stored in a fixed, reusable class bank. The authors also introduce COOPER-CLS, which uses joint optimization with a regularization term that keeps the learned embeddings close to their original representations for image classification. The paper further analyzes the differences between local aggregation and joint optimization from a theoretical perspective. It also studies different embedding adaptation strategies through prompt-space analysis, segmentation experiments, cross-domain transfer, and image classification. The experiments cover 11 segmentation datasets with SAM3 and 9 classification backbones. COOPER improves upon the original SAM3 on all 11 segmentation datasets, with an average gain of 16.30 mIoU points. The problem studied in this paper is relevant, and the experimental analysis is fairly extensive. However, the main method is closely related to existing techniques, and the current results do not yet provide enough evidence that per-example optimization followed by aggregation offers a consistent advantage over well-designed joint optimization methods.

Critical strengths

S1. Rather than introducing another trainable module, the paper studies how different strategies for optimizing class embeddings affect generalization when the model weights are frozen. In particular, the authors observe that joint optimization can continue to reduce training loss without necessarily improving test performance. This is an interesting problem worth investigating.

S2. The experiments cover 11 segmentation datasets spanning natural images, remote sensing, and medical imaging, as well as several classification backbones from the CLIP, PE, and SigLIP families. The authors also evaluate cross-dataset transfer, image-to-video transfer, and changes in embedding direction and magnitude. These experiments provide useful evidence about the behavior of the learned embeddings.

S3. The theoretical analysis explains how local aggregation and joint optimization can lead to different solutions by examining the relationship between loss curvature and local optima. The authors also analyze how ineffective local updates can affect aggregation. The appendix includes additional controlled comparisons, ablation studies, and failure cases, which help clarify when the proposed approach works and where it falls short.

Critical weaknesses

W1. Limited novelty of the main method. The main claimed contribution is to optimize an embedding independently for each training example and then construct a fixed class bank through weighted aggregation. However, local parameter optimization, parameter averaging, class prototype aggregation, and performance-based weighting have all been studied in prior work. For example, FedFomo [1] uses performance improvements relative to a reference model to determine aggregation weights. ProDA [2] directly learns output prompt embeddings rather than input prompt embeddings. DP²FL [3] independently trains prompt parameters with a frozen foundation model and aggregates them using validation-based weights. Although these methods differ from COOPER in their tasks, optimization granularity, and the parameters being aggregated, the underlying ideas are closely related. The paper mainly discusses local training and aggregation through FedAvg [4] and Model Soups [5], but gives less attention to prior work on performance-weighted prompt aggregation and output embedding learning. Furthermore, the class bank itself is simply a fixed collection of learned embeddings, one for each class, and does not appear to introduce a new mechanism. I would therefore like the authors to clarify what is technically new about per-example embedding refinement beyond applying existing local optimization and weighted aggregation techniques to SAM3.

W2. The effectiveness of gain-weighted aggregation is not sufficiently supported. COOPER assigns aggregation weights based on the improvement in training IoU obtained by each independently optimized embedding. However, a larger improvement on the training example does not necessarily mean that the resulting embedding will generalize better to unseen images. Table 3 shows that most of the benefit from gain weighting comes from Kvasir-SEG (+37.53 pp). The improvements on other datasets are much smaller, and weighting even hurts performance in some cases. Moreover, the component ablation in Appendix E.4 shows that gain weighting does not outperform uniform averaging on CamVid, VOC, or LoveDA. These results raise questions about how generally useful the proposed weighting scheme is. The authors should explain why training IoU gain is used as the weighting criterion and whether it offers any clear advantage over alternatives such as final IoU, performance on separate validation examples, or other measures of embedding quality.

W3. The local–joint comparisons do not provide sufficient evidence for the claimed advantage of the proposed method. A central question of this paper is whether optimizing embeddings independently for each example and then aggregating them leads to better generalization than joint optimization. However, the methods compared in Table 2 do not use fully matched training budgets, candidate selection procedures, or optimization settings. Although Appendix E.1 provides more carefully controlled experiments across three random seeds, these experiments cover only three datasets: VOC, LoveDA, and CamVid. The results show that local aggregation performs slightly better on VOC and LoveDA, while joint optimization performs substantially better on CamVid. These findings suggest that local aggregation does not consistently outperform joint optimization. The current evidence is insufficient to establish that local aggregation offers a reliable advantage over well-tuned joint optimization across different domains and support set sizes. Furthermore, the paper does not provide a reliable criterion for deciding when local aggregation should be preferred over joint optimization.

Questions for authors

Q1. Could the authors clarify the key differences between COOPER and prior methods such as ProDA [2], FedFomo [1], and DP²FL [3], which have already explored output embedding learning or performance-weighted parameter aggregation? Beyond applying local optimization and weighted averaging to SAM3 class embeddings, is there a more specific technical contribution?

Q2. Could the authors report GPU-hours, wall-clock time, and final mIoU for COOPER and joint optimization under the same hardware and total compute budget? In particular, how do the computational cost and segmentation performance change as the number of support examples per class increases from 4 to 16, 32, 64, or more?

Q3. Could the authors report GPU-hours, wall-clock time, and final mIoU for COOPER and joint optimization under the same hardware and total compute budget? In particular, how do the computational cost and segmentation performance change as the number of support examples per class increases from 4 to 16, 32, 64, or more?

I would be willing to reconsider my rating if the authors can address these concerns, particularly those regarding the novelty of the proposed method.

[1] Zhang, M., Sapra, K., Fidler, S., Yeung, S., and Alvarez, J. M. Personalized Federated Learning with First Order Model Optimization. ICLR, 2021. https://arxiv.org/abs/2012.08565 [2] Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. Prompt Distribution Learning. CVPR, 2022. https://openaccess.thecvf.com/content/CVPR2022/html/Lu_Prompt_Distribution_Learning_CVPR_2022_paper.html [3] Chang, Y., Shi, X., Zhao, X., Chen, Z., and Ma, D. Dual Prompt Personalized Federated Learning in Foundation Models. Scientific Reports, 15, 28026, 2025. https://doi.org/10.1038/s41598-025-11864-4 [4] Chen, T., et al. SAM3-Adapter: Efficient Adaptation of Segment Anything 3 for Camouflage Object Segmentation, Shadow Detection, and Medical Image Segmentation. arXiv:2511.19425, 2025. https://arxiv.org/abs/2511.19425

Reviewer_4WjdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Authors introduce COOPER, a method for improving the performance of frozen vision-language models by optimizing class embeddings without updating the model's parameters. Unlike conventional approaches that jointly optimize class embeddings across training examples, COOPER independently refines embeddings for individual image–class pairs and subsequently aggregates them using gain-based weighting to construct reusable class embeddings. The authors also introduce COOPER-CLS, a classification counterpart that employs joint optimization with an angular regularizer. The paper provides a theoretical analysis of why local aggregation and joint optimization may produce different solutions and investigates when weighting can reduce bias from ineffective refinements. Experiments across 11 segmentation datasets using SAM3 demonstrate an average improvement of 16.30 mIoU percentage points over the original embeddings. The authors additionally evaluate cross-domain and image-to-video transfer, along with classification experiments across nine backbones. While COOPER generally performs favorably against the tested joint optimization baselines in segmentation, the classification experiments suggest that joint optimization is more effective than local aggregation in that setting.

Critical strengths

Strengths -

  1. Introduction and the whole paper were easy to follow.
  2. The problem the authors are trying to solve is timely.
Critical weaknesses

The paper has two critical weaknesses -

  1. Insufficient empirical evidence supporting the central motivation of local optimization over joint optimization - I find that the central motivation of the paper is not sufficiently supported by the experimental results.
  • The authors argue that joint optimization can lead to overfitting and poor generalization, thereby motivating per-example optimization followed by aggregation. However, Table 2 shows that joint optimization outperforms COOPER on several datasets, including CamVid, Cityscapes, PascalPart, and Kvasir-SEG.
  • Furthermore, the authors themselves acknowledge that joint optimization performs better in the classification setting. Although Section 3 provides a theoretical explanation based on curvature-endpoint coupling, I do not see experiments establishing whether these theoretical conditions hold for the evaluated datasets or explain the observed performance differences.

I believe the authors need to demonstrate when and why local aggregation provides better generalization than joint optimization, rather than relying primarily on average improvements across datasets. Without this analysis, I find the central methodological justification insufficiently convincing.

  1. Limited improvements over competitive baselines and inadequate computational cost comparisons - I am not convinced that the reported performance improvements justify the additional complexity of the proposed approach.
  • Although COOPER achieves an average improvement of +16.30 mIoU points over the original SAM3 embeddings, its improvement over the stronger Joint-O baseline is only 2.36 percentage points (+16.30 versus +13.94). Moreover, COOPER underperforms Joint-O on four of the eleven datasets.
  • I also find the computational comparison insufficient, particularly because COOPER requires independent optimization for multiple training examples, potentially introducing considerable computational overhead. Although the authors report construction times, these measurements are not obtained under matched GPU budgets, and Joint-O's reported time excludes the initial construction of the COOPER bank.

I would like to see a comprehensive comparison of total training time, GPU-hours, peak memory consumption, and inference throughput under identical computational settings. Additionally, I believe comparisons with more directly relevant SAM3 adaptation methods, such as FS-CPL and M2C, are necessary to establish whether COOPER offers a meaningful advantage over existing approaches. Without stronger baselines and fair computational comparisons, I find it difficult to assess the practical significance of the proposed method.

Questions for authors

See Weaknesses section.

Reviewer_2Ju2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript studies how to adapt the class embeddings consumed by frozen vision-language models, contrasting joint fitting of a shared embedding across labelled examples with per-example fitting followed by class-level aggregation. The proposed method, COOPER, freezes SAM3, fits one embedding per training image-class pair, selects the checkpoint with the highest training IoU, merges the selected fits with gain weights subject to a three-improvement gate, and applies a training-only guard choosing between a blend, the original embedding, and the incumbent, fixing the bank before deployment. Evaluation covers eleven segmentation datasets with SAM3, nine classification backbones, cross-domain bank reuse over nine dataset pairs, and image-to-video transfer to 343 VSPW clips, reporting a mean gain of 16.30 mIoU points over the original embeddings.

Critical strengths
  1. The central question is well posed and answered against the authors' own interest, since the controls in Figure 10 and Appendix J.3 show joint fitting leading by 4.43 points at sixteen shots, and the manuscript reports the reversal.
  2. The reachability claim is certified rather than asserted: a Frank-Wolfe gap supplies a valid lower bound on the distance to the convex hull of sampled prompts, the bound is positive for all 76 reported classes, and Appendix B.3 verifies the arithmetic in extended precision.
  3. The evaluation is broad, spanning natural, aerial, medical, and long-tailed segmentation, nine classification backbones across CLIP, PE, and SigLIP, transfer over nine source-target pairs without target fitting, and video transfer with no VSPW frames used for fitting or selection.
Critical weaknesses
  1. The 16.30-point mean is dominated by one degenerate baseline. On Kvasir-SEG the original scores 0.00 mIoU, so its gain of 78.46 is almost half of the eleven-dataset sum; excluding it the mean falls to 10.08 and the median is 9.88, while Table 2 reports no interval for the mean and the per-dataset intervals condition on one constructed bank, as Appendix E.3 states.
  2. Baseline parity is not established in Table 2, and the failures follow the recipe rather than the estimator class. CoOp-C falls to 55.52 points below Original on Cityscapes and 33.80 below on CamVid, and the manuscript itself shows in Section 5 and Appendix E that smaller updates or query fallback recover the LoveDA probe from 0.00 to about 41 IoU. Numbers from a configuration the authors can repair are not evidence that joint fitting overfits support-specific patterns, yet they drive the mean column and Figure 7. A fixed tuning budget for every baseline, with the selected learning rate, step cap, and fallback reported per dataset, is needed before the ranking can be read as a property of the estimator.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.