acceptodds
Withdrawn from ICLR 2027

Preserving Manifold Capacity: A Geometry-Aware Dual-Prompt Framework for Open-World Cross-Modal Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In large-scale cross-modal data mining, constructing an embedding space that generalizes to open-set distributions is a fundamental challenge. We investigate a geometric failure mode, termed “Feature Space Cannibalization," in which excessive adaptation to seen categories may reduce the representational flexibility required for generalization to unseen categories. This paper proposes a Geometry-Aware Dual-Prompting (GADP) framework to address this topology distortion. We introduce Modality-Adaptive Contextual Tokens as distribution anchors to adapt heterogeneous inputs (e.g., sparse sketches and dense images) to a pre-trained manifold without catastrophic forgetting. Crucially, we reinterpret an established angular-margin objective as a compactness regularizer for zero-shot cross-modal retrieval. By synergizing global alignment with angular margin regularization, we actively compact the distributions of seen classes, encouraging compact seen-class distributions and preserving representational flexibility for unseen categories. Experiments on three datasets under four standard evaluation settings demonstrate the effectiveness of GADP. In particular, the method reports 34.5% mAP@all on QuickDraw Ext. under our evaluation setup.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-World Cross-Modal Retrieval, Geometry-Aware Prompting, Manifold Capacity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_DMeDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Geometry-Aware Dual-Prompting (GADP) for zero-shot sketch-based image retrieval. The authors identify a phenomenon termed "Feature Space Cannibalization," arguing that excessive adaptation to seen classes may reduce the representation capacity available for unseen classes.

To address this issue, the proposed framework combines modality-specific prompts with contrastive alignment, triplet loss, angular-margin regularization, and center regularization. The method aims to improve cross-modal alignment while compacting seen-class representations. Experiments on three benchmark datasets under four evaluation settings report substantial improvements in zero-shot retrieval performance.

Critical strengths

1. Meaningful motivation and reasonable method design.

The paper addresses an important challenge in zero-shot cross-modal retrieval: adapting pretrained representations to seen categories without sacrificing generalization to unseen categories. The idea of encouraging more compact seen-class representations is intuitively reasonable, and the proposed components are designed to support cross-modal alignment and class discrimination.

2. Promising empirical performance.

The method reports substantial improvements across several standard ZS-SBIR benchmarks. The framework is also relatively parameter-efficient, updating only modality-specific prompts and LayerNorm parameters while keeping the main CLIP backbone frozen. These results suggest that the proposed combination of techniques may be practically useful.

Critical weaknesses

1. Insufficient justification of the central geometric claim.

The paper argues that compacting seen-class representations preserves manifold capacity for unseen classes. However, reducing intra-class variance does not necessarily imply that the corresponding classification or confidence regions become smaller. Moreover, the theoretical discussion appears to conflate additive cosine margins with additive angular margins (Eqs. 14–16), and the claimed guarantee of non-zero void volume is not rigorously established.

Since manifold capacity preservation is presented as a central contribution, these issues substantially weaken the theoretical justification of the proposed insight.

2. Limited novelty and insufficient experimental attribution.

Several key components, including modality-specific prompting, contrastive alignment, and angular-margin regularization, are established techniques. Although their combination is reasonable, the methodological novelty is relatively limited.

The paper reports substantial performance gains, but some comparisons involve different CLIP backbone capacities, and the experiments use only a single random seed. Furthermore, the ablation study does not clearly separate the contributions of AM-Softmax and center regularization. Consequently, it remains difficult to determine whether the reported improvements are specifically attributable to the proposed manifold capacity preservation mechanism.

Questions for authors

Q1. Could the authors clarify the theoretical relationship between angular-margin regularization, intra-class compactness, and manifold capacity preservation? In particular, how do Eqs. 14–16 establish a reduction in the occupied regions of seen classes, and under what assumptions is a non-zero void volume guaranteed?

Q2. Could the authors clarify which loss terms are included in the "Capacity Reservation" component in Table 2? Section 5.3 appears inconsistent with Eq. 19 regarding the role of Lcap. How much of the improvement comes from AM-Softmax and center regularization individually?

Q3. Since some competing methods use different CLIP backbones, could the authors clarify which comparisons are directly matched in model capacity? A controlled same-backbone comparison would help establish how much of the reported improvement comes from the proposed method.

Reviewer_pj1fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Geometry-Aware Dual-Prompting (GADP) for zero-shot sketch-based image retrieval. It introduces separate sketch and image prompts into a largely frozen CLIP visual encoder, updates LayerNorm parameters, and combines cross-modal contrastive learning, hard-negative triplet constraints, AM-Softmax, and center regularization. Its central interpretation is that compacting seen-class angular distributions mitigates “Feature Space Cannibalization” and preserves representational capacity for unseen categories. Evaluation covers two Sketchy Ext. splits, TU-Berlin Ext., and QuickDraw Ext., with component ablations under a fixed prompt architecture, t-SNE visualizations, and qualitative retrieval examples. The reported retrieval results are strong, although their connection to the proposed capacity-reservation mechanism remains insufficiently established.

Critical strengths
  1. Strong reported results across several standard zero-shot retrieval protocols. Table 1 reports the corresponding mAP metrics of 80.4%, 92.2%, 81.0%, and 34.5% on TU-Berlin Ext., Sketchy Ext. Split1, Sketchy Ext. Split2, and QuickDraw Ext., respectively. These evaluations cover different levels of sketch abstraction and category partitions, indicating worthwhile empirical value in the training recipe. The retrieval results constitute the principal evidence supporting acceptance.
  2. Ablations under a fixed prompt architecture support the effectiveness of the joint training recipe. Table 2 compares global alignment, local constraints, and the capacity-regularization combination while keeping the prompt architecture fixed. Relative to global alignment alone, the complete model improves mAP by 9.4, 4.7, 6.9, and 11.1 percentage points across the four settings. Adding either the local constraints or the capacity-regularization combination also improves performance. This supports complementary effects among the training constraints and provides evidence beyond final-model comparisons alone.
Critical weaknesses
  1. The central capacity-reservation argument does not follow from the actual optimization objective. Equations (14)–(15) use the cosine margin cos(θ) − m, whereas Section 4.4 subsequently reasons about cos(θ + m) and directly subtracts m from the acceptance-region angular radius in Eq. (16). These margins are not equivalent, so the stated contraction does not follow from the implemented objective as formulated. Equation (17) also asserts that the total occupied volume is much smaller than the sphere’s surface measure without establishing sufficient conditions. Although Section 3 describes volume as a conceptual proxy and the introduction limits the guarantee, Section 4.4 explicitly claims guaranteed nonzero vacant capacity. Figure 3 only visualizes selected unseen categories through two-dimensional t-SNE and cannot establish contraction of seen-class occupied regions in the original space. Because this mechanism is the paper’s principal conceptual contribution, the gap materially weakens its scientific argument.
  2. The experimental definition and independent contribution of capacity regularization are unclear. Equation (19) explicitly includes L_cap, but Section 5.3 states that it is “implicitly optimized via CLIP alignment” and describes the capacity-reservation ablation as adding L_center. The alignment loss in Eqs. (8)–(9) contains neither the class prototypes nor the explicit cosine margin in Eq. (15), leaving the objective actually evaluated unclear. Table 2 supports the regularization combination but does not distinguish the contribution of the AM-Softmax margin from center regularization. This distinction is consequential because modality-specific shallow prompts and LayerNorm adaptation already appear in prior ZS-SBIR work, making the proposed regularization mechanism and interpretation central to the incremental contribution. Sain et al., 2023 Without a consistent objective specification and independent attribution, the evidence does not yet establish a substantive methodological advance beyond combining existing prompting and metric-learning components.
Questions for authors
  1. Clarification—geometric justification: For the actual cos(θ) − m objective, what justifies the angular-radius contraction in Eq. (16) and the volume conclusion in Eq. (17)? Is there a valid derivation or existing quantitative analysis in the original embedding space connecting contraction of the defined seen-class acceptance regions to preserved unseen-class capacity, independently of empty regions in a two-dimensional t-SNE plot?
  2. Clarification—actual objective and attribution: Is L_cap in Eq. (15) explicitly optimized in the experiments reported in Tables 1 and 2, and which losses are included in each ablation row? With the encoder, prompt architecture, and other losses fixed, what evidence separates the benefit of a nonzero cosine margin from that of center regularization?
Reviewer_GscMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GADP, a geometry-aware dual-prompting framework for zero-shot sketch-based image retrieval (ZS-SBIR). The method employs modality-specific learnable prompts to bridge the gap between sketches and images, while combining global contrastive learning, hard-negative separation, and angular-margin regularization to optimize the cross-modal embedding space. The authors argue that compacting the feature distributions of seen classes can preserve representational capacity for unseen categories, thereby improving zero-shot retrieval performance.

Critical strengths

Well-motivated research problem.

The paper addresses the potential degradation of unseen-class generalization caused by over-adaptation to seen categories during cross-modal alignment. This is an important challenge in zero-shot cross-modal retrieval.

Reasonable integration of geometric constraints and cross-modal alignment.

The framework combines modality-specific prompts, global alignment, and class compactness regularization to adapt heterogeneous modalities while attempting to preserve the generalization capability of the pretrained model. The overall design is intuitive and well motivated.

Competitive experimental performance.

The method achieves promising retrieval performance across multiple ZS-SBIR benchmarks. The ablation studies also provide empirical support for the effectiveness of the proposed optimization objectives.

Critical weaknesses

Insufficient theoretical justification for the manifold capacity reservation hypothesis.

The central hypothesis is that compacting the feature distributions of seen classes preserves representational capacity for unseen categories. However, reduced intra-class variance does not necessarily imply better separability for unseen classes. The relationship between seen-class compactness and unseen-class generalization remains insufficiently justified.

Gap between the geometric interpretation and the actual optimization objectives.

The paper interprets InfoNCE, Triplet Loss, and AM-Softmax as mechanisms for global isometry, local geometric rectification, and capacity reservation, respectively. However, these conventional objectives do not directly guarantee the corresponding geometric properties. In particular, it remains unclear whether angular-margin regularization genuinely reduces the manifold volume occupied by seen classes, rather than simply improving discriminative boundaries.

Insufficient analysis of the independent contribution of dual prompting.

The proposed framework adopts separate modality-specific prompts for sketches and images, but the ablation studies mainly focus on different loss functions. The benefits of dual prompting over shared prompting, as well as the respective contributions of prompt learning and LayerNorm adaptation, are not sufficiently investigated.

Questions for authors
  1. Could the authors further explain why compacting the feature distributions of seen classes should improve generalization to unseen categories? Under what conditions might such compactness constraints negatively affect unseen-class discriminability?

  2. How can the authors demonstrate that angular-margin regularization actually preserves manifold capacity, rather than merely improving class compactness or discriminative boundaries? Are there more direct geometric measurements supporting this mechanism?

  3. Could the authors further justify the necessity of modality-specific dual prompting and provide comparisons with shared prompting or single-modality prompting to better isolate the contributions of individual components?

Reviewer_cug2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the generalization problem in Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) from a geometric perspective. The authors introduce the concept of "Feature Space Cannibalization," arguing that excessive optimization on seen categories may occupy the limited representational capacity of the hyperspherical embedding space, thereby degrading generalization to unseen categories. To address this issue, the paper proposes Geometry-Aware Dual-Prompting (GADP), which combines modality-specific learnable prompts with contrastive alignment, hard-negative separation, and angular-margin-based regularization. The central idea is to compact the distributions of seen classes while preserving representational capacity for unseen categories. Experiments are conducted on three datasets under four evaluation settings. The proposed method achieves competitive retrieval performance and shows improvements over several existing approaches on multiple benchmarks.

Critical strengths
  1. Relevant problem formulation and potentially useful geometric perspective. The paper addresses an important challenge in zero-shot cross-modal retrieval: balancing adaptation to seen categories with generalization to unseen categories. Interpreting this trade-off through the geometric compactness of learned representations provides an interesting perspective that may motivate further investigation.
  2. Competitive empirical performance with a relatively simple framework. The proposed method achieves competitive results across several standard ZS-SBIR benchmarks, with particularly strong performance on TU-Berlin Ext. and Sketchy Ext. The combination of parameter-efficient prompt tuning and existing metric learning objectives also makes the framework relatively straightforward to implement. These empirical results suggest that the proposed combination of components may have practical value, even though its theoretical interpretation requires further justification.
Critical weaknesses
  1. Insufficient theoretical justification for the core geometric claims. The paper's central concepts, particularly "Manifold Capacity" and "Feature Space Cannibalization," lack rigorous mathematical foundations. The metric structure of the hypersphere is not clearly specified, and several key claims rely primarily on intuition rather than formal derivations or supporting references. For instance, the paper does not establish why increasing intra-class angular variance necessarily enlarges the occupied region, or why reduced remaining capacity inevitably causes retrieval collisions and degraded zero-shot generalization. Moreover, Eq. (3) does not account for overlaps between class-occupied regions, while the capacity preservation claims in Eqs. (16)–(17) are insufficiently justified.
  2. Limited methodological novelty and insufficient experimental validation. The proposed Distribution Anchoring mechanism essentially concatenates learnable prompts with input tokens, but the claim that it maps representations toward the manifold center lacks a clear mathematical definition or justification. The overall framework also largely combines existing techniques, including contrastive learning, triplet loss, and AM-Softmax. Furthermore, the experiments do not directly verify the claimed preservation of manifold capacity, and the ablation study does not sufficiently isolate the contributions of individual components. Finally, all results are reported from single runs without standard deviations across multiple random seeds, raising concerns about reproducibility and statistical reliability.
Questions for authors

Q1. Geometric formulation and metric definition Could the authors explicitly define the metric used on the hyperspherical embedding space and clarify the relationship among Euclidean distance, angular distance, and geodesic distance in the proposed framework? In particular, what justifies approximating the posterior-defined occupied region in Eq. (2) as a hyperspherical cap?

Q2. Validity of manifold capacity measurement How does Eq. (3) account for overlaps among the occupied regions of different seen classes? Would the measure of their union be more appropriate than the sum of their individual measures? Additionally, is the proposed manifold capacity quantitatively measured during training or evaluation, or does it serve only as a conceptual interpretation?

Q3. Theoretical justification for capacity reservation Can the authors provide a formal derivation or appropriate theoretical support showing that reducing intra-class angular variance necessarily decreases the occupied region of a seen class and improves the generalization of unseen classes? In particular, how do Eqs. (14)–(17) establish that AM-Softmax guarantees a reduction in occupied volume? Could the authors also clarify the discrepancy between the additive cosine margin and additive angular margin formulations?

Q4. Mechanism and novelty of Distribution Anchoring What is the precise mathematical definition of the "manifold center" or "stable regions" mentioned in Section 4.1? How does concatenating modality-specific learnable prompts move the resulting representations toward these regions? What distinguishes the proposed Distribution Anchoring mechanism from conventional modality-specific prompt tuning, beyond its geometric interpretation?

Q5. Direct experimental evidence and ablation consistency Can the authors provide quantitative geometric analyses, such as intra-class angular variance, occupied-region estimates, or inter-class overlap statistics, to directly validate the proposed capacity reservation hypothesis? Additionally, could the authors clarify the treatment of L_cap and L_center in Table 2 and Section 5.3, and report separate ablations for these two terms?

Q6. Reproducibility and statistical reliability Can the authors report results averaged over multiple independent runs with different random seeds, including standard deviations for the main evaluation metrics? Additionally, could the authors provide the complete training configuration, particularly the batch size, training epochs, loss weights, hard-negative mining strategy, and model selection criteria?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.