acceptodds
Under review as a conference paper at ICLR 2027

Semantic Purification for Conditional Representation Learning

Abstract

Conditional representation learning aims to extract criterion-specific features for customized tasks. Recent methods construct conditional subspaces spanned by criterion-specific text bases in the embedding space of vision-language models (VLMs). Image embeddings are then projected onto these subspaces to obtain conditional representations. However, since VLMs are not explicitly trained to disentangle semantics associated with different criteria, the corresponding conditional subspaces remain coupled. This coupling induces semantic leakage during projection, thereby degrading the semantic purity of conditional representations. To suppress semantic leakage, we propose Semantic Purification for Conditional Representation Learning (SP-CRL). Specifically, SP-CRL first decomposes the original text basis and performs curvature-based adaptive truncation on the resulting basis vectors to construct a purer conditional subspace. It then identifies an appropriate noise subspace and projects image embeddings onto its null space to remove irrelevant semantic components. Extensive experiments across customized clustering, customized few-shot classification, and customized retrieval tasks demonstrate that SP-CRL achieves state-of-the-art performance with superior generalization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Conditional Representation Learning, Semantic Leakage, Subspace Projection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_odzaOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the semantic leakage problem in conditional representation learning, the authors propose Criterion-Specific Basis Refinement (CSBR) and Null-Space Denoising Projection (NSDP) on top of CRL to refine the conditional representations, thereby improving the semantic purity of conditional representations. Extensive experiments on customized clustering, customized few-shot classification, and customized retrieval tasks demonstrate the superiority and generalization of the proposed method.

Critical strengths
  1. The paper achieves consistent performance improvements on all three downstream tasks, and the gains persist across three VLM backbones (CLIP, BLIP2, SigLIP).
  2. The proposed CSBR and NSDP perform basis refinement and null-space denoising of the conditional subspace, and the paper validates the effectiveness of the method through extensive comparative and ablation experiments.
Critical weaknesses
  1. The self-constructed Flowers dataset reaches 100% on the Species criterion. Does this indicate that the dataset itself is too simple?
  2. The paper emphasizes being training-free and efficient, but lacks an analysis of the cost of the NSDP greedy search as the number of criteria grows.
Questions for authors

Please see the weaknesses.

4 comments

Response to W1: Difficulty of the Flowers Dataset

Thank you for raising this important concern. We acknowledge that the Species criterion in the original Flowers dataset exhibits performance saturation, as several baseline methods already achieve near-perfect clustering results. To address this concern, we construct an expanded dataset, Flowers v2, containing 1,756 images with 40 species categories, compared with 533 images and 4 species categories in the original dataset.

To ensure a fair comparison, we follow the same experimental protocol as the original Flowers dataset, including the VLM backbone, clustering procedure, evaluation metrics, and repeated evaluations with different random seeds. All methods are evaluated under identical settings. The clustering results on Flowers v2 are summarized below:

MethodSpecies NMISpecies ACCSpecies ARIColor NMIColor ACCColor ARI
TAC79.0064.8157.6178.5181.8370.76
Multi-MaP69.8556.2443.2267.3073.4964.42
Multi-Sub88.4178.6776.4383.6284.2382.49
CRL86.4479.0474.1984.7984.4481.61
SP-CRL (Ours)90.1781.3778.9488.0486.0184.93

Flowers v2 provides a more challenging evaluation under the Species criterion, where SP-CRL achieves 90.17% NMI, 81.37% ACC, and 78.94% ARI, rather than the saturated performance observed on the original dataset. Importantly, SP-CRL consistently outperforms all compared methods across all six evaluation metrics under both Species and Color criteria. These results further demonstrate the effectiveness of SP-CRL on a fine-grained dataset with substantially more species categories. We have uploaded both Flowers v1 and Flowers v2, organized by species, to the supplementary materials.

Response to W2: Computational Cost of NSDP greedy search

Thank you for this insightful question. Although SP-CRL is training-free, we acknowledge that NSDP's greedy search introduces additional computational overhead. We therefore provide a theoretical complexity analysis and additional experiments evaluating its runtime as the number of candidate non-target criteria increases.

Computational Complexity of NSDP. Let mm denote the number of candidate non-target criteria, s≤ms\leq m the number of greedy search rounds, nn the number of initial text basis vectors per criterion, dd the embedding dimension, and NN the number of image embeddings. At round jj, NSDP evaluates m−j+1m-j+1 candidates, each involving at most jnjn basis vectors. Accounting for subspace refinement and denoising priority computation, the overall time complexity is upper-bounded by

=O\left(\sum_{j=1}^{s}(m-j+1)jnd(N+d)\right) =O(ms^2nd(N+d)).$$ In the worst case, $s=m$, yielding an $O(m^3nd(N+d))$ upper bound. **Scalability of NSDP.** To evaluate how the computational cost of NSDP scales with the number of candidate non-target criteria, we conduct a stress test on CLEVR4-10k with $m\in\{2,4,6,8,16,32\}$ simulated non-target criteria, each containing 512 initial text basis vectors. All experiments are conducted on an NVIDIA RTX 5070 Ti GPU, with runtimes averaged over 100 runs after 30 warm-up iterations. We disable early stopping (i.e., terminating the search when the denoising priority no longer improves) to evaluate the full greedy search procedure. All results are reported as NSDP greedy search runtime (s). | Target Criterion | 2 | 4 | 6 | 8 | 16 | 32 | |:--|--:|--:|--:|--:|--:|--:| | Shape | 0.227 | 0.538 | 0.873 | 1.721 | 6.415 | 37.509 | | Color | 0.163 | 0.438 | 1.029 | 1.433 | 6.935 | 36.070 | | Count | 0.183 | 0.581 | 1.047 | 1.506 | 5.989 | 37.019 | | Texture | 0.218 | 0.467 | 1.127 | 1.440 | 6.951 | 37.906 | As shown above, NSDP's runtime increases with the number of candidate non-target criteria, consistent with the theoretical analysis. Even with early stopping disabled, the search completes within 1.73 s for up to 8 criteria and within 38 s for 32 criteria. These results demonstrate that NSDP's greedy search incurs manageable one-time computational overhead. Moreover, the number of non-target criteria is typically limited in practical scenarios, and early stopping can further reduce the computational cost.

I thank the authors for their response. The Flowers v2 experiment addresses my concern about the saturation of the original dataset, and the complexity analysis together with the scalability test clarifies the cost of the NSDP greedy search as the number of criteria grows. Both of my concerns are now resolved, so I keep my rating at 4.

AuthorsOpenReview ↗

We are glad that your concerns are fully resolved.

Dear Reviewer odza,

We are glad that your concerns are fully resolved. Thanks again for your time and effort in reviewing this paper. And thanks for your supportive and positive rating.

Regards from the authors.

Reviewer_KG5qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper focuses on the coupling between criterion-specific conditional subspaces and the resulting semantic leakage during projection. The authors refer to the phenomenon that information of non-target criteria can still be detected in target conditional representations as semantic leakage. They propose SP-CRL, which consists of two steps: CSBR refines the target subspace via SVD decomposition and adaptive truncation, and NSDP selects an appropriate noise subspace after potential denoising gain and loss to reduce non-target semantic information through null-space projection. Experiments are conducted on customized clustering, few-shot classification and retrieval tasks. The experimental results demonstrate that the proposed method achieves overall performance improvements compared with existing baselines. Component ablation experiments verify the complementary effect of the two modules. The appendix also presents sensitivity analyses on text basis quantity, LLM selection, and VLM selection.

Critical strengths
  1. The problem is clearly stated, with definitions and experimental evidence of semantic leakage provided. The authors point out that the conditional subspaces of different criteria are coupled, resulting in target conditional representations still containing non-target information. They demonstrate the geometric coupling between criterion-specific subspaces through analyses of subspace affinity and the semantic discriminability of SVD basis vectors, offering a relatively clear explanation for the limitations in existing projection-based conditional representation learning.
  2. The method is simple and does not require fine-tuning the VLM backbone. SP-CRL improves conditional representations via relatively straightforward linear algebraic operations and demonstrates practical performance across multiple tasks.
Critical weaknesses
  1. The independent contribution of the semantic purification mechanism has not been fully verified. Although the paper conducted ablation experiments on components, truncation ratios and noise subspace selection via Table 4–6, it still lacks controls such as dimension-matched random subspace filtering. Therefore, it cannot completely rule out performance gains brought by general dimensionality reduction or feature filtering. This will affect the judgment of the core mechanism of the method and its innovative contributions.

  2. Limited evaluation of non-target criterion inference. NSDP assumes that non-target criteria are specified in advance. Although the Appendix text introduces an automatic inference procedure, its effectiveness is not experimentally evaluated. The performance of SP-CRL with inferred non-target criteria remains unclear.

Questions for authors
  1. Could you add random subspace filtering matching the preserved dimensions of SP-CRL or other semantic-agnostic controls to further verify that the performance improvement mainly stems from the proposed semantic purification mechanism rather than general dimensionality reduction or feature filtering?
  2. According to Equations (16)–(17), after the denoised representations are re-projected onto the target subspace, the final representations may no longer be completely orthogonal to the selected noise subspace. Could the authors further explain whether this will lead to residuals of non-target semantic information and its impact on the denoising performance of NSDP?
  3. How does SP-CRL perform with automatically inferred non-target criteria instead of predefined criteria?
8 comments

Response to W1 & Q1: Semantic Purification versus General Dimensionality Reduction and Feature Filtering

Thank you for this valuable suggestion. To verify that the performance gains of SP-CRL stem from its semantic purification mechanisms rather than general dimensionality reduction or feature filtering, we conduct additional dimension-matched control experiments on customized clustering tasks for both CSBR and NSDP. All randomized experiments are repeated 10 times with independent random seeds, and the results are averaged across these runs.

Comparison of Dimensionality Reduction Strategies. To verify that CSBR's effectiveness stems from its adaptive basis truncation rather than general dimensionality reduction, we compare it with four dimension-matched alternatives: Random Subspace (projecting onto a random kk-dimensional orthogonal subspace), Bottom-k (retaining the last kk singular directions), Random-k (randomly selecting kk singular directions), and Uniform-k (uniformly sampling kk singular directions). The latter three strategies select directions from the SVD of the original criterion-conditioned text basis, onto which the image embeddings are then projected. Here, kk denotes the number of basis vectors retained by CSBR, and all strategies retain the same subspace dimensionality. All results below are reported as NMI / ACC / ARI (%).

StrategyCLEVR ShapeCLEVR ColorCards NumberCards Suit
Random Subspace (k-dimensional)69.17 / 70.35 / 59.416.13 / 18.58 / 3.4411.88 / 18.85 / 5.915.06 / 37.04 / 4.87
Bottom-k55.81 / 60.32 / 45.152.50 / 15.73 / 1.2132.28 / 38.11 / 23.102.08 / 31.03 / 1.96
Random-k68.31 / 71.04 / 59.4340.73 / 43.58 / 29.0133.60 / 38.04 / 21.8016.11 / 45.21 / 14.49
Uniform-k74.70 / 77.57 / 66.9132.56 / 36.88 / 19.4534.27 / 38.10 / 22.1011.55 / 41.95 / 11.71
Top-k (CSBR)82.62 / 85.65 / 77.8587.87 / 85.06 / 80.3538.10 / 40.76 / 24.0839.65 / 70.34 / 39.00

CSBR consistently achieves the best performance across all four criteria and all evaluation metrics. These results indicate that its improvements stem from effective basis refinement rather than dimensionality reduction.

Comparison of Noise Subspace Filtering Strategies. To further investigate whether the improvements of NSDP arise from its noise subspace identification strategy rather than general feature filtering, we introduce a dimension-matched random noise projection baseline. Specifically, we replace the noise subspace identified by NSDP with a randomly generated orthogonal subspace of the same dimension, while keeping the CSBR-refined target subspace unchanged. All results below are reported as NMI / ACC / ARI (%).

MethodCLEVR ShapeCLEVR ColorCards NumberCards Suit
CSBR Only82.62 / 85.65 / 77.8587.87 / 85.06 / 80.3538.10 / 40.76 / 24.0839.65 / 70.34 / 39.00
CSBR + Random Noise Projection81.60 / 84.07 / 76.0287.21 / 85.29 / 79.7636.83 / 40.14 / 23.5138.92 / 68.97 / 37.84
SP-CRL86.67 / 90.69 / 84.1189.99 / 89.88 / 85.0938.78 / 44.45 / 28.4041.61 / 73.22 / 42.20

Random noise projection generally degrades performance compared with CSBR alone, whereas SP-CRL consistently achieves better results across all criteria and metrics. These findings indicate that NSDP's effectiveness stems from suppressing irrelevant semantic components within the identified noise subspace, rather than random feature (subspace) filtering.

Response to Q2: Projection Order and Residual Semantic Leakage

Thank you for this insightful question. We agree that the final representations are not necessarily orthogonal to the selected noise subspace due to the inherent coupling between criterion-specific subspaces. However, geometric non-orthogonality does not necessarily imply the presence of residual non-target semantic information.

Theoretical Analysis. Consider an idealized decomposition of the image embeddings as I=It+In\mathbf{I}=\mathbf{I}_t+\mathbf{I}_n, where It\mathbf{I}_t and In\mathbf{I}_n denote the target and non-target semantic components, respectively. Let Pt=(Tt∗)⊤Tt∗\mathbf{P}_t=(\mathbf{T}_t^{\ast})^\top\mathbf{T}_t^{\ast} and Pn=(Tn∗)⊤Tn∗\mathbf{P}_n=(\mathbf{T}_n^{\ast})^\top\mathbf{T}_n^{\ast} denote the orthogonal projection matrices onto the target and noise subspaces, respectively. Assuming ItPt=It\mathbf{I}_t\mathbf{P}_t=\mathbf{I}_t and InPn=In\mathbf{I}_n\mathbf{P}_n=\mathbf{I}_n, Equations (16)–(17) yield

Rt=(It+In)(E−Pn)Pt=It(E−Pn)Pt,\mathbf{R}_t=(\mathbf{I}_t+\mathbf{I}_n)(\mathbf{E}-\mathbf{P}_n)\mathbf{P}_t=\mathbf{I}_t(\mathbf{E}-\mathbf{P}_n)\mathbf{P}_t,

where E\mathbf{E} is the identity matrix. Under this idealized assumption, the original non-target semantic component is eliminated before target projection. Nevertheless, the final representation may still have a nonzero projection onto the noise subspace:

RtPn=It(E−Pn)PtPn.\mathbf{R}_t\mathbf{P}_n=\mathbf{I}_t(\mathbf{E}-\mathbf{P}_n)\mathbf{P}_t\mathbf{P}_n.

Therefore, under this idealized decomposition, the component of the final representation lying within the noise subspace originates from the projection of target semantic information, rather than from residual non-target semantic information.

Ablation Study on Projection Order. Reversing the projection order ensures that the final representations are strictly orthogonal to the noise subspace. To investigate whether this guarantees better denoising performance, we compare three strategies: Target Only (direct projection onto the target subspace), Reversed Order (target projection followed by noise removal), and NSDP (Ours) (noise removal followed by target projection). All strategies use the same target subspace, while the two denoising strategies also share the same noise subspace. The customized clustering results (NMI / ACC / ARI, %) are summarized below:

MethodCLEVR ShapeCLEVR ColorCards NumberCards Suit
Target Only82.62 / 85.65 / 77.8587.87 / 85.06 / 80.3538.10 / 40.76 / 24.0839.65 / 70.34 / 39.00
Reversed Order83.40 / 84.92 / 78.3287.72 / 84.15 / 80.1238.40 / 42.64 / 25.5840.36 / 71.59 / 40.46
NSDP (Ours)86.67 / 90.69 / 84.1189.99 / 89.88 / 85.0938.78 / 44.45 / 28.4041.61 / 73.22 / 42.20

NSDP achieves the best performance across all four criteria and all evaluation metrics. Although Reversed Order guarantees strict orthogonality to the noise subspace, it consistently underperforms NSDP. These results indicate that strict geometric orthogonality does not necessarily yield better denoising performance, as it may excessively remove target-relevant semantic information.

Response to W2 & Q3: Effectiveness of Automatic Non-Target Criterion Inference

Thank you for raising this important concern. To assess the applicability of SP-CRL without predefined non-target criteria, we conduct additional experiments on CLEVR4-10k, Cards, and Flowers, comparing automatically inferred non-target criteria with predefined ones.

Automatic Non-Target Criterion Inference. Following the procedure described in Appendix D, we first remove target-subspace components from image embeddings and apply k-means clustering to the residual embeddings to select representative images. We then use MLLM (GPT-6) to infer candidate non-target criteria, from which NSDP identifies the noise subspace without manually specifying non-target criteria. The inferred candidates and selected noise criteria are summarized below:

DatasetTargetInferred Candidate Non-Target CriteriaSelected Noise Criteria
CLEVR4-10kShapeColor, Texture, Count, SizeColor
CLEVR4-10kColorShape, Texture, Count, SizeShape
CardsNumberSuit, Ink Color, Illustration Style, Background Appearance, Card OrientationSuit, Illustration Style
CardsSuitNumber, Ink Color, Illustration Style, Background Appearance, Card OrientationNumber
FlowersSpeciesColor, Background Appearance, Bloom Stage, Flower Count, ViewpointBackground Appearance, Viewpoint, Flower Count
FlowersColorSpecies, Petal Shape, Background Appearance, Bloom Stage, Flower Count, ViewpointBackground Appearance, Viewpoint, Petal Shape

The inferred candidates are semantically plausible and include factors beyond the predefined criterion sets.

Performance with Inferred Non-Target Criteria. We compare the clustering performance of SP-CRL using automatically inferred and predefined non-target criteria, along with a baseline without NSDP. The results (NMI / ACC / ARI, %) are summarized below:

DatasetTargetw/o NSDPNSDP (Predefined)NSDP (Inferred)
CLEVR4-10kShape82.62 / 85.65 / 77.8586.67 / 90.69 / 84.1186.67 / 90.69 / 84.11
CLEVR4-10kColor87.87 / 85.06 / 80.3589.99 / 89.88 / 85.0989.99 / 89.88 / 85.09
CardsNumber38.10 / 40.76 / 24.0838.78 / 44.45 / 28.4040.49 / 44.19 / 28.73
CardsSuit39.65 / 70.34 / 39.0041.61 / 73.22 / 42.2041.61 / 73.22 / 42.20
FlowersSpecies99.27 / 99.14 / 99.03100.00 / 100.00 / 100.0099.30 / 99.17 / 99.11
FlowersColor60.22 / 70.62 / 51.8963.47 / 79.93 / 59.5761.29 / 72.68 / 52.93

Automatic inference achieves identical performance to the predefined setting on CLEVR4-10k (Shape and Color) and Cards (Suit), as NSDP selects the same noise subspaces in both settings. Notably, on Cards (Number), automatic inference identifies Illustration Style as an additional noise criterion and achieves better overall performance than the predefined setting, suggesting that automatically inferred criteria can capture useful semantic factors beyond manually predefined ones. Although automatic inference underperforms the predefined setting on Flowers, it consistently improves upon the baseline without NSDP. These results demonstrate that SP-CRL remains effective with automatically inferred non-target criteria, supporting its applicability without manually predefined non-target criteria.

Reviewer_gP8xOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Semantic Purification for Conditional Representation Learning (SP-CRL), aiming to mitigate semantic leakage caused by the coupling between different conditional subspaces. Specifically, the method introduces two components: i) Criterion-Specific Basis Refinement (CSBR), which employs SVD and curvature-based adaptive truncation to obtain more criterion-specific textual bases; and ii) Null-Space Denoising Projection (NSDP), which identifies non-target semantic subspaces and suppresses irrelevant components in image representations while preserving target-related information. Extensive experiments on clustering, few-shot classification, and retrieval demonstrate the effectiveness of the proposed approach across diverse datasets and criteria.

Critical strengths
  1. Effective refinement of textual representations. The SVD-based basis refinement, together with curvature-based truncation, provides an effective way to filter redundant components and obtain more criterion-specific textual representations.
  2. Thoughtful denoising of image representations. The proposed NSDP explicitly addresses semantic interference from non-target criteria. Its joint consideration of denoising gain and target-semantic preservation makes the image refinement strategy well motivated.
  3. Comprehensive experimental evaluation. Extensive experiments are conducted across multiple tasks, datasets, and criteria. The ablation studies and sensitivity analyses further support the effectiveness and robustness of the proposed method.
Critical weaknesses
  1. Intermediate visualizations. The paper could benefit from additional visualizations of intermediate results. Although the paper introduces two modules, CSBR and NSDP, the processes of basis refinement and noise subspace identification are mainly explained through mathematical formulations. Visual illustrations would make the proposed method easier to understand.
  2. Computational efficiency. The computational overhead of the proposed method could be further analyzed, particularly considering the iterative greedy search and basis decomposition in NSDP. A discussion of computational complexity or runtime would help clarify its practical efficiency.
  3. Figure presentation. There seems to be a potential inconsistency in Figure 1(d). For the color-based retrieval example, the retrieved results should correspond to "pink", whereas the examples on the right appear to be predominantly red. In addition, trimming some redundant gray background could improve the overall visual presentation.
Questions for authors

See the above weaknesses.

5 comments

Response to W1: Intermediate Visualizations

Thank you for this helpful suggestion. For CSBR, Appendix A (Figure 3) already visualizes the semantic discriminability of singular directions, together with the curvature-based truncation boundaries. It shows that leading singular directions exhibit stronger target-criterion discriminability, while semantic selectivity gradually diminishes as singular values decrease. Moreover, the curvature-based truncation boundaries generally lie near the regions where semantic selectivity begins to diminish, providing empirical support for the rationale behind our proposed adaptive truncation strategy.

For NSDP, to make its greedy noise subspace selection process easier to understand, we have added a visualization of this process in Appendix G of the revised manuscript. Specifically, the new figure illustrates the denoising priorities of candidate noise subspaces across successive search iterations under different target criteria. It highlights the selected and unselected candidates, as well as the stopping condition determined by the previous best priority.

We hope these visualizations will help you better understand the basis refinement and noise subspace selection processes in SP-CRL.

Response to W2: Computational Efficiency

Thank you for this important question. We analyze the theoretical computational complexity of SP-CRL and conduct additional runtime experiments to evaluate its practical efficiency.

Theoretical Complexity Analysis. We analyze the computational complexity of CSBR, NSDP, and inference.

CSBR. Let nn denote the number of initial text basis vectors per criterion and dd the embedding dimension. Given a text basis matrix of size n×dn \times d, economy-sized SVD dominates the computational cost, yielding CCSBR=O(ndmin⁡(n,d))C_{\mathrm{CSBR}}=O(nd\min(n,d)).The subsequent energy computation, curvature estimation, and adaptive truncation introduce only lower-order costs.

NSDP. Let mm denote the number of candidate non-target criteria, s≤ms\leq m the number of greedy search rounds, and NN the number of image embeddings used to evaluate denoising priorities. At round jj, NSDP evaluates m−j+1m-j+1 candidates, each containing at most jnjn initial basis vectors. SVD-based refinement requires at most O(jnd2)O(jnd^2) time per candidate, while denoising gain and loss require O(Njnd)O(Njnd) and O(jnd2)O(jnd^2), respectively. Thus, the overall complexity across ss search rounds is CNSDP=O(∑j=1s(m−j+1)jnd(N+d))=O(ms2nd(N+d)).C_{\mathrm{NSDP}}=O\left(\sum_{j=1}^{s}(m-j+1)jnd(N+d)\right)=O(ms^2nd(N+d)).

Inference. CSBR and NSDP are performed only once during offline subspace construction. Since the refined subspaces remain fixed, the null-space denoising and target projection operations can be combined into a precomputed transformation. Each image then requires only two low-rank matrix multiplications, yielding an inference complexity of O(dkt)O(dk_t), where kt≤nk_t \leq n is the number of retained target basis vectors. Compared with CRL (O(nd)O(nd)), SP-CRL can achieve more efficient inference as ktk_t is typically smaller than nn.

Empirical Runtime Evaluation. We evaluate the computational efficiency of SP-CRL on CLEVR4-10k using an NVIDIA RTX 5070 Ti GPU. All measurements are averaged over 100 runs after 30 warm-up iterations.

Preprocessing Cost. We measure the preprocessing costs of CSBR and NSDP with n=512n=512 initial text basis vectors per criterion. The results are summarized below:

TargetCSBR (ms)NSDP (ms)Total (ms)Memory (MiB)
Shape102.732334.516437.24879.097
Color98.815336.629435.44478.946
Count100.112367.639467.75178.993
Texture99.356331.304430.66078.929

Preprocessing takes only 430.660–467.751 ms, with approximately 79 MiB of peak GPU memory, representing a modest one-time offline cost.

Scalability of NSDP. To investigate how the number of candidate non-target criteria affects the computational overhead of NSDP's greedy search, we vary the number of simulated non-target criteria m∈{2,4,6,8,16,32}m\in\{2,4,6,8,16,32\} while fixing n=512n=512. Early stopping (i.e., terminating the search when the denoising priority no longer improves) is disabled to execute all greedy search rounds. The runtime (s) is reported below:

Targetm=2m=2m=4m=4m=6m=6m=8m=8m=16m=16m=32m=32
Shape0.2270.5380.8731.7216.41537.509
Color0.1630.4381.0291.4336.93536.070
Count0.1830.5811.0471.5065.98937.019
Texture0.2180.4671.1271.4406.95137.906

As expected, runtime increases with the number of non-target criteria. Nevertheless, even with 32 non-target criteria and early stopping disabled, the full greedy search completes within 38 s as a one-time offline cost. In practical scenarios, such a large number of non-target criteria is unlikely, and the early stopping mechanism can further reduce computational overhead.

Inference Efficiency. We compare the inference efficiency of CRL and SP-CRL with different initial text basis sizes n∈{256,512,1024,2048}n\in\{256,512,1024,2048\}, excluding VLM encoding and subspace construction. The table reports inference time (μs/image\mu\mathrm{s}/\mathrm{image}) as CRL / SP-CRL.

nnShapeColorCountTexture
2560.024 / 0.0210.019 / 0.0190.021 / 0.0200.033 / 0.026
5120.035 / 0.0290.032 / 0.0190.033 / 0.0190.033 / 0.022
10240.059 / 0.0280.058 / 0.0260.057 / 0.0240.058 / 0.023
20480.118 / 0.0480.115 / 0.0250.115 / 0.0290.113 / 0.023

SP-CRL achieves comparable inference times across all settings. The speedup reaches 2.112.11–2.522.52× at n=1024n=1024 and 2.462.46–4.914.91× at n=2048n=2048, demonstrating increasing inference efficiency with larger text bases.

Overall, these results show that SP-CRL incurs manageable one-time preprocessing costs while providing efficient inference.

Response to W3: Figure presentation

Thank you for your careful observation. We would like to clarify that the retrieval results in Figure 1(d) are intentional rather than erroneous. As explicitly indicated by the label "Same-Color Candidates Excluded" in the figure, candidates sharing the query color (pink) are deliberately excluded from the retrieval pool. This setting aims to more clearly illustrate the differences in semantic purity of the representations obtained by different methods. Specifically, when no candidates share the same color as the query (pink), retrieval based on pure color-conditional representations should prioritize samples with similar colors (e.g., red objects), rather than those sharing the same shape (e.g., gear-shaped objects with dissimilar colors). As illustrated in Figure 1(d), SP-CRL retrieves predominantly red-colored objects with diverse shapes, whereas CLIP and the orthogonal projection method tend to retrieve gear-shaped objects with dissimilar colors. These observations provide qualitative evidence that SP-CRL produces purer conditional representations. Additionally, following your suggestion, we have trimmed the redundant gray background of Figure 1(d) in the revised manuscript to improve its visual clarity and readability.

Reviewer_44QCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Semantic Purification for Conditional Representation Learning (SP-CRL), a training-free framework designed to reduce semantic leakage in criterion-specific representations derived from vision-language models. SP-CRL introduces two components. Criterion-Specific Basis Refinement (CSBR) applies singular value decomposition and curvature-based adaptive truncation to retain basis vectors considered more relevant to the target criterion. Null-Space Denoising Projection (NSDP) then selects non-target subspaces through a greedy strategy that balances potential denoising gain against target-information loss, and removes their components before extracting conditional representations. The reported results show consistent improvements over CRL and several task-specific baselines. Additional ablations and sensitivity analyses examine the two proposed components, basis truncation, noise subspace selection, and different LLM/VLM backbones.

Critical strengths
  1. Well-motivated problem. The paper identifies a meaningful limitation of projection-based conditional representation learning: existing methods focus on the target subspace but pay less attention to its relationship with non-target subspaces. The authors clearly explain how this coupling can introduce unwanted semantic components during projection.
  2. Coherent and practical framework. CSBR and NSDP address semantic leakage in complementary ways, by refining the target basis and removing components associated with non-target subspaces. Both use standard linear algebra without additional representation-model training, making the framework straightforward to integrate with existing vision-language embeddings. Table 4 supports the complementary contributions of the two components.
  3. Broad experimental evaluation. The experiments cover customized clustering, few-shot classification, and retrieval. Results with CLIP, BLIP2, and SigLIP suggest that the method is not restricted to a single backbone. Sensitivity analyses across description generators and basis quantities provide further evidence of its robustness.
Critical weaknesses
  1. The connection between singular-value magnitude and semantic purity needs stronger justification. CSBR assumes that high-singular-value directions capture criterion-relevant semantics, while low-singular-value directions contain more irrelevant information. However, SVD ranks directions by embedding variance, which does not necessarily reflect semantic relevance. Although Appendix A provides empirical evidence, Figure 3 does not show a consistent sharp transition at the selected truncation boundary. The authors should better justify why maximum curvature provides a reliable semantic boundary and discuss when this assumption may fail.
  2. The noise subspace selection objective lacks direct semantic validation. NSDP uses image-embedding energy and overlap with the target subspace to estimate denoising gain and loss. However, high embedding energy does not necessarily correspond to irrelevant semantics, and low geometric overlap does not guarantee preservation of target information. While Table 6 shows improvements on the evaluated clustering tasks, it remains unclear whether the proposed priority score reliably captures the actual trade-off between removing irrelevant information and preserving target semantics.
  3. The practical applicability of NSDP depends on the availability of non-target criteria. The main method assumes that non-target criteria are known in advance. Although Appendix D introduces an MLLM-based procedure to infer them automatically, the main experiments do not evaluate this setting separately or compare it with manually specified criteria. The sensitivity to clustering choices and the quality of inferred criteria also remain unclear. This limits the evidence for applying the complete framework to less structured real-world scenarios.
  4. The performance gains are uneven across tasks, and computational costs are insufficiently evaluated. SP-CRL achieves clear improvements in clustering and few-shot classification, but the gains in retrieval are relatively modest. On DeepFashion, training-free SP-CRL improves mean mAP only from 7.93% to 8.18% over CRL, while the fabric result is slightly worse. Moreover, NSDP involves repeated subspace construction and greedy search, but the paper does not report preprocessing time, memory consumption, or total computational cost compared with CRL. These results make it difficult to assess whether the additional complexity is justified, particularly for retrieval.
Questions for authors

Q1. What supports the assumption that singular-value magnitude reflects criterion-specific semantic information? Could the authors compare curvature-based truncation with fixed cumulative-energy thresholds or singular-value-gap-based selection? How closely does the selected boundary align with the point where target and non-target discriminability begin to converge in Appendix A?

Q2. Does the proposed denoising priority reliably reflect semantic leakage reduction and target-information preservation? Could the authors compare priority scores with target and non-target linear probing results, and provide ablations using gain-only, loss-only, and the full objective?

Q3. Could the authors evaluate the automatic criterion inference procedure in Appendix D against the predefined-criterion setting? Reporting downstream performance, criterion quality, and sensitivity to the clustering parameter K would help clarify its practical robustness.

Q4. Could the authors report preprocessing time, peak memory usage, and inference cost for SP-CRL and CRL under the same hardware and backbone settings? Also, what explains the relatively modest retrieval improvements compared with the larger gains in clustering? An analysis of criterion overlap and its relationship with retrieval performance would help clarify when SP-CRL is most beneficial.

Final Comment after Rebuttal

The authors' responses have adequately addressed most of my major concerns and substantially clarified the proposed method and its effectiveness. While some limitations remain, I acknowledge the improvements and will raise my rating from 2 to 3.

7 comments

We sincerely thank you for your thorough review of our manuscript and for raising important and insightful questions. We address each concern in detail below.

Response to W1 & Q1: Singular Values and Semantic Relevance

Motivation and Empirical Evidence. We acknowledge that singular values reflect embedding variance rather than semantic relevance directly. However, the text basis in CSBR (a set of text embeddings) is constructed from descriptions explicitly conditioned on the target criterion. Consequently, variations among these text embeddings are expected to be primarily driven by differences in criterion-relevant semantics. We therefore hypothesize that high-singular-value directions, along which these text embeddings exhibit high variance, are more likely to capture the dominant criterion-relevant semantic information. As shown in Figure 3 (Appendix A), target-criterion discriminability generally decreases with singular-value rank, while non-target discriminability becomes relatively more prominent. These empirical observations support our hypothesis.

Comparison with Alternative Truncation Strategies. We conduct additional experiments comparing our curvature-based strategy with fixed cumulative-energy thresholds (20%, 60%, 80%, 95%, 98%, and 100%) and singular-value-gap-based selection (using either the maximum or minimum gap between consecutive singular values). The results (NMI / ACC / ARI, %) are summarized below:

MethodShapeColorNumberSuit
Maximum singular-value gap18.66 / 26.85 / 10.0033.24 / 32.81 / 17.6010.17 / 14.75 / 3.9624.44 / 54.56 / 20.85
Minimum singular-value gap81.03 / 81.34 / 74.0772.44 / 67.34 / 61.1437.06 / 39.23 / 23.3437.36‾\underline{37.36} / 57.62 / 30.38
Energy 20%63.42 / 69.14 / 52.0880.06 / 76.40 / 67.7011.95 / 15.38 / 4.4314.11 / 45.24 / 10.72
Energy 60%79.66 / 84.57 / 75.0887.37‾\underline{87.37} / 84.74‾\underline{84.74} / 79.74‾\underline{79.74}22.57 / 26.95 / 11.8536.91 / 69.38‾\underline{69.38} / 36.08‾\underline{36.08}
Energy 80%80.82 / 82.91 / 74.6386.55 / 83.52 / 78.1831.74 / 34.72 / 18.3036.77 / 64.38 / 35.07
Energy 95%82.17 / 84.65 / 77.5379.66 / 76.17 / 69.9137.76 / 40.22 / 23.1836.89 / 57.82 / 30.74
Energy 98%84.20 / 86.28 / 79.1977.74 / 74.47 / 67.9937.80‾\underline{37.80} / 40.28‾\underline{40.28} / 23.57‾\underline{23.57}37.15 / 59.35 / 31.40
Energy 100% (Full)81.72 / 82.12 / 75.0073.91 / 70.78 / 63.9936.86 / 39.38 / 23.2735.99 / 55.41 / 28.44
Adaptive (Ours)82.62‾\underline{82.62} / 85.65‾\underline{85.65} / 77.85‾\underline{77.85}87.87 / 85.06 / 80.3538.10 / 40.76 / 24.0839.65 / 70.34 / 39.00

The results show that appropriate truncation boundaries vary across target criteria, making a single fixed threshold unsuitable for all criteria. Our adaptive strategy achieves the best performance on three criteria and the second-best on the remaining criterion, demonstrating its effectiveness and robustness across different target criteria.

Alignment with Semantic Discriminability Convergence. We further examine the alignment between the curvature-based truncation boundary and the discriminability crossover point in Appendix A. Using a moving average with a window size of 5, we identify the first intersection of the target and non-target discriminability curves as an empirical reference for the point where target and non-target discriminability begin to converge. Across four target criteria, the mean absolute distance is only 13 out of 200 singular directions (6.5%), indicating relatively close alignment. Notably, this reference depends on the image dataset, whereas CSBR determines its boundary solely from the text basis. Therefore, exact alignment is not expected.

Response to W2 & Q2: Semantic Validation of Denoising Priority

Linear Probing Analysis of Denoising Priority. We conduct linear probing experiments to provide empirical justification for the proposed denoising priority P=G(1−L)P=G(1-L). We define ΔAt=Atafter−Atbefore\Delta A_t=A_t^{\mathrm{after}}-A_t^{\mathrm{before}} and ΔAnt=∑n≠t(Anafter−Anbefore)\Delta A_{\mathrm{nt}}=\sum_{n\ne t}(A_n^{\mathrm{after}}-A_n^{\mathrm{before}}), representing changes in target accuracy and cumulative non-target accuracy induced by denoising, respectively. We use ΔAt−ΔAnt\Delta A_t-\Delta A_{\mathrm{nt}} to quantify the empirical trade-off between target-information preservation and non-target semantic suppression.

TargetNoiseGain ↑\uparrowLoss ↓\downarrowPriority ↑\uparrowΔAt↑\Delta A_t \uparrowΔAnt↓\Delta A_{\mathrm{nt}} \downarrowTrade-off ↑\uparrow
ColorShape0.1780.3570.115-0.46-7.597.13
ColorCount0.0810.0940.073-0.30-1.321.02
ColorTexture0.0860.1100.076-0.23-2.191.96
TextureColor0.0930.2190.072-0.21-3.743.53
TextureShape0.1780.6080.070-7.21-6.28-0.93
TextureCount0.0810.1710.067-1.30-2.080.78

When Color is the target criterion, Shape achieves the highest gain and the largest reduction in non-target accuracy, supporting the effectiveness of the gain term. When Texture is the target criterion, Shape yields the highest denoising gain but substantially reduces target accuracy. Introducing the loss term significantly lowers its denoising priority, thereby avoiding excessive loss of target-relevant information. In both cases, the highest-priority candidate achieves the best empirical trade-off, supporting the complementary roles of gain and loss.

Ablation Study on Denoising Priority. We compare gain-only, loss-only, and the full objective for noise subspace selection. The results (NMI / ACC / ARI, %) are summarized below:

MethodTextureShapeColorCount
Gain-only10.95 / 23.59 / 6.3584.89‾\underline{84.89} / 87.63‾\underline{87.63} / 80.66‾\underline{80.66}89.33‾\underline{89.33} / 87.15‾\underline{87.15} / 83.09‾\underline{83.09}28.50 / 29.82 / 15.54
Loss-only11.23‾\underline{11.23} / 26.00‾\underline{26.00} / 7.19‾\underline{7.19}82.55 / 84.16 / 76.9287.94 / 86.67 / 81.4226.38‾\underline{26.38} / 29.33‾\underline{29.33} / 14.88‾\underline{14.88}
Full (Ours)14.27 / 28.27 / 9.5386.67 / 90.69 / 84.1189.99 / 89.88 / 85.0928.50 / 29.82 / 15.54

The full objective achieves the best performance on three criteria and matches gain-only on the remaining criterion, as both select the same noise subspace. These results further demonstrate the effectiveness of balancing denoising gain and target-information loss in identifying appropriate noise subspaces.

Response to W3 & Q3: Practical Applicability of NSDP

Automatic Non-Target Criterion Inference. To evaluate NSDP without predefined non-target criteria, we follow Appendix D: remove target-subspace components from image embeddings, apply k-means with KK clusters to the residual embeddings to select representative images, and use MLLNM (GPT-6) to infer candidate non-target criteria. NSDP then identifies the noise subspace from these candidates without manually specifying non-target criteria. The inferred criteria are summarized below:

DatasetTargetInferred Candidate Non-target CriteriaSelected Noise Criteria
CLEVR4-10kShapeColor, Texture, Count, SizeColor
CLEVR4-10kColorShape, Texture, Count, SizeShape
CardsNumberSuit, Ink Color, Illustration Style, Background Appearance, Card OrientationSuit, Illustration Style
CardsSuitNumber, Ink Color, Illustration Style, Background Appearance, Card OrientationNumber
FlowersSpeciesColor, Background Appearance, Bloom Stage, Flower Count, ViewpointBackground Appearance, Viewpoint, Flower Count
FlowersColorSpecies, Petal Shape, Background Appearance, Bloom Stage, Flower Count, ViewpointBackground Appearance, Viewpoint, Petal Shape

The inferred candidates are semantically plausible and include factors beyond the predefined criterion sets.

Comparison with Predefined Non-Target Criteria. We compare clustering performance using inferred and predefined non-target criteria against a baseline without NSDP. The results (NMI / ACC / ARI, %) are summarized below:

DatasetTargetw/o NSDPNSDP (Predefined)NSDP (Inferred)
CLEVR4-10kShape82.62 / 85.65 / 77.8586.67 / 90.69 / 84.1186.67 / 90.69 / 84.11
CLEVR4-10kColor87.87 / 85.06 / 80.3589.99 / 89.88 / 85.0989.99 / 89.88 / 85.09
CardsNumber38.10 / 40.76 / 24.0838.78‾\underline{38.78} / 44.45 / 28.40‾\underline{28.40}40.49 / 44.19‾\underline{44.19} / 28.73
CardsSuit39.65 / 70.34 / 39.0041.61 / 73.22 / 42.2041.61 / 73.22 / 42.20
FlowersSpecies99.27 / 99.14 / 99.03100.00 / 100.00 / 100.0099.30‾\underline{99.30} / 99.17‾\underline{99.17} / 99.11‾\underline{99.11}
FlowersColor60.22 / 70.62 / 51.8963.47 / 79.93 / 59.5761.29‾\underline{61.29} / 72.68‾\underline{72.68} / 52.93‾\underline{52.93}

Automatic inference matches the predefined setting on CLEVR4-10k (Shape and Color) and Cards (Suit), as NSDP selects the same noise subspaces in both settings. Notably, on Cards (Number), automatic inference identifies Illustration Style as an additional noise criterion and outperforms the predefined setting in NMI and ARI. This suggests that automatic inference can discover useful semantic factors beyond manually predefined criteria, potentially further improving performance. Although it underperforms the predefined setting on Flowers, it consistently improves over the baseline without NSDP. These results support the practical applicability of NSDP without predefined non-target criteria.

Sensitivity to the Number of Representative Clusters. We evaluate K∈{8,16,32,64}K\in\{8,16,32,64\} on CLEVR4-10k with Color as the target criterion and Cards with Number as the target criterion. For CLEVR4-10k, the inferred non-target criteria remain unchanged across all values of KK. For Cards, Card Orientation is additionally inferred at K=32K=32 and 6464. Despite this variation, NSDP selects the same noise subspaces and achieves identical downstream clustering performance across all tested values of KK, demonstrating robustness to the choice of KK in these settings.

Comments on the paper

1 comment

General Response

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback and thoughtful discussions. We would like to summarize the strengths highlighted by reviewers, the major concerns addressed during discussion, and the reviewer feedback after discussion.

1. Strengths Highlighted by Reviewers

We appreciate the reviewers' positive assessments of several key aspects of our work:

  • Well-motivated problem and clear formulation (44QC, KG5q). The reviewers recognized the importance of addressing semantic leakage in conditional representation learning, highlighting the clear problem formulation and supporting empirical evidence.

  • Well-designed and practical framework (44QC, gP8x, KG5q). The reviewers acknowledged the coherent design of CSBR and NSDP, their complementary roles in semantic purification, and the practical advantage of improving conditional representations through straightforward linear algebraic operations without additional VLM training.

  • Comprehensive experimental validation (44QC, gP8x, odza). The reviewers highlighted the extensive evaluations across customized clustering, few-shot classification, and retrieval, as well as the consistent improvements across multiple VLM backbones. The ablation studies and sensitivity analyses were also recognized as supporting the effectiveness and robustness of our method.

2. Major Concerns Addressed During Discussion

We conducted additional experiments and analyses to address the reviewers' major concerns:

  • Semantic purification mechanism and methodological justification (44QC, KG5q). We conducted dimension-matched control experiments to distinguish semantic purification from generic dimensionality reduction and random feature filtering. Additional analyses of adaptive truncation, denoising priority, and projection order further support the design and effectiveness of CSBR and NSDP.

  • Generalizability and practical applicability (44QC, KG5q, odza). We evaluated the performance of SP-CRL in scenarios without predefined non-target criteria. The results show consistent improvements over the baseline without NSDP across three datasets. Notably, on the Cards dataset, SP-CRL with automatically inferred non-target criteria even outperforms SP-CRL with predefined non-target criteria on certain metrics. Additionally, we introduced the more challenging Flowers v2 dataset with 40 species categories, where SP-CRL consistently outperforms all compared methods.

  • Computational efficiency and scalability (44QC, gP8x, odza). We provided theoretical complexity analyses and comprehensive runtime evaluations covering offline preprocessing, inference efficiency, and scalability with increasing numbers of candidate non-target criteria. The results demonstrate that SP-CRL incurs modest one-time preprocessing overhead, achieves efficient inference, and maintains manageable computational costs as the number of non-target criteria increases, demonstrating its computational efficiency and scalability.

3. Reviewer Feedback After Discussion

We are grateful that Reviewer 44QC increased their rating from 2 to 3, acknowledging that most major concerns had been adequately addressed. Reviewers gP8x and odza confirmed that their concerns had been resolved and maintained their positive ratings of 3 and 4, respectively. Meanwhile, Reviewer KG5q increased their soundness score from 2 to 3, acknowledging that some concerns had been addressed.

Overall, we believe that the additional experiments, analyses, and clarifications provided during the rebuttal further strengthen the technical soundness, empirical validity, and practical applicability of SP-CRL. We sincerely appreciate the Area Chair's and reviewers' time and consideration during the author-reviewer discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.