acceptodds
Under review as a conference paper at ICLR 2027

Aligning within Pair-Specific Subspaces for Noisy Correspondence Learning

Abstract

Noisy correspondence introduces semantically mismatched image-text pairs as positives, inducing erroneous attraction while degrading the supervision available from correct correspondences. Existing methods mitigate this problem through reliability estimation, robust optimization, and correspondence correction. However, pair-level weighting mainly controls the learning strength of pairs with different confidence levels, without directly identifying suitable directions for auxiliary alignment. Auxiliary constraints based on similarities between samples may also propagate correspondence errors to other pairs and interfere with learning reliable correspondences. We propose APS (Aligning within Pair-Specific Subspaces), which extracts multiview consistency evidence from each observed pair to determine the space for auxiliary alignment. Specifically, a frozen teacher model, obtained through reliability-weighted warm-up, evaluates the consistency of image-text view combinations. Their selected joint representations are then used to construct a pair-specific subspace. Correspondence confidence jointly weights the global contrastive objective and the auxiliary subspace objective, strengthening image-text consistency for higher-confidence pairs while suppressing unreliable auxiliary constraints. Experiments on Flickr30K, MS-COCO, and CC120K demonstrate the effectiveness and robustness of APS under synthetic and real-world correspondence noise.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
noisy correspondence learning, cross-modal retrieval, pair-specific subspaces, contrastive learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VRDXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Aligning within Pair-Specific Subspaces (APS) for robust image-text retrieval under noisy correspondences. The method first trains a teacher model using GMM-based confidence weighting to reduce the influence of unreliable image-text pairs. For each pair, the frozen teacher evaluates multiple augmented image and text views, selects the most consistent combinations, and constructs a pair-specific subspace using SVD. During student training, an auxiliary alignment loss encourages agreement within these subspaces, alongside the confidence-weighted global contrastive objective. Inference uses the student's original global representations without additional computation. Experiments on Flickr30K, MS-COCO, and CC120K show competitive retrieval performance under synthetic and real-world correspondence noise.

Critical strengths
  1. The paper addresses a practical problem in cross-modal retrieval and offers a reasonable motivation for complementing confidence-based sample weighting with more selective feature alignment. Instead of applying additional alignment constraints to the entire representation space, APS uses information from individual image-text pairs to determine where auxiliary alignment should take place. The overall approach is straightforward and fits naturally into the existing contrastive learning framework.

  2. The experimental analysis is reasonably thorough, particularly with the supplementary material. Beyond the main component ablations, the authors compare different view selection strategies, shuffled and batch-shared subspaces, and single-modality versus joint subspace construction. These experiments provide useful evidence that the improvements are related to the proposed view selection and pair-specific alignment, rather than simply adding an auxiliary loss.

  3. APS achieves competitive results across multiple datasets and noise levels, including the real-world CC120K benchmark. The paper also examines hyperparameter sensitivity and provides additional results on MS-COCO 5K. Another practical advantage is that the subspace construction is only used during training, so the method introduces no additional parameters or computational overhead at inference time.

Critical weaknesses
  1. Table 3 shows that GMM-based confidence weighting accounts for most of the improvement, while the additional gain from subspace alignment is relatively small. The supplementary ablations with Random-2, Bottom-2, and shared subspaces do provide useful evidence for the proposed design. Still, subspace projection and pair-conditioned feature selection have been explored in related works such as SSP and SimLAP. Since APS mainly combines these ideas with multi-view selection and auxiliary alignment, its methodological contribution appears relatively incremental.

  2. The paper constructs subspaces by selecting highly similar image-text views and applying SVD to their joint representations, but it is not clear whether these directions actually capture reliable shared semantics. For normalized image and text features, their sum is naturally orthogonal to their difference, so projection along these directions can reduce cross-modal disagreement without necessarily identifying meaningful shared information. Moreover, with the default setting of (m=r=2), SVD simply orthogonalizes the span of two selected representations. Although the supplementary ablations support the effectiveness of this construction, they offer limited insight into what information the resulting subspaces actually preserve.

  3. The SOTA results in Table 1 are taken from different papers using different encoders and training configurations, and even the CLIP baseline differs considerably between Tables 1 and 3. The supplementary material also refers to a binary-weight GMM baseline evaluated at epoch 5, while the main text describes GMM-soft/GMM+CW and APS is evaluated at epoch 10. The relationship between these configurations is not entirely clear. Given the relatively small improvements over recent methods, these differences make it difficult to attribute the reported gains solely to APS.

  4. Most synthetic experiments rely on randomly shuffled image-text correspondences, whereas real mismatches may share much of their semantic content and differ only in specific attributes or relationships. This is particularly relevant to APS, since selecting highly similar views could preserve superficial semantic overlap while overlooking the details responsible for the mismatch. Although CC120K provides an evaluation under real-world noise, the comparisons there are relatively limited, leaving the effectiveness of the proposed view selection under more challenging correspondence errors less clear.

Questions for authors
  1. Appendix A.2 mentions a binary-weight GMM baseline evaluated at epoch 5, while the main text refers to GMM-soft and GMM+CW. Are these the same experiments? Please clarify the weighting scheme and training configuration, and report how the GMM baseline performs when trained for the same number of epochs as APS.

  2. Tables S2–S4 provide useful comparisons of different view selection and subspace construction strategies. What happens if pair-specific subspaces are constructed directly from the original teacher features without multi-view selection? A comparison with simple teacher-based alignment would also help isolate the contribution of the proposed construction. How much additional training time does the complete procedure require, including warm-up and offline subspace construction?

  3. Have the authors examined the projected feature norms, singular value distributions, or auxiliary alignment gradients for clean and noisy pairs? Such analyses could help explain whether the selected directions preserve meaningful shared information and whether the constructed subspaces remain stable when the joint representations are nearly linearly dependent.

  4. Both the confidence values and subspace bases remain fixed during student training. What happens when the teacher initially assigns an incorrect confidence or the student representations change substantially? Also, low-confidence samples still contribute as negatives to other samples' contrastive losses. Does this affect the effectiveness of confidence-based noise suppression?

  5. How does APS perform on incorrect image-text pairs that share most of their semantic content but differ in attributes, object counts, or spatial relationships? Examining the selected views for these cases, alongside those of correctly matched pairs, would help clarify what the proposed multi-view consistency criterion actually captures.

7 comments
AuthorsOpenReview ↗

Response to Reviewer VRDX — Part 1/5

Thank you for recognizing the motivation, subspace comparisons, and inference efficiency, and for your detailed suggestions on basis construction, attribution of the gains, and fine-grained mismatches. We have reorganized the component ablations across three datasets and added diagnostics on captions not used in basis construction, singular values, auxiliary gradients, per-epoch updates, and fine-grained discrimination. These results support using each pair's own multi-view evidence to determine auxiliary alignment directions and correspondence confidence to regulate supervision strength, while preserving efficient retrieval with global representations.

Responses to Weaknesses

W1. Additional contribution of the subspace term and relation to SSP and SimLAP

APS addresses learning from potentially incorrect image–text correspondences by constructing an auxiliary alignment space from each pair's own multi-view evidence. Correspondence confidence jointly regulates the global and subspace objectives. The evidence used to construct the space, the level at which it is defined, and its role in training are central to this design.

SSP uses local visual features to construct a shared visual subspace and class-specific language subspaces, improving image–text alignment through projection for few-shot classification. SimLAP uses class-pair-conditioned learnable gating for contrastive learning between visual samples in the corresponding spaces. APS selects evidence from augmented views of each observed image–text pair, constructs a fixed pair-specific subspace, and uses it for auxiliary positive-pair alignment. Global contrastive learning and final retrieval remain in a common representation space. We will clarify these differences in the evidence used for space construction and the role of the resulting spaces in training.

The existing controls support constructing subspaces at the pair level. On Flickr30K with 40% noise, using the same Top-2 evidence, pair-specific subspaces achieve 553.2 RSum, compared with 551.4 for a batch-shared subspace. Keeping the full collection of subspaces while shuffling their assignments to pairs yields 548.2. The corresponding gaps of 1.8 and 5.0 RSum points support associating the auxiliary space with the content of the current pair.

The updated ablations further show that APS improves over GMM+CW by 1.5, 1.8, and 2.8 RSum points at the three noise rates on Flickr30K, by 1.1, 1.1, and 0.8 points on COCO 1K, and by 1.6 points on CC120K. Full results are provided in Q1. These matched comparisons identify the additional contribution of the subspace term beyond continuous weighting and extend the evidence across datasets and to real-world correspondence noise.

W2. Orthogonality of the sum and difference and information retained by the subspace

APS uses the selected joint directions to provide pair-relevant auxiliary constraints for the student. The geometric property you identified holds: for unit vectors v,tv,t,

(v+t)⊤(v−t)=0.(v+t)^\top(v-t)=0.

Under the default setting m=r=2m=r=2, when the basis construction matrix has full rank, SVD provides an orthonormal basis for the span of the two selected directions. In the dedicated comparison, GS and SVD achieve 553.2±0.5553.2\pm0.5 and 553.2±0.3553.2\pm0.3 RSum, respectively, consistent with the equivalence of the auxiliary objective within the same subspace.

To examine what information this space retains, we construct the basis using a separate original caption for each of 900 test images. We then evaluate correct captions not used in basis construction, captions with a single factual mismatch, and captions from different images. The mean text projection norms in the Top-2 subspaces are:

Candidate captionMean projection norm
Correct caption not used in basis construction0.489
Fine-grained mismatched caption with a single factual change0.471
Caption from a different image0.302

The mean projection norm of correct captions exceeds that of captions from different images by 0.187. Fine-grained mismatched captions also retain substantial components, as they preserve much of the image-relevant content. These results support the retention of content relevant to the current pair. Together with the matched training comparisons in Q1, they support the use of pair-relevant subspaces for effective auxiliary supervision. The trained global representations also achieve 69.5% accuracy in the fine-grained diagnostic, compared with 68.2% for GMM+CW, as reported in Q5.

In the revision, we will describe the role of the subspace more precisely as “retaining representation components relevant to the current pair to provide selective auxiliary alignment constraints for the student,” supported by the geometric and empirical analyses.

AuthorsOpenReview ↗

Response to Reviewer VRDX — Part 2/5

W3. Comparisons with prior work, internal CLIP baselines, and training settings

The CLIP results in Table 1 are taken from the fine-tuning baseline reported by NPC, using ViT-B/32, 5 training epochs, and a batch size of 256. The CLIP baseline in Table 3 is fine-tuned under our protocol, using the same backbone, 10 training epochs, and a batch size of 128. These results serve to present published benchmark performance and support internal component analysis, respectively.

To clarify attribution of the gains, we have reorganized and extended the binary GMM, GMM+CW, and APS comparisons under a common training protocol, reporting means and standard deviations over three seeds. The revision will identify the sources and configurations of the published results and use matched internal controls to quantify the contributions of continuous weighting and the subspace term. The corrections and detailed results are provided in Q1.

W4. Randomly permuted correspondences and semantically similar fine-grained mismatches

APS selects highly consistent views within each pair to provide content-relevant constraints for auxiliary training. To evaluate the trained model's ability to distinguish local factual changes, we have added a diagnostic set with single-fact caption edits covering color, count, and spatial relations, with 300 test images per category and 900 images in total.

For models trained on Flickr30K with 40% random correspondence noise, APS achieves 69.5% overall two-choice accuracy, exceeding GMM+CW by 1.3 percentage points, with higher mean accuracy in all three categories. This diagnostic directly evaluates discrimination between complete captions using the student's global representations, providing additional evidence for APS under semantically similar mismatches. Detailed results are given in Q5.

Responses to Questions

Q1. Configurations of binary GMM, GMM-soft, and GMM+CW, and results under the same training schedule

GMM-soft and GMM+CW denote the same continuous-weighting baseline. Binary GMM uses binary 0/1 weights derived from the reliability estimates. APS adds pair-specific auxiliary subspace alignment to GMM+CW, with confidence weighting applied to both losses.

Thank you for identifying the reporting issue in the original manuscript. The original table mislabeled the epoch-5 GMM+CW results as GMM and omitted the corresponding binary GMM results. We have corrected this mapping and rerun the control that uses binary weights during student training while retaining the auxiliary subspace term. We explicitly label this configuration “GMM (binary) + subspace.”

The updated comparisons use the same 10-epoch training protocol as APS, with seeds 99, 1994, and 1995 on an A100 GPU. We report mean RSum and standard deviation at epoch 10. The component ablations on Flickr30K and CC120K are:

ConfigurationFlickr30K 20%Flickr30K 40%Flickr30K 60%CC120K real-world noise
CLIP fine-tuning under matched settings542.0±0.4542.0\pm0.4537.7±0.9537.7\pm0.9525.4±0.8525.4\pm0.8502.6±1.0502.6\pm1.0
GMM (binary)550.9±0.2550.9\pm0.2549.1±0.0549.1\pm0.0544.7±0.5544.7\pm0.5511.8±0.6511.8\pm0.6
GMM+CW553.1±0.1553.1\pm0.1551.4±0.3551.4\pm0.3546.9±0.8546.9\pm0.8520.8±1.0520.8\pm1.0
GMM (binary) + subspace553.6±0.6553.6\pm0.6551.4±0.4551.4\pm0.4548.2±0.3548.2\pm0.3—
GMM+CW + global-space auxiliary alignment553.0±0.2553.0\pm0.2551.7±0.2551.7\pm0.2546.9±1.1546.9\pm1.1520.6±0.7520.6\pm0.7
APS554.6±0.2554.6\pm0.2553.2±0.5553.2\pm0.5549.7±0.1549.7\pm0.1522.4±1.1522.4\pm1.1

“—” indicates that we do not report a CC120K result for the binary-weighted subspace configuration in this comparison. On Flickr30K with 40% noise, global-space auxiliary alignment uses λ=0.01\lambda=0.01, selected on the validation set, while APS uses its default λ=0.7\lambda=0.7.

The corresponding results on COCO 1K are:

Configuration20% noise40% noise60% noise
GMM (binary)533.8±0.9533.8\pm0.9531.8±0.2531.8\pm0.2527.9±0.0527.9\pm0.0
GMM+CW535.1±0.8535.1\pm0.8533.2±0.2533.2\pm0.2529.7±0.1529.7\pm0.1
APS536.2±0.7536.2\pm0.7534.3±0.2534.3\pm0.2530.5±0.2530.5\pm0.2

These comparisons support the complementary contributions of continuous weighting and auxiliary subspace alignment. Across all three datasets, GMM+CW achieves higher mean RSum than binary GMM, and APS provides further gains over GMM+CW. On Flickr30K, full APS also improves over “GMM (binary) + subspace” by 1.0, 1.8, and 1.5 RSum points. Compared with global-space auxiliary alignment, APS achieves gains of 1.6, 1.5, and 2.8 points at the three noise rates on Flickr30K and 1.8 points on CC120K. We will make the configuration names, training schedules, and statistical reporting consistent throughout the revision.

AuthorsOpenReview ↗

Response to Reviewer VRDX — Part 3/5

Q2. Bases constructed from original teacher features, direct teacher alignment, and total training cost

The multi-view construction in APS provides multiple content combinations for each pair. The teacher then selects highly consistent combinations to determine auxiliary alignment directions. To evaluate this construction, we have added two teacher-based controls:

  • Two-dimensional subspace from original teacher features: We use the teacher features of the unaugmented image and complete caption as the two rows of a matrix, construct a two-dimensional basis via SVD, and apply the same projected cosine alignment loss to the student.
  • Direct cross-modal teacher alignment: We align the student image feature with the frozen teacher text feature, and the student text feature with the frozen teacher image feature. The auxiliary term is the average of the two cosine losses.

On Flickr30K with 40% noise, all three auxiliary configurations use the same confidence weighting and λ=0.7\lambda=0.7, with K=3,m=r=2K=3,m=r=2 for APS. Epoch-10 results over three seeds in this dedicated comparison are:

MethodRSum
GMM+CW551.4±0.4551.4\pm0.4
Two-dimensional subspace from original teacher features550.2±0.4550.2\pm0.4
Direct cross-modal teacher alignment548.7±0.1548.7\pm0.1
Multi-view APS553.2±0.7553.2\pm0.7

APS improves over the subspace constructed from original teacher features and direct teacher alignment by 3.0 and 4.5 RSum points, respectively, and over GMM+CW by 1.8 points. These results support the complete design of constructing pair-specific auxiliary spaces through multi-view selection.

For efficiency, APS performs multi-view encoding and basis construction once, offline. The accumulated training-stage times on a single A100 GPU, averaged over three seeds, are:

MethodTotal training-stage time, including warm-up and offline construction
GMM+CW2034.5±13.52034.5\pm13.5 s
APS2348.6±13.32348.6\pm13.3 s

Both totals include approximately 578 seconds of teacher warm-up, initialization, and final checkpoint saving. Validation and testing are timed separately. APS adds approximately 314 seconds, or 15.4%. Multi-view generation and encoding take 306.0 seconds, and basis construction with related I/O takes 13.8 seconds. This one-time precomputation supports subsequent auxiliary training. Inference directly uses the student's global features, with zero additional parameters or FLOPs.

Q3. Projection norms, singular values, auxiliary gradients, and near-linear dependence

In APS, the subspace determines the directions of the auxiliary constraints, while confidence determines their strength for each pair. To examine these two components, we conduct offline diagnostics on Flickr30K with 40% synthetic noise over three seeds. We distinguish clean and mismatched pairs using the ground-truth correspondence labels recorded during noise injection.

First, using the epoch-10 student and a fixed selection of training pairs, we measure projection norms for additional captions and auxiliary gradient norms:

Diagnostic measureClean pairsMismatched pairs
Image projection norm0.739±0.0010.739\pm0.0010.608±0.0050.608\pm0.005
Caption projection norm0.777±0.0010.777\pm0.0010.630±0.0060.630\pm0.006
Projection norm of another correct caption0.617±0.0020.617\pm0.0020.430±0.0060.430\pm0.006
Projection norm of a caption from a different image0.277±0.0050.277\pm0.0050.316±0.0060.316\pm0.006
Auxiliary gradient norm before weighting by qq0.1636±0.00180.1636\pm0.00180.3510±0.01440.3510\pm0.0144
Auxiliary gradient norm after weighting by qq0.1590±0.00180.1590\pm0.00180.0026±0.00040.0026\pm0.0004

The gradient statistic is the joint norm of the gradients of the auxiliary loss with respect to the normalized image and text output features. It includes λ\lambda and omits the common batch-level denominator for all samples.

For clean pairs, another correct caption that was not used in basis construction has a mean projection norm 0.340 higher than a caption from a different image, supporting the retention of representations relevant to the same image. After confidence weighting, clean pairs retain approximately 97% of their mean gradient magnitude, while the mean gradient norm of mismatched pairs falls to approximately 1.6% of that of clean pairs. These results support the role of confidence weighting in concentrating auxiliary supervision on reliable pairs.

Reviewer_tGscOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

APS addresses image-text retrieval with noisy correspondence by adding a pair-specific alignment loss to confidence-weighted contrastive training. After a short warm-up, a frozen CLIP teacher scores combinations of augmented image and text views. The highest-scoring combinations are combined into joint representations, whose span defines an auxiliary subspace for each observed pair. The student is trained to align the projected image and text features, with GMM-derived confidence weighting both losses. Retrieval uses the student's global features, so the auxiliary spaces add no inference cost. Experiments cover synthetic noise on Flickr30K and MS-COCO and real-world noise on CC120K. The supplement includes controls for view selection, subspace assignment, shared bases, and modality evidence, as well as an analysis of the one-dimensional loss.

Critical strengths
  1. The subspace controls are useful. In Table S2, Top-2 selection consistently outperforms random selection, bottom-ranked selection, and shuffled assignments across all three noise levels. The shared-basis and single-modality controls in Tables S3-S4 also help isolate which parts of the construction matter. These experiments make a reasonable case that the specific auxiliary space affects performance.
  2. The improvement over the internal GMM+CW baseline is consistent on Flickr30K: 2.6, 2.7, and 3.4 RSum at 20%, 40%, and 60% noise. This is a more informative comparison than the gap over published CLIP results. The method also keeps retrieval simple, using only the student encoders and global cosine similarity.
Critical weaknesses
  1. I am not yet convinced that the gains establish a sufficiently strong contribution across datasets. On COCO 1K, APS leads the strongest listed method by 1.3 and 0.4 RSum at 20% and 40% noise, and trails DCL by 0.4 at 60%. The CC120K gain over HaNa is 1.2. These are small margins between results obtained under different training settings, without enough uncertainty information to judge their stability. The stronger internal comparison against GMM+CW is only shown on Flickr30K. Corresponding controls on COCO and CC120K would help establish how much the proposed subspace term contributes there.
  2. The ablation descriptions do not agree. Table 3 and its discussion identify the 544.2/540.7/531.3 row as removing continuous weighting during student training, but Table S1 labels the same results as "GMM+CW+Auxiliary Subspaces." The baseline reporting also needs clarification: Table 3 gives an exception to three-run reporting for the GMM baseline, while Appendix A.2 refers to a two-run binary-weight baseline at epoch 5 without clearly mapping it to the displayed rows. These may be reporting errors, but they affect the evidence used to separate the effects of weighting and subspace alignment. I would want them resolved before relying on those conclusions.
Questions for authors
  1. Which configuration produced the 544.2/540.7/531.3 results? Was continuous weighting used during student training? Please reconcile Table 3 and Table S1, and identify the run count and evaluation epoch for each GMM baseline.
  2. Do you have matched-setting GMM+CW versus APS results on COCO and CC120K, including run-to-run variability? These would directly affect my assessment of the additional contribution.
  3. Was the auxiliary coefficient tuned separately for the global-space alignment baseline? Does its performance deficit remain with a comparable validation-based sweep?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tGsc — Part 1/3

Thank you for recognizing the subspace controls and the ablation analysis on Flickr30K, and for highlighting the need to clarify the contribution across datasets, experimental configurations, and baseline tuning. APS determines auxiliary alignment directions from each pair's own multi-view evidence and combines this supervision with confidence weighting. Our additional experiments show improvements in mean RSum from the subspace term on Flickr30K, COCO, and CC120K. We have also corrected the configuration labels and added component ablations under matched settings.

Responses to Weaknesses

W1. Improvements across datasets and training settings for comparisons with prior work

Our experiments evaluate both overall retrieval performance on standard benchmarks and the additional contribution of the subspace term under a common training protocol. Tables 1 and 2 report published results for the compared methods, retaining their original encoders and training settings. APS uses CLIP ViT-B/32, a batch size of 128, and 10 training epochs: 2 epochs of teacher warm-up followed by 8 epochs of student training. We report epoch-10 results over three seeds. The revision will explicitly identify the sources and training settings of the compared results.

To address attribution of the gains, we have added GMM+CW versus APS comparisons under the same training protocol, reporting means and standard deviations. The results show positive gains at all three noise rates on COCO and under real-world correspondence noise on CC120K; the values are provided in Q2. These matched comparisons directly support the contribution of APS across datasets, while inference continues to use only the student's global features.

W2. GMM+CW controls on COCO and CC120K

GMM+CW and APS use the same confidence-weighted training framework. APS adds an auxiliary constraint defined by the pair's multi-view evidence, so the comparison directly measures the contribution of the subspace term beyond confidence weighting.

We have extended this comparison to COCO and CC120K. Across three seeds, APS improves mean RSum on COCO 1K by 1.1, 1.1, and 0.8 points at 20%, 40%, and 60% noise, respectively, and by 1.6 points on CC120K. Together with the 1.5, 1.8, and 2.8-point gains at the three noise rates on Flickr30K, these results support pair-specific auxiliary alignment across datasets and noise conditions. Means and standard deviations are provided in Q1 and Q2.

W3. Inconsistent ablation labels in Tables 3 and S1

Thank you for identifying the inconsistent configuration labels. The ablation intended to examine the removal of continuous weighting during student training was still labeled with “CW” in Table S1, creating ambiguity. We apologize for this reporting error.

To make the configuration explicit, we have rerun the control with binary 0/1 weights during student training while retaining pair-specific auxiliary subspace alignment. We consistently label this configuration “GMM (binary) + subspace.” Its updated RSum values on Flickr30K are 553.6, 551.4, and 548.2 at the three noise rates. These rerun results with the explicitly specified configuration will replace the corresponding row in the original manuscript.

Comparing this control with full APS evaluates continuous weighting when the subspace term is present. Comparing GMM+CW with full APS evaluates the additional contribution of the subspace term. We will make the configuration names and results consistent across Table 3, Table S1, and the text.

W4. Number of runs and evaluation epochs for the GMM baselines

Thank you for pointing out the baseline reporting issue. Our audit found that the epoch-5 GMM+CW results were mislabeled as GMM, while the corresponding binary GMM results were omitted. We have corrected this mapping and reorganized the component ablations.

The updated comparisons distinguish binary GMM, GMM+CW, binary GMM + subspace, and full APS. All use seeds 99, 1994, and 1995, follow the same 10-epoch training protocol as APS, and are evaluated at epoch 10. Q1 provides the configuration definitions and results so that the contributions of continuous weighting and auxiliary subspace alignment can be checked directly.

AuthorsOpenReview ↗

Response to Reviewer tGsc — Part 2/3

Responses to Questions

Q1. Configuration of the original 544.2/540.7/531.3 row and reporting of the GMM baselines

The original row was intended to examine student training without continuous weighting, but its labels were inconsistent between Tables 3 and S1. We have rerun the control with binary 0/1 weights during student training while retaining auxiliary subspace alignment. It is now explicitly labeled “GMM (binary) + subspace,” with updated RSum values of 553.6/551.4/548.2.

GMM-soft and GMM+CW denote the same continuous-weighting baseline, which uses GMM-derived correspondence confidence to weight the global contrastive loss. Binary GMM uses binary weights; “GMM (binary) + subspace” retains the subspace term under binary weighting; full APS uses continuous confidence to weight both the global and subspace objectives.

The original table mislabeled the epoch-5 GMM+CW results as GMM and omitted the binary GMM results. The corrected comparisons consistently use three seeds, 10 training epochs, and evaluation at epoch 10:

ConfigurationStudent-stage weightsAuxiliary subspace alignmentSeedsEvaluation epoch
GMM (binary)Binary 0/1No310
GMM+CW (GMM-soft)Continuous confidence weightsNo310
GMM (binary) + subspaceBinary 0/1Yes310
APSContinuous confidence weightsYes310

The seeds are 99, 1994, and 1995, and the experiments are run on an A100 GPU. Mean RSum and standard deviation on Flickr30K are:

Configuration20% noise40% noise60% noise
GMM (binary)550.9±0.2550.9\pm0.2549.1±0.0549.1\pm0.0544.7±0.5544.7\pm0.5
GMM+CW553.1±0.1553.1\pm0.1551.4±0.3551.4\pm0.3546.9±0.8546.9\pm0.8
GMM (binary) + subspace553.6±0.6553.6\pm0.6551.4±0.4551.4\pm0.4548.2±0.3548.2\pm0.3
APS554.6±0.2554.6\pm0.2553.2±0.5553.2\pm0.5549.7±0.1549.7\pm0.1

Continuous weighting improves over binary GMM by 2.2, 2.3, and 2.2 RSum points. Adding the subspace term to GMM+CW yields further gains of 1.5, 1.8, and 2.8 points. With the subspace term retained in both configurations, full APS also outperforms binary weighting by 1.0, 1.8, and 1.5 points. These comparisons support the complementary contributions of continuous weighting and pair-specific auxiliary alignment.

Q2. Matched-setting results and run-to-run variability on COCO and CC120K

We have completed these additional comparisons. Within each dataset, we use a common training protocol and evaluate the effect of adding pair-specific auxiliary subspace alignment to GMM+CW. Mean RSum and standard deviation over three seeds are:

Dataset and evaluationNoise rateGMM (binary)GMM+CWAPSAPS − GMM+CW
COCO 1K20%533.8±0.9533.8\pm0.9535.1±0.8535.1\pm0.8536.2±0.7536.2\pm0.7+1.1
COCO 1K40%531.8±0.2531.8\pm0.2533.2±0.2533.2\pm0.2534.3±0.2534.3\pm0.2+1.1
COCO 1K60%527.9±0.0527.9\pm0.0529.7±0.1529.7\pm0.1530.5±0.2530.5\pm0.2+0.8
CC120KReal-world noise511.8±0.6511.8\pm0.6520.8±1.0520.8\pm1.0522.4±1.1522.4\pm1.1+1.6

APS achieves higher mean RSum in all four settings. The gains cover all three noise rates on COCO and extend to real-world correspondence noise on CC120K. These results support the subspace term as an effective complement to confidence-weighted training.

AuthorsOpenReview ↗

Response to Reviewer tGsc — Part 3/3

Q3. Separate tuning of the global-space alignment baseline

APS uses the evidence within each pair to focus auxiliary alignment on representation directions supported by that pair. The original Flickr30K experiments used λ=0.7\lambda=0.7 for both global-space and subspace alignment to compare the alignment spaces at a fixed auxiliary loss weight. Following your suggestion, we have additionally tuned the global-space alignment baseline using a separate validation-based search.

On Flickr30K with 40% noise, we search λ∈{0.005,0.01,0.05,0.1,0.5}\lambda\in\{0.005,0.01,0.05,0.1,0.5\}, retaining λ=0\lambda=0 and the original default λ=0.7\lambda=0.7 as references. We select a common value of λ=0.01\lambda=0.01 based on mean validation RSum across three seeds. This configuration achieves 551.7±0.2551.7\pm0.2 test RSum at epoch 10. APS, using its default λ=0.7\lambda=0.7, achieves 553.2±0.5553.2\pm0.5, a mean improvement of 1.5 RSum points.

The global-space and subspace alignment results across settings are summarized below. The separate parameter search described above applies to the Flickr30K 40% noise column.

ConfigurationFlickr30K 20%Flickr30K 40%Flickr30K 60%CC120K real-world noise
GMM+CW + global-space auxiliary alignment553.0±0.2553.0\pm0.2551.7±0.2551.7\pm0.2546.9±1.1546.9\pm1.1520.6±0.7520.6\pm0.7
APS554.6±0.2554.6\pm0.2553.2±0.5553.2\pm0.5549.7±0.1549.7\pm0.1522.4±1.1522.4\pm1.1
APS − global-space auxiliary alignment+1.6+1.5+2.8+1.8

The comparison after separate tuning, together with the mean gains in the other settings, supports determining auxiliary alignment directions from pair-specific evidence. We will include the search range, validation-based selection rule, and auxiliary loss weights for each configuration in the revision.

Thank you for these suggestions on evaluating the contribution. The additional experiments support the complementary benefits of continuous weighting and pair-specific subspace alignment across datasets and under real-world correspondence noise. We will update the configuration names, run details, and statistical reporting to present these contributions clearly.

I thank the authors for the thorough rebuttal and additional experiments. The configuration labeling and baseline reporting issues have been properly addressed. Nevertheless, the matched-setting improvements remain small—0.8–1.1 RSum on COCO and 1.6 on CC120K—and are comparable to the corresponding standard deviations. So I maintain my original rating of weak rejection.

Reviewer_zJP3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

An Aligning within Pair-Specific Subspace (APS) method is proposed to solve the image-text misalignment problem in real world dataset where the paired samples in contrastive learning step might include the noisy pairs. The GMM is used to give different weights to different confidence level pairs for model training, and the SVD method is used to generate the subspace for each pair to enforce the sample level feature consistence for text-image pair. Ablation studies on different datasets and different noisy levels were conducted to strengthen the soundness of the paper.

Critical strengths
  1. In this paper, the authors tried to solve the noisy example problem in image-text contrastive learning task. Two existing techniques, sample pair weighting and subspace alignment, are used to address this problem.
  2. Ablation studies on different datasets and different noisy levels were conducted to strengthen the soundness of the paper.
Critical weaknesses
  1. The contrastive loss does not mean the high text-image pair correctness. The authors should provide more evidence to prove this.
  2. The authors did not provide sufficient evidence to show the effectiveness of the selected subspaces.
  3. The computational cost of the training part does not include time cost for teacher warm-up and multi-view subspace construction.
Questions for authors

1.The necessity of SVD is unclear. There are other subspace creation methods, e.g. Gram-Schmidt. The authors should clarify whether simpler orthogonalization methods would achieve similar results. 2. The low-confidence samples can still contribute to the negative terms in the contrastive loss of other samples. The authors should investigate on the effects of the negative sample pairs.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zJP3 — Part 1/3

Thank you for recognizing our experimental analysis and for your suggestions regarding confidence estimation, the effectiveness of the selected subspaces, and computational cost. In noisy correspondence learning (NCL), APS uses correspondence confidence to weight the supervision for each image–text pair and constructs an auxiliary alignment subspace from the pair's own multi-view evidence. Our existing and additional experiments support this design through confidence diagnostics, information retention for captions not used in basis construction, and retrieval performance. We have also measured training time including teacher warm-up and offline subspace construction. We address your concerns below.

Responses to Weaknesses

W1. Contrastive loss as an indicator of correspondence reliability

APS estimates correspondence reliability from the loss distribution during warm-up, allowing correctly matched image–text pairs to receive greater training weights. Specifically, we fit a two-component Gaussian mixture model (GMM) to the per-pair contrastive losses and use the posterior probability of the lower-mean component as a continuous confidence score. This score controls the contributions of both the global contrastive loss and the auxiliary subspace alignment loss. The design builds on the tendency of correctly matched pairs to be easier for the model to fit.

To evaluate this premise, we assess the GMM confidence scores on Flickr30K with 40% synthetic correspondence noise, using the ground-truth correspondence labels recorded during noise injection. The mean AUROC over three random seeds is 0.9988. Under this noise setting, the confidence scores derived from the warm-up losses effectively distinguish clean from mismatched pairs, providing direct empirical support for continuous confidence weighting.

W2. Effectiveness of the selected subspaces

APS aims to concentrate auxiliary alignment on representation directions supported by the current image–text pair. The teacher first selects view pairs with high cross-modal consistency, and their joint representations define the pair-specific subspace. The retrieval ablations in the supplementary material and our additional caption diagnostics support this design in three ways.

First, selecting highly consistent view pairs improves retrieval. Table S2 keeps the number of selected view pairs, subspace dimension, orthogonalization method, and confidence weighting fixed. At 20%, 40%, and 60% noise on Flickr30K, Top-2 outperforms Random-2 by 2.2, 2.2, and 2.0 RSum points, and Bottom-2 by 4.1, 4.0, and 3.7 points, respectively. These results support using the teacher's consistency scores to select the evidence for basis construction.

Second, assigning each subspace to its corresponding pair matters. Keeping the entire collection of subspaces while shuffling their assignments to pairs reduces RSum by 6.6, 5.0, and 6.5 points at the three noise rates. At 40% noise, constructing a batch-shared subspace from the same Top-2 evidence yields 551.4 RSum, compared with 553.2 for pair-specific APS. These controls support determining the auxiliary alignment directions from the content of the current pair.

Third, the pair-specific subspace retains more information from another caption of the same image that was not used in basis construction. On Flickr30K with 40% noise, we select clean pairs from a fixed subset of 2,048 training pairs and restrict the comparison to samples for which both subspaces are valid. Using the epoch-10 student, we encode another ground-truth caption of the same image and a caption from a different image, normalize their features, and project them onto the correctly assigned and shuffled subspaces. The mean projection norms, ∥B⊤t∥2\lVert B^\top t\rVert_2, over three seeds are:

Caption evaluatedCorrectly assigned subspaceShuffled subspace
Another caption of the same image, not used in basis construction0.6170.253
Caption from a different image0.2770.279

The correctly assigned subspace increases the mean projection norm of the additional same-image caption by 0.364, while the norms of captions from different images remain nearly unchanged. This supports the selective retention of information relevant to the current image. Together with the retrieval ablations, these results support both high-consistency view selection and pair-specific auxiliary alignment.

AuthorsOpenReview ↗

Response to Reviewer zJP3 — Part 2/3

W3. Training cost including teacher warm-up and subspace construction

APS limits additional training cost through one-time offline computation. After warm-up, the teacher is frozen, multi-view encoding and subspace construction are performed once, and the resulting bases are reused throughout student training. Retrieval directly uses the student's global features.

The approximately 0.6% overhead reported in the original manuscript refers to student training and its associated evaluation. Following your suggestion, we additionally measured teacher warm-up and offline subspace construction on Flickr30K with 40% noise, using a single A100 GPU and three seeds. The table reports the mean and standard deviation of the time accumulated over the training stages in each run, including initialization and final checkpoint saving. Validation and testing are timed separately.

StageGMM+CW (s)APS (s)
Teacher warm-up578.1±1.6578.1\pm1.6578.1±1.6578.1\pm1.6
Multi-view generation and encoding0306.0±2.6306.0\pm2.6
Basis construction and I/O013.8±1.313.8\pm1.3
Student training stage1456.3±11.91456.3\pm11.91450.6±18.91450.6\pm18.9
Total training-stage time2034.5±13.52034.5\pm13.52348.6±13.32348.6\pm13.3

Including these stages, GMM+CW and APS take approximately 33.9 and 39.1 minutes, respectively. APS adds 314.1±26.8314.1\pm26.8 seconds on average, corresponding to 15.4%±1.4%15.4\%\pm1.4\% overhead. Multi-view generation and encoding account for approximately 96% of the additional precomputation time; basis construction and related I/O take approximately 13.8 seconds. Student training times are comparable, and peak allocated GPU memory is 7.6 GiB for both methods. APS adds no parameters or FLOPs at inference.

These measurements characterize the efficiency of APS: approximately 5.2 minutes of one-time additional training cost enable pair-specific auxiliary supervision while preserving student-stage memory usage and inference efficiency. We will include the stage-wise breakdown and timing protocol in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer zJP3 — Part 3/3

Responses to Questions

Q1. Necessity of SVD and comparison with simpler orthogonalization

We use singular value decomposition (SVD) because it provides a unified procedure for constructing an orthonormal basis and extracting dominant directions when the number of input directions exceeds the target dimension. Under the default setting m=r=2m=r=2, when the two selected joint representations are linearly independent, Gram–Schmidt (GS) orthogonalization and SVD produce orthonormal bases for the same subspace. The projected coordinates differ only by a shared orthogonal transformation, so their cosine similarity and the auxiliary alignment objective are identical.

On Flickr30K with 40% noise, GS and SVD achieve 553.2±0.5553.2\pm0.5 and 553.2±0.3553.2\pm0.3 RSum, respectively, in this dedicated comparison. Their comparable mean performance is consistent with the equivalence above.

When m>rm>r, SVD uses all selected joint representations to identify the rr-dimensional subspace with the largest total projected energy. With each row of HiH_i representing a selected joint feature, this construction solves:

max⁡B⊤B=Ir∥HiB∥F2.\max_{B^\top B=I_r}\lVert H_iB\rVert_F^2.

In this comparison, GS orthogonalizes the inputs in order and retains the first rr directions, whereas SVD extracts the dominant directions from all selected representations. The results are:

mmrrGS RSumSVD RSum
22553.2±0.5553.2\pm0.5553.2±0.3553.2\pm0.3
42552.8±0.5552.8\pm0.5553.3±0.5553.3\pm0.5
62552.7±0.4552.7\pm0.4553.3±0.5553.3\pm0.5
64552.8±0.2552.8\pm0.2553.1±0.2553.1\pm0.2
82552.9±0.2552.9\pm0.2553.3±0.5553.3\pm0.5
84552.6±0.1552.6\pm0.1552.9±0.1552.9\pm0.1

SVD achieves higher mean RSum in all five settings with m>rm>r. These results support SVD as a unified basis construction procedure: it represents the selected subspace under the default setting and uses all selected evidence to extract dominant directions when dimensionality reduction is needed. We will clarify these two roles in the revision.

Q2. Effect of retaining low-confidence samples as negatives for other anchors

APS retains the images and texts from low-confidence pairs as negative candidates for other anchors to preserve useful cross-sample discrimination while downweighting unreliable pair supervision. The confidence qjq_j measures the reliability of the observed pair (Ij,Tj)(I_j,T_j), whereas whether TjT_j is a valid negative for another image IiI_i depends on the correspondence between IiI_i and TjT_j. An image or text from a low-confidence pair can therefore still provide useful contrastive information.

On Flickr30K with 40% noise, we compare retaining all negative candidates with masking low-confidence negative candidates in the denominator of the contrastive loss. The results over three seeds are:

Negative-candidate strategyRSum
Retain all negative candidates (APS)553.21±0.75553.21\pm0.75
Mask low-confidence negative candidates552.34±0.45552.34\pm0.45

Retaining all negative candidates improves mean RSum by 0.87 points, supporting the use of confidence weighting to regulate pair supervision while preserving useful cross-sample discrimination.

Thank you for prompting these additional analyses. The confidence diagnostics, subspace ablations, and evaluation of captions not used in basis construction support APS's use of reliable supervision. The orthogonalization and negative-candidate experiments provide further evidence for the implementation choices. We will incorporate these results and the complete timing breakdown into the revised manuscript.

I would like to thank the authors for the additional experiments. However, the experiments of SVD-based orthogonalization or retaining all negative samples did not show a significant improvement. For example, retaining all negative candidates increases the mean RSum marginally, but with a higher std. I would like to keep my original rating.

Comments on the paper

2 comments

Dear Reviewers, thank you again for your valuable feedback. As the discussion period ends in one day, we would like to check whether our response has addressed your concerns. We are happy to discuss any further questions and would appreciate it.

AuthorsOpenReview ↗

Global Response

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for carefully reviewing our submission and the additional experiments. APS achieves state-of-the-art retrieval performance on the noisy correspondence learning (NCL) benchmarks evaluated in our paper, attaining the highest reported RSum among the compared methods in six of the seven settings in Tables 1 and 2. These include all three noise rates on Flickr30K, 20% and 40% noise on MS-COCO 1K, and real-world correspondence noise on CC120K. In the remaining setting, MS-COCO 1K with 60% noise, APS ranks second, 0.4 RSum points behind the best result.

Our work asks how a model can draw useful alignment supervision from the noisy training data itself. Confidence weighting determines how much each image–text pair contributes to training, and APS uses agreement across views of that same pair to determine which directions to align. Prior work has used shared or class-specific subspaces for tasks such as few-shot classification. APS constructs its auxiliary alignment space from each observed pair’s own evidence. On Flickr30K, retrieval performance dropped when we kept the same subspaces and changed only which pairs they were assigned to.

We also provide a simple, strong baseline for NCL. GMM+CW uses the confidence estimated by a Gaussian mixture model as a soft weight on each pair’s contrastive loss when fine-tuning a pretrained CLIP backbone. On CC120K, this baseline already outperformed standard CLIP fine-tuning by 18.2 RSum points under matched training conditions. APS added a further 1.6 points. Across the seven matched settings, its gains over GMM+CW ranged from 0.8 to 2.8 points in mean RSum. These modest margins quantify what pair-specific alignment adds once both models already benefit from the same strong backbone and confidence-weighted training.

The discussion also helped us identify a next step. Our fine-grained evaluation measured caption discrimination at test time. We plan to study how to identify a local mismatch when a training caption contains one incorrect detail and the rest still provides useful supervision.

We thank the reviewers for the suggestions that motivated these experiments and the chairs for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.