- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Aligning within Pair-Specific Subspaces (APS) for robust image-text retrieval under noisy correspondences. The method first trains a teacher model using GMM-based confidence weighting to reduce the influence of unreliable image-text pairs. For each pair, the frozen teacher evaluates multiple augmented image and text views, selects the most consistent combinations, and constructs a pair-specific subspace using SVD. During student training, an auxiliary alignment loss encourages agreement within these subspaces, alongside the confidence-weighted global contrastive objective. Inference uses the student's original global representations without additional computation. Experiments on Flickr30K, MS-COCO, and CC120K show competitive retrieval performance under synthetic and real-world correspondence noise.
Critical strengths
-
The paper addresses a practical problem in cross-modal retrieval and offers a reasonable motivation for complementing confidence-based sample weighting with more selective feature alignment. Instead of applying additional alignment constraints to the entire representation space, APS uses information from individual image-text pairs to determine where auxiliary alignment should take place. The overall approach is straightforward and fits naturally into the existing contrastive learning framework.
-
The experimental analysis is reasonably thorough, particularly with the supplementary material. Beyond the main component ablations, the authors compare different view selection strategies, shuffled and batch-shared subspaces, and single-modality versus joint subspace construction. These experiments provide useful evidence that the improvements are related to the proposed view selection and pair-specific alignment, rather than simply adding an auxiliary loss.
-
APS achieves competitive results across multiple datasets and noise levels, including the real-world CC120K benchmark. The paper also examines hyperparameter sensitivity and provides additional results on MS-COCO 5K. Another practical advantage is that the subspace construction is only used during training, so the method introduces no additional parameters or computational overhead at inference time.
Critical weaknesses
-
Table 3 shows that GMM-based confidence weighting accounts for most of the improvement, while the additional gain from subspace alignment is relatively small. The supplementary ablations with Random-2, Bottom-2, and shared subspaces do provide useful evidence for the proposed design. Still, subspace projection and pair-conditioned feature selection have been explored in related works such as SSP and SimLAP. Since APS mainly combines these ideas with multi-view selection and auxiliary alignment, its methodological contribution appears relatively incremental.
-
The paper constructs subspaces by selecting highly similar image-text views and applying SVD to their joint representations, but it is not clear whether these directions actually capture reliable shared semantics. For normalized image and text features, their sum is naturally orthogonal to their difference, so projection along these directions can reduce cross-modal disagreement without necessarily identifying meaningful shared information. Moreover, with the default setting of (m=r=2), SVD simply orthogonalizes the span of two selected representations. Although the supplementary ablations support the effectiveness of this construction, they offer limited insight into what information the resulting subspaces actually preserve.
-
The SOTA results in Table 1 are taken from different papers using different encoders and training configurations, and even the CLIP baseline differs considerably between Tables 1 and 3. The supplementary material also refers to a binary-weight GMM baseline evaluated at epoch 5, while the main text describes GMM-soft/GMM+CW and APS is evaluated at epoch 10. The relationship between these configurations is not entirely clear. Given the relatively small improvements over recent methods, these differences make it difficult to attribute the reported gains solely to APS.
-
Most synthetic experiments rely on randomly shuffled image-text correspondences, whereas real mismatches may share much of their semantic content and differ only in specific attributes or relationships. This is particularly relevant to APS, since selecting highly similar views could preserve superficial semantic overlap while overlooking the details responsible for the mismatch. Although CC120K provides an evaluation under real-world noise, the comparisons there are relatively limited, leaving the effectiveness of the proposed view selection under more challenging correspondence errors less clear.
Questions for authors
-
Appendix A.2 mentions a binary-weight GMM baseline evaluated at epoch 5, while the main text refers to GMM-soft and GMM+CW. Are these the same experiments? Please clarify the weighting scheme and training configuration, and report how the GMM baseline performs when trained for the same number of epochs as APS.
-
Tables S2–S4 provide useful comparisons of different view selection and subspace construction strategies. What happens if pair-specific subspaces are constructed directly from the original teacher features without multi-view selection? A comparison with simple teacher-based alignment would also help isolate the contribution of the proposed construction. How much additional training time does the complete procedure require, including warm-up and offline subspace construction?
-
Have the authors examined the projected feature norms, singular value distributions, or auxiliary alignment gradients for clean and noisy pairs? Such analyses could help explain whether the selected directions preserve meaningful shared information and whether the constructed subspaces remain stable when the joint representations are nearly linearly dependent.
-
Both the confidence values and subspace bases remain fixed during student training. What happens when the teacher initially assigns an incorrect confidence or the student representations change substantially? Also, low-confidence samples still contribute as negatives to other samples' contrastive losses. Does this affect the effectiveness of confidence-based noise suppression?
-
How does APS perform on incorrect image-text pairs that share most of their semantic content but differ in attributes, object counts, or spatial relationships? Examining the selected views for these cases, alongside those of correctly matched pairs, would help clarify what the proposed multi-view consistency criterion actually captures.
7 comments
Response to Reviewer VRDX — Part 2/5
W3. Comparisons with prior work, internal CLIP baselines, and training settings
The CLIP results in Table 1 are taken from the fine-tuning baseline reported by NPC, using ViT-B/32, 5 training epochs, and a batch size of 256. The CLIP baseline in Table 3 is fine-tuned under our protocol, using the same backbone, 10 training epochs, and a batch size of 128. These results serve to present published benchmark performance and support internal component analysis, respectively.
To clarify attribution of the gains, we have reorganized and extended the binary GMM, GMM+CW, and APS comparisons under a common training protocol, reporting means and standard deviations over three seeds. The revision will identify the sources and configurations of the published results and use matched internal controls to quantify the contributions of continuous weighting and the subspace term. The corrections and detailed results are provided in Q1.
W4. Randomly permuted correspondences and semantically similar fine-grained mismatches
APS selects highly consistent views within each pair to provide content-relevant constraints for auxiliary training. To evaluate the trained model's ability to distinguish local factual changes, we have added a diagnostic set with single-fact caption edits covering color, count, and spatial relations, with 300 test images per category and 900 images in total.
For models trained on Flickr30K with 40% random correspondence noise, APS achieves 69.5% overall two-choice accuracy, exceeding GMM+CW by 1.3 percentage points, with higher mean accuracy in all three categories. This diagnostic directly evaluates discrimination between complete captions using the student's global representations, providing additional evidence for APS under semantically similar mismatches. Detailed results are given in Q5.
Responses to Questions
Q1. Configurations of binary GMM, GMM-soft, and GMM+CW, and results under the same training schedule
GMM-soft and GMM+CW denote the same continuous-weighting baseline. Binary GMM uses binary 0/1 weights derived from the reliability estimates. APS adds pair-specific auxiliary subspace alignment to GMM+CW, with confidence weighting applied to both losses.
Thank you for identifying the reporting issue in the original manuscript. The original table mislabeled the epoch-5 GMM+CW results as GMM and omitted the corresponding binary GMM results. We have corrected this mapping and rerun the control that uses binary weights during student training while retaining the auxiliary subspace term. We explicitly label this configuration “GMM (binary) + subspace.”
The updated comparisons use the same 10-epoch training protocol as APS, with seeds 99, 1994, and 1995 on an A100 GPU. We report mean RSum and standard deviation at epoch 10. The component ablations on Flickr30K and CC120K are:
| Configuration | Flickr30K 20% | Flickr30K 40% | Flickr30K 60% | CC120K real-world noise |
|---|---|---|---|---|
| CLIP fine-tuning under matched settings | ||||
| GMM (binary) | ||||
| GMM+CW | ||||
| GMM (binary) + subspace | — | |||
| GMM+CW + global-space auxiliary alignment | ||||
| APS |
“—” indicates that we do not report a CC120K result for the binary-weighted subspace configuration in this comparison. On Flickr30K with 40% noise, global-space auxiliary alignment uses , selected on the validation set, while APS uses its default .
The corresponding results on COCO 1K are:
| Configuration | 20% noise | 40% noise | 60% noise |
|---|---|---|---|
| GMM (binary) | |||
| GMM+CW | |||
| APS |
These comparisons support the complementary contributions of continuous weighting and auxiliary subspace alignment. Across all three datasets, GMM+CW achieves higher mean RSum than binary GMM, and APS provides further gains over GMM+CW. On Flickr30K, full APS also improves over “GMM (binary) + subspace” by 1.0, 1.8, and 1.5 RSum points. Compared with global-space auxiliary alignment, APS achieves gains of 1.6, 1.5, and 2.8 points at the three noise rates on Flickr30K and 1.8 points on CC120K. We will make the configuration names, training schedules, and statistical reporting consistent throughout the revision.
Response to Reviewer VRDX — Part 3/5
Q2. Bases constructed from original teacher features, direct teacher alignment, and total training cost
The multi-view construction in APS provides multiple content combinations for each pair. The teacher then selects highly consistent combinations to determine auxiliary alignment directions. To evaluate this construction, we have added two teacher-based controls:
- Two-dimensional subspace from original teacher features: We use the teacher features of the unaugmented image and complete caption as the two rows of a matrix, construct a two-dimensional basis via SVD, and apply the same projected cosine alignment loss to the student.
- Direct cross-modal teacher alignment: We align the student image feature with the frozen teacher text feature, and the student text feature with the frozen teacher image feature. The auxiliary term is the average of the two cosine losses.
On Flickr30K with 40% noise, all three auxiliary configurations use the same confidence weighting and , with for APS. Epoch-10 results over three seeds in this dedicated comparison are:
| Method | RSum |
|---|---|
| GMM+CW | |
| Two-dimensional subspace from original teacher features | |
| Direct cross-modal teacher alignment | |
| Multi-view APS |
APS improves over the subspace constructed from original teacher features and direct teacher alignment by 3.0 and 4.5 RSum points, respectively, and over GMM+CW by 1.8 points. These results support the complete design of constructing pair-specific auxiliary spaces through multi-view selection.
For efficiency, APS performs multi-view encoding and basis construction once, offline. The accumulated training-stage times on a single A100 GPU, averaged over three seeds, are:
| Method | Total training-stage time, including warm-up and offline construction |
|---|---|
| GMM+CW | s |
| APS | s |
Both totals include approximately 578 seconds of teacher warm-up, initialization, and final checkpoint saving. Validation and testing are timed separately. APS adds approximately 314 seconds, or 15.4%. Multi-view generation and encoding take 306.0 seconds, and basis construction with related I/O takes 13.8 seconds. This one-time precomputation supports subsequent auxiliary training. Inference directly uses the student's global features, with zero additional parameters or FLOPs.
Q3. Projection norms, singular values, auxiliary gradients, and near-linear dependence
In APS, the subspace determines the directions of the auxiliary constraints, while confidence determines their strength for each pair. To examine these two components, we conduct offline diagnostics on Flickr30K with 40% synthetic noise over three seeds. We distinguish clean and mismatched pairs using the ground-truth correspondence labels recorded during noise injection.
First, using the epoch-10 student and a fixed selection of training pairs, we measure projection norms for additional captions and auxiliary gradient norms:
| Diagnostic measure | Clean pairs | Mismatched pairs |
|---|---|---|
| Image projection norm | ||
| Caption projection norm | ||
| Projection norm of another correct caption | ||
| Projection norm of a caption from a different image | ||
| Auxiliary gradient norm before weighting by | ||
| Auxiliary gradient norm after weighting by |
The gradient statistic is the joint norm of the gradients of the auxiliary loss with respect to the normalized image and text output features. It includes and omits the common batch-level denominator for all samples.
For clean pairs, another correct caption that was not used in basis construction has a mean projection norm 0.340 higher than a caption from a different image, supporting the retention of representations relevant to the same image. After confidence weighting, clean pairs retain approximately 97% of their mean gradient magnitude, while the mean gradient norm of mismatched pairs falls to approximately 1.6% of that of clean pairs. These results support the role of confidence weighting in concentrating auxiliary supervision on reliable pairs.
Response to Reviewer VRDX — Part 1/5
Thank you for recognizing the motivation, subspace comparisons, and inference efficiency, and for your detailed suggestions on basis construction, attribution of the gains, and fine-grained mismatches. We have reorganized the component ablations across three datasets and added diagnostics on captions not used in basis construction, singular values, auxiliary gradients, per-epoch updates, and fine-grained discrimination. These results support using each pair's own multi-view evidence to determine auxiliary alignment directions and correspondence confidence to regulate supervision strength, while preserving efficient retrieval with global representations.
Responses to Weaknesses
W1. Additional contribution of the subspace term and relation to SSP and SimLAP
APS addresses learning from potentially incorrect image–text correspondences by constructing an auxiliary alignment space from each pair's own multi-view evidence. Correspondence confidence jointly regulates the global and subspace objectives. The evidence used to construct the space, the level at which it is defined, and its role in training are central to this design.
SSP uses local visual features to construct a shared visual subspace and class-specific language subspaces, improving image–text alignment through projection for few-shot classification. SimLAP uses class-pair-conditioned learnable gating for contrastive learning between visual samples in the corresponding spaces. APS selects evidence from augmented views of each observed image–text pair, constructs a fixed pair-specific subspace, and uses it for auxiliary positive-pair alignment. Global contrastive learning and final retrieval remain in a common representation space. We will clarify these differences in the evidence used for space construction and the role of the resulting spaces in training.
The existing controls support constructing subspaces at the pair level. On Flickr30K with 40% noise, using the same Top-2 evidence, pair-specific subspaces achieve 553.2 RSum, compared with 551.4 for a batch-shared subspace. Keeping the full collection of subspaces while shuffling their assignments to pairs yields 548.2. The corresponding gaps of 1.8 and 5.0 RSum points support associating the auxiliary space with the content of the current pair.
The updated ablations further show that APS improves over GMM+CW by 1.5, 1.8, and 2.8 RSum points at the three noise rates on Flickr30K, by 1.1, 1.1, and 0.8 points on COCO 1K, and by 1.6 points on CC120K. Full results are provided in Q1. These matched comparisons identify the additional contribution of the subspace term beyond continuous weighting and extend the evidence across datasets and to real-world correspondence noise.
W2. Orthogonality of the sum and difference and information retained by the subspace
APS uses the selected joint directions to provide pair-relevant auxiliary constraints for the student. The geometric property you identified holds: for unit vectors v,t,
(v+t)⊤(v−t)=0.
Under the default setting m=r=2, when the basis construction matrix has full rank, SVD provides an orthonormal basis for the span of the two selected directions. In the dedicated comparison, GS and SVD achieve 553.2±0.5 and 553.2±0.3 RSum, respectively, consistent with the equivalence of the auxiliary objective within the same subspace.
To examine what information this space retains, we construct the basis using a separate original caption for each of 900 test images. We then evaluate correct captions not used in basis construction, captions with a single factual mismatch, and captions from different images. The mean text projection norms in the Top-2 subspaces are:
The mean projection norm of correct captions exceeds that of captions from different images by 0.187. Fine-grained mismatched captions also retain substantial components, as they preserve much of the image-relevant content. These results support the retention of content relevant to the current pair. Together with the matched training comparisons in Q1, they support the use of pair-relevant subspaces for effective auxiliary supervision. The trained global representations also achieve 69.5% accuracy in the fine-grained diagnostic, compared with 68.2% for GMM+CW, as reported in Q5.
In the revision, we will describe the role of the subspace more precisely as “retaining representation components relevant to the current pair to provide selective auxiliary alignment constraints for the student,” supported by the geometric and empirical analyses.