- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper targets dense retrieval for conjunctive queries ("has a, b, c and does not have x, y"). It proposes GOLD mining — synthesising a natural-language query from a (positive, hard-negative) document pair with Gemini 2.5 Flash so that the negative is known to violate exactly d of the query's conditions — and GOLD reweighting, which modifies Margin-MSE, InfoNCE, SigLIP and CoSENT so that hard negatives receive partial credit as a function of d/V. Datasets are built from Amazon ESCI (text, 40k triples) and DeepFashion In-Shop (images, 26k triples), with 133 / 83 human-written test queries. all-mpnet-base-v2 and CLIP ViT-B/32 are fine-tuned for one epoch over three seeds, compared against the unmodified losses and NV-Retriever mining. Appendix A shows that, for Margin-MSE without easy negatives, low GOLD loss implies attributes are approximately affinely recoverable from the embeddings and vice versa, that binary Margin-MSE gives no such guarantee, and gives a capacity construction adapted from Garg et al. (2026). The paper reports that mining plus reweighting improves recall over the baseline for every loss and modality, achieves "state-of-the-art recall," and that mining raises hard-negative win rate while reweighting trades win rate for recall.
Critical strengths
-
A cheap, generic source of graded supervision, applied uniformly across four loss families. Deriving a violation count from LLM-extracted attributes and plugging it into Margin-MSE, InfoNCE, SigLIP and CoSENT is a clean idea, and testing it on both text and image modalities with the same recipe is the right way to show it is loss-agnostic. The construction pipeline, prompts (Appendix E) and annotator instructions (Appendix F) are released.
-
Honest reporting of negative and null results, with paired statistics. The paper states that mining alone reduces text recall (L408–409), that reweighting lowers hard-negative win rate (L418–424), and that the InfoNCE improvement is not significant (L412–414); Table 3 gives per-query paired permutation tests over three seeds for every comparison, and Table 2 reports seed ranges. The theorems in Appendix A are stated carefully. Overall write up is easy to follow.
Critical weaknesses
-
The results do not support the headline claim, and reweighting moves the motivating metric the wrong way.
- On the strongest loss, InfoNCE, the baseline already matches GOLD: +0.001 recall in both modalities, not significant (Table 2, Table 3). The significant gains (SigLIP, Margin-MSE, CoSENT) bring weaker losses up to roughly InfoNCE's level, and NV-Retriever does the same on text for two of them. "State-of-the-art recall" is not shown; "lifts weaker losses" is. The paper should say why GOLD is needed if plain InfoNCE does as well.
- Win rate against the paired hard negative is the conjunctive-discrimination metric, and reweighting lowers it (Figure 4, L418–424) while raising Recall@K over mostly easy negatives. That questions the design: partial credit reduces pressure on exactly the near-miss negatives the paper is about. The CoSENT variant drops d entirely (all hard negatives get rank ½, L267–268) yet has the largest text gain (+.044), so the gain there comes from a hard/easy split, not from counting violations. No graded metric (nDCG with a d-based gain, or precision over graded negatives) is reported, though d is available for every test triple.
- On human image queries, reweighting lowers recall for three of four losses (up to −.060, Table 2); the main text does not mention it, and the synthetic-to-human correlation in Figure 5 is driven by differences between losses, not by the method effects being claimed.
-
The theory is not tested, and the contribution is not positioned against the relevant literature.
- Appendix A proves that low GOLD loss implies attributes are linearly recoverable by a probe, but no probe is ever trained on the paper's own embeddings before or after GOLD training. Section 4.3 and Appendix A contain statements and proofs only. The theory covers Margin-MSE, while the best empirical loss is InfoNCE, and its condition on V is unrelated to the tuned V = 20.
- Graded relevance is standard in learning to rank (RankNet/LambdaRank; LambdaLoss, CIKM 2018) and in contrastive learning (RINCE, AAAI 2022); ESCI itself ships four-level labels that the paper discards without running a graded-label baseline. Hard-vs-easy negative treatment is established in retrieval and in recommender systems. The abstract's "binary training objective" framing does not hold against this. It will be helpful to show some other approaches which have used graded judgement for improving retrieval.
- Conjunctive queries have also been handled through a standard non-embedding route — Boolean retrieval, LLM-formulated Boolean clauses (SIGIR 2023), filtered dense retrieval (WWW 2023). The paper quotes filtering in one sentence ("ontologies are fixed"), yet its own pipeline extracts open-vocabulary attributes per document and composes queries with explicit AND/NOT — the pieces of a parse-then-filter baseline it never runs. By Section 3, what is new beyond attaching a counted label to standard LLM query synthesis and BM25 / same-category negatives is hard to identify.
Overall paper mentions multiple relevant IR and Recommender Systems directions that were developed over time. However, the baselines used, and other aspects fall short of truly standing out. Refer to questions for further discussion.
Questions for authors
Clarification:
- Why is GOLD needed when the unmodified InfoNCE baseline matches it (Table 2)? Is there any setting — base model, dataset, cutoff — where GOLD-InfoNCE separates from InfoNCE outside seed noise?
- Please report a graded metric (nDCG with gain derived from d, or precision over graded negatives) alongside Recall@K and win rate, so the trade-off in Figure 4 can be evaluated on the quantity the paper is about.
- Were linear probes trained on the baseline and GOLD-trained embeddings to test affine recoverability of attributes? If not, can this be added — it is the experiment that connects Appendix A to Section 5.
- Is the Main test set generated by the GOLD pipeline (queries conditioned on the negative) while baseline training queries are not (L216–217)? If so, how much of the gap is train/test distribution match?
- What is the retrieval corpus for Recall@5 / Recall@20, and how many unique test products are there (Table 3 notes text test queries share products)? Why @5 for text and @20 for images?
- CoSENT: why collapse all hard negatives to rank ½ rather than use rank k − d, given the loss only needs an ordering? Does the +.044 text gain survive with graded ranks, and does it survive with a binary hard/easy flag in the other losses (i.e. is d itself doing work)?
- Eq. 5: the SigLIP target T = e^{−s·d/V} uses the learned scale s. Is s detached in the target? If not, the regression target is non-stationary.
- L280–282 require V ≥ max(η, K*), but the sweep (L371–373) includes V = 20 with η = 40. Were those cells excluded, or is the constraint not enforced? Please also report sensitivity to V rather than only the selected value.
- Was the accuracy of Gemini's attribute buckets (A-only / B-only / both / neither) ever checked? d is only correct if those buckets are, and the method rests on d.
- Human queries (L219–221): how many annotators, how recruited and compensated, what agreement, and how exactly were 133 / 83 queries used? Appendix F gives instructions only.
- GOLD-reweighted InfoNCE on human image (0.574 [.542–.590]) is identical to NV-Retriever InfoNCE on the same column, including seed range. Is this a copy error?
- SigLIP is cited as Tschannen et al. 2025 in Section 4 and defined from Zhai et al. 2023 in Appendix C; which was used?
Suggestions:
- Run a parse-then-filter baseline (LLM clauses over the extracted attributes, optionally reranked by the dense model) and a graded-label baseline on ESCI's native labels (LambdaLoss or RINCE), and reposition the contribution against LTR, Boolean/filtered retrieval, and graded contrastive losses.
- Test semantic hard negatives for GOLD in addition to BM25 (text currently uses lexical negatives only, L190–192), and grade in-batch "easy" negatives by attribute distance rather than treating all as maximally wrong — the extracted attributes make this cheap.
- Address how the pipeline extends when labelled positives are unavailable or new documents arrive; text positives are drawn from ESCI's labelled pairs (L188–190), and every query derives from a single seed positive. More importantly, the pipeline is highly dependent on the attributes of positive example, and can face topic/attribute drift based problems.
- Use a second attribute extractor or a human audit sample for d; use a stronger, current base embedder for at least one configuration.
- Notation: ℓ is used for elements of S ⊆ D (Eq. 8, L353) while L = Q ∪ D (L322) and Appendix A uses x ∈ S; pick one. Clarify L345: z_i on a query means "requires attribute i," on a document "has attribute i." Define "Main" once and use it consistently (Table 1 says "Synthetic," Table 2 and Figure 5 say "Main").
- Add the reproducibility and ethics checklist after the AI-use statement, and drop the "to yell at when frustrated" line from that statement (I admit that it mildly amused me).
4 comments
Graded relevance is standard in learning to rank (RankNet/LambdaRank; LambdaLoss, CIKM 2018) and in contrastive learning (RINCE, AAAI 2022); ESCI itself ships four-level labels that the paper discards without running a graded-label baseline.
Numerous works including those leverage ranking to improve retrieval. GOLD differs from the listed works in that GOLD scores can be calculated singularly, i.e. in the absence of a paired example, although we do not do so in this paper. This distinction makes the training independent of the pair selection algorithm, which may be difficult or biased in some contexts.
ESCI labels are tied to specific queries. Having rewritten the queries in our dataset creation, the four-level labels no longer apply, so would not be useful to use. We also decline to compare to them because they are human generated. Moreover, the comparison would show whether BM25/GOLD-mining are superior or inferior to human annotators under a particular regime, which we consider out of scope for our contributions.
Conjunctive queries have also been handled through a standard non-embedding route — Boolean retrieval, LLM-formulated Boolean clauses (SIGIR 2023), filtered dense retrieval (WWW 2023).
The parse-then-filter (we understand this to mean applying an independent embedding lookup for each parsed attribute, then intersecting results) is awkward in production environments because it requires at least one generative model call, followed by arbitrarily many embedding model calls. Compared to a single, unified embedding call, this is slow, expensive, and inconsistent in resource use. Moreover, it cannot be cleanly swapped with a traditional embedding system.
Questions For Authors:
Clarification:
Why is GOLD needed when the unmodified InfoNCE baseline matches it (Table 2)? Is there any setting — base model, dataset, cutoff — where GOLD-InfoNCE separates from InfoNCE outside seed noise?
No.
Please report a graded metric (nDCG with gain derived from d, or precision over graded negatives) alongside Recall@K and win rate, so the trade-off in Figure 4 can be evaluated on the quantity the paper is about.
Please see above.
Were linear probes trained on the baseline and GOLD-trained embeddings to test affine recoverability of attributes? If not, can this be added — it is the experiment that connects Appendix A to Section 5.
After training the probes, we find that the linear recoverability does not increase wth GOLD. We will remove the proof from the paper.
Is the Main test set generated by the GOLD pipeline (queries conditioned on the negative) while baseline training queries are not (L216–217)? If so, how much of the gap is train/test distribution match?
Correct. To address this concern, we rerun the experiment using an evaluation set composed of 50/50 baseline-selected and gold-selected queries. The results do not differ significantly.
New candidate corpora: Text Recall@5
| Loss | Group | GOLD-selected | Baseline-selected | 50/50 mixed |
|---|---|---|---|---|
| – | Untrained | 0.692 | 0.689 | 0.689 |
| InfoNCE | Baseline | 0.886 [0.885–0.888] | 0.874 [0.871–0.876] | 0.879 [0.877–0.882] |
| InfoNCE | NV-Retriever | 0.883 [0.882–0.884] | 0.872 [0.870–0.873] | 0.876 [0.875–0.878] |
| InfoNCE | GOLD mined | 0.884 [0.882–0.886] | 0.874 [0.872–0.877] | 0.876 [0.874–0.879] |
| InfoNCE | GOLD reweighted (ours) | 0.888 [0.885–0.890] | 0.877 [0.874–0.879] | 0.881 [0.878–0.884] |
| SigLIP | Baseline | 0.847 [0.845–0.849] | 0.836 [0.834–0.839] | 0.840 [0.838–0.842] |
| SigLIP | NV-Retriever | 0.853 [0.852–0.853] | 0.846 [0.846–0.847] | 0.847 [0.845–0.849] |
| SigLIP | GOLD mined | 0.840 [0.835–0.844] | 0.829 [0.825–0.831] | 0.834 [0.829–0.839] |
| SigLIP | GOLD reweighted (ours) | 0.866 [0.865–0.867] | 0.857 [0.855–0.859] | 0.860 [0.858–0.862] |
| Margin-MSE | Baseline | 0.849 [0.847–0.852] | 0.837 [0.834–0.843] | 0.841 [0.838–0.845] |
| Margin-MSE | NV-Retriever | 0.868 [0.864–0.870] | 0.857 [0.856–0.859] | 0.860 [0.858–0.861] |
| Margin-MSE | GOLD mined | 0.839 [0.838–0.841] | 0.829 [0.826–0.830] | 0.831 [0.829–0.833] |
| Margin-MSE | GOLD reweighted (ours) | 0.862 [0.861–0.863] | 0.856 [0.854–0.857] | 0.857 [0.856–0.859] |
| CoSENT | Baseline | 0.817 [0.812–0.822] | 0.811 [0.806–0.814] | 0.815 [0.812–0.818] |
| CoSENT | NV-Retriever | 0.863 [0.861–0.865] | 0.856 [0.855–0.857] | 0.857 [0.855–0.858] |
| CoSENT | GOLD mined | 0.809 [0.802–0.814] | 0.797 [0.786–0.804] | 0.802 [0.795–0.807] |
| CoSENT | GOLD reweighted (ours) | 0.861 [0.861–0.861] | 0.852 [0.849–0.853] | 0.857 [0.857–0.857] |
New candidate corpora: Image Recall@20
| Loss | Group | GOLD-selected | Baseline-selected | 50/50 mixed |
|---|---|---|---|---|
| – | Untrained | 0.246 | 0.261 | 0.255 |
| InfoNCE | Baseline | 0.492 [0.480–0.499] | 0.505 [0.492–0.514] | 0.501 [0.487–0.511] |
| InfoNCE | NV-Retriever | 0.487 [0.485–0.490] | 0.501 [0.499–0.503] | 0.495 [0.493–0.495] |
| InfoNCE | GOLD mined | 0.493 [0.491–0.498] | 0.505 [0.503–0.508] | 0.501 [0.496–0.505] |
| InfoNCE | GOLD reweighted (ours) | 0.493 [0.492–0.494] | 0.504 [0.503–0.507] | 0.499 [0.496–0.501] |
| SigLIP | Baseline | 0.446 [0.440–0.454] | 0.461 [0.455–0.472] | 0.455 [0.450–0.463] |
| SigLIP | NV-Retriever | 0.441 [0.433–0.451] | 0.457 [0.451–0.465] | 0.450 [0.443–0.458] |
| SigLIP | GOLD mined | 0.448 [0.443–0.453] | 0.465 [0.462–0.467] | 0.456 [0.453–0.460] |
| SigLIP | GOLD reweighted (ours) | 0.465 [0.451–0.473] | 0.479 [0.466–0.485] | 0.472 [0.460–0.480] |
| Margin-MSE | Baseline | 0.431 [0.428–0.436] | 0.442 [0.438–0.448] | 0.437 [0.434–0.440] |
| Margin-MSE | NV-Retriever | 0.413 [0.412–0.415] | 0.425 [0.423–0.427] | 0.420 [0.418–0.422] |
| Margin-MSE | GOLD mined | 0.437 [0.432–0.442] | 0.447 [0.439–0.454] | 0.442 [0.435–0.448] |
| Margin-MSE | GOLD reweighted (ours) | 0.446 [0.444–0.448] | 0.460 [0.456–0.462] | 0.454 [0.452–0.456] |
| CoSENT | Baseline | 0.379 [0.375–0.385] | 0.388 [0.381–0.395] | 0.384 [0.378–0.394] |
| CoSENT | NV-Retriever | 0.335 [0.330–0.344] | 0.347 [0.340–0.356] | 0.342 [0.335–0.350] |
| CoSENT | GOLD mined | 0.361 [0.350–0.368] | 0.375 [0.362–0.382] | 0.371 [0.359–0.378] |
| CoSENT | GOLD reweighted (ours) | 0.381 [0.368–0.390] | 0.398 [0.383–0.406] | 0.391 [0.375–0.400] |
What is the retrieval corpus for Recall@5 / Recall@20, and how many unique test products are there (Table 3 notes text test queries share products)? Why @5 for text and @20 for images?
The original Main evaluation ranks each query against a shared held-out corpus of 3,173 text documents or 2,318 image files.
The corpus is the deduplicated union of all test positives, hard negatives, and random easy negatives. Training and validation documents are excluded. Every model uses the same corpus within a modality.
Text retrieval approaches saturation at larger K, so reducing recall@k to 5 separates high-performing strategies better.
CoSENT: why collapse all hard negatives to rank ½ rather than use rank k − d, given the loss only needs an ordering? Does the +.044 text gain survive with graded ranks, and does it survive with a binary hard/easy flag in the other losses (i.e. is d itself doing work)?
Ordering based on the number of attributes violated slightly decreases recall on both text and images compared to the collapsed 1/2 case. We stress that this result does not necessarily translate to other loss functions.
| CoSENT supervision | Text Recall@5 | Image Recall@20 |
|---|---|---|
| GOLD-mined | 0.8151 | 0.3633 |
| Hard negatives rank ½ | 0.8663 | 0.3839 |
| Graded ranks: positive 11, hard 11 − d, easy 0 | 0.8608 | 0.3781 |
Eq. 5: the SigLIP target T = e^{−s·d/V} uses the learned scale s. Is s detached in the target? If not, the regression target is non-stationary.
We accidentally used s for two distinct scales. The target uses a fixed α = 20; only the logit scale s is learned. Thus targets remain constant, and no detachment is needed. We will correct Eq. 5.
L280–282 require V ≥ max(η, K), but the sweep (L371–373) includes V = 20 with η = 40. Were those cells excluded, or is the constraint not enforced? Please also report sensitivity to V rather than only the selected value.*
The V = 20, η = 40 cells were included. Our implementation clips normalized distances to [0, 1], so the easy-negative margin is min(η/V, 1). Thus this cell uses a margin of 1 rather than 2. The paper omitted the clipping operation; we will correct the equation and clarify that V ≥ max(η, K*) is sufficient to avoid clipping, rather than an enforced constraint.
The sweep sizes differ because Margin-MSE has two hyperparameters. It uses V ∈ {20, 40, 80} and η ∈ {10, 20, 40}, giving nine configurations per modality, or 18 across text and images. InfoNCE and SigLIP each use V ∈ {10, 20, 40, 80}, giving four configurations per modality, or eight across both modalities. Thus “eight versus nine” compares different scopes; across both modalities, the comparison is eight versus 18. CoSENT uses neither parameter.
The following tables report validation recall from seed 42: Recall@5 for text and Recall@20 for images.
| Loss / modality | V = 10 | V = 20 | V = 40 | V = 80 |
|---|---|---|---|---|
| InfoNCE / text | .8913 | .8920 | .8867 | .8875 |
| InfoNCE / image | .5292 | .5370 | .5370 | .5316 |
| SigLIP / text | .8433 | .8456 | .8502 | .8593 |
| SigLIP / image | .4778 | .4817 | .4887 | .5027 |
For Margin-MSE, the full grids are:
| Text: η | V = 20 | V = 40 | V = 80 |
|---|---|---|---|
| 10 | .8707 | .8730 | .8259 |
| 20 | .7711 | .8639 | .8510 |
| 40 | .7711 | .7612 | .8411 |
| Image: η | V = 20 | V = 40 | V = 80 |
|---|---|---|---|
| 10 | .4856 | .4856 | .4832 |
| 20 | .4123 | .4762 | .4754 |
| 40 | .4123 | .3882 | .4606 |
InfoNCE varies little across the tested V values. SigLIP improves through V = 80 in both modalities, so its selected value is at the grid boundary. Margin-MSE depends on both V and η: at η = 10, image recall changes little, while text recall falls at V = 80. Clipping makes η = 20 and η = 40 equivalent at V = 20, explaining their identical scores. These single-seed sweeps describe validation sensitivity; they do not establish statistical significance.
Was the accuracy of Gemini's attribute buckets (A-only / B-only / both / neither) ever checked? d is only correct if those buckets are, and the method rests on d.
We annotated 50 examples on each dataset and find that attributes are correct on text 97.3% and on images 98.4%.
Human queries (L219–221): how many annotators, how recruited and compensated, what agreement, and how exactly were 133 / 83 queries used? Appendix F gives instructions only.
We recruited 11 annotators from our university. Annotators were compensated in the form of reciprocal annotation by the authors on unrelated research. The queries are used as the queries in the human datasets.
GOLD-reweighted InfoNCE on human image (0.574 [.542–.590]) is identical to NV-Retriever InfoNCE on the same column, including seed range. Is this a copy error?
It is not. We verified that although summary results are identical, the models differ in which examples are inferred correctly.
SigLIP is cited as Tschannen et al. 2025 in Section 4 and defined from Zhai et al. 2023 in Appendix C; which was used?
We will correct both to use Zhai et al. 2023
Response to Reviewer eVSA
We thank the reviewer for their thoughtful feedback. It is clear the reviewer is well-read in this field and thought deeply about the paper, and the suggestions will surely improve our research going forward.
Critical Weaknesses:
The results do not support the headline claim, and reweighting moves the motivating metric the wrong way.
This is a fair weakness. We will update the main framing in the paper to reflect it.
Can the reviewer clarify the purpose of precision over graded negatives? With only one graded negative per triple, the value of this metric is not clear, and we find defining precision among easy and hard negatives to be ambiguous.
We report nDCG@2 below and will add it to the paper.
We would like to note that CoSENT is unique among loss functions in that it uses ordinal rankings. It is possible that with access to multiple ranked hard negatives, the gain would be even larger, but we did not want to dilute the focus by treating it specially. Being unique in this regard, we emphasize that this behavior may not apply to other loss strategies.
We will update the text to emphasize the human trial results. When correlations are taken within loss type groups then averaged, the text correlation remains high but image does not:
The theory is not tested, and the contribution is not positioned against the relevant literature.
We attempted to train these probes but found no significant difference between baseline and ours with respect to linear recoverability. We will remove the proof.