acceptodds
Under review as a conference paper at ICLR 2027

RIMU: Ranking-Aware Dense Retrieval Adaptation via Convex Optimization

Abstract

Retrieval underpins many modern agents, but dense retrievers built with frozen encoders do not automatically improve as verified query feedback accumulates. Query–corpus misalignment may therefore persist even when the correct target for a query is known. We introduce RIMU, a lightweight method that corrects stored corpus vectors based on the ranking of gold targets relative to their competitors. RIMU computes minimum-displacement updates through one convex quadratic program per query, then renormalizes the vectors to preserve cosine retrieval. Across ten retrieval pools and five frozen encoders, it improves N@10 over the frozen index by 13.4–27.1%. With Qwen3 Embedding-8B, it achieves the highest N@10 on seven of ten pools. Learning curves show gains as feedback accumulates, while gain retention analyses assess whether later updates preserve earlier improvements. Updates take 2.3–27.3 ms per query on CPU, without encoder retraining or additional index entries.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Retrieval-Augmented Generation, Retrieval Adaptation, Continual Retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8SvYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RIMU, a document vector editing method for frozen dense retrieval models. The method leverages verified query feedback to directly adjust target document vectors through convex optimization, thereby improving their ranking relative to competing documents. It also employs tangent-space constraints and an ownership mechanism to reduce unnecessary vector changes and inter-query interference. Experiments cover ten retrieval pools and five encoders, and the results show that the method achieves a notable improvement in N@10 compared with frozen indexes.

Critical strengths

The problem has practical value. Without retraining the encoder, retrieval feedback is absorbed directly through vector editing, which offers certain deployment advantages.

The optimization objective is clear. Through ranking constraints and a minimum-displacement objective, it balances retrieval effectiveness and preservation of the original vectors.

The experimental coverage is relatively broad. The method is evaluated on multiple encoders and retrieval pools, and performance changes, ablation results, and update overhead are analyzed.

Critical weaknesses

Weakness 1. There is an obvious connection to the document-vector editing work Embedding Surgery [1], which shares much of RIMU's motivation, yet the paper does not discuss or compare with it. In particular, it is not explained what substantive innovations RIMU offers over this closest work in terms of ranking constraints, continual feedback updates, and optimization mechanisms. A direct comparison with the closest method is lacking, and without it readers cannot judge what RIMU substantively adds.

Weakness 2. The ownership constraint's benefit is confined to guarded queries (training N@10 +0.69, invasions −0.012/query), but the paper does not quantify how often an invasion actually pushes a gold out of the top 10. Appendix C states that "fewer than one invasion in twenty displaces a gold from a top ten," and describes this as the mechanism working as designed on a quantity that costs retrieval almost nothing. This is confusing, because the whole motivation of the ownership constraint is to finely control and reduce the probability of such invasions. If an invasion almost never displaces a gold from the top 10 (fewer than 5%), this conditional probability should be surfaced in the main text, with an honest statement that ownership mainly reduces harmless invasions. Otherwise readers cannot judge whether the module deserves to be on by default.

[1] Amendola M, Mallia A, Perego R. Embedding Surgery: Localized Updates for Adaptive Ranking Correction in Dense Retrieval. arXiv preprint arXiv:2609.05110. 2026 Sep 4.

Questions for authors

See the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8SvY (1/2)

We thank the reviewer for the valuable feedback and for bringing this important concurrent work to our attention. We answer the two weaknesses in turn: novelty with respect to Embedding Surgery (W1) and the ownership term (W2). In the revised PDF, changes made in response to your comments are shown in purple.

W1: The argument for novelty needs further supplementation. There is an obvious connection to the document vector editing work Embedding Surgery, and it has not yet been fully explained what substantive innovations RIMU offers in terms of ranking constraints, continual feedback updates, and optimization mechanisms. A direct comparison with the closest method is lacking.

Both methods use ranking-constrained convex editing of corpus vectors; RIMU makes five different design choices (Table R10). We regard Embedding Surgery as concurrent work, and the revised paper now discusses it as such in the Related Work section, next to the direct comparison in Table 1.

Table R10: Design choices of Embedding Surgery and RIMU.

Embedding SurgeryRIMU
Writable setthe gold and its top-20 competitors (Symmetric, the default; the Promotion and Demotion variants move only one side)only the query's golds
Ranking constraintshard, every out-of-order pair in the top-20soft (slack), only for queries whose ranking needs repair
Spherical geometryno normalization in the updatetangent step + renormalization
Feedback streamone query at a timebuffered stream (B = 64 per snapshot)
Cross-query protectionnone explicitownership rows for earlier similar queries

Experimental setup

We reproduced Embedding Surgery in its default symmetric configuration (the gold and its top-20 competitors writable, one feedback query at a time) on the paper's pools, folds and feedback with both encoders. Two variants test single design choices (Table R11): post-update renormalization only, without RIMU's tangent constraint, and Surgery's own Promotion variant, which moves only the relevant documents, as RIMU does.

Table R11: Embedding Surgery vs. RIMU, test N@10 per pool, mean over 5 folds. Surgery: symmetric (default) configuration; Promotion: only the relevant documents move. Best updated method in bold.

PoolBGE FrozenBGE SurgeryBGE + renorm.BGE PromotionBGE RIMUQwen FrozenQwen SurgeryQwen + renorm.Qwen PromotionQwen RIMU
SRA52.574.674.138.282.661.387.283.055.989.2
ToolRet-Cust.33.644.843.735.461.249.862.762.557.466.0
ToolRet-Web37.444.142.440.651.746.651.449.647.256.2
ToolRet-Code34.235.936.632.246.253.256.756.352.658.7
MAIR-Code32.537.036.234.447.849.555.852.953.358.0
MAIR-Legal30.129.230.027.233.345.644.245.642.349.3
MAIR-Finance55.555.955.854.956.049.350.850.250.451.1
MAIR-Medical49.049.949.548.749.740.042.441.939.840.4
MAIR-Academic40.039.340.033.539.844.944.244.742.743.9
MAIR-Web48.448.348.546.448.045.245.745.644.143.7
Mean41.345.945.739.251.648.554.153.248.655.6

Findings

(1) RIMU is better on the mean with both encoders, and on seven of ten pools against all three Surgery variants. The margin is +5.7 N@10 with BGE and +1.5 with Qwen3-8B, and RIMU repairs more of the feedback (training N@10 77.9 vs. 61.7). Surgery, or its renormalized variant, is stronger on MAIR-Medical, -Academic and -Web with both encoders. Post-update renormalization alone does not close the gap; the two variants are reported only in this response, not in the paper's Table 1.

(2) The gain comes from RIMU's choices together, not from the writable set alone. The Promotion variant moves only the relevant documents, as RIMU does, but under hard constraints and without normalization or protection; it is the weakest variant (mean 39.2 / 48.6 N@10) and loses 14.3 / 13.2 N@10 on unseen test queries (BGE / Qwen3-8B), against 1.2 / 0.6 for the symmetric default, which agrees with Surgery's choice of the symmetric form as its default. RIMU's writable set pays off together with its tangent geometry, its soft constraints triggered only when a ranking needs repair, and ownership.

In the revised paper: the Emb. Surgery rows of Table 1 and of the timing table (Table 21, Appendix G.1).

AuthorsOpenReview ↗

Response to Reviewer 8SvY (2/2)

W2: The ownership constraint's benefit is confined to guarded queries (training N@10 +0.69, invasions −0.012/query), but the paper does not quantify how often an invasion actually pushes a gold out of the top 10. App. C.3 mentions "fewer than one invasion in twenty displaces a gold"; if so, this conditional probability should be surfaced in the main text, with an honest statement that ownership mainly reduces harmless invasions. Otherwise readers cannot judge whether the module deserves to be on by default.

Experimental setup

Appendix C.3 counts invasion events. In the revision we also count per test query (Table R12), for RIMU with and without ownership, for ERM and for Embedding Surgery.

Table R12: Ownership and intrusion harm, mean over 10 pools × 5 folds. Intrusion: a rewritten non-gold enters a test query's top-10 above the query's best gold, which it did not outscore before the update; harmful: this lowers the query's N@10; any gold out: at least one gold that was in the query's top-10 before the update is no longer; all golds out: the query had a gold in its top-10 and has none after (% of test queries). RIMU and ERM: same runs as the paper's Table 24; Emb. Surgery: symmetric configuration, same pools and folds.

MethodN@10 TrainN@10 TestIntrusion %Harmful %Any gold out %All golds out %
BGE-large
ERM91.746.536.817.09.37.0
Emb. Surgery61.745.912.65.31.10.8
RIMU w/o ownership77.151.621.57.62.81.6
RIMU77.951.620.77.12.51.5
Qwen3-Emb.-8B
ERM91.550.633.417.88.96.9
Emb. Surgery71.354.19.55.41.00.6
RIMU w/o ownership77.055.618.28.72.81.5
RIMU77.355.617.88.52.71.4

Findings

(1) The conditional probability the reviewer asks for is about 11–12%. Among test queries that suffer an intrusion, at least one gold that was in the top-10 is pushed out in 11.2% with BGE (493 of 4,397) and 12.1% with Qwen3-8B (338 of 2,803), against 21.2% and 18.1% with ERM (pooled over all test queries; dividing the mean shares of Table R12 gives 12.1% and 15.2%). The "fewer than one in twenty" of Appendix C.3 (C.4 in the revision) counts per invasion event, i.e. per rewritten non-gold entering a top-10, most of which stay below the gold; per query the rate is about one in nine. The revised paper reports both, next to Figure 5 and in Appendix H.3.

(2) Ownership's effect is small, but keeping it on by default has no visible downside. It leaves test N@10 unchanged and lowers the intrusions it targets with both encoders: harmful intrusions 7.6 → 7.1% (BGE) and 8.7 → 8.5% (Qwen3-8B), any gold out 2.8 → 2.5% and 2.8 → 2.7%, all golds out 1.6 → 1.5% and 1.5 → 1.4%, training N@10 +0.8 and +0.4 (unrounded 76.95 → 77.31). Of the 0.8-point drop in intrusions with BGE, 0.5 points are harmful ones; under the per-event definition of Appendix C.3, ownership removes 0.77 points of invaded queries, 0.55 of them harmful (0.26 and 0.19 with Qwen3-8B), so it does not mainly remove harmless invasions. Over the 50 (pool, fold) cells the reductions are significant with BGE (paired t-test: harmful t = −4.8, any gold out t = −4.2) and weaker with Qwen3-8B (t = −3.0 and −1.7); test N@10 does not change (t = 0.7 and 0.9). Its update-time cost is small (a few ownership rows per query's QP), so we keep it on by default.

(3) The gap to ERM comes from RIMU's update, not from ownership. Without ownership, RIMU already lowers the share of test queries that lose a gold from about 9% (ERM) to 2.8%, because it ranks a gold above its competitors instead of pulling it onto the query. Ownership's effect is largest where many feedback queries ask for nearby targets (MAIR-Code, ToolRet-Code, MAIR-Medical, SRA) and close to zero on MAIR-Legal, -Web and -Academic.

(4) Embedding Surgery intrudes least because it repairs least. It repairs less of the feedback (training N@10 61.7 vs. 77.9 with BGE, 71.3 vs. 77.3 with Qwen3-8B), so fewer rewritten rows reach other queries' top-10 (harmful 5.3 / 5.4%), and its test N@10 stays lower (45.9 / 54.1 vs. 51.6 / 55.6).

In the revised paper: Figure 5 in the main text and Appendix H.3 (Table 24).

Thank you for your careful and detailed response. In light of the fact that I have already given a positive evaluation, I will maintain my original recommendation.

Thank you for the time and care you have devoted to reviewing our work. Your suggestions on the additional comparison and on the ownership term have helped us strengthen the paper and make it more comprehensive. We sincerely appreciate your continued support.

Reviewer_72AGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

RIMU performs continual adaptation of dense retrieval with a frozen encoder. Given a feedback query carrying a verified gold set, it does not simply pull the gold vector toward the query; instead it imposes ranking constraints against the top-k non-gold competitors the query retrieved, requiring each gold to lead its competitor by a specified margin, with conflicting constraints priced through slack variables. Tangency constraints and ownership constraints that suppress cross-query interference are layered on top, and a minimum-displacement convex quadratic program is solved. The program is solved in its dual form, so the kernel size depends only on the number of constraint rows and is decoupled from embedding dimension and corpus size; each displacement is renormalized onto the unit sphere before write-back to preserve cosine retrieval. Across ten retrieval pools and five frozen encoders it outperforms lightweight update baselines such as NUDGE and ERM, at millisecond-level CPU cost per query.

Critical strengths

1、easy to follow 2、sound like a very innovative idea

Critical weaknesses

1、Only targets labeled as gold in training feedback are writable; a test query whose gold no training query claims cannot benefit. Gains are almost entirely determined by gold overlap (r = 0.93): training N@10 rises 60.2% while test rises only 13.4%, and the retention metric is in-sample at both ends. No criterion is given for when to stop updating.

2、Each query solves its own QP, later writes overwrite earlier ones, and convexity belongs to individual subproblems rather than to the update sequence. Order sensitivity is measured only at B = 64; once the cache grows, components merge into 948,246 rows and 3.3 TiB, and three pools fail to run at all. The counterintuitive finding that joint solving is worse remains under-explained.

3、Write-back presumes an index supporting frequent in-place updates with immediate visibility, yet HNSW reinsertion is costly and IVF-PQ, once quantized, cannot store arbitrary real-valued vectors at all. Experiments run only on 3K–111K targets with exact search; write amplification and rebuild cycles under ANN are never validated. Feedback is also assumed to be verified gold, with no noise experiments.

Questions for authors

If RIMU is deployed on an HNSW or IVF-PQ (product-quantized) index, every write-back implies either a structural update or, once quantized, the outright inability to store a real-valued vector. What are the write amplification, the retrieval-quality decay, and the index rebuild cycle of your method on such indexes? In addition, on corpora of tens of millions to a billion vectors, each cache requires a fresh O(B|T|d) retrieval pass — can the 2.3–27.3 ms per-query update cost still hold?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 72AG (1/4)

We thank the reviewer for the detailed and constructive review. Many of the comments were valuable for improving the paper, and we have added a substantial set of new experiments in response. We first answer the two questions for the authors (Q1, Q2), then the weaknesses, one claim at a time (W1–W8; the weakness on in-place index updates is answered together with Q1). All new results use the paper's 10 pools × 5 folds and BGE unless stated otherwise. In the revised PDF, changes made in response to your comments are shown in orange.

Q1: If RIMU is deployed on an HNSW or IVF-PQ index, every write-back implies either a structural update or, once quantized, the outright inability to store a real-valued vector. What are the write amplification, the retrieval-quality decay, and the index rebuild cycle of your method on such indexes?

A weakness raises the same point (frequent in-place updates with immediate visibility; 3K–111K targets with exact search only); we answer both here.

Experimental setup

We re-ran all pools behind HNSW and IVF-PQ with four ways of keeping the index in sync with the rows RIMU rewrites (Table R5, the same as Table R4 for ebQG Q4). To test scale we added distractors: an SRA pool plus the other 9 pools (0.44M rows) and an SRA or ToolRet-Code pool plus the 11 BRIGHT corpora (1.58M rows), and ran the 5 folds one by one on the same index, never rebuilt (each fold's feedback is written in place, scored on that fold's test queries and reverted before the next fold, so no feedback carries across folds).

Table R5: RIMU behind ANN indexes, mean N@10 over 10 pools × 5 folds. Maintenance: wall time to bring the index up to date after RIMU (4 CPU threads); ms/query: 1 thread, BGE. HNSW: M = 32, ef 128; IVF-PQ: 4√N lists, 64-byte codes, nprobe 16, exact re-rank of the top 100.

IndexModel / index updateN@10 BGEN@10 Qwen3-8BMaint. (s) BGEMaint. (s) Qwen3-8Bms/query
Exactfrozen41.348.5––14.7
RIMU51.655.6––14.7
HNSWfrozen40.848.2––0.9
RIMU, no maintenance50.655.1000.9
RIMU, re-insert moved rows50.855.31.97.00.9
RIMU, moved rows in exact buffer51.155.4001.5
RIMU, full rebuild51.155.414.145.70.9
IVF-PQfrozen38.746.6––0.5
RIMU, no maintenance45.652.6000.5
RIMU, re-insert moved rows46.353.00.10.20.5
RIMU, moved rows in exact buffer49.554.1001.1
RIMU, full rebuild48.053.822.8102.30.5

Findings

(1) RIMU changes little, so an update is cheap: HNSW loses almost nothing, IVF-PQ is best served by the buffer of (2). What we measure is the logical update volume and the maintenance time, not the physical write amplification of a particular storage engine: each fold rewrites only 407–1,058 rows, at most 0.1% of the corpus, and re-inserting them into HNSW takes 2–5 s, against 92–621 s for a full rebuild. A write-back need not even be a structural update: RIMU's moves are small (cosine ≈0.985 to the old vector), so replacing only the moved rows' vectors and keeping the graph already keeps 96% of the exact-search gain, and re-inserting them keeps 97–100%. The quality loss stays small when one graph goes through 5 folds of in-place writes at the larger scale (88–94% of the gain kept; the lower share comes from HNSW itself, which at the same settings recalls 0.91–0.92 of the exact top-10 on these larger graphs against 0.98 on the paper's pools); this tests repeated writes, not months of accumulated feedback, which we have not run behind an ANN index. Behind IVF-PQ the reviewer is right that a code cannot hold the exact new vector; re-encoding the moved rows with the existing quantizer still keeps at least 74% of the gain (BGE; 90% with Qwen3-8B; 81–89% at the larger scale).

(2) RIMU's rows simply follow the index's normal maintenance. Deployed ANN indexes are maintained periodically anyway, because documents are added and changed all the time: new or changed rows are held uncompressed next to the index (e.g., the growing segments of Milvus), visible to queries at once, and merged at the next compaction. RIMU's moved rows take exactly this path; under the update loads we tested, no full rebuild was needed. While in this exact buffer they keep all of the gain (HNSW 51.1 / 55.4 N@10, the same as a full rebuild; IVF-PQ 49.5 / 54.1, above a full rebuild) at a small latency cost (HNSW 0.9 → 1.5 ms, IVF-PQ 0.5 → 1.1 ms per query), and after the merge they stay at the re-insert level above.

In the revised paper: Table 3 in the main text and Appendix E.3 (Table 15).

AuthorsOpenReview ↗

Response to Reviewer 72AG (2/4)

Q2: On corpora of tens of millions to a billion vectors, each cache requires a fresh O(B|T|d) retrieval pass. Can the 2.3–27.3 ms per-query update cost still hold?

Yes, with the ANN index a deployment at that scale already uses. We estimate it at 1.58M rows, 14 times our largest pool: the update took 1,352 s per fold for 4,320 feedback queries, of which 1,228 s was the exact competitor retrieval, leaving at most 28.6 ms per query for everything other than retrieval. This residual is the difference of two separately timed runs, so it also absorbs the run-to-run variation of a 20-minute retrieval pass; at 0.44M rows it is only 2.5 ms. Timed directly, the QP solve takes 0.6 ms per feedback query at both 0.44M and 1.58M rows, so it does not grow with |T|. Retrieving competitors through HNSW instead of exact search took 1.7 s per fold (0.4 ms per query), and syncing the ANN index takes 2–5 s per fold (Q1). All these timings are in Table 16 of the revised paper (Appendix E.3).

W1: Only targets labeled as gold in training feedback are writable; a test query whose gold no training query claims cannot benefit. Gains are almost entirely determined by gold overlap (r = 0.93): training N@10 rises 60.2% while test rises only 13.4%.

Experimental setup

Table R6 (the same as Table R1 for ebQG Q1) splits the test queries by whether their gold received feedback (seen) or not (unseen).

Table R6: Qwen3-Embedding-8B, change in N@10 vs. frozen on seen / unseen test queries, mean over 5 folds. †: the method's grid chose no update (no vector moved). The Surgery columns are reported only in this response.

PoolSeen shareFrozen seen / unseenRIMU seenRIMU unseenNUDGE-N seenNUDGE-N unseenERM seenERM unseenSurgery seenSurgery unseen
SRA97%60.4 / 87.1+29.0−5.2+28.5−16.3+21.4−11.1+26.7+2.3
ToolRet-Cust.86%44.8 / 81.6+19.1−1.5+14.9−1.5+12.0−4.1+15.6−3.9
ToolRet-Web59%38.8 / 57.7+17.1−1.2+15.5−1.7+17.2−1.9+9.3−1.4
ToolRet-Code36%54.5 / 52.5+22.4−4.00.0†0.0†+20.6−14.9+15.3−3.1
MAIR-Code45%48.2 / 50.5+19.7−0.5+15.9−1.3+16.2−2.1+14.7−0.4
MAIR-Legal23%16.4 / 54.2+22.2−1.7+8.9−3.4+13.3−8.4+0.6−2.0
MAIR-Finance12%46.4 / 49.6+17.9−0.3+18.1−2.7+26.5−7.0+8.3+0.7
MAIR-Medical39%25.3 / 49.2+3.5−1.50.0†0.0†+3.7−17.4+3.1+2.0
MAIR-Academic6%12.0 / 47.0+8.4−1.60.0†0.0†+9.6−4.0−1.7−0.6
MAIR-Web17%27.4 / 48.9−4.9−0.8−1.7−0.8−4.5−3.0+1.4+0.4
Mean42%37.4 / 57.8+15.4−1.8+10.0−2.8+13.6−7.4+9.3−0.6

Findings

(1) By design, RIMU improves only the targets that received feedback; this is the trade-off of staying lightweight and drawing its gain from the feedback. We agree with the reviewer. RIMU writes each correction only into the targets that received feedback: later held-out queries for those targets can be retrieved better, while the gold of a test query that received no feedback is not directly adapted. Table R6 shows this on average: the gain comes from the seen queries (+15.4 N@10), while the unseen ones lose 1.8 on average (up to 5.2 on SRA); seen queries can also lose, as on MAIR-Web (−4.9). This explains r = 0.93 and the 60.2% vs. 13.4% gap: the training number is measured on the feedback queries themselves, the test number averages over pools where 6–97% of the test queries hit a seen target.

(2) Among methods that edit corpus vectors, RIMU gives the best balance between seen and unseen queries. Every such method trades one against the other. ERM pulls the golds onto the query and loses most on unseen queries (−7.4); NUDGE-N gains less on seen ones (+10.0) and still loses more on unseen (−2.8). Embedding Surgery is the most conservative: it loses least on unseen queries (−0.6), but gains far less on seen ones (+9.3 vs. +15.4), so its test N@10 stays lower (54.1 vs. 55.6). RIMU gains the most on seen queries at a small unseen cost (−1.8).

In the revised paper: Appendix D.4 (Table 9).

AuthorsOpenReview ↗

Response to Reviewer 72AG (3/4)

W2: The retention metric is in-sample at both ends.

Experimental setup

We cut each fold's feedback into 5 rounds, write them one after another into the same index, and after every round score the three groups of Table R7; the middle one (new queries for a target written in round 1) is out of sample.

Table R7: Retention when each fold's feedback is written in 5 rounds into the same index (round 0 = frozen). N@10, mean over 10 pools × 5 folds, BGE-large.

After roundRound-1 feedback queries (in-sample)Held-out test queries whose gold was written in round 1All held-out test queries
041.031.741.3
177.849.748.8
277.752.349.8
377.553.950.9
477.254.751.3
576.955.151.7

Findings

(1) Out of sample, the gain is kept and keeps growing. The held-out test queries whose gold was written in round 1, never used as feedback, rise from 31.7 to 49.7 N@10 after round 1 and reach 55.1 after round 5 (Table R7): RIMU carries feedback to new queries for the same targets, and later rounds add to it rather than erase it. Rows that received no feedback are never written; the small loss on unseen targets (Table R6) comes from rewritten golds that intrude into their top-10.

(2) In sample, the round-1 gain is also kept. The round-1 feedback queries keep 76.9 of their 77.8 N@10 through four later rounds. Since RIMU writes only the targets it is given (W1), this in-sample number is what an update must keep, and the out-of-sample column above checks the rest.

In the revised paper: Appendix F.1 (Table 20); seen and unseen targets in Appendix D.4 (Table 9).

W3: No criterion is given for when to stop updating.

Per query, RIMU already has a stopping rule. A query triggers an update only if its best gold is not ranked first, some gold lies outside the top-10, or the best gold leads the strongest non-gold by less than 0.05 (Section 3 of the paper); otherwise nothing is written. When a triggered query's rows already hold at zero displacement, the minimum-displacement solution is zero as well. We state the missing pool-level criterion as a limitation in the revised Conclusion.

W4: Each query solves its own QP, later writes overwrite earlier ones, and convexity belongs to individual subproblems rather than to the update sequence.

Convexity guarantees each step only, but in practice later rounds undo little. Convexity guarantees the optimality of each step only; a later write can in principle undo an earlier one, and we do not claim a sequence-level guarantee. Over the 5-round streams, the round-1 feedback queries lose only 0.9 N@10 through four later rounds (Table R7).

In the revised paper: Appendix F.1 (Table 20).

W5: Order sensitivity is measured only at B = 64.

Experimental setup

We re-ran all pools and folds with caches of B = 16, 64 and 256 queries, each with 5 arrival orders (Table R8).

Table R8: RIMU test N@10 by cache size B, and the spread (max−min) of N@10 over 5 arrival orders. N@10 is the mean over 5 folds and 5 orders, the spread the mean over folds; BGE-large. With the paper's fixed order, B = 64 is Table 1 of the paper (mean 51.6).

PoolN@10 B=16N@10 B=64N@10 B=256Spread B=16Spread B=64Spread B=256
SRA82.082.683.00.71.01.0
ToolRet-Cust.62.462.159.91.61.92.4
ToolRet-Code46.446.446.40.61.01.3
ToolRet-Web51.151.551.80.60.60.6
MAIR-Code47.747.647.51.61.61.7
MAIR-Legal33.633.633.61.01.01.4
MAIR-Finance55.955.955.90.10.20.5
MAIR-Medical49.950.050.10.40.60.5
MAIR-Academic39.839.839.80.20.20.2
MAIR-Web48.148.148.10.20.30.3
Mean51.751.851.60.70.91.0

Findings

Neither the cache size nor the arrival order matters much. Changing B by a factor of 16 moves the mean N@10 by only 0.2, and all pools but two by at most 0.7. The two exceptions are ToolRet-Customized, where many feedback queries compete for the same targets and a larger cache lets more of them share one stale snapshot, and SRA, where a larger cache helps slightly. Across the 5 arrival orders, a single run moves by 0.7–1.0 N@10 (max−min), within the fold-to-fold noise. Small caches are therefore both the safe and the cheap choice.

In the revised paper: Appendix E.5 (Table 18).

Reviewer_ebQGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RIMU, a method to update stored document vectors from verified query feedback, with the encoder frozen. Per query, the idea is to move the gold/relevant docs as little as possible so they beat the retrieved competitors by a margin. Unlike prior corpus-update methods (NUDGE, ERM), which move gold vectors toward their queries, RIMU sets each update from gold-versus-competitor score margins (like contrastive update). It solves a minimum-displacement convex QP per query with two additional constraints. The tangency constraint restricts the update to the tangent space of the current vector, keeping the program convex; the vector is then renormalized. The ownership constraint is a soft penalty against moving a gold toward similar observed queries with disjoint golds.

Critical strengths
  • The paper explores an interesting direction of updating stored document embeddings from a stream of verified query feedback. The proposed method allows to cheaply correct errors without retraining the embedding model.

  • Nice formulation of the problem as a QP problem with three constraints. The three constraints are reasonable.

  • The method demonstrates improved performance on tool retrieval datasets where the target set is small and mostly stationary.

Critical weaknesses
  • The application scenario is quite narrow because it requires the target space to be stationary where the target documents during test time are mostly seen during training. In other words, the method could not generalize to unseen documents or unfamiliar queries.

  • For seen feedback, a simpler approach is to build a semantic cache that map queries or semantically similar ones to gold documents. The author should compare their approach to this simple baseline. The authors should also consider a fine-tuning baseline trained on the same data used for training their method to quantify the accuracy–cost trade-off.

  • All experiments use exact search, and reported update times exclude any index maintenance. Practical deployments typically use ANN indexes (e.g., HNSW, IVF-PQ), where repeated in-place updates can degrade graph or cluster quality and require periodic rebuilds. The authors should evaluate whether gains hold under an ANN index at scales with in-place updates, and report the maintenance cost.

Questions for authors
  1. Can you report results separately for test queries whose gold received training feedback and for those whose gold did not?
  2. How does RIMU compare to a semantic-cache baseline that maps a test query to the golds of its nearest training queries?
  3. How does it compare to fine-tuning the encoder (or LoRA) on the same feedback, in both accuracy and total cost, including corpus re-encoding?
  4. Do the gains hold under an ANN index (e.g., HNSW) at larger corpus scale with in-place updates, and what are the index maintenance and rebuild costs?
  5. Can you show qualitative examples of the top-10 results before and after an update, for both a training query and a related test query, including a case where an update causes intrusion into another query's results?
9 comments
AuthorsOpenReview ↗

Response to Reviewer ebQG (1/5)

We thank the reviewer for the careful reading. Below, Q1–Q5 follow the order of the review: (Q1) seen vs. unseen targets, (Q2) semantic cache, (Q3) encoder fine-tuning, (Q4) ANN indexes, (Q5) concrete top-10 examples. All new results use the paper's 10 pools, the same train-query feedback and the same test splits, and report the mean over 5 folds unless stated otherwise. We see RIMU as complementary to the alternatives raised here: it writes feedback into the index within seconds, between the periodic encoder fine-tunes that reach targets without feedback, and it combines with a semantic cache, a bounded cache on top of RIMU being the best configuration in our experiments. In the revised PDF, changes made in response to your comments are shown in teal.

Q1: Can you report results separately for test queries whose gold received training feedback and for those whose gold did not?

Experimental setup

No test query is ever used for feedback. A test query is seen if one of its gold documents was also the gold of some feedback query in the same fold, and unseen otherwise (feedback-covered / uncovered golds). Table R1 compares RIMU with three methods that also edit corpus vectors: NUDGE-N, ERM (which pulls the golds toward the query) and Embedding Surgery.

Table R1: Qwen3-Embedding-8B, change in N@10 vs. frozen on seen / unseen test queries, mean over 5 folds. †: the method's grid chose no update (no vector moved). The Surgery columns are reported only in this response (the paper's Table 9 has no Surgery).

PoolSeen shareFrozen seen / unseenRIMU seenRIMU unseenNUDGE-N seenNUDGE-N unseenERM seenERM unseenSurgery seenSurgery unseen
SRA97%60.4 / 87.1+29.0−5.2+28.5−16.3+21.4−11.1+26.7+2.3
ToolRet-Cust.86%44.8 / 81.6+19.1−1.5+14.9−1.5+12.0−4.1+15.6−3.9
ToolRet-Web59%38.8 / 57.7+17.1−1.2+15.5−1.7+17.2−1.9+9.3−1.4
ToolRet-Code36%54.5 / 52.5+22.4−4.00.0†0.0†+20.6−14.9+15.3−3.1
MAIR-Code45%48.2 / 50.5+19.7−0.5+15.9−1.3+16.2−2.1+14.7−0.4
MAIR-Legal23%16.4 / 54.2+22.2−1.7+8.9−3.4+13.3−8.4+0.6−2.0
MAIR-Finance12%46.4 / 49.6+17.9−0.3+18.1−2.7+26.5−7.0+8.3+0.7
MAIR-Medical39%25.3 / 49.2+3.5−1.50.0†0.0†+3.7−17.4+3.1+2.0
MAIR-Academic6%12.0 / 47.0+8.4−1.60.0†0.0†+9.6−4.0−1.7−0.6
MAIR-Web17%27.4 / 48.9−4.9−0.8−1.7−0.8−4.5−3.0+1.4+0.4
Mean42%37.4 / 57.8+15.4−1.8+10.0−2.8+13.6−7.4+9.3−0.6

Findings

(1) By design, RIMU improves only the targets that received feedback; this is the trade-off of staying lightweight and drawing its gain from the feedback. RIMU leaves the encoder untouched and stores each correction in the vectors of the targets that received feedback, so later held-out queries for those targets can be retrieved better. This is what lets an update take seconds on a CPU; the price is that targets without feedback are not reached, which would require changing the encoder (Q3: training plus re-encoding the whole corpus). Table R1 shows this on average: the gain comes from the seen queries (+15.4 N@10), while the unseen ones lose 1.8 on average (up to 5.2 on SRA); seen queries can also lose, as on MAIR-Web (−4.9).

(2) Among methods that edit corpus vectors, RIMU gives the best balance between seen and unseen queries. Every such method trades one against the other. ERM pulls the golds onto the query and loses most on unseen queries (−7.4); NUDGE-N gains less on seen ones (+10.0) and still loses more on unseen (−2.8). Embedding Surgery is the most conservative: it loses least on unseen queries (−0.6), but gains far less on seen ones (+9.3 vs. +15.4), so its test N@10 stays lower (54.1 vs. 55.6). RIMU gains the most on seen queries at a small unseen cost (−1.8).

In the revised paper: Appendix D.4 (Table 9).

AuthorsOpenReview ↗

Response to Reviewer ebQG (2/5)

Q2: How does RIMU compare to a semantic-cache baseline that maps a test query to the golds of its nearest training queries?

Experimental setup

We implemented this baseline on the same splits as a kNN cache mixed with the retriever's ranking: the k nearest cached feedback queries with cosine ≥ τ vote for their golds, which are re-ranked together with the retriever's top-10 by retriever score + λ·similarity (k, τ, λ chosen on a dev split of the training queries: 20% are scored against a cache built from the other 80%, so no query can match its own entry; the cache is then rebuilt from all training queries and scored once on the test set; returning the neighbours' golds first, as in the question, is λ = 100). Table R2 gives the cache alone, RIMU alone (no cache) and a cache on top of RIMU (see below); BGE gives the same picture.

Table R2: Bounded semantic cache per pool, Qwen3-Embedding-8B, N@10 mean over 5 folds. Column groups: share of the feedback the cache can hold (most recent kept); cache + RIMU at 100% equals cache only (nothing is evicted) and is omitted. Bold: best of each row.

PoolFrozenRIMU aloneCache 10%Cache 25%Cache 50%Cache 100%Cache+RIMU 10%Cache+RIMU 25%Cache+RIMU 50%
SRA61.389.279.985.087.988.389.190.491.3
ToolRet-Cust.49.866.064.867.568.469.471.771.071.9
ToolRet-Web46.656.250.253.255.755.457.057.958.7
ToolRet-Code53.258.757.358.457.758.058.658.759.0
MAIR-Code49.558.053.454.655.956.458.258.158.5
MAIR-Legal45.649.346.345.646.246.849.848.947.7
MAIR-Finance49.351.149.249.249.649.650.950.750.7
MAIR-Medical40.040.440.039.940.040.840.039.839.9
MAIR-Academic44.943.944.844.744.644.544.043.944.1
MAIR-Web45.243.745.145.145.145.143.944.244.4
Mean48.555.653.154.355.155.456.356.456.6

Findings

(1) RIMU needs no cache that grows with the queries. A cache stores every feedback query and adds a lookup to every request, so it grows with the query traffic without bound (10⁶ cached queries take 15 GB with Qwen3-8B). RIMU writes the feedback into the vectors the index already holds, so serving needs no feedback-query cache and no extra lookup, and the index keeps its size (the ownership term only compares the feedback queries of the current cache while an update is computed, off the query path).

(2) With every feedback query kept, the cache is comparable to RIMU; with a bounded cache it falls behind. Keeping all feedback, the cache reaches about the same mean N@10 as RIMU (55.4 vs. 55.6): it is clearly stronger on ToolRet-Cust. (69.4 vs. 66.0), and RIMU is stronger on MAIR-Code and MAIR-Legal. Once the cache holds half of the feedback or less, its mean falls below RIMU (55.1, 54.3, 53.1). The cache's k, τ and λ are tuned per pool, while RIMU uses one configuration for all pools.

(3) A cache and RIMU combine. Under a storage budget a cache has to drop feedback; RIMU can write what the cache evicts into the vectors instead of discarding it. A bounded cache on top of RIMU gives the best configuration in our experiments (56.6 with half of the feedback cached).

In the revised paper: Appendix E.1 (Table 13).

AuthorsOpenReview ↗

Response to Reviewer ebQG (3/5)

Q3: How does it compare to fine-tuning the encoder (or LoRA) on the same feedback, in both accuracy and total cost, including corpus re-encoding?

Experimental setup

We fine-tuned the encoder with LoRA on exactly RIMU's feedback (r = 16 on the attention and dense layers; InfoNCE with 7 (BGE) / 3 (Qwen3-8B) hard negatives from the frozen top-50; 3 epochs), then re-encoded the whole pool and rebuilt the index (with exact search the re-encoded matrix is the index, so the times below are training plus re-encoding; behind an ANN index a full rebuild comes on top, Table R4). RIMU runs on a CPU (8 threads, no GPU); results in Table R3.

Table R3: RIMU vs. LoRA with all feedback (BGE: mean over 5 folds, 4 for SRA and MAIR-Web; Qwen3-8B: fold 0 only, so these frozen and RIMU columns differ from the 5-fold means of Table 1). N@10, better of RIMU and LoRA in bold. LoRA time: GPU training + re-encoding the whole pool on one A100 (RIMU: seconds on a CPU, 1.4–24 s per pool with BGE, 2–49 s with Qwen3-8B).

PoolDocsBGE FrozenBGE RIMUBGE LoRABGE LoRA time (min)Qwen FrozenQwen RIMUQwen LoRAQwen LoRA time (min)
SRA26,26252.482.789.349.6 + 2.661.389.390.6270.7 + 104.6
TR-cust.3,36733.661.273.313.3 + 0.251.466.782.358.8 + 0.9
TR-code3,79434.246.254.513.5 + 0.354.458.365.770.9 + 2.3
TR-web37,29237.451.760.357.1 + 3.246.256.367.0150.4 + 17.7
M-Code23,46332.547.851.39.7 + 2.248.756.759.250.5 + 22.0
M-Legal52,74330.133.334.927.5 + 5.344.248.246.463.6 + 182.9
M-Fin.49,08355.556.059.44.4 + 3.044.446.361.423.1 + 39.8
M-Med.30,37149.049.748.041.2 + 4.142.442.544.6108.45 + 179.0
M-Acad.111,35140.039.845.39.8 + 10.841.840.856.725.8 + 338.3
M-Web100,14948.748.447.0136.2 + 8.844.242.650.2721.3 + 198.4
Mean41.451.756.34047.954.862.4263.0

Findings

(1) LoRA is more accurate, at a much higher cost. Changing the encoder itself helps on most pools. But every update needs a GPU and must re-encode and re-index the whole corpus, up to 15.3 h for one update (Qwen3-8B on MAIR-Web).

(2) RIMU covers the time between encoder updates. Fine-tuning is run periodically, with a GPU and a re-encoding of the whole corpus, which for collections with millions of documents makes each run costly. RIMU takes seconds on a CPU and rewrites only the rows that received feedback, so feedback takes effect at once, in between those runs.

In the revised paper: Appendix E.2 (Table 14).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful reviews and constructive suggestions. The reviewers noted the following strengths:

  • Practical value without retraining (ebQG S1, 8SvY S1). ebQG found updating stored embeddings from verified feedback an interesting direction that corrects errors cheaply without retraining the encoder. 8SvY noted that absorbing feedback through vector editing offers deployment advantages.
  • A clear and innovative formulation (72AG S1, S2, ebQG S2, 8SvY S2). 72AG found the paper easy to follow and the idea innovative. ebQG found the QP formulation and its three constraints reasonable. 8SvY noted that the ranking constraints and the minimum-displacement objective balance retrieval effectiveness against preserving the original vectors.
  • Broad experimental coverage (8SvY S3, ebQG S3). The method is evaluated on multiple encoders and retrieval pools, with ablations and update overhead. ebQG noted the improved performance on tool retrieval.

We conducted new experiments to address each reviewer's comments. All results are in the revised paper, with changes highlighted in colour (one colour per reviewer).

Comparison with related methods

  • Embedding Surgery, concurrent work (8SvY W1). RIMU outperforms Embedding Surgery with both encoders. It differs in five design choices, among them writing only the golds under soft ranking constraints, a spherical update and ownership protection. Its mean N@10 is 51.6 vs. 45.9 with BGE and 55.6 vs. 54.1 with Qwen3-8B. The revised paper discusses it as concurrent work in Related Work and adds it to Table 1.
  • Semantic cache (ebQG Q2). RIMU reaches the accuracy of a cache without its storage. A cache that keeps every feedback query is on par with RIMU (55.4 vs. 55.6), but its storage grows with the query traffic; once bounded, it falls below RIMU. The two combine: a bounded cache on top of RIMU is the best configuration (56.6).
  • Encoder fine-tuning with LoRA (ebQG Q3). Fine-tuning is more accurate but far more costly; RIMU corrects errors cheaply without retraining. LoRA reaches 62.8 vs. 55.6 N@10 with Qwen3-8B, but each update needs a GPU run and re-encoding the whole corpus (4.6 h on average), while RIMU takes seconds on a CPU. This trade-off is expected: RIMU gives up some accuracy for an update cheap enough to run between fine-tuning runs.

Deployment behind ANN indexes and at scale

  • HNSW and IVF-PQ (ebQG Q4, 72AG Q1). RIMU's gain survives an ANN index because it changes little. It rewrites only the golds that received feedback, by small angles (cosine ≈0.985 to the old vector). Re-inserting the moved rows keeps 97–100% of the exact-search gain with HNSW, and at least 74% with compressed IVF-PQ codes (90% with Qwen3-8B); an exact buffer of the moved rows keeps all of it.
  • Larger corpora (ebQG Q4, 72AG Q2). An update stays cheap at scale. At 1.58M documents, each fold rewrites at most 0.1% of the corpus. Syncing HNSW takes 2–5 s against 92–621 s for a rebuild, and the QP solve, timed directly, takes 0.6 ms per feedback query at both 0.44M and 1.58M rows.

Scope of the claims

  • Seen vs. unseen targets (ebQG Q1, 72AG W1). As stated in the paper, RIMU's gain comes from the targets that received feedback. Seen test queries gain 15.4 N@10, at a small cost on unseen ones (−1.8). This is how RIMU remembers feedback without adding a cache. Among vector-editing methods, it balances seen and unseen queries best.
  • Qualitative examples and harmful intrusion (ebQG Q5, 8SvY W2). RIMU harms less than half as many test queries as ERM. Figure 8 shows top-10 lists before and after an update. A rewritten non-gold lowers a test query's N@10 for 7.1% of test queries, against 17.0% with ERM. Among queries with an intrusion, a gold is pushed out of the top-10 in 11–12%; ownership lowers harmful intrusions (7.6 → 7.1% with BGE) without changing test N@10.

Robustness

  • Noisy feedback (72AG W8). RIMU degrades least under wrong golds. With 30% hard wrong golds it keeps 83% of its clean gain over the frozen index.
  • Cache size and arrival order (72AG W5). Neither matters much. Changing B from 16 to 256 moves the mean N@10 by at most 0.2, and the 5 arrival orders by 0.7–1.0.
  • Multiple feedback rounds (72AG W2, W4). Later rounds keep earlier gains. Round-1 training N@10 stays at 76.9 of 77.8, and the held-out gain keeps growing (49.7 → 55.1).
  • Memory at scale (72AG W6). The 3.3 TiB figure belongs to the whole-stream joint ablation, not to RIMU. RIMU solves one QP per query within a cache, so its memory is bounded by one query's constraints; it runs on all pools at every B from 16 to 256 and on 1.58M-row corpora.

Our response to each reviewer gives the details and the locations in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.