acceptodds
Under review as a conference paper at ICLR 2027

When Familiarity Is Not Enough: Geometry and Regret in Dual-Process Memory Routing

Abstract

Searching again can recover a missing memory, but a router must decide whether to incur the cost before seeing the results. RF-Mem makes this decision from initial similarity summaries, even though Recollection changes the query direction. We show that this mismatch creates an information barrier: corpora with identical complete indexed scores can require opposite optimal actions, with exact Bayes and minimax regret gaps. The separation persists under published Bag semantics across explicit single-beam parameter ranges, including arbitrarily many finite rounds. We trace the missing information to memory–memory geometry and ask which geometric observations can improve the decision before Recollection begins. In a constructed family, even the initial Top- Gram block is uninformative, yet seed–candidate inner products yield an exact acquisition-cost versus regret tradeoff. Small observation budgets can leave the optimal action unchanged; a cached centroid resolves it through one scalar probe whose precision requirement tightens with candidate count. A continuous Bag model further connects stable retrieval and probabilistic relevance to a learned-probe guarantee under explicit sampling conditions. Together, these results link RF-Mem's routing barrier to the observations, costs, and precision needed to overcome it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Adaptive memory retrieval, Retrieval routing, Regret analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zb7YOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies what an RF-Mem router can infer before executing Recollection. It constructs corpora with identical complete query–memory similarity profiles but opposite cost-optimal routing actions, showing that memory–memory geometry omitted by the initial scores can determine the value of Recollection. The paper quantifies the resulting Bayes and minimax regret, extends the separation to explicit single-beam ranges of the published Bag procedure, characterizes adaptive Gram blocks sufficient for exact execution, and decomposes routing loss across geometry, truncation/identity, summary compression, and gate error. It further formulates geometric observation as a priced acquisition problem, derives exact tradeoffs for individual Gram probes and cached aggregate probes, and revisits the mean–entropy frontier, entropy certification and calibration, and the Top-K mean as an upper-tail statistic. The validation is entirely theoretical, through explicit constructions, proofs, and a synthetic logistic-relevance learning example.

Critical strengths

The strongest contribution is the clean separation between classifier error and missing information: the score-identical constructions show that improving a router trained on the same initial summaries cannot universally recover information required by the subsequent query update, while the exact Bayes/minimax gaps and information-ladder decomposition make this distinction quantitative. The paper is also unusually careful about implementation semantics, normalization, tie-breaking, duplicate-free perturbations, and the limits of each theorem. The correction of the low-entropy inversion, the treatment of the Top-K mean as a dependent upper-tail functional, and the acquisition-cost analysis further turn the negative result into useful guidance about what additional observations could resolve the ambiguity.

Critical weaknesses
  1. The paper does not establish whether the identified information barrier is prevalent or consequential in realistic memory corpora. The principal separations are constructed examples, while the positive learnability result assumes a specially constructed stable Bag family with logistic relevance and directly observed labels. There is no audit on RF-Mem’s original benchmarks showing how often similar initial-score profiles conceal materially different Recollection value, nor any evaluation showing that the proposed geometric probes improve an actual router. The results therefore establish possibility and worst-case necessity, but leave the practical significance of the mechanism unresolved.

  2. The most mechanism-specific conclusions cover a narrower implementation contract than the deployed system motivating the work. The Bag separations and probe results rely on single-beam settings, exact full-corpus search, fixed ties, the literal first-insertion Bag semantics, and a unique-relevant-item binary retrieval loss. Appendix A.3 explicitly states that the released PersonaBench backend instead uses MMR-based cumulative beam scores, epsilon-regularized normalization, and an insertion-prefix return, while Appendix A.4 leaves competing beams unresolved. Consequently, the current results do not directly cover at least one released RF-Mem backend or the practically important B>1 setting.

Questions for authors
  1. Can the authors report, on an existing RF-Mem benchmark, how frequently score-matched or near-score-matched instances exhibit substantially different Recollection advantages, and whether a seed–candidate or centroid probe improves routing risk after accounting for its measured cost? This evidence would materially affect my assessment of the contribution’s practical importance.

  2. Which of the principal separation and probe results remain valid for the released PersonaBench backend or for B > 1 with competing children? If no theorem currently transfers, can the authors provide a controlled experiment demonstrating that the same phenomenon occurs under those semantics?

4 comments

Thank you for identifying the two missing tests: whether the ambiguity matters on a benchmark, and whether it survives the beam and output rules of the released system. We ran both tests. On PersonaBench, the two actions differed in Recall@5 on 19 of 230 scored questions. One seed–candidate probe lowered held-out, cost-adjusted loss relative to the same gate trained on scores alone, but its recall gain came from only two questions; a centroid probe did not improve the primary result. Separately, we found an explicit pair with identical complete indexed scores and opposite optimal actions under both the released backend and a competing-beam instance of published Algorithm 3. These findings support different claims, so we address them separately.

1. How often does the ambiguity matter on PersonaBench? Does a probe pay for itself?

We ran both retrieval actions on every scored question, rather than inferring the unchosen action from a routing log. We used the six user corpora and both communities in the RF-Mem release, noise level 0.0, and the released gold-evidence Recall@5 rule, which allows multiple evidence items and alternative valid references. The evaluator excludes 33 subjective questions, leaving 230 of 263. We used the multi-qa-MiniLM-L6-cos-v1 encoder (FP32 ONNX, masked mean pooling, unit normalization), the release's FAISS index and retrieval methods, and K=5K=5, B=3B=3, F=1F=1, α=0.8\alpha=0.8, MMR weight 0.90.9, score floor 00, and depth cap 10. Familiarity reproduced the published Recall@5 to four decimals (0.4484). Recollection gave 0.4586 versus 0.4491 in Zhang et al., Table 2. We report this as an audit at the specified operating point, not an exact replication of every original result.

Let Ai=riR−riFA_i=r_{iR}-r_{iF} be the paired difference in Recall@5. Recollection helped on 14, hurt on 5, and tied on 211 questions. A hindsight selector that knew both recalls would achieve n−1∑imax⁡(riF,riR)=0.4718n^{-1}\sum_i\max(r_{iF},r_{iR})=0.4718, only 0.0132 above always using Recollection. This quantifies how much recall there was to recover at this operating point, before charging for observations. It does not assume the logistic model of Proposition 4.

What counts as a score match? We compared unordered pairs within the same user's corpus, so a coordinate of the complete score vector VV refers to the same memory for both queries. We separately examined the sorted top-five scores SS, which discard identities. For a representation XX, we used ∥Xi−Xj∥∞≤ε\lVert X_i-X_j\rVert_\infty\leq\varepsilon. Of 4,309 possible within-user pairs, none had exactly equal VV or SS. We report both the number of eligible pairs and the number of distinct questions they cover:

Matched representationε\varepsilonPairsQuestions covered (of 230)Pairs with ∣Ai−Aj∣≥0.20\lvert A_i-A_j\rvert\geq0.20Pairs preferring different actions at the primary price
Complete indexed VV0.0500——
Complete indexed VV0.108161/81/8
Complete indexed VV0.15781119/789/78
Sorted top-five SS0.0249705/495/49
Sorted top-five SS0.0547920866/47952/479
Sorted top-five SS0.101,578229217/1,578163/1,578

The last column compares the signs of Di=Ai−β(tiR−tiF)D_i=A_i-\beta(t_{iR}-t_{iF}), where tiat_{ia} is the measured warm retrieval time in ms and β=0.001\beta=0.001 recall units/ms. The distinction between columns matters. At ε=0.05\varepsilon=0.05 in SS, 13.8% of pairs differed in recall advantage by at least 0.20 and 10.9% preferred different cost-adjusted actions. In all 52 of the latter pairs, one member tied in recall, making Familiarity preferable on cost. No pair at the listed radii had a cost-adjusted advantage of at least +0.10+0.10 on one side and at most −0.10-0.10 on the other. Near matches thus reveal variation in the size of the benefit; we found no evidence here that large, two-sided reversals are common. These are descriptive pair frequencies: pairs overlap, distinct queries can have different relevant items, and near equality of SS or VV is not the exact observational equivalence in Theorem 1.

A held-out test at measured cost. The score gate used the five initial scores, their mean, and the released entropy summary. We added just one scalar to this same gate. With z(j)z_{(j)} the memory at initial rank jj, the alternatives were

P_{\rm cent}=\left\langle z_{(1)},\frac15\sum_{j=6}^{10}z_{(j)}\right\rangle.$$ Initial rank selected every candidate without access to relevance labels. The centroid was formed online. Each gate used standardized ridge regression (penalty 10) to predict $D_i$ and chose Recollection for a positive prediction. We trained on one entire community and evaluated on the other, then reversed the roles. Standardization and fitting used training data only. Nothing in this procedure presumes that relevance is logistic or that the benchmark admits a single marked item. For policy $\pi$, we measured $\widehat R_\beta(\pi)=n^{-1}\sum_i[1-r_{i,\pi(i)}+\beta T_i(\pi)]$. The online timer includes the initial candidate search for gates, vector gathering, any centroid construction, the dot product, the gate decision, and the selected retrieval path. The forced-arm timings used for the training targets were measured separately. The policy times below are means of per-question medians over five interleaved warm CPU runs. Query embeddings and the common memory index were available to all policies. These are **post-encoding retrieval costs**; they exclude offline gate training and generation, and are not comparable to end-to-end GPU latencies. | Policy, $\beta=0.001$ | Recollection calls / 230 | Held-out Recall@5 | Mean ms | Cost-adjusted loss ↓ | |---|---:|---:|---:|---:| | Always Familiarity | 0 | 0.44841 | 0.030 | 0.55162 | | Always Recollection | 230 | 0.45862 | 6.516 | 0.54789 | | Released mean/entropy gate (function defaults, probe width 5) | 178 | 0.45022 | 5.131 | 0.55491 | | Learned score gate | 129 | 0.45065 | 3.738 | 0.55309 | | Same gate + seed–candidate probe | 124 | 0.45587 | 3.603 | 0.54773 | | Same gate + centroid probe | 131 | 0.45065 | 3.765 | 0.55311 | The seed probe reduced loss versus its matched score-only baseline by $$(0.455870-0.450652)+0.001(3.738291-3.602960)\simeq 0.00535.$$ It changed 31 decisions; two improved recall, both in the same held-out community, and the other 29 tied in recall. The loss reduction was $-0.00033$ in one community and $+0.01113$ in the other. The seed gate was effectively tied with always using Recollection (loss difference $0.00016$), and the centroid did not improve on the learned score gate. At ten times the time price ($\beta=0.01$), always Familiarity had lower loss (0.55190) than either learned probe gate (0.55266 seed; 0.55265 centroid). We therefore see a modest, setting-specific benefit over a matched score gate, **not** evidence of a generally superior router. With two communities, we do not claim a population-level improvement or an effect on generated answers. **Proposed revision.** We will report the 14/5/211 outcome split and the matched-gate risk comparison in the main text, and put the matching grid, fold results and timing protocol in the appendix. We will distinguish three quantities in the discussion: variation in realized advantages, the value of an observation to an optimal decision, and the performance of a fitted gate. In particular, if $d_{X,P}=\mathbb E[L_F-L_R\mid X,P]$ and $d_X=\mathbb E[d_{X,P}\mid X]$, the gross Bayes value of observing $P$ is $$R_X^\star-R_{X,P}^\star =\mathbb E[(d_{X,P})_+-(d_X)_+] =\mathbb E\!\left[\min\{\mathbb E[(d_{X,P})_+\mid X],\mathbb E[(-d_{X,P})_+\mid X]\}\right].$$ The last equality follows by writing $d_X$ as the difference of the two conditional nonnegative parts. Information has decision value when it resolves **which action to take**; variation confined to one sign can be substantial yet have zero decision value. A priced probe must also repay its acquisition cost. This is why we report both columns of the matching audit and measure the actual gate. Proposition 4 will remain explicitly a learnability example under its stated model, rather than the basis for the benchmark claim.

2. Does separation survive the released backend and genuinely competing children?

What transfers from the submission. Theorem 2's parameter ranges and Theorem 5's exact mm-candidate optimum were proved for the single-beam published Bag contract. Neither proof transfers verbatim to the released backend. Theorem 3 already covers multiple beams for trace reconstruction under its own contract; reconstructing a run does not by itself show that two runs require opposite actions. For exact inner-product search and distance-based clustering with fixed randomness and ties, a full-Gram reconstruction can also account for the release's MMR and regularization: writing x=Wax=Wa and Γ=W⊤W\Gamma=W^\top W gives ⟨zi,x⟩=(Γa)i\langle z_i,x\rangle=(\Gamma a)_i and ∥Wa∥=a⊤Γa\lVert Wa\rVert=\sqrt{a^\top\Gamma a}. Pairwise redundancy uses Γij\Gamma_{ij}; division by ∥Wa∥+ϵ\lVert Wa\rVert+\epsilon, accumulated branch scores, Seen membership, and prefix order can then be followed explicitly.

One pair, two distinct implementations. Set q=(1,0,0)q=(1,0,0), index the eight memories by i=0,…,7i=0,\ldots,7, and let the only relevant index be y=5y=5. Define

z_i=(v_i,\sqrt{1-v_i^2}\cos\theta_i,\sqrt{1-v_i^2}\sin\theta_i).$$ Use the angle indices $$k^+=(12,2,1,3,9,12,0,10),\qquad k^-=(12,2,1,3,9,0,0,10).$$ Only the direction of item 5 perpendicular to $q$ changes. Thus $\langle q,z_i^+\rangle=\langle q,z_i^-\rangle=v_i$ for **every** index, and the entire initial top-five Gram block is unchanged. All initial scores are distinct, and neither corpus has duplicate embeddings. We evaluated the final returned indices, including deduplication and truncation, with $K=5$, $B=3$, $F=1$, and $\alpha=0.8$: | Retrieval contract | Final output in $+$ | Final output in $-$ | |---|---|---| | Familiarity, both contracts | $(0,1,2,3,4)$ | $(0,1,2,3,4)$ | | Released PersonaBench `mean_group`, actual MMR and prefix return | $(0,1,2,5,4)$ | $(0,1,2,4,7)$ | | Published Algorithm 3, two rounds, minimum-SSE nonempty $K$-means and first-insertion Bag scores | $(0,1,2,5)$ | $(0,1,2,4,7)$ | For the [released retrieval method](https://github.com/Zhang-Yingyi/ICLR2026_RF-Mem/blob/d6a2cb3416e8320bcb1439921d319729aeccec8e/RF_mem/personabench_data/retri_mdoel/EmbdRetri.py), we supplied the vectors directly and executed its retrieval method with the same MMR weight, score floor and depth cap as in the audit. Both runs stopped after two rounds. **Three distinct parents actually compete.** In $+$, seven children compete for three places; the target-bearing child ranks first (cumulative score 1.08333) and enters the returned prefix. In $-$, six children compete; the best target-bearing child scores 0.96772, below the third retained score 0.99045. The target is retrieved as a candidate and then pruned. The third-to-fourth score gaps exceed 0.0612 and 0.0142, respectively. For the published contract, second-round retrieval uses four memories per parent and partitions them into three nonempty clusters. The minimum-SSE solution merges the closest pair: a two-point cluster contributes $\lVert z_i-z_j\rVert^2/2$ to SSE, and the two singletons contribute zero. We enumerated all six possible pairs per parent. Nine children compete for three places in each corpus. We also checked the comparisons determining both implementations' traces with 45-decimal interval arithmetic; the published-contract beam cutoff gaps exceed 0.7057 and 0.6175. The distinction therefore survives genuine pruning and does not depend on a tie. **Decision and probe consequences.** Let $0<\Delta_c<1$ be a fixed extra Recollection cost in binary miss-loss units for this controlled pair. A score-only router chooses Recollection with the same probability $p$ in both corpora. The output table gives $$\operatorname{Reg}_+(p)=(1-p)(1-\Delta_c),\qquad \operatorname{Reg}_-(p)=p\Delta_c.$$ Minimizing their average over $p$ gives $\tfrac12\min\{\Delta_c,1-\Delta_c\}$. Equalizing them gives $p=1-\Delta_c$ and minimax regret $\Delta_c(1-\Delta_c)$. Thus the numerical trace is consequential for the *final decision*, not merely a change in a retrieved candidate. Moreover, the seed and candidate here are exactly initial **ranks 1 and 6**, the same observable used in the benchmark: $$\langle z_0,z_5^\pm\rangle =v_0v_5\ \pm\ \sqrt{(1-v_0^2)(1-v_5^2)}.$$ Thresholding this one scalar at $v_0v_5$ identifies the optimal action on the pair. With equal priors it is worth purchasing precisely when its cost is less than the unprobed Bayes gap, if this scalar and abstaining from acquisition are the available alternatives. This establishes a one-probe result for the demonstrated competing-beam case. **Proposed revision.** We will add the coordinates, the two return traces, and their decision losses to Appendix A.4.

Thank you for addressing my concerns adequately and I decide to raise my score

Reviewer_KxvTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors examine a routing decision in RF-Mem where Familiarity retrieves memories directly from the initial query and Recollection performs additional searches after updating the query using retrieved memories. The router must select between these strategies before observing the outcome of Recollection. The authors show that two memory corpora can have identical complete initial query-memory similarity profiles yet require opposite optimal routing decisions. This yields exact Bayes and minimax regret gaps for routers based only on initial scores. Also, the paper identifies memory-memory geometry as the missing information and studies whether additional Gram-matrix observations can improve the routing decisions. Under an explicitly constructed prior, it derives an exact trade-off between the cost of probing candidate similarities and the regret of an uninformed decision. Further results consider cached geometric summaries and a logistic relevance model.

Critical strengths
  1. The paper clearly distinguishes an information limitation from a classifier-capacity limitation. Its similarity score indistinguishability construction and exact regret calculations make the central routing failure mode precise.

  2. The analysis of information acquisition goes beyond identifying an impossibility result. The exact cost-regret trade-off for Gram probes gives a useful metric of when additional geometric observations are worthwhile.

Critical weaknesses
  1. The strongest theoretical guarantees depend on explicitly designed memory corpora, retrieval budgets, and single-beam configurations. The exact acquisition result also assumes a known candidate family and a specialized prior.

  2. The paper does not demonstrate that the indistinguishability phenomenon is prevalent or consequential in real memory datasets. The theoretically relevant memory is externally designated, and there is limited empirical evidence that geometric probing improves end-to-end routing quality after accounting for its actual acquisition cost.

Questions for authors
  1. Can the authors extend the separation or cost-regret analysis to more general multi-beam retrieval tasks and less structured candidate distributions?

  2. Can the authors evaluate real RF-Mem queries to measure the frequency of score-similar cases with different optimal actions and the net benefit of acquiring memory-memory geometric information?

6 comments

Thank you for asking for two tests of the paper's central claim: can the better action still change when several beams actually compete, and does observing geometry buy anything on unmodified benchmark queries? We now have evidence for both, with different implications. At each tested width B=2,…,6B=2,\ldots,6, changing one memory while preserving every initial score reversed the better action under the released PersonaBench backend. On unmodified PersonaBench questions, the benefit of buying a probe was much smaller: it saved 0.181 ms per query relative to a gate that saw the same six scores, at the same mean Recall@5. Always using Recollection still had the lowest measured loss at our chosen time price. Below we distinguish those results from the larger saving in a controlled family.

Q1 / Weakness 1. What survives genuine beam competition, and which assumptions on candidate distributions can be relaxed?

The target can lose in beam selection even when all initial scores are identical. For each B∈{2,3,4,5,6}B\in\{2,3,4,5,6\} we constructed two corpora with K=B+2K=B+2, fanout F=1F=1, α=0.8\alpha=0.8 and B+5B+5 distinct memory vectors. The query, initial top-KK vectors, indexed scores of all memories, and gold target at rank K+1K+1 are identical across the pair. Only the target's direction perpendicular to the query changes. We evaluated the final return under (i) two rounds of published Algorithm 3 with nonempty minimum-SSE clustering and first-insertion Bag scores, and (ii) the released PersonaBench method with MMR, cumulative beam scores, regularized normalization and insertion-prefix return. The release filled the budget in two rounds at both depth caps 2 and 10.

For all five widths under both contracts, Familiarity missed in both corpora and Recollection returned the target only in the positive corpus. In every negative released trace, a child carrying the target was proposed and pruned; all BB parents contributed proposals. For example, at B=3B=3, the common Familiarity output was (0,1,2,3,4)(0,1,2,3,4), the gold item was 5, and the two final Recollection outputs were (0,1,2,5,4)(0,1,2,5,4) and (0,1,2,4,7)(0,1,2,4,7). The comparisons at the beam cutoff are strict: interval checks gave minimum retained-versus-discarded gaps above 0.60630.6063 for the published contract and 0.001400.00140 for the released contract across the ten base instances. A separate released B=3,F=2,K=8B=3,F=2,K=8 example had first-round cluster sizes 3,2,13,2,1 and eight versus nine second-round proposals. It too gave opposite actions; for this last pair we have an execution trace rather than an interval certificate.

Why does the output difference matter for routing? With binary miss loss, extra Recollection cost 0<Δc<10<\Delta_c<1 and probability uu of choosing Recollection on the common observation, the two regrets are

r+(u)=(1−u)(1−Δc),r−(u)=uΔc.r_+(u)=(1-u)(1-\Delta_c),\qquad r_-(u)=u\Delta_c.

Minimizing their equal-prior average gives 12min⁡{Δc,1−Δc}\tfrac12\min\{\Delta_c,1-\Delta_c\}; equalizing them gives minimax regret Δc(1−Δc)\Delta_c(1-\Delta_c). A cross inner product between the seed and target distinguishes the two base corpora. The reversal also need not depend on a single gold item: give the B=3B=3 pair the same two references {0,5}\{0,5\}. All outputs contain 0, and only positive-corpus Recollection contains 5. The gross recall gains are 1/21/2 and 00, so the optimal actions differ for every 0<Δc<1/20<\Delta_c<1/2.

We also perturbed the five base pairs, embedding them in 384 dimensions and padding to 512 memories. At maximum angular displacements of 0.01, 0.05 and 0.20 rad, the original direction of separation survived in 160/160, 135/160 and 112/160 pairs, respectively (32 paired draws per width and radius). Thus the examples survive actual pruning and perturbations. We would state the five widths and the tested budgets explicitly: this is an existence result for those settings, not a transfer of the single-beam theorem to arbitrary implementations or widths.

Nonuniform priors change the optimal policy. In Theorem 5's known pool, let p0p_0 be the probability of no mark and pjp_j the unconditional probability that candidate jj is the unique mark. Sort the pjp_j in descending order, write Pt=∑j=1tp(j)P_t=\sum_{j=1}^t p_{(j)}, P0=0P_0=0, and set a=1−Δca=1-\Delta_c, b=Δcb=\Delta_c. With exact entry observations of common price cc, the minimum expected acquisition cost plus regret relative to the state-informed action is

G∗(p,c)=min⁡0≤t≤m{c∑s=0t−1(1−Ps)+min⁡[bp0,a(1−p0−Pt)]}.G^* (p,c) = \min_{0 \le t \le m} \{c \sum_{s=0}^{t-1} (1 - P_s) + \min \big[ b p_0, a (1-p_0 - P_t) \big] \}.

A hit settles the action. Only the all-negative path continues, so test s+1s+1 is paid for with probability 1−Ps1-P_s. After tt negatives, choosing Recollection errs only if no mark exists, whereas choosing Familiarity errs on the remaining marked mass. Descending order maximizes every prefix mass; for any cutoff it therefore minimizes both testing cost and terminal error. Optimizing the cutoff after sorting takes O(mlog⁡m)O(m\log m) time in total. For (p0,p1,p2)=(0.50,0.45,0.05)(p_0,p_1,p_2)=(0.50,0.45,0.05), Δc=0.5\Delta_c=0.5 and c=0.1c=0.1, testing zero, one and two candidates gives total risks 0.2500.250, 0.1250.125, and 0.1550.155. The best policy really does stop after one negative. The fair, uniform-location prior recovers Theorem 5's endpoint formula.

We can also relax exclusivity. In a separate two-round, single-beam Gram construction, a random set MM of marked candidates can have any known joint law. All candidates are fixed acceptable gold alternatives; the final Bag contains one if and only if M≠∅M\ne\varnothing. For a set UU of negative tests let s(U)=Pr⁡(M∩U=∅)s(U)=\Pr(M\cap U=\varnothing), p∅=Pr⁡(M=∅)p_\varnothing=\Pr(M=\varnothing), and let W(U)W(U) be the remaining optimal risk weighted by s(U)s(U). With unequal prices cjc_j,

W(U)=min⁡{bp∅, a[s(U)−p∅],min⁡j∉U[cjs(U)+W(U∪{j})]},W([m])=0.W(U)=\min \{ bp_\varnothing,\ a[s(U)-p_\varnothing], \min_{j\notin U}\left[c_j s(U)+W(U\cup\{j\})\right] \},\qquad W([m])=0.

The first two terms stop and choose an action; the third pays for one test and continues only on a negative. This is a standard cost-sensitive acquisition argument. The RF-Mem contribution is a Gram realization with those observations and final retrieval outcomes. The two extensions relax different assumptions: we obtain competing-beam separations and an exact cost analysis for more general known candidate laws. We do not infer a universal multi-beam cost formula or a way to discover the candidate pool from routine logs.

Q2 / Weakness 2. How often do similar scores hide different actions, and is a measured probe worth buying?

Unmodified questions: estimate the frequency before intervening. We forced both released actions on all 230 eligible questions from the two noise-0.0 PersonaBench communities (six users; the evaluator excludes 33 subjective questions). PersonaBench is a public synthetic benchmark, so this audits the motivating system's evaluation queries rather than production traffic. We used the benchmark's own gold evidence, including multiple or alternative references, with B=3,F=1,K=5B=3,F=1,K=5, α=0.8\alpha=0.8, MMR weight 0.9 and depth cap 10. Recollection increased Recall@5 on 14/230 questions, decreased it on 5/230, and tied on 211/230. Mean recall rose from 0.44841 to 0.45862; an oracle choosing the higher recall on each question would reach 0.47181 before costs.

Let Ai=QiR−QiFA_i=Q_{iR}-Q_{iF} be the recall gain and Di=Ai−β(tiR−tiF)D_i=A_i-\beta(t_{iR}-t_{iF}) its value after charging β=0.001\beta=0.001 recall units per measured ms. We counted all 4,309 within-user pairs, so full indexed vectors VV refer to the same memories. S5S_5 is the sorted top-five score vector. A match means ∥Xi−Xj∥∞≤ε\lVert X_i-X_j\rVert_\infty\le\varepsilon; a disagreement requires DiDj<0D_iD_j<0.

Profile and radiusMatched pairsDistinct questions coveredPairs with opposite priced actions
Exact VV or exact S5S_5000
VV, 0.05000
VV, 0.108161
VV, 0.15781119
S5S_5, 0.0249705
S5S_5, 0.0547920852
S5S_5, 0.101,578229163

At top-five radius 0.05, 52/479 pairs (10.9%) preferred different priced actions, involving 51 distinct questions; 66 of the 479 pairs differed in raw recall gain by at least 0.20. Each of the 52 pairs with opposite actions included a question for which the two actions tied on recall and Familiarity was cheaper. No tested radius produced a pair with a priced gain of at least +0.10+0.10 on one side and at most −0.10-0.10 on the other. Pairs share questions, and two naturally different questions can differ in ways other than their geometry. These counts answer the near-match question; they do not estimate the prevalence of exact collisions in production traffic.

Net routing value with the same search width and a measured price. We trained on one community, evaluated on the other, then reversed the split. Both gates saw the same six ranked scores and the same summaries from the same search. The sole additional feature was ⟨z(1),z(6)⟩\langle z_{(1)},z_{(6)}\rangle; ranks were chosen without using the gold labels. Both gates used ridge regression with penalty 10 and training targets priced on the evaluation CPU. For policy π\pi we measured

R^β(π)=1230∑i[1−Qi,π(i)+βTi(π)].\widehat R_\beta(\pi)=\frac1{230}\sum_i[1-Q_{i,\pi(i)}+\beta T_i(\pi)].

We timed candidate search, vector reads, any online probe construction, gate evaluation and the chosen retrieval path. Each time is the mean of per-question medians over five interleaved warm CPU runs. The common encodings and index were already available; offline training and answer generation are outside this measure.

Policy on unmodified PersonaBenchRecollection callsMean Recall@5Mean online msPriced loss
Always Familiarity00.448410.0150.55161
Always Recollection2300.458622.4460.54382
Six scores1540.455871.7150.54585
Six scores + seed–candidate inner product1400.455871.5340.54566
Ten scores1530.455871.7080.54584
Ten scores + uncached centroid1530.455871.7160.54585

The seed probe changed 28 decisions: one question gained 0.2 recall, one lost 0.2, and 26 tied. Its 0.000181 reduction in loss relative to the six-score gate therefore comes entirely from saving 1.71543−1.53447=0.180961.71543-1.53447=0.18096 ms per query. The uncached centroid gave no net gain. Always using Recollection was better at this price. This benchmark supports a small conditional saving over a learned score gate; it does not demonstrate superior generated-answer quality or a generally better production policy.

There is also a useful break-even calculation. Holding the tested policies, their per-question outputs and their times fixed, let PP denote the seed gate. Its empirical loss is below both constant actions exactly when

\frac{\bar Q_P-\bar Q_F}{\bar T_P-\bar T_F}.$$ The unrounded measurements give endpoints 0.00302128 and 0.00491261 recall units/ms. Below the lower endpoint Recollection's recall advantage outweighs the saved time; above the upper endpoint Familiarity's speed wins. This is a descriptive sensitivity analysis of the same measurements, not a newly selected primary price or a test of policies retrained at those prices.
Reviewer_B5gwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate how the memory system in RF-Mem can determine whether an additional retrieval step is worth its cost before executing it, and provide extensive theoretical analysis of this question.

Critical strengths

The authors provide extensive and detailed theoretical analysis, including very detailed derivations

Critical weaknesses
  1. The most apparent issue is that the entire paper appears to be merely a theoretical analysis of a single method, RF-Mem, attempting only to justify some of its design choices. It has no generality whatsoever and is essentially disconnected from other work.

  2. The paper contains no experiments and provides no evidence to support the authors’ arguments.

Questions for authors

Could the authors provide appropriate experiments to substantiate their conclusions?

7 comments

Thank you for asking both what the theory says beyond RF-Mem and what evidence supports it. The submitted version contained no benchmark experiments. We now have a direct test that does not use RF-Mem: on the official SciFact test set, two standard rerankers, applied to lexical and dense representations, can change the better routing action while all initial scores and gold references stay fixed. We also measured the size of the effect on unmodified RF-Mem questions. We would revise the paper to distinguish the general observation about routing, the claims specific to RF-Mem's final output, and the measured value of a probe.

Weakness 1. Is the contribution confined to justifying one algorithm?

The decision we study is whether to accept an initial retrieval or buy a second stage before seeing what it returns. This is a familiar question in information retrieval. Cronen-Townsend, Zhou and Croft (CIKM 2004) studied selective query expansion by comparing models of the original and expanded retrievals. Jeong et al. (NAACL 2024) learned to select retrieval strategies of different complexity for a question. Our question addresses what a router can infer before running the second stage if its observation is restricted to the initial scores. A router that reads question text or other features has a different observation, so our lower bound is not a performance claim about Adaptive-RAG. Likewise, Greiner, Grove and Roth (2002) already formulated the tradeoff between the cost of acquiring an attribute and the loss from choosing wrongly. The RF-Mem contribution there is to exhibit actual Gram observations and final retrieval outcomes for which that tradeoff can be solved exactly.

One counterexample, two retrieval methods with opposite incentives. Let qq and the document vectors ziz_i have unit norm, vi=q⊤ziv_i=q^\top z_i and Gji=zj⊤ziG_{ji}=z_j^\top z_i. Consider a second stage that retains the first-ranked seed z0z_0 and selects one other document by hi=avi+bG0ih_i=a v_i+bG_{0i}. In positive-only, one-seed Rocchio feedback, ranking by the updated query aq+bz0a q+b z_0 gives this rule with a,b>0a,b>0; normalizing the updated query does not change its order. In maximal marginal relevance, the first choice after the seed has a=λa=\lambda, b=−(1−λ)b=-(1-\lambda). The first method rewards proximity to what it already found; the second penalizes redundancy. Both compare two documents using an inner product missing from the initial query scores.

Take an orthonormal basis q,e1,e2q,e_1,e_2. Give the seed, a distractor and one gold target initial scores 1>s>d>v>01>s>d>v>0 and let

z_x=dq+\sqrt{1-d^2}\,e_2,\qquad z_y^\pm=vq\pm\sqrt{1-v^2}\,e_1 .$$ The cheap action returns $(z_0,z_x)$ in both corpora. The complete indexed profile is $(s,d,v)$ and even the initial top-two vectors themselves are identical. Put $\tau=\sqrt{(1-s^2)(1-v^2)}$. Since $G_{0y}^\pm=sv\pm\tau$ and $G_{0x}=sd$, the second-stage margin is $$h_y^\pm-h_x=(a+bs)(v-d)\pm b\tau .$$ For any fixed $a>0$ and $b\ne0$, choose $d$ sufficiently close to $v$ that $|a+bs|(d-v)<|b|\tau$. The two margins have opposite signs. When $b=0$, changing this geometry cannot change the ranking. This condition identifies the feature of the rule that matters. For our executed three-document instance $(s,d,v)=(.70,.51,.50)$, the margins in the two orientations were $+.296,-.323$ under Rocchio $(a,b)=(1,.5)$, and $-.190,+.181$ under MMR with $\lambda=.7$. The *same* pair favors purchasing the second stage in opposite orientations for the two methods. The seed stays first for positive Rocchio because it has the largest $v_i$ and $G_{00}=1$; MMR selects it first by definition. The separation also admits nonnegative vectors. Keep the seed tangent along $e_1$, use $e_1$ and $e_2$ as the two unit target tangents, and give the distractor a nonnegative tangent whose $e_1$ component is $\sqrt{1-v^2}/2$ and whose norm is $\sqrt{1-d^2}$. Such a tangent exists when $d$ is close enough to $v$. The margins become $(a+bs)(v-d)\pm b\tau/2$, which still straddle zero. Thus negative coordinates are not necessary for the existence result. With binary miss loss and an extra cost $0<\Delta<1$, any score-only router chooses the second stage with the same probability $u$ in both environments. Its regrets in the success and failure environments are $(1-u)(1-\Delta)$ and $u\Delta$. Minimizing their equal-prior average gives $\tfrac12\min\{\Delta,1-\Delta\}$; minimizing their maximum gives $\Delta(1-\Delta)$. Thus opposite optimal actions and the regret calculation do not require a beam, a Bag, a stop rule, or RF-Mem.

More generally, let D=Lstop−LsecondD=L_{\rm stop}-L_{\rm second}, including action costs, and write XX for the initial observation and ZZ for additional geometry. Before paying to observe ZZ, its Bayes value is

RX∗−RX,Z∗=E[(E[D∣X,Z])+−(E[D∣X])+].R_X^* - R_{X, Z}^* = \mathbb E [(\mathbb E [D \mid X, Z])_+ - (\mathbb E[D \mid X])_+ ].

Indeed, at either information level the best rule starts from the loss of stopping and subtracts the positive part of the conditional advantage. Geometry can change the return on many queries yet have zero decision value if the conditional advantages remain on the same side of zero. We use this elementary identity to define the empirical test; we do not claim the decision-theoretic identity as new.

Weakness 2 and question. What happens in experiments?

1. Outside RF-Mem: two rerankers on scientific evidence retrieval. We used all 300 queries in the BEIR SciFact test split and all 5,183 abstracts, with SciFact's external relevance judgments. There were no training queries or relevance labels used to set parameters. For each claim and representation we ranked the full corpus by cosine, took the first 100 candidates, and compared the original top five with (i) reranking by q+0.5 mean(z(1),z(2),z(3))q+0.5\,\mathrm{mean}(z_{(1)},z_{(2)},z_{(3)}) or (ii) five greedy MMR selections with λ=.7\lambda=.7. The representations were corpus-fitted, normalized TF–IDF and frozen multi-qa-MiniLM-L6-cos-v1. The candidate pool and output budget were identical across the two methods within a representation. No RF-Mem retrieval routine was used.

Representation / rerankerDifferent top-five set / 300Recall improved / worsened / tiedMean Recall@5: original → rerankedPaired mean change [95% bootstrap interval]Mean nDCG@5: original → reranked
TF–IDF / Rocchio2079 / 8 / 283.7276 → .7319+.0043 [−.0213, +.0293].6112 → .6082
TF–IDF / MMR2583 / 33 / 264.7276 → .6452−.0823 [−.1157, −.0508].6112 → .5672
MiniLM / Rocchio2137 / 2 / 291.6018 → .6132+.0113 [−.0054, +.0287].5235 → .5199
MiniLM / MMR2904 / 19 / 277.6018 → .5621−.0398 [−.0687, −.0128].5235 → .5031

The four rows use the same 300 claims. The intervals resample claims in pairs of original and reranked outcomes (10,000 draws); they do not treat the four rows as independent datasets. Depending on the setting, 69–97% of final sets changed, yet most claims tied on recall. MMR reduced mean recall here even though it recovered missed evidence for particular claims. Neither Rocchio interval establishes a positive mean effect. This separation between a changed ranking and a worthwhile action is precisely what a routing evaluation must measure.

2. Does missing geometry cause some of the action reversals? An observational change in evidence alone cannot answer that. For each query we kept every initial score, document identity and gold label fixed. Within its reranking pool we removed only the alignment between documents' components perpendicular to the query:

G~=vv⊤+diag⁡(1−vi2).\widetilde G=vv^\top+\operatorname{diag}(1-v_i^2).

This is feasible: choose distinct orthonormal eie_i perpendicular to qq and take z~i=viq+1−vi2ei\widetilde z_i=v_iq+\sqrt{1-v_i^2}e_i. Only the 100 pool documents need modification; the other 5,083 can remain fixed. The pool fits in 101 dimensions, including in the original 384-dimensional dense space. We ran the rerankers both on the resulting Gram matrix and on explicitly constructed vectors. Both methods then returned the original cosine order on all 300 queries under both representations. Setting the feedback weight to zero or setting MMR's λ\lambda to one also recovered the original order throughout. Direct-vector and full-Gram implementations agreed on every native top-five ordering.

Define the advantage of reranking by D=Recall@5rerank−Recall@5cosine−0.02D=\mathrm{Recall@5}_{\rm rerank}-\mathrm{Recall@5}_{\rm cosine}-0.02. The fixed surcharge is in recall units; it is not a timing estimate. We counted only strict reversals, DnativeDcontrol<0D_{\rm native}D_{\rm control}<0. In table order there were 9, 3, 7 and 4, representing 22 distinct claims across the four comparisons. For example, dense cosine missed SciFact claim 415's sole gold abstract (ID 6309659). Native MMR returned it third, and the orthogonal-residual control missed it: the priced advantage went from 1−.02=.981-.02=.98 to −.02-.02 at the very same initial scores. These are interventions on vectors with the original labels held fixed, not rewritten scientific documents or an estimate of how often two natural corpora have identical profiles.

3. Does the analysis suggest a way to reduce observation costs? For nonnegative TF–IDF, every redundancy term is nonnegative. At any MMR step with retained set SS,

λvi−(1−λ)max⁡j∈SGij ≤ λvi.\lambda v_i-(1-\lambda)\max_{j\in S}G_{ij}\ \le\ \lambda v_i .

We visited candidates in descending order of this known upper bound, acquired their similarities to SS on demand, and stopped when the next bound fell strictly below the best fully evaluated score. An unseen item then cannot win; cached pairs need not be bought twice. Compared with eager MMR using the same pool and caching, this reproduced the identical five IDs in the identical order on 300/300 claims, with 27,492 rather than 117,000 distinct document–document inner products (76.5% fewer; median 78 rather than 390 per query). This is a concrete benefit outside RF-Mem for a specified observation interface. It counts inner products after the pool and query scores are available; it is not a runtime or answer-quality gain.

4. What the original RF-Mem setting adds. The generic two-stage argument does not tell us whether a missed document survives competing beams, enters the Bag, or appears in the returned prefix. We tested these separately:

  • Final returns and beam competition. Reconstructing from the augmented Gram matrix reproduced both actions' ordered outputs for all 230 PersonaBench and 589 PersonaMem unmodified queries. On explicit score-identical pairs at each B=2,3,4,5,6B=2,3,4,5,6, both the published two-round contract and the released PersonaBench backend produced opposite optimal actions. In every negative released trace the target child was proposed and pruned. With benchmark-labelled PersonaBench questions, varying only the rank-six item's tangent orientation (chosen without labels) gave 0, 1 and 21 strict action reversals out of 230 at intervention strengths .25,.50.25,.50 (primary), and .90.90, at the fixed .02 surcharge.
  • Unmodified benchmark frequency and measured cost. Recollection improved/tied/worsened benchmark Recall@5 on 14/211/5 of 230 PersonaBench questions. There were no exact indexed-score matches among 4,309 within-user pairs. At top-five score distance at most .05, 52/479 pairs preferred different priced actions; each included one raw-recall tie. Holding out each community in turn, adding one rank-1/rank-6 inner product to the same six-score gate preserved mean recall at .45587, cut Recollection calls 154 → 140 and reduced online time 1.715 → 1.534 ms/query, including acquisition. At .001 recall units/ms, the measured loss saving was .000181 relative to the matched gate. Always Recollection still had lower loss (.54382 versus .54566). These results concern evidence retrieval on a synthetic memory benchmark, not generated answers.
  • Independent paid-policy test and quantitative predictions. In 1,280 fresh, constructed environments a policy frozen before testing retained every one of 661 hits, cut Recollection calls 1,280 → 896 and lowered measured online time 2.480 → 1.849 ms (25.4%) versus always recollecting. Candidate search, vector reads, the probe and the chosen retrieval branch were timed. This larger effect belongs to the tested family. In Theorem 5's separately specified single-mark model, exact enumeration of actual retrieval outcomes also matched the predicted regret at all 393 budget settings and the optimal priced risk at all 54 price settings. The latter checks the cost formula under its assumptions; it does not turn the formula into a cost law for every retriever.

Proposed manuscript changes. We would frame the introduction around selective second-stage retrieval, use the three-document argument to identify precisely what transfers, and place the SciFact table and the measured PersonaBench comparison in the main text. We would put the explicit constructions, candidate and cost definitions, score-preserving controls, complete outcomes and timing protocol in the appendix. The revised claims would distinguish an information limit, its occurrence under a controlled intervention, and a measured net saving on an unmodified benchmark.

Comments on the paper

1 comment

We thank the reviewers for asking when a limitation of score-only routing changes a real retrieval decision, and whether it is worth paying for the missing information. All additions and revisions relative to the submitted manuscript are marked in red. Section 8 and Tables 1–2 (pp. 8–9) distinguish three outcomes: a change in the returned evidence, a change in the preferred action, and a gain after paying for the probe.

Reviewer 1 — Released backend and benchmark frequency. For each beam width B=2,…,6B=2,\ldots,6, with F=1F=1 and K=B+2K=B+2, we give two corpora with identical complete initial scores but opposite optimal actions under both published Algorithm 3 and the released PersonaBench backend (binary miss loss, extra cost 0<Δc<10<\Delta_c<1). In every negative example on the released backend, a child carrying the gold item was proposed and then pruned (Appendix A.4).

On all 230 eligible, unmodified PersonaBench questions, Recollection improved Recall@5 on 14, lowered it on 5, and tied on 211. None of the 4,309 within-user pairs matched exactly on the full or top-five scores. Of 479 pairs within 0.05 on each of the five leading scores, 52 favored different actions after charging for time; each included a question on which recall tied and the cheaper action won (Appendix G.2). A separate intervention held every initial score and the benchmark gold labels fixed: at a fixed surcharge of 0.02 recall units, changing only the sixth-ranked memory reversed the decision on 1/230 questions at the primary perturbation strength and 21/230 at the strongest (Appendix G.4). These intervention counts do not estimate natural prevalence.

Reviewer 2 — Candidate assumptions and the net benefit of probing. We solve the acquisition problem for any known prior on the location or absence of a single mark. With a common probe price, candidates are tested in decreasing order of mark probability, and stopping after one negative test can be strictly optimal. A separate recursion covers correlated marks and unequal prices; both results still require a known candidate pool (Appendices F.2–F.3).

On PersonaBench, with each community held out in turn, adding a seed–candidate inner product to a gate given the same six scores reduced Recollection calls from 154 to 140 and mean online time from 1.715 to 1.534 ms at unchanged mean Recall@5 (0.45587). Timing includes candidate search, the probe, and the chosen retrieval. The uncached centroid gave no net gain, and always running Recollection had the lowest loss at the tested price of 0.001 recall units/ms (Table 1; Appendix G.3). The larger time saving of 25.4% was measured separately on 1,280 held-out environments from the constructed family, where the gate retained all 661 baseline hits (Appendix G.6).

Reviewer 3 — Generality beyond RF-Mem and independent experiments. The same three-document construction applies to Rocchio feedback, which rewards similarity to the seed, and maximal marginal relevance (MMR), which penalizes it. For either method, changing the seed–candidate inner product alone can reverse whether reranking is worth its cost (Appendix F.1).

We tested both methods with TF–IDF and MiniLM on the same 300 SciFact claims, without RF-Mem code. At fixed initial scores and gold judgments, changing the geometry reversed the preferred action for 22 distinct claims across the four settings at a surcharge of 0.02 recall units. On the unmodified corpus, MMR lowered mean recall with both representations. For lexical MMR, we also reproduced all 300 ordered top-five lists with 76.5% fewer document–document inner products, an operation count rather than a latency estimate (Table 2; Appendix H).

Appendix G.5 reports a separate check of the theory: Gram matrices reproduced both actions’ final ordered outputs on all 819 unmodified PersonaBench and PersonaMem queries. The source package includes the protocols, query-level results, and scripts for reproducing both main tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.