- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies what an RF-Mem router can infer before executing Recollection. It constructs corpora with identical complete query–memory similarity profiles but opposite cost-optimal routing actions, showing that memory–memory geometry omitted by the initial scores can determine the value of Recollection. The paper quantifies the resulting Bayes and minimax regret, extends the separation to explicit single-beam ranges of the published Bag procedure, characterizes adaptive Gram blocks sufficient for exact execution, and decomposes routing loss across geometry, truncation/identity, summary compression, and gate error. It further formulates geometric observation as a priced acquisition problem, derives exact tradeoffs for individual Gram probes and cached aggregate probes, and revisits the mean–entropy frontier, entropy certification and calibration, and the Top-K mean as an upper-tail statistic. The validation is entirely theoretical, through explicit constructions, proofs, and a synthetic logistic-relevance learning example.
Critical strengths
The strongest contribution is the clean separation between classifier error and missing information: the score-identical constructions show that improving a router trained on the same initial summaries cannot universally recover information required by the subsequent query update, while the exact Bayes/minimax gaps and information-ladder decomposition make this distinction quantitative. The paper is also unusually careful about implementation semantics, normalization, tie-breaking, duplicate-free perturbations, and the limits of each theorem. The correction of the low-entropy inversion, the treatment of the Top-K mean as a dependent upper-tail functional, and the acquisition-cost analysis further turn the negative result into useful guidance about what additional observations could resolve the ambiguity.
Critical weaknesses
-
The paper does not establish whether the identified information barrier is prevalent or consequential in realistic memory corpora. The principal separations are constructed examples, while the positive learnability result assumes a specially constructed stable Bag family with logistic relevance and directly observed labels. There is no audit on RF-Mem’s original benchmarks showing how often similar initial-score profiles conceal materially different Recollection value, nor any evaluation showing that the proposed geometric probes improve an actual router. The results therefore establish possibility and worst-case necessity, but leave the practical significance of the mechanism unresolved.
-
The most mechanism-specific conclusions cover a narrower implementation contract than the deployed system motivating the work. The Bag separations and probe results rely on single-beam settings, exact full-corpus search, fixed ties, the literal first-insertion Bag semantics, and a unique-relevant-item binary retrieval loss. Appendix A.3 explicitly states that the released PersonaBench backend instead uses MMR-based cumulative beam scores, epsilon-regularized normalization, and an insertion-prefix return, while Appendix A.4 leaves competing beams unresolved. Consequently, the current results do not directly cover at least one released RF-Mem backend or the practically important B>1 setting.
Questions for authors
-
Can the authors report, on an existing RF-Mem benchmark, how frequently score-matched or near-score-matched instances exhibit substantially different Recollection advantages, and whether a seed–candidate or centroid probe improves routing risk after accounting for its measured cost? This evidence would materially affect my assessment of the contribution’s practical importance.
-
Which of the principal separation and probe results remain valid for the released PersonaBench backend or for B > 1 with competing children? If no theorem currently transfers, can the authors provide a controlled experiment demonstrating that the same phenomenon occurs under those semantics?
4 comments
A held-out test at measured cost. The score gate used the five initial scores, their mean, and the released entropy summary. We added just one scalar to this same gate. With the memory at initial rank , the alternatives were
P_{\rm cent}=\left\langle z_{(1)},\frac15\sum_{j=6}^{10}z_{(j)}\right\rangle.$$ Initial rank selected every candidate without access to relevance labels. The centroid was formed online. Each gate used standardized ridge regression (penalty 10) to predict $D_i$ and chose Recollection for a positive prediction. We trained on one entire community and evaluated on the other, then reversed the roles. Standardization and fitting used training data only. Nothing in this procedure presumes that relevance is logistic or that the benchmark admits a single marked item. For policy $\pi$, we measured $\widehat R_\beta(\pi)=n^{-1}\sum_i[1-r_{i,\pi(i)}+\beta T_i(\pi)]$. The online timer includes the initial candidate search for gates, vector gathering, any centroid construction, the dot product, the gate decision, and the selected retrieval path. The forced-arm timings used for the training targets were measured separately. The policy times below are means of per-question medians over five interleaved warm CPU runs. Query embeddings and the common memory index were available to all policies. These are **post-encoding retrieval costs**; they exclude offline gate training and generation, and are not comparable to end-to-end GPU latencies. | Policy, $\beta=0.001$ | Recollection calls / 230 | Held-out Recall@5 | Mean ms | Cost-adjusted loss ↓ | |---|---:|---:|---:|---:| | Always Familiarity | 0 | 0.44841 | 0.030 | 0.55162 | | Always Recollection | 230 | 0.45862 | 6.516 | 0.54789 | | Released mean/entropy gate (function defaults, probe width 5) | 178 | 0.45022 | 5.131 | 0.55491 | | Learned score gate | 129 | 0.45065 | 3.738 | 0.55309 | | Same gate + seed–candidate probe | 124 | 0.45587 | 3.603 | 0.54773 | | Same gate + centroid probe | 131 | 0.45065 | 3.765 | 0.55311 | The seed probe reduced loss versus its matched score-only baseline by $$(0.455870-0.450652)+0.001(3.738291-3.602960)\simeq 0.00535.$$ It changed 31 decisions; two improved recall, both in the same held-out community, and the other 29 tied in recall. The loss reduction was $-0.00033$ in one community and $+0.01113$ in the other. The seed gate was effectively tied with always using Recollection (loss difference $0.00016$), and the centroid did not improve on the learned score gate. At ten times the time price ($\beta=0.01$), always Familiarity had lower loss (0.55190) than either learned probe gate (0.55266 seed; 0.55265 centroid). We therefore see a modest, setting-specific benefit over a matched score gate, **not** evidence of a generally superior router. With two communities, we do not claim a population-level improvement or an effect on generated answers. **Proposed revision.** We will report the 14/5/211 outcome split and the matched-gate risk comparison in the main text, and put the matching grid, fold results and timing protocol in the appendix. We will distinguish three quantities in the discussion: variation in realized advantages, the value of an observation to an optimal decision, and the performance of a fitted gate. In particular, if $d_{X,P}=\mathbb E[L_F-L_R\mid X,P]$ and $d_X=\mathbb E[d_{X,P}\mid X]$, the gross Bayes value of observing $P$ is $$R_X^\star-R_{X,P}^\star =\mathbb E[(d_{X,P})_+-(d_X)_+] =\mathbb E\!\left[\min\{\mathbb E[(d_{X,P})_+\mid X],\mathbb E[(-d_{X,P})_+\mid X]\}\right].$$ The last equality follows by writing $d_X$ as the difference of the two conditional nonnegative parts. Information has decision value when it resolves **which action to take**; variation confined to one sign can be substantial yet have zero decision value. A priced probe must also repay its acquisition cost. This is why we report both columns of the matching audit and measure the actual gate. Proposition 4 will remain explicitly a learnability example under its stated model, rather than the basis for the benchmark claim.2. Does separation survive the released backend and genuinely competing children?
What transfers from the submission. Theorem 2's parameter ranges and Theorem 5's exact -candidate optimum were proved for the single-beam published Bag contract. Neither proof transfers verbatim to the released backend. Theorem 3 already covers multiple beams for trace reconstruction under its own contract; reconstructing a run does not by itself show that two runs require opposite actions. For exact inner-product search and distance-based clustering with fixed randomness and ties, a full-Gram reconstruction can also account for the release's MMR and regularization: writing and gives and . Pairwise redundancy uses ; division by , accumulated branch scores, Seen membership, and prefix order can then be followed explicitly.
One pair, two distinct implementations. Set , index the eight memories by , and let the only relevant index be . Define
z_i=(v_i,\sqrt{1-v_i^2}\cos\theta_i,\sqrt{1-v_i^2}\sin\theta_i).$$ Use the angle indices $$k^+=(12,2,1,3,9,12,0,10),\qquad k^-=(12,2,1,3,9,0,0,10).$$ Only the direction of item 5 perpendicular to $q$ changes. Thus $\langle q,z_i^+\rangle=\langle q,z_i^-\rangle=v_i$ for **every** index, and the entire initial top-five Gram block is unchanged. All initial scores are distinct, and neither corpus has duplicate embeddings. We evaluated the final returned indices, including deduplication and truncation, with $K=5$, $B=3$, $F=1$, and $\alpha=0.8$: | Retrieval contract | Final output in $+$ | Final output in $-$ | |---|---|---| | Familiarity, both contracts | $(0,1,2,3,4)$ | $(0,1,2,3,4)$ | | Released PersonaBench `mean_group`, actual MMR and prefix return | $(0,1,2,5,4)$ | $(0,1,2,4,7)$ | | Published Algorithm 3, two rounds, minimum-SSE nonempty $K$-means and first-insertion Bag scores | $(0,1,2,5)$ | $(0,1,2,4,7)$ | For the [released retrieval method](https://github.com/Zhang-Yingyi/ICLR2026_RF-Mem/blob/d6a2cb3416e8320bcb1439921d319729aeccec8e/RF_mem/personabench_data/retri_mdoel/EmbdRetri.py), we supplied the vectors directly and executed its retrieval method with the same MMR weight, score floor and depth cap as in the audit. Both runs stopped after two rounds. **Three distinct parents actually compete.** In $+$, seven children compete for three places; the target-bearing child ranks first (cumulative score 1.08333) and enters the returned prefix. In $-$, six children compete; the best target-bearing child scores 0.96772, below the third retained score 0.99045. The target is retrieved as a candidate and then pruned. The third-to-fourth score gaps exceed 0.0612 and 0.0142, respectively. For the published contract, second-round retrieval uses four memories per parent and partitions them into three nonempty clusters. The minimum-SSE solution merges the closest pair: a two-point cluster contributes $\lVert z_i-z_j\rVert^2/2$ to SSE, and the two singletons contribute zero. We enumerated all six possible pairs per parent. Nine children compete for three places in each corpus. We also checked the comparisons determining both implementations' traces with 45-decimal interval arithmetic; the published-contract beam cutoff gaps exceed 0.7057 and 0.6175. The distinction therefore survives genuine pruning and does not depend on a tie. **Decision and probe consequences.** Let $0<\Delta_c<1$ be a fixed extra Recollection cost in binary miss-loss units for this controlled pair. A score-only router chooses Recollection with the same probability $p$ in both corpora. The output table gives $$\operatorname{Reg}_+(p)=(1-p)(1-\Delta_c),\qquad \operatorname{Reg}_-(p)=p\Delta_c.$$ Minimizing their average over $p$ gives $\tfrac12\min\{\Delta_c,1-\Delta_c\}$. Equalizing them gives $p=1-\Delta_c$ and minimax regret $\Delta_c(1-\Delta_c)$. Thus the numerical trace is consequential for the *final decision*, not merely a change in a retrieved candidate. Moreover, the seed and candidate here are exactly initial **ranks 1 and 6**, the same observable used in the benchmark: $$\langle z_0,z_5^\pm\rangle =v_0v_5\ \pm\ \sqrt{(1-v_0^2)(1-v_5^2)}.$$ Thresholding this one scalar at $v_0v_5$ identifies the optimal action on the pair. With equal priors it is worth purchasing precisely when its cost is less than the unprobed Bayes gap, if this scalar and abstaining from acquisition are the available alternatives. This establishes a one-probe result for the demonstrated competing-beam case. **Proposed revision.** We will add the coordinates, the two return traces, and their decision losses to Appendix A.4.Thank you for addressing my concerns adequately and I decide to raise my score
Thank you for identifying the two missing tests: whether the ambiguity matters on a benchmark, and whether it survives the beam and output rules of the released system. We ran both tests. On PersonaBench, the two actions differed in Recall@5 on 19 of 230 scored questions. One seed–candidate probe lowered held-out, cost-adjusted loss relative to the same gate trained on scores alone, but its recall gain came from only two questions; a centroid probe did not improve the primary result. Separately, we found an explicit pair with identical complete indexed scores and opposite optimal actions under both the released backend and a competing-beam instance of published Algorithm 3. These findings support different claims, so we address them separately.
1. How often does the ambiguity matter on PersonaBench? Does a probe pay for itself?
We ran both retrieval actions on every scored question, rather than inferring the unchosen action from a routing log. We used the six user corpora and both communities in the RF-Mem release, noise level 0.0, and the released gold-evidence Recall@5 rule, which allows multiple evidence items and alternative valid references. The evaluator excludes 33 subjective questions, leaving 230 of 263. We used the multi-qa-MiniLM-L6-cos-v1 encoder (FP32 ONNX, masked mean pooling, unit normalization), the release's FAISS index and retrieval methods, and K=5, B=3, F=1, α=0.8, MMR weight 0.9, score floor 0, and depth cap 10. Familiarity reproduced the published Recall@5 to four decimals (0.4484). Recollection gave 0.4586 versus 0.4491 in Zhang et al., Table 2. We report this as an audit at the specified operating point, not an exact replication of every original result.
Let Ai=riR−riF be the paired difference in Recall@5. Recollection helped on 14, hurt on 5, and tied on 211 questions. A hindsight selector that knew both recalls would achieve n−1∑imax(riF,riR)=0.4718, only 0.0132 above always using Recollection. This quantifies how much recall there was to recover at this operating point, before charging for observations. It does not assume the logistic model of Proposition 4.
What counts as a score match? We compared unordered pairs within the same user's corpus, so a coordinate of the complete score vector V refers to the same memory for both queries. We separately examined the sorted top-five scores S, which discard identities. For a representation X, we used ∥Xi−Xj∥∞≤ε. Of 4,309 possible within-user pairs, none had exactly equal V or S. We report both the number of eligible pairs and the number of distinct questions they cover:
The last column compares the signs of Di=Ai−β(tiR−tiF), where tia is the measured warm retrieval time in ms and β=0.001 recall units/ms. The distinction between columns matters. At ε=0.05 in S, 13.8% of pairs differed in recall advantage by at least 0.20 and 10.9% preferred different cost-adjusted actions. In all 52 of the latter pairs, one member tied in recall, making Familiarity preferable on cost. No pair at the listed radii had a cost-adjusted advantage of at least +0.10 on one side and at most −0.10 on the other. Near matches thus reveal variation in the size of the benefit; we found no evidence here that large, two-sided reversals are common. These are descriptive pair frequencies: pairs overlap, distinct queries can have different relevant items, and near equality of S or V is not the exact observational equivalence in Theorem 1.