- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper adapts retrieval using feedback about which chunks are useful. It combines per-chunk Beta beliefs with temporal decay and multiple semantic keys fitted to successful query embeddings through a von Mises–Fisher mixture. On a 20,640-chunk personal corpus, a simulation with 1,405 synthetic queries reports Recall@5 improving from 0.73 to 0.97.
Critical strengths
-
Adapting retrieval to recurring information needs is a useful problem.
-
Queries are scored before their feedback is applied, which supports the evaluation of adaptation across different queries.
Critical weaknesses
-
The novelty of adapting document vectors from query feedback is not sufficiently established. NUDGE (ICLR 2025) already adjusts document embeddings using labeled queries, but is neither discussed nor compared against.
-
The main experiment does not demonstrate a benefit from combining Bayesian beliefs with adapted keys. In Table 1, Keys alone and full BRAG both achieve 0.97 Recall@5 and 3.8 average rank.
-
The Bayesian formulation needs a clearer technical contribution. The conjugate updates and vMF mixture fitting are standard, while Appendix B.4 ultimately adopts the prediction recursion as a definition after approximations to the OU dynamics.
-
The evaluated feedback does not match the stated notion of utility closely enough. Utility is defined through actual downstream use, whereas Appendix A.6 labels whether a chunk contains information answering the query. Retrieval scores alone therefore do not establish better generated answers or better estimates of actual use.
Questions for authors
see weakness
4 comments
Response to Reviewer oBS3 (2/2)
Response to Reviewer oBS3 (2/2)
W3. "The Bayesian formulation needs a clearer technical contribution. The conjugate updates and vMF mixture fitting are standard, while Appendix B.4 ultimately adopts the prediction recursion as a definition after approximations to the OU dynamics."
The revised version makes the main contribution of the paper clearer. It is a generative model, inspired by, but quite different from, classic library and human memory models. The model illuminates optimality conditions for inference and control. The approximation methods are not novel. They are the Laplace bridge, to move between a logit-Gaussian and a Beta belief, and a factorized posterior with one Beta per chunk. However, these methods had not been used in the classic library and human-memory literature, and they are essential for real-time practical use.
The wording of the paper was confusing regarding the relationship between the OU dynamics in logit space and the Beta recursion. We made this clearer in the revised submission, which now reads "The discrete-time OU process on the logit of utility defines the dynamics of the model. The Laplace bridge approximation turns it into a recursion on the Beta parameters (e, n). Once the bridge is applied, the filter no longer needs the underlying logit process and propagates the belief directly in the Beta family."
To measure the approximations we simulated data from BRAG's own generative model (30 or 50 chunks, 2 keys each, 200 steps, 5 seeds, 55 runs) and compared BRAG with a joint particle filter with 65.5K particles (doubling to 131K moved posterior means by less than 0.003). Since the exact posterior is calibrated under the model, the distance between BRAG's belief and that posterior measures both the approximation error and the loss of calibration.
The simplified recursion. BRAG's beliefs stay within 0.02-0.09 of the exact posterior. Factorizing the posterior across chunks adds a mean error of 0.002-0.007, at the particle filter's own noise floor. The Beta projection adds 0.02 nats KL, and the Laplace bridge another 0.01 nats, with top-3 agreement with the exact filter of 0.975 (0.988 without it). Recall@3 stays within ±0.015 of the exact filter.
W4. "The evaluated feedback does not match the stated notion of utility closely enough. Utility is defined through actual downstream use, whereas Appendix A.6 labels whether a chunk contains information answering the query. Retrieval scores alone therefore do not establish better generated answers or better estimates of actual use."
Yes, this was confusing. The revised paper defines utility through the generative model, not through use. Each query q_t is generated by one chunk, and U_t = i means q_t was drawn from p(q | i). A chunk's utility X_i is then the rate at which it generates queries, its demand. The curator's label ("does this chunk contain the information asked for?") is the observable evidence about which chunk generated the query. When two chunks both contain the answer, both labels are noisy evidence for the same latent cause, which the responsibility γ splits between them. The feedback we evaluate is therefore an observation of the quantity the model defines.
To test answers as well as retrieval, we gave the same LLM the chunks retrieved by plain search or by BRAG and graded its answers with LongMemEval's official judge prompt. Accuracy rose from 0.552 to 0.702 on LongMemEval-M (+0.150 [+0.118, +0.183]) and from 0.457 to 0.553 on PersonaMem-v2 (+0.096 [+0.086, +0.107]).
Dear Reviewer oBS3,
Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.
Follow-up before the discussion closes tonight
Dear Reviewer oBS3,
The discussion period closes tonight (November 18, 11:59 pm AoE), so we wanted to follow up once more. In our response we compared BRAG with NUDGE on three of NUDGE's own benchmarks, using its code and splits, and added NUDGE to the two public datasets. We also showed where the Bayesian beliefs add to the adapted keys, which is when a new chunk makes an old one obsolete, and clarified the generative model behind the formulation. Every change is marked in blue in the revised PDF.
Could you let us know whether these results address your concerns? If they do, we would be grateful if you would consider updating your score. If anything is still unclear, we will answer right away.
Best regards, The Authors
Response to Reviewer oBS3 (1/2)
Response to Reviewer oBS3 (1/2)
Thank you for pointing us to NUDGE.
BRAG is a generative framework that clarifies the conditions under which previous approaches are optimal and suggests new ones. For example, NUDGE can be interpreted as BRAG with one key per chunk (K = 1), infinite utility lifetime, and a hard prior on the embedding moves. What BRAG adds is the overall generative framework, including a step that grows with evidence, several keys beside the original embedding, online updates, and a utility that fades with time. We made this clearer in the revised related work section.
We compared against NUDGE on three of its seven text benchmarks, using its code, splits and one of its embedders (BGE-small), with NDCG@10 (Table 3 of the revised paper).
NUDGE's step size is chosen on validation, so we tuned BRAG's settings the same way and read the test split once. BRAG is ahead of the better NUDGE variant on FEVER (+0.013) and NQ (+0.015), and the two are tied on NFCorpus (+0.005). The NUDGE rows on the two public benchmarks are in Table 2 of the revised paper.
The revised paper contains a new experiment with public datasets that investigates the joint contribution of the system dynamics and the adaptation of the observation model.
Table 2 (Section 6.2) runs keys only and full BRAG side by side on LongMemEval-M and PersonaMem-v2. On queries asked again the two tie (0.858 on LongMemEval-M, 0.659 and 0.658 on PersonaMem-v2). The belief matters when a new chunk makes the information in an old one obsolete (a fact updated on LongMemEval-M, a forget request on PersonaMem-v2). There, adding the system dynamics raised Recall@5 of the new chunk from 0.711 to 0.860 and from 0.669 to 0.801, and BRAG ranks the new chunk above the old one in 0.557 and 0.567 of cases, against 0.047 and 0.006 for keys only.
On the original paper's corpus, the rerun of Table 1 (labels applied only within the top 100) shows the belief at the top of the ranking. Recall@1 rises by +0.027 [0.009, 0.048] over keys alone, while Recall@5 and average rank barely move (+0.001 and −0.05).