- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
TrawMem argues that relevance-based memory retrieval ignores the functional role a memory plays for a given query: answering it directly versus bridging to weakly relevant but necessary evidence. The method segments dialogue history into LLM-summarized local threads linked in a thread graph. At query time it propagates scores over a bounded neighborhood and assigns each candidate turn a soft distribution over three roles (anchor/bridge/context) through fixed feature mappings. It then selects at most B turns by greedily maximizing a four-term monotone submodular objective, which carries a 1−1/e guarantee. On LoCoMo and Mem-Gallery with four answer backbones, it reports the best macro F1/BLEU-1/J against five memory systems (LoCoMo J +2.19–6.34 over the strongest baseline). It also includes ablations, sensitivity, efficiency, and mechanism diagnostics.
Critical strengths
Broad, consistent gains with reasonable robustness checks. TrawMem has the best average in all eight backbone × benchmark settings (Tables 1–2). The advantage holds under cross-model judging (Table 4: +2.26 / +5.31 J over SimpleMem) and against two more recent systems (Table 3). It also uses fewer answer tokens than SimpleMem (~2,050 vs. ~2,351 tok/q; Table 5). As a practical retrieval pipeline, it is credible. Diagnostics beyond the leaderboard. Stage-wise gold recall (Fig. 13) and the coverage–redundancy frontier against Top-B relevance (Fig. 15) give a real view of what the selection step does.
Critical weaknesses
The titular role mechanism is heuristic and its contribution is not established. Roles come from hand-designed signals (temporal/relational cues, a query-length flag) mapped through fixed, unreported matrices (App. A.3); nothing is learned. Removing role conditioning costs only 1.31–1.82 J on LoCoMo and 1.21–1.73 on Mem-Gallery (Figs. 4–5), from single runs with no variance reported. By a rough binomial estimate, the standard error of macro-J on LoCoMo is about 1.5 pp, given OpenDomain n=96. Role distributions are close to uniform (entropy 0.75–0.90 nats vs. ln 3≈1.10; Fig. 10b). Most importantly, the motivating case does not hold up. On MultiHop, final-workspace gold recall (61.3%) is below that of the initially routed threads alone (67.1%), and the same holds for Temporal (79.6% vs. 82.1%; Fig. 13). H.1.2 shows that ρ shifts across paired queries, but not that the shift changes which evidence is retrieved.
The protocol leaves the gains unattributed. (i) The answer prompt is method-specific, with CoT and per-category rules (App. J.1, R2), and the paper does not say whether baselines used it. Table 3 reports the same method on the same backbone (Qwen3.8-27B) at F1 59.74 / 54.51 under different runners, versus 51.23 in Table 1. That pipeline effect is larger than any reported method gain. (ii) Hyperparameters appear to be selected on the test set. The Fig. 6 peaks (63.26 / 62.95 / 68.56 / 74.52) exactly match Table 1, with backbone-specific (λm, λr), and B=16 is chosen on the same LoCoMo data. No held-out split is used. (iii) There is no token-matched control, such as raw-turn RAG at ~2k tokens, even though the strongest baseline, LightMem, uses about half the context (Table 5). The full-context baseline appears only in App. H.3.1 (63.70% with GPT-5.6-Luna), which is unusually low, and it is not discussed.
Questions for authors
Were all baselines run with the same answer prompt (R2) and the same judge? Please report the baselines re-run under R2, and explain the 51.23 vs. 59.74 F1 gap for the same backbone between Tables 1 and 3.
How were λm, λr, B, γ, K, H, M_max, A, and M chosen? Please report all values, along with results from a single configuration shared across backbones and fixed on a held-out split.
Please add matched-budget controls: (a) raw-turn Naive RAG at ~2,050 tokens; (b) thread retrieval + top-B by π, with no roles and no submodular selection. Please also give seed variance or bootstrap CIs for Table 1 and the ablations.
Does the role term causally recover low-α bridge evidence? For example, report gold-evidence recall stratified by α, with and without role terms, and explain the drop in MultiHop recall after selection.
12 comments
Response to Reviewer mXTV (Part 2/6)
Weakness 1.b & Question 3 (second half: variance across multiple runs): Removing role conditioning reduces by only 1.21–1.82 points (close to a rough binomial standard error), and multiple-run seed variance is missing for the main results in Table 1 and all ablation variants
Thank you for your comment and question. We report the per-category results for the role ablation (Table R2) and seed variance for the original main results in Table 1 and all ablation variants in Figure 4 (Table R3), covering both paired sampling error at the question and conversation levels and the variability across independent end-to-end runs (with different random seeds for Stage 1 thread induction, answer generation, and the evaluator; note that we report the mean standard deviation over independent seeds here, so normal seed-level differences of from the single default runs in the original main table and Figure 4, such as , are expected):
Table R2: Per-category comparison of the full model and No Role Cond. over independent end-to-end runs (mean standard deviation)
| Dataset and evaluation scope | Full model () | No Role Cond. () | Difference |
|---|---|---|---|
| LoCoMo - MultiHop (, Qwen3.5-9B) | 54.68 0.39 | 51.84 0.30 | |
| LoCoMo - Temporal (, Qwen3.5-9B) | 65.36 0.34 | 63.18 0.41 | |
| LoCoMo - OpenDomain (, Qwen3.5-9B) | 58.54 0.87 | 58.33 0.74 | |
| LoCoMo - SingleHop (, Qwen3.5-9B) | 74.55 0.19 | 74.34 0.21 | |
| LoCoMo - Mean excluding OpenDomain (Qwen3.5-9B) | 64.86 0.18 | 63.12 0.19 | |
| LoCoMo - Macro- over all four categories (Qwen3.5-9B) | 63.28 0.21 | 61.92 0.19 | |
| LoCoMo - Macro- over all four categories (GPT-5.6-Luna) | 74.47 0.23 | 72.66 0.25 | |
| MemGallery - Overall Macro- (Qwen3.5-9B) | 49.09 0.24 | 47.89 0.21 |
Table R3: Comparison of the original main results in Table 1 and all ablations in Figure 4 over independent runs (LoCoMo, mean standard deviation)
| Comparison / ablation variant | Control or variant | Full TrawMem model | Difference |
|---|---|---|---|
| Original Table 1: Qwen3.5-9B (vs. LightMem) | 60.01 0.24 | 63.28 0.21 | |
| Original Table 1: Qwen3.8-27B (vs. SimpleMem) | 60.79 0.17 | 62.91 0.22 | |
| Original Table 1: DeepSeek-V4-Flash (vs. LightMem) | 65.49 0.25 | 68.59 0.19 | |
| Original Table 1: GPT-5.6-Luna (vs. SimpleMem) | 68.22 0.19 | 74.47 0.23 | |
| Ablation (Qwen3.5-9B): Fixed Seg. | 61.91 0.24 | 63.28 0.21 | |
| Ablation (Qwen3.5-9B): No Cross-Edge | 61.71 0.23 | 63.28 0.21 | |
| Ablation (Qwen3.5-9B): Seed-only | 62.48 0.16 | 63.28 0.21 | |
| Ablation (Qwen3.5-9B): No Role Cond. | 61.92 0.19 | 63.28 0.21 | |
| Ablation (Qwen3.5-9B): Identity K | 62.19 0.22 | 63.28 0.21 | |
| Ablation (Qwen3.5-9B): No Multi-thread | 61.37 0.26 | 63.28 0.21 |
Response to Reviewer mXTV (Part 2/6)
1) Across question types, variation across random seeds is far below the points from an independent-binomial estimate (Tables R2–R3): The points correspond to two independent samples. For paired comparisons on the same questions, the four backbones had, respectively, , , , and questions that changed from incorrect to correct / correct to incorrect relative to their respective strongest baselines (net gains of questions, with a per-question paired standard error of only percentage points). Relative to No Role Cond., about %–% of questions had the same outcome (incorrect-to-correct / correct-to-incorrect counts ranged from to questions, with a paired per-question Macro- standard error of only points). Moreover, when aggregating each of the long conversations as a whole, the full model performs better on conversations (two-sided sign test ). Meanwhile, the standard deviation of across the end-to-end runs is only points for Table 1 and all ablation variants, and the ranking of ablation importance exactly matches Figure 4.
2) Roles are especially indispensable for multi-hop, temporal, and cross-task cases (Table R2): The event-thread graph retrieves the candidate pool across sessions (hence the largest drop for No Multi-thread; we will clarify the division of responsibilities in Section 4.2), while the role module retains weakly related but complementary evidence. SingleHop and OpenDomain () can be answered by direct matching (only points; after excluding OpenDomain, the mean for the other three categories instead rises from to ), whereas roles improve by and points on MultiHop and Temporal, respectively. On the question categories in MemGallery (Qwen3.5-9B), No Role Cond. () falls below the strongest baseline, Mem0 (); adding role modulation raises it to (outperforming in all runs, ).
The revision will briefly summarize these multi-run stability and category-level findings in Sections 4.1–4.2, and add the complete standard deviations, per-category role ablations, and paired statistics (Tables R2–R3) to Appendices E and H.1.
Response to Reviewer mXTV (Part 1/6)
Weakness 1.a & Question 2 (first half: matrices A,M): The roles are obtained by mapping manually designed signals through fixed, unreported matrices, with no learning process. What are the specific values of matrices A and M, and how were they determined?
Thank you for your comment and question. The complete values of A and M are given below; both are specified once according to the role definitions in Appendix A.3 and shared across all four backbones and both datasets, with no trainable parameters. Table R1 shows that the gains come from “which feature is assigned to which role,” rather than from the specific numerical values.
Values. The rows correspond to (anchor, bridge, context), respectively; the columns of A correspond to the query features xq=(1,ztemp,zrel,zlen)⊤ in Eq. (17), and the columns of M(δq) correspond to the role features fq(v)=(fqa,fqb,fqc,t(v),1−fqa,1)⊤ in Eq. (20). The temperature in Eq. (5) is TR=0.50 (see Table R6 in Part 5/6 for the other hyperparameters):
M(\delta_q)=\begin{bmatrix}2.20&0&0&0&0&0.35\delta_{q,a}\\\\0&1.45+0.85\delta_{q,b}&0&0&0.15&0\\\\0&0&1.30&0.85\delta_{q,c}&0.15&0\end{bmatrix}.$$ **How they were determined.** The locations of the nonzero entries follow directly from the role definitions in Appendix A.3: in $A$, the relational cue $z_{\mathrm{rel}}$ primarily increases bridge demand, while the temporal cue $z_{\mathrm{temp}}$ primarily increases context demand; in $M$, each role reads only its own features (anchor reads $f^a_q$, bridge reads $f^b_q$, and context reads $f^c_q$ and $t(v)$), and increases with the corresponding role demand $\delta_q$. The numerical values of the nonzero entries were set manually; Table R1 tests whether the results are sensitive to these values: **Table R1: Comparison of different $A$ and $M$ settings (Qwen3.5-9B)** | Matrix setting | Specific configuration | LoCoMo $J$ | MultiHop recall (%) | MemGallery $J$ | | --- | --- | --- | --- | --- | | No Role Cond. | Disable role terms | 61.95 | 55.0 | 47.85 | | Uniform $0/1$ matrices | Keep the locations of nonzero entries and set all their values to $1$ | 62.89 | 58.5 | 48.82 | | Coefficient perturbation $\pm 25$% (10 runs) | Independently multiply each coefficient by $\mathcal{U}(0.75, 1.25)$ | 63.18 $\pm$ 0.14 | 59.1 $\pm$ 0.2 | 49.02 $\pm$ 0.15 | | Random row/column permutation (10 runs) | Shuffle the feature-to-role correspondences | 60.84 $\pm$ 0.31 | 52.5 $\pm$ 0.5 | 46.62 $\pm$ 0.35 | | Original matrices (default) | As above | 63.26 | 59.2 | 49.06 | **1) The gains come from the locations of the nonzero entries, not their values (Table R1)**: When all nonzero entries are set to $1$ (with no tunable coefficients), $J$ is $62.89$ / $48.82$, already capturing $+0.94$ / $+0.97$ of the original matrices’ gains over `No Role Cond.` ($+1.31$ / $+1.21$). After randomly perturbing all coefficients by $\pm 25$% over $10$ runs, $J$ is $63.18 \pm 0.14$ (range $62.94\text{–}63.35$, with only small fluctuations around the default value). Conversely, randomly permuting rows and columns to disrupt the feature-to-role correspondence yields a result below `No Role Cond.` ($60.84$ vs. 61.95), showing that this correspondence is what matters. **2) Roles produced by the fixed mapping agree with human annotations and approach the results of per-item LLM judgments**: On $450$ samples independently annotated by two annotators ($\kappa=0.81$), the fixed mapping achieves a macro-averaged role F1 of $76.4$%, $23.9$ percentage points higher than the baseline that assigns roles by binning only $\alpha_q(v)$ ($52.5$%), and close to the $79.8$% achieved by zero-shot, per-item judgments from GPT-5.6-Luna. The latter requires one LLM call for each of approximately $145$ candidates per question, whereas the fixed mapping makes no LLM calls at query time (the F1 scores for anchor / bridge / context are $83.1$% / $71.1$% / $74.9$%, respectively). **3) The absence of training is consistent with all comparison baselines**: LoCoMo and MemGallery provide neither training sets nor role annotations; baselines such as `LightMem`, `SimpleMem`, and `Mem0` likewise have no trainable parameters during retrieval and selection. The revision will add the values of $A$ and $M$ and Table R1 to Appendices A.3 and H.1.