- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper combines a multi-index episodic store with a LoRA-based latent memory for multimodal conversational agents. An inner next-token-prediction loop internalizes a history, while an outer QA objective trains retrieval-conditioned use of the adapted memory. At inference, a first-token Jensen–Shannon-divergence proxy and latent-answer confidence select episodic, latent or hybrid inference. Experiments use Qwen2.5-VL-3B/7B on Mem-Gallery and a LoCoMo subset. The combined system substantially improves several reported scores; for example, the 7B Mem-Gallery Judge score rises from 70.61 for Episodic to 75.28 for Adaptive. The protocol reserves some original test questions for supervised alignment and evaluates the remaining questions.
Critical strengths
-
The complementary roles of explicit evidence and an internalized prior are well motivated. The method retains links back to original records rather than treating generated summaries as the only available evidence.
-
Table 1 includes standalone Episodic and Latent branches, fixed Hybrid composition, Adaptive routing, multiple external baselines and two model sizes. The large improvements from combining the branches, particularly on LoCoMo, are potentially important rather than being limited to a negligible aggregate change.
-
The task-category breakdown and constructed retrieval-failure experiments help characterize when combination is useful. The authors also show cases where a hybrid can hurt, which is a more informative analysis than assuming that additional memory is always beneficial.
Critical weaknesses
-
The contribution of memory internalization is confounded with supervised QA adaptation. Section 5.1 uses a subset of the original test set for outer-loop training and calibrates routing thresholds on that training set. Such a support/query protocol can be legitimate, but its interpretation depends on exactly what is shared across scenes, conversations, entities and question templates. The answer-overlap statistics in Table 3 and the three resampled training subsets in Figure 8 do not establish the absence of semantic or relational overlap. They also do not separate history internalization from learning the answer interface using QA labels. A matched QA-trained adapter without history internalization, together with explicit split sizes and scene-level separation, is needed to attribute the reported gains.
-
The adaptive router is less conclusively supported than the fixed hybrid. On Mem-Gallery with 3B, Adaptive improves Judge over Hybrid by only 0.14 points. With 7B on LoCoMo, Judge improves from 71.50 to 77.50, but F1 decreases from 67.49 to 65.56 and EM from 56 to 52. These differences need paired uncertainty and an explanation of the judge/lexical disagreement, rather than a general claim that routing improves answer quality. First-token distribution change measures responsiveness, not necessarily the usefulness of the latent memory; the overlapping proxy distributions in Figure 5 do not establish routing calibration. Threshold values, calibration objectives and performance against a simple fixed or confidence-only routing rule are not reported sufficiently clearly.
-
The efficiency evidence does not evaluate the complete proposed system. Table 4 reports Latent and Episodic costs but omits Hybrid and Adaptive. Equation (7) requires generating a latent answer to compute sequence confidence, while the first stage compares base and adapted first-token distributions. On a hybrid-routed query, this may precede retrieval and another generation. Branch-only latency therefore does not substantiate the end-to-end efficiency claim. The complete ingestion cost, routing/prefill cost and any reuse of the provisional answer need to be included.
Questions for authors
-
What are the exact training, calibration and evaluation splits, including scene/conversation overlap? How do results compare with an equally QA-supervised adapter that omits history internalization and with evaluation on entirely unseen scenes?
-
What thresholds and calibration objective are used by the router? Please report its paired improvement over fixed Hybrid and a simple routing control, and reconcile the Judge-versus-F1/EM trade-off on 7B LoCoMo.
-
Can the missing implementation details be supplied, along with end-to-end Hybrid and Adaptive latency that includes all probe generations, retrieval and final decoding?
5 comments
Response to vc28 Part2. for Generalization
(3) Generalization to Unseen Scenarios
To evaluate scenario-level separation, we fix the source adapter trained on AI_Robotics and transfer it to the remaining 19 scenarios and 1,046 questions. These target scenarios do not participate in source-adapter training. Target-history inner-loop updates use only the target dialogue history.
| Method | F1 | Judge |
|---|---|---|
| Episodic | 62.09 | 70.6 |
| Source adapter + Episodic | 63.28 | 71.6 |
| Source adapter + target-history Inner + Episodic | 64.96 | 71.99 |
Direct transfer of the source prior improves F1 by 1.19 pp and Judge by 1.00 pp over Episodic. Encoding the target history through the inner loop adds a further 1.68 pp F1, for total improvements of 2.87 pp F1 and 1.39 pp Judge over Episodic, without using target QA labels.
Together, these results support contributions beyond standard QA supervision: meta-learning optimizes a transferable initialization that enables the inner loop to encode raw dialogue history at test time and combine that latent context with retrieved evidence.
Response to vc28 Part3. for Adaptive Routing Effectiveness
W2 & Q2: Adaptive Routing Effectiveness
Implementation, Thresholds, and Calibration
Section 4.2, Eqs. (6)-(7), defines the two routing signals: first-token JSD () measures responsiveness to latent activation, while sequence confidence () estimates whether latent memory can answer the query sufficiently. Together, they provide a lightweight assessment of query-memory compatibility.
As detailed in Appendix E (Table 13), we use default thresholds of and . We also evaluate a calibrated variant, using one shared threshold pair within each model-benchmark setting to maximize the mean Judge score on the training QA split. This split is shared with outer-loop training and strictly excludes test questions. The following table compares the combined router with calibrated single-proxy routers.
| Method | Mem-Gallery 7B | Mem-Gallery 3B | LoCoMo 7B | LoCoMo 3B |
|---|---|---|---|---|
| Latent | 40.02 | 38.54 | 50 | 48.5 |
| Episodic | 71.61 | 67.28 | 63.5 | 42 |
| Hybrid | 73.29 | 70.56 | 71.5 | 73 |
| Divergence-only, calibrated | 74.15 | 70.59 | 74 | 73 |
| Confidence-only, calibrated | 72.951 | 70.32 | 72.5 | 73 |
| Adaptive, default thresholds | 74.972 | 71.07 | 75 | 73 |
| Adaptive, calibrated | 75.28 | 70.7 | 77.5 | 74 |
The calibrated combined router exceeds fixed Hybrid and both calibrated single-proxy routers in all four settings. Default thresholds improve over Hybrid in three settings and match it in the fourth. These answer-level comparisons complement Figure 5 by directly evaluating memory selection across threshold choices.
Proxy Validity and Stability
We agree that first-token JSD measures responsiveness and does not directly establish memory usefulness. Our router combines this signal with confidence as a joint proxy for query-memory compatibility, using first-token divergence to avoid full-sequence probing when unnecessary.
Figure 5 shows stronger discriminability at the distribution extremes: low JSD and high confidence. As the thresholds move toward these tails, the observed Judge gains increase monotonically. There is one loss under the loosest threshold and zero losses under the two tighter thresholds.
| Condition | No. of queries / Coverage | Judge gain [95% CI] | Wins / Ties / Losses |
|---|---|---|---|
| 108 / 12.24% | +6.187 [+1.416, +11.615] | 9 / 98 / 1 | |
| 90 / 10.20% | +7.422 [+2.739, +12.746] | 7 / 83 / 0 | |
| 56 / 6.35% | +10.714 [+3.571, +18.333] | 6 / 50 / 0 |
We additionally use 5,000 bootstrap resamples to estimate 95% confidence intervals for the gains over Hybrid on Mem-Gallery (7B). Both the default router (+1.68 pp; 95% CI [+0.92, +2.60]) and calibrated router (+1.99 pp; 95% CI [+1.24, +2.87]) have strictly positive intervals, supporting statistically significant improvements in this setting.
Complementary Analysis and Metric Trade-offs
On LoCoMo 7B, Adaptive improves Judge by 6.00 pp, from 71.50 to 77.50, while EM and F1 decline by 4.00 pp and 1.93 pp, respectively. The examples below illustrate two patterns. First, bypassing noisy retrieved contexts can help correct factual errors using internalized priors, improving both Judge and F1, as in the Gina case. Second, latent or hybrid pathways can produce longer responses whose wording differs from short reference answers. In the Andrew and John cases, Judge scores remain unchanged while the additional tokens substantially reduce lexical overlap, with F1 decreases of approximately 78-85 points and EM decreases of 100 points.
| Question | Hybrid response -> Adaptive response (abridged or translated) | Judge change | F1 change (approx.) |
|---|---|---|---|
| Where does Gina intern? | "Paris" -> "The fashion department of an international company..." | +75 | +66 |
| Has Andrew moved? | "No" -> "No, Andrew hasn't mentioned..." | 0 | -84 |
Model capacity may also affect the ability to extract information from latent parameters and reconcile memory conflicts. The baseline Episodic scores on LoCoMo differ substantially between 3B and 7B (42.00 vs. 63.50). The lower confidence and weaker tail separation observed for 3B are consistent with more conservative routing. On Mem-Gallery 3B, the gains over Hybrid are modest (+0.14 pp calibrated; +0.51 pp default), while high-confidence queries can still avoid retrieval.
Overall, the router shows statistically significant improvements over Hybrid on Mem-Gallery 7B and higher calibrated scores than the single-proxy baselines across the four settings. Gains over Confidence-only reach 2.33 pp on Mem-Gallery 7B and 5.00 pp on LoCoMo 7B. The tail analysis and examples identify where routing helps, while the metric trade-offs show why semantic and lexical scores should be considered together. End-to-end efficiency is detailed below.
Response to vc28 Part1. for Supervision Ablations, Split Protocols
We thank the reviewer for recognizing the "well motivated" complementary roles of explicit evidence and internalized priors, and the value of our detailed analyses. We address the concerns through matched-supervision ablations, explicit data splits, unseen-scenario evaluation, routing comparisons, and complete cost accounting.
W1 & Q1: Supervision Ablations, Split Protocols, and Generalization
We thank the reviewer for asking how memory internalization is separated from learning the QA answer interface, and for requesting clarification of the split protocols and scenario-level separation. We address each aspect below.
(1) Data Splits and Overlap Clarification
Appendix B.2 and Table 6 report the data splits, constructed using a fixed seed (42), with the same split used across methods and model scales. Section 5.5 and Figure 8(c) further examine sensitivity to training-set sampling. Both outer-loop training and router threshold calibration use only the support split, strictly excluding evaluation QA pairs.
All methods access the same complete dialogue history for memory construction, as required by the task protocol. Sharing this history does not itself imply evidence leakage. Outer-loop QA evidence is retrieved from episodic memory and remains fixed during training; gold-clue annotations are not used as training supervision (Appendix C.4).
We additionally examine evidence overlap at two levels:
The gains persist on the following subsets with no gold-clue overlap with training retrieval contexts, supporting effective evidence use beyond potential reuse of familiar QA evidence.
(2) Component Ablations vs. Matched QA-SFT
To distinguish history internalization from learning the answer interface through QA labels, we compare our framework against a QA-supervised adapter trained on the same QA split and retrieved evidence, alongside an Inner-only ablation. All Judge scores are evaluated using DeepSeek-V4.1-Flash.
Episodic + SFT improves lexical metrics (EM/F1), consistent with learning the answer interface. Unsupervised history internalization alone (Episodic + Inner-only) exceeds it on Judge score (72.57 vs. 72.27) and improves over Episodic by 1.38 percentage points (pp) without using QA labels.
Optimizing the meta-initialization, ϕinit∗, through the outer loop (Hybrid) improves the Judge score by 3.54 pp over the matched Episodic + SFT adapter, guiding history encoding toward more effective evidence utilization and answer alignment.
We further compare Hybrid with UniversalRAG and Mem0 under the same SFT setting.
QA supervision improves EM by 8.88 pp for UniversalRAG and 23.47 pp for Mem0. Nevertheless, Hybrid exceeds their Judge scores by 1.85 pp and 15.02 pp, respectively. UniversalRAG also starts from a higher pre-SFT Judge score than Episodic (71.49 vs. 71.19). These controls indicate that QA supervision alone does not explain the gains: meta-learning shapes a prior initialization, ϕinit∗, that supports subsequent unsupervised history encoding and collaboration with retrieved evidence.