acceptodds
Under review as a conference paper at ICLR 2027

Intuition with Evidence: Adaptive Dual Memory for Multimodal Agents

Abstract

As multimodal agents engage in sustained interactions, long-term memory serves two distinct but complementary functions: fine-grained evidence recall and internalized interpretation for global understanding. Although episodic and latent memory address these demands through explicit historical records and internalized experience, neither paradigm alone can accommodate the diverse memory affinities across tasks, owing to limitations in episodic evidence coverage and integration, and in latent factual grounding. To bridge this divide, we propose a dual-memory framework, unifying a model-native latent memory as an intuitive prior with a multi-indexed episodic memory for exact evidence reconstruction via retrieval-conditioned joint alignment. At inference, the framework adaptively composes these memories through a training-free routing mechanism driven by Jensen-Shannon Divergence (JSD) sensitivity and confidence proxies for query-dependent memory affinity. Extensive evaluations across multimodal benchmarks demonstrate state-of-the-art performance, outperforming single-paradigm baselines overall (+4.67% over vanilla Episodic), with further analysis highlighting episodic strengths in evidence-localized tasks and latent contributions to integrating dispersed information and compensating for imperfect retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
memory, agent harness, self-evolving, latent memory, multimodal

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vc28OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines a multi-index episodic store with a LoRA-based latent memory for multimodal conversational agents. An inner next-token-prediction loop internalizes a history, while an outer QA objective trains retrieval-conditioned use of the adapted memory. At inference, a first-token Jensen–Shannon-divergence proxy and latent-answer confidence select episodic, latent or hybrid inference. Experiments use Qwen2.5-VL-3B/7B on Mem-Gallery and a LoCoMo subset. The combined system substantially improves several reported scores; for example, the 7B Mem-Gallery Judge score rises from 70.61 for Episodic to 75.28 for Adaptive. The protocol reserves some original test questions for supervised alignment and evaluates the remaining questions.

Critical strengths
  1. The complementary roles of explicit evidence and an internalized prior are well motivated. The method retains links back to original records rather than treating generated summaries as the only available evidence.

  2. Table 1 includes standalone Episodic and Latent branches, fixed Hybrid composition, Adaptive routing, multiple external baselines and two model sizes. The large improvements from combining the branches, particularly on LoCoMo, are potentially important rather than being limited to a negligible aggregate change.

  3. The task-category breakdown and constructed retrieval-failure experiments help characterize when combination is useful. The authors also show cases where a hybrid can hurt, which is a more informative analysis than assuming that additional memory is always beneficial.

Critical weaknesses
  1. The contribution of memory internalization is confounded with supervised QA adaptation. Section 5.1 uses a subset of the original test set for outer-loop training and calibrates routing thresholds on that training set. Such a support/query protocol can be legitimate, but its interpretation depends on exactly what is shared across scenes, conversations, entities and question templates. The answer-overlap statistics in Table 3 and the three resampled training subsets in Figure 8 do not establish the absence of semantic or relational overlap. They also do not separate history internalization from learning the answer interface using QA labels. A matched QA-trained adapter without history internalization, together with explicit split sizes and scene-level separation, is needed to attribute the reported gains.

  2. The adaptive router is less conclusively supported than the fixed hybrid. On Mem-Gallery with 3B, Adaptive improves Judge over Hybrid by only 0.14 points. With 7B on LoCoMo, Judge improves from 71.50 to 77.50, but F1 decreases from 67.49 to 65.56 and EM from 56 to 52. These differences need paired uncertainty and an explanation of the judge/lexical disagreement, rather than a general claim that routing improves answer quality. First-token distribution change measures responsiveness, not necessarily the usefulness of the latent memory; the overlapping proxy distributions in Figure 5 do not establish routing calibration. Threshold values, calibration objectives and performance against a simple fixed or confidence-only routing rule are not reported sufficiently clearly.

  3. The efficiency evidence does not evaluate the complete proposed system. Table 4 reports Latent and Episodic costs but omits Hybrid and Adaptive. Equation (7) requires generating a latent answer to compute sequence confidence, while the first stage compares base and adapted first-token distributions. On a hybrid-routed query, this may precede retrieval and another generation. Branch-only latency therefore does not substantiate the end-to-end efficiency claim. The complete ingestion cost, routing/prefill cost and any reuse of the provisional answer need to be included.

Questions for authors
  1. What are the exact training, calibration and evaluation splits, including scene/conversation overlap? How do results compare with an equally QA-supervised adapter that omits history internalization and with evaluation on entirely unseen scenes?

  2. What thresholds and calibration objective are used by the router? Please report its paired improvement over fixed Hybrid and a simple routing control, and reconcile the Judge-versus-F1/EM trade-off on 7B LoCoMo.

  3. Can the missing implementation details be supplied, along with end-to-end Hybrid and Adaptive latency that includes all probe generations, retrieval and final decoding?

5 comments
AuthorsOpenReview ↗

Response to vc28 Part1. for Supervision Ablations, Split Protocols

We thank the reviewer for recognizing the "well motivated" complementary roles of explicit evidence and internalized priors, and the value of our detailed analyses. We address the concerns through matched-supervision ablations, explicit data splits, unseen-scenario evaluation, routing comparisons, and complete cost accounting.

W1 & Q1: Supervision Ablations, Split Protocols, and Generalization

We thank the reviewer for asking how memory internalization is separated from learning the QA answer interface, and for requesting clarification of the split protocols and scenario-level separation. We address each aspect below.

(1) Data Splits and Overlap Clarification

Appendix B.2 and Table 6 report the data splits, constructed using a fixed seed (42), with the same split used across methods and model scales. Section 5.5 and Figure 8(c) further examine sensitivity to training-set sampling. Both outer-loop training and router threshold calibration use only the support split, strictly excluding evaluation QA pairs.

All methods access the same complete dialogue history for memory construction, as required by the task protocol. Sharing this history does not itself imply evidence leakage. Outer-loop QA evidence is retrieved from episodic memory and remains fixed during training; gold-clue annotations are not used as training supervision (Appendix C.4).

We additionally examine evidence overlap at two levels:

  • Clue-annotation level: Among the 966 Mem-Gallery test questions with gold-clue annotations, 704 (72.88%) share no gold clues with training questions of the same task category.
  • Retrieved-context level: For 116 Mem-Gallery test questions (12.01%) and 22 LoCoMo test questions (44.0%), none of their gold-clue turns appears in the retrieved evidence of any training QA pair, across all task types.

The gains persist on the following subsets with no gold-clue overlap with training retrieval contexts, supporting effective evidence use beyond potential reuse of familiar QA evidence.

BenchmarkTest subsetMethodF1 (%)Judge (%)
Mem-Gallery116/966 (12.01%)Episodic58.7268.1
Mem-Gallery116/966 (12.01%)Hybrid63.68 (+4.96 pp)70.47 (+2.37 pp)
LoCoMo22/50 (44.00%)Episodic62.3970.45
LoCoMo22/50 (44.00%)Hybrid82.14 (+19.75 pp)79.55 (+9.09 pp)

(2) Component Ablations vs. Matched QA-SFT

To distinguish history internalization from learning the answer interface through QA labels, we compare our framework against a QA-supervised adapter trained on the same QA split and retrieved evidence, alongside an Inner-only ablation. All Judge scores are evaluated using DeepSeek-V4.1-Flash.

MethodF1EMJudge
Episodic61.8942.3371.19
Episodic + SFT70.853.9772.27
Episodic + Inner-only70.7753.7972.57
Hybrid: Episodic + Inner + Outer71.2754.8175.81

Episodic + SFT improves lexical metrics (EM/F1), consistent with learning the answer interface. Unsupervised history internalization alone (Episodic + Inner-only) exceeds it on Judge score (72.57 vs. 72.27) and improves over Episodic by 1.38 percentage points (pp) without using QA labels.

Optimizing the meta-initialization, ϕinit∗\phi_{\mathrm{init}}^{*}, through the outer loop (Hybrid) improves the Judge score by 3.54 pp over the matched Episodic + SFT adapter, guiding history encoding toward more effective evidence utilization and answer alignment.

We further compare Hybrid with UniversalRAG and Mem0 under the same SFT setting.

MethodF1EMJudge
UniversalRAG + SFT70.9251.0273.96
Mem0 + SFT57.5741.7760.79
Hybrid71.2754.8175.81

QA supervision improves EM by 8.88 pp for UniversalRAG and 23.47 pp for Mem0. Nevertheless, Hybrid exceeds their Judge scores by 1.85 pp and 15.02 pp, respectively. UniversalRAG also starts from a higher pre-SFT Judge score than Episodic (71.49 vs. 71.19). These controls indicate that QA supervision alone does not explain the gains: meta-learning shapes a prior initialization, ϕinit∗\phi_{\mathrm{init}}^{*}, that supports subsequent unsupervised history encoding and collaboration with retrieved evidence.

AuthorsOpenReview ↗

Response to vc28 Part2. for Generalization

(3) Generalization to Unseen Scenarios

To evaluate scenario-level separation, we fix the source adapter trained on AI_Robotics and transfer it to the remaining 19 scenarios and 1,046 questions. These target scenarios do not participate in source-adapter training. Target-history inner-loop updates use only the target dialogue history.

MethodF1Judge
Episodic62.0970.6
Source adapter + Episodic63.2871.6
Source adapter + target-history Inner + Episodic64.9671.99

Direct transfer of the source prior improves F1 by 1.19 pp and Judge by 1.00 pp over Episodic. Encoding the target history through the inner loop adds a further 1.68 pp F1, for total improvements of 2.87 pp F1 and 1.39 pp Judge over Episodic, without using target QA labels.

Together, these results support contributions beyond standard QA supervision: meta-learning optimizes a transferable initialization that enables the inner loop to encode raw dialogue history at test time and combine that latent context with retrieved evidence.

AuthorsOpenReview ↗

Response to vc28 Part3. for Adaptive Routing Effectiveness

W2 & Q2: Adaptive Routing Effectiveness

Implementation, Thresholds, and Calibration

Section 4.2, Eqs. (6)-(7), defines the two routing signals: first-token JSD (dd) measures responsiveness to latent activation, while sequence confidence (cc) estimates whether latent memory can answer the query sufficiently. Together, they provide a lightweight assessment of query-memory compatibility.

As detailed in Appendix E (Table 13), we use default thresholds of τd=0.10\tau_d=0.10 and τc=0.95\tau_c=0.95. We also evaluate a calibrated variant, using one shared threshold pair within each model-benchmark setting to maximize the mean Judge score on the training QA split. This split is shared with outer-loop training and strictly excludes test questions. The following table compares the combined router with calibrated single-proxy routers.

MethodMem-Gallery 7BMem-Gallery 3BLoCoMo 7BLoCoMo 3B
Latent40.0238.545048.5
Episodic71.6167.2863.542
Hybrid73.2970.5671.573
Divergence-only, calibrated74.1570.597473
Confidence-only, calibrated72.95170.3272.573
Adaptive, default thresholds74.97271.077573
Adaptive, calibrated75.2870.777.574

The calibrated combined router exceeds fixed Hybrid and both calibrated single-proxy routers in all four settings. Default thresholds improve over Hybrid in three settings and match it in the fourth. These answer-level comparisons complement Figure 5 by directly evaluating memory selection across threshold choices.

Proxy Validity and Stability

We agree that first-token JSD measures responsiveness and does not directly establish memory usefulness. Our router combines this signal with confidence as a joint proxy for query-memory compatibility, using first-token divergence to avoid full-sequence probing when unnecessary.

Figure 5 shows stronger discriminability at the distribution extremes: low JSD and high confidence. As the thresholds move toward these tails, the observed Judge gains increase monotonically. There is one loss under the loosest threshold and zero losses under the two tighter thresholds.

ConditionNo. of queries / CoverageJudge gain [95% CI]Wins / Ties / Losses
d≤0.1,c>0.95d\le 0.1,\\ c>0.95108 / 12.24%+6.187 [+1.416, +11.615]9 / 98 / 1
d≤0.05,c>0.975d\le 0.05,\\ c>0.97590 / 10.20%+7.422 [+2.739, +12.746]7 / 83 / 0
d≤0.01,c>0.99d\le 0.01,\\ c>0.9956 / 6.35%+10.714 [+3.571, +18.333]6 / 50 / 0

We additionally use 5,000 bootstrap resamples to estimate 95% confidence intervals for the gains over Hybrid on Mem-Gallery (7B). Both the default router (+1.68 pp; 95% CI [+0.92, +2.60]) and calibrated router (+1.99 pp; 95% CI [+1.24, +2.87]) have strictly positive intervals, supporting statistically significant improvements in this setting.

Complementary Analysis and Metric Trade-offs

On LoCoMo 7B, Adaptive improves Judge by 6.00 pp, from 71.50 to 77.50, while EM and F1 decline by 4.00 pp and 1.93 pp, respectively. The examples below illustrate two patterns. First, bypassing noisy retrieved contexts can help correct factual errors using internalized priors, improving both Judge and F1, as in the Gina case. Second, latent or hybrid pathways can produce longer responses whose wording differs from short reference answers. In the Andrew and John cases, Judge scores remain unchanged while the additional tokens substantially reduce lexical overlap, with F1 decreases of approximately 78-85 points and EM decreases of 100 points.

QuestionHybrid response -> Adaptive response (abridged or translated)Judge changeF1 change (approx.)
Where does Gina intern?"Paris" -> "The fashion department of an international company..."+75+66
Has Andrew moved?"No" -> "No, Andrew hasn't mentioned..."0-84

Model capacity may also affect the ability to extract information from latent parameters and reconcile memory conflicts. The baseline Episodic scores on LoCoMo differ substantially between 3B and 7B (42.00 vs. 63.50). The lower confidence and weaker tail separation observed for 3B are consistent with more conservative routing. On Mem-Gallery 3B, the gains over Hybrid are modest (+0.14 pp calibrated; +0.51 pp default), while high-confidence queries can still avoid retrieval.

Overall, the router shows statistically significant improvements over Hybrid on Mem-Gallery 7B and higher calibrated scores than the single-proxy baselines across the four settings. Gains over Confidence-only reach 2.33 pp on Mem-Gallery 7B and 5.00 pp on LoCoMo 7B. The tail analysis and examples identify where routing helps, while the metric trade-offs show why semantic and lexical scores should be considered together. End-to-end efficiency is detailed below.

Reviewer_JkrROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a dual-memory framework, unifying a model-native latent memory as an intuitive prior with a multi-indexed episodic memory for exact evidence reconstruction via retrieval-conditioned joint alignment. Evaluations across multimodal benchmarks demonstrate state-of-the-art performance, outperforming single-paradigm baselines overall (+4.67% over vanilla Episodic), with further analysis highlighting episodic strengths in evidence-localized tasks and latent contributions to integrating dispersed information and compensating for imperfect retrieval.

Critical strengths

S1: The authors have done comprehensive baseline comparison, 10+ baselines across two benchmarks (Mem-Gallery, LoCoMo) at 3B/7B scales. Also, the cost analysis is included. S2: Good writing, the figures and tables are well presented and clearly illustrate the framework's workflow.

Critical weaknesses

W1:The combination of the two memory types appears to be more of an incremental contribution, and the algorithm leans towards being heuristic. The added contribution is essentially the JSD-based router, which is modest. I would expect to see more theoretical innovations. W2: The practical scalability is poor. Ingest latency of ~3244s/scene for latent memory and ~1958s for episodic (Table 4) is prohibitive for any realistic deployment. The paper dismisses this but doesn't seriously address it. W3: Only Mem-Gallery and LoCoMo are evaluated, both from similar paradigms. More generalization to other long-horizon agent benchmarks (e.g., LongMemEval, HotpotQA-style reasoning) would help to broad the diversity.

Questions for authors

Q1:What are the threshold values τ_d and τ_c, and how sensitive are they across the two benchmarks and model sizes? Q2: See in weakness.

4 comments
AuthorsOpenReview ↗

Response to JkrR Part1. for Novelty and Cost

We thank the reviewer for recognizing our "comprehensive baseline comparison" and clear presentation. We address the concerns through additional ablations, cost accounting, and evaluations on two additional benchmarks.

W1 & Q2: Novelty

Our primary contribution is the first co-active framework that synergizes latent memory (for intuition emergence) and episodic memory (for fine-grained evidence). In contrast to the unidirectional evolution or single-memory usage in MemVerse and UniMem, our framework is grounded in a systematic analysis of the properties and task preferences of each memory mechanism.

Guided by this framework, we develop tailored implementations featuring multi-dimensional episodic retrieval and latent memory capable of rapid test-time adaptation across diverse contexts. We also introduce a meta-learning recipe for joint training that couples history encoding with evidence utilization: after the adapter encodes a history, the outer objective evaluates how well the adapted model answers questions about that history given the evidence retrieved for each query. Let H\mathcal{H} denote the historical trajectory, eq=Retrieve⁡(q,H)e_q=\operatorname{Retrieve}(q,\mathcal{H}) denote the retrieved evidence, and UHK(ϕ)U_{\mathcal{H}}^{K}(\phi) represent the adapter state obtained from ϕ\phi after KK steps of historical language modeling updates. Our unified objective is:

= \frac{1}{|\mathcal D_{\mathrm{align}}|} \sum_{(q,a)\in \mathcal D_{\mathrm{align}}} \ell_{\mathrm{QA}}(p_{\theta,U_{\mathcal H}^{K}(\phi)}(\cdot \mid q,e_q),a).$$ Crucially, this objective guides history encoding, $U_{\mathcal{H}}^{K}(\phi)$, toward more effective evidence utilization and answer alignment. We further design a lightweight router that couples latent memory states with query properties, alongside extensive empirical analyses of memory preferences, synergy-driven robustness, and hyperparameter sensitivity across benchmarks. Together, these contributions provide a foundation for the co-evolution and role differentiation of heterogeneous memory systems. **W2 & Q2: Deployment Costs** We thank the reviewer for pointing out this practical scalability issue. We apologize for omitting the explicit combined costs of Hybrid and Adaptive from Table 4. We first clarify a reporting ambiguity: the reported **3,244.72 s** Latent ingestion time included offline meta-training for five epochs, with four inner-loop steps, one outer-loop step, and meta-gradient updates across training scenes. Actual test-time ingestion requires only four inner-loop steps and takes **309.67 s/scene**. With parallel construction of the two memory branches, ingestion latency for Hybrid and Adaptive is governed by the slower Episodic branch: $$\max(309.67,1958.61)=1958.61\\ \mathrm{s/scene}.$$ Memory construction is performed once per scenario, and the memories are reused across subsequent queries, amortizing the construction cost. During QA, Hybrid's latency (**10.11 s/query**) is dominated by episodic retrieval, which could be further optimized using lighter retrievers. | Method | Ingestion latency (s/scene) | QA latency (s/query) | Input tokens/query | | --- | ---: | ---: | ---: | | Hybrid | 1,958.61 | 10.11 | 2,503.00 | | Adaptive | 1,958.61 | 10.87 | 2,322.27 | Adaptive offers a practical balance between performance and efficiency. It maintains comparable QA latency (**10.87 s/query; +0.76 s/query**) while reducing input tokens by **7.22%**. The probing mechanism reuses existing computations to limit additional overhead. The measured increase of 0.76 s/query therefore accompanies a reduction in input tokens and the flexibility to select different memory branches.
AuthorsOpenReview ↗

Response to JkrR Part2. for Novelty and Cost

W3 & Q2: Benchmark Diversity

We extend the evaluation to LongMemEval-S and HotpotQA, covering long-term conversational memory and multi-hop question answering. We follow their official evaluation protocols, reporting answer EM and F1 for HotpotQA and LLM-judge accuracy for LongMemEval-S, with DeepSeek-V4.1-Flash as the judge.

MethodHotpotQA EMHotpotQA F1LongMemEval-S Judge
Episodic6675.8753
Hybrid6578.3957
Mem05867.5740
FuMemory5059.7310

Across both benchmarks, Hybrid outperforms Mem0 and FuMemory on every reported metric and achieves the best HotpotQA F1 (78.39) and LongMemEval-S accuracy (57). On HotpotQA, Episodic retrieval slightly leads on strict EM (66 vs. 65), while Hybrid improves token-level F1 by 2.52 points. Exact match is sensitive to minor phrasing differences for short extractive answers, whereas F1 captures token overlap with the reference. These results extend the evidence for the framework's effectiveness to additional scenarios and task types.

Q1: Threshold Calibration

Appendix E and Table 13 already report the threshold values and calibration objective. One threshold pair, (τd,τc)(\tau_d,\tau_c), is shared across tasks within each model-benchmark setting and selected to maximize the mean Judge score on training QA pairs. Test QA pairs are excluded from calibration. The calibration objective is:

\in \underset{(\tau_d,\tau_c)\in\mathcal{T}_d\times\mathcal{T}_c}{\operatorname{argmax}} \frac{1}{N}\sum_{i=1}^{N} s_i\left(\pi_{\tau_d,\tau_c}(d_i,c_i)\right).$$ We additionally compare the shared default thresholds, $(\tau_d,\tau_c)=(0.1,0.95)$, with the calibrated thresholds: | Setting | Fixed Hybrid | Adaptive, default | Adaptive, calibrated | | --- | ---: | ---: | ---: | | Mem-Gallery 7B | 73.29 | 74.97 | **75.28** | | Mem-Gallery 3B | 70.56 | **71.07** | 70.7 | | LoCoMo 7B | 71.5 | 75 | **77.5** | | LoCoMo 3B | 73 | 73 | **74** | Adaptive remains competitive with fixed Hybrid under the shared default thresholds. Calibration further improves performance in three of four settings, with the largest gain on LoCoMo 7B. This comparison shows that routing benefits extend across threshold choices, while supporting calibration for different models and benchmarks. We hope this response clarifies that our contribution extends beyond the router to an effective dual-memory framework whose benefits generalize across scenarios and task types. The additional cost analysis also clarifies the distinction between offline training and test-time memory construction, along with the modest QA overhead of adaptive routing. We respectfully invite you to reconsider your score based on these findings.
AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer JkrR

Thank you again for your constructive review of our work. With the discussion phase moving forward, we wanted to briefly follow up and see if our responses and additional analyses have resolved your questions. We remain fully available and happy to provide any further clarifications if needed!

Thanks for your response and I intend to maintain my score.

Reviewer_1SVGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dual-memory framework for multimodal agents, combining a structured episodic store with latent memory encoded in a LoRA adapter. Episodic retrieval provides specific evidence, while history adaptation and retrieval-conditioned alignment train the adapter to support broader understanding. At inference, a router uses first-token distribution divergence and sequence confidence to select episodic, latent, or hybrid inference. Evaluations on Mem-Gallery and LoCoMo show strong hybrid performance and complementary behavior across task categories, with additional analyses of retrieval failures and adaptation settings.

Critical strengths
  1. A coherent design with strong hybrid performance. The framework combines complementary roles for explicit evidence and adapted parameters. Hybrid inference consistently outperforms standalone branches across benchmarks and model scales, demonstrating the value of integrating both memory mechanisms.

  2. Useful analysis of memory complementarity. Task-category comparisons reveal distinct strengths of the two memory branches. Experiments with missing or misleading retrieval further show how hybrid memory mitigates episodic retrieval limitations, highlighting the benefits of their complementarity.

Critical weaknesses
  1. Memory gains are not clearly separated from supervised adaptation. It is unclear whether baselines receive equivalent QA supervision. The answer-overlap analysis does not rule out shared underlying evidence. Without matched supervision and clearer data splits, some gains may stem from task adaptation rather than memory organization.

  2. The adaptive router's benefits and efficiency remain unclear. Gains over fixed hybrid inference are small or inconsistent, and proxy distributions do not establish reliable routing. Sequence confidence also requires additional generation, while the cost analysis excludes Hybrid and Adaptive, leaving their overall efficiency unverified.

Questions for authors
  1. What are the exact support, calibration, and evaluation splits? Do baselines receive equivalent QA supervision, and is supporting evidence shared across splits?

  2. How does the method compare with standard retrieval-conditioned adapter fine-tuning under matched supervision? Can ablations isolate history encoding and meta-learning?

  3. How do divergence-only, confidence-only, and simple routing baselines perform? Please report branch-selection frequencies and variability across splits.

  4. What are the full ingestion and inference costs of Hybrid and Adaptive, including confidence generation? Please clarify the missing training and calibration details.

7 comments
AuthorsOpenReview ↗

Response to 1SVG Part1.

We sincerely thank the reviewer for highlighting important perspectives for strengthening our evaluation and for recognizing our "coherent design with strong hybrid performance" and "useful analysis of memory complementarity." To address these concerns, we expand our evaluation with matched-supervision controls, component ablations, evidence-coverage analysis, routing comparisons, and cost accounting. We address each point below.

W1, Q1 & Q2: Beyond QA Supervision and Component Ablations

We thank the reviewer for asking whether the performance gains arise from QA supervision or memory internalization and for suggesting additional verification. We address this concern through the following experiments and analyses.

(1) Component Ablations vs. Standard SFT

Using the same training set and retrieved evidence, Hybrid outperforms standard SFT adapters by 3.54 percentage points (pp) in Judge score. The component ablations further distinguish their contributions: starting from the default random initialization, unsupervised history encoding (Inner-only) improves the Judge score by 1.38 pp over Episodic, supporting effective context internalization. Adding meta-learning (+Outer) further improves the Judge score by 3.24 pp over Inner-only by optimizing the initialization for contextual feature extraction and collaboration with episodic evidence. All Judge scores are evaluated using DeepSeek-V4.1-Flash.

MethodF1EMJudge
Episodic61.8942.3371.19
Episodic + SFT70.853.9772.27
Episodic + Inner-only70.7753.7972.57
Hybrid: Episodic + Inner + Outer71.2754.8175.81

(2) Matched-Supervision Baselines (Baseline + SFT)

Under the same SFT setting, competing baselines show substantial improvements in exact match, consistent with improved output-style alignment (UniversalRAG: +8.88 pp EM; Mem0: +23.47 pp EM). Nevertheless, Hybrid exceeds UniversalRAG + SFT by 1.85 pp in Judge score, even though UniversalRAG led our Episodic baseline by 0.30 pp before SFT. These controls indicate that matched QA supervision alone does not explain the gains from history encoding and synergistic training.

MethodF1EMJudge
UniversalRAG + SFT70.9251.0273.96
Mem0 + SFT57.5741.7760.79
Hybrid71.2754.8175.81

(3) Dataset Splits and Evidence Coverage

We split the datasets using a fixed seed (42), with strictly disjoint train/test QA IDs: 629/1,082 for Mem-Gallery and 200/50 for LoCoMo. We also evaluate sampling-seed variance (Sec. 5.5, Fig. 8c). Scenario-specific details appear in Appendix B.2, Table 6. All baselines receive the same complete dialogue history for memory construction, as required by the task protocol. Importantly, outer-loop training does not use gold-clue annotations; it uses only fixed evidence retrieved from episodic memory (Appendix C.4).

We additionally audit evidence overlap at two levels:

  • Clue-annotation level: 72.88% (704/966) of annotated Mem-Gallery test QAs share no target gold clues with training QAs in the same category.
  • Retrieved-context level, across all task categories: For 12.01% (116/966) of annotated Mem-Gallery test QAs and 44.0% (22/50) of LoCoMo test QAs, none of their required gold-clue turns appears in the retrieved RAG prompts of any training QA.

The following results on subsets with no gold-clue overlap with training retrieval contexts show that gains persist beyond the evidence retrieved for QA supervision.

BenchmarkTest subsetMethodF1 (%)Judge (%)
Mem-Gallery116/966 (12.01%)Episodic58.7268.1
Mem-Gallery116/966 (12.01%)Hybrid63.68 (+4.96 pp)70.47 (+2.37 pp)
LoCoMo22/50 (44.00%)Episodic62.3970.45
LoCoMo22/50 (44.00%)Hybrid82.14 (+19.75 pp)79.55 (+9.09 pp)

Together, these controls support benefits beyond direct QA supervision: the inner loop encodes test-time context into parameters, while the meta-learned prior ϕinit∗\phi_{\mathrm{init}}^{*} learns strategies for organizing and using factual evidence, supporting history encoding and collaboration between the two memory types.

AuthorsOpenReview ↗

Response to 1SVG Part 2. for Efficiency and Implementation Details

W2 & Q4: Efficiency and Implementation Details

We apologize for omitting the combined costs of Hybrid and Adaptive from Table 4, which initially reported only the standalone Episodic and Latent branch costs. We also clarify a reporting ambiguity: the reported 3,244.72 s Latent ingestion time included offline meta-training for five epochs, with four inner-loop steps, one outer-loop step, and meta-gradient updates across training scenes. Actual test-time ingestion requires only four inner-loop steps and takes 309.67 s/scene.

With parallel construction of Latent and Episodic memory, ingestion latency for Hybrid and Adaptive is governed by the slower Episodic branch:

max⁡(309.67,1958.61)=1958.61s/scene.\max(309.67, 1958.61) = 1958.61\\ \mathrm{s/scene}.

Hybrid's QA latency (10.11 s/query) is dominated by episodic retrieval, which could be reduced using lighter retrievers.

MethodIngestion latency (s/scene)QA latency (s/query)Input tokens/query
Hybrid1,958.6110.112,503.00
Adaptive1,958.6110.872,322.27

Adaptive maintains comparable QA latency (10.87 s/query; +0.76 s/query) while reducing input tokens by 7.22% through selective skipping of episodic retrieval. The probing overhead is largely offset by retrieval time savings. Confidence is calculated from token probabilities already produced during generation, requiring no additional forward passes for scoring:

= \exp\left(\frac{1}{T}\sum_{t=1}^{T}\log p_{\theta}(y_t\mid x,y_{<t})\right) = \left(\prod_{t=1}^{T}p_{\theta}(y_t\mid x,y_{<t})\right)^{1/T}.$$ Generation continues from the first-token probe, and complete Latent outputs are directly reused when the Latent branch is selected. **Implementation details.** Appendix C.3 (Table 8) reports LoRA configurations, inner-/outer-loop hyperparameters, seeds, and infrastructure. Appendix C.4 describes meta-training, and Appendix E (Table 13) details router calibration. We will add explicit references to these sections in the main text.
AuthorsOpenReview ↗

Response to 1SVG Part 3. for Adaptive Routing Effectiveness and Selection

W2 & Q3: Adaptive Routing Effectiveness and Selection

We thank the reviewer for prompting a deeper analysis of the adaptive router. We examine signal effectiveness, statistical stability, reliability at the distribution tails, and branch selection across splits. Inference costs are reported above.

(1) Routing Ablations and Simple Baselines

We compare our router under calibrated thresholds (τd=0.40\tau_d=0.40, τc=0.86\tau_c=0.86) and fixed thresholds (τd=0.1\tau_d=0.1, τc=0.95\tau_c=0.95) against single-signal ablations and baselines calibrated using the same protocol.

Divergence-only routing improves over Hybrid, whereas Confidence-only routing performs slightly below Hybrid in the calibrated setting. Combining both signals achieves the best calibrated performance, and the combined router also improves over Hybrid under fixed thresholds. Calibrated lookup and short-draft baselines remain close to Hybrid, whereas LLM-prior routing performs worse. The latter cannot directly observe the internal latent memory state.

MethodMem-Gallery (7B)
Latent40.02
Episodic71.61
Hybrid73.29
Divergence-only, calibrated74.15
Confidence-only, calibrated72.951
Query-length lookup, calibrated73.611
Query-type lookup, calibrated73.668
Short-draft routing, fixed47.307
Short-draft routing, calibrated73.611
LLM-prior routing70.69
Adaptive (ours), fixed74.972
Adaptive (ours), calibrated75.81

(2) Statistical Stability and Reliability at the Distribution Tails

We conduct a 5,000-resample bootstrap analysis of the gains over fixed Hybrid. For Mem-Gallery (7B), the 95% confidence intervals are strictly positive under both fixed and calibrated routing thresholds.

SettingFixed routing - Hybrid: gain [95% CI]Calibrated routing - Hybrid: gain [95% CI]
Mem-Gallery (7B)+1.68 [+0.92, +2.60]+1.99 [+1.24, +2.87]

Our distribution plots show stronger discriminability at the signal tails: low JSD and high confidence. As the thresholds move toward these extremes, the observed Judge gains increase monotonically, with almost no losses. Threshold calibration therefore helps balance selection reliability and coverage.

ConditionNo. of queries / CoverageJudge gain [95% CI]Wins / Ties / Losses
d≤0.1,c>0.95d\le 0.1,\\ c>0.95108 / 12.24%+6.187 [+1.416, +11.615]9 / 98 / 1
d≤0.05,c>0.975d\le 0.05,\\ c>0.97590 / 10.20%+7.422 [+2.739, +12.746]7 / 83 / 0
d≤0.01,c>0.99d\le 0.01,\\ c>0.9956 / 6.35%+10.714 [+3.571, +18.333]6 / 50 / 0

(3) Branch Selection and Cross-Split Stability

On Mem-Gallery (7B), the Latent/Episodic/Hybrid selection ratios are 12.2%/52.6%/35.15%. The substantial use of Episodic memory is consistent with the benchmark's query composition, where 58% of queries require at most two gold clues. Single-scenario evaluations across random-seed splits further show stable Adaptive performance and branch-selection patterns.

SeedRAG / Latent / Hybrid selectionsAdaptive F1Adaptive EM
118 / 5 / 765.0846.67
322 / 5 / 362.8443.83
4521 / 3 / 664.8946.67

Together with the cost analysis above, these results support a practical balance between accuracy and inference efficiency.

We hope these responses clarify that the improvements are not explained solely by direct QA supervision, but also reflect the complementary contributions of history encoding and meta-learned alignment. The additional analyses support the effectiveness of adaptive routing for different query demands at a modest inference overhead. In light of this additional evidence, we respectfully ask you to consider revising your score upward.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.