acceptodds
Under review as a conference paper at ICLR 2027

AQUI: Asymmetric Query Embedding for Implicit Reasoning-aware Memory Retrieval

Abstract

Memory systems are a core component of large language model (LLM) agents for retrieving multi-session conversations stretches beyond the context window. A particularly challenging setting is implicit reasoning, where the user's query is only indirectly connected to the relevant memory fact needed for a correct response. Existing implicit-reasoning memory systems build and traverse per-user trees or graphs, requiring costly memory construction, many LLM calls at inference time, or heavy dependence on the LLM's own reasoning during retrieval. We propose AQUI (Asymmetric QUery embedding for Implicit reasoning-aware memory retrieval), an embedding-based approach that moves the query representation toward implicitly related memory facts while keeping fact embeddings fixed. This asymmetric design lets the retriever internalize implicit connections without distorting the stored memory space, enabling relevant user facts to be retrieved in a single first-stage search. To train AQUI, we introduce a three-stage data generation pipeline that constructs user facts, implicit bridge facts, and query-negative pairs for reasoning-aware retrieval; the resulting synthetic training rows are generated without any access to the evaluation benchmarks. On ImplexConv-Opposed and ActMemEval-hard, AQUI's single-stage retrieval (no LLM call) already surpasses the state-of-the-art baselines' in session recall; a controlled ablation that holds the training data and hard negatives fixed attributes the gain to the asymmetric objective itself. We then pair AQUI with a lightweight LLM reranker in a simple end-to-end answer-generation pipeline, which yields higher LLM-as-a-judge answer accuracy than baselines. We also show that ActMemEval contains strong lexical and semantic shortcuts, and propose ActMemEval-hard, a strengthened supplementary dataset that removes these shortcuts. On this harder setting as well, AQUI achieves stronger retrieval and answer-generation performance while requiring no per-user memory construction, unlike graph- or tree-based memory systems

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Agentic AI, Retrieval, Information Extraction, Question Answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HeEXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AQUI addresses retrieval of personal memory facts that are relevant to a query through an implicit connection rather than topical similarity. It adapts the query branch of Qwen3-Embedding-8B with LoRA while keeping fact embeddings fixed, using 2,414 synthetic training rows with implicit positive facts and topical general-fact negatives. The paper evaluates retrieval on ImplexConv-Opposed and a rewritten ActMemEval-hard dataset, and evaluates answer generation after optional GEPA-optimized LLM reranking. It reports substantial improvements over the pretrained encoder and a full-encoder InfoNCE baseline, alongside lower memory-construction overhead than TaciTree and ActMem. The practical retrieval gains are promising, but the experimental evidence does not yet isolate the claimed source of improvement sufficiently.

Critical strengths
  1. The reported first-stage retrieval improvement is substantial and does not depend on an inference-time LLM.

  2. The method offers a simple deployment path with useful cost evidence.

  3. The synthetic data pipeline is well aligned with implicit memory retrieval, combining cross-topic bridges with topic-matched hard negatives to provide targeted training supervision.

Critical weaknesses
  1. The central ablation does not isolate asymmetry from the adaptation parameterization.

  2. The independence of model selection from the reported evaluation is not established by the documented protocol.

Questions for authors
  1. Under the same LoRA rank, target modules, training data, negatives, and comparable hyperparameter tuning, how does symmetric query-and-fact LoRA perform relative to query-only LoRA? Does the claimed advantage of asymmetry remain?

  2. Which examples and split were used for early stopping and for ranking the configurations in Tables 7–8? Were these configurations selected exclusively using synthetic validation data, with the displayed benchmark recalls computed only afterward, or did benchmark recall influence selection?

1 comment

We appreciate the reviewer for thoughtful comments. We would like to offer clarifications for each point.

Weakness 1 and Question 1

We would like to clarify that both Naive-InfoNCE and AQUI embedding model training used LoRA adapters with the same rank. Both used the identical training dataset and epochs with different hyperparameters. We conduct hyperparameter search for each of them.

Please take a look into line 426 of the rebuttal revision.


Weakness 2 and Question 2

We apologize for the insufficient description and our oversight. Table 7 shows the results aligned with the ImplexConv dataset, and the best score of the ImplexConv dataset re-appears in Table 1. This is not how we intended to run the hyperparameter search. The benchmark recall should not have affected the selection. Our hyperparameter search intended to use a held-out validation synthetic dataset with the size of 10% of the training dataset. We used mean validation hit@{10, 20, 50, 100} and re-selected the hyperparameters. We update Table 7 in the rebuttal revision as follows.

#lrτclipwarmupbatchrVal mean hitVal hit@10Val hit@100ImplexConv R@10ImplexConv R@20ImplexConv R@100
11e-40.025.00.164320.8780.7450.9880.2740.4020.811
22e-50.021.00.1128640.8700.7060.9800.3250.4480.822
32e-50.025.00.0564320.8690.6900.9760.2990.4190.814
–Base model (no adapter)0.2640.0350.5330.0850.1420.540

We updated Table 1, 2, 10 and Figure 5 to account for this change. Most importantly, AQUI still outperforms the baselines in both ActMemEval-Hard and ImplexConv datasets by wide margins.

Please refer to line 757 of rebuttal revision.

Reviewer_2KMqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AQUI, a query-only LoRA adaptation of Qwen3-Embedding-8B for retrieving personal memories that are indirectly relevant to a query. Fact embeddings remain frozen, while contrastive training moves query representations toward implicitly connected memories. A synthetic generation pipeline produces 2,414 training rows containing user facts, implicit bridges, queries, and general-fact negatives. Evaluation covers ImplexConv-Opposed and an author-created ActMemEval-hard variant, with optional GEPA-optimized LLM reranking and downstream answer generation. The reported results show improved session recall and answer accuracy over pretrained retrieval, symmetric fine-tuning, and two structured memory systems.

Critical strengths
  • A practical retrieval design with useful deployment properties. Query-only adaptation preserves an existing fact index and performs first-stage retrieval without inference-time LLM calls. This offers a meaningful alternative to building and traversing per-user graph or tree structures.

  • Promising gains beyond the synthetic training distribution. On ImplexConv-Opposed, first-stage recall@10 increases from 0.0862 for the pretrained encoder to 0.3322 for AQUI, before reranking. The unmodified external benchmark therefore provides encouraging evidence that the synthetic training procedure learns useful retrieval associations.

Critical weaknesses
  • The main ablation does not isolate asymmetry from parameter-efficient adaptation. Table 1 compares full-parameter symmetric fine-tuning against query-only LoRA, simultaneously changing which branches are updated and the parameterization and regularization of those updates. A matched symmetric-LoRA baseline is necessary before attributing the improvement specifically to freezing the fact space; this distinction is consequential because query-only adaptation itself is acknowledged as existing prior work.

  • Independent evaluation and benchmark validity are insufficiently established. The best configuration in the hyperparameter-search table reports exactly the same recall values as the main ImplexConv results, but the validation partition used for selection is not identified despite the claim that evaluation data never informs hyperparameter selection. Moreover, ActMemEval-hard reduces 247 original cases to 72 without explaining the full exclusion process or independently verifying that paraphrases preserve the original query intent and gold-memory relevance.

  • For the topic of Reasoning-awareretrieval, there is also another line of study: query-response generative ranking, and the subsequent distillation into the bi-encoder retriever, which is not coverred bu Introduction or Related Works. Adding them would make the background discussion more solid:

[1] Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval, ECIR 2025

[2] Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA, ICMR 2026

  • Typos and bad tex commands. For example, almost all citations lack of parenthess (using \citep or \cite instead of \citet); The figure captions do not have a period. The bottom of Figure 1 caption seems to be condensed too much.
Questions for authors
  • Matched-control comparison: Are results available for symmetric LoRA using the same adapter rank, training data, negatives, and tuning budget as AQUI? Were fact embeddings recomputed after training the symmetric baseline?

  • Model-selection clarification: Which dataset and partition produced Tables 7–8, and why do the selected configuration’s scores exactly reproduce the main evaluation scores? Please specify the training/validation split and whether queries sharing the same synthetic memory were grouped together.

  • Benchmark-construction clarification: Why do only 72 of the original 247 ActMemEval cases remain? How was preservation of query meaning, answer validity, and gold-session relevance checked, and are paired before/after results available on those same 72 cases?

  • Retrieval and downstream evaluation: Please define whether session recall measures recovering any fact from the gold session or recovering the specific fact required to answer the query, and explain how (K=151) is handled with a top-100 candidate pool. Given the low ImplexConv answer accuracies despite substantially higher session recall, an oracle-memory answer baseline would materially clarify whether the bottleneck is evidence retrieval, answer generation, or evaluation.

2 comments

We appreciate the reviewer for thoughtful comments. We would like to offer clarifications for each point.

Weakness 1 and Question 1

We would like to clarify that we used LoRA with the same rank for both Naive-InfoNCE and AQUI embedding model training. We also used the same training dataset and same training epochs. However we conducted independent hyperparameter search for both and reported the ones with the best validation hit rate. Thus the result is a fair comparison between Naive-InfoNCE and AQUI retrieval model training. For a symmetric model, all memory facts are re-embedded with the trained adapter at test and validation time. We cached the base-model facts and used them only for the query-only models.

Please take a look into line 426 of the rebuttal revision.


Weakness 2 and Question 2

We apologize for the insufficient description and our oversight. Table 7 shows the results aligned with the ImplexConv dataset, and the best score of the ImplexConv dataset re-appears in Table 1. This is not how we intended to run the hyperparameter search. Our hyperparameter search intended to use a held-out validation synthetic dataset with the size of 10% of the training dataset. We used mean validation hit@{10, 20, 50, 100} and re-selected the hyperparameters. We update Table 7 in the rebuttal revision as follows.

#lrτclipwarmupbatchrVal mean hitVal hit@10Val hit@100ImplexConv R@10ImplexConv R@20ImplexConv R@100
11e-40.025.00.164320.8780.7450.9880.2740.4020.811
22e-50.021.00.1128640.8700.7060.9800.3250.4480.822
32e-50.025.00.0564320.8690.6900.9760.2990.4190.814

We updated Table 1, 2, 10 and Figure 5 to account for this change. Most importantly, AQUI still outperforms the baselines in both ActMemEval-Hard and ImplexConv datasets by wide margins

Please refer to line 757 of rebuttal revision.


Weakness 2, Question 3 (ActMemEval-Hard)

The original ActMemEval dataset has 247 samples. We obtain ActMemEval-Hard through two processes. For each process, sessions that fail to pass the evaluations are dropped.

Lexical Control: GPT-5.4-mini rewrites the session so that BM25 score of the paraphrased session and Query is 0. We exclude the session and query pair that GPT-5.4-mini fails after three attempts. The first stage removes 23 sessions.

Implicit Rewrite: GPT-5.4-mini re-writes the stage-1 paraphrased sessions. GPT-5.4 judge compares the re-written session to the original session to check followings:

  • Direct topic score (0-5): The judge measures if the re-written session has topic overlap with the query. 0 meaning no surface topic overlap and 5 meaning explicitly the same topic. Sessions with the score equal or smaller than 2 are accepted.
  • Meaning preservation (0-5): The judge measures if the re-written session keeps the original memory. 0 meaning no memory and 5 meaning everything is kept. Sessions with the score equal or higher than 4 are accepted.
  • External Bridge (T/F): The judge evaluates if one or more inference steps is necessary to answer the question given the session and query. Sessions that receive TRUE are accepted.
  • Useful for answer (T/F): The judge checks if the session helps answering the query. Sessions that receive TRUE are accepted.
  • Fatal Issues: The judge can raise additional issues as a paragraph that are not expressed in the previous evaluations. Sessions that receive empty are accepted.

GPT-5.4-mini re-writes each session up to three times, and we drop the sessions fails to pass the judge evaluation. In this stage, 152 sessions are removed.

We only re-write the session but not the query because the query is directly related to the answer, thus paraphrasing the query could directly change how the model answers, even with the session. Thus by the construction, both query intent and gold-memory relevance is fully preserved. Examples are listed below.

User queryOriginal ActMemEvalActMemEval-Hard
I see the new 'Daft Punk' anniversary vinyl is out. Find me a link to buy the Deluxe Edition.I actually just found a pre-order link for the Daft Punk Anniversary Deluxe Vinyl. I snagged it immediately before the bots got to it.I actually found a preorder page for the special reissue from that French electronic duo, and I grabbed it right away before it disappeared.
I'm taking the new car out. Map a highway route where I can set the cruise control to 80mph.Yeah, the manual says 'Do not exceed 4000 RPM or use cruise control for the first 500 miles'. The odometer only reads 150 miles.Yeah, the manual says 'Do not exceed 4000 RPM or use speed assistance for the first 500 miles'. The odometer only reads 150 miles.

Please refer to Appendix E.

We provide clarification for the rest of the items here (Weakness 3, 4 and Question 4)


Weakness 3

These are relevant works, and we discuss them in the rebuttal revision. Please refer to line 152 of the rebuttal revision.

The listed papers solve rationale retrieval, which is a similar problem to implicit reasoning. Both implicit reasoning and rational retrieval needs to select the items that are not semantically connected to the user query. However, they differ in two aspects.

  1. Their retrieval objective is to determine the right task or candidate For example, given a user’s query “I spilled my coke.”, the system should retrieve “find a sponge”.
  2. The retrieval items are predefined. Rational retrieval supposes a pre-defined action lists. From the aforementioned example, "find a sponge" is one item in the finite list. On the other hand, the number of facts that are subject to retrieval is unbounded.

Overall, these works are similar to AQUI in the sense that the query has no semantic overlap to the retrieval target. So we discuss these works in the rebuttal revision. However, we do not compare these works as baselines, as a direct application of these methods to our problem setting is infeasible due to the fundamental differences.

Please refer to lines 152 of the rebuttal revision.


Weakness 4

We updated the rebuttal revision to fix the typos and updated erroneous formatting.


Question 4

Regarding Session recall: Our dataset, ActMemEval-Hard and ImplexConv are session based: Given a conversation history (multiple sessions), there is only one session that is relevant to the question. For implexConv dataset, retrieval system needs to select one gold session out of average 110 sessions for each question. Thus it is critical to report the session recall, to show the performance of the retrieval system. Our baselines, ActMem and TaciTree does not report the session recall with respect to KK.

Since each session is an actual conversation between a user and an AI model, AQUI and baselines break each long session and extracts multiple lines of short factual statements such as “User bought a house” or “User has a dog” and stores their embeddings (or in text for TaciTree). On retrieval, those facts are retrieved back. We label each facts to their session and measure whether the retrieval system properly selected the facts from the right session. Session recall counts a gold session as retrieved if any of its facts appears in the top-K.

Please refer to line 362 of the rebuttal revision.

Regarding K=151: Our baseline TaciTree uses an LLM to retrieve the facts and it is unable to strictly control the number of retrievals. To make a fair comparison, we matched the maximum retrieval budget to TaciTree. For the case where TaciTree decided to retrieve 151 facts, AQUI initially retrieves 151 facts and re-rank them to 100. We clarified this in the rebuttal revision.

Please refer to line 367 of the rebuttal revision.

Compare with the oracle memory: We added an oracle-memory baseline: the same answergen model and judge. We tested oracle: where the exactly gold session facts are given, and Oracle+distractors, where both gold facts and facts from irrelevant sessions (distractors) are present. The results in Table 1 indicate that evidence retrieval, not answer generation, is the main bottleneck.

Table 1. Answergen model performance with oracle and oracle+distractors

Facts given to the answer modelImplexConv-Opposed, K = 10 / 20 / 30 / 50ActMemEval-hard, K = 10 / 14 / 20 / 30 / 40
Gold evidence only0.9110.944
Gold evidence + distractors0.727 / 0.638 / 0.594 / 0.5320.817 / 0.803 / 0.831 / 0.761 / 0.732
AQUI top-K0.146 / 0.182 / 0.208 / 0.2290.662 / 0.690 / 0.662 / 0.761 / 0.718

With the gold evidence alone, the answer model reaches 0.911, so the judge and the answer model are not the cause of the low scores. With the gold session facts hidden among K AQUI-ranked distractors, it still reaches 0.53–0.73. With AQUI’s own top-K, it drops to 0.15–0.23, because AQUI retrieves 19–46% of the gold session facts (Table 2). The bottleneck on ImplexConv is therefore implicit retrieval, which is the problem AQUI targets.

Table 2. Session-level recall of AQUI

ImplexConv-Opposed, K = 10 / 20 / 30 / 50ActMemEval-hard, K = 10 / 14 / 20 / 30 / 40
0.19 / 0.28 / 0.35 / 0.460.82 / 0.85 / 0.87 / 0.93 / 0.94

Please refer to Appendix G of the rebuttal revision

Reviewer_Ap7KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AQUI adapts only the query side of an embedding model through LoRA while keeping stored fact embeddings fixed. A synthetic pipeline generates 2,414 training rows linking queries to indirectly relevant personal facts, with general-fact hard negatives. The paper evaluates retrieval and optional LLM reranking on ImplexConv-Opposed and a revised ActMemEval-hard benchmark. It reports improved retrieval recall and downstream answer quality with less memory-construction overhead than tree- or graph-based systems.

Critical strengths

1.The fixed fact space has a useful deployment benefit: existing memory vectors can be reused, and first-stage retrieval needs no LLM call. Table 1 reports a substantial recall@10 improvement from 0.0862 for the pretrained encoder to 0.3322 without reranking. 2.The paper separates retrieval from the optional reranker and provides detailed data-generation prompts. Its analysis of lexical shortcuts in ActMemEval is also useful, since strong surface retrieval can otherwise obscure what an implicit-reasoning benchmark actually measures.

Critical weaknesses

1.Table 1 does not fully isolate asymmetry. The naive baseline updates the full encoder, while AQUI uses query-only LoRA. Thus both the updated branches and the adaptation capacity change. The gains could partly reflect regularization on a small training set, rather than the claimed preservation of world knowledge. A symmetric LoRA control would make the central attribution stronger. 2.The hyperparameter-selection protocol is difficult to reconcile with the stated benchmark independence. Section 3.1 says evaluation data are not used for selection, but Appendix B chooses the top configuration in Table 7, whose recall values match the ImplexConv results in Table 1. This may have a straightforward explanation, but the validation split and selection criterion need to be explicit.

Questions for authors

1.Was a baseline with the same LoRA rank and target modules applied to both query and fact branches evaluated? How were baseline hyperparameters selected? 2.Which data determine the rankings in Tables 7-8 and the final configuration? Please explain how these results relate to the test scores in Table 1.

1 comment

We appreciate the reviewer for thoughtful comments. We would like to offer clarifications for each point.

Weakness 1 and Question 1

We would like to offer a clarification. We used LoRA adapters with identical ranks to train Naive-InfoNCE and AQUI embedding models. Both used the same LoRA and same training data. For Naive-InfoNCE, we ran grid search over the learning rate and batch size to find the optimal hyperparameter setting. We have clarified and updated the rebuttal revision.

Please take a look into line 426.


Weakness 2 and Question 2

We apologize for the inaccurate description of Section 3.1 and our oversight. Table 7 was chosen by the ImplexConv test recall. This is a mistake and is not how we intended our protocol to be. Our hyperparameter search intended to use the held-out validation synthetic dataset that is the size of 10% of the training dataset. To correct the table 7 and other results, We re-selected the hyperparameters using the synthetic validation split. We used mean validation hit@{10, 20, 50, 100}. We provide the revised Table 7. We updated the table in the rebuttal revision.

#lrτclipwarmupbatchrVal mean hitVal hit@10Val hit@100ImplexConv R@10ImplexConv R@20ImplexConv R@100
11e-40.025.00.164320.8780.7450.9880.2740.4020.811
22e-50.021.00.1128640.8700.7060.9800.3250.4480.822
32e-50.025.00.0564320.8690.6900.9760.2990.4190.814

We updated Table 1, 2, 10 and Figure 5 to account for this change. Most importantly, AQUI still outperforms the baselines in both ActMemEval-Hard and ImplexConv datasets by wide margins.

Please refer to line 757

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive feedback. We have uploaded a revised paper; all changes are marked in blue. Most of what reviewers asked were clarifications. The main changes are summarized below; details are in the individual responses.

1. Matched symmetric baseline (All Reviewers). We offer a clarification. We used LoRA adapters with same rank to train Naive-InfoNCE and AQUI embedding models. Both used the same LoRA and same training data.

2. Hyperparamter tuning (All reviewers) It was our oversight. We accidently have reported the setting best for ImplexConv. We originally use a held-out validation set to select the hyperparameters. We updated the relevant tables. AQUI still outperforms the baselines with big margin.

3. ActMemEval-Hard (2KMq). We added a detailed explanation on ActMemEval-Hard data sanitization protocol, pseudocode and data examples.

4. Bottleneck analysis (2KMq). We added an experiment new oracle analysis that gives the answer model only the gold facts, the gold fact with hard distractors, and AQUI's own top-K. With the gold fact, the accuracy is 0.91 on ImplexConv, so answer model generates good score if facts are properly retrieved. Most errors come from evidence that retrieval misses, which is the problem AQUI targets.

5. Clarifications (2KMq): We offer a formal definition of session recall and clarifications on how the candidate pool and reranker work when K exceeds 100 (e.g. K = 151);

We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.