acceptodds
Under review as a conference paper at ICLR 2027

PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory

11 views

Abstract

Long-context reasoning remains a critical bottleneck for large language models, as recent recurrent-memory approaches face two inherent challenges: sequential chunk-wise updates can overwrite early critical evidence with later irrelevant content, and serial inter-chunk dependencies limit parallelism and cause latency to increase with context length. To address these issues, we propose PI-Mem (Parallel-Iterative Memory), a mechanism that processes all chunks in parallel and iteratively refines a shared memory over a bounded number of turns. In each turn, PI-Mem reads all chunks in parallel conditioned on the current memory, selects new or complementary evidence from each chunk, and merges the selected evidence into a compact shared memory for the next turn. To discourage redundant turns, we optimize the workflow through reinforcement learning with an auxiliary turn-efficiency reward, enabling the model to adaptively exit once sufficient evidence has been accumulated. We evaluate PI-Mem with Qwen3.5-35B-A3B and Qwen2.5-7B on the HotpotQA benchmark across context lengths up to 3.6 million tokens and find that it outperforms the recurrent-memory baseline by +6.25 and +7.81 absolute points while achieving 6.1 and 2.1 inference speedups, respectively. These results demonstrate that PI-Mem breaks the accuracy–efficiency trade-off in long-context reasoning and provides a scalable approach to complex multi-hop question answering over extremely long documents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Context LLM, Memory, Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iMWTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work find that Long-context reasoning remains a critical bottleneck for large language models, as recent recurrent-memory approaches face two inherent challenges: sequential chunk-wise updates can overwrite early critical evidence with later irrelevant content, and serial inter-chunk dependencies limit parallelism and cause latency to increase with context length. Therefore, this ork propose PI-Mem that process all chunks ina parallel and iteratively refines a shared memory.

Critical strengths
  1. This work uses Figure 1 to explain the core idea, which is easy to understand.
  2. The paper propose the PI-memory, which use a shared memory so that the model could parallel process the long-context.
  3. The experiment support the effectiveness of PI-memory.
  4. The latency analysis proves that the PI-memory could be 16.9× faster than Vanilla and 6.1× faster than MemAgent, this suggests that the PI-memory could be deployed to speed up long context processing.
Critical weaknesses
  1. It seems that the model is Qwen, and it is not sure whether the method works well for other models.
  2. As the PI-memory is used to improve RL training for long-context. It is not sure whether the PI-memory is sensitive to the RL methods.
  3. What is the training cost of PI-memory, compared to baselines.
Questions for authors

NA

5 comments
AuthorsOpenReview ↗

Response to Reviewer iMWT (Part 1)

Dear reviewer iMWT,

Thank you for your careful review and constructive feedback. We address your concerns with additional experiments on a non-Qwen backbone, comparisons across RL training configurations, and a direct comparison of training costs.

W1: Generalization beyond Qwen

PI-Mem operates at the workflow level and is not tied to a specific LLM architecture, so it can be applied to different LLM backbones without architectural changes. Stronger instruction-following and evidence-processing capabilities help a model select relevant evidence and refine the shared memory, enabling it to make better use of PI-Mem for long-context tasks.

Training-free evaluation on Gemma 4 E4B. We apply Vanilla, MemAgent, and PI-Mem to the same instruction-tuned checkpoint gemma-4-E4B-it, without any additional training. We evaluate 64 questions at each HQA length. Both memory workflows use 15K-token chunks and a 4,096-token output budget, with K=3K=3 for PI-Mem. Vanilla uses the same output budget.

DocumentsLength settingVanillaMemAgentPI-Mem
507K78.1373.4476.56
10014K68.7567.1968.75
20028K64.0665.6367.19
40056K75.0065.6362.50
800112K62.5051.5657.81
1600224K23.4457.8162.50
3200448K15.6353.1367.19
6400896K0.0046.8859.38
128001.8M0.0056.2559.38
256003.6M1.5650.0068.75
Average—38.9158.7565.00

Scores are Sub-EM (%). Length labels follow the paper's document-count settings; actual token counts depend on the tokenizer. PI-Mem achieves an average score of 65.00, compared with 58.75 for MemAgent. At the longest setting, PI-Mem scores 68.75 versus MemAgent's 50.00. These results demonstrate that the workflow's benefits extend beyond Qwen to Gemma 4 E4B without backbone-specific RL training.

AuthorsOpenReview ↗

Response to Reviewer iMWT (Part 2)

W2: Sensitivity to the RL method

To examine PI-Mem's sensitivity to the RL method, we additionally train Qwen2.5-7B PI-Mem with standard GRPO and Dr. GRPO configurations.

Because repeating full RL training is expensive, we conduct these additional experiments on Qwen2.5-7B for 80 steps, whereas the Qwen2.5 PI-Mem checkpoint reported in the paper was trained to step 240 (Appendix A, Table 5). All three configurations below are evaluated at step 80 on the eight HQA lengths from 7K to 896K, with 64 questions per length.

Training configurationDivide advantages by group stdLoss aggregationKL coefficientReward: step 0Reward: step 80Reward gainHQA 7K–896K (step 80)
Standard GRPO [1]Yesseq-mean-token-mean0.0010.5640.826+0.26269.14
Dr. GRPO [2]Noseq-mean-token-sum-normNo KL0.5650.787+0.22268.75
Original configurationNotoken-mean0.0010.5510.799+0.24869.34

All three configurations show substantial reward improvements and achieve similar downstream HQA scores, ranging from 68.75 to 69.34. This supports robustness across the tested RL configurations: the workflow can be trained effectively with both standard GRPO and Dr. GRPO.

AuthorsOpenReview ↗

Response to Reviewer iMWT (Part 3)

W3: Training cost and inference efficiency

Thank you for raising the concern about training cost. We acknowledge that PI-Mem requires more training time than MemAgent for the same number of training steps in our Qwen3.5 experiments.

PI-Mem takes longer to train in our Qwen3.5 experiments. We compare Qwen3.5-35B-A3B MemAgent and PI-Mem using the same training data and hyperparameters, as described in Appendix A. Both use 32 H200 GPUs and are trained for 80 steps. The training-time estimates from the logs are:

MethodH200 GPUsTraining stepsTraining time
MemAgent328056 h 48 min 18 s
PI-Mem328074 h 38 min 07 s

PI-Mem takes approximately 31.4% longer. Its multi-turn workflow introduces additional processing during training, including repeated chunk reads and memory refinement. We acknowledge this additional training cost; these additional calls produce more call-level training sequences within each trajectory and increase the number of training tokens, while our efficiency claim concerns end-to-end inference latency, not faster training. For Qwen2.5 MemAgent, we use the officially released checkpoint and do not have a corresponding local training-time measurement. For context, a contributor to the official repository reports approximately 32 GPU-days for 150 training steps and continued performance improvements over the first 400 steps [3]. This comment does not specify the training step of the released checkpoint.

The additional training cost is distinct from inference latency. Appendix B.3, Table 11 reports the complete end-to-end inference workload, including all executed turns. For the 64-question HQA evaluation at 3.6M, using eight H200 GPUs with TP=2:

MethodQwen3.5 latency (s)Qwen2.5 latency (s)
MemAgent12,060.715,535.50
PI-Mem1,974.652,682.48

PI-Mem completes this workload faster by parallelizing chunk processing, despite performing more total FLOPs than MemAgent. The additional one-time training cost is therefore accompanied by lower latency during inference.

The workflow can also provide gains without additional training. The Gemma 4 E4B results above show a training-free HQA advantage over MemAgent. The existing Qwen3.5 results in Appendix B.2 show the same pattern: without RL, PI-Mem achieves 69.84 versus MemAgent's 64.37 on HQA (Table 7), and 96.25 versus 65.26 on RULER OOD tasks (Table 8). These results show that backbones with sufficient instruction-following and evidence-processing capabilities can already benefit from PI-Mem without additional training. RL remains useful for further improvement: Qwen3.5 PI-Mem reaches 81.25 on HQA and 98.04 on the OOD tasks after training.

References

[1] Zhihong Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024.

[2] Zichen Liu et al. Understanding R1-Zero-Like Training: A Critical Perspective. 2025.

[3] huiyeruzhou. Training-cost clarification in the official MemAgent repository, Issue #8. July 15, 2025.

Reviewer_4hb6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PI-Mem, a parallel and iterative memory method for long-context reasoning. Unlike MemAgent, which reads document chunks one by one, PI-Mem reads all chunks in parallel. In each turn, it selects useful evidence and merges it into a shared memory. The process repeats until no new evidence is found or the maximum number of turns is reached. PI-Mem keeps much of MemAgent’s data construction and RL setup, but replaces sequential memory updates with parallel read-select-merge iterations.

The full workflow is trained with reinforcement learning. The reward combines answer accuracy with a bonus for using fewer turns. Experiments with Qwen3.5-35B-A3B and Qwen2.5-7B show improvements over MemAgent on RULER and LongBench. PI-Mem also reports large inference speedups at very long context lengths.

Critical strengths
  1. Strong accuracy and latency results at very long context lengths. PI-Mem performs better than MemAgent with both tested backbones, especially on contexts containing millions of tokens. It also provide large inference speedups over MemAgent. These results show that parallel memory processing is a useful alternative to sequential memory updates.

  2. The paper provides useful ablations and evaluations. It studies reinforcement learning, selection, merging, the turn reward, the maximum number of turns, and memory size. It also evaluates two different model backbones on two long-context benchmarks. These experiments provide useful support for the main parts of the method.

Critical weaknesses
  1. It is hard to tell how stable the reported improvements are. Each model is trained only once, and each sample is evaluated only once with stochastic decoding. RULER has only 64 samples for each task and context length. At 3.6M tokens, the reported gains mean only four or five more correct answers. More runs or confidence intervals are needed to show that these gains are reliable.

  2. The reported speedups rely heavily on parallel hardware and do not mean that PI-Mem uses less total compute. PI-Mem reads all chunks again in each turn, so it uses more FLOPs than Mem-agent. The experiments use eight H200 GPUs and do not report single-query latency, peak memory use, or results with fewer GPUs. Therefore, the claim that PI-Mem breaks the accuracy-efficiency trade-off may be too strong.

Questions for authors
  1. Are the accuracy improvements stable across different training seeds or repeated decoding runs? Can the authors provide confidence intervals or uncertainty estimates for the main HQA and LongBench v2 results?

  2. How does the latency improvement change with the number of GPUs and with a single input query? Can the authors also report peak GPU memory and total inference cost?

  3. How many chunk observations are usually selected in each turn, and how long is the input to the merge call? Could the merge input exceed the model’s context limit when many chunks contain useful information?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4hb6 (Part 1)

Dear reviewer 4hb6,

Thank you for your careful review and constructive suggestions. We have supplemented the evaluation with repeated decoding runs, independent training runs, and single-query latency and memory measurements across GPU configurations.

W1/Q1: Stability across repeated evaluation and training runs

Repeated decoding with the fully RL-trained models. We summarize three decoding runs per model–method pair, including the original reported evaluation, using the fully trained final checkpoints. For HQA, each run covers all ten lengths from 7K to 3.6M, with 64 questions per length. We report the mean, sample standard deviation, and 95% confidence interval across runs.

HQA: average across all ten lengths

Model / MethodMean ± Std (%)95% CI
Qwen3.5 PI-Mem80.42 ± 1.18[77.48, 83.36]
Qwen3.5 MemAgent75.89 ± 0.59[74.42, 77.35]
Qwen2.5 PI-Mem84.17 ± 0.09[83.94, 84.39]
Qwen2.5 MemAgent78.12 ± 0.95[75.76, 80.49]

Qwen3.5: HQA by context length

Method / Statistic7K14K28K56K112K224K448K896K1.8M3.6M
MemAgent — Mean ± Std82.81 ± 1.5680.21 ± 0.9081.77 ± 3.2582.29 ± 0.9078.12 ± 1.5671.35 ± 3.2572.40 ± 1.8069.27 ± 2.3970.31 ± 1.5670.31 ± 1.56
MemAgent — 95% CI[78.93, 86.69][77.97, 82.45][73.69, 89.85][80.05, 84.53][74.24, 82.01][63.27, 79.43][67.91, 76.88][63.34, 75.20][66.43, 74.19][66.43, 74.19]
PI-Mem — Mean ± Std86.98 ± 0.9081.77 ± 0.9082.29 ± 2.3981.25 ± 4.1381.77 ± 3.2578.12 ± 4.1381.77 ± 1.8077.60 ± 1.8074.48 ± 0.9078.12 ± 2.71
PI-Mem — 95% CI[84.74, 89.22][79.53, 84.01][76.36, 88.22][70.98, 91.52][73.69, 89.85][67.86, 88.39][77.29, 86.25][73.12, 82.09][72.24, 76.72][71.40, 84.85]

Qwen2.5: HQA by context length

Method / Statistic7K14K28K56K112K224K448K896K1.8M3.6M
MemAgent — Mean ± Std83.33 ± 1.8082.29 ± 0.9076.04 ± 1.8081.25 ± 1.5677.60 ± 0.9075.00 ± 1.5677.08 ± 0.9076.56 ± 4.1378.12 ± 4.6973.96 ± 0.90
MemAgent — 95% CI[78.85, 87.82][80.05, 84.53][71.56, 80.52][77.37, 85.13][75.36, 79.85][71.12, 78.88][74.84, 79.32][66.29, 86.83][66.48, 89.77][71.72, 76.20]
PI-Mem — Mean ± Std83.33 ± 0.9082.29 ± 0.9079.69 ± 1.5682.81 ± 0.0089.58 ± 0.9087.50 ± 1.5683.85 ± 0.9083.85 ± 2.3986.98 ± 2.3981.77 ± 0.90
PI-Mem — 95% CI[81.09, 85.57][80.05, 84.53][75.81, 83.57][82.81, 82.81][87.34, 91.82][83.62, 91.38][81.61, 86.10][77.93, 89.78][81.05, 92.91][79.53, 84.01]

LongBench v2: Qwen3.5, three runs on the same 503 questions

MethodMean ± Std (%)95% CI
MemAgent52.29 ± 1.19[49.32, 55.25]
PI-Mem54.27 ± 0.53[52.97, 55.58]

The Qwen2.5 HQA results show a consistent advantage across repeated decoding: PI-Mem averages 84.17 ± 0.09, compared with 78.12 ± 0.95 for MemAgent. At 3.6M, the corresponding scores are 81.77 ± 0.90 and 73.96 ± 0.90. The reported Qwen3.5 and LongBench v2 summaries also favor PI-Mem.

Independent training runs. To separately examine training variability, we conducted two independent Qwen2.5-7B PI-Mem training runs with different seeds. Because repeating full RL training is expensive, both runs were evaluated at step 80, whereas the Qwen2.5 PI-Mem checkpoint reported in the paper was trained to step 240 (Appendix A, Table 5). We evaluated the step-80 models on all eight HQA lengths from 7K to 896K.

Training runReward (step 0)Reward (step 80)Reward gainHQA 7K–896K (step 80)
Run10.5350.810+0.27569.92
Run20.5510.799+0.24869.34

Both training runs show substantial reward improvements and similar downstream HQA scores at the same training step.

AuthorsOpenReview ↗

Response to Reviewer 4hb6 (Part 2)

W2/Q2: Single-query latency, GPU scaling, memory, and compute

Thank you for asking whether the latency advantage persists with fewer GPUs and for raising the concern about computational cost. PI-Mem does require more inference FLOPs than MemAgent; its benefit is improved accuracy and lower latency under the evaluated hardware configurations.

Existing full-workload measurements. Appendix B.3, Table 11 already reports estimated algorithmic FLOPs and measured end-to-end latency for the complete 64-question HQA evaluation sets. At 3.6M, using eight H200 GPUs with TP=2, the results are:

MethodQwen3.5 latency (s)Qwen3.5 PFLOPsQwen2.5 latency (s)Qwen2.5 PFLOPs
Vanilla33,365.1471,573.3538,653.00175,515.76
MemAgent12,060.712,054.655,535.504,462.91
PI-Mem1,974.653,365.352,682.487,633.65

PI-Mem performs more FLOPs than MemAgent but completes the workload sooner by parallelizing chunk processing. These are full-set completion times, not single-query response times. The FLOPs estimates include all workflow stages and executed turns, using the paper's convention of two FLOPs per multiply–accumulate operation.

Single-query measurements. We use the same 3.6M HQA query for both methods and both backbones. TP is fixed at 1, and DP denotes the number of independently loaded model replicas, equal to the GPU count. Measurements follow warmup and cover the complete workflow, excluding model loading. We report latency as Mean ± Std over three repeated runs, estimated PFLOPs per query, and the mean NVML total peak memory across the reported runs.

Qwen2.5

MethodDP / GPUsLatency (s)PFLOPs / queryNVML total peak (GiB)
MemAgent11880.95 ± 21.8364.06120.87
MemAgent2*1902.86 ± 12.4563.98241.36
MemAgent4*1889.77 ± 10.1364.57485.09
MemAgent8*1894.28 ± 16.3764.22973.98
PI-Mem1195.83 ± 0.21112.07121.80
PI-Mem2104.55 ± 1.04111.50243.17
PI-Mem460.43 ± 1.06111.71486.67
PI-Mem839.51 ± 0.34111.55974.26

Qwen3.5

MethodDP / GPUsLatency (s)PFLOPs / queryNVML total peak (GiB)
MemAgent15046.47 ± 10.9439.66122.89
MemAgent2*5054.46 ± 12.3838.91243.34
MemAgent4*5048.92 ± 3.1039.27489.98
MemAgent8*5057.09 ± 6.1138.98979.15
PI-Mem1152.87 ± 3.9948.11122.82
PI-Mem280.66 ± 3.9347.72244.74
PI-Mem448.74 ± 0.4947.56491.99
PI-Mem835.74 ± 4.1447.64979.71

*For MemAgent, increasing DP does not parallelize the recurrent chain of a single query. Only one replica can advance that chain at a time, leaving the additional replicas idle with respect to useful concurrent work on this query. Consequently, extra DP replicas do not provide a systematic latency reduction.

PI-Mem is already faster in the single-GPU setting: independent chunk calls can be batched within one replica. Adding replicas further reduces latency, from 195.83 to 39.51 seconds for Qwen2.5 and from 152.87 to 35.74 seconds for Qwen3.5 when scaling from one to eight GPUs. The estimated computation per query remains approximately constant across DP configurations; the reduced time comes from executing that work with greater concurrency.

Peak GPU memory. NVML total peak measures GPU-memory occupancy across all deployed replicas, not the query's live KV-cache usage alone. vLLM preallocates a KV-cache pool, and loaded replicas retain weights, cache allocations, and buffers even when idle. This explains why the total footprint grows approximately with the replica count, including for the starred MemAgent configurations. No out-of-memory errors occurred in the experiments reported in our paper.

These measurements separate total computation from response latency: PI-Mem trades additional computation for parallel execution and lower latency, with a benefit observed here even on one GPU.

AuthorsOpenReview ↗

Response to Reviewer 4hb6 (Part 3)

Q3: Observation counts and the merge-input budget

Thank you for pointing out this potential limitation of the merge step. We examined the post-RL HQA trajectories at all ten lengths, with 64 questions at each length, to quantify the observations produced and the input passed to the merge call.

The table counts chunk responses reporting <check>yes</check> over all executed turns, with the mean per question in parentheses. These are cumulative per-question counts, not per-turn averages. Merge-input lengths include the question, previous memory, retained observations, and the complete prompt formatting.

DocumentsApprox. lengthQwen2.5 chunks reporting yes (total / per question)Qwen3.5 chunks reporting yes (total / per question)Qwen2.5 max. merge inputQwen3.5 max. merge input
507K131 (2.05)65 (1.02)7324,547
10014K155 (2.42)65 (1.02)8634,395
20028K189 (2.95)120 (1.88)1,03112,720
40056K243 (3.80)192 (3.00)1,12713,119
800112K278 (4.34)260 (4.06)1,66517,904
1600224K319 (4.98)269 (4.20)1,92917,647
3200448K409 (6.39)325 (5.08)5,28734,896
6400896K630 (9.84)341 (5.33)17,45144,165
128001.8M855 (13.36)398 (6.22)10,87278,413
256003.6M1503 (23.48)526 (8.22)22,29180,055

Observed input lengths. Qwen2.5's largest recoverable merge input contains 22,291 tokens. With the 1,024-token output allowance, the total is 23,315 tokens, within a 32K budget. For Qwen3.5, the largest estimated input is 80,055 tokens, or 84,151 including its 4,096-token output allowance, below the 262,144-token context window. We found no evidence of merge-context overflow in these evaluated trajectories, including the 3.6M setting.

Although the evaluated trajectories showed no overflow, we acknowledge your concern about larger merge inputs and propose the following extension.

Handling merge-input overflow within the existing workflow. If the selected observations exceed the merge-input budget, we can introduce a length check before the existing merge call. We keep the observations in their original chunk order and remove them from the end until the total merge-input length no longer exceeds the budget. Only the current merge input is shortened, while all source chunks remain available for rereading in the next turn. PI-Mem's existing multi-turn mechanism can therefore incorporate evidence progressively. In the next turn, the model is expected to return <check>no</check> for chunks whose relevant evidence is already covered by memory and <check>yes</check> for those containing complementary evidence not yet incorporated into memory.

Reviewer_1gEHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets two problems of recurrent-memory agents: later chunks can overwrite early evidence, and the chunk-by-chunk dependency makes latency grow linearly with the number of chunks. PI-Mem replaces the single sequential pass with a few "read-select-merge" turns. In each turn, every chunk is read in parallel against the same global memory and emits a flag. The "yes" observations are merged into a new memory, and the loop stops when no chunk says "yes" or after KK turns. The whole workflow is trained with GRPO using one trajectory-level reward: answer accuracy plus a turn-efficiency bonus λturn(K−k)/(K−1)\lambda_{turn}(K-k)/(K-1). On RULER-HQA up to 3.6M tokens, RULER OOD tasks up to 1M, and LongBench v2, PI-Mem beats MemAgent with both Qwen3.5-35B-A3B and Qwen2.5-7B, and it is 6.1× / 2.1× faster at 3.6M tokens.

Critical strengths
  1. A simple, new decomposition of long-context memory

Reading in parallel against a shared memory, then iterating, cuts the critical path from O(N)O(N) to O(K)O(K) while still letting chunks exchange information through the memory between turns. The adaptive exit through the check signal reuses the same output, so it costs nothing extra. Few memory designs remove the serial dependency without giving up cross-chunk reasoning, so this matters.

  1. Consistent gains, strongest at the longest contexts and outside the training distribution

On Qwen3.5 with the same training data and hyperparameters as MemAgent, the HQA average rises from 76.56 to 81.25, and RULER OOD at 1M from 83.86 to 96.88. Training uses only synthetic HQA at 140K tokens, yet the gains carry over to 3.6M tokens and to the 11 OOD RULER tasks. This makes it less likely that RL simply overfits the QA format.

  1. The mechanism is supported directly, not only by end accuracy.

Table 4 shows final-memory coverage on MV-NIAH (MemAgent 58.6% vs. PI-Mem 100% at 1M). The paired case studies show the overwrite failure concretely. Ablations cover select/merge, KK, the turn reward, and chunk/memory size.

Critical weaknesses
  1. It is unclear what RL actually improves

One scalar reward, based only on the final answer and the turn count, is broadcast to every read, merge, and final call. As a result, the paper cannot tell whether RL improves evidence selection and memory construction or mainly the final QA call. This matters because the "+RL" gain is larger than the workflow gain on Qwen2.5: PI-Mem w/o RL (42.30) is below MemAgent w/o RL (44.53), and RL then lifts it to 84.06 (Table 7). The OOD transfer (Table 8) partly answers this concern, but it is indirect.

  1. Order dependency inside a document is dropped by design

In real documents, earlier text often conditions how later text should be read. PI-Mem reads chunks independently within a turn, and the ReadCall/MergeCall templates carry no chunk index. The merged memory therefore cannot tell which fact came later. Iteration restores cross-chunk information flow, but not order, and most samples stop after 2 turns (App. C.1: 75.94% under K=5K=5). HQA and RULER are largely order-insensitive, so they cannot reveal this limitation. Tasks where later information supersedes earlier information (state tracking, knowledge updates, evolving user preferences) would.

  1. The stopping rule and the memory budget are underspecified

The loop stops only if all NN chunks output no. At 3.6M tokens that is 240 chunks (Qwen3.5) or 720 chunks (Qwen2.5), so one false yes forces a full extra reread. Turn 1 almost always finds evidence, so turn 2 is in practice a confirmation pass that rereads everything. The memory is free-form text with no schema (App. D). Its 4K size appears only in App. C.2 and seems to be just the output-length cap. The merge input $|O^{(k)}|\times$4,096 tokens could exceed the context window when many chunks say yes, and the paper does not say how this case is handled.

Questions for authors

Please see the weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1gEH (Part 1)

Dear reviewer 1gEH,

Thank you for taking the time to carefully review our paper and for your constructive feedback. Below, we respond to your three concerns with stage-level diagnostics of RL, additional experiments on conflicting factual updates, and an audit of the stopping behavior and merge-input budget.

W1: What does RL improve?

We appreciate your suggestion to examine what RL improves beyond the final answer. To address this concern, we provide stage-level diagnostics below. Our training uses a trajectory-level reward to optimize the entire workflow. To examine where improvements occur, we compare intermediate outputs and final answers for the same questions before and after RL.

Controlled tasks enable quantitative stage-level diagnostics. We use two RULER tasks with explicitly identifiable target values, allowing quantitative diagnostics across questions without an additional LLM judge. Multi-key NIAH-3 (MK3) asks for a target UUID, making it suitable for tracking whether an extracted target survives selection and memory construction. Multi-query NIAH (MQ) has four target values per question, allowing us to measure the completeness of multi-target extraction and integration rather than only whole-question correctness.

Target coverage tracks evidence across workflow stages. We define target coverage as the fraction of gold target values present in the output at a given stage, counting each target only once per question. First-turn read coverage uses the union of all chunk outputs; post-selection coverage uses only outputs actually passed to merge; merged-memory coverage measures the first merged memory; final memory coverage is measured immediately before the final QA call. End-to-end Sub-EM is the original evaluation score.

For each model and task, we pair the pre-/post-RL trajectories across all eight lengths (8K–1M), with 64 questions per length: 512 question instances, containing 512 target UUIDs for MK3 or 2,048 target values for MQ. Lengths are equally weighted. The tables report percentages, with arrows indicating before → after RL.

MQ: all-length average

Metric (%)Qwen2.5: before → after RLQwen3.5: before → after RL
First-turn read coverage97.07 → 98.8395.61 → 99.41
First-turn post-selection coverage95.21 → 98.8395.56 → 99.41
First-turn merged-memory coverage94.19 → 98.8395.56 → 99.37
Final memory coverage (before QA)98.49 → 99.4699.12 → 100.00
End-to-end Sub-EM98.14 → 98.9794.58 → 100.00

MK3: all-length average

Metric (%)Qwen2.5: before → after RLQwen3.5: before → after RL
First-turn read coverage97.85 → 97.6699.80 → 100.00
First-turn post-selection coverage97.85 → 97.6699.80 → 100.00
First-turn merged-memory coverage76.56 → 97.4699.61 → 99.02
Final memory coverage (before QA)86.13 → 98.2499.80 → 100.00
End-to-end Sub-EM60.16 → 97.8599.41 → 100.00

These results provide three complementary observations:

  1. Evidence extraction improves before any merge or final answer. On MQ, first-turn read coverage increases for both models, showing improved evidence extraction before any merging or final QA takes place.
  2. Evidence retention and memory construction also improve. On Qwen2.5 MK3, first-turn read and post-selection coverage remain nearly unchanged, while merged-memory coverage increases from 76.56% to 97.46% and final memory coverage from 86.13% to 98.24%.
  3. RL can help the model use stored evidence to generate the final answer. For Qwen3.5 MQ, pre-RL final memory coverage is already 99.12%, but end-to-end Sub-EM is 94.58%; both reach 100% after RL. This suggests that RL improves not only evidence extraction and memory construction, but also the use of stored evidence to generate the final answer.

Regarding the backbone difference in Table 7, we believe Qwen3.5’s stronger instruction-following and evidence-processing capabilities enable it to execute the more complex PI-Mem workflow more effectively. Qwen2.5’s training-free PI-Mem performance falls below MemAgent’s, and it relies more heavily on RL to improve workflow execution. Although Qwen3.5 already approaches the ceiling on several diagnostic metrics before RL, training further improves its evidence processing and final-answer performance, helping it execute the workflow more reliably.

Overall, these controlled diagnostics support our focused claim: trajectory-level RL improves observable behavior at multiple stages, not only the final answer. MK3 and MQ provide explicit targets that make these stage-level changes quantitatively traceable; attributing improvements to individual stages in open-ended tasks is more difficult because intermediate evidence can take many valid forms.

AuthorsOpenReview ↗

Response to Reviewer 1gEH (Part 2)

W2: Order dependence and later facts superseding earlier ones

Thank you for highlighting the importance of document order and valid factual updates. We agree that retrieval-focused evaluations alone are insufficient to address this concern. We provide an implementation clarification and an additional conflict-resolution evaluation below.

Parallel execution does not imply shuffled evidence. In our actual implementation, selected chunk outputs are assembled in their original relative order and labeled as [Section 1], [Section 2], etc., before merging. These labels are assigned after selection: they preserve the relative order of retained outputs, but are not absolute original chunk indices or timestamps. Thank you for pointing this out: we omitted this formatting detail when presenting the prompt template in the paper. We will update the MergeCall template in Appendix D to explicitly show these section labels.

More importantly, our goal is to reduce the destructive overwriting of relevant evidence by irrelevant later content, not to prevent valid updates. The merge call regenerates memory from the previous memory and newly selected observations, allowing information to be revised or replaced rather than only appended.

Additional benchmark evaluation. We evaluate the FactConsolidation task from MemoryAgentBench [1], which contains conflicting facts with explicit identifiers: higher-numbered facts supersede lower-numbered ones. Answering correctly therefore requires resolving updates, not merely retaining more facts. Single-hop (SH) questions test direct updated relations, while multi-hop (MH) questions require reasoning over the updated relations. We use both official length settings, 64K and 262K, so the inputs span multiple chunks. We retain the original contexts, fact identifiers, questions, and gold answers, and evaluate each question independently in a query-conditioned QA setting. Each setting contains all 100 questions, giving 400 question instances per method. Answers are evaluated using deterministic Sub-EM, without an LLM judge.

MemAgent and PI-Mem use their respective existing RL step-80 checkpoints, without additional training on FactConsolidation. Vanilla, YaRN, and RAG use the original instruction-tuned weights. The inference configurations for all baselines and PI-Mem largely follow those used for LongBench v2 in Table 3.

FactConsolidation results: Qwen3.5-35B-A3B, Sub-EM (%)

MethodSH 64KSH 262KMH 64KMH 262KAverage
Vanilla95.0066.0030.005.0049.00
YaRN85.0055.0020.003.0040.75
RAG63.0050.0010.001.0031.00
MemAgent87.0080.0038.0019.0056.00
PI-Mem98.0091.0051.0045.0071.25

PI-Mem outperforms MemAgent in all four settings, including multi-hop questions that require using updated facts. This supports the compatibility of parallel reading with conflict resolution: preserving earlier evidence does not require treating outdated evidence as permanently valid.

A representative example. In an MH-64K question asking where Blériot's founder died, a later fact changes the founder to Juliette Gordon Low, while the relevant fact about her death city, Pune, appears earlier in the document. MemAgent updates the founder but fails to recover the earlier death-city information and answers that it is not mentioned. PI-Mem also identifies the updated founder in its first turn, then uses the updated shared memory to reread earlier chunks in the second turn and correctly answers Pune. This illustrates why favoring recent information alone is insufficient: a new fact can change the reasoning path and make previously overlooked evidence relevant. Iterative rereading allows PI-Mem to recover that evidence.

AuthorsOpenReview ↗

Response to Reviewer 1gEH (Part 3)

W3: Stopping rule and memory budget

Thank you for raising these practical concerns. We address the confirmation cost and the potential merge-input bottleneck below.

Confirmation cost and iterative refinement. We agree that confirmation rereads and false-positive check signals can add inference FLOPs. Our reported end-to-end latency includes these costs and remains substantially lower than that of Vanilla and MemAgent at ultra-long context lengths under the same evaluation setup (Appendix B.3, Table 11). This reflects a trade-off between additional computation and greater parallelism. Subsequent turns also support useful refinement: in Appendix C.1, Table 12, Qwen2.5's HQA score at 3.6M increases from 70.31 with K=1K=1 to 81.25 with K=3K=3.

Merge-input budget and scalability. We acknowledge that the model's context window is a genuine capacity limit: if many chunks return long relevant observations, their combined merge input can exceed the available budget. To examine whether this occurs in our experiments, we audit the existing post-RL HQA trajectories across all ten lengths, up to 3.6M tokens, with 64 questions per length.

The table reports the number of chunk responses beginning with <check>yes</check>, summed across all executed turns, with the per-question mean in parentheses. Merge-input lengths are in tokens and include the prompt, previous memory, selected observations, and chat template.

DocumentsApprox. lengthQwen2.5 chunks reporting yes (total / per question)Qwen3.5 chunks reporting yes (total / per question)Qwen2.5 max. merge inputQwen3.5 max. merge input
507K131 (2.05)65 (1.02)7324,547
10014K155 (2.42)65 (1.02)8634,395
20028K189 (2.95)120 (1.88)1,03112,720
40056K243 (3.80)192 (3.00)1,12713,119
800112K278 (4.34)260 (4.06)1,66517,904
1600224K319 (4.98)269 (4.20)1,92917,647
3200448K409 (6.39)325 (5.08)5,28734,896
6400896K630 (9.84)341 (5.33)17,45144,165
128001.8M855 (13.36)398 (6.22)10,87278,413
256003.6M1503 (23.48)526 (8.22)22,29180,055

The largest recoverable Qwen2.5 merge input is 22,291 tokens, or 23,315 after reserving 1,024 output tokens, which fits within even a 32K context budget. For Qwen3.5, the largest estimated input is 80,055 tokens; reserving 4,096 output tokens gives 84,151, below the model's 262,144-token context window. We did not observe merge-context overflow in the audited experiments.

Although we did not observe this issue in our experiments, we agree that it is a valid concern and propose the following extension.

Handling merge-input overflow within the existing workflow. If the selected observations exceed the merge-input budget, we can add a length check before the existing merge call. We keep observations in their original chunk order and remove observations from the end until the complete merge input no longer exceeds the input budget. Only the current merge input is shortened, while all source chunks remain available for rereading in the next turn. This allows PI-Mem's existing multi-turn mechanism to incorporate evidence progressively. In the next turn, the model is expected to return <check>no</check> for chunks whose relevant evidence is already covered and <check>yes</check> for those containing complementary evidence not yet incorporated into memory.

References

[1] Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions. International Conference on Learning Representations (ICLR), 2026.

Comments on the paper

1 comment

We thank all reviewers for their encouraging assessments and constructive feedback, and the chairs for their time and consideration. We appreciate the positive evaluations of our work and the suggestions for strengthening its analysis and experimental validation. During the rebuttal period, we provided point-by-point responses, conducted additional experiments, and revised the manuscript. Below, we summarize the main responses and revisions for the reviewers. Changes in the revised manuscript are highlighted in blue.

  1. What RL improves (Reviewer 1gEH, W1). We provide paired, stage-level diagnostics on MK3 and MQ across all eight RULER lengths. These controlled tasks have explicitly identifiable targets, enabling quantitative coverage measurements without an additional LLM judge. The results show improvements in evidence extraction, memory construction, and the use of stored evidence for final answers, supporting benefits beyond the final QA call alone.

  2. Conflicting factual updates and evidence order (Reviewer 1gEH, W2). We add FactConsolidation from MemoryAgentBench, where newer facts supersede conflicting older ones. PI-Mem outperforms MemAgent in all four settings, both with and without RL, extending the evaluation beyond static evidence retention. The complete comparison with Vanilla, YaRN, and RAG is included in Appendix B.2, Table 10. We also clarify that selected observations are assembled in their original relative order and update the MergeCall prompt template with section labels matching the implementation (Appendix E).

  3. Merge-input capacity (Reviewer 1gEH, W3; Reviewer 4hb6, Q3). We audit post-RL HQA trajectories at all ten lengths, reporting cumulative check-yes counts and recovered or estimated merge-input lengths. The audit found no evidence of merge-context overflow, including at 3.6M tokens. We also describe a possible length-check extension that shortens only the current merge input while retaining all source chunks for subsequent rereading (Appendix D, Table 15).

  4. Evaluation and training variability (Reviewer 4hb6, W1/Q1). Our response reports repeated HQA and LongBench v2 evaluations with means, standard deviations, and 95% confidence intervals. We separately examine training robustness through two independent Qwen2.5 training runs. These results support evaluation stability and training robustness.

  5. Single-query latency, GPU scaling, and computational cost (Reviewer 4hb6, W2/Q2; Reviewer iMWT, W3). In our rebuttal, we supplement the full-workload results already reported in our original submission (Appendix B.3) with single-query latency, estimated FLOPs, and peak GPU-memory measurements across GPU configurations. PI-Mem retains a latency advantage even on one GPU and benefits from additional replicas. We also quantify and acknowledge its longer training time than MemAgent in our Qwen3.5 experiments. We clarify in the revised manuscript that, despite higher FLOPs than MemAgent, PI-Mem achieves lower end-to-end latency through parallel chunk processing, offering a favorable accuracy–efficiency trade-off.

  6. Transfer beyond Qwen (Reviewer iMWT, W1). Our response adds training-free HQA evaluation on Gemma 4 E4B, supporting the applicability of PI-Mem beyond Qwen without backbone-specific RL training.

  7. Sensitivity to the RL method (Reviewer iMWT, W2). We additionally train Qwen2.5 PI-Mem with standard GRPO and Dr. GRPO configurations. All tested configurations show substantial reward improvements and similar downstream HQA performance, supporting robustness across the evaluated RL configurations. The configurations and results are reported in our response to Reviewer iMWT.

We are pleased to have addressed all concerns raised by the reviewers during the rebuttal. All three reviewers have maintained their scores of 3 (Weak Acceptance). We again thank the reviewers for their encouraging evaluations and constructive suggestions, and the chairs for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.