acceptodds
Under review as a conference paper at ICLR 2027

Task-Oriented Memory Compilation: Executable State Representations for Long-Context Language Models

Abstract

Answering questions over long interaction histories often requires tracking updated facts, following relations, and aggregating scattered evidence. Retrieval and compression methods supply selected text, which can leave these operations to the answering model, and typically rely on auxiliary neural models to produce it. We propose Task-Oriented Memory Compilation (TOMC), which instead performs these operations while building memory. TOMC computes state, relation, or count records from the history and packs them with selected source text under a fixed token budget. It is training-free, runs on CPU, and supplies plain text to black-box LLM APIs. On the BEAM long-term memory benchmark, with 100K–1M-token histories and three strong LLMs as API readers (GPT-5.1, Rednote preview, and DeepSeek 4.1 Flash), TOMC achieves higher overall mean scores than six retrieval, memory, and compression baselines. It also uses 32.4–36.6% fewer input tokens than the strongest baseline, LIGHT, and 11–17× less construction time than per-call neural compressors. At 500K and 1M tokens, TOMC scores higher in all 36 reader–baseline–length comparisons; its largest gains over BEAM-RAG and LIGHT are in temporal reasoning, which ablation links to the records. These results suggest that executing task operations during memory construction complements retrieval and offers an alternative to token compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-context language models, long-term conversational memory, memory construction, context compression, retrieval-augmented generation, state tracking, training-free methods

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X3sSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TOMC (Task-Oriented Memory Compilation) is a training-free, CPU-only method that executes task operations while building memory for long-context QA and passes plain text to black-box LLM API readers. Given a history, question, and token budget, it segments the history into line units, selects evidence with fixed lexical rules, routes the question to one of STATE / RELATION / COUNT / RAW, executes the corresponding operation (an ordered SET/COPY state machine, typed relation-path joins, or term counts), and packs the compiled records with selected source text under the budget (records receive αB, α=0.42). On BEAM (100K–1M tokens) with three recent API readers (GPT-5.1, Rednote preview, DeepSeek 4.1 Flash), TOMC reports higher paired mean scores than six retrieval/memory/compression baselines in all 36 reader–baseline–length comparisons at 500K and 1M, uses 32.4–36.6% fewer input tokens than the strongest baseline (LIGHT), and builds memory in 4.21s/call on CPU with no GPU or auxiliary neural model (11–17× faster than per-call neural compressors). The validation includes BEAM answer-quality tables, RULER diagnostics (NIAH/VT/QA2), a BEAM compilation ablation, and construction-cost measurements.

Critical strengths

1.A genuine, reproducible systems contribution. TOMC needs no training, GPU, or auxiliary neural inference; construction is 4.21s/call on CPU versus 47.94–72.07s for per-call neural compressors, with a text-only interface to black-box APIs. Because it is a deterministic rule engine rather than learned weights, the released reference implementation and frozen per-question outputs make the reported numbers independently reproducible offline. This is consequential because it is a practical, verifiable alternative to expensive neural memory pipelines (e.g., LIGHT's 32B writer/filter), and it is the part of the paper most likely to be useful to the community.

2.A clearly isolated positive result on state tracking. On the RULER VT diagnostic, the STATE route reaches 100% answer-item recall for all three readers versus 23.8–58.4% for query selection at 89.2% fewer tokens, and compiling only selected evidence drops recall to ~20% — correctly attributing the gain to executing the update chain over the full history. This is consequential because it cleanly demonstrates the one setting where the core idea provides a large, well-explained benefit.

Critical weaknesses

1.The headline contribution — compiled records — is not statistically supported on the aggregate benchmark. The BEAM compilation ablation (Appendix F) shows restoring compiled records changes overall scores by only +1.20 / −0.91 / +3.01 points across the three readers (negative for Rednote preview), with Holm-adjusted p=0.575, and the benefit is concentrated almost entirely in temporal reasoning (+8.33/+7.50/+4.86). Meanwhile TOMC's margins over the strong baselines (LIGHT, BEAM-RAG) are only ~7–12% overall and near zero at 100K, with the large gains coming against weaker neural compressors. Together this indicates that most of TOMC's advantage comes from retained-source selection and its long-history cost/robustness profile, not from the "compiled records" the framing centers on. This is consequential because the paper's main claim ("executing task operations during memory construction improves answer quality") is broader than the evidence supports; the claim would be sound if rescoped to temporal/state tracking.

2.A published same-line method is not empirically compared, and the key claim is not exclusive. APEX-MEM (ACL 2026), which also builds structured conversational memory and performs temporal/relational operations, is cited but only contrasted in prose. Concurrently, executable-memory work (User-as-Code, arXiv Jun 2026) independently reports the same "execution beats retrieval" effect (6–43% vs ~99% on aggregation questions). This is consequential because, without a comparison to an already-published structured-memory baseline, the paper's positioning and novelty relative to the structured-operations line remain unestablished. (In fairness, TEM, arXiv Oct 2026, post-dates submission and is not a required comparison.)

Questions for authors

1.Clarification affecting assessment. Given Appendix F (p=0.575; benefit concentrated in temporal reasoning, negative for one reader), can you show a statistically significant compiled-records gain under a single shared question set and judge, and state explicitly which abilities the claim should be scoped to? 2.Clarification affecting assessment. Can you add an empirical comparison to APEX-MEM (ACL 2026) on BEAM/RULER, or justify why a structured/temporal-memory system is not an appropriate baseline? A brief positioning against executable-memory approaches (e.g., User-as-Code) would also help. 3.Clarification affecting assessment. Can you report Table 1 under one shared valid-question set and one judge per length tier, so that absolute scores and the "all 36 comparisons" statement are comparable across rows?

2 comments
AuthorsOpenReview ↗

Response to Reviewer X3sS (Part 1 of 2: W1 & Q1)

Thank you very much for your constructive review and thoughtful questions, and for recognizing TOMC's CPU-only efficiency and state-tracking design. Your comments directly help make our presentation much clearer, sharpen our attribution, and make our experimental validation more solid, while also guiding future extensions of our work. Below we address W1 & Q1 in Part 1 and W2, Q2, & Q3 in Part 2.

Regarding W1 & Q1: Scoping the Compilation Claim and Controlled Attribution

For W1 and Q1, we agree that our submission stated the compilation claim too broadly: we will narrow our compilation claim in the manuscript specifically to temporal/state tracking.

Thanks for your comment; separating these factors makes our attribution much clearer and directly informs our future work. First, to isolate compilation from retrieval across all 600 BEAM questions under the 8,192-token cap and one blind judge per reader, we hold the retrieved BM25 candidate pool identical and compare Raw Pool, zero-API CPU Rules, and an LLM Writer calling DeepSeek 4.1 Flash:

Same-Pool Arm (600q)Extra Writer API CostDeepSeek 4.1 Flash Reader (All; 9-Abil.; 1M)GPT-5.1 Reader (All; 9-Abil.; 1M)
Raw PoolUSD 0.00 (0 writer tok, 0 calls)53.63; 56.72; 47.4153.55; 57.57; 48.42
CPU RulesUSD 0.00 (0 writer tok, 0 calls)54.68; 58.07; 50.4053.98; 58.22; 50.83
LLM Writer (DeepSeek 4.1 Flash)+USD 2.26 (+5.73M writer tok, 5.94 s/q)57.85; 58.24; 51.7556.45; 57.49; 50.63

From this table, when candidate evidence is held fixed, zero-cost CPU Rules improves million-token histories and target state/temporal slices such as Multi-Session and Temporal Reasoning over Raw Pool. Meanwhile, pairing DeepSeek 4.1 Flash with a DeepSeek 4.1 Flash writer benefits from homogeneous two-agent alignment, whereas transferring its synthesized records to GPT-5.1 yields no statistically significant overall gain over CPU Rules: LLM Writer leads mainly on the 60 Contradiction items, while zero-cost CPU Rules preserves verbatim lines and timestamps, matching or edging it out across the nine non-contradiction abilities pooled and at 1M on GPT-5.1, especially on Event Ordering and Temporal Reasoning.

Second, to test compiled records directly on the same 600 questions under one blind judge per reader, identical neutral instructions, and identical token length (mean 5,192.47 cl100k tokens), we compare:

  • TOMC (Neutral): compiled records + selected source text;
  • Lexical (no-record control): selected source text + equal-token extra BM25 lines replacing the records (358 record-bearing questions).
Reader (N = 600)TOMC (Neutral)LexicalDiff.Key Subset Gain (TOMC (Neutral) vs. Lexical)
GPT-6 Luna58.4856.75+1.73+2.27 on compiled (432q, p = 0.044); +5.00 Temporal
Rednote preview56.7556.39+0.36+1.51 on replaced (358q); +1.38 at 1M; +5.98 Info Extr.
GPT-5.162.1162.13-0.02+1.29 at 100K; +3.75 Preference; +1.88 Contradiction
DeepSeek 4.1 Flash62.6163.05-0.44+3.75 Contradiction; +3.33 Knowledge Update

From the table, at matched length and neutral framing, TOMC (Neutral) improves GPT-6 Luna's compiled and Temporal subsets and lifts record-replaced items on Rednote, while tying overall on GPT-5.1 and DeepSeek 4.1 Flash when removed records are refilled with extra BM25 lines, confirming why narrowing our compilation claim is warranted; we will revise the paper accordingly.

AuthorsOpenReview ↗

Response to Reviewer X3sS (Part 2 of 2: W2, Q2, & Q3)

Regarding W2 & Q2: APEX-MEM and Head-to-Head Comparison with User-as-Code

For W2 and Q2, APEX-MEM (ACL 2026) has not released public code, so we will clarify its design in the related work section and compare TOMC head-to-head against open-source executable memory User-as-Code (UaC), where TOMC outperforms UaC on BEAM and edges it out on LoCoMo.

First, on our much longer 100K–1M-token BEAM benchmark (600 questions) under the same 8,192-token budget and one blind rubric judge, we compare UaC and TOMC across two readers:

Reader (BEAM, N = 600)UaCTOMCOverall GainKey Ability Gains (TOMC over UaC)
DeepSeek 4.1 Flash51.4061.29+9.89 (p = 0.00006)Temporal +31.39, Info Extr. +22.61, Multi-Session +16.44
GPT-5.1 (DeepSeek 4.1 Flash-built UaC)51.0960.21+9.12 (p = 0.00006)Temporal +33.75, Info Extr. +24.16, Multi-Session +16.69

Second, without retuning any rules or budgets, we evaluate TOMC against UaC, LIGHT, and BEAM-RAG on the shorter 2024 LoCoMo benchmark (~16K–26K tokens per chat; 1,540 questions) with DeepSeek 4.1 Flash using official judge-free lexical token F1 x 100 (where full-sentence answers against concise gold spans score in the 20–30 range):

Reader (LoCoMo 2024, N = 1,540)BEAM-RAGLIGHT (DeepSeek 4.1 Flash)UaC (June 2026)TOMCRank
DeepSeek 4.1 Flash24.1126.0128.2828.62Ranks 1st

This length contrast is revealing: on shorter 2024 LoCoMo chats (~16K–26K tokens), zero-LLM TOMC edges out UaC (28.62 vs. 28.28) and beats LIGHT (26.01) and BEAM-RAG (24.11), whereas on much longer 100K–1M BEAM histories, TOMC outperforms UaC by a wide margin (+9.89 and +9.12 overall, p = 0.00006; +31 to +34 on Temporal, +16 to +17 on Multi-Session) because UaC synthesizes Python state over bounded session windows via LLM calls while TOMC combines full-history CPU reductions with budgeted verbatim text. Thank you for suggesting User-as-Code; we will add this comparison to the paper.

Regarding Q3: Table 1 Under One Shared Question Set and One Unified Blind 7-Method Judge

For Q3, yes: scoring all seven methods jointly in one blind 7-way rubric panel per (reader, question) under unified Gemini 3.8 Flash on the strict 561/600 3-reader common set (including 0-empty repaired 1M LLMLingua-2-D) gives every method one standalone score per cell, where TOMC wins all 36/36 long-history (500K–1M) comparisons (mean 59.30 vs. 53.74 for LIGHT [+5.56] and 51.83 for BEAM-RAG [+7.47]) and matches Table 1's overall win rate.

Scoring all seven anonymized, shuffled candidates (A-G) together (0, 0.5, or 1 per official rubric criterion, averaged x 100) removes pairwise drift so absolute scores are directly comparable across all rows on the exact same shared valid set (185/200 at 100K, 186/200 at 500K, 190/200 at 1M):

Method (7-Way, 500K/1M)500K GPT-5.1500K Rednote500K DeepSeek 4.1 Flash1M GPT-5.11M Rednote1M DeepSeek 4.1 Flash500K–1M Mean
TOMC62.8257.0661.5658.6756.8858.8459.30 (36/36)
LIGHT (2026)50.3553.7856.4555.8051.3554.7553.74 (+5.56)
BEAM-RAG (2026)52.8149.9252.2452.8151.9151.3251.83 (+7.47)
BRIEF-Pro (2026)41.6038.4440.4140.9642.2538.8040.41 (+18.89)
LongLLMLingua38.0837.0937.5541.8641.4740.5939.44 (+19.86)
LLMLingua-2-H37.6636.1935.3743.1338.8638.3638.26 (+21.04)
LLMLingua-2-D (0-empty 1M)36.5932.9633.7138.6937.9334.9935.81 (+23.49)

Thank you again for your time and thorough review, which substantially strengthens our paper; as a zero-GPU CPU module, our method could also be directly deployed with modern frontier LLM APIs! If our clarifications and new experiments address your concerns, we kindly hope you will consider raising your score.

Reviewer_cahFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the topic of context management for long-context language modeling. Broadly, they motivate their work by recounting how language modeling tasks are increasingly requiring long contexts and that the manner in which this history is managed affects downstream task success. They develop a method: Task-Oriented Memory Compilation which uses a predefined rule set to compute “task-specific” records. By providing these task-specific records to an LLM API, they improve performance. They evaluate TOMC against 6 context management baselines, finding that their method is superior. TOMC is lightweight, and does not require expensive operations (e.g., additional inference calls or embeddings).

Critical strengths

(I do have a point of confusion around how the results reported in Table 1 and Figure 2A are supposed to be interpreted with respect to the baselines. But I will enumerate strengths based on my understanding of the results. Please correct my understanding in the weaknesses section.)

  1. The TOMC method consistently boosts performance across models, baseline context management methods, and tasks.

  2. TOMC is a lightweight and efficient method (cpu only) which manages long-context without a large overhead cost (compared to the baseline methods).

The above two strengths mean that TOMC enables a good resource vs. performance tradeoff for long context tasks (although it would be good to plot resource usage vs. performance curves).

Critical weaknesses

The main weakness is that I am having a hard time understanding how TOMC was compared to the 6 context management baselines: I cannot tell if TOMC is deployed on top of the baselines or compared directly to these baselines. I think in Table 1, each baseline (e.g., LIGHT) is evaluated with and without TOMC where the numerator is the baseline performance and the denominator is baseline+TOMC. I have a similar point of confusion with Figure 2A. If my interpretation is correct, then it looks like TOMC does consistently boost the performance of current context management schemes but also should be compared directly to the baseline context management schemes. If my understanding of table 1 is wrong and this is a direct comparison of TOMC to the baselines, I do not understand why the denominator is changing within each subsection.

If the evaluation is done with a baseline w/ and w/o TOMC, then the context management strategy would consume the resources of both the baseline method and TOMC (but TOMC seems pretty cheap).

Questions for authors

Line 92: what is the definition of a “reader”? Is a “reader” the same thing as doing inference with an LLM?

3 comments

Thank you very much for your positive review. Your suggestions make our experimental presentation clearer, sharpen our cost-quality attribution, and guide future extensions.

For W1a, although TOMC is modular and can plug into existing retrieval or memory pipelines, every comparison in Table 1 and Figure 2A is a direct standalone evaluation without building on any baseline.

Each baseline and TOMC start independently from the same raw history under the same budget and pass context directly to the LLM API; we will split Base. / TOMC into separate columns and clarify both captions.

For W1b, whenever two baselines share a single-answer-judged batch and succeeded on the same questions, TOMC's standalone score is identical; shifts across rows come from runtime API rejections on a few items and protocol-specific paired/dated batches (Table 7).

Comparing 100K vs. Overall in the same GPT-5.1 Table 1 batch shows this:

PairTierSucceededBaselineTOMCGain
BEAM-RAG vs. TOMC100K200/20053.5862.26+16.2%
LIGHT vs. TOMC100K200/20060.5062.26+2.9%
BEAM-RAG vs. TOMCOverall598/60052.4759.25+12.9%
LIGHT vs. TOMCOverall599/60053.7559.48+10.7%

In rows 1 and 2, every call succeeded and TOMC's standalone score is identical. In this batch, a few 500K/1M calls hit runtime API content filters (not early stopping). Under early token quotas, we ran head-to-head paired batches (Table 7) rather than one joint pass. Because failures differ by method, we averaged each pair over mutually succeeded items rather than shrinking the shared set (restricting to the shared subset across all methods still gives TOMC 36/36 long-history wins). We will note this beneath Table 1.

For W2, because TOMC runs standalone in all reported comparisons rather than building on a baseline, deploying TOMC incurs only its own CPU cost and zero baseline GPU or model overhead.

Across three strong, recent LLM APIs (GPT-5.1, Aug 2026 Rednote preview, Sep 2026 DeepSeek 4.1 Flash) on 600 BEAM questions:

MethodExtra ModelBuild TimePeak GPUHost RSSTOMC GainInput Diff.
TOMC (Ours)None (CPU)4.21 s/call0 GiB3.75 GiBRef.Ref.
LIGHT (2026)*32B AWQ41.53 h/batch80.82 GiB6.04 GiB+7.0% (+3.71 pts)-34.8%
BEAM-RAG (2026)BGE-small23.94 s/conv.0 GiB1.45 GiB+10.5% (+5.46 pts)+13.0%
BRIEF-Pro (2026)3B LLM47.94 s/call11.81 GiB5.45 GiB+45.7% (+17.02 pts)+275.5%
LongLLMLingua7B LLM72.07 s/call18.50 GiB10.24 GiB+49.4% (+17.90 pts)-9.3%
LLMLingua-2-HBERT10.19 s/conv.1.55 GiB1.62 GiB+43.8% (+17.48 pts)-34.3%
LLMLingua-2-D*BERT9.16 s/conv.1.55 GiB1.85 GiB+63.8% (+22.28 pts)+7.9%*

(LIGHT*: 600-memory batch, 1 worker KV pool; LLMLingua-2-D*: empty 1M memories raise overall ratio, though TOMC uses fewer tokens at 100K and 500K; native stage units.)

From the table, TOMC needs zero GPUs or auxiliary models and runs over an order of magnitude faster per call than LLM compressors, beating all six baselines in accuracy while cutting input tokens by ~35% (32.4-36.6%) vs. LIGHT (2026). Per your suggestion, we will add resource-vs.-performance curves (including BRIEF-Pro) for build cost and input tokens vs. accuracy.

To further strengthen our 2026 comparisons, we also evaluated CPU-only TOMC against User-as-Code (UaC, June 2026) on BEAM and against BEAM-RAG, LIGHT, and UaC on LoCoMo (1,540q, F1 x 100):

BEAM (100K-1M, 600q)UaC (June 2026)TOMCGain
DeepSeek 4.1 Flash51.4061.29+9.89 (p = 0.00006)
GPT-5.1 (DeepSeek 4.1 Flash-built UaC)51.0960.21+9.12 (p = 0.00006)
LoCoMo (2024, 1,540q F1)BEAM-RAG (2026)LIGHT (DeepSeek 4.1 Flash)UaC (June 2026)TOMC
DeepSeek 4.1 Flash24.1126.0128.2828.62 (1st)

For Q1, yes, on Line 92 a "reader" simply means the downstream LLM API performing inference to answer from the prepared context and question.

Memory/RAG papers use "reader" to distinguish the strong answering LLM from upstream memory-building models (which TOMC removes); we will define it at Line 92.

Thank you again for your constructive review, which makes our paper stronger! Because many agent memory frameworks operate at different layers rather than compressing context, our method can also combine with other memory management methods; as a zero-GPU CPU module, our method could also be directly deployed with modern LLM APIs!

Thanks for your response. I don't have further questions. I will maintain my positive score.

Thank you so much for your reply! From both your review and your follow-up, we can clearly feel that you took the time to read our paper and response closely yourself, and that kind of genuine engagement really made our day. Thank you again for your time and support, and we will make sure all the clarifications we promised are cleanly incorporated into the final revision!

Reviewer_2ZXUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Task-Oriented Memory Compilation (TOMC), a training-free method for answering questions over long conversational histories. Its central idea is to execute supported task operations during memory construction, reducing the need for the answering LLM to reconstruct task state from retrieved text. Given a history and a question, TOMC uses predefined rules to select evidence and construct state, relation, or count records, with a raw-text route that skips compilation. It combines these records with selected source text under a fixed token budget. Memory construction runs on CPU without auxiliary neural-model inference, and the resulting textual memory supports black-box LLM readers. The BEAM experiments use a separate implementation with simplified state and relation operators and deterministic prefiltering applied in 380 of the 600 memory constructions (Appendix B.1).

The main evaluation covers 600 BEAM questions over histories of 100K, 500K, and 1M tokens, using GPT-5.1, Rednote preview, and DeepSeek 4.1 Flash. The authors report higher overall scores than six retrieval, memory, and compression baselines, including higher scores in all 36 reader–baseline–length comparisons at 500K and 1M tokens. However, ability-level results averaged over these two tiers show lower scores than BEAM-RAG and LIGHT on instruction following, BEAM-RAG on abstention, and LIGHT on summarization. TOMC uses 32.4–36.6% fewer reader-input tokens than LIGHT and has a reported mean CPU construction time of 4.21 seconds per question.

BEAM compilation ablations show consistent improvements in temporal reasoning but mixed overall effects across readers. Additional diagnostics on three RULER tasks demonstrate successful variable tracking when state records are constructed from the full history and show that retaining source text remains important for key–value retrieval and multi-hop QA. On the latter task, TOMC uses fewer input tokens but scores below query selection, with 40.85 versus 47.49 F1.

Critical strengths
  1. Practical memory construction without auxiliary neural inference. TOMC uses predefined rules to construct memory on CPU and supplies plain text through a model-agnostic reader interface. It requires neither training nor access to reader weights. Table 3 reports a mean construction time of 4.21 s per call on CPU, compared with recorded compression times of 72.07 s and 47.94 s for LongLLMLingua and BRIEF-Pro, which reach peak GPU allocations of 18.50 and 11.81 GiB, respectively. This strengthens my assessment of its practical value because it supports black-box LLM APIs in settings where running an auxiliary neural model is costly or infeasible.

  2. Positive system-level results on long conversational histories. The BEAM evaluation covers histories up to 1M tokens and three API readers. The paired analysis reports statistically significant overall improvements over BEAM-RAG for all three readers and over LIGHT for two of the three (Appendix D.3, Table 8). TOMC also uses 32.4–36.6% fewer reader-input tokens than LIGHT. These findings strengthen my assessment of the complete system because its gains extend across multiple readers and include improvements over competitive retrieval and memory baselines, with reduced reader input relative to LIGHT.

Critical weaknesses
  1. Limited and reader-dependent evidence for the central compilation mechanism. The BEAM compilation ablation changes overall scores by +1.20 points for GPT-5.1 and −0.91 for Rednote preview, both with Holm-adjusted p = 0.575; only DeepSeek Flash shows a significant improvement of +3.01 points (Appendix F). Although temporal reasoning improves for all three readers, these results provide limited evidence of a consistent overall benefit from compilation. For calibration, the 168 raw-text-route memories, which are identical in both arms, differ by +0.69 and −1.02 points for GPT-5.1 and Rednote preview, which the authors attribute to answer and judging variation (Figure 4b). Event-ordering scores also decline for all three readers, although these ability-level differences are descriptive. The ablation removes accompanying instructions and reduces input length, leaving these factors intertwined with the contribution of compiled records. Furthermore, the BEAM implementation omits COPY and typed relation-path joins (Appendix B.1), so its results do not directly validate several operations described in Section 3. This is important because executing task operations during memory construction is the main conceptual contribution, whereas the stronger system-level results do not establish how much of the advantage this mechanism explains.

  2. Insufficient evidence of generalizability and comparative advantage under controlled conditions. The long-conversation evaluation is restricted to BEAM, and RULER uses a separate task-specific implementation. Given the reliance on predefined routing and extraction rules, the absence of rule-coverage and correctness measurements leaves their effectiveness on varied natural-language state updates unclear. In addition, Table 1 combines different scoring protocols and reader batches, limiting comparisons across rows, although individual paired comparisons remain informative. A lexical-retrieval control with matched packing and input budget, together with an LLM-executed control using the same supporting evidence, would better establish when deterministic compilation is preferable. These limitations materially affect my assessment because the current evidence does not establish whether the proposed approach transfers beyond the evaluated patterns or offers a consistent advantage over simpler alternatives under comparable conditions.

Questions for authors

Clarifications and additional evidence for the current claims

  1. Can the effect of compiled records be separated from instructions and memory length? The BEAM ablation removes the compiled records together with their instructions and does not refill the freed budget, reducing mean memory length from 5,192 to 4,724 tokens (Appendix F). This corresponds to approximately 650 tokens of records and instructions per changed memory, averaged over the 432 compiled memories. Section 3.4 states that the reference implementation does not transfer unused budget between the record and source-text blocks. Does the BEAM constructor, which reserves 3,440 tokens for records (Appendix B.1), follow the same policy? Could the authors report a length-matched comparison with identical, representation-neutral instructions and a common budget, reporting the actual memory length of each arm? The arm without records would also serve as a lexical-retrieval control with matched packing.

  2. How were the rules developed, and how accurate are they on BEAM? On what data were the routing cues and extraction patterns developed, and how were the reference compiler’s α = 0.42 (Appendix B.2) and the 3,440-token record reservation on BEAM (Appendix B.1) chosen? Were any evaluation conversations used in this process? On a manually annotated sample, what is the precision of the state extractor, what fraction of question-relevant updates does it cover, and how often does executing the extracted updates produce the correct state? How are the 432 compiled and 168 raw-text-route memories distributed across abilities and history tiers, and what does the extreme-length routing cue test?

  3. Do the BEAM results reflect the method described in Section 3? Appendix B.1 states that the BEAM constructor omits snapshot COPY, replaces typed path joins with within-unit entity co-occurrence, and applies a deterministic head, tail, and query-signal prefilter in 380 of the 600 memory constructions because the histories exceed two million characters. How are these constructions distributed across history tiers? Could the authors report BEAM results with the reference compiler of Section 3 and a controlled ablation of the prefilter?

  4. How does rule-based compilation compare with an LLM performing the same operations? The paper motivates predefined rules partly through the cost of auxiliary neural inference, but does not compare them with an LLM compiling the same selected evidence. Appendix B.1 mentions an optional language-model compiler that was disabled in the experiments. Could the authors report results with this compiler enabled, or with an LLM producing records from the same evidence under the same budget, together with its construction cost?

  5. Are the comparisons sensitive to the scoring protocol? Table 1 combines five protocols (A to E, Table 7), and TOMC’s overall GPT-5.1 score varies from 54.71 to 60.45 across rows. Reader batches and valid question sets also differ, so this variation cannot be attributed to scoring alone. Could the authors rescore the existing answers under one common protocol to assess sensitivity to judging? Why are all 200 direct LLMLingua-2 memories at 1M empty (Table 6)?

  6. What does construction cost over a full conversation? The reported 4.21 s is the mean of six replays (Appendix G), and TOMC constructs a new memory for each question (Appendix B.1). Could the authors report construction time across all 600 constructions by tier, clarify whether prefiltering is included, and compare total memory-construction and retrieval cost across several questions about one conversation with BEAM-RAG, which reports 23.94 s per conversation index and 9.71 ms per retrieval query?

Questions 1 to 4 would most directly affect my assessment.

Suggestions for future work

  1. Evaluate TOMC on additional conversational memory benchmarks, such as LongMemEval and LoCoMo, including implicit or paraphrased state updates that the predefined patterns may not match.
  2. Study an agentic setting with recurrent memory writing and reading, maintaining memory incrementally rather than rebuilding it for each question, and report end-to-end latency including reader inference.
  3. Provide a failure analysis separating routing, extraction, packing, and reader errors. This could explain why compilation improves temporal reasoning but lowers event-ordering scores for all three readers (Appendix F), and why TOMC trails both BEAM-RAG and LIGHT on instruction following when averaged over the 500K and 1M tiers (Section 4.1).
  4. Compare TOMC with structured-memory approaches that use an LLM to query or manipulate records, such as APEX-MEM, or with hybrids that apply rules to supported operations and use an LLM for the remaining cases.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer 2ZXU (Part 1 of 2: Q1–Q3, Equal-Length Controls, Rule Audit, & Reference/Prefilter Ablations)

Thank you very much for your thorough review and constructive questions. Your comments help make our method presentation clearer, sharpen our attribution between compiled records and source selection, and guide both our controlled experiments and future extensions of our work. (Part 1 covers Q1–Q3; Part 2 covers Q4–Q6 and Suggestions 1–4.)

For Q1, yes, the BEAM constructor follows the same policy, and per your request we ran a controlled comparison with identical neutral instructions and matched memory length: TOMC-Neutral improves compiled and long-horizon subsets on GPT-6 Luna and Rednote while tying overall on GPT-5.1 and DeepSeek 4.1 Flash.

Across all 600 BEAM questions and four readers, with one blind judge per reader, identical neutral instructions, and matched length on every question at mean 5,192.47 tokens, we compare:

  • TOMC-Neutral: compiled records + selected source text;
  • Lexical (no-record control): selected source text + equal-token extra BM25 lines replacing the records (358 record-bearing questions).
Reader (N = 600)TOMC-NeutralLexicalDiff.Key Subset Observation (TOMC-Neutral vs. Lexical)
GPT-6 Luna58.4856.75+1.73+2.43 replaced (358q); +2.27 compiled (432q, p = 0.044); +5.00 Temporal
Rednote preview56.7556.39+0.36+1.51 replaced (358q); +1.38 at 1M; +5.98 Info Extr.
GPT-5.162.1162.13-0.02+1.29 at 100K; +3.75 Preference; +1.88 Contradiction
DeepSeek 4.1 Flash62.6163.05-0.44+3.75 Contradiction; +3.33 Knowledge Update

From the table, at matched length and neutral framing, compiled records improve GPT-6 Luna's compiled and Temporal subsets and lift record-replaced items on Rednote, while tying overall on GPT-5.1 and DeepSeek 4.1 Flash when freed record budget is refilled with BM25 lines; we thus scope compilation to structured state tracking and conversational temporal anchoring.

For Q2, our method design comes from temporal signal representation principles and was prototyped and frozen on LongBench v2, without ever considering or using any of the 600 BEAM evaluation items or rubrics.

Treating histories as non-stationary event streams, TOMC adapts three temporal-signal methods: (1) adaptive piecewise sampling (PAA, Keogh et al.; APCA, Chakrabarti et al.) for order-preserving event sampling with neighbors; (2) symbolic, graph, and histogram spaces (SAX, Lin et al.; Visibility Graphs, Lacasa et al.; SFA/BOSS, Schäfer et al.) for routing to STATE, RELATION, COUNT, or RAW; and (3) multiresolution decomposition (Mallat) for allocating 3,440 record + 4,752 verbatim tokens under 8,192 (alpha = 0.42). All rules and the >=2,000,000-character prefilter/routing cue (bounding CPU time and 3.75 GiB RAM) were fixed on >4M-token LongBench v2 logs:

Slice / AuditRaw RouteCompiled (Hdr + Payload)Payload (Graph, State, Freq)Prefiltered
All BEAM (600q)168 (28.0%)432 (74 + 358)186, 164, 8380 (63.3%)
100K, 500K, 1M (200q)159, 9, 041, 191, 200 (30, 165, 163 payload)2,25,3; 91,71,3; 93,68,20, 180, 200
10 Abilities (60q each)10–2139–50 (22–42 payload)Highest: Multi-Session (42), Update (41)38 (63.3%)
State Parser Audit0 COPY142/164 (86.59%) no cand.22/164 cand. (human audit planned)N/A

From the table, 100K chats mostly stay on raw selection whereas >=2M-char 500K/1M chats trigger prefiltering and compiled routing. Because our exact key = value state rules came from structured LongBench v2 code/docs rather than tuning on BEAM-600 dialogues, 142/164 state-routed items emit no exact answer_candidates, so conversational BEAM relies on date-entity co-occurrence anchors plus timestamped source text.

For Q3, controlled ablations confirm that the 2M-character prefilter halves 1M CPU latency without inflating scores, while conversational TOMC outperforms the generic Section 3 reference compiler because generic count cues over-trigger on open-ended chat.

Ablation Study (DeepSeek 4.1 Flash)QuestionsDefault TOMCAblated VariantKey Finding
w/o 2M-Char Prefilter380 (>2M chars)59.3260.75No sig. diff. (p = 0.308); 1M CPU: 6.89 vs. 15.76 s/q
Generic Sec. 3 RulesAll 60061.0053.12Default +7.88 (p = 0.00002); +35.50 on COUNT (107q)

Disabling prefiltering leaves overall accuracy not significantly changed while doubling million-token CPU latency; meanwhile, both compilers score similarly on the reference RAW subset (60.26 vs. 58.58), whereas generic COUNT cues misfire on conversational wording. We will clearly distinguish the structured-log and conversational instantiations in the paper.

AuthorsOpenReview ↗

Reply to Reviewer 2ZXU (Part 2 of 2: Q4–Q6 & Suggestions 1–4, Same-Pool LLM vs. Rules, Unified Judge, CPU Cost, & Multi-Benchmark)

(Part 2: Q4–Q6 & Suggestions 1–4.)

For Q4, holding retrieved evidence fixed across all 600 BEAM questions, even calling reader-scale DeepSeek 4.1 Flash as an LLM compiler brings little benefit outside Contradiction over zero-cost CPU Rules: the two perform on par across the nine non-contradiction abilities, and the LLM compiler's overall edge on cross-model GPT-5.1 is not statistically significant.

On the same BM25 pool per question under the 8,192-token cap, we compare Raw Pool, zero-API CPU Rules, and an LLM Writer calling DeepSeek 4.1 Flash:

Same-Pool Arm (600q)Extra Writer CostDeepSeek 4.1 Flash (All; 9-Abil.; 1M)GPT-5.1 (All; 9-Abil.; 1M)
Raw PoolUSD 0.00, 0 writer tok53.63; 56.72; 47.4153.55; 57.57; 48.42
CPU RulesUSD 0.00, 0 writer tok54.68; 58.07; 50.4053.98; 58.22; 50.83
LLM Writer (DeepSeek 4.1 Flash)+USD 2.26, +5.73M writer tok, 5.94 s/q57.85; 58.24; 51.7556.45; 57.49; 50.63

From the table, pairing DeepSeek 4.1 Flash with a DeepSeek 4.1 Flash writer benefits from homogeneous two-agent alignment, whereas transferring its synthesized records to GPT-5.1 yields no significant overall gain over CPU Rules. Specifically, LLM Writer wins mainly by paraphrasing implicit conflicts on the 60 Contradiction items, while zero-cost CPU Rules preserves verbatim lines and timestamps, matching or edging it out across the nine non-contradiction abilities pooled and at 1M on GPT-5.1, especially on Event Ordering and Temporal Reasoning.

**For Q5, blind 7-way scoring under unified Gemini 3.8 Flash on the 561/600 shared valid set gives every method one standalone score per cell, preserving all 36/36 wins at 500K/1M and Table 1's overall win rate, while fixing upstream LLMLingua-2-D's 100th-percentile tie filter for the 200 empty 1M memories **

Method (7-Way, 500K/1M)500K GPT-5.1500K Rednote500K DeepSeek 4.1 Flash1M GPT-5.11M Rednote1M DeepSeek 4.1 Flash
TOMC62.8257.0661.5658.6756.8858.84
LIGHT (2026)50.3553.7856.4555.8051.3554.75
BEAM-RAG (2026)52.8149.9252.2452.8151.9151.32
BRIEF-Pro (2026)41.6038.4440.4140.9642.2538.80
LongLLMLingua38.0837.0937.5541.8641.4740.59
LLMLingua-2-H37.6636.1935.3743.1338.8638.36
LLMLingua-2-D (0-empty)36.5932.9633.7138.6937.9334.99

For Q6, across all 600 uncached CPU constructions including prefiltering, TOMC averages 4.27 s/q overall, beating BEAM-RAG's total CPU time for up to 20, 3, and 6 questions/chat across tiers while motivating session caching for 10-question chats at 500K/1M.

Tier (200q)PrefilteredTOMC Mean (Median)BEAM-RAG Index (Median)RAG / QueryBreak-Even
100K0/2000.31 s (0.07 s)6.05 s (5.78 s)9.54 ms~20 q/chat (20.04)
500K180/2005.61 s (5.08 s)21.91 s (21.92 s)9.57 ms~3.9 q/chat (3.91)
1M200/2006.89 s (6.95 s)43.85 s (42.74 s)10.02 ms~6 q/chat (6.37)

For Suggestions 1–4, thank you for these constructive suggestions. Without retuning TOMC's rules or 8,192-token CPU budget, we added both a new structured-memory baseline on BEAM and a new benchmark case study on LoCoMo, where TOMC beats User-as-Code (UaC, mid-June 2026) on BEAM and ranks 1st on LoCoMo.

We considered APEX-MEM from the outset and cited it in our paper, but found no public code; per Reviewer X3sS's suggestion, we compared TOMC with User-as-Code (UaC, mid-June 2026) on all 600 BEAM items:

BEAM (100K–1M, 600q)UaC (June 2026)TOMC (Ours)Overall Gain
DeepSeek 4.1 Flash51.4061.29+9.89 (p = 0.00006)
GPT-5.1 (DeepSeek 4.1 Flash-built UaC)51.0960.21+9.12 (p = 0.00006)

We also added 2024 LoCoMo (1,540q, ~16K–26K tok/chat, lexical F1) as a case study against our two 2026 baselines (BEAM-RAG, LIGHT) and mid-June 2026 UaC:

LoCoMo Case Study (1,540q F1)BEAM-RAG (2026)LIGHT (2026, DeepSeek 4.1 Flash)UaC (June 2026)TOMC (Ours)
DeepSeek 4.1 Flash24.1126.0128.2828.62 (1st)

For Suggestions 2–3, we will add our header-softening and prefiltering failure breakdown and discuss incremental maintenance and latency profiling in the paper. Overall, TOMC delivers highly competitive performance on a lightweight, zero-GPU CPU foundation using modern LLM APIs directly, and our method could also be directly deployed with modern LLM APIs! Thank you again for your thorough review; if our clarifications and new experiments address your concerns, we kindly hope you will consider raising your score.

Thanks for the new experiment and the effort!

In your response to Q4 (Part 2), the LLM writer improves overall accuracy over CPU rules by about 3.2 points with DeepSeek 4.1 Flash and 2.5 points with GPT-5.1, and it costs just $2.26. With such a low cost, what is the advantage of using rules instead of an LLM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.