- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TOMC (Task-Oriented Memory Compilation) is a training-free, CPU-only method that executes task operations while building memory for long-context QA and passes plain text to black-box LLM API readers. Given a history, question, and token budget, it segments the history into line units, selects evidence with fixed lexical rules, routes the question to one of STATE / RELATION / COUNT / RAW, executes the corresponding operation (an ordered SET/COPY state machine, typed relation-path joins, or term counts), and packs the compiled records with selected source text under the budget (records receive αB, α=0.42). On BEAM (100K–1M tokens) with three recent API readers (GPT-5.1, Rednote preview, DeepSeek 4.1 Flash), TOMC reports higher paired mean scores than six retrieval/memory/compression baselines in all 36 reader–baseline–length comparisons at 500K and 1M, uses 32.4–36.6% fewer input tokens than the strongest baseline (LIGHT), and builds memory in 4.21s/call on CPU with no GPU or auxiliary neural model (11–17× faster than per-call neural compressors). The validation includes BEAM answer-quality tables, RULER diagnostics (NIAH/VT/QA2), a BEAM compilation ablation, and construction-cost measurements.
Critical strengths
1.A genuine, reproducible systems contribution. TOMC needs no training, GPU, or auxiliary neural inference; construction is 4.21s/call on CPU versus 47.94–72.07s for per-call neural compressors, with a text-only interface to black-box APIs. Because it is a deterministic rule engine rather than learned weights, the released reference implementation and frozen per-question outputs make the reported numbers independently reproducible offline. This is consequential because it is a practical, verifiable alternative to expensive neural memory pipelines (e.g., LIGHT's 32B writer/filter), and it is the part of the paper most likely to be useful to the community.
2.A clearly isolated positive result on state tracking. On the RULER VT diagnostic, the STATE route reaches 100% answer-item recall for all three readers versus 23.8–58.4% for query selection at 89.2% fewer tokens, and compiling only selected evidence drops recall to ~20% — correctly attributing the gain to executing the update chain over the full history. This is consequential because it cleanly demonstrates the one setting where the core idea provides a large, well-explained benefit.
Critical weaknesses
1.The headline contribution — compiled records — is not statistically supported on the aggregate benchmark. The BEAM compilation ablation (Appendix F) shows restoring compiled records changes overall scores by only +1.20 / −0.91 / +3.01 points across the three readers (negative for Rednote preview), with Holm-adjusted p=0.575, and the benefit is concentrated almost entirely in temporal reasoning (+8.33/+7.50/+4.86). Meanwhile TOMC's margins over the strong baselines (LIGHT, BEAM-RAG) are only ~7–12% overall and near zero at 100K, with the large gains coming against weaker neural compressors. Together this indicates that most of TOMC's advantage comes from retained-source selection and its long-history cost/robustness profile, not from the "compiled records" the framing centers on. This is consequential because the paper's main claim ("executing task operations during memory construction improves answer quality") is broader than the evidence supports; the claim would be sound if rescoped to temporal/state tracking.
2.A published same-line method is not empirically compared, and the key claim is not exclusive. APEX-MEM (ACL 2026), which also builds structured conversational memory and performs temporal/relational operations, is cited but only contrasted in prose. Concurrently, executable-memory work (User-as-Code, arXiv Jun 2026) independently reports the same "execution beats retrieval" effect (6–43% vs ~99% on aggregation questions). This is consequential because, without a comparison to an already-published structured-memory baseline, the paper's positioning and novelty relative to the structured-operations line remain unestablished. (In fairness, TEM, arXiv Oct 2026, post-dates submission and is not a required comparison.)
Questions for authors
1.Clarification affecting assessment. Given Appendix F (p=0.575; benefit concentrated in temporal reasoning, negative for one reader), can you show a statistically significant compiled-records gain under a single shared question set and judge, and state explicitly which abilities the claim should be scoped to? 2.Clarification affecting assessment. Can you add an empirical comparison to APEX-MEM (ACL 2026) on BEAM/RULER, or justify why a structured/temporal-memory system is not an appropriate baseline? A brief positioning against executable-memory approaches (e.g., User-as-Code) would also help. 3.Clarification affecting assessment. Can you report Table 1 under one shared valid-question set and one judge per length tier, so that absolute scores and the "all 36 comparisons" statement are comparable across rows?
2 comments
Response to Reviewer X3sS (Part 2 of 2: W2, Q2, & Q3)
Regarding W2 & Q2: APEX-MEM and Head-to-Head Comparison with User-as-Code
For W2 and Q2, APEX-MEM (ACL 2026) has not released public code, so we will clarify its design in the related work section and compare TOMC head-to-head against open-source executable memory User-as-Code (UaC), where TOMC outperforms UaC on BEAM and edges it out on LoCoMo.
First, on our much longer 100K–1M-token BEAM benchmark (600 questions) under the same 8,192-token budget and one blind rubric judge, we compare UaC and TOMC across two readers:
| Reader (BEAM, N = 600) | UaC | TOMC | Overall Gain | Key Ability Gains (TOMC over UaC) |
|---|---|---|---|---|
| DeepSeek 4.1 Flash | 51.40 | 61.29 | +9.89 (p = 0.00006) | Temporal +31.39, Info Extr. +22.61, Multi-Session +16.44 |
| GPT-5.1 (DeepSeek 4.1 Flash-built UaC) | 51.09 | 60.21 | +9.12 (p = 0.00006) | Temporal +33.75, Info Extr. +24.16, Multi-Session +16.69 |
Second, without retuning any rules or budgets, we evaluate TOMC against UaC, LIGHT, and BEAM-RAG on the shorter 2024 LoCoMo benchmark (~16K–26K tokens per chat; 1,540 questions) with DeepSeek 4.1 Flash using official judge-free lexical token F1 x 100 (where full-sentence answers against concise gold spans score in the 20–30 range):
| Reader (LoCoMo 2024, N = 1,540) | BEAM-RAG | LIGHT (DeepSeek 4.1 Flash) | UaC (June 2026) | TOMC | Rank |
|---|---|---|---|---|---|
| DeepSeek 4.1 Flash | 24.11 | 26.01 | 28.28 | 28.62 | Ranks 1st |
This length contrast is revealing: on shorter 2024 LoCoMo chats (~16K–26K tokens), zero-LLM TOMC edges out UaC (28.62 vs. 28.28) and beats LIGHT (26.01) and BEAM-RAG (24.11), whereas on much longer 100K–1M BEAM histories, TOMC outperforms UaC by a wide margin (+9.89 and +9.12 overall, p = 0.00006; +31 to +34 on Temporal, +16 to +17 on Multi-Session) because UaC synthesizes Python state over bounded session windows via LLM calls while TOMC combines full-history CPU reductions with budgeted verbatim text. Thank you for suggesting User-as-Code; we will add this comparison to the paper.
Regarding Q3: Table 1 Under One Shared Question Set and One Unified Blind 7-Method Judge
For Q3, yes: scoring all seven methods jointly in one blind 7-way rubric panel per (reader, question) under unified Gemini 3.8 Flash on the strict 561/600 3-reader common set (including 0-empty repaired 1M LLMLingua-2-D) gives every method one standalone score per cell, where TOMC wins all 36/36 long-history (500K–1M) comparisons (mean 59.30 vs. 53.74 for LIGHT [+5.56] and 51.83 for BEAM-RAG [+7.47]) and matches Table 1's overall win rate.
Scoring all seven anonymized, shuffled candidates (A-G) together (0, 0.5, or 1 per official rubric criterion, averaged x 100) removes pairwise drift so absolute scores are directly comparable across all rows on the exact same shared valid set (185/200 at 100K, 186/200 at 500K, 190/200 at 1M):
| Method (7-Way, 500K/1M) | 500K GPT-5.1 | 500K Rednote | 500K DeepSeek 4.1 Flash | 1M GPT-5.1 | 1M Rednote | 1M DeepSeek 4.1 Flash | 500K–1M Mean |
|---|---|---|---|---|---|---|---|
| TOMC | 62.82 | 57.06 | 61.56 | 58.67 | 56.88 | 58.84 | 59.30 (36/36) |
| LIGHT (2026) | 50.35 | 53.78 | 56.45 | 55.80 | 51.35 | 54.75 | 53.74 (+5.56) |
| BEAM-RAG (2026) | 52.81 | 49.92 | 52.24 | 52.81 | 51.91 | 51.32 | 51.83 (+7.47) |
| BRIEF-Pro (2026) | 41.60 | 38.44 | 40.41 | 40.96 | 42.25 | 38.80 | 40.41 (+18.89) |
| LongLLMLingua | 38.08 | 37.09 | 37.55 | 41.86 | 41.47 | 40.59 | 39.44 (+19.86) |
| LLMLingua-2-H | 37.66 | 36.19 | 35.37 | 43.13 | 38.86 | 38.36 | 38.26 (+21.04) |
| LLMLingua-2-D (0-empty 1M) | 36.59 | 32.96 | 33.71 | 38.69 | 37.93 | 34.99 | 35.81 (+23.49) |
Thank you again for your time and thorough review, which substantially strengthens our paper; as a zero-GPU CPU module, our method could also be directly deployed with modern frontier LLM APIs! If our clarifications and new experiments address your concerns, we kindly hope you will consider raising your score.
Response to Reviewer X3sS (Part 1 of 2: W1 & Q1)
Thank you very much for your constructive review and thoughtful questions, and for recognizing TOMC's CPU-only efficiency and state-tracking design. Your comments directly help make our presentation much clearer, sharpen our attribution, and make our experimental validation more solid, while also guiding future extensions of our work. Below we address W1 & Q1 in Part 1 and W2, Q2, & Q3 in Part 2.
Regarding W1 & Q1: Scoping the Compilation Claim and Controlled Attribution
For W1 and Q1, we agree that our submission stated the compilation claim too broadly: we will narrow our compilation claim in the manuscript specifically to temporal/state tracking.
Thanks for your comment; separating these factors makes our attribution much clearer and directly informs our future work. First, to isolate compilation from retrieval across all 600 BEAM questions under the 8,192-token cap and one blind judge per reader, we hold the retrieved BM25 candidate pool identical and compare
Raw Pool, zero-APICPU Rules, and anLLM WritercallingDeepSeek 4.1 Flash:From this table, when candidate evidence is held fixed, zero-cost
CPU Rulesimproves million-token histories and target state/temporal slices such as Multi-Session and Temporal Reasoning overRaw Pool. Meanwhile, pairingDeepSeek 4.1 Flashwith aDeepSeek 4.1 Flashwriter benefits from homogeneous two-agent alignment, whereas transferring its synthesized records toGPT-5.1yields no statistically significant overall gain overCPU Rules:LLM Writerleads mainly on the 60 Contradiction items, while zero-costCPU Rulespreserves verbatim lines and timestamps, matching or edging it out across the nine non-contradiction abilities pooled and at 1M onGPT-5.1, especially on Event Ordering and Temporal Reasoning.Second, to test compiled records directly on the same 600 questions under one blind judge per reader, identical neutral instructions, and identical token length (mean 5,192.47 cl100k tokens), we compare:
TOMC (Neutral): compiled records + selected source text;Lexical(no-record control): selected source text + equal-token extra BM25 lines replacing the records (358 record-bearing questions).From the table, at matched length and neutral framing,
TOMC (Neutral)improvesGPT-6 Luna's compiled and Temporal subsets and lifts record-replaced items onRednote, while tying overall onGPT-5.1andDeepSeek 4.1 Flashwhen removed records are refilled with extra BM25 lines, confirming why narrowing our compilation claim is warranted; we will revise the paper accordingly.