- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LEDGER separates fact extraction, program induction, execution, and answering for conversational-memory aggregation tasks: counting, summation, differences, ratios, latest-state selection, and listing. An offline extractor produces source-linked facts session by session without seeing future queries. A Phi-3 router fine-tuned on synthetic MSC data uses candidate facts to specify an operator, filters, and operand bindings. A deterministic executor evaluates the program over the complete fact store and appends the result, supporting fact IDs, and source IDs to Parent RAG’s original top-10 sessions. NONE, empty support, or invalid results return the original retrieval payload.
Critical strengths
-
Fact extraction is independent of future queries. The router explicitly specifies operations and filters, and the executor returns results with supporting sources. Candidate selection bounds the program-induction context while execution covers the complete fact store. This separation supports auditing omissions, binding errors, and execution errors independently and reusing the same facts for subsequent queries.
-
LEDGER and Parent RAG share the raw retrieval policy, with gains of 4.5 percentage points on both benchmarks. In Table 8, replacing the executor with an LLM reduced test402 accuracy to 66.2%, and removing the raw-text overlay reduced it to 64.2%. These comparisons support combining execution with raw evidence. The paper also distinguishes earlier privileged diagnostics from final results.
Critical weaknesses
-
Section 6’s 11/13 operator accuracy and 4/13 execution equivalence concern early SFT, not V11.3. The final system lacks measurements of non-NONE coverage, support-set accuracy, operand binding, and execution equivalence. The results therefore do not identify which computations produce the end-to-end gains or establish whether the final router resolves the binding bottleneck identified in the paper.
-
Table 2 compares retrieval variants, PAL-style, and Fact Store + LLM, but no structured-memory method. PAL also changes both fact-construction timing and input context. Table 8 supports the executor’s value within this system, but does not separate the contributions of reusable extraction, router training, and the restricted language relative to related alternatives.
-
The final payload does not add program-support sessions. If supporting sources fall outside Parent’s top-10, the answer model cannot necessarily check the side channel against raw text. Structural validation cannot detect semantic errors, and the reported configuration disables confidence gating. The overlay ablation supports retaining raw text but does not measure how often incorrect computations are accepted, corrected, or cause regressions.
-
LEDGER adds per-session fact extraction, router training, and per-query program induction while retaining ten complete raw sessions. Construction cost, fact-store size, query tokens, latency, and amortized cost across query counts are unreported. Without these measurements, the practical value and operating range of reuse relative to query-time PAL or Fact Store + LLM remain unresolved.
Questions for authors
-
What are V11.3’s non-NONE coverage, support-set precision/recall, operand-binding accuracy, and execution equivalence by operator? How often do supporting sources fall outside Parent’s top-10? For structurally valid but semantically incorrect records, please trace their acceptance or correction rates and the resulting answer changes relative to Parent.
-
With identical facts, raw-text payloads, and training supervision, how do prompted program induction and the fine-tuned router compare? Could you add a structured-memory baseline with the same answering model and quantify the separate gains from the fixed fact store and restricted execution interface relative to existing combinations?
-
Please report per-question improvements/regressions and paired intervals against Parent RAG and the executor ablation, resampling LoCoMo by conversation. Among the remaining 232 questions, are regressions offset by improvements, and in which question types? Do overall and target-type differences persist across repeated runs or evaluation by an independent judge?
-
Please report extraction/training cost, routing and answering tokens per query, fact-store storage, and latency, with amortization across queries over the same history. Against query-time PAL and Fact Store + LLM, what total costs are required to achieve the reported accuracy, and at what query counts do the costs cross over?