- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Question-Grammar Compilation (QGC), a structured supervision framework for optimizing per-document KV memories. By systematically generating question–answer pairs across three linguistic axes (query frame, lexical variation, and word order) the method improves factual accessibility in compressed memories. Experiments demonstrate that systematic fact coverage and phrasing diversity substantially improve QA accuracy compared to unconstrained self-study generation.
Critical strengths
Systematic Experimental Analysis: The paper provides controlled experiments examining how different dimensions of question diversity affect memory accessibility, offering useful empirical insights.
Practical Supervision Guidelines: The findings highlight the importance of fact coverage and linguistic diversity, providing actionable guidance for constructing effective QA supervision.
Promising Empirical Results: QGC achieves strong QA performance with substantially fewer attention slots than full-document inference, demonstrating the potential of structured supervision for compact KV memories.
Critical weaknesses
Limited Technical Novelty: The proposed method primarily optimizes a task-specific data generation and supervision workflow using predefined question templates. It introduces neither a fundamentally new memory representation nor a novel optimization algorithm, limiting its methodological contribution.
Questionable Generalizability: The approach relies heavily on predefined relation schemas, structured fact extraction, and single-hop factual QA. It remains unclear whether the findings generalize to unstructured knowledge, complex reasoning, or broader agent memory applications.
Insufficient Presentation Clarity: The paper's presentation is unnecessarily complex relative to its core methodological contribution. The extensive discussion of factorial interventions and statistical analyses obscures the main intuition and practical significance. The authors should substantially improve the organization, clarity, and accessibility of the presentation to communicate the contributions more effectively.
Questions for authors
Beyond systematically controlling fact coverage and question diversity, what is the fundamental methodological contribution of QGC compared with existing synthetic QA generation and curriculum construction approaches?
How would QGC extend to unstructured documents without predefined relation schemas, or to tasks requiring multi-hop reasoning and integration of multiple facts? Are there experimental results supporting such generalization?
Given that the proposed approach requires fact extraction, predefined grammar construction, and per-document optimization, how does its overall computational and implementation cost compare with simpler alternatives? Could the authors also clarify the core technical contribution and better distinguish methodological innovations from empirical observations?
Holding the memory fixed is deliberate: the compiler changes one question property at a time to isolate what a memory needs
Technical novelty (and Q1, second half of Q3). The paper asks which properties of its training questions a per-document memory needs. We therefore deliberately hold the representation, frozen reader and optimizer fixed: changing them alongside the supervision would confound its effect. The methodological contribution is the instrument that isolates it: a compiler that builds the supervision with exact counts for every fact. One property of the questions can then change while all else stays fixed, each fact's exposure and question length included. We use it for a full factorial over three phrasing axes and a reweighting that changes only how the axes are combined while every axis and every pair of axes keeps its frequency. Synthetic-QA pipelines filter generated data, and curricula order examples; neither is built to hold exposure fixed, so phrasing stays confounded with coverage and answer quality: repairing only the answers and exposure of unconstrained generation recovers 47–86% of its gap to the grammar (Z2MW, Q2). The empirical contribution is the answer:
The design rule: give every fact correct answers, adequate exposure and varied phrasing, above all its entity at several positions in the question; how exposure is split across axis combinations, the tested reweighting included, can be left to chance without detected loss. Exact control is what the measurement needs; the recipe is simpler.
Generalizability (and Q2). Our claims concern single-hop factual access over a predefined relation schema. Extending the schema needs facts in entity–relation–value form, not hand-written templates: per-fact LLM paraphrases compiled at the same exposure match or exceed the grammar on both synthetic platforms (§5.4) and on ClinicalTrials.gov (0.933 vs 0.910). The facts need not be given either: on ClinicalTrials.gov an LLM extracts them from the record text (F1 0.974), and the compiled code exceeds both the full document (0.869) and unconstrained generation at the same budget (0.377). Unstructured knowledge and multi-hop reasoning lie outside the tested scope.
Cost (first half of Q3). The grammar is written once per schema, so its questions cost no LLM calls; unconstrained generation takes 4–5× as long as the optimization (Table 8). On the 100 ClinicalTrials.gov records (batch size 1, medians over 914 queries):
Against re-reading the document it replaces, the code saves 212.7 ms per query and recovers its 27.8 s preparation after about 130 queries. Against the facts it is compiled from, it costs the same per query at 10 fields and stays at 64 slots as they grow: at 55–57 facts per document (synthetic platforms) it answers in 71 ms, against 153 ms for re-reading the facts or 81–82 ms cached in 17–18× more memory. The revised version will add this table.
Presentation clarity. The factorial and the tests let each finding be read as an effect of the supervision. The intuition: what a per-document code can answer depends on the questions it is trained on, and QGC fixes those questions so each property can be tested alone. In short:
The revised version will separate established from speculative conclusions, explain the λ-path in plain language, and move the details of the statistical tests to the appendix.