acceptodds
Under review as a conference paper at ICLR 2027

Compile What Counts: How Question Supervision Shapes Queryable Memory

Abstract

A per-document key–value (KV) code is memory for a frozen reader: optimized once against the document, it is queried in place of the document thereafter. Optimized to store the document, however, such a code need not expose it through the reader's question-answering interface: storage and access are different objectives. What makes the memory queryable is the supervision that bridges them, question–answer pairs that current practice samples from a language model and that guarantee neither per-fact coverage nor question-distribution structure. Question-Grammar Compilation (QGC) instead compiles the supervision: a closed question grammar with three phrasing axes and 27 cells per relation is expanded at exact per-fact exposure, which turns the question distribution into something we can intervene on. A full factorial over axis widths and a path that moves only third-order joint mass at fixed low-order marginals show, on two synthetic platforms, that widening each axis causes gains of up to accuracy with subadditive interactions, while third-order structure and cell-level exactness show no detected response. At matched budget, unconstrained self-study-style generation loses – against the compiled grammar, whereas LLM paraphrases compiled per fact match or exceed it. Axis directions replicate on ClinicalTrials.gov under a prospective gate, where compiled codes reach accuracy against for the zero-shot full document at fewer attention slots than the document. The dominant factor is systematic, fact-grounded supervision with adequate question support; who writes the questions matters only where the grammar's coverage of interrogative forms is thin.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, context compression, key-value memory, synthetic supervision, question answering

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_dKR2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Question-Grammar Compilation (QGC), a structured supervision framework for optimizing per-document KV memories. By systematically generating question–answer pairs across three linguistic axes (query frame, lexical variation, and word order) the method improves factual accessibility in compressed memories. Experiments demonstrate that systematic fact coverage and phrasing diversity substantially improve QA accuracy compared to unconstrained self-study generation.

Critical strengths

Systematic Experimental Analysis: The paper provides controlled experiments examining how different dimensions of question diversity affect memory accessibility, offering useful empirical insights.

Practical Supervision Guidelines: The findings highlight the importance of fact coverage and linguistic diversity, providing actionable guidance for constructing effective QA supervision.

Promising Empirical Results: QGC achieves strong QA performance with substantially fewer attention slots than full-document inference, demonstrating the potential of structured supervision for compact KV memories.

Critical weaknesses

Limited Technical Novelty: The proposed method primarily optimizes a task-specific data generation and supervision workflow using predefined question templates. It introduces neither a fundamentally new memory representation nor a novel optimization algorithm, limiting its methodological contribution.

Questionable Generalizability: The approach relies heavily on predefined relation schemas, structured fact extraction, and single-hop factual QA. It remains unclear whether the findings generalize to unstructured knowledge, complex reasoning, or broader agent memory applications.

Insufficient Presentation Clarity: The paper's presentation is unnecessarily complex relative to its core methodological contribution. The extensive discussion of factorial interventions and statistical analyses obscures the main intuition and practical significance. The authors should substantially improve the organization, clarity, and accessibility of the presentation to communicate the contributions more effectively.

Questions for authors

Beyond systematically controlling fact coverage and question diversity, what is the fundamental methodological contribution of QGC compared with existing synthetic QA generation and curriculum construction approaches?

How would QGC extend to unstructured documents without predefined relation schemas, or to tasks requiring multi-hop reasoning and integration of multiple facts? Are there experimental results supporting such generalization?

Given that the proposed approach requires fact extraction, predefined grammar construction, and per-document optimization, how does its overall computational and implementation cost compare with simpler alternatives? Could the authors also clarify the core technical contribution and better distinguish methodological innovations from empirical observations?

1 comment
AuthorsOpenReview ↗

Holding the memory fixed is deliberate: the compiler changes one question property at a time to isolate what a memory needs

Technical novelty (and Q1, second half of Q3). The paper asks which properties of its training questions a per-document memory needs. We therefore deliberately hold the representation, frozen reader and optimizer fixed: changing them alongside the supervision would confound its effect. The methodological contribution is the instrument that isolates it: a compiler that builds the supervision with exact counts for every fact. One property of the questions can then change while all else stays fixed, each fact's exposure and question length included. We use it for a full factorial over three phrasing axes and a reweighting that changes only how the axes are combined while every axis and every pair of axes keeps its frequency. Synthetic-QA pipelines filter generated data, and curricula order examples; neither is built to hold exposure fixed, so phrasing stays confounded with coverage and answer quality: repairing only the answers and exposure of unconstrained generation recovers 47–86% of its gap to the grammar (Z2MW, Q2). The empirical contribution is the answer:

  • varying word order adds 0.20–0.22 accuracy, wording 0.11–0.17 and frame 0.06–0.10; the estimates keep this order with a second reader and on ClinicalTrials.gov;
  • widening two axes together gains less than the sum of the two;
  • at one budget, the same LLM's questions lose 0.24–0.40 to the grammar when generated freely and match or exceed it when compiled per fact;
  • word order acts mainly through position: changing only position indices, tokens in place, reverses which training order answers better (new; 9RVJ, W2).

The design rule: give every fact correct answers, adequate exposure and varied phrasing, above all its entity at several positions in the question; how exposure is split across axis combinations, the tested reweighting included, can be left to chance without detected loss. Exact control is what the measurement needs; the recipe is simpler.

Generalizability (and Q2). Our claims concern single-hop factual access over a predefined relation schema. Extending the schema needs facts in entity–relation–value form, not hand-written templates: per-fact LLM paraphrases compiled at the same exposure match or exceed the grammar on both synthetic platforms (§5.4) and on ClinicalTrials.gov (0.933 vs 0.910). The facts need not be given either: on ClinicalTrials.gov an LLM extracts them from the record text (F1 0.974), and the compiled code exceeds both the full document (0.869) and unconstrained generation at the same budget (0.377). Unstructured knowledge and multi-hop reasoning lie outside the tested scope.

Cost (first half of Q3). The grammar is written once per schema, so its questions cost no LLM calls; unconstrained generation takes 4–5× as long as the optimization (Table 8). On the 100 ClinicalTrials.gov records (batch size 1, medians over 914 queries):

Context given to the readerQuery latencyKept per documentPreparationAccuracy (§6.2)
Full document, re-read313.4 mstext onlynone0.869
Full document, KV cached114.6 ms126.9 MiB KVone prefill0.869
Extracted facts as text102.3 mstext onlyextraction 17.8 s0.948
64-slot code100.7 ms3.5 MiB codeextraction 17.8 s + optimization 10 s0.910

Against re-reading the document it replaces, the code saves 212.7 ms per query and recovers its 27.8 s preparation after about 130 queries. Against the facts it is compiled from, it costs the same per query at 10 fields and stays at 64 slots as they grow: at 55–57 facts per document (synthetic platforms) it answers in 71 ms, against 153 ms for re-reading the facts or 81–82 ms cached in 17–18× more memory. The revised version will add this table.

Presentation clarity. The factorial and the tests let each finding be read as an effect of the supervision. The intuition: what a per-document code can answer depends on the questions it is trained on, and QGC fixes those questions so each property can be tested alone. In short:

QuestionInterventionFinding
Which phrasing axes matter?Widen frame, wording and word order (2×2×2)All help, word order most; pairs subadditive
Does finer structure matter?Reweight third-order structure at fixed marginals; sample cells at randomNo detected effect
Do the gains need the correct facts?Derange the answersAccuracy falls to the no-context level
Does the question source matter?Grammar, per-fact LLM paraphrases, unconstrained generation at one budgetPer-fact LLM paraphrases match or beat the grammar; unconstrained generation loses 0.24–0.40
Do the directions replicate?Second reader family; ClinicalTrials.govSame directions; code 0.910 vs full document 0.869

The revised version will separate established from speculative conclusions, explain the λ-path in plain language, and move the details of the statistical tests to the appendix.

Reviewer_9RVJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how the structure of question–answer supervision affects the queryability of per-document key–value (KV) memory codes optimized against a frozen language model. The authors introduce Question-Grammar Compilation (QGC), which constructs a closed question grammar over three axes - query frame, lexical variation, and word order - and guarantees exact per-fact supervision exposure. Using factorial interventions and a marginal-preserving intervention on third-order joint structure, the paper finds that widening each phrasing axis improves held-out QA accuracy, with subadditive pairwise interactions and no detected benefit from the tested third-order changes. At a matched supervision budget, unconstrained self-study-style generation substantially underperforms compiled supervision, whereas per-fact LLM paraphrases within the compiler match or outperform the hand-designed grammar. The authors further report positive effects of axis widening on ClinicalTrials.gov records, where the compiled code achieves 0.910 accuracy versus 0.869 for zero-shot full-document access, while using substantially fewer document-side attention slots.

The paper’s central contribution is an intervention-based analysis of supervision design for queryable memory, rather than a new memory architecture.

Critical strengths

A well-motivated and experimentally tractable research question.

The paper makes a useful distinction between storing document information and making that information accessible through a frozen reader’s QA interface. Treating the supervision distribution as an experimental variable is a promising direction.

Strong control over the experimental interventions.

The QGC framework controls per-fact exposure, question-token length, and selected low-order marginals. The factorial design is substantially more informative than comparing independently generated QA datasets with uncontrolled differences in coverage and phrasing.

Useful comparisons between structured and generated supervision.

The matched-budget experiments show that unconstrained self-study-style generation can perform substantially worse than compiled supervision, while LLM paraphrases incorporated into the compiler can improve performance. These results suggest that reliable coverage and fact-grounded answers matter more than requiring every question to follow a hand-written template.

Appropriate qualification of several findings.

The authors distinguish a lack of detected third-order effects from a universal claim that higher-order structure never matters. They also acknowledge that the mechanism experiments cover only a limited set of phrasing axes and distributions.

Evidence beyond a single synthetic dataset.

The use of a second synthetic domain, an exploratory reader-family comparison, and a prospective direction gate on real registry documents makes the results more compelling than a purely synthetic demonstration.

Critical weaknesses
  1. The scope of the conclusions is broader than the demonstrated task setting.

The experiments primarily concern single-hop factual retrieval over a predefined relation schema. The compiler receives this schema as an input, and the natural-document evaluation uses structured ClinicalTrials.gov fields. This is a useful controlled setting, but it leaves open whether the same conclusions hold for less structured documents, compositional questions, multi-hop reasoning, temporal or cross-document queries, and questions requiring inference rather than direct fact retrieval.

The paper would be stronger if it more clearly separated conclusions about the tested QGC setting from broader claims about queryable memory. In particular, success on a closed grammar does not establish that the same supervision properties are sufficient for open-ended question distributions.

  1. The paper demonstrates the value of phrasing diversity more clearly than it identifies the underlying mechanism.

Widening the query-frame, lexical, and word-order axes improves accuracy, and the word-order effect is consistently largest. The paper proposes a positional-geometry explanation involving RoPE phases, but presents this as a hypothesis rather than a directly tested mechanism.

Alternative explanations remain plausible. For example, different word orders may change which entity and relation tokens receive effective attention, rather than primarily teaching invariance to positional phase. A targeted intervention that perturbs query positions at inference time, or compares semantically equivalent variants while controlling token positions, would help distinguish these explanations.

This distinction matters because the paper’s most interesting result may not simply be that diverse supervision helps, but that particular forms of diversity interact with the geometry of the frozen reader’s attention.

  1. The evidence for the irrelevance of higher-order structure is necessarily narrow.

The paper reports no detected response to the tested third-order intervention while preserving single-axis and pairwise marginals. This is a valuable result for the specified intervention, but the authors themselves note that the lambda-path explores only one direction in the eight-dimensional space of third-order structure.

Consequently, the evidence does not establish that higher-order interactions are generally unimportant. The factorial width experiment and the lambda-path also answer related but distinct questions. The paper should emphasize more prominently that its conclusion concerns the tested interaction and intervention family, rather than higher-order structure in general.

  1. The practical compression claim needs a more complete cost accounting.

The reported 36× reduction concerns document-side attention slots, not a 36× reduction in total memory footprint or end-to-end cost. The paper reports that a code occupies approximately 3.5 MiB per document and that optimization takes about 50 seconds per document on the reported hardware. It also uses a large compiled supervision stream, with roughly 180,000 training tokens per synthetic document on average.

These costs do not invalidate the method: per-document optimization may be amortized across many future queries. However, the practical benefit depends on query volume, document-update frequency, storage requirements, and the cost of constructing and optimizing each code. The comparison against full-document inference would be more informative with a break-even analysis that includes preprocessing, optimization, storage, and repeated-query costs.

  1. The natural-document comparison does not isolate the source of the performance advantage.

The compiled code is optimized using document-specific QA supervision, whereas the zero-shot full-document baseline is not trained or optimized for the document. The comparison therefore demonstrates that a document-specific optimized representation can outperform direct zero-shot access under the reported evaluation, but it does not isolate whether the gain comes from compression, the supervision design, or document-specific optimization itself.

The extracted-facts and gold-facts text baselines are useful diagnostic references, but they do not fully resolve this issue. Additional controls, such as alternative supervision strategies using the same per-document optimization budget, or comparisons with other optimized memory representations, would clarify the incremental contribution of QGC.

  1. The evaluation protocol leaves some questions about robustness and generalization.

The synthetic evaluations use programmatically constructed facts and exact-match scoring, while the natural-document evaluation covers one structured domain. The paper also uses a relatively small number of documents in its primary synthetic platform and only three seeds per code arm.

The reported document-level confidence intervals and preregistered direction gate are useful safeguards, but the paper would benefit from stronger evidence on domain diversity and on question distributions that differ more substantially from the compiled grammar. In particular, a low n-gram overlap threshold does not by itself establish semantic or structural novelty of the evaluation questions.

Questions for authors

The reviewer provide the questions in the weakness part.

3 comments
AuthorsOpenReview ↗

A direct test shows the word-order penalty is mainly positional; same-budget controls place the gain in the supervision design

W1. Scope. Our claims concern single-hop factual access over a predefined relation schema; the revised version will separate established from speculative conclusions. Beyond the closed grammar, an independently generated question distribution (W6) preserves the conclusions about all three alternative supervision sources, also below the 4-gram overlap threshold (Table 2, App. F); and on ClinicalTrials.gov the extractor reads the record text (F1 0.974), with the fields serving as schema and gold.

W2. Mechanism. A test you propose supports the positional hypothesis in §7. For a code trained on one word order, we keep each token of a held-out question in place and permute only its position indices, moving the entity phrase's indices to the other end, as in the other word order; the answer position is unchanged. Without the permutation, codes trained on the question's word order lead those trained on another by +0.207 [+0.167, +0.248] / +0.170 [+0.105, +0.234]; with it, the lead reverses to −0.129 [−0.176, −0.081] / −0.116 [−0.172, −0.061]: with identical tokens and indices, codes do better when the entity's place matches their training than when the token order does. In this 2×2, position has a main effect of +0.168 / +0.143, token order of +0.039 [+0.006, +0.073] / +0.027 [−0.023, +0.077]. The word-order penalty is thus mainly positional, the interaction with the frozen reader's attention geometry you highlight: diversity lets the code be read with the entity at several places in the question. The revised version will add this test to §5.1 and state the positional account in §7 as tested.

W3. Third-order scope. The λ-path moved one direction in the eight-dimensional space, in one of its four two-dimensional components; we have now moved one direction in each of the other three: their mean change is +0.006 [−0.010, +0.022] / +0.018 [−0.007, +0.042], and no component survives Holm correction. The revised abstract and introduction will say "the tested third-order family", and §5.3 will state its resolution: a factorial three-way effect of 0.016 / 0.023, or a change of 0.025 / 0.046 along the λ-path over λ ∈ [0, 0.75], would be detected with about 80% power.

W4. Cost accounting. Here are query latency and full preparation cost on the 100 ClinicalTrials.gov records (batch size 1, medians over 914 queries):

Context given to the readerQuery latencyKept per documentPreparationAccuracy (§6.2)
Full document, re-read313.4 mstext onlynone0.869
Full document, KV cached114.6 ms126.9 MiB KVone prefill0.869
Extracted facts as text102.3 mstext onlyextraction 17.8 s0.948
64-slot code100.7 ms3.5 MiB codeextraction 17.8 s + optimization 10 s0.910

Against re-reading the document, the code saves 212.7 ms per query and recovers its 27.8 s preparation after about 130 queries; against a cached document it is as fast with 36× less memory. The 50 s you cite is for 55-fact synthetic documents; ten facts take 10 s. A changed document incurs the preparation again, so amortization depends on the queries between rebuilds. The revised version will add this as App. H (Table 11).

W5. Source of the natural-document gain. At fixed compression and optimization budget, the supervision decides whether a code falls below or rises above the full document. On the 60 gate records, a code trained with one phrasing per relation scores 0.708, below the zero-shot full document (0.857), while the compiled code reaches 0.899 (the last two from the released per-document results). New: on all 100 records at the same budget, unconstrained generation scores 0.377 and the compiled code 0.910, or 0.933 (+0.023 [0.011, 0.035]) with per-fact LLM paraphrases. The gain also needs the correct facts: deranged labels bring a code close to the no-context level on both synthetic platforms (0.131 / 0.138 vs 0.119 / 0.117, §5.3). Other memory representations lie outside our question, which fixes the representation and varies the supervision. The revised version will add both new controls.

W6. Robustness. Intervals are over documents (40 and 20 synthetic documents, 60 or 100 registry records), with the three code seeds averaged per document; at 20 documents every main and pairwise interval in Table 1 excludes zero. The point estimates rank word order > wording > frame in all four settings. Novelty of the evaluation does not rest on the n-gram threshold: the independent questions are written by Llama-3.1-8B for the same entity–relation targets, screened for intent by gemma-2-9b-it, and exclude every grammar question verbatim. New: on them, one phrasing per relation trails the full grammar by 0.394 / 0.447, as on held-out grammar questions (0.383 / 0.469). On ClinicalTrials.gov the evaluation phrasings are verbalized by a model that never sees gold values (§6.2).

Reviewer_9RVJOpenReview ↗

Further questions and concerns about the comments.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to directly address the concerns raised in my review.

In particular, the positional intervention in W2 provides meaningful evidence that the word-order effect is substantially influenced by position assignments rather than token order alone. The additional third-order interventions and statistical power analysis in W3 also strengthen the conclusions within the tested intervention family. Furthermore, the matched-budget controls in W5 substantially clarify the contribution of supervision design, addressing one of my original concerns.

However, two points remain worth clarifying.

  1. Interpretation of the positional mechanism.

The new experiment convincingly demonstrates positional sensitivity. However, permuting position indices while preserving token order may alter multiple relative positional relationships within the reader's attention computation. Therefore, I would distinguish evidence for positional dependence from a more specific mechanistic explanation involving RoPE geometry. The former is supported by the new results, while the latter may still require more targeted analysis.

  1. Practical advantage over extracted-fact representations.

The newly reported cost analysis is useful, but it raises an additional question. The extracted-facts-as-text baseline achieves higher accuracy (0.948 vs. 0.910), with nearly identical query latency (102.3 ms vs. 100.7 ms), while avoiding the additional code optimization step.

I recognize that the paper's main contribution is the study of supervision design under a fixed memory representation, rather than proposing a universally superior representation. Nevertheless, this comparison appears important when interpreting the practical benefits of optimized queryable memory codes. A clearer discussion of the trade-offs between storage, accuracy, and preprocessing cost would strengthen the paper.

Overall, the rebuttal addresses several of my original concerns, particularly regarding experimental controls and the interpretation of higher-order effects. I view these additions positively and believe they strengthen the empirical contribution. My remaining concerns primarily relate to the scope of the mechanistic interpretation and the practical significance of the learned representation relative to simpler alternatives.

AuthorsOpenReview ↗

The word-order effect lies within the question; combined supervision brings the code level with the extracted facts at a smaller footprint

Thank you for the follow-up.

1. Interpretation of the positional mechanism. We agree that the permutation establishes positional dependence, not a specific geometry. The test your review suggested, perturbing query positions at inference time, separates the two: together with the permutation, it places the word-order effect within the question. We shift every position index of the query by k ∈ {2, 4, 8, 16, 32}: its tokens and the distances within it stay the same, and only its RoPE phase relative to everything before it, the code slots included, changes. On the second synthetic platform (20 documents, three seeds), the shift is not inert: both codes lose 0.18–0.19 accuracy at k = 32. The gain from training on all three word orders rather than one, however, stays between +0.173 and +0.192 at every shift: +0.173 [+0.132, +0.213] unshifted and +0.183 [+0.153, +0.214] at k = 32 (mean change over k = 4–16: +0.011 [−0.033, +0.055]). The permutation, by contrast, keeps every token in place and changes only position indices within the question, yet reverses which training order reads it better. Shifting the query relative to the code leaves the effect intact; rearranging positions inside the question flips it. Across shifts of up to 32 we detect no dependence on the query's phase relative to the code; the revised §7 states the mechanism at this level, leaving open which relation within the question the reader uses, and App. I adds the shift test.

2. Practical advantage over extracted-fact representations. We agree with your framing, and the new results change the balance of this comparison: for a one-time cost of about 40 s more preparation per record, combining the grammar's questions with per-fact LLM paraphrases brings the code level with the extracted facts in accuracy, at the same cost per query and in about half their context.

Accuracy. The 0.910 comes from grammar-only supervision. To test whether its gap to the extracted facts is inherent to the representation, we vary only the supervision, with the 64-slot code, frozen reader and evaluation fixed. Per-fact LLM paraphrases already raised the code to 0.933 (W5). At twice the grammar's per-fact exposure, training on the grammar's questions and the paraphrases together brings it to 0.950, level with the extracted facts (0.948; difference +0.002 [−0.007, +0.012]). The same doubled exposure spent on the paraphrases alone gives 0.937, so the gain comes from combining the two sources (+0.013 [+0.004, +0.022]), not from more training. The gap you point to is thus closed by supervision design, the variable the paper studies.

Storage. What the code saves is the state the reader must hold to answer: 64 slots and 3.5 MiB for a record, against 116 positions and 6.3 MiB for the text, whether the text is cached or recomputed at each query. Shortening the text cannot match it: even the bare values, without field names, need a median of 78 tokens, more than the code's 64 positions. Held to those 64 positions, the extracted facts fall from 0.948 to 0.557 even when the reader rewrites them as compactly as it can (0.496 when truncated), against 0.910 for the code (+0.353 [+0.319, +0.387]; ahead on 98 of 100 records).

Preprocessing. Per record:

Given to the readerContextAccuracyOne-time preparation
Extracted facts as text116 tokens (median)0.94817.8 s: extraction
64-slot code, grammar64 slots0.91027.8 s: + 10 s optimization
64-slot code, grammar + paraphrases64 slots0.950about 58 s: + about 20 s paraphrasing, 20 s optimization

Per query, reading a 64-slot code costs what reading the text costs (100.7 vs 102.3 ms). The extra preparation is paid once per record, while the context saving recurs at every query: the text is thus the simpler choice when records change often, the code when context per record is the binding budget. The revised PDF adds this comparison to §6.2 and App. H.

Reviewer_Z2MWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how QA training data affect document-specific KV memories for a frozen language model. They propose Question-Grammar Compilation (QGC), which generates questions from facts extracted from a document by varying the question frame, wording, and word order while keeping the training exposure per fact fixed. Experiments on two synthetic datasets and ClinicalTrials.gov show that greater question diversity improves accuracy, with diminishing gains when combining variations. More precise control over question frequencies and combinations did not significantly improve accuracy. The evaluation mainly measures whether each trained memory can answer new phrasings of questions about the trained document’s facts.

Critical strengths
  1. The experiments vary the question phrasing while keeping the number of training examples per fact and question length fixed. This provides clearer evidence that the accuracy gains come from question diversity rather than more training examples.

  2. The benefits of question diversity are observed across two synthetic domains, another reader model, and real documents. This suggests that the main finding is not limited to one dataset or model.

Critical weaknesses

The practical benefit of the learned KV memory remains unclear. On ClinicalTrials.gov, directly providing the extracted facts achieves 94.8% accuracy with a median of 116 tokens, while the KV memory uses 64 slots, achieves 91.0%, and requires separate optimization for each document. The paper does not show whether the reduction in attention slots provides enough savings in query latency or total cost to justify the additional optimization and lower accuracy.

Questions for authors
  1. How does the KV approach compare with the extracted-fact baseline in query latency and total cost, including preparation? How many queries per document, if any, are needed to offset the additional optimization cost?

  2. Can the authors evaluate the unconstrained self-study baseline using corrected answers and the same fact coverage and training exposure as QGC? This would help clarify how much of the performance gap is due to question phrasing rather than answer quality or coverage.

2 comments
AuthorsOpenReview ↗

The code's fixed footprint pays back as documents and fact lists grow; with answers and exposure corrected, the gap sits on facts asked in few ways

Thank you for the precise questions.

Weakness and Q1. Latency, total cost and break-even. The code is built to replace the document in the reader's context, at 64 slots however much the document holds. Here are query latency and full preparation cost on the 100 ClinicalTrials.gov records (batch size 1, medians over 914 queries):

Context given to the readerQuery latencyKept per documentPreparationAccuracy (§6.2)
Full document, re-read313.4 mstext onlynone0.869
Full document, KV cached114.6 ms126.9 MiB KVone prefill0.869
Extracted facts as text102.3 mstext onlyextraction 17.8 s0.948
64-slot code100.7 ms3.5 MiB codeextraction 17.8 s + optimization 10 s0.910
  • Against the document it replaces, the code is more accurate, saves 212.7 ms per query over re-reading and recovers its 27.8 s preparation after about 130 queries; against a cached document it is as fast with 36× less memory.
  • Against the extracted facts it is compiled from, better supervision brings the code within 0.015 of their accuracy. Trained on per-fact LLM paraphrases (Table 2's recipe) instead of the grammar's questions, it reaches 0.933 (+0.023 [0.011, 0.035]), for about 20 s more preparation per record. Per query the two cost the same on these 10-field records (100.7 vs 102.3 ms, 64 slots against a median of 116 tokens), so the 10 s optimization takes about 6,000 queries to recover; the code's advantage grows with the fact count, because it stays at 64 slots. On our synthetic platforms (55–57 facts per document on average), the fact list has a median of 1,116 / 1,145 tokens and takes 153 ms per query to re-read, or 82 / 81 ms cached in 61 / 63 MiB; the code answers in 71 ms with 17–18× less memory than the cached list and recovers its 50 s optimization after about 600 queries against re-reading.

The revised version will add this table and the break-even analysis.

Q2. Corrected answers, coverage and exposure. Corrected (fact-graph) answers alone recover 64% / 19% of the grammar's lead over unconstrained generation. The self-study questions recover 86% / 47% once every fact they ask about also gets QGC's training exposure, leaving 0.035 [0.002, 0.067] / 0.211 [0.165, 0.258]:

Training questions, same budgetPlatform 1Platform 2
Unconstrained generation0.5880.390
Same questions; fact-graph answers wherever a question targets an admitted fact0.7440.464
Same questions, those targeting an admitted fact; fact-graph answers; each fact they ask about trained at the grammar's exposure0.7960.578
Per-fact LLM paraphrases, fact-graph answers, the grammar's coverage and exposure (Table 2)0.8670.857
Compiled grammar0.8310.789

The rest sits on facts that the generated questions ask in few ways or not at all. On independently written questions asking for a fact's value (Llama-3.1-8B, same facts), the grammar's lead falls steadily with the number of distinct generated questions per fact. For facts asked in one or two ways (5.5% / 16.3% of the facts they ask about) it is 0.292 [0.115, 0.468] / 0.606 [0.480, 0.731]; for facts asked in eleven or more (49% / 41%), −0.068 [−0.107, −0.029] / −0.116 [−0.208, −0.025]: there the generated questions do better. Generated phrasings thus serve a code as well as the grammar's once a fact has enough of them, as the same generator's 27 paraphrases per fact show. What a code needs is every fact, with correct answers, asked in enough ways at adequate exposure. The revised version will add these conditions to the matched-budget comparison.

AuthorsOpenReview ↗

Combined supervision closes the remaining gap to the extracted facts in about half their context

In our reply to your weakness and Q1, per-fact LLM paraphrases brought the code within 0.015 of the extracted facts. Two new results on the same 100 records close that gap and show what the 64 slots save.

Accuracy. To test whether the remaining gap is inherent to the representation, we vary only the supervision, with the 64-slot code, frozen reader and evaluation fixed. At twice the grammar's per-fact exposure, training on the grammar's questions and the paraphrases together brings the code to 0.950, level with the extracted facts (0.948; difference +0.002 [−0.007, +0.012]). The same doubled exposure spent on the paraphrases alone gives 0.937, so the gain comes from combining the two sources (+0.013 [+0.004, +0.022]), not from more training examples, the distinction your review credits in the factorial.

Context. What the reduction in slots saves is room in the reader's context: the code takes 64 positions where the text takes a median of 116, at the same cost per query (100.7 vs 102.3 ms, as in our reply). Shortening the text cannot match it: even the bare values, without field names, need a median of 78 tokens, more than the code's 64 positions. Held to those 64 positions, the extracted facts themselves score only 0.557 even when the reader rewrites them as compactly as it can (0.496 when truncated), against 0.910 for the code trained on the grammar alone (+0.353 [+0.319, +0.387]; ahead on 98 of 100 records).

Trade-off. Of the two costs you weigh, the lower accuracy no longer holds; what remains is preparation, about 40 s more per record than the facts, for the same accuracy in about half the context. The text is the simpler choice when records change often, the code when context per record is the binding budget. The revised PDF adds this comparison to §6.2 and App. H.

Reviewer_1b4tOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a small memory for each document so a frozen 7B model can answer questions from it without reading the full document. The training questions are built from facts extracted from the document, using a fixed set of templates that vary how each question is phrased. Every fact gets the same number of training questions. On two synthetic datasets, more varied phrasing improves accuracy a lot, while finer details of how questions are mixed make no detected difference. LLM-written paraphrases tied to the extracted facts work as well as or better than the templates, while questions the model writes freely about the document do much worse. On 100 ClinicalTrials.gov records, the trained memory scores 0.910, compared with 0.869 when the model reads the full document.

Critical strengths
  1. The paper reports full results tables and confidence intervals and releases code and per-document results.
  2. Good experiment and especially control experiment setup, which increases the trustworthiness of the results: When the answers are shuffled between facts, accuracy drops to the same level as having no context. This shows the codes actually store the facts rather than just learning how questions are phrased. On the real data, the authors also show which errors come from the code and which come from fact extraction.
Critical weaknesses
  1. Training on more varied phrasings helping the model answer unseen phrasings is already known from work on paraphrase diversity (e.g., Allen-Zhu & Li, 2024). Is this problem novel enough?
  2. The self-study baseline feels weaker than it should be. The same 7B model writes the questions and answers, and those answers are used without any checking. Only 21–34% of them actually match a fact in the document. It’s also trained differently from Cartridges, which uses context distillation. So I’m not sure the big drop shows that generated questions don’t work. It may just reflect a weak setup, and a fairer comparison would follow the original Cartridges recipe more closely.
  3. I also noticed that the best results come from LLM paraphrases tied to the extracted facts, not from the hand-written grammar. To me, the main takeaway is cover every fact and use the correct answers, which makes the grammar itself seem less important than the paper suggests.
Questions for authors
  1. Giving the model the extracted facts as text already scores higher than the trained code. When would someone choose to train a code instead? Is there a case where the code does better, such as with many documents, very long documents, or documents with many facts?
  2. Do the main results still hold if you remove the yes/no questions? Also, how large would a third-order effect need to be for your test to detect it?
  3. Can you compare with a faithful Cartridges setup (stronger generator, context-distillation objective) at the same budget?
1 comment
AuthorsOpenReview ↗

Generated questions match the grammar once compiled per fact; neither better answers, a larger generator nor context distillation closes the gap

Thank you for the careful reading, and for singling out the label derangement and the error accounting as the controls that make the results trustworthy.

W1. Novelty relative to paraphrase diversity. We agree that phrasing diversity helps facts become extractable; Allen-Zhu & Li show this for knowledge stored in weights during pretraining (§2). What is new is which structure of the question distribution a frozen reader's per-document code needs, measured at exact per-fact exposure, which that study does not isolate:

  • the axes are not equal: the point estimates rank word order > wording > frame in all four settings (two platforms, a second reader family, ClinicalTrials.gov);
  • every pair is subadditive, while the tested third-order family and exact cell balance show no detected response;
  • at the same budget, unconstrained generation loses 0.24–0.40;
  • the word-order penalty is mainly positional: permuting only a question's position indices reverses which single-order code does better (9RVJ, W2).

W2 and Q3. Strength of the generated-question baseline. At the same budget, both changes you name, a stronger generator and the context-distillation objective, leave most of the gap; corrected answers and exposure narrow it further, and compiling the questions per fact closes it:

Training questions, same budgetPlatform 1Platform 2
Unconstrained generation0.5880.390
Same, Cartridges' context-distillation objective0.6060.428
Same recipe, Qwen2.5-14B (twice the reader's size) as generator0.6690.523
Same questions; fact-graph answers wherever a question targets an admitted fact0.7440.464
Same questions, those targeting an admitted fact; fact-graph answers; each fact they ask about trained at the grammar's exposure0.7960.578
Per-fact LLM paraphrases, fact-graph answers, the grammar's coverage and exposure (Table 2)0.8670.857
Compiled grammar0.8310.789

Context distillation trains the code to imitate the reader given the full document (0.778 / 0.728, Table 2), and both distilled codes stay below that teacher, which the compiled code exceeds. Under distillation the grammar's questions still lead the generated ones by 0.067 [0.025, 0.110] / 0.170 [0.099, 0.242], and the compiled code leads distilled generation by 0.225 [0.175, 0.274] / 0.361 [0.315, 0.407].

W3. "Cover every fact and use the correct answers." Agreed, with one condition the factorial adds: adequate question support. Coverage and correct answers are necessary but not sufficient: a code trained on every fact with correct answers but one phrasing per relation trails the full grammar by 0.38 / 0.47 (Table 3). Likewise, with fact-graph answers and the grammar's exposure, self-study questions trail the grammar on independently written questions asking for a fact's value by 0.292 / 0.606 for facts they ask in one or two ways, but lead it by 0.068 / 0.116 for facts asked in eleven or more. The grammar is the instrument that makes these effects identifiable, and a container for any phrasing source: per-fact LLM paraphrases are QGC with an LLM filling the cells.

Q1. When to train a code rather than give the facts as text. On these 10-field records, adding per-fact LLM paraphrases to the grammar's questions brings the code level with the extracted facts (0.950 vs 0.948), as fast per query; held to the code's 64 positions, the facts themselves score only 0.557. The fixed 64 slots matter in the three cases you list:

  • Long documents. Re-reading costs grow with the document; the code's do not: on these 2,320-token records, the code saves 212.7 ms per query and recovers its 27.8 s preparation after about 130 queries.
  • Many documents. Against a cached document the code is as fast with 36× less memory (3.5 vs 126.9 MiB), and memory grows with the number of resident documents.
  • Many facts. On our synthetic platforms (55–57 facts per document on average), the fact list has a median of 1,116 / 1,145 tokens and takes 153 ms per query to re-read, or 82 / 81 ms cached in 61 / 63 MiB. The code answers in 71 ms with 17–18× less memory than the cached list and recovers its 50 s optimization after about 600 queries against re-reading.

Q2. Yes/no items and the detectable third-order effect. Without the yes/no items, every conclusion holds and most strengthen: main effects keep the order O > L > F on both platforms (+0.280 / +0.135 / +0.077 and +0.262 / +0.218 / +0.136), all pairwise terms stay negative, the three-way term remains undetected (0.000 [−0.011, +0.010] / −0.005 [−0.025, +0.015]), and the gap to unconstrained generation widens to 0.361 / 0.556. On the original evaluation, a factorial three-way effect of 0.016 / 0.023, or a change of 0.025 / 0.046 along the λ-path over λ ∈ [0, 0.75], would be detected with about 80% power at α = 0.05; the weakest pairwise terms are −0.028 / −0.022 (Table 1).

The revised version will add all analyses reported here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.