acceptodds
Under review as a conference paper at ICLR 2027

Conditioning the Write: Semantic Cues for Continual Long-Context Parametric Memory

Abstract

Continual parametric memory stores source information in model parameters so that questions can be answered without the source text. We study whether the input conditions under which fixed question–answer (QA) targets are written affect later recall. Our method supplies question-derived (QD) cues that describe a question group's shared setting, or evidence-removed (ER) cues that retain source background after designated answer evidence is deleted. Because these cues are unavailable at evaluation, we alternate cued and bare questions while keeping answer supervision and optimizer updates fixed. Across 100 matched LIFT paths, QD and ER improve mean final F1 from 41.54 to 44.81 and 44.70. On ten whole-dialogue LoCoMo writes, compact ER reaches 25.68 F1 using 8.0% of full ER's total training tokens, although full ER has a lower peak drop. An eight-seed study on three fixed scenes shows that cue gains also persist with on-policy context distillation, whose additional benefit depends on the cue. These results support input conditioning for improved recall of written answers.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parametric memory, semantic conditioning, continual learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9CW3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether training-time input conditioning improves continual parametric memory when evaluation provides only questions. It introduces question-derived (QD) and evidence-removed (ER) cues, alternating cued and bare-question presentations while keeping QA supervision and optimizer updates matched. Across 100 sequential LIFT paths, QD and ER improve final F1 from 41.54 to 44.81 and 44.70 respectively. On independent whole-dialogue LoCoMo writes, compact ER achieves similar final recall to full ER using approximately 8% of its training tokens. Additional experiments examine cue schedules, context distillation, and local optimization effects.

Critical strengths

The compact-cue experiment provides a useful practical result. Compact ER achieves 25.68 final F1 compared with 25.43 for full ER. Reporting the accompanying peak-drop tradeoff makes this comparison more informative than reporting final recall alone.

Critical weaknesses

None

Questions for authors

What differences between the main experiment and Table 6 explain the reversal between QD and Generic?

3 comments

Thank you for recognizing the practical value of compact cues and for asking about the QD/Generic reversal. We clarify the comparison below. Please let us know if any point remains unclear. F1 is on a 0–100 scale. ± denotes sample SD. We will incorporate the corresponding revisions and clarifications into the main text.

Q1. What differences between the main experiment and Table 6 explain the reversal between QD and Generic?

  • The reversal concerns template QD under a different writing protocol. Table 6 and the main experiment both use Qwen3-8B with rank-128 all-linear LoRA and question-only evaluation, but differ in how the background is constructed and used:

  • QD construction: The main study generates a cue from the full question group, with an anchor-based fallback. Table 6 uses a fixed template with four frequent question anchors. The main cue can describe the shared setting and question dimensions. The template represents the group through four selected anchors.

  • QA coverage and evaluation: The main study generates up to ten QA per source sentence and evaluates multiple original questions per context. Table 6 generates ten QA for one target sentence per context and evaluates one question per context.

  • Writing protocol: The main study uses 100 ten-context paths, five epochs per context, and deterministic QA-by-epoch alternation. Table 6 uses one ten-context sequence with two seeds and 150 total updates, alternating 75 cued and 75 bare optimizer updates.

  • The acquisition and retention results explain the observed ranking. In Table 6, template QD achieves higher immediate acquisition than Generic, but also much greater subsequent forgetting. Its initial advantage therefore does not survive to the final question-only evaluation. The two-run means are:

CueAcquisitionForgettingFinal F1
Template QD73.9033.7043.70
Generic60.3015.4052.40
Matched ER64.9014.9055.20
Mismatched ER59.0024.6041.90
  • Matched ER achieves the highest final F1 in this same probe. It exceeds Generic (55.20 vs. 52.40) and mismatched ER (55.20 vs. 41.90). Thus, Table 6 supports the value of episode-matched background even where template QD shows greater acquisition and forgetting. In the original main experiment, QD instead reaches 44.81 versus 42.82 for Generic. This reversal concerns different QD constructions and writing protocols. Because these choices changed together, we test cue content directly using the matched controls below.

  • Under the main protocol, QD and ER exceed all corresponding controls in mean final F1. We completed nine conditions on the same 100 paths, sharing QA, initialization, write order, and update counts. All prefixed conditions use the same alternation and answer-only supervision. Generic repeats/truncates a fixed, episode-independent instruction. Generic, Random, and mismatched cues match QD or ER separately in token length and answer position.

Length settingQA-onlyGenericRandomMismatchedMatched cue (Ours)
QD length41.98±6.9842.85±6.8541.93±6.8542.79±6.9644.16±6.97
ER length41.98±6.9843.03±6.4442.81±7.8243.15±7.0744.70±6.62
  • Scores are mean±path SD. Matched cue denotes QD/ER in the respective row, and QA-only is shared. Paired gains over Generic are 1.31±5.85 for QD and 1.67±5.45 for ER. The mean advantage remains after matching prefix length, answer position and presentation schedule, supporting an added contribution from episode-matched content. These new runs are separate from the original main results and their statistical tests.

  • The schedule ablation independently supports how the cues are used. On the original 100 paths, always-cued QD/ER reach 41.68/41.43 final F1, close to QA-only at 41.54. Alternation reaches 44.81/44.70 with QA, answer exposure and updates unchanged. Bare-question presentations help preserve access when cues are removed. The contribution is therefore the joint design of useful background and training presentations that support later question-only recall.

  • The mean document advantage also holds at 14B. On the same 100 paths:

PrefixFinal F1
QA-only45.84±7.07
Generic (QD length)46.67±6.88
QD47.95±6.71
Generic (ER length)46.18±7.19
ER48.09±7.40
  • QD and ER exceed their length-matched Generic controls in mean final recall. Together, the matched 8B and 14B comparisons support their value in the tested document-writing setting. We will label Table 6's cue as template QD, clarify its 150-update total budget, and distinguish this probe from the main generated-QD experiment and new matched controls.

Thank you for your clarification, and I would like to maintain my original rating as Weak Acceptance.

Thank you for your follow-up and positive comments of our work on semantic cues for continual parametric memory. Your question helped us clarify the experimental protocols and more precisely explain the scope of our findings. We sincerely appreciate your time and constructive feedback throughout the review process.

Reviewer_HPuMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether training-time input conditioning can improve the question-only recall of information stored in language model parameters during continual memory writing. The authors propose two semantic cues: Question-Derived (QD) cues, which describe the shared context of a question group without accessing the source text or answers, and Evidence-Removed (ER) cues, which preserve background information after designated answer evidence is removed. To prevent the model from becoming dependent on prefixes that are unavailable at inference time, the method alternates between cue-conditioned and bare-question training while keeping the QA supervision and optimizer updates matched. The method is evaluated using Qwen3-8B with LoRA adaptation on sequential LIFT memory-writing paths and whole-dialogue LoCoMo settings. Across 100 matched LIFT paths, QD and ER improve mean final F1 from 41.54 to 44.81 and 44.70, respectively. On ten LoCoMo dialogues, both methods also improve final recall over QA-only training. The authors further introduce compact ER (cER), which achieves a final F1 of 25.68 while using approximately 8% of the total training tokens required by full ER. Additional experiments investigate cue schedules, cue-content sensitivity, context distillation, and local gradient effects. Overall, the paper provides evidence that carefully designed training-time context can improve later recall without requiring that context at inference time. However, the generality of the findings and the extent to which the gains arise specifically from semantic rather than generic input conditioning require further validation.

Critical strengths
  1. Controlled experimental design that isolates training-time input conditioning. The authors make a meaningful effort to control confounding factors by keeping QA supervision, initialization, training order, and optimizer updates matched across the main comparisons. The alternating schedule also directly addresses a realistic deployment constraint: the cue is available during memory writing but not during evaluation. The improvements observed on both LIFT and LoCoMo provide useful evidence that the effect is not restricted to a single experimental setting.
  2. Practical analysis of the efficiency–recall trade-off. The compact ER variant provides a useful efficiency result. It achieves a final F1 of 25.68 on the reported LoCoMo evaluation while using approximately 8% of the total training tokens required by full ER. The comparisons between direct and alternating cue use, together with the analyses of forgetting and cue-content sensitivity, also help characterize when input conditioning may be useful. These results make the paper more informative than a simple demonstration that adding a prompt can improve recall.
Critical weaknesses
  1. Limited evidence for generalization beyond the evaluated model and data. All principal QA-based experiments use Qwen3-8B with LoRA adaptation. The main sequential LIFT evaluation uses 100 matched writing paths, but these paths do not constitute 100 independent datasets. The whole-dialogue LoCoMo evaluation contains only ten dialogues, while the context-distillation study uses three fixed scenes. Repeated seeds on the same scenes primarily measure optimization and rollout variability, not variability across different contexts. Consequently, it remains unclear whether the observed gains generalize to other model families, model scales, or substantially different memory-writing settings. The empirical evidence supports the proposed approach in the tested configurations, but broader claims about continual parametric memory should be made cautiously.
  2. The semantic contribution and statistical robustness of the improvements are not fully established. On LIFT, the reported mean final-F1 improvements over QA-only are 3.27 points for QD and 3.16 points for ER. These are encouraging but moderate gains. The paper reports mean paired improvements, yet the main results would be more convincing with confidence intervals or paired significance tests that account for the experimental sampling structure. More importantly, the evidence that the semantic content of the cues is the principal source of improvement is mixed. A generic prefix also improves performance in the main comparison and outperforms QD in the small two-run cue-content study. Although matched ER performs better than mismatched ER in that study, the limited number of runs makes it difficult to establish a robust ranking of cue types. The authors also acknowledge that residual text in ER may contain redundant answer support, leaving some uncertainty about the extent to which the cues are genuinely answer-blind. Together, these issues limit the strength of the causal interpretation and the confidence with which the gains can be attributed specifically to semantic conditioning.
Questions for authors
  1. Statistical reliability of the reported gains. Could the authors provide paired confidence intervals and appropriate statistical tests for the improvements over QA-only and the generic-prefix baseline, particularly the +3.27 and +3.16 final-F1 gains on LIFT? For the 100 matched paths, how are dependencies arising from shared or overlapping source contexts handled? Given that the LoCoMo evaluation contains only ten dialogues, how sensitive are the conclusions to individual dialogues? This information would help determine whether the reported improvements are robust rather than specific to the sampled writing paths or contexts.
  2. Validation of answer-blind cue construction. The paper notes that residual ER text may still contain redundant answer support. Beyond lexical filtering, how do the authors verify that QD and ER do not inadvertently expose the target answer or provide sufficient alternative evidence to recover it directly? Could the authors quantify the frequency of residual answer support and clarify whether the main results persist under stricter semantic leakage checks? This clarification is important for distinguishing improved parametric memory writing from effects caused by answer-relevant information remaining in the training inputs. Additional validation that would materially strengthen the assessment
  3. Generalization across model families or scales. Have the authors evaluated the proposed approach on another base model or model scale? If not, could they clarify the extent to which the findings are expected to depend on Qwen3-8B and qualify the broader claims accordingly? Validation on an additional model would strengthen the case that the method is generally applicable to continual parametric memory.
3 comments

We sincerely thank you for recognizing the controlled design and efficiency analysis. We will incorporate the corresponding revisions and clarifications into the main text. Please let us know if any point remains unclear.

W1 & Q3. Limited evidence for generalization beyond the evaluated model and data. Generalization across model families or scales.

  • New 14B experiment. In response, we completed a new Qwen3-14B experiment with five conditions on 100 matched document paths. QA-only scores 45.84±7.07 F1, QD 47.95±6.71, and ER 48.09±7.40. QD- and ER-length Generic score 46.67±6.88 and 46.18±7.19. LoRA capacity is identical across conditions. Both cues exceed QA-only and Generic in mean F1, extending the result beyond Qwen3-8B to a second model scale.

  • Writing settings. This new scale test complements the original document and whole-dialogue experiments and the three-scene objective study. Eight paired seeds measure optimization and rollout variability within those scenes. These evaluations test cue conditioning across model scale, writing granularity, and training objectives.

W2. The semantic contribution and statistical robustness.

  • QD/ER exceed all length-matched controls in mean document F1. To address the small two-run probe, we completed nine conditions on 100 matched paths (900 condition–path runs). QA, initialization, order, and updates are shared. All prefixed conditions use the same alternation. Generic repeats/truncates a fixed, episode-independent instruction. Generic, Random and mismatched cues match QD/ER separately in token length and answer position.
Length settingQA-onlyGenericRandomMismatchedMatched cue (ours)
QD length41.98±6.9842.85±6.8541.93±6.8542.79±6.9644.16±6.97
ER length41.98±6.9843.03±6.4442.81±7.8243.15±7.0744.70±6.62
  • Scores are mean±path SD. Matched cue is QD/ER in its row. QA-only is shared. Separate sequential LoCoMo controls favor Generic over QD/ER (30.29 vs. 28.55/27.93), so the additional semantic-benefit claim is specific to documents. Within this setting, the controls support a contribution from matched background beyond prefix length and cue presentation.

Q1. Statistical reliability of the reported gains.

  • Paired gains. Values below are mean±sample SD of paired F1 differences over 100 paths or ten dialogues from the original runs.
Contrastdocumentswhole-dialogue LoCoMo
QD − QA-only3.27±5.781.08±0.83
ER − QA-only3.16±5.861.78±1.39
QD − Generic1.99±5.74−0.54±1.48
ER − Generic1.88±5.660.16±1.57
  • Paired tests. Paired label swaps use 200,000 document assignments and all 1,024 dialogue assignments. Holm correction adjusts p-values to control the chance of at least one false-positive finding across five comparisons per protocol. These tests concern original runs, not the new controls.
CueDocument p: QA-only / GenericLoCoMo p: QA-onlyLoCoMo leave-one-out gain
QD<0.0001 / 0.002190.023440.94–1.25
ER<0.0001 / 0.002190.023441.54–2.05
  • Statistical support. Table p-values are Holm-adjusted. All four original document contrasts remain significant after correction, supporting the gains over both QA-only and Generic.

  • Dialogue sensitivity. LoCoMo gains over QA-only stay positive when any dialogue is removed, so their direction is stable to omitting a single dialogue. An advantage over Generic is not established on LoCoMo.

Q2. Validation of answer-blind cue construction.

  • Construction. QD generation sees questions without source text or answers. Lexical checks use answers afterward. ER removes designated evidence but may retain redundant support.

  • Direct recovery. Frozen Qwen3-8B without adapters generated 44,220 answers (7,370 QA × six inputs), greedily with a 96-token limit. Each input includes the question. F1 is mean±SD across 119 context means. Cap is the pooled percentage reaching 96 generated tokens without EOS.

Additional inputRecovery F1Cap (%)
None (question only)9.97±3.4661.15
QD11.02±3.7166.24
QD-length Generic12.58±4.7728.56
ER11.97±4.0354.40
ER-length Generic11.66±4.2729.82
Answer-supporting evidence28.05±6.8330.28
  • Evidence control. The reader receives source text designated as evidence for that question. This positive control raises recovery from 9.97 to 28.05 F1. QD/ER differ from matched Generic by −1.56/+0.31 F1. These frozen-reader results are separate from trained-memory recall. Evidence exact match is only 0.85%, and unequal cap rates limit interpretation.

  • Interpretation. The trained document gains over Generic are not mirrored by a consistent increase in direct recovery in this diagnostic.

Thank you for the detailed response. I appreciate the authors' efforts to address the review comments. These revisions clarify some of my concerns and improve the completeness of the paper. However, considering the paper's contributions, experimental quality, and remaining limitations, I decide to maintain my original rating.

We are pleased that our response and revisions have helped clarify some of your concerns and improve the completeness of the paper. Our motivation is that conditioning QA fine-tuning on semantic context may help incorporate new knowledge while limiting interference with previously learned knowledge. So, we combine QD/ER cue construction with alternating cued and uncued training to support recall after continual updates, without requiring cues or source context at inference.

In our LIFT-based document experiments, with identical QA supervision and update budgets, we obtain the following final-F1 gains:

ModelCueOver QA-onlyOver Generic
Qwen3-8BQD+2.18±5.46+1.31±5.85
Qwen3-8BER+2.72±4.88+1.67±5.45
Qwen3-14BQD+2.11±5.76+1.28±5.16
Qwen3-14BER+2.25±6.11+1.91±6.40

Values are mean gains±paired SD in F1 points. Generic uses the same alternation and matched prefix length.

These results show overall gains over QA-only and additional gains beyond generic conditioning across both cue types and model scales. Could you clarify which remaining concern carries the most weight in your assessment? We sincerely thank you for your time and constructive comments throughout the review process, and appreciate your suggestions on the key remaining concerns. This would help us prioritize further improvements to the method.

Reviewer_zd7COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether adding input context during parametric memory writing improves later question-only recall. The authors introduce two types of cues: question-derived (QD) generated from a question group’s shared setting, and evidence-removed (ER) derived from source context after removing explicit answer evidence. Cued and bare questions are alternated during training. Compared to the QA-only baseline, QD/ER improve final F1 from 41.54 to 44.81 and 44.70 on LIFT, and improve from 23.66 to 24.73 and 25.43 on LoCoMo.

Critical strengths

Strengths:

  • The paper presents a clean and controlled empirical study.
  • The alternating training schedule is useful.
Critical weaknesses

Weaknesses:

  • Methodological novelty is limited, the main contribution is an empirical study, but the current evaluation and comparisons are narrow and limited: one base model, two benchmarks, not fully test paraphrased or human written questions. The paper also provides limited comparison to existing memory-writing, continual-learning, or retrieval methods.
  • The contribution of semantic cue is unclear. In small fixed content control, generic and random id both achieve better final F1 score than QD (table 6). This suggests part of the gain may come from extra conditioning/input diversification/regularization rather than semantic conditioning itself.
  • Weak statistical support.
Questions for authors

Questions:

  1. Do QD/ER outperform a same length random/generic prefix or a mismatched cue in the 100-path setting?
2 comments

Thank you for recognizing the controlled design and alternating schedule. We address your concerns below; please let us know if any point remains unclear. F1 is on a 0–100 scale; ± denotes sample SD. We will incorporate the corresponding revisions and clarifications into the main text.

W1. Methodological novelty, evaluation and comparisons are limited

  • QD constructs background from training questions; ER removes designated answer evidence from source text. We alternate cued and bare QA for later question-only recall. LIFT already uses contextualized auxiliary QA; our contribution is to test background construction and presentation under fixed answer supervision and updates.

  • The schedule ablation tests this design directly. On the same 100 paths, always-cued QD/ER reach 41.68/41.43 final F1, close to QA-only at 41.54. Alternation raises these scores to 44.81/44.70 with QA, answer exposure and updates unchanged. Including bare questions therefore improves access when cues are absent.

  • The mean gains also hold at 14B. On 100 matched document paths, Qwen3-14B reaches 45.84±7.07 with QA-only, 47.95±6.71 with QD and 48.09±7.40 with ER. Paired gains over length-matched Generic are 1.28±5.16 and 1.91±6.40. This extends the document result to a second model scale within Qwen3.

  • Recall improves under different question wording. Document evaluation uses original human-written SQuAD questions after training on source-generated QA. In a separate LoCoMo study, an answer-free rewriter independently generates training and evaluation paraphrases of each original question. We train on one version and evaluate original questions or held-out paraphrases, excluding normalized exact matches to training:

EvaluationQA-onlyQDER
Original questions24.48±2.3029.64±1.9827.44±1.42
Held-out paraphrases22.39±3.6226.28±2.3524.77±3.05
  • Both cues improve recall on original questions and held-out paraphrases after exact training matches are removed. The gains therefore persist when the same information is queried with different wording, supporting access beyond the training question form.

W2. The contribution of semantic cues is unclear & Q1. Do QD/ER outperform same-length generic, random or mismatched cues on 100 paths?

  • The new document experiment compares QD/ER with trained, length-matched Generic, Random and mismatched cues. Both exceed every corresponding control in mean final F1. The mean advantage remains after controlling prefix length and cue presentation.

  • We completed nine conditions sharing the same 100 paths, QA, initialization, write order and updates. All prefixed conditions use the same alternation. Generic repeats/truncates a fixed, episode-independent memory instruction; Generic, Random and mismatched cues match QD or ER separately in token length and answer position. All receive the same answer-only QA supervision.

Length settingQA-onlyGenericRandomMismatchedMatched cue
QD length41.98±6.9842.85±6.8541.93±6.8542.79±6.9644.16±6.97
ER length41.98±6.9843.03±6.4442.81±7.8243.15±7.0744.70±6.62
  • Scores are mean±path SD from the new runs. Matched cue denotes QD/ER in the respective row, and QA-only is shared. The higher mean F1 supports an added benefit from episode-matched content under the same prefix length and training schedule. Combined with the schedule ablation, this supports both parts of the method: cue/bare alternation improves access without cues, and matched background further improves document recall.

W3. Statistical support is weak

  • We use 200,000 paired label swaps for the original 100-path gains. Holm correction adjusts p-values to control the chance of at least one false-positive finding across the five comparisons per protocol. All four contrasts remain significant after correction (adjusted p<0.05).
ContrastGain ± paired SDAdjusted p
QD − QA-only3.27±5.78<0.0001
ER − QA-only3.16±5.86<0.0001
QD − Generic1.99±5.740.00219
ER − Generic1.88±5.660.00219
  • Tests assume within-pair exchangeability. They concern 119 contexts from two articles, are not source-cluster-adjusted, and do not apply to the new control runs.

  • Original whole-dialogue LoCoMo gains over QA-only also remain significant for QD/ER (adjusted p=0.02344). Leaving each dialogue out retains gains of 0.94–1.25 and 1.54–2.05, respectively, and both remain positive.

  • The paired tests support the original recall gains within the evaluated source pool. The dialogue sensitivity check further shows that the positive LoCoMo result is not driven by any single dialogue.

Dear Reviewer zd7C,

Thank you again for your valuable feedback on our paper, particularly your suggestion to include length-matched controls to better assess the contribution of semantic cue content. As the discussion period approaches the November 18 deadline, we would like to briefly follow up on our response.

In the suggested 8B LIFT-based comparison, QD/ER improve mean final F1 by 2.18/2.72 points over QA-only and 1.31/1.67 over Generic, while also exceeding the random and mismatched controls. We additionally evaluated the methods at 14B and provided paired statistical analyses of the original results to broaden the evaluation and strengthen statistical support.

We would greatly appreciate hearing whether our responses have addressed your concerns. We remain available and ready to respond promptly to any further questions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.