acceptodds
Under review as a conference paper at ICLR 2027

Deterministic Addresses Constrain Reads, Not Ordinary Writes: Deletion Boundaries in Conditional-Memory Language Models

Abstract

Conditional-memory language models retrieve from deterministic n-gram-addressed tables, making table state directly inspectable and removable. Yet an address identifies where a model reads, not where gradient-based adaptation writes. Unlike causal-tracing locations, these addresses are computed by the architecture rather than estimated. We test whether they form a deletion boundary by grafting exact and hashed n-gram memory into pretrained Pythia (410M, 1.4B) and Qwen2.5 (0.5B, 1.5B), fine-tuning rare trigger–continuation mappings at preregistered exposure counts, and intervening on rows, grafts, and backbones. Across both families, behavior reaches near-ceiling success when the graft is frozen. In Pythia, symmetric swaps localize the learned mapping to the backbone: a clean graft preserves it, a poisoned graft on a clean backbone yields zero success in all ten swaps, and target-row deletion is inert. The same deletion produces model-level mean target-specific attack-success reductions of 98.3 to 100 percentage points for writes deliberately confined to addressed rows. In a separate preregistered durability assay, deleted mappings remain at zero observed exact-match success through 512 later optimization steps, retained Adam state, and one constructed exact 16-row collision. A paired four-rate sweep shows optimizer policy can move causal dependence into the table at a sharp, scale-dependent threshold. At the largest rate, whole-table state exceeds the registered 0.15 attack-success sufficiency contrast in 13/16 1.4B and 11/16 410M runs, both observed majorities; only 1.4B passes the registered majority-confidence rule. Clean perplexity rises 8.0% at 410M and 7.1% at 1.4B, while final-row sufficiency exceeds 0.15 in only 7/16 runs at each size. Our scope is retrofitted 0.4B–1.5B models and rare-string, single-token probes. Deterministic addressing constrains reads but does not by itself create an auditable deletion boundary; that requires an enforced write path and post-adaptation location verification.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
conditional memory, memory-augmented language models, knowledge localization, machine unlearning, deletion guarantees, write locality, model editing, mechanistic interpretability, backdoor learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qjWJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper investigates whether deterministic n-gram memory addresses provide a reliable deletion boundary after language-model fine-tuning. On memory-augmented Pythia and Qwen models, this authors study synthetic trigger–continuation mappings through freezing, row deletion, and component-intervention experiments. They find that ordinary fine-tuning can encode these mappings in the backbone, while explicitly row-constrained writes can be removed by deleting the corresponding rows. Moreover, higher table learning rates can increase memory dependence, but incur quality costs and do not guarantee row-level locality.

Critical strengths

S1. The paper conducts an extensive set of experiments to investigate the storage and deletion of learned mappings.

S2. It shows that deterministic read addresses do not guarantee that information learned through ordinary fine-tuning remains confined to the corresponding table rows.

Critical weaknesses

W1. Limited novelty of the central finding. When parameters remain trainable, the backbone’s ability to learn new mappings is an expected consequence of joint optimization. The authors acknowledge that prior work does not claim that ordinary fine-tuning preserves row-level locality. It therefore remains unclear what substantive new insight the paper offers beyond empirically confirming that deterministic read addresses alone do not guarantee localized writes.

W2. Lack of a unified explanation and a coherent argument. The experiments appear loosely organized and lack a clear logical progression. While component swaps and learning-rate sweeps provide useful observations, these are not synthesized into a coherent explanation of storage locality or generalizable design principles. The paper therefore reads likely as a collection of experiments on specific implementations, with an unclear broader contribution.

Questions for authors

Q1. What substantive new insight does the paper offer beyond the expected observation that a trainable backbone can learn new mappings without updating the memory table?

Q2. What conditions determine whether learned information resides in the backbone or becomes localized to specific memory rows, and how do the component-swap and learning-rate experiments jointly support this explanation?

Reviewer_8Pd9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines whether deterministic memory addresses define reliable deletion boundaries after fine-tuning. Through retrofitted memory modules, backbone–graft swaps, deletion interventions, and learning-rate sweeps, it shows that trigger mappings can reside outside the addressed rows and that storage location depends on the training configuration.

Critical strengths
  1. Symmetric component swaps and row-restricted positive controls provide useful evidence about where the tested mappings are stored.
  2. The paper transparently documents negative results, failed extensions, and implementation limitations.
Critical weaknesses
  1. Limited novelty and significance. Deterministic read addresses do not constrain updates to a trainable dense backbone. Demonstrating that unrestricted fine-tuning stores information outside those addresses largely confirms an expected limitation, rather than overturning an established assumption. The paper explicitly acknowledges that prior conditional-memory systems do not claim unrestricted fine-tuning preserves row locality.
  2. Key comparisons do not isolate the claimed factors. Ordinary fine-tuning and direct row writes differ in objective, exposure count, and learning rate, preventing attribution to write-path restriction alone. G6/G11 use BF16 updates without FP32 master parameters, leaving the learning-rate transition confounded with numerical update resolution. Some G11 arms additionally match intended rather than realized parameter displacement. Consequently, the optimizer-based explanation remains insufficiently established.
Questions for authors
  1. Does backbone-dominated storage persist in a model with substantial, measured dependence on conditional memory?
  2. Can matched objectives, exposure counts, and learning rates isolate the effect of restricting the write path?
  3. Would the reported learning-rate transition survive FP32 master parameters and matched realized update budgets?
  4. Is the deletion result reproducible for multiple interacting mappings or semantically equivalent triggers beyond rare-string, single-token probes?
Reviewer_PmmKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests if deterministic memory addresses actually give reliable deletion boundaries once a language model has been fine-tuned. In experiments on Pythia and Qwen retrofits, the authors show the mappings can end up learned in the backbone instead, so deleting the addressed rows doesn’t work. They use component swaps, row-restricted positive controls and a paired learning-rate sweep to tell apart memory capacity from where the adaptation really stores the behavior. Raising the table learning rate increases dependence on the memory, but it costs quality and doesn’t consistently concentrate the mapping in it’s final addressed rows.

Critical strengths

Swapping the poisoned graft for its clean counterpart keeps 99.65% and 99.49% attack success, and the reverse swaps give zero in all ten Pythia runs. This is strong support for backbone storage

The row-restricted controls show deletion works when the storage is genuinely localized, with target-specific removal reaching 98.3–99.6 percentage points in Pythia and 100 in Qwen

The four-rate sweep uses 16 paired seeds per size and reports routing as well as quality costs. It shows table dependence can go up without reliable final-row sufficiency

Critical weaknesses

Sections 4.1 and 6 report that memory contributes only 0.001395 nats to clean prediction in Pythia-1.4B, while the joint-pretraining experiments do not establish routing. Backbone dominance therefore remains untested in a successfully evaluated, jointly trained memory model.

Section 6 and Appendix I show that G6 and G11 update BF16 parameters without FP32 master weights, while G11’s custom arms match intended rather than realized updates. Numerical precision therefore remains an unresolved explanation for the learning-rate transition and policy differences.

Section 6 and Appendix A show that successful routing experiments use rare-string, single-token mappings. The three-token successor fails calibration, so no routing experiment is conducted for it. The findings therefore provide limited evidence about deleting natural, multi-token knowledge.

Questions for authors

Does the learning-rate transition still happen with FP32 master weights and optimizer states, in particular at the ordinary table learning rate?

Across the existing checkpoints, is a stronger clean-language dependence on the graft linked to more table storage after fine-tuning?

Could the released artifact include the checkpoint states and exact memory banks needed to reproduce the component swaps independently, since Appendix K says they are stored seperately?

Comments on the paper

1 comment
Area_Chair_S9mMOpenReview ↗

Halfway through the discussion period

Dear reviewers,

Thank you for your continued participation in the review process. The authors have responded to the reviews. When you get a chance, please acknowledge their response and, where appropriate, please respond to their comments.

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.