- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A source model carries a LoRA "memory adapter" trained on past interactions; the interaction history is lost, and only the source model + adapter and the questions remain. FMC migrates the memory to a new backbone in three steps: query the source at the retained questions to recover answers, keep the prompt-stable ones, then fit a single closed-form ridge regression (no backprop) that turns the target model's internal states into logit boosts on the recovered answer tokens. The result is a small output-head-only adapter, installed in seconds. On three memory benchmarks it beats transferred and distilled baselines and reproduces the recovered answers exactly.
Critical strengths
- Real problem, clean abstraction. Memory continuity across backbone upgrades without replay data is legitimate and under-served; "transfer observable answers, not parameter coordinates" is crisp.
- Transparent/honest reporting. Pre-declared selection rules, sealed states, input hashes, bootstrap CIs, and disclosure of every confound — including the ones that indict the results. A model of honest reporting.
- The analytic install is genuinely cheap and verified equivalent to the iterative solve of the same objective at lower cost; the mechanism ablations (e.g., address–answer shuffling) are well designed.
Critical weaknesses
W1 — The headline result is a unit test, and the paper's own control shows a text file matches the adapter. The "250/250 recovered values reproduced verbatim" result is forced by construction: the ridge constraints literally set the boosted token's logit at those prefixes, and the address pool was itself admitted using temporary FMC writes with readback fidelity as the criterion. More importantly, the paper's own control — feeding the same recovered values to the same reasoner as text — produces identical final strings and identical F1 on all questions. In the evaluated pipeline, a small text file matches the compiled adapter; the one regime where parametric storage could win (adapter-only serving, no context window) is never evaluated. The paper concedes this ("not superiority to text storage") — but then the central experiment does not support the paper's thesis.
W2 — The one superiority claim over a standard alternative rests on a budget the authors admit was chosen against the baseline. Distilled LoRA receives exactly 2 epochs with min-train-loss checkpointing — 1/16th of the 32 epochs, over six-layer Q/K/V/O LoRA, that built the source memory itself (App. A.1) — and the appendix states the reporting scope "was chosen after inspecting earlier budget experiments, which showed that longer training can reverse comparisons." The abstract's "improves task scores over same-content distillation" therefore generalizes a result the authors' own archive contradicts at longer budgets. The seconds-scale installation savings are also immaterial next to the shared selection cost (thousands of seconds). And the mechanism that would justify "compilation" over ordinary distillation is never isolated: FMC is functionally a one-step closed-form distillation on self-generated data, and the single-gradient-step distillation baseline that would separate the two is absent.
W3 — Generalization and side effects are unmeasured. Much of KnowEdit "test" is paraphrase-level generalization of supervised items (most canonical addresses sit directly in the compiled pool). Behavior on unrelated prompts with the adapter active — precisely the risk of a head that boosts 250 answer strings' logits over the full vocabulary — is explicitly unmeasured. The MAB test set is 60 queries with no CI, LoCoMo F1 gaps carry no significance tests, and the two extra model pairs lack the Distilled baseline, so the central comparison exists for exactly one source–target pair.
Questions for authors
Major:
- Report the FMC vs. Distilled-LoRA comparison at the baseline's best budget (the archive experiments that reverse the comparison), and rewrite the abstract to match what survives.
- Evaluate the adapter-only serving regime (no context window, no BM25 + text path): can the compiled head answer the questions on its own? If not, the contribution reduces to head-only probe distillation matched by a text file.
Minor:
- Measure side effects: perplexity and instruction-following on generic prompts with the adapter active vs. inactive; add CIs for the MAB and LoCoMo gaps; report how many KnowEdit test paraphrases are near-paraphrases of compiled items.
- Add the single-gradient-step distillation baseline (matched recovered data, one SGD step) — if it matches FMC, "compilation" reduces to one-step distillation in closed form. Also compare against head + last-block LoRA to justify the head-only restriction.
5 comments
Response to Reviewer obb8 [2/3]
W2. On the distillation budget, the cost claim, the relation to one-step distillation, and the head-only restriction.
Response. We ran the comparison at the larger budgets, the single-step baseline, and the head + last-block comparison, and we rewrote the abstract.
Budget (Major question 1). In one fit, FMC matches 8 epochs of distillation on KnowEdit and stays ahead of 16 epochs on MAB and on the admitted LoCoMo pool, in less time than any budget (Table R2).
Table R2. Budget sweep on the primary pair. Parentheses give the fixed-pool installation time, excluding shared preparation.
| Task (metric) | Distilled LoRA: 2 epochs | 4 epochs | 8 epochs | 16 epochs | FMC |
|---|---|---|---|---|---|
| KnowEdit (accuracy, %) | 34.92 (5.10 s) | 43.75 (6.53 s) | 53.53 (9.47 s) | 64.54 (15.43 s) | 53.53 (3.44 s) |
| MAB (accuracy, %) | 60.00 (3.25 s) | 75.00 (4.15 s) | 85.00 (6.03 s) | 88.33 (9.74 s) | 90.00 (2.34 s) |
| LoCoMo (shared-reader F1) | 6.10 (7.44 s) | 6.84 (10.15 s) | 8.27 (15.30 s) | 8.70 (25.97 s) | 8.74 (6.30 s) |
FMC is at or above distillation in 11 of the 12 cells. Distillation's score also depends strongly on the epoch budget, which has to be chosen on the target, whereas FMC has no such choice. Distillation is ahead on KnowEdit at 16 epochs, so the abstract no longer claims higher scores from a single budget. Tables 1 and 2 of the revised paper report the sweep, and the abstract's sentence on distillation now reads: "Across the evaluated memory tasks, FMC matches or exceeds same-content distillation in most training-budget settings and, given fixed content, reaches comparable scores with a 2.8–4.2× shorter fitting step."
Cost. On LoCoMo the selection stage dominates the end-to-end time, as the reviewer says: it takes 2,626.5 s, both methods are fitted on the pool it selects, and the end-to-end times are 2,632.8 s and 2,634.0 s. KnowEdit and MAB have no such stage (Section 3.3), so there the fitting step of Table 2 is the only target-side computation in which the methods differ. The revised Section 4.3 states the claim as a cheaper fitting step given fixed content.
Mechanism. We agree with the reviewer's description: FMC is response distillation whose installation step is solved in closed form, and Section 2.3 places it there. What the closed form computes is the converged solution of its objective, not the first gradient step. On the same objective, the iterative solver reaches the same 54/60 after 21.36 s of solving, against 0.01 s for the closed form (Table 12), the equivalence the reviewer notes among the strengths. A single gradient step omits the inverse Gram factor that separates correlated keys. Table R3 reports the requested single-step baseline, one SGD step on the matched recovered data.
Table R3. Single-step baseline (accuracy, %).
| Task | Target Base | One SGD step | FMC |
|---|---|---|---|
| MAB | 23.33 | 31.67 | 90.00 |
| KnowEdit | 10.87 | 11.01 | 53.53 |
Head-only restriction. The restriction is what permits the closed-form fit: with frozen features, collection and fitting separate (Section 3.3). For the requested comparison we trained LoRA matrices in the output head and in the last transformer block on the same recovered data (Table R4).
Table R4. Head + last-block LoRA against FMC (accuracy, %).
| Task | Head + last block: 2 epochs | 4 epochs | 8 epochs | 16 epochs | FMC |
|---|---|---|---|---|---|
| KnowEdit | 41.03 (21.28 s) | 52.17 (42.90 s) | 60.73 (85.08 s) | 67.80 (170.42 s) | 53.53 (3.44 s) |
| MAB | 86.67 | 83.33 | 90.00 | 90.00 | 90.00 |
On MAB the larger adapter reaches FMC's score at 8 and 16 epochs and does not exceed it. On KnowEdit it exceeds FMC from 8 epochs, at about 25× FMC's fitting time, because once the last block is trained the final hidden states can no longer be collected once and reused. The restriction is thus what buys an installation with no target backpropagation. Both baselines are in Appendix I.6 of the revised paper (Tables 22 and 23).
Response to Reviewer obb8 [3/3]
W3. On generalization, side effects, uncertainty, and the additional model pairs.
Response. We thank the reviewer for the specific list. Each item is now measured.
Paraphrase-level generalization. This is what KnowEdit measures here: Section 4.1 states that the target is evaluated on held-out paraphrases, not on new facts. To the reviewer's question: 450 of the 736 test paraphrases (61.14%) are near-paraphrases of a compiled item. They belong to the 225 test facts whose canonical address is in the compiled pool (Table 11), and each is an alias variant of that address: the template is the same and only the subject's name changes, for example to a short form. Table R5 refines Table 11 by character similarity to the compiled address.
Table R5. KnowEdit accuracy by similarity to the compiled address.
| KnowEdit test paraphrases | Count | FMC (%) | Distilled LoRA, 2 epochs (%) |
|---|---|---|---|
| Compiled fact, similarity ≥ 0.90 | 182 | 57.14 | 42.31 |
| Compiled fact, similarity < 0.90 | 268 | 69.40 | 47.01 |
| Fact outside the compiled pool | 286 | 36.36 | 18.88 |
Accuracy does not rise with character overlap: FMC is not more accurate on the closer variants. The result supports transfer of the installed answers across subject aliases.
Side effects. We measured perplexity and instruction following with the MAB adapter active on inputs unrelated to the memory (Table R6).
Table R6. Side effects with the MAB adapter active.
| State | MAB accuracy (%) | Perplexity | Strict instruction pass rate (%) |
|---|---|---|---|
| Target Base | 23.33 | 9.16 | 88.28 |
| Distilled LoRA, 2 epochs | 60.00 | 29.09 | 47.66 |
| FMC, as submitted | 90.00 | 66.08 | 21.09 |
| FMC with preservation term | 91.67 | 9.57 | 57.81 |
The submitted adapter is not neutral on such inputs. The update changes only the output rows of the requested tokens (Section 3.4), but Eq. (1) fits their logits at the compiled positions alone and does not constrain them elsewhere. We therefore added one preservation term to the same fit, as MEMIT does for the keys it preserves (Meng et al., 2023): hidden states of generic text enter the ridge problem as additional rows with zero targets. The boosts are unchanged, and installation is still one closed-form fit with no backpropagation. With the term, perplexity returns close to Target Base, the pass rate rises by 36.72 points, MAB accuracy is preserved, and all three measures are better than for two-epoch distillation.
Uncertainty. The gaps to two-epoch distillation now have intervals on all three tasks, and none includes zero (Table R7).
Table R7. Intervals for the gaps to two-epoch distillation.
| Gap | Estimate | 95% interval | Basis |
|---|---|---|---|
| MAB accuracy | 30.00 points | [18.33, 41.67] | paired bootstrap over the 60 queries (counts of Figure 3(b)) |
| KnowEdit accuracy | 18.61 points | [13.45, 23.51] | paired bootstrap over 368 facts (Section 4.2) |
| LoCoMo F1 | 2.64 points | [2.0, 3.4] | bootstrap over the nine streams; FMC is above on all nine (exact two-sided sign test, p = 0.004; Table 13) |
Other model pairs. The two additional model pairs of Section 4.5 show the pattern of the primary pair (Table R8): distillation passes FMC only on KnowEdit at 16 epochs. Over the primary and additional pairs, FMC is at or above distillation in 25 of 28 cells.
Table R8. The same sweep on the additional pairs (source Qwen3-1.7B).
| Target model | Task (accuracy, %) | Distilled LoRA: 2 epochs | 4 epochs | 8 epochs | 16 epochs | FMC |
|---|---|---|---|---|---|---|
| Qwen3-4B | KnowEdit | 19.57 | 24.32 | 39.95 | 50.41 | 46.60 |
| Qwen3-4B | MAB | 35.00 | 53.33 | 70.00 | 76.67 | 85.00 |
| SmolLM3-3B | KnowEdit | 27.17 | 43.75 | 52.31 | 56.93 | 55.43 |
| SmolLM3-3B | MAB | 20.00 | 41.67 | 70.00 | 85.00 | 85.00 |
In the revised paper these results are in Appendix I: the additional pairs in I.1 (Table 18), the intervals in I.3, the similarity split in I.4 (Table 20), and the preservation term in I.5 (Table 21).
We thank the reviewer again for the detailed review. All changes are marked in blue in the revised PDF, and we are happy to answer further questions.
Response to Reviewer obb8 [1/3]
We thank the reviewer for the careful reading, and for the remarks that the abstraction is clean and the reporting transparent. We added a 2/4/8/16-epoch budget sweep on all three tasks, the same sweep on KnowEdit and MAB for the two additional model pairs, the adapter-only LoCoMo evaluation that the reviewer asks for, and side-effect measurements with the adapter active, and we rewrote the claims to match the new evidence. The contribution of the paper is now:
Each question is answered under the weakness it belongs to.
Response. We thank the reviewer for naming the regime in which parametric storage could matter. We evaluated it, and the answer to Major question 2 is yes: the compiled head answers on its own (Table R1). The LoCoMo part of the revised Section 4.2 now leads with this result and not with the readback or the text control, with every budget in Appendix I.2 (Table 19).
Table R1. LoCoMo on the 250 admitted values. Adapter-only means that the target answers the 1,201 natural questions directly, with the stream's adapter active, empty context, and no index or reader.
Adapter-only F1 nearly doubles over Target Base, with no index and no stored text to consult. FMC obtains this in one fit, whereas distillation needs 8 to 16 epochs to reach the same level.
On whether the readback is forced: it is not forced by the fit. The ridge constraints do not set the logits. They are fitted jointly, in least squares and under a rank limit, so the requested logits are approximated, and exact decoding needs a sufficient margin at every prefix (Appendix D.2). Admission does select values that can be written one at a time, and Section 4.2 reports the result as conditional on it. What the result shows is that such values stay exact when installed jointly, which no evaluated distillation budget achieves.
Under the shared reader a text file matches the adapter, as the reviewer says, so we claim no superiority to text storage; Section 4.2 states this and cites the 250/250 readback with its condition. What LoCoMo does support is the thesis that a parametric memory can be carried into the successor's parameters with one analytic fit: the admitted multi-token values are installed jointly without loss, and the installed adapter can be used without an index.