- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This empirical paper distinguishes target-answer emission from newly correcting an initially wrong answer, transferring the change, and preserving other knowledge. It studies localization-derived controllers, audits existing edited outputs, and examines pre-edit target-token rank as a predictor of correction. The direct-controller experiments find no budget-admissible positive selective control in the evaluated valid subset; other learned interventions do produce admissible gains. A separate fixed-adapter selection experiment finds that rank-based selection yields more corrections than random or answer-frequency-matched selection. Context, additional training exposure, and an exploratory masked MoE update illustrate further intervention-specific conditions.
Critical strengths
A useful accounting framework for actual correction
Section 2, PDF page 3, lines 116–160, explicitly decomposes target hits into retained correctness and new corrections, subtracts regressions for net gain, distinguishes wording/entity/relation transfer, and defines collateral budgets. The paired rescoring in Figure 3 uses the same 100 outputs per model and reports 60–70 target hits but only 7–22 new corrections, with smaller counts after the available preservation checks (Section 4, PDF page 5, lines 229–250). These definitions and the paired audit make clear what an editing result actually establishes. Holding outputs and the scorer fixed makes the retention-versus-correction discrepancy particularly easy to interpret.
Controlled tests of whether read-derived structure helps an update
Section 3, PDF page 5, lines 219–224, constructs geometry using training data, chooses checkpoints using validation data, and evaluates 302 test entities with matched random projections and an unmasked arm. It includes five LoRA arms, three Qwen models, and three seeds, and reports that the prespecified advantages remain unconfirmed while acknowledging that the intervals permit differences. This moves beyond comparing read and write directions geometrically: it asks whether the proposed structure improves an intervention against relevant controls. The cautious handling of inconclusive comparisons is a methodological strength.
A separate prospective test of the rank-selection observation
Section 5.2 and Table 1, PDF page 7, lines 326–346, measure ranks before inspecting edited outputs, hold the shared adapters and 200-entity budget fixed, and compare rank selection with blind and answer-frequency-matched subsets. Rank selection increases the reported mean correction count in all three comparisons, for example from 50.06 to 63.67 relative to frequency matching for Work–country/Qwen14B. The text explicitly limits the interpretation of repeated subset sampling. This supplies a useful predictive check rather than relying solely on the descriptive rank bins in Figure 4. Frequency matching is also a meaningful control for one plausible source of the association.
Critical weaknesses
The main direct-controller result needs a more assessable protocol and calibration
positive selective control
Source: Section 3, Readable, but Not Reliably Writable; PDF p. 4; manuscript lines 173–179.
Concern. The main text specifies the collateral thresholds but does not give an operational definition of selective control, the fact population for the 42-cell comparison, or the rules for choosing each operator’s site, direction, and intervention strength. Figure 2 therefore establishes a result for selected configurations whose coverage and calibration are difficult to assess from the main body. The explicitly disclosed 0/21 unflagged result is the relevant localization evidence; the other 21 cells cannot expand that evidential coverage. The separate dose trajectories and the random-projection controls for learned LoRA updates are useful, but they do not by themselves calibrate all seven direct-controller families. This limits how much the central negative result tells us about the tested localization interfaces beyond those particular configurations.
Suggested clarification or experiment. Please add a compact main-text protocol specifying the selective-control equation, evaluation population, localization-validity criterion, and configuration-selection rule. For the valid direct-controller subset, show uncertainty over evaluated facts or independent runs and an equal-budget sweep of intervention strength, with matched random site/direction controls where applicable. Keep these controls distinct from the existing learned-update experiment. This would strengthen the inference about localization-derived control.
The reliable-correction criteria are not closed on a common evaluation population
The final rung omits WD and is a partial preservation check.
Source: Section 4, The Answer Changes. Does the Edit Hold?; PDF p. 5; manuscript lines 231–235.
Concern. The paper is transparent about this omission. However, the main evidence remains distributed across different checks and populations: Figure 3 combines new correction with KL/unrelated preservation but omits wrong-relation damage; the synthetic held-out-fact test does not measure persistence of the same correction under rewording; and Figure 4 reports new corrections with KL rather than full-budget success. The reported denominators and retained hits permit computation of 2/6/5 regressions and 20/1/3 net corrections for Qwen14B/32B/A3B, respectively. The unresolved issue is how these gains combine with paraphrase transfer and the complete collateral budget. Consequently, the main text does not quantify, on one common population, how many useful corrections remain once complete-answer transfer, regressions, and all stated collateral criteria are considered together.
Suggested clarification or experiment. Please complete one coherent evaluation on the existing paired-audit cohort, or on a clearly specified representative learned-update cohort: report retained correctness, new corrections, regressions and net gain; test held-out paraphrases of the same edited facts using complete-answer scoring; and measure KL, wrong-relation damage and unrelated damage under one stated aggregate budget. A joint table with sample sizes and uncertainty would be sufficient.
Rank’s incremental predictive value and generality remain insufficiently tested
three relation–model comparisons
Source: Section 5.2, Can We Tell Before We Try?; PDF p. 7; manuscript lines 329–346.
Concern. Table 1 is encouraging, but it covers two relations, two model families, and three fixed shared adapters. Answer-frequency matching addresses one alternative explanation, while first-token rank may also proxy sequence likelihood, tokenization, or familiarity already captured by other predictors. Figure 4 contains small, non-monotone bins, so it cannot resolve this issue by itself. The manuscript correctly states that first-token support is not complete factual knowledge and that 10,000 subset draws are not training replications. My concern is therefore the remaining validation gap: the central actionable predictor has not yet been shown to add information over stronger alternatives or to retain its selection advantage on new relations. Section 6 already identifies a related comparison as open, but it is important for judging the contribution beyond established familiarity-based prediction.
Suggested clarification or experiment. Please fix the selection policy before evaluating several held-out relations, compare rank with full-answer likelihood and a target-probability margin under the same eligibility, entity budget and cost gates, and report per-adapter paired gains with uncertainty across training seeds. A small set of fresh independently trained adapters would also test whether the advantage survives beyond the currently fixed output pools. These experiments would test incremental predictive usefulness.
Questions for authors
-
How is selective control computed, and how were sites, directions and doses selected for the valid direct-controller configurations in Figure 2? How sensitive is the conclusion to equal-budget tuning of those choices?
-
Can one main-text cohort report net gain, held-out paraphrase performance, and all three collateral criteria together, with regression counts explicitly tabulated and the currently unmeasured wrong-relation damage added to the paired audit?
-
Does rank outperform full-answer likelihood or probability-margin predictors on held-out relations, and what are the paired gains and uncertainty across the three shared adapters or fresh training runs?