acceptodds
Under review as a conference paper at ICLR 2027

Same Count, Different Memorisation: Spacing and Time Since the Last Copy

Abstract

Deduplication acts on one quantity: how often a sequence appears. How much leakage a count produces is not fixed by that count. Duplicates clustered into consecutive steps, which we call massed, leak less than spaced ones. We hold the count, pin the last copy and the learning-rate position, and vary only when earlier copies occur. Against massed placement, the leakiest of four spacings, selected in sample, adds +0.733 bits of sampled exposure, a rank-based measure, at 10.6M non-embedding parameters. That is a lower bound: at the top counts the exposure estimator censors at 9 bits. This massed penalty is positive in all ten arms, 3.2M to 1B parameters. We widen each interval for variation between canary panels. It still excludes zero in seven to ten arms on exposure, depending on the component, and six on depth, a likelihood gap measured in nats. Per count cell, within panel, the penalty excludes zero from count k >= 8 on exposure and k >= 16 on depth. The penalty appears on five separately generated canary panels, each excluding zero alone, and on a calibrated attack. So leakage at a fixed count is schedule-dependent in direction: within panel the fitted gain is 1.22 to 1.95x over the drop-free nine. We claim the direction, not the size: the two confounds that could manufacture it leave the penalty at 0.59 to 1.29 times its value. A second axis, retention, appears at one configuration. Time since the last copy flattens the count exponent on exposure, +0.375 to -0.002 as retention grows tenfold. It is present at 20 tokens per parameter once retention is matched, on exposure within panel. Depth does not follow on the first panel: matching moves it by +0.0006 nats, unmoved rather than weakened. Exposure resolves on three panels separately, not pooled; depth on one. Clustering is not a defence: it loses to conditional random placement, on the attack from k >= 8. So quote leakage with its schedule and retention rather than intervening.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.