Depth-Ordered Feature Memory: From Snapshots to Trajectories

Sparse dictionaries describe transformers layer by layer and crosscoders assign cross-layer identities, yet neither maintains state across depth. **We introduce Depth-Ordered Feature Memory (DOFM)**, a sparse recurrent observer trained to predict normalized residual updates across a frozen transformer. DOFM integrates three mechanisms: a shared decoder fixing directional coordinates across depth _**(identity)**_; a per-token recurrent state tracking writes, retention, and attenuation _**(history)**_; and cumulative master gates organizing coordinates into a retention hierarchy _**(address)**_. Across diverse backbones, accumulated state reads word sense as accurately as the residual stream itself while revealing what probes cannot: which coordinate carries the sense, when it was written, and how long it persists. The retention prior hierarchically organizes coordinates, allowing a persistent tier to match full-dictionary accuracy and support addressable causal intervention; lexical competition resolves via write-time asymmetric recruitment, where competing senses receive reduced writes and persist without downstream erasure. Feature lifespan indexes persistence and generality over abstraction, turning cross-layer computation into an addressable state trajectory.