A Bit at a Time: Sparse Delta Updates for Indexing Fast-Evolving Codebases

Modern code retrieval systems increasingly operate over codebases that evolve continuously: functions are reformatted, locally edited, refactored, or rewritten while preserving their underlying intent. Standard retrieval pipelines usually treat each version as an independent target, optimizing ranking quality but ignoring whether related versions can share stable index-facing representational structure. We study this problem as efficient continual code retrieval, where a model should retrieve a behaviorally similar version of a code object while enabling representation and index reuse across code evolution. To evaluate this setting, we construct CODECONTINUUM-R, an evolution-aware benchmark built by expanding standard code-search examples into version families that span surface-level rewrites, local structural edits, and higher-level refactorings. Surprisingly, we find that sparse representations already provide a strong foundation for this setting: their active latent supports form discrete routing signatures that remain substantially shared across evolved code variants while maintaining competitive retrieval quality. Building on this observation, we propose continuality-preserved sparse coding, which augments sparse autoencoder training with two lightweight evolution-aware regularizers: a neuron-overlap regularizer that encourages behaviorally similar variants to activate shared Top-K latents, and an activation-order regularizer that preserves the relative priority of shared active neurons. Across Jina-Code-Embedding and Qwen3-Embedding backbones, our method improves sparse-support overlap and activation-order consistency while largely preserving retrieval accuracy. These improvements enable more compact neuron-reuse storage and more reliable small-budget routing in multi-stage sparse retrieval for evolving codebases.