Anon: Extrapolating Adaptivity Beyond SGD and Adam

Adaptive optimizers such as Adam and non-adaptive methods like SGD, the dominant first-order element-wise optimizers, exhibit distinct generalization capabilities across different architectures. Prior tunable optimizers attempt to bridge this gap by strictly interpolating between SGD and Adam, effectively confining adaptivity within the 0-to-1 bound. However, this restricted interpolation leaves performance on the table: the optimal adaptivity of some tasks lies outside this interval, and no existing tunable optimizer can reach it. Extrapolating adaptivity violates the strict non-decreasing pre-conditioner assumption underlying the convergence guarantees of existing adaptive methods, so standard guarantees no longer apply. To break this barrier, we propose Anon, an optimizer that achieves fully continuous adaptivity extrapolation across the entire real-number spectrum. To guarantee provable stability in these out-of-bound regimes, we introduce Incremental Delay Update (IDU), a novel mechanism that bypasses hard max-tracking strategies. We theoretically establish Anon's convergence in both convex and non-convex settings. Empirically, Anon demonstrates highly competitive and scalable performance among state-of-the-art element-wise optimizers on representative image classification, diffusion, and large language modeling tasks.