Smooth Spectral Operators for Indefinite Hessians: Relative Resolution, Stability, and Matrix-Free Approximation

Curvature-aware optimization in high-dimensional non-convex problems relies on matrix-free spectral transformations of indefinite Hessians. A fundamental difficulty in designing these operators is that a regularization parameter measured in absolute curvature units changes its operational meaning when the overall objective scale changes. Under homogeneous curvature rescaling, a fixed absolute parameter alters the relative transition geometry and arbitrarily changes the matrix-free computational difficulty. To resolve this ambiguity, we introduce relative resolution, parameterizing the spectral transition scale relative to an admissible reference spectral scale. This dimensionless formulation yields a normalized spectral geometry whose polynomial approximation complexity is independent of the absolute curvature scale under admissible normalization. Our analysis connects relative resolution to fixed-$\tau$ operator-perturbation stability, finite polynomial-approximation budgets, and the operational scale mismatch. Experiments support the predicted approximation-complexity scaling, quantify sensitivity to operational scale mismatch, and characterize the distinct realized-action regimes observed across controlled non-convex and stiff physics-informed optimization settings.