PSG: Preserving Activation Structure for Low-Bit Image Restoration

Post-training quantization (PTQ) is a practical route to deploying image restoration models, yet restoration fidelity degrades sharply at ultra-low bit widths. A key challenge is channel-wise outliers, which are particularly pronounced in Mamba-based models and substantially inflate the activation range. Suppressing these extremes can tighten the range, but is not a reliable remedy, as even limited removal can severely degrade reconstruction quality. We show that these outliers are not random spikes, but exhibit strong persistence: their dominant direction remains stable across inputs, and even within each input, the response along this direction is effectively shared across tokens. We term this shared directional component the pedestal. Crucially, the pedestal not only inflates the range, but also defines a direction highly sensitive to quantization error, making it a structure that should be preserved rather than suppressed. Building on this, we propose PSG, a PTQ framework that combines Pedestal Shifting and GRILL. Pedestal Shifting keeps the pedestal outside the quantizer while reducing the activation range, and GRILL calibrates quantization bounds for the residual activations and weights through closed-form updates. Across diverse image restoration tasks and architectures, PSG consistently outperforms existing methods, delivering up to $3.73$ dB higher PSNR at 2 bits, $4.33\times$ smaller models, and $6.13\times$ fewer bit-operations, while deploying as standard TensorRT engines with no additional inference overhead. Our code is available at: https://anonymous.4open.science/r/PSG-AD60.