Recovering Evicted Response for Progressive Sparse Attention in Image Super-Resolution

Progressive sparse attention achieves efficiency in image super-resolution by iteratively pruning the candidate set across depth, forming a nested sequence of shrinking supports. However, this pruning paradigm imposes an irrecoverable information bottleneck, giving rise to evidence eviction: response information from discarded candidates is irreversibly lost, both immediately when a fixed budget forces removal and in later layers when relevance emerges only after pruning has taken place. Either way, deeper layers are permanently denied access to interactions that may carry critical high-frequency cues for structure recovery. To address this, we propose EvRC (Evicted Response Compensation), which recovers pruned response information through two complementary mechanisms at different timescales. Instantly, Immediate Recovery folds the discarded response directly into the current-stage representation, ensuring that no scored interaction is silently dropped. Subsequently, Deferred Recovery decouples the evicted response into an eviction-magnitude scalar and a content residual, and deposits it into a shared Trace Bank, enabling subsequent layers to selectively recover evidence conditioned on their current representation. Together, the two components ensure that the active support may shrink, but the response evidence from the broader scored support remains accessible throughout the network. Experiments on standard benchmarks under both classical and lightweight settings demonstrate state-of-the-art performance, with particularly notable gains in recovering fine structures within complex scenes.