Adaptive Discounting of Compression Error Feedback for Decentralized Learning

Decentralized learning enables collaborative model training across multiple nodes without relying on a central server, where each node performs local updates using its own data and exchanges compressed updates with neighboring nodes. Although numerous methods have been proposed for decentralized non-convex optimization under communication constraints, effectively correcting compressed error and mitigating the impact of network topology, particularly in sparse networks, remains an open challenge. To address this issue, we propose A-DEFEAT-GT (adaptively discounted error feedback algorithm for decentralized learning with gradient tracking), which integrates error-feedback with an adaptively tuned discount coefficient to dynamically control the influence of accumulated compression errors, together with gradient tracking technique to deal with heterogeneous data, especially in sparse networks, without relying on momentum propagation or consensus model update that require additional computation. Experiments demonstrate that A-DEFEAT-GT consistently outperform existing decentralized methods by achieving higher test accuracy under the same communication budget, particularly under sparse network conditions. These results highlight both the effectiveness and practical efficiency of the proposed method.