CINN: A Communication-Informed Neural Network

As models grow, distributed pre-training must repeatedly transfer model weights and parameter fragments across devices or data centers, so the data volume increases and transfer time lengthens. The next compute step then waits for the data to arrive, leaving the computing devices idle. To address this problem, this paper proposes a Communication-Informed Neural Network (CINN), which improves communication efficiency by reducing the dimensionality of the model weights and parameter fragments to be transmitted. Unlike existing dimensionality-reduction methods that mainly target numerical reconstruction or rely on fixed linear maps, CINN reshapes the local geometry of model parameter fragments through an input-dependent nonlinear transformation followed by a compressive projection, with both components adapted to the task. Jacobian–gradient response strengthens the sensitivity of the low-dimensional representation to task-relevant directions and reduces information loss under the compressive projection, while an orthogonality regularizer reduces directional overlap among transmitted coordinates so that limited communication resources preferentially retain task-relevant information. In CIFAR-10 distributed training experiments with ResNet-56, CINN compresses the accumulated outer-loop parameter difference after every 10 local optimization steps, mapping each 1028-dimensional parameter block to a low-dimensional latent plus one scale factor. At 8× compression, CINN is only 4.72 points below uncompressed transmission. When each block is compressed to 16 dimensions, CINN reaches 72.78%, 9.98 points above linear projection and 17.46 points above PCA. Under a 200 ms / 1 Gbps wide-area network (WAN) model, CINN at 1/8 compression reduces the mean epoch time of a 1B-parameter FP16 payload from 346.1 s to 62.6 s.