acceptodds
Under review as a conference paper at ICLR 2027

CINN: A Communication-Informed Neural Network

Abstract

As models grow, distributed pre-training must repeatedly transfer model weights and parameter fragments across devices or data centers, so the data volume increases and transfer time lengthens. The next compute step then waits for the data to arrive, leaving the computing devices idle. To address this problem, this paper proposes a Communication-Informed Neural Network (CINN), which improves communication efficiency by reducing the dimensionality of the model weights and parameter fragments to be transmitted. Unlike existing dimensionality-reduction methods that mainly target numerical reconstruction or rely on fixed linear maps, CINN reshapes the local geometry of model parameter fragments through an input-dependent nonlinear transformation followed by a compressive projection, with both components adapted to the task. Jacobian–gradient response strengthens the sensitivity of the low-dimensional representation to task-relevant directions and reduces information loss under the compressive projection, while an orthogonality regularizer reduces directional overlap among transmitted coordinates so that limited communication resources preferentially retain task-relevant information. In CIFAR-10 distributed training experiments with ResNet-56, CINN compresses the accumulated outer-loop parameter difference after every 10 local optimization steps, mapping each 1028-dimensional parameter block to a low-dimensional latent plus one scale factor. At 8× compression, CINN is only 4.72 points below uncompressed transmission. When each block is compressed to 16 dimensions, CINN reaches 72.78%, 9.98 points above linear projection and 17.46 points above PCA. Under a 200 ms / 1 Gbps wide-area network (WAN) model, CINN at 1/8 compression reduces the mean epoch time of a 1B-parameter FP16 payload from 346.1 s to 62.6 s.

Primary area:
optimization
Keywords:
distributed pre-training, gradient compression, communication bottleneck, deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4cFDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CINN, a learned compression method designed to address network bandwidth bottlenecks in distributed learning. CINN consists of an input-dependent nonlinear mapping followed by a learnable projection. The evaluation is limited to ResNet-56 on CIFAR-10 using a two-GPU setup, where CINN outperforms PCA and linear projection baselines.

Critical strengths
  1. Addresses a relevant problem in distributed learning.
  2. Provides an implementation of the proposed method.
Critical weaknesses
  1. Mismatch between motivation and evaluation. The paper is motivated by frontier-scale models and cross-datacenter training, yet the evaluation is limited to ResNet-56 on CIFAR-10 using only two GPUs.
  2. Weak positioning relative to existing work. Quantization and sparsification are dismissed based on superficial arguments, overlooking numerous hybrid and adaptive methods that address the stated limitations. Furthermore, Section 2.2 discusses only general dimensionality-reduction techniques while omitting low-rank compression methods developed specifically for distributed training (e.g., PowerSGD and ATOMO). Apart from DiLoCo, none of the cited references is more recent than 2019.
  3. Limited theoretical contributions. The theorems establish only elementary properties and provide neither convergence guarantees nor evidence of an advantage over linear compression. Theorem 1 also applies to CINN, whose Jacobian has an ((N-K))-dimensional null space. Theorem 2 merely establishes that this null space varies with the input (x), without demonstrating that task-relevant directions are preserved.
  4. Unjustified assumptions. Equation (3.11) lacks sufficient justification, and the invertibility of the Jacobian assumed in Theorem 2 is not guaranteed by the proposed architecture.
  5. Missing implementation details. Several key components are insufficiently specified, including the decoder architecture and the compressor training procedure.
  6. Lack of ablation studies. No ablation experiments are provided to assess the contributions of the individual components of CINN.
  7. Missing baselines. The experimental evaluation lacks comparisons with established compression methods, including Top-k, PowerSGD, quantization, and more recent low-rank and adaptive compression techniques.
  8. Substantial accuracy degradation. The reported 4.72-percentage-point accuracy drop at only an 8× compression ratio on CIFAR-10 is substantial. Moreover, the paper does not report results across multiple random seeds or provide measures of variability.
  9. Insufficient runtime and overhead analysis. The paper does not provide a comprehensive wall-clock time comparison or quantify the computational overhead introduced by CINN. Any compressor transmitting the same number of bytes could achieve a similar communication speedup, making it difficult to attribute the reported gains specifically to CINN. In particular, compressor training, encoding/decoding costs, and synchronization overhead are not evaluated.
  10. Ambiguous terminology. Terms such as "batch," "state," and "parameter fragments" are used ambiguously. More precise terminology (e.g., model parameters or gradients) should be adopted, and key terms should be clearly defined and used consistently.
Questions for authors
  1. What are the decoder architecture and the loss function used to train the compressor?
  2. Can the authors provide ablation studies examining the contributions of Qjac,RorthQ_{\mathrm{jac}}, R_{\mathrm{orth}}, and the design of fθf_\theta?
  3. Is the Jacobian invertible in practice, as assumed in Theorem 2? Can the authors provide any convergence guarantees for the proposed method?
  4. How does CINN compare with Top-k sparsification, PowerSGD, quantization, and recent low-rank and adaptive compression methods under equal communication budgets? Please include wall-clock training time and compressor overhead.
  5. Can the authors report results across multiple random seeds, including means and standard deviations?
  6. Does CINN maintain its performance advantages when evaluated on larger models, transformers, larger datasets, and systems with more workers? How does it perform at higher synchronization intervals ((H = 100)–(500)) under realistic network conditions and actual data transfers?
Reviewer_TRrLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a Communication-Informed Neural Network (CINN) to reduce communication overhead in distributed training. The proposed method combines an input-dependent nonlinear transformation, a Jacobian–gradient response, and an orthogonality regularizer to preserve task-relevant information during compression. Experiments on CIFAR-10 demonstrate the effectiveness of the proposed method.

Critical strengths
  • The key codes are released.
  • The experimental results show the effectiveness.
Critical weaknesses
  • Limited experimental evaluation.
  • Insufficient explanation of key technical details.
Questions for authors
  • In Theorem 2, the authors prove that the nonlinear transformation can change the orientation of the local null space. However, this does not directly demonstrate that the proposed transformation can preserve more task-relevant information.
  • The experimental evaluation is limited to CIFAR-10. The authors should conduct additional experiments on more challenging datasets, such as CIFAR-100 and ImageNet.
  • The experimental comparisons are insufficient. More comprehensive comparisons with representative methods are necessary to demonstrate the advantages of CINN.
  • The proposed method is only evaluated on ResNet-56. Considering that the paper primarily targets communication efficiency in large-scale distributed training, experiments on additional network architectures, particularly larger models, are needed. Moreover, can the proposed method be applied to Transformer-based architectures?
  • The computational overhead introduced by CINN is not sufficiently analyzed. Since the proposed method involves nonlinear transformations, Jacobian–gradient computations, and decoding operations, the authors should provide more ablation studies of the compression and decompression costs to demonstrate the overall efficiency in practical distributed training scenarios.
Reviewer_De8aOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose a compression approach for the DiLoCo outer-loop parameter difference before replicas exchange. Each block of coordinates is passed through a dimension-preserving nonlinear map fθf_\theta (residual blocks with LayerNorm) and then a learnable linear projection to dd coordinates, plus one scale factor per block; a decoder reconstructs the block after the all-gather. Training combines the task loss with a Jacobian reward encouraging the code to respond to perturbations along the task-gradient direction and an orthogonality penalty on WW.

Experiments are on CIFAR-10 / ResNet-56, two replicas, 10 inner steps per sync, 200 epochs. At d=128d = 128 (≈8×\approx 8\times compression) CINN reaches 85.44% vs. 90.16% uncompressed; at d=16d = 16 it reaches 72.78% vs. 62.80% (linear projection) and 55.32% (PCA). Two-GPU all-gather latency for the 3.4 MB update drops from 2.5 ms to 0.3 ms at d=128d = 128.

Critical strengths

The problem area is relevant and topical - as compute becomes more and more constrained, being able to effectively combine clusters that are not colocated (and hence do not have fast interconnects), is of increasing interest

Critical weaknesses

The experiments and baselines do not contain any appropriate comparisons and no decisions are ablated. Top-kk sparsification, PowerSGD, quantization, and error feedback are all cited but none is compared. Quantization is dismissed because it “does not change the dimensionality,” yet the important factor communication overhead is bytes: 4-bit quantisation of a float32 block already gives the same ≈8×\approx 8\times reduction as d=128d = 128 and is a standard baseline for outer updates. There is no ablation of the Jacobian reward or the orthogonality penalty, so it is unknown whether the gain over the linear baseline comes from the nonlinearity, from training with the task loss rather than a reconstruction loss, or from the regularisers. This is especially important as the motivation/intuition behind why such an approach would work is not clear. Two replica's for most experiments is also damaging and not representative of how such approaches would typically be used.

Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.