- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies decentralized zeroth-order optimization with sparse communication. Its central observation is that the coordinates required for exact one-hop peer-state mixing are coupled to the sparse query support: to recover a target coordinate block from ordered value messages, every positive-weight neighbor's message support must contain that target block. Building on this condition, the paper proposes ZO-COSMO, which aligns the sparse query direction with the coordinates used for mixing, so each active link sends only ordered values and no recurring coordinate indices. An edge-local matching variant assigns supports per active edge and reduces per-node payload relative to global-support mixing. The paper analyzes matching contraction and matching-class optimality, and gives convergence results for global-support, matching, and sparse-momentum updates. Communication accounting compares value-only payloads with indexed sparse and dense alternatives. Experiments include Rosenbrock synthetic tests, direction-correlation and heterogeneity studies, Qwen2-7B LoRA experiments on QNLI/SST-2, and a single-worker full-model sparse ZO study. The paper claims that removing coordinate-index overhead buys additional optimization rounds under a fixed bit budget and yields fixed-budget gains in decentralized LLM fine-tuning, although the LLM evidence is limited to 2--8 logical workers and two tasks.
Critical strengths
1.The support-compatibility characterization is clear and useful. Theorem 1 gives a sharp condition for when a receiver can recover selected coordinates of a one-hop neighbor average from ordered value messages alone, and it identifies the worst-case coordinate error when the condition fails. This turns “which coordinates must be communicated” into a precise protocol design question rather than a generic compression issue.
2.The index-free, edge-local design has concrete communication value, and the empirical controls are informative. By aligning query and mixing supports, ZO-COSMO removes the recurring coordinate-index field and charges only value payloads per active link; the edge-local matching variant further reduces per-node payload. The experiments include indexed controls that isolate representation cost, direction-correlation controls, placement audits for logical workers, and reproducibility details, making the empirical claims more interpretable than a simple end-to-end comparison.
Critical weaknesses
Weakness 1: Empirical evidence for the central LLM claim is not yet strong enough.
The authors report several key accuracy contrasts whose CIs include zero, e.g., ZO-COSMO vs.\ seed-aware Rand- (, CI ; Section 5.3, Appendix D.3, Tables 13--14) and the edge-local gain (, CI ; Appendix D.3). The placement audit further confirms that sharing a frozen backbone does not alter the simulated trajectories (Appendix D.3). These defenses are valid, but the abstract and conclusion still present value-only communication as yielding gains in decentralized LLM fine-tuning, whereas the Qwen evidence covers only 2--8 logical workers, two tasks, and a single-worker full-model run (Section 5.3; Appendix D.3). ZO-MARINA-type results are reported at initialization because its first dense refresh exceeds the fixed budget (Appendix C.5; Appendix D.3), and the momentum benefit reverses at a larger step size (Section 5.2; Appendix B.10). The claim should therefore be downgraded or supported by stronger evidence.
Weakness 2: The theory relies on idealized public information and lacks a robustness analysis for support mismatch and asynchrony.
The authors explicitly state the ideal public-coin model (Appendix A.4) and provide seed amortization and transfer checks (Appendix C.2--C.4), which is appropriate. What remains unaddressed is how the exact-mixing condition of Theorem 1 (Section 4.3) degrades when seed desynchronization, round misalignment, packet loss, private supports, or non-uniform supports violate . Theorem~1 only gives a worst-case coordinate error ; the paper does not provide an expectation or high-probability error bound, nor does it analyze how such mismatch affects convergence. The main comparisons charge the value payload (Section 4.3; Appendix C), and although Appendix C.4 notes that a missed round recovers the support but not the missed values or optimizer state, the effect of synchronization failure and retransmission on exact mixing is not incorporated into the theory or the main experiments. Therefore, the practical extrapolation of ``exact mixing'' lacks stated boundaries.
Questions for authors
On Weakness 1 (empirical evidence and scope) For the key LLM contrasts whose 95% CIs include zero---e.g., ZO-COSMO vs.\ seed-aware Rand- (, CI ; Section 5.3, Appendix D.3, Tables 13--14) and the edge-local gain (, CI ; Appendix D.3)---please report per-seed paired differences, multiple-comparison correction, and a power analysis. If the primary CI still crosses zero, do the authors agree to downgrade the claim from gains in decentralized LLM fine-tuning to more rounds under a fixed bit budget, with task-dependent accuracy gains?
The Qwen evidence covers only 2--8 logical workers, two tasks, and a single-worker full-model run (Section 5.3; Appendix D.3). Please add real multi-process, non-IID, and at least one additional-task evidence, or state the extrapolation boundary explicitly in the limitations. In particular, does the placement audit establish simulation equivalence only, rather than sufficiency of scale and task coverage?
Please separate ``infeasible under budget'' baselines from equal-round comparisons. For ZO-MARINA-type, whose first dense refresh exceeds the fixed budget (Appendix C.5; Appendix D.3), please clarify that it is not an effective baseline under the QNLI fixed-budget setting, and report the equal-round result separately. Also provide a step-size sensitivity grid and failure-run statistics for the momentum gain, given that the comparison reverses at a larger step (Section~5.2; Appendix B.10).
On Weakness 2 (theory boundary and communication modeling) Theorem 1 only gives a worst-case coordinate error when (Section 4.3). Please provide an expectation or high-probability error bound under support mismatch, or at least a sensitivity experiment measuring disagreement and accuracy degradation. Without such a bound, how should readers interpret ``exact mixing'' beyond the ideal public-coin model?
Please state precisely which results hold only under the synchronous public-seed model, and whether Theorems 2, 4, and 5 remain valid under seed desynchronization, round misalignment, packet loss, or private/non-uniform supports. If they do not, please give a counterexample or explicit conditions.
The main comparisons charge value payload (Section 4.3; Appendix C), while Appendix C.4 notes that a missed round recovers the support but not the missed values or optimizer state. Please provide an end-to-end communication trace including seed initialization, synchronization/retransmission, and headers, or explicitly restrict the communication advantage to the synchronous reliable-delivery model.