- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work studies the identifiability of training grouping of losses with gradient descent. If the losses are quadratic functions and matrix-valued spectral weights (defined in Section 4) are known then the local dynamics is determined; bounds are given for non quadratic losses.
Critical strengths
I believe that a strength of the paper is to characterize theoretically what is needed in order to identify the training dynamic by gradient descent starting from a weight w_0 of groupings of quadratic losses and to have bounds in the non quadratic case.
Critical weaknesses
My feeling is that for a non specialist of federate learning like me, it is very difficult to understand the setup and the claims of the paper and one has to persist in order to get to understand the contribution of the paper. Table 1 helped me a lot to wrap up my understanding of the paper.
Questions for authors
No question that could materially affect my assessment.
Response to t34A: the setup and how to read Table 1
Thank you for pointing out that Table 1 helped. We have rewritten the opening around the computation below and added an explanation of that table.
Divide a fixed set of training losses into groups. Start each group at the same parameters, take E gradient steps on its average loss, then average the final parameters by group size. We compare two groupings with the same losses, overall objective, starting point and update rule. Each test starts afresh; different horizons are not successive server rounds.
The question is what information lets us determine whether the two computations return the same parameters. The aim is to identify the update, not to recover the unique grouping that produced it.
Figure 1 gives the simplest example. Two groupings use exactly the same losses and have the same separate lists of gradients and Hessians at the starting point. After two steps, their differences from training on the unsplit data are 0 and 0.014. The difference comes from which gradient is paired with which Hessian. Knowing the two lists without those pairings loses information needed to predict the update.
Section 4 asks whether observing more updates removes the ambiguity. Even matching every horizon from one starting point need not give the same result elsewhere. The quadratic results specify what spectral information and observations suffice to determine later updates. With noisy observations, the paper bounds how different a later update can still be across the permitted losses.
For example, a participant may assign its data to synchronized workers. An unchanged objective does not ensure that regrouping preserves the returned update. We isolate that distinction with deterministic, stateless, full-batch gradient descent.
The opening, Table 1 and Figure 1 now provide a short route into the problem before the longer-horizon results in Section 4. Section 6 tests where pairing predicts nonlinear updates and where it fails.