- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GIGA addresses frequency-dependent loss of sparse update opportunities as worker count and global batch grow. It forms active-support means within fixed worker groups, sums those means for the parameter update, and separately accumulates their element wise squares in Adagrad state. Group size controls retained granularity. Experiments cover two industrial CTR workloads, 32–512 worker scaling, matched aggregation baselines, multi-day stability, and warm-start/from-scratch settings.
Critical strengths
- The paper identifies a precise and practically important failure mode of large-batch sparse optimization. The method follows directly from the analysis and cleanly couples the update numerator with the Adagrad-state increment.
- The controlled experiments are strong. They isolate aggregation, compare plausible alternatives, ablate the state update, show multi-day and month-long behavior, generalize to a second workload, and report full-system scaling and overhead.
Critical weaknesses
- All efficacy evidence uses private industrial CTR workloads and Adagrad. It remains unclear whether the phenomenon and update generalize to public benchmarks or other adaptive optimizers.
- Group size is consequential and tuned separately by scale; the method lacks an automatic selection rule. The one-step approximation preserves state accumulation but not sequential parameter updates, and no convergence analysis is provided.
Questions for authors
- Can you provide a public-data or controlled synthetic reproduction?
- Can group size be selected from observed active-group statistics without quality sweeps?
- How does the method extend to row-wise Adagrad, Adam-family optimizers, or asynchronous updates?
9 comments
[Q3 / W1] Row-wise Adagrad, Adam, and asynchronous updates
How does the method extend to row-wise Adagrad, Adam-family optimizers, or asynchronous updates?
Response: The Criteo-1TB experiments in Q1 show AUC gains of 0.17 pt with row-wise Adagrad and 0.06 pt with AdamW. We also evaluate warm-start incremental training on industrial CTR data with 32 workers, local batch 600, and GIGA . Gains over the corresponding RBI runs are reported in percentage points:
| Optimizer | ΔAUC (pt) | ΔGAUC (pt) |
|---|---|---|
| AdamW | +0.002 | +0.048 |
| Row-wise Adagrad | +0.006 | +0.041 |
AdamW warm-starts model parameters and initializes optimizer states from scratch, with results measured after three days of incremental training. Row-wise Adagrad initializes its accumulator by averaging the existing Adagrad state within each row.
These results extend the optimizer evidence to warm-start industrial training, with gains in both AUC and GAUC. Appendix F defines the optimizer extensions below.
Row-wise Adagrad. For a mean-square scalar accumulator, GIGA gives
\qquad e_j^{+} = e_j - \eta\frac{\sum_{k\in\mathcal{A}_j}\bar{g}_{j,k}}{\sqrt{h_j^{+}}+\epsilon}.$$ Here $e_j$ is the embedding row, $h_j$ is its scalar accumulator, $d$ is the embedding dimension, and $\mathcal{A}_j$ is the set of groups containing ID $j$. Within group $k$, reduce-by-ID averages the locally aggregated gradients $g_{j,w}$ over the workers $S_{j,k}$ that contain ID $j$, giving $\bar{g}_{j,k}=\frac{1}{|S_{j,k}|}\sum_{w\in S_{j,k}}g_{j,w}$. The resulting scalar accumulator equals the final accumulator obtained by sequentially accumulating the same prescribed group gradients (Appendix F.1). **Adam.** For a given row, let $S$ be its active workers and $\mathcal A$ its active groups. Using the group means defined above, RBI and GIGA construct their moment inputs as $$r=\frac{1}{|S|}\sum_{w\in S}g_w,\qquad M=\sum_{k\in\mathcal A}\bar g_k,\qquad Q=\sum_{k\in\mathcal A}\bar g_k^{\odot 2}.$$ RBI uses $(r,r^{\odot 2})$ as the first- and second-moment inputs. GIGA feeds $M$ and $Q$ to the first moment $m$ and second moment $v$, respectively: $$m^+=\beta_1m+(1-\beta_1)M,\qquad v^+=\beta_2v+(1-\beta_2)Q,$$ $$e^+=e-\eta\frac{m^+/(1-\beta_1^t)}{\sqrt{v^+/(1-\beta_2^t)}+\epsilon}.$$ Both methods advance $t$ once per physical table update and use the same normalization and sparse-row mask, updating present rows even when their gradients cancel and leaving inactive rows unchanged. One active group recovers the corresponding masked Adam update (Appendix F.2). **Asynchronous updates.** We outline a protocol with group-local synchronization and independent progress across groups. For a given row, let $z_b$ be the active-worker mean in completed group record $b$: | Step | Stage | Proposed rule | | --- | --- | --- | | 1 | **Group completion** | Each group submits its active-worker means after all its workers finish the batch. | | 2 | **Staleness check** | Before committing an update, check sparse-row age against $\tau$ local owner updates. Return expired groups for recomputation. | | 3 | **Owner update** | Combine accepted records into $M=\sum_b z_b$ and $Q=\sum_b z_b^{\odot 2}$, then apply one atomic GIGA update. Consume each record once. | Separate-square state accumulation remains exact for admitted gradients. Appendix F.3 describes this asynchronous protocol. # [W2] Sequential approximation and convergence > The one-step approximation preserves state accumulation but not sequential parameter updates, and no convergence analysis is provided. **Response:** **The new analysis establishes convergence of GIGA's actual fused iterates without assuming equivalence to sequential parameter updates.** Appendix A.4 proves the following rate for a smooth, lower-bounded objective, unbiased bounded worker gradients, stationary IID worker sampling, fixed grouping and parameter set, positive initial accumulators, and fixed learning rates: $$\frac{1}{T}\sum_{t=1}^{T}\mathbb E\|\nabla f(x_t)\|_2^2 =O\left(\frac{\log(1+T)}{\sqrt T}\right).$$ Gradients are recomputed at each iterate. The key inequality $M_{t,i}^2\le KQ_{t,i}$, where $K=W/G$, bounds the accumulated squared steps by a logarithmic term. The proof accounts for the activity-dependent mean of $M_t$ and the dependence between the current gradient and adaptive denominator. The sequential comparison establishes a separate property: identical state accumulation for prescribed group gradients.Changes in the revised manuscript (highlighted in red)
- Added a convergence theorem and proof in Appendix A.4 (Proposition 1, Equation A.3), summarized in Section 3.4.
- Added an activity-based group-size selection rule and comparisons at 128 and 256 workers in Appendix C.2 (Tables C.4–C.5).
- Added row-wise Adagrad updates in Appendix F.1.
- Defined Adam moment updates, clock, and sparse-row masking in Appendix F.2.
- Specified an asynchronous group-completion and owner-update protocol in Appendix F.3.
- Added public Criteo-1TB results for Adagrad, row-wise Adagrad, and AdamW in Appendix G.
Thank you for your constructive feedback. We hope the additional experiments, convergence analysis, and clarifications address your concerns, and we would appreciate your reconsideration of the final score.
We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. Below, we address these concerns point by point and try our best to update the manuscript accordingly.
[Q1 / W1] Public-data or controlled synthetic reproduction
Response: We train DeepFM from scratch on public Criteo-1TB with 32 workers and local batch 4096. With$G=1$ throughout, GIGA improves AUC over RBI across all three optimizers:
AdamW values are means of five runs. Appendix G reports this public-data evaluation (Table G.1).
These results extend the evidence beyond private industrial workloads and standard Adagrad, with consistent gains across all three tested optimizers.
[Q2 / W2] Selecting G from active-group statistics
Response: We provide a group-size selection rule based on sampled worker activity. Let$W$ be the number of workers and$G$ the number of workers per group, giving$W/G$ groups. Each record$i$ corresponds to an item ID in a sampled step. A worker is active if it contains the ID, and a group is active if at least one of its workers is active. Let$n_i$ and$a_i(G)$ be the numbers of active workers and active groups, respectively. We replay the same worker hits for every candidate
G.We define hot records as$\mathcal H={i:n_i\geq W/2}$ and cold records as$\mathcal C={i:1\leq n_i\leq3}.Letq_{99}(G)$ be the 99th percentile of$a_i(G)$ over$\mathcal H$. The three costs are:
Phot(G)=q99(1)q99(G)
Shot(G)=∣H∣1∑i∈H1[ai(G)=W/G]
Lcold(G)=1−∑i∈Cni∑i∈Cai(G)
Each active group supplies one virtual update, contributing its mean gradient to the update numerator and its squared mean to the Adagrad state. The costs capture how grouping balances compensation for lost head-ID updates with preservation of contribution granularity:
G.Increasing
Gmoderates head-ID amplification but merges more contributions within groups. The score balances these effects while preserving tail-ID contributions.We select the group size by minimizing the worst of the three costs. For each candidate group size G, we compute
\mathrm{Score}(G) = \max\left\\{P_{\mathrm{hot}}(G),\\ S_{\mathrm{hot}}(G),\\ L_{\mathrm{cold}}(G)\right\\}.
We select the G in the candidate set G with the lowest score, breaking ties in favor of the smaller group size. The tables below use the same notation and report AUC and GAUC differences in percentage points relative to the 32-worker reference.
128 workers:
G256 workers:
GThe rule selects
G=2at 128 workers andG=4at 256 workers using activity statistics alone. Both recover positive AUC and GAUC gains, whereasG=1leaves GAUC below the reference at both scales. Further increasingGraises hot-ID saturation and yields no additional quality gain. These results support a practical, mechanism-based choice ofG: moderate head-ID compensation without excessive coalescing, preserve tail-ID contributions, and recover both ranking metrics without using quality sweeps to selectG.