- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how to recover MGDA directions from partial gradient geometry. It develops a conditional minimax characterization and presents two methods: Direct computes the full Gram matrix using structured contractions, while Stream acquires partial information and uses a certificate to decide when to stop.
Critical strengths
-
The theoretical analysis provides a principled way to quantify uncertainty and certify approximate MGDA solutions.
-
The paper evaluates computational cost and geometric accuracy, and reports several limitations and negative results.
Critical weaknesses
-
Structured NTK contractions [1] and ghost-gradient products [2] already compute gradient inner products without materializing full gradients. While the paper acknowledges these works, it is unclear what Direct specifically adds. To properly evaluate its engineering novelty, the paper needs more implementation details, accessible code, and direct comparisons with these existing methods.
-
ParetoTrace-Stream’s practical advantage over Direct is not clearly established. Methods like Direct exploit network structure to compute the full Gram matrix and derive an exact MGDA direction. In contrast, Stream introduces approximations, sample acquisition costs, and certification overhead. While Stream can be faster in certain cases, the current results do not clearly show when its runtime gains outweigh the memory overhead and loss in solution quality compared to structured contraction baselines.
-
The paper is difficult to follow. The connections between the theoretical claims, the stopping rules, and the empirical results are not clearly framed. Key technical details are scattered across the main text and appendix, and inconsistent experimental setups make it hard to evaluate the empirical claims.
[1] Novak, R., Sohl-Dickstein, J., & Schoenholz, S. S. (2022). Fast finite width neural tangent kernel. In Proceedings of the 39th International Conference on Machine Learning, PMLR, 162, 17018–17044. https://proceedings.mlr.press/v162/novak22a.html
[2] Wang, J. T., Mittal, P., Song, D., & Jia, R. (2025). Data Shapley in one training run. In The Thirteenth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2406.11011
Questions for authors
-
What specific implementation details distinguish Direct from [1] and [2]? Could the authors provide comparisons with appropriately adapted implementations of these approaches?
-
Under what conditions should a practitioner choose Stream over Direct? Can the authors give practical guidance based on the number of objectives, gradient agreement, and acquisition and certification costs?
[1] Novak, R., Sohl-Dickstein, J., & Schoenholz, S. S. (2022). Fast finite width neural tangent kernel. In Proceedings of the 39th International Conference on Machine Learning, PMLR, 162, 17018–17044. https://proceedings.mlr.press/v162/novak22a.html
[2] Wang, J. T., Mittal, P., Song, D., & Jia, R. (2025). Data Shapley in one training run. In The Thirteenth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2406.11011
2 comments
Thank you for the rebuttal. My main concern remains Stream’s practical advantage over Direct. Stream gives an approximate solution, while Direct is exact. Moreover, Steam's time advantage is demonstrated only under specific conditions. The memory results in the rebuttal also differ from those reported in the paper. I appreciate the theoretical contribution, but these practical concerns remain unresolved, so I will keep my original rating.
Thank you for the review and for the questions on implementation details, code, and direct comparisons.
What ParetoTrace-Direct adds (first weakness and first question). ParetoTrace-Direct shares the contraction arithmetic of [1] and [2]. Our contribution is the minimax characterization and the certified partial recovery built on it, and ParetoTrace-Direct is its full-information endpoint (Theorem 4), built for task Grams with aggregation by task inside the contraction, a FLOP-based contraction order, embeddings restricted to observed vocabulary rows, tied-weight summation, online factor retirement and one batched reverse pass for shared inputs. Against [1] and [2] adapted to task Grams, it is 1.19 and 1.22 times faster than structured NTK contraction and 1.7 and 4.6 times faster than ghost-gradient products on Pythia-160M at the same memory, and 1.03 and 2.3 times faster at Pythia-1B, where per-task backpropagation runs out of memory. Code, adapters and logs are in the repository.
When to choose Stream (second weakness and second question). The paper's routing rule applies (p. 7, App. F). Use ParetoTrace-Direct when the layer factors fit, ParetoTrace-Stream when contracting every block costs more than a short sketch or memory is short, certify descent rather than the objective when gradients conflict, and certify nothing at two or three tasks. Stream wins on time at large M with agreeing gradients (1.97× at M = 2048) and for strict descent on CIFAR-10 groups at M = 512 (1.24×), and on memory (0.54 GiB against 8.04 GiB on CelebA-40), with training quality level with ParetoTrace-Direct (+0.8 points [−3.5, +5.2] in worst-attribute accuracy). Target-aware queries cut its live recovery from 1.85 s to 1.08 s.
Presentation (third weakness). The final version specifies each method in one place, splits Table 1 by protocol, defines terms on first use, and opens its experiments with a theory-to-experiment map. We are happy to hear if there are any other suggestion that would aid in the clarity of the paper.