acceptodds
Under review as a conference paper at ICLR 2027

ParetoTrace: Minimax Recovery of MGDA Directions from Partial Gradient Geometry

Abstract

Multiple gradient descent (MGDA) finds a direction that descends on every objective at once. It needs the Gram matrix of task gradients, and building that matrix exactly is the bottleneck as objectives multiply, so a prevailing response is to substitute cheaper geometry. We ask what the observed geometry determines, and answer exactly. Observing Gram-vector products leaves a family of consistent Grams, and we characterize which quantities are identical across it. The MGDA objective is not: it can only be narrowed to a computable range. The worst-task descent margin is, at any weighting the observations already span, and reading it costs no action beyond those already taken. Two methods follow. ParetoTrace-Direct contracts every parameter block, leaving one consistent Gram and returning the exact direction with no queries. It is faster than serial task gradients at objectives, where four of six external implementations exhaust an NVIDIA A100 40 GB card. ParetoTrace-Stream buys products one at a time and stops once the range certifies. It is faster than contraction where materializing the Gram exhausts the card, and has lower worst-task loss and higher held-out accuracy than the strongest cheap baseline on all three seeds. That baseline lets one task run away to a loss of while ours stays under . In replay on a training run it drives, the sharp endpoint certifies of updates against the cheaper rule's , never later, while a numerical guard limits the live run to , and a line search on its exact responses keeps every task loss non-increasing on its minibatch at of updates. Used as an instrument, the characterization shows that MGDA-UB sener2018multi, the standard cheap substitute, increases at least one task loss at forty CelebA objectives on every trunk and batch size we tested.

Primary area:
optimization
Keywords:
Multi-Objective Optimization, Multi-Task Learning, Multiple Gradient Descent Algorithm (MGDA), Gradient-Based Optimization, Minimax Recovery, Gram Matrix Sketching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZNzrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how to recover MGDA directions from partial gradient geometry. It develops a conditional minimax characterization and presents two methods: Direct computes the full Gram matrix using structured contractions, while Stream acquires partial information and uses a certificate to decide when to stop.

Critical strengths
  1. The theoretical analysis provides a principled way to quantify uncertainty and certify approximate MGDA solutions.

  2. The paper evaluates computational cost and geometric accuracy, and reports several limitations and negative results.

Critical weaknesses
  1. Structured NTK contractions [1] and ghost-gradient products [2] already compute gradient inner products without materializing full gradients. While the paper acknowledges these works, it is unclear what Direct specifically adds. To properly evaluate its engineering novelty, the paper needs more implementation details, accessible code, and direct comparisons with these existing methods.

  2. ParetoTrace-Stream’s practical advantage over Direct is not clearly established. Methods like Direct exploit network structure to compute the full Gram matrix and derive an exact MGDA direction. In contrast, Stream introduces approximations, sample acquisition costs, and certification overhead. While Stream can be faster in certain cases, the current results do not clearly show when its runtime gains outweigh the memory overhead and loss in solution quality compared to structured contraction baselines.

  3. The paper is difficult to follow. The connections between the theoretical claims, the stopping rules, and the empirical results are not clearly framed. Key technical details are scattered across the main text and appendix, and inconsistent experimental setups make it hard to evaluate the empirical claims.

[1] Novak, R., Sohl-Dickstein, J., & Schoenholz, S. S. (2022). Fast finite width neural tangent kernel. In Proceedings of the 39th International Conference on Machine Learning, PMLR, 162, 17018–17044. https://proceedings.mlr.press/v162/novak22a.html

[2] Wang, J. T., Mittal, P., Song, D., & Jia, R. (2025). Data Shapley in one training run. In The Thirteenth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2406.11011

Questions for authors
  1. What specific implementation details distinguish Direct from [1] and [2]? Could the authors provide comparisons with appropriately adapted implementations of these approaches?

  2. Under what conditions should a practitioner choose Stream over Direct? Can the authors give practical guidance based on the number of objectives, gradient agreement, and acquisition and certification costs?

[1] Novak, R., Sohl-Dickstein, J., & Schoenholz, S. S. (2022). Fast finite width neural tangent kernel. In Proceedings of the 39th International Conference on Machine Learning, PMLR, 162, 17018–17044. https://proceedings.mlr.press/v162/novak22a.html

[2] Wang, J. T., Mittal, P., Song, D., & Jia, R. (2025). Data Shapley in one training run. In The Thirteenth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2406.11011

2 comments

Thank you for the review and for the questions on implementation details, code, and direct comparisons.

  1. What ParetoTrace-Direct adds (first weakness and first question). ParetoTrace-Direct shares the contraction arithmetic of [1] and [2]. Our contribution is the minimax characterization and the certified partial recovery built on it, and ParetoTrace-Direct is its full-information endpoint (Theorem 4), built for task Grams with aggregation by task inside the contraction, a FLOP-based contraction order, embeddings restricted to observed vocabulary rows, tied-weight summation, online factor retirement and one batched reverse pass for shared inputs. Against [1] and [2] adapted to task Grams, it is 1.19 and 1.22 times faster than structured NTK contraction and 1.7 and 4.6 times faster than ghost-gradient products on Pythia-160M at the same memory, and 1.03 and 2.3 times faster at Pythia-1B, where per-task backpropagation runs out of memory. Code, adapters and logs are in the repository.

  2. When to choose Stream (second weakness and second question). The paper's routing rule applies (p. 7, App. F). Use ParetoTrace-Direct when the layer factors fit, ParetoTrace-Stream when contracting every block costs more than a short sketch or memory is short, certify descent rather than the objective when gradients conflict, and certify nothing at two or three tasks. Stream wins on time at large M with agreeing gradients (1.97× at M = 2048) and for strict descent on CIFAR-10 groups at M = 512 (1.24×), and on memory (0.54 GiB against 8.04 GiB on CelebA-40), with training quality level with ParetoTrace-Direct (+0.8 points [−3.5, +5.2] in worst-attribute accuracy). Target-aware queries cut its live recovery from 1.85 s to 1.08 s.

  3. Presentation (third weakness). The final version specifies each method in one place, splits Table 1 by protocol, defines terms on first use, and opens its experiments with a theory-to-experiment map. We are happy to hear if there are any other suggestion that would aid in the clarity of the paper.

Thank you for the rebuttal. My main concern remains Stream’s practical advantage over Direct. Stream gives an approximate solution, while Direct is exact. Moreover, Steam's time advantage is demonstrated only under specific conditions. The memory results in the rebuttal also differ from those reported in the paper. I appreciate the theoretical contribution, but these practical concerns remain unresolved, so I will keep my original rating.

Reviewer_6wXkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the high computational and memory costs of MGDA in multi-objective optimization by proposing ParetoTrace, a framework that recovers the MGDA update direction from partial observations of gradient geometry and provides theoretical guarantees on optimal objective value bounds and recovery error. Based on this framework, the authors develop two algorithms, ParetoTrace-Direct and ParetoTrace-Stream, for exact solving and progressive certification-based solving, respectively. Experiments on multi-task learning, language model training, and synthetic datasets demonstrate the accuracy of direction recovery and computational efficiency advantages in certain settings. However, the benefits in practical training remain dependent on the specific scenario.

Critical strengths
  1. This paper studies MGDA direction recovery from partial observations of gradient geometry. It establishes upper and lower bounds on the optimal objective value, provides conditional minimax recovery guarantees, and introduces a certification mechanism for common descent directions. Compared with traditional methods based on approximating the Gram matrix, this framework provides clear theoretical guarantees on recovery quality, making it a valuable theoretical contribution.

  2. The authors propose two methods, ParetoTrace-Direct and ParetoTrace-Stream. The experiments evaluate these methods in terms of direction recovery accuracy, computation time, memory usage, and task performance. The results demonstrate significant computational efficiency advantages in certain settings, while also examining the applicability and limitations of the proposed methods.

Critical weaknesses
  1. Although this paper achieves more accurate recovery of the MGDA update direction, ParetoTrace shows higher test cross-entropy than Uniform and CAGrad in the Pythia-160M multi-task training experiments. The large-scale experiments mainly rely on synthetic data or optimization objectives constructed from sample groups. There is insufficient validation on larger language models and real-world multi-task settings, so the practical benefits and generalization ability of the proposed methods remain unclear.

  2. The speedup of ParetoTrace-Stream largely depends on gradient correlation and the query strategy. In the real CIFAR-10 experiments, the original query strategy failed to achieve early certification, and its computation time was even longer than that of ParetoTrace-Direct. Appendix G shows that the theoretical stopping criterion achieved 37 early certifications in offline replay, but only 19 early stops were achieved during actual training due to numerical safeguards. This suggests that the theoretical savings in queries do not consistently translate into practical computational benefits, limiting the method's usefulness.

Questions for authors
  1. In the Pythia-160M experiments, ParetoTrace achieves higher test cross-entropy than Uniform and CAGrad. Why does more accurate MGDA direction recovery not lead to better generalization performance?
  2. How does ParetoTrace perform in terms of computational efficiency and actual training performance on larger language models?
  3. In Appendix G, the number of early certifications during actual training is much lower than in offline replay. How do numerical safeguards affect certification efficiency, and how can this be improved?
  4. The choice of query vectors significantly affects the certification efficiency of ParetoTrace-Stream. How should query vectors be selected in practical training to balance certification efficiency and computational cost?
3 comments

Thank you for the thoughtful review and questions that leads to a new theorem. More on code, proofs and checks are at https://anonymous.4open.science/r/ParetoTrace-anonymous-37B6/README.md

  1. Generalization on Pythia-160M (first weakness and first question). MGDA optimizes common descent on the current minibatch, not held-out loss, and without the verified step uniform is a strong baseline on held-out averages (Kurin et al., 2022, Xin et al., 2022), as our CelebA and QM9 studies confirm. With the verified step no task's minibatch loss rose, uniform moved at only 8 of 900 updates on CelebA with frozen heads and 151 of 600 at Pythia-1B, and ParetoTrace-Direct moved at every update (reply to P8RF, first weakness).

  2. Larger language models (second question). At Pythia-1B ParetoTrace-Direct recovers the exact direction in 1.92 s and 15.7 GiB, 1.03 and 2.3 times faster than structured NTK contraction and ghost-gradient products, while per-task backpropagation runs out of memory, and a training update costs 1.6× uniform. Without the verified step uniform reached held-out worst-task loss 0.774 and ParetoTrace-Direct 1.026 (prespecified primary +0.252 [+0.167, +0.337]). With it ParetoTrace-Direct reached 0.927 against 1.069 for uniform, and the prespecified audit-first primary did not separate (reply to P8RF, first weakness).

  3. Live certification versus replay (second weakness and third question). App. G's guard subtracted a slack scaled to the Gram, a median 7.1× the upper endpoint, which blocked early stops. A guard without it certified 889 of 900 live CelebA updates early, and target-aware Stream 894 of 900 at 1.14 s against 1.79 s per update. On CIFAR-10 groups Stream with strict descent beats ParetoTrace-Direct at M = 512 (1.24×, Table 27), and at M = 100 ParetoTrace-Direct is faster.

  4. Choosing query vectors (fourth question). This question led to a new theorem. The completion query rule closes the certification interval in at most min⁡(M,rank⁡R+1)\min(M,\operatorname{rank}R+1) exact actions, where RR is the geometry not yet contracted, and its weights are optimal for every completion of the transcript. The proof idea is that each query that does not close the interval raises dim⁡(R1/2S)\dim(R^{1/2}\mathcal S) by one, where S\mathcal S is the queried span. The full proof and a check on 608 random instances are at https://anonymous.4open.science/r/ParetoTrace-anonymous-37B6/theory/residual_rank_theorem.md. In practice, once the objective is certified, Stream queries the task with the smallest response, which with strict descent cuts live recovery from 1.85 s to 1.08 s.

Please edit the rebuttal to use an anonymous GitHub repo; the URL may violate ICLR rules.

The code is now at a new link https://anonymous.4open.science/r/ParetoTrace-anonymous-37B6/README.md. To our knowledge the previous link was also posted under an alias account.

Reviewer_P8RFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks what a set of Gram-vector products Kv determines about the MGDA solution, where K = G^T G is the Gram of task gradients. The Grams consistent with k observed products are the shorted completion K_sk plus any PSD B with BΩ = 0 (Thm 1). The MGDA optimum is then known only to lie in [L_sk, U_sk), and the in-span minimizer λ_sk attains the conditional minimax regret U_sk - L_sk. A linear functional of K is determined if and only if its block on the unobserved subspace vanishes (Thm 2), so the worst-task margin at an in-span weighting is exact. The lower endpoint L_sk gives a stopping test that dominates the Frank-Wolfe gap (eq. 12, 21). Two methods follow. ParetoTrace-Direct contracts every parameter block to obtain the exact Gram, and is 14.9× faster than serial per-task gradients at M = 32 on Pythia-160M. ParetoTrace-Stream acquires products until U_sk/L_sk \leq 1 + ε or the margin is positive, holds the weighting across updates and re-audits it with one product; it is 1.97× faster than contraction on synthetic near-rank-two panels at M = 2048. The evidence is seven separate protocol blocks (Table 1) and appendix studies: an audit showing that MGDA-UB returns an ascent direction at 40 CelebA objectives, a CelebA-40 training run in which exact MGDA has the best worst-attribute accuracy, Pythia-160M runs at 8 to 32 objectives, replay on 103 recorded Grams, CIFAR-10 example groups and an NYUv2 pilot.

Critical strengths
  1. A correct characterization with a stopping rule others can adopt. Thms 1 and 2 are short but exact, and the endpoint L_sk dominates the Frank-Wolfe gap pointwise using only the responses already observed (eq. 21). In replay on projected-gradient queries it halves the actions to a 5% stop, from 22 to 11 (Table 16); the paper makes no wall-clock claim for that setting. This matters because any matrix-free MGDA solver that stops on the Frank-Wolfe gap can switch to L_sk and can only stop earlier.
  2. A clear first-order failure of the standard cheap substitute. At 40 CelebA objectives MGDA-UB returns a direction that raises some task's loss to first order on 3 of 3 trunks at all three batch sizes, and the paper identifies the mechanism. This matters because MGDA-UB is how MGDA is usually run at scale, and the method never checks its direction.
Critical weaknesses
  1. The practical case for recovering the exact MGDA direction is not made. The paper shows that cheap substitutes lose geometric fidelity as objectives multiply, but not that recovering this fidelity improves a trained model, nor a multi-task setting where recovery is both needed and affordable. By the paper's own account the downstream evidence is mixed. Stream's speed advantage is confined to strongly agreeing gradients, where uniform weighting is often already adequate, and where gradients conflict the certificate costs most of a full Gram. The certified quantity is also less stable across minibatches than the precision it is certified to. This is the main reason for my rating: a methods paper needs evidence that the method is worth running.
  2. The headline claims go beyond the recorded evidence. The abstract and Table 1 combine numbers from separate protocols, several of them single-seed, replay-only or tuned asymmetrically, and the caveats that qualify them appear only in the appendices. The paper's central theoretical object, the sharp endpoint, decides few of the reported results, and some Table 1 certificates have no recorded stopping rule. The efficiency headline rests on prior contraction arithmetic without the matching baseline. A reader who stops at the abstract will take these numbers as the paper's result.
Questions for authors

Questions whose answers could change my assessment

  1. The one positive downstream result (CelebA-40, Table 18) computes exact MGDA from 40 backward passes rather than with either ParetoTrace method, uses two seeds, and leaves the margin over MGDA-UB unresolved (C.9.2). On Pythia-160M, uniform has lower worst-task test CE than ParetoTrace-Direct on all three seeds (C.1), and C.6.6 refers to "the null training result" (l. 1636-1637). Can you train ParetoTrace-Direct on CelebA-40 with enough seeds to separate it from MGDA-UB (C.9.2 estimates about twenty per arm), and add a standard benchmark with more than three tasks, such as QM9, against Nash-MTL, FairGrad, Aligned-MTL and FAMO?
  2. Stream's speed advantage holds at effective rank near two (App. E.2, l. 1988-1989), where uniform already descends on 101 of 102 recorded Grams (Table 12). Where gradients conflict, the objective certificate needs 33 of 40 actions on CelebA-40 and 227 of 512 on CIFAR-10 groups, the deployed block rule certifies 0 of 3 at three times contraction's time (Table 15), and the descent certificate stops at about twice the MGDA objective with normalized gain 0.02 to 0.04 (Table 13). On the block-4 panels where Stream is 1.97× faster than contraction, what are uniform's objective ratio and worst-task gain? Apart from strict descent at M = 512 on two-example CIFAR-10 groups (Table 27, 1.24×, lost at M = 1024), is there a multi-task setting where uniform fails to descend and Stream beats Direct in wall-clock time?
  3. The minibatch MGDA objective changes by a factor of 1.257 across minibatches on CelebA-40 (C.9.1) and 5.07 on CIFAR-10 groups (C.6.6), against a 5% certification target. What does a 5% certificate buy on CelebA-40, and does a noise-calibrated ε change Stream's cost there?
  4. The run behind the abstract's "every task loss non-increasing on its minibatch at 60 of 60 updates" reaches worst-task training loss 65.91, against 5.45 for the same direction at the tuned step, at a median step of 5.86 × 10^{-7} against 3 × 10^{-4}; one of the 60 updates is a refusal that held the parameters (Table 25). Is there an acceptance rule between these two that keeps the guarantee at a smaller cost in progress? Will the abstract report the tuned-step comparison next to the 60 of 60?
  5. The abstract's comparison with "the strongest cheap baseline" is block 5: two arms, no appendix protocol, no recorded stopping rule (Table 3), and Stream's cadence tuned on a development seed while MGDA-FA ran at its default (l. 461-462). Which rule and margin gate were used, and how many refreshes occurred? How does 0.206 s per update at M = 32 square with 6.24 s per recovery at M = 16 in App. G? How do the results change with MGDA-FA tuned on the same seed and with FAMO, CAGrad and uniform arms? What supports calling MGDA-FA the strongest cheap baseline, given that FAMO beats ParetoTrace-Direct in block 6 and uniform and CAGrad beat both methods on test CE in block 2?

Clarifications 6. The sharp endpoint L_sk decides only block 7 (replay) and the App. G run. Blocks 2 and 4 stop on the cheaper \ell_c, and blocks 1 and 5 record neither the endpoint nor the margin gate (B.5, l. 1134-1136), although the ungated rule can certify at five times the optimum (A.5.7). Can the gate be recovered from the logs for blocks 1 and 5? If not, should Table 1 mark those Stream rows as uncertified? 7. In the live run L_sk stops early on 19 of 100 steps, all of them steps where float32 round-off makes the measured q non-positive (App. G, l. 2110-2117); the 37-against-27 advantage comes from replay. Does the live certificate ever stop early at a state away from Pareto-stationarity? 8. The 14.9× speedup comes from the factor contraction of Novak et al. (2022) (l. 199, 359), measured against a serial per-task loop. How does ParetoTrace-Direct compare in time and memory with a ghost-Gram computation (Wang et al., 2025), which the paper lists as open (l. 480)?

1 comment

Thank you for the careful review. We ran the requested studies and the full results are in the repository.

  1. Exact recovery (weakness 1). Our verified step requires positive first-order response for every task (Prop. 1), then backtracks until all minibatch losses pass Eq. (27), otherwise skipping. Pythia-1B (8 fully shared tasks, 3 seeds × 200 updates):
MethodLoss risesMovesWorst lossMean loss
Uniform, fixed577/600600/6000.6850.503
Uniform, verified0151/6001.0690.543
Direct, verified0600/6000.9270.455

The primary audit-first (certified uniform, otherwise exact MGDA) versus verified uniform was inconclusive (−0.019 [−0.931, +0.892]). Uniform failed the first-order gate 448/600 times. Direct moved 600/600 times (3.1 s/update versus 1.20 s for plain uniform), beat verified uniform on 3/3 seeds (secondary −0.142 [−0.426, +0.143]), and minimized mean loss. Fixed uniform had lower worst loss but raised some minibatch loss 577/600 times (235–285 per designated task).

On CelebA-40 with frozen heads, the primary worst-attribute training BCE difference (Direct − uniform) was −0.018 [−0.021, −0.014]. Uniform/MGDA-UB/Direct moved 8/6/900 of 900 times; Direct finished within 0.0002 of fixed-step BCE. With trained heads, uniform passed 898/900 gates and did slightly better (+0.013 [+0.002, +0.023]). The guarantee is per update and minibatch, in floating point for SGD; fixed-set audit losses never rose. In exact arithmetic, MGDA passes whenever any weighting does.

  1. Headline claims (weakness 2). We will identify each protocol and replace 14.9× with the updated timings (item 10).

  2. CelebA-40 and QM9 (question 1). Over 20 CelebA seeds, the primary worst-attribute accuracy difference (Direct − MGDA-UB) was +0.9 points [−4.1, +5.8], inconclusive. Direct won on mean/balanced accuracy in 19/20 seeds each and cross-entropy in 20/20. Uniform led on worst-attribute accuracy (67.7% vs. 62.6%), mean accuracy and cross-entropy; these replace Table 18. On QM9 (7 methods, 2 epochs, 3 seeds), Direct beat MGDA-UB in 3/3 seeds and Nash-MTL in 2/3, was within 0.012 of FairGrad/Aligned-MTL, and trailed FAMO.

  3. Block 4 and Stream (question 2). Uniform descends on all nine panels (objective factor 1.15–4.26; worst-task gain 0.11–0.97), versus Stream's factor ≤1.002 and gain ≥0.955 (Table 20). Beyond Table 27's M=512 case, Stream is not faster than Direct when uniform fails; its benefit is memory.

  4. 5% certificate (question 3). This certifies the current minibatch step with relative direction error ≤√0.05 ≈22%; Stream's positive-margin stop ensures every task's first-order descent. On CelebA, cost depends on the descent target, not ε. Strict descent and target-aware queries cut recovery from 1.85 to 1.08 s.

  5. Intermediate rule (question 4). Any c∈(0,1) in Eq. (27) ensures strict decrease. On Pythia-1B, c=10⁻⁴ yielded a 6× larger median step than c=1/2 and lower validation loss at 100 updates, but no separation at 200 (+0.006 [−0.531, +0.542]). We will pair 60/60 with the tuned-step comparison in the abstract.

  6. Periodic comparison (question 5). Block 5 adds an exact correction to MGDA-FA every fifth update, with a damped Frank–Wolfe step and no margin gate (cadence 5, damping 0.5 tuned on a development seed; 20 refreshes). Its 0.206 s is average time/update at M=32, versus Appendix G's 6.24 s median per certified recovery at M=16 in another run. On Pythia-1B, all methods used learning rates selected on the same development seed. The primary periodic − MGDA-FA result was inconclusive (−0.037 [−0.464, +0.390]). Worst-task losses: uniform 0.774, periodic 0.781, MGDA-FA 0.818, CAGrad 0.846, FAMO 0.878. This replaces Block 5 and the “strongest cheap baseline” claim.

  7. Block gates (clarification 6). Blocks 2, 3, 6, 7 have run output; Blocks 1 and 4 lack transcripts and will be marked uncertified; Block 5 had no certificate. The positive-part bound holds for either margin sign: the gate specifies a descent target. We will clarify B.5 and the text after Eq. (21).

  8. Live stops (clarification 7). Appendix G's 19 non-positive measured objectives arose because its Gram-scaled guard had median 7.1× the upper endpoint. Removing slack enabled early certification on 889/900 CelebA updates (894/900 target-aware), with fallback near stationarity and all audits passing.

  9. NTK/ghost-gradient (clarification 8) Direct contracts stored layer factors into task Grams. On Pythia-160M (8/32 tasks), it was 1.19×/1.22× faster than structured NTK (Novak et al., 2022) and 1.7×/4.6× faster than ghost-gradient products (Wang et al., 2025), at equal memory, both adapted to task Grams. On Pythia-1B, gains were 1.03×/2.3×; per-task backprop ran out of memory. On CelebA-40, Stream used 0.54 GiB versus 2.09 GiB (per-task backprop) and 8.04 GiB (Direct). Full table in repository.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all the reviewers for their time and thoughtful feedback! We have conducted all the requested studies, along with several additional ones. They are listed below, together with the questions they address. Detailed numerical results are provided in our individual responses. Code, proofs and a checker for all the claims can be found at, https://anonymous.4open.science/r/ParetoTrace-anonymous-37B6/README.md.

  1. We benchmarked ParetoTrace-Direct against structured NTK contraction, ghost-gradient products and per-task backpropagation on Pythia-160M and Pythia-1B. ParetoTrace-Direct is the fastest of the four at every size and runs at Pythia-1B, where per-task backpropagation runs out of memory (ZNzr first weakness and first question, P8RF clarification 8, 6wXk second question).

  2. Training under the verified step on CelebA-40 and Pythia-1B. The verified step takes an update only if every task's minibatch loss is verified to decrease. Under it no task's minibatch loss rose, and where tasks share every trained parameter ParetoTrace-Direct moved at every update while uniform stalled (P8RF first weakness and fourth question, 6wXk first weakness and first question).

  3. A 20-seed CelebA-40 study, QM9 with seven methods, and uniform controls on the nine block-4 panels, regenerated from the same generator and seeds (P8RF first and second questions, 6wXk first weakness and first question).

  4. A six-rule Pythia-1B training study under equal tuning (6wXk second question, P8RF fifth question).

  5. A new guard and target-aware queries for ParetoTrace-Stream, which certified 894 of 900 live CelebA-40 updates early and cut live recovery from 1.85 s to 1.08 s (6wXk second weakness and third question, P8RF third question and clarification 7, ZNzr second weakness and second question).

  6. A new theorem on choosing Stream's queries. The completion query rule closes the certification interval in at most min(M, rank R + 1) exact actions, and the full proof is in the repository (6wXk fourth question).

  7. Revision. The paper will add these results and replace Table 18 and §6.1's ordering against uniform with the 20-seed results and the 14.9× figure with the new timings. It will remove the block-5 periodic result, mark the Stream rows of blocks 1 and 4 uncertified (P8RF second weakness and clarification 6), and specify each method in one place (ZNzr third weakness).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.