acceptodds
Under review as a conference paper at ICLR 2027

LISC: Balancing Sample Influence with Spectral Reweighting

Abstract

Modern datasets are highly heterogeneous, with predictive signals unevenly distributed across samples. As a consequence, gradients estimated as averages over mini-batches tend to reinforce dominant learning signals, leaving other potentially useful directions weakly represented. This issue cannot be directly addressed by optimizers that operate strictly on gradients already aggregated across samples. We therefore propose a sample-reweighting method, Local Inverse Spectral Curriculum (LISC), which re-balances gradient directions while remaining fully compatible with existing optimizers. Starting from a local linear approximation of per-example losses, LISC assigns weights that reduce the relative influence of strongly represented gradient directions. Its underlying spectral transformation is equivalent to preconditioning with a local gradient second moment, yet it avoids constructing a full parameter-space preconditioner. Across vision and language tasks, architectures, and optimizers, LISC improves early-training test performance, with gains exceeding five percentage points in multiple settings, while matching final performance in vision and improving final generalization in several language tasks. These findings indicate that geometry-aware sample aggregation can complement optimizer-level transformations to accelerate learning without sacrificing, and in some settings even improving, final generalization.

Primary area:
optimization
Keywords:
Sample Reweighting, Curriculum Learning, Gradient Geometry, Spectral Methods, Stochastic Optimization, Training Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CaeDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose to use LISC as a sample reweighting method to emphasize less pronounced yet potentially relevant gradient directions, which is especially relevant for the setting of heterogeneous data. They show that this reweighting is equivalent to preconditioning with a local gradient second moment, whilst avoiding the construction of the parameter-space preconditioning matrix. Empirically, LISC improves early-training test performance and is on par with the provided baselines in final performance.

Critical strengths
  1. Methodological approach: I really like the general idea and the corresponding methodological depth, analyzing the relationship between eigenvalues and the contribution to the gradient, as well as the analysis of the effect of clipping and normalization.
  2. Versatility of LISC
  3. The theory section is very nice to read and tidy and states the scope of the approach pretty clearly.
Critical weaknesses

To me, it feels like the empirical findings do not fully match the story:

  1. A major claim is LISC's advantage in the heterogeneous data setting. Yet I am not sure how heterogeneous the selected datasets are, so I feel the evaluation setup doesn't match the story and might even overshadow a strength of LISC. See question 1).

  2. I feel like the claim about optimization budgets and "how quickly (the model) acquires useful predictive structures" might collapse when taking into account the results from B.9 about the computational overhead of LISC. See question 2).

  3. I see how the generalization claim holds up in early epochs, but its advantage doesn't consistently hold for the final model. Thus, with the results presented as is, together with the concerns stated in weakness 2), I am doubtful about the actual advantage of LISC. See questions 1) and 4) for some potentially underexplored advantages of LISC.

  4. Further comparison baselines are missing. See question 3).

Questions for authors
  1. A major claim is the heterogeneous data setting. In what way are the selected evaluation datasets heterogeneous? Consider adding a more obviously heterogeneous dataset to the evaluation.

  2. Please add test accuracy vs. wall-clock time curves.

  3. Any thoughts on why Aligned-MTL performs comparatively worse than both the base and LISC? What are the chosen hyperparameters of Aligned-MTL? Also, please add further baseline methods.

  4. What is the effect of LISC on the predicted output distribution? Does LISC lead to better or worse calibration than other methods? Does LISC end up in a different local optimum compared to other methods, leading to similar performance yet different strengths?

3 comments

We thank Reviewer CaeD for highlighting the analysis of gradient directions and the effects of clipping and normalization.

Q1. Data Heterogeneity

Our experiments measure how strongly examples reinforce the same gradient directions. Our claim about heterogeneity is based on gradient energy distributed to different bases, which can be one proxy to measure heterogeneity in terms of concepts, structures, or patterns.

Following your suggestion, we evaluated LISC on PACS [1]. We pooled 3 styles of a class for training, allowing a minibatch to contain several styles, and tested on the fourth. We repeated this for each heldout style. Averaged across heldout styles, LISC gains 4.16 pp at epoch 5 and 3.89 pp at epoch 20. The final difference is 0.6 pp. This gives us evidence of the gain due to LISC on unseen styles. We have added this result in Table 2.

Q2. Equal wall-clock time comparison

We agree that LISC adds a computational overhead because of the spectral transformation. To assess the acceleration in terms of the wall-clock, we have added a test accuracy vs wall-clock plot. The plot (Fig 4) shows that LISC can also be pareto optimal compared to the baseline even when we compare against the wall-clock. For instance, for MMLU, LISC achieves 58.78% in 7.11 min compared to AdamW which achieves 58.41% in 7.69 min. Similarly, for CIFAR-10, LISC reaches 83.73% in 1.61 min, whereas SGD reaches 81.52% at the same wall-clock time.

W3. Early and final gains consistency

Across 11 settings in Table 1, we observe that LISC has higher mean heldout performance vs Base at the first reported checkpoint in all 11, at the middle checkpoint in 8, and at the final checkpoint in 8. Its mean AUC is also higher in all 11 settings. The three lower final means occur on C10, by 0.03, 0.09, and 0.20 pp. On C100 with SGD, the gain narrows from 5.02 pp at epoch 5 to 0.71 pp at epoch 200. We agree that the most consistent benefit is earlier improvement at a fixed update budget. We acknowledge that the final gains are smaller and depend on the setting.

Q3. Aligned-MTL performance and configuration

Aligned-MTL was proposed to balance task gradients, while we treat each example as an objective. To make the evaluation feasible, we applied Aligned-MTL’s spectral rule also in the same measurement coordinates used by LISC (Appendix B.2). Aligned-MTL’s transformation brings every singular value down to the smallest retained value. Consequently, a single small retained value can reduce the aggregate gradient in the classifier-head parameters. During epochs 1–5, the median ratio of the Aligned-MTL gradient norm to the ordinary gradient norm was 0.42% on CIFAR-10. In addition, Aligned-MTL was primarily proposed for scene understanding, and the authors didn’t evaluate their performance in the image classification settings.

We would also like to clarify that Aligned-MTL does have competitive results in some cases. For instance, on CIFAR-10/SGD, it exceeds Base at epochs 5 and 20. On GSM8K, its final exact match is 49.89% vs 49.33% for AdamW and 50.59% for LISC (Table 1).

Aligned-MTLOptimizerPhysical / effective batchPeak LRWD
C10 and C100, RN-18SGD, Nesterov 0.9128 / 1280.10.0005
GSM8K, Qwen3-0.6B + LoRAAdamW4 / 320.00020.002
CommonsenseQA, Qwen3-0.6B + LoRAAdamW8 / 320.00040.01

In addition, our Aligned-MTL runs use the Base optimizer settings, without an independent Aligned-MTL search. We have clarified this in Appendix B.2 in the revised manuscript. 

Additional baselines. 

We evaluated lagged-loss weighting, in-batch class balancing, and adaptations of GradNorm, GradVac, CAGrad, and IMTL-G [2–5]. LISC has the highest mean epoch-5 accuracy on both datasets and the highest mean final accuracy on C-100. We have provided the complete result in Table 9 in the revised manuscript.

Output Distribution and Calibration

On MMLU, we observed that the paired prediction analysis shows a trade-off. At the final checkpoint, AdamW and LISC differ by only 0.32 pp in accuracy, yet their answers differ on 23.38% of questions. When wrong, LISC assigns its chosen answer 61.86% confidence on average, compared to 52.72% for AdamW. Expected calibration error (ECE) [6], NLL, and Brier score are all higher with LISC at checkpoints examined. We have summarized the results in Table 11.

Yes, LISC converges to a different optimum compared to base. We have added a plot (Fig 9) that depicts this.

[1] Li et al. Deeper, Broader and Artier Domain Generalization. ICCV 2017.

[2] Chen et al. GradNorm. ICML 2018.

[3] Wang et al. Gradient Vaccine. ICLR 2021.

[4] B. Liu et al. Conflict-Averse Gradient Descent for Multi-task Learning. NeurIPS 2021.

[5] L. Liu et al. Towards Impartial Multi-task Learning. ICLR 2021.

[6] Guo et al. On Calibration of Modern Neural Networks. ICML 2017.

Thank you for the thorough response, the additional experiments, and the transparent communication of the results. Overall, these experiments answer my questions. However, my general concern about the practical benefit in the current evaluation setting remains, and it is partly even reinforced by the warm-up ablation in Table 10. I am also still doubtful about the claim/usefulness on heterogeneous data. While it is now addressed with PACS, the benefit there does not seem substantially different from that on the original datasets. I will therefore keep my score.

Furthermore, from how I’ve understood Table 11 and B.10, LISC yields a model with worse calibration (at least for MMLU). I think this is a very interesting insight. Whilst it might not favour LISC at first glance, understanding the underlying mechanism, together with the different convergence optimum of LISC, could give LISC a potential that the current accuracy evaluation does not show (e.g., for noisy label data sets?). Given how principled the theoretical approach is, I think a more holistic analysis of the solutions LISC converges to could reveal use cases where the method really shines. I am looking forward to seeing what this direction holds, and I believe that convincing results there could turn LISC into a very meaningful and relevant contribution.

Thanks a lot for the follow-up and for appreciating our theoretical approach. We would like to clarify the following points.

Heterogeneous data. On PACS, LISC achieves higher mean accuracy than the baseline at every evaluated checkpoint, including 35.33 ± 2.20 vs 34.71 ± 1.24 at the final checkpoint. The final gain is modest, and the consistent improvements support LISC’s usefulness when training across styles and evaluating on an unseen style.

EpochSGDSGD + LISC
522.83 ± 1.0226.99 ± 1.75
2027.26 ± 4.2531.15 ± 2.49
5034.71 ± 1.2435.33 ± 2.20

Practical benefit and warm-up. Table 10 shows that warm-up improves performance in the image settings and also benefits LISC, suggesting that the two are complementary. Warm-up hurts performance in the evaluated language settings, whereas LISC’s early improvements are consistent across vision and language tasks. The wall-clock results also demonstrate practical gains in the reported settings despite LISC's additional computational cost.

Calibration and learned solutions. We agree that calibration results are not positive for LISC. Our claim about the regularizing effect of spectral balancing does not imply improved calibration. The prediction disagreement despite similar accuracy also motivates us to examine the learned solutions beyond accuracy.

We appreciate this constructive suggestion. We believe LISC offers a useful contribution through both its principled approach to gradient preconditioning using data geometry and its demonstrated gains in early performance across various settings.

Reviewer_3rViOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LISC, which reweights the examples in each mini-batch using the Gram matrix of their per-example gradients, by default measured on the classifier head or on the LoRA adapters. A damped inverse-square-root filter gives less weight to the dominant gradient directions. The authors show this is equivalent to a form of inverse-square-root preconditioning in the measurement coordinates, and that the reweighted gradient is still a descent direction. Across 11 vision and language settings with SGD, AdamW and Muon, LISC improves early-training accuracy by up to about 5 points, with similar or slightly better final accuracy.

Critical strengths
  • The link between sample weighting and preconditioning is clean. Filtering the small b x b Gram matrix gives the same result as preconditioning in the measurement coordinates. This is a principled reason to act at the aggregation step rather than in the optimizer, and it is cheap to compute. I found this the most interesting part of the paper.

  • The evaluation is broad and consistent in direction. Early accuracy and AUC improve in all 11 settings, across three optimizer families and both full training and LoRA. A norm-matched control and a hyperparameter sweep help rule out "it is just a smaller learning rate".

Critical weaknesses
  • The practical benefit is unclear once cost and noise are counted. The gains are per optimization step, but LISC adds 7-21% wall-clock time on CIFAR and about 40% on ImageNet and the LoRA tasks. Many early gains are within one standard deviation over 3 seeds, and final differences are mostly a few tenths of a point. A comparison at equal wall-clock time is needed to support the claim that LISC accelerates learning.

  • It is unclear whether the spectral mechanism explains the gains. In Table 6, even gamma = 0.99 barely changes the weights, yet it already reaches 75.9 at epoch 5 versus 71.1 for the baseline. Any mild down-weighting of easy examples might work just as well. Gamma was also tuned on test accuracy, and the weight decay differs between Base and LISC in three settings. Finally, the only competing reweighting method is Aligned-MTL, although the released code already contains an InfoBatch implementation.

Questions for authors

Clarifications:

  • Can you plot test accuracy against wall-clock time for at least CIFAR-100, ImageNet and one LoRA task?
  • Why does gamma = 0.99 already give most of the early gain? Could you extend Table 6 to larger gamma values?
  • Does a short learning-rate warmup for the SGD baseline close the early gap?
  • How do simple baselines compare at the same cost, such as in-batch class balancing, loss-based weights or InfoBatch?
  • How does the effect change with batch size, given that the Gram matrix has at most b modes?
  • The released code has no configs for the Muon, CCT-7 and GPT-2 runs, and the README's CIFAR-10 command uses a different weight decay from Table 3. Could you add the missing configs and fix the mismatch?

Suggestions:

  • Choose gamma on a validation split, and report the main table with identical hyperparameters for Base and LISC.
  • Consider dropping "curriculum" from the name, or explain why it fits.
3 comments

We thank Reviewer 3rVi for highlighting the connection between sample weighting and preconditioning. 

W1. Practical benefit and variability. Table 1 shows that LISC improves mean early performance and AUC in all 11 settings at matched numbers of updates and this lead is preserved in most of the cases till the end.

We agree that LISC adds a computational overhead because of the spectral transformation. To assess the acceleration in terms of the wall-clock, we have added a test accuracy vs wall-clock plot. The plot (Figure 4) shows that LISC checkpoints can also be pareto optimal compared to the baseline even when we compare against the wall-clock. For instance, for MMLU, LISC achieves 58.78% in 7.11 min compared to AdamW alone that achieves 58.41% in 7.69 min. Similarly, for CIFAR-10, LISC reaches 83.73% in 1.61 min, whereas SGD only reaches 81.52% in the same measured time.

W2. Spectral mechanism and larger γ\gamma. The filter retains a fraction γ/(1+γ)\gamma /(1+\gamma ) of the original leading-mode energy (Section 4.2). This is 2.9% at γ=0.03\gamma =0.03, 49.7% at 0.990.99, and 99.0% at γ=100\gamma =100. So, we would like to mention that γ=0.99\gamma =0.99 does change the weights. However, with γ=100\gamma =100, the transformation is almost similar to plain SGD. The empirical results for γ=100\gamma =100 also shows SGD like behavior. We have extended Table 7 to include the results for γ=2, 10, 100 \gamma =2,\ 10,\ 100{\ }. The extended CIFAR-10 sweep gives mean epoch-5 accuracies of 74.28%, and 71.90% at γ=2,\gamma =2, and 100100, compared with 71.14% for SGD. This trend is consistent with weaker filtering at large γ\gamma.

Does SGD warmup close the gap? On CIFAR, largely yes. With five epochs of linear warmup from 0.01 to 0.10, SGD reaches 76.35% on CIFAR-10 and 47.39% on CIFAR-100 at epoch 5. LISC without warmup reaches 76.48% and 47.70%. However, adding warm-up also improves LISC. So, warmup and LISC can also be used as complementary to improve acceleration for image classification.

However, we do not observe such acceleration in language tasks. Adding warmup actually hurts the performance of both SGD and LISC over all epochs. On GSM8K, adding LISC to SGD with the same 21-update warmup improves the performance from 46.89% to 47.57% at epoch 1 and from 48.60% to 49.13% at epoch 3. Similarly, warmup also hurts the performance in all the epochs for MMLU. We have provided the complete result in Table 10 in the revised manuscript.

Additional baselines. 

We evaluated lagged-loss weighting, in-batch class balancing, and adaptations of GradNorm, GradVac, CAGrad, and IMTL-G [2–5] on CIFAR-10, CIFAR-100, and MMLU. Results are summarized in the table below. LISC has the highest mean epoch-5 accuracy on all three cases and the highest mean final accuracy on CIFAR-10 and MMLU. We have provided the complete result in Table 9 in the revised manuscript.

Validation selection and weight decay. We chose γ=0.03\gamma =0.03 for all the datasets using the early CIFAR-10 test AUC (Table 7). So, for CIFAR-10, we agree that the test curve influenced this choice. However, this is not true for other cases, because we stick with γ=0.03\gamma =0.03 for all the experiments. We do agree that, for different cases, if we choose γ\gamma based on hold-out performance for each datasets separately, then the chosen γ\gamma could have been different.

To check if this is the case, in a separate experiment, we selected γ\gamma by mean epoch-200 accuracy over a stratified 5K holdout set from the training set, then retrained on all 50K examples. The best holdout final accuracy were observed with γ=0.10\gamma =0.10 on CIFAR-10 and 0.03 on CIFAR-100. With this γ\gamma, the mean official-test epoch-5 accuracy improves from 71.14% (SGD) to 76.44% (LISC, previously 76.48 for γ=0.03\gamma =0.03) for CIFAR-10 and from 42.68% to 47.70% for CIFAR-100. Similarly, the corresponding final means are 95.40% vs 95.32% and 78.26% vs 78.97%.

Batch size.

Figure 7 already shows CIFAR-100/SGD epoch five gains of +5.02, +10.28, and +14.69 pp at batch sizes 128, 256, and 512 with fixed WD and linearly scaled LR wrt the batch size. On GSM8K, holding effective batch size at 32, LR at 0.0002, and WD at 0.002 gives epoch one gains of +1.29 and +0.94 pp for physical batches of 4 and 8. We have clarified this distinction in Appendix B.7.

Code and terminology.

Thanks for noticing the missing configurations. We have added the missing configurations and fixed the README discrepancy. We have clarified “curriculum” at its first use. LISC keeps the sampling order fixed and recomputes each example’s weight, so its emphasis on an example can change during training.

[1] Qin et al. InfoBatch. ICLR 2024.

[2] Chen et al. GradNorm. ICML 2018.

[3] Wang et al. Gradient Vaccine. ICLR 2021.

[4] B. Liu et al. Conflict-Averse Gradient Descent for Multitask Learning. NeurIPS, 2021.

[5] L. Liu et al. Towards Impartial Multitask Learning. ICLR 2021.

Thank you for the detailed response and the additional experiments. The holdout-based selection of γ, the extended γ sweep, the new reweighting baselines, and the updated code address several of my concerns.

However, I will keep my score, because my main concern about practical significance remains. First, the warmup experiment shows that a standard SGD warmup already matches most of LISC's early gain on CIFAR-10 and CIFAR-100 (76.35% vs. 76.48% and 47.39% vs. 47.70%). The most visible improvement in the paper therefore seems largely replaceable by a common, cost-free technique, and the rebuttal does not report the numbers for LISC combined with warmup. Second, the wall-clock evidence is limited. The MMLU difference (58.78% vs. 58.41%) is within seed variance, and no wall-clock result is given for ImageNet, where the overhead is largest. On the language tasks, where warmup does not help, the gains of LISC are small (about 0.5-0.7 points on GSM8K).

AuthorsOpenReview ↗

Tables are presented in the

Thank you for your engagement in a discussion with us.

We would kindly like to point out that all additional experiments are added to the updated manuscript. You are specifically looking for Table 10 (Appendix B.9), where LISC + warmup is reported, confirming our earlier statement that LISC+warmup achieves higher performance than SGD + warmup. Warmup is generally improving generalization, but LISC still adds value.

Furthermore, we would like to highlight that wall clock improvements are not the central contribution of our technique but the conceptual insights. The performance gains are consistent across a wide range of experiments.

Reviewer_ybbeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Local Inverse Spectral Curriculum (LISC), a sample-reweighting method that applies a damped inverse-square-root filter to the mini-batch gradient Gram matrix to reduce the relative influence of dominant gradient directions. The authors establish a connection to local second-moment preconditioning and provide an efficient implementation using selected measurement parameters. Experiments across vision and language tasks show improved early-training generalization per optimization step, with largely preserved or modestly improved final performance.

Critical strengths

The paper proposes a new gradient aggregation method to address the dominance of strongly represented gradient directions in mini-batch training. The approach is supported by both theoretical analysis and experiments across vision and language tasks. The presentation is clear and well organized.

Critical weaknesses
  1. The conceptual novelty should be framed more cautiously, particularly in the title, since spectral gradient reweighting is already present in methods such as Aligned-MTL. The main methodological distinction appears to lie in the damped spectral filter and its conversion into nonnegative, normalized sample weights.
  2. Why spectral balancing improves generalization remains insufficiently explained. Weak modes may contain noise, while dominant modes may encode useful signals. The theoretical analysis establishes spectral energy redistribution but does not generally connect it to improved test performance.
Questions for authors

What mechanism explains LISC’s improved test performance despite its sometimes higher unweighted training loss? Does spectral reweighting act as an implicit regularizer, and under what conditions should it improve the trade-off between training fit and generalization?

2 comments

We would like to thank Reviewer ybbe for the constructive feedback and for recognizing the clarity of the presentation.

W1. Novelty relative to Aligned-MTL Senushkin et al. (2023) introduced Aligned-MTL to balance gradients across different tasks while we balance the gradient energy along different bases using per-example gradients information within a mini-batch. We discuss these connections in Section 2 and Appendix D. We have revised the introduction to clarify the specific contributions of LISC. The main distinction is how the two methods treat differences in gradient strength. Aligned-MTL equalizes the nonzero singular values of the gradient matrix. LISC uses a damped filter that reduces these differences while preserving their ordering. In Proposition 2, we also show the equivalence between data side transformation to parameter side preconditioning, which in itself is novel, and an efficient way to precondition gradients. Similarly, in Section 4.2 we show how to compute the classifier-head Gram matrix exactly from the head inputs and loss derivatives, without forming individual parameter gradients. We also demonstrate empirically that using the same measurement coordinates for both methods, LISC achieves higher mean early and final held-out performance than per-example Aligned-MTL in each of the 11 settings in Table 1.

W2. Generalization despite higher unweighted training loss We agree that a mode’s eigenvalue alone cannot tell us whether it carries a useful signal or noise. We have revised the discussion following Proposition 4 to make explicit when spectral reweighting can help generalization and when it can hurt. In Appendix C we provide a concrete example where two features are predictive during training. The first produces the dominant gradient mode, but its correlation with the label disappears at test time. The second produces a weaker mode, but its correlation persists. At initialization, the uniformly weighted gradient gives the steepest local decrease in training loss for a step of fixed length. LISC gives the second feature a larger share of that step. This sacrifices some immediate decrease in training loss, but makes more progress along the feature that remains predictive at test time. We have established this trade-off on the test side in Proposition 10. At equal update norm, LISC followed by Muon gives a strictly larger first-order decrease in population test loss than Muon alone. In this sense, LISC can act as an implicit regularizer by changing which directions the model fits first. The benefit depends on what those directions represent. If the dominant mode carries the most reliable signal and weaker modes mainly contain noise, the same reweighting can hurt. Finite damping limits how strongly LISC changes their relative contributions, but it cannot identify which features will generalize.

In addition, we demonstrate empirically across tasks, datasets, models, and optimizers that the addition of LISC in almost all of the cases aids generalization.

Revision We have revised the introduction to acknowledge prior spectral transformation proposed by Aligned-MTL. We have revised the discussion after Proposition 4 to explain how LISC can improve early test performance even when unweighted training loss decreases more slowly. We have connected this discussion to Appendix C, where reweighting favors the feature that remains predictive at test time, and explain why the same reweighting can hurt when the dominant mode carries the more reliable signal.

Please engage with the authors during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Paper Revision

Dear AC and Reviewers,

Thank you for the careful reviews and continued discussion. We have revised the manuscript to clarify LISC’s contribution and strengthen the evidence behind it. The revision addresses the following concerns:

ConcernsNew evidence and clarification
Additional BaselinesWe have added six weighting and gradient-balancing baselines (Table 9). LISC has the highest mean epoch-5 accuracy on all the datasets tested and the highest mean final accuracy on CIFAR 100, and MMLU.
Pareto OptimalityDespite its extra computational cost, LISC can achieve higher accuracy in the same or less wall-clock time in multiple settings (Figure 4).
Extension to heterogeneous dataOn PACS, LISC improves unseen-style accuracy by 4.16 points at epoch 5 and 0.62 points at the final epochs (Table 2).
Differences with prior spectral methodsWe differentiate our spectral transformation against Aligned-MTL. In addition, we explicitly formulate the equivalence between sample weighting and gradient preconditioning, which is novel in itself.
Why spectral balancing improves generalization?Appendix C and Proposition 10 gives a setting where LISC favors a feature that remains predictive at test time, improving first-order test-loss reduction at equal update norm.
Warmup AblationWarmup recovers much of the early CIFAR gain, but also benefits LISC; it does not help in the evaluated language settings (Table 10).

The revised paper supports LISC’s value through its principled connection between sample weighting and preconditioning, together with consistent early improvements across diverse training settings. Thank you again for your careful feedback and continued discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.