acceptodds
Under review as a conference paper at ICLR 2027

Auto-LTOpt: Automated Optimizer Evolution for Extreme Class Imbalance

Abstract

Practical learning tasks frequently exhibit imbalanced long-tailed data distributions, posing significant challenges for standard optimizers to generalize to tail classes. Existing long-tailed optimizers rely heavily on hand-crafted rules and limited gradient adjustment strategies, resulting in poor flexibility and generalization. In this paper, we propose Auto-LTOpt, a novel automated framework that uses large language models to automatically design tailored optimizers for long-tailed learning without human intervention. By exploring a rich search space of gradient manipulation operations through program evolution, Auto-LTOpt can autonomously discover effective optimization strategies to balance head and tail classes. Extensive experiments on standard long-tailed benchmarks show that Auto-LTOpt outperforms state-of-the-art handcrafted optimizers, while achieving strong and stable performance across diverse imbalanced settings. Our code is available at Supplement.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long-tailed learning, optimizer, large language model, program evolution, imbalanced classification

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_eWXjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Auto-LTOpt, a framework that uses large language models and evolutionary search to discover optimizers for learning under class imbalance. It explores modifications to update rules, utility functions, and optimizer classes, using validation performance to retain promising candidates. Experiments on CIFAR, ImageNet-LT, and iNaturalist 2018 show competitive performance across several training objectives, supported by component ablations and analyses of the search process. Quite interesting.

Critical strengths
  1. The paper explores automated optimizer design for class imbalance, offering greater flexibility than tuning existing update rules.
  2. Experiments cover four benchmarks and diverse training objectives, providing encouraging evidence of broad applicability.
  3. The discovered optimizer generally improves overall accuracy over strong baselines, including Muon, with promising results on ImageNet-LT and iNaturalist 2018.
Critical weaknesses
  1. The contribution beyond adapting ReEvo could be clearer. Comparisons with LLM sampling and ReEvo under equal budgets would strengthen the evaluation.
  2. The proposed connection between small gradient norms and tail classes needs validation. The implementation also appears to overwrite the direct row scaling. More clarification is encouraged.
  3. Several gains over Muon are small, and tail accuracy occasionally decreases. More explanation is encouraged.
  4. The stated search configuration is difficult to reconcile with the 109 evaluated candidates. The cost analysis also excludes LLM overhead.
  5. The theoretical results mainly describe general properties of elitist search. They provide limited explanation of the discovered optimizer’s effectiveness for rare classes.
Questions for authors

Please refer to weaknesses.

Reviewer_VhhvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents Auto-LTOpt, an application of LLM-driven program evolution to optimizer discovery for long-tailed classification. It utilizes ReEvo to synthesize and evaluate optimizer variants and reports improvements across several datasets and training objectives.

Critical strengths
  1. The paper addresses a relevant problem
  2. It shows improvements over several datasets and training objectives
Critical weaknesses
  1. The paper lacks methodological novelty
  2. The implementation does not use the operation that it touts as its central mechanism
  3. The implementation is not sound
  4. The theoretical contribution does not connect with the empirical contribution
Questions for authors

Please provide the actual candidate count, complete search cost, and search-versus-transfer protocol for every benchmark?

Reviewer_JMerOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the automated design of optimizers for long-tailed classification. Building on ReEvo, the framework uses an LLM to generate, cross over, and mutate optimizer programs at three granularities: the core update function, utility functions, and the optimizer class. Candidates are evaluated through proxy-network validation performance, and an elite reservoir retains the best programs across generations. The paper reports results under several losses and training frameworks, together with the discovered optimizer implementation, prompts, component ablations, and search-cost analysis. The problem is practically relevant, but important inconsistencies remain between the claimed mechanism, the listed implementation, and the interpretation of the ablations.

Critical strengths

Promising empirical utility across complementary long-tailed training settings. The reported optimizer improves overall accuracy over Muon in 20 of the 21 CIFAR-100-LT configurations in Table 1 and all eight ImageNet-LT/iNaturalist configurations in Table 2. For example, ImageNet-LT accuracy improves from 43.9 to 44.6 under CE, and iNaturalist accuracy improves from 73.4 to 74.0 under BCL. These reported gains across different objectives and dataset scales are the principal positive evidence that automated optimizer adaptation could provide a useful component for existing long-tailed training methods. This empirical utility is the main consideration supporting acceptance, although statistical repeatability and whether a single fixed optimizer is reused across settings remain to be clarified.

Critical weaknesses
  1. Executed implementation and ablations: Is the code used to produce Tables 1, 2, and 5 identical to Appendix D.2? Through what path do the row-wise modifications affect parameter updates after g = X.flatten()? Which optimizer handles 1D parameters? Please provide the corresponding code and ablations with other settings and random seeds held fixed to reconcile the results in Table 5. 
  2. Incremental contribution and repeatability: Figures 2 and 6 provide useful comparisons with LLM-generated optimizers and different search granularities. Which specific components are new relative to the ReEvo configuration used here, and which reported comparisons isolate their effects? For key accuracy differences relative to Muon, what variability is observed across repeated runs? These details would help assess whether the contribution lies primarily in a useful discovered optimizer, an improved discovery framework, or both.
Questions for authors
  1. Executed implementation and ablations: Is the code used to produce Tables 1, 2, and 5 identical to Appendix D.2? Through what path do the row-wise modifications affect parameter updates after g = X.flatten()? Which optimizer handles 1D parameters? Please provide the corresponding code and ablations with other settings and random seeds held fixed to reconcile the results in Table 5. 
  2. Incremental contribution and repeatability: Figures 2 and 6 provide useful comparisons with LLM-generated optimizers and different search granularities. Which specific components are new relative to the ReEvo configuration used here, and which reported comparisons isolate their effects? For key accuracy differences relative to Muon, what variability is observed across repeated runs? These details would help assess whether the contribution lies primarily in a useful discovered optimizer, an improved discovery framework, or both. 
  3. Search and transfer protocol: Which dataset, imbalance ratio, loss, and proxy-training duration were used to discover the final optimizer? Do Tables 1, 2, and 4 reuse a single fixed discovered optimizer, or is search repeated for each configuration? Clarifying this would establish the extent to which the reported results demonstrate reuse of an optimizer across tasks or successful task-specific discovery.
Reviewer_Gxw5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Auto-LTOpt, an LLM-driven framework for automatically discovering optimizers tailored to long-tailed learning. It introduces a three-level optimizer search space and employs program evolution with initialization, crossover, and mutation. Experiments on four long-tailed benchmarks demonstrate competitive performance across various training objectives, generally outperforming existing optimizers.

Critical strengths
  1. Auto-LTOpt achieves consistent improvements across four long-tailed benchmarks and multiple training objectives, demonstrating promising generalization and applicability.

  2. The framework reduces reliance on manually designed optimization rules by combining LLM-based program generation with evolutionary search.

Critical weaknesses
  1. Inconsistency between the claimed method and implementation. The optimizer in Appendix D.2 does not explicitly identify tail classes. More importantly, g = X.flatten() overwrites the preceding row-wise gradient adjustments and 1D boosting. This raises serious concerns about the claimed mechanisms and the validity of the corresponding ablations.

  2. Limited improvements over Muon and insufficient comparisons. The discovered optimizer largely builds upon Muon's Newton–Schulz orthogonalization, with relatively small improvements. Comparisons against simple Muon variants and ProMO are missing, making the additional methodological contribution difficult to establish.

  3. The metho scheme is more like engineering optimization.

Questions for authors
  1. Is the implementation in Appendix D.2 identical to the one used in experiments? How do the row-wise adjustments remain effective after g = X.flatten(), and how are 1D gradients handled?

  2. Can the authors compare against Muon with simple gradient clipping/reweighting and ProMO, preferably across multiple random seeds, to establish that the improvements arise from the discovered optimization strategy?

  3. The proposed framework appears to primarily extend conventional evolutionary program search by incorporating LLMs. Beyond engineering improvements, is there any theoretical justification or principled insight that distinguishes the proposed approach from existing automated optimization methods? If the author can solve this confusion, I will improve my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.