acceptodds
Under review as a conference paper at ICLR 2027

TraceNAS: Neural Architecture Search for LLM Pruning via Gradient Trace Correlation

Abstract

Structured pruning is essential for efficient deployment of Large Language Models (LLMs). The varying sensitivity of LLM sub-blocks to pruning necessitates the identification of optimal non-uniformly pruned models. Existing methods evaluate the importance of layers, attention heads, or weight channels in isolation, ignoring the complex global structural dependencies that exist across the model. Training aware search for structured pruning addresses these dependencies by fine-tuning candidates during the search, making the search nearly as expensive as the recovery training that follows. We propose TraceNAS, a Neural Architecture Search (NAS) framework for joint depth and width structured pruning that replaces this with a training free search: each candidate is scored without any weight updates, using a scale-invariant zero-shot proxy whose alignment with the pretrained model correlates with its potential to recover accuracy during post-pruning training. By removing weight updates from the search loop, TraceNAS discovers high-potential pruned architectures on a single A100 GPU in 8.5 hours, yielding up to 7.5 reduction in search wall clock, 1.35 reduction in search peak memory, and 10 fewer tokens processed during search compared to training-based search baselines. Evaluations on the Llama and Qwen families show TraceNAS matches training aware search performance at a fraction of the search cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Compression, Neural Architecture Search, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NvmqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TraceNAS, a training-free proxy for neural architecture search in LLM pruning. It uses gradient-trace correlation to estimate how well a pruned architecture preserves the optimization behavior of the original model, then searches candidate architectures before applying recovery training. Experiments on several LLM families evaluate the proxy, the quality of selected architectures, and the search cost. The main claim is that TraceNAS can reduce the cost of architecture search while maintaining competitive post-recovery performance.

Critical strengths
  1. The paper addresses an important computational bottleneck. Training-aware architecture search can be expensive when each candidate requires substantial recovery training. A proxy that can rank candidates without training each one is potentially useful.

  2. The proposed proxy is evaluated rather than simply assumed to work. The paper studies its correlation with recovered-model performance and includes comparisons with alternative proxy choices. This provides some empirical support for the central idea.

  3. The evaluation covers multiple LLM families and search-cost measures. The experiments report model quality as well as search time, token usage, and memory consumption. This offers a more complete picture than evaluating final accuracy alone.

Critical weaknesses
  1. The proxy validation does not fully establish reliability in the actual search setting. Some correlation results are obtained in a different architecture space or under different recovery settings from the main search experiments. The paper should more clearly demonstrate that the proxy ranks candidates reliably under the same conditions in which it is used.

  2. The efficiency comparisons are not fully matched. The reported search-time and resource savings involve different baselines and hardware configurations. Wall-clock time, total GPU-hours, and token usage should be reported separately under clearly specified settings. Otherwise, the size of the practical efficiency advantage is difficult to assess.

  3. The final quality gains are not sufficiently supported by variability analysis, and some textual claims do not match the tables. For example, the discussion of the LLaMA-3.1 comparison says that DarwinLM is worse only on MMLU, while the reported results also show lower scores on PIQA and HellaSwag. In addition, the relatively small differences in some final accuracy results need stronger evidence that they are robust across independent recovery runs.

Questions for authors
  1. Can you validate the gradient-trace correlation proxy within the same architecture search space and recovery setup used in the main experiments, and report how stable the candidate rankings are across calibration samples?

  2. Can you provide a clearer efficiency comparison that reports wall-clock time and total GPU-hours separately, using the same hardware and accounting for the full search and recovery costs?

  3. Can you report variability across independent recovery runs for the main quality comparisons and correct the discussion of the LLaMA-3.1 results so that it matches the reported table?

Reviewer_2PHwOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TraceNAS, which uses gradient correlation with the dense model to score candidates in an evolutionary search over model depth and width. Candidate weights are frozen during search, and the selected architecture subsequently undergoes continued pretraining.

Critical strengths

The paper includes downstream evaluation, controlled comparisons under a common recovery schedule, and useful ablations of the scoring function. Reducing the cost of evaluating candidate architectures is a reasonable objective.

Critical weaknesses
  • The methodological advance over existing search methods seems limited. Previous work like Tyr-the-Pruner already searches without training candidates and uses alignment with the dense model for selection. The main addition here is gradient-based scoring. A controlled comparison against output KL within the same search pipeline would help establish whether the extra gradient computation is worthwhile.

  • The practical advantage remains unclear. The paper emphasizes cheaper search, while the selected model still requires substantial recovery training. Comparisons with readily available smaller models at similar deployment budgets are missing. The combined cost also matters: Table 16 shows DarwinLM reaching similar quality with less recovery training.

  • The explanation of functional preservation goes too far. Local gradient correlation is described as keeping candidates in the pretrained convergence basin and preserving their functional state. The paper provides little support for those interpretations. On Llama-3.1, substantially better perplexity also comes with considerably worse MMLU performance than DarwinLM, which deserves discussion.

  • The recovery-potential analysis mixes architectures of different sizes. Parameter count itself ranks these candidates well, and TraceNAS already shows strong correlation before recovery starts. I would like to see whether the score remains informative among similarly sized candidates after accounting for their initial loss.

  • Most comparative evidence still comes from older models. The main experiments center on Llama-2, Llama-3.1, and Qwen2.5. Qwen3 is a useful addition, but its experiments only include the dense baseline. Comparing against existing pruning methods would make the evidence much more relevant (and up-to-date).

Questions for authors
  • How does the total search and recovery cost compare with DarwinLM when targeting similar final quality?

  • Under the same search space and recovery setup, how does gradient correlation compare with output KL?

Reviewer_tjsHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

TraceNAS proposes using Evolutionary Architecture Search to jointly prune the depth and width of pre-trained LLMs. Candidate architectures are scored based on the weighted correlation between their low-rank gradients and those of the original pre-trained model. No parameter updates occur during the search process. The selected architecture subsequently undergoes continued pre-training to recover performance. The study experiments on the Llama and Qwen model families, evaluating the ranking quality of this proxy metric, downstream performance, and the computational cost of the search.

Critical strengths
  1. Clear writing, easy to follow.
  2. The author provides a detailed discussion of design choices and their justification.
  3. The idea of ​​using the pre-trained model's gradients as a reference to rank pruned architectures while avoiding intermediate training is interesting.
  4. The paper evaluates multiple baseline methods under a consistent performance-recovery setup, and the experiments cover a diverse range of model families.
Critical weaknesses
  1. Performance gains are inconsistent. Please see question 1.
  2. The incremental benefit of the proxy metric is unclear. Please see question 2.
Questions for authors
  1. For the Llama-3.1 model, although TraceNAS achieves better perplexity, its MMLU performance is significantly worse than DarwinLM. I find this puzzling; could the authors explain why this might be the case?
  2. Table 1 shows that this proxy metric's ranking correlation is similar to GradNorm's. How do the two methods compare when ranking similarly sized architectures, especially among the higher-performing candidates?
  3. Both TraceNAS and the gradient-based global pruning approach leverage gradient information from the model’s loss function to guide the finding of non-uniform structured parameter subsets. Could the authors discuss how these methods relate and compare them with other recent gradient-based structured pruning techniques, such as GISP [1], GRASPrune [2], and DDP [3]?

[1] Ziyan Wang et al. From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models. ACL 2026. [2] Ziyang Wang et al. GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models. ACL 2026. [3] Weiyu Huang et al. Deterministic Differentiable Structured Pruning for Large Language Models. ICML 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.