acceptodds
Withdrawn from ICLR 2027

Recursive Self-Improvement Agent for Adaptive Symbolic Pruning Strategy Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training pruning is indispensable for deploying large language models (LLMs), and the importance metric that ranks weights largely determines the quality of the sparsified model. Hand-crafted metrics such as Wanda and SparseGPT encode fixed human intuition and do not adapt to new architectures or high-sparsity regimes, while automated symbolic discovery methods such as Pruner-Zero run genetic programming (GP) with frozen hyperparameters and a frozen evaluation pipeline, which leads to premature convergence and a prohibitive GPU cost per candidate. We propose PrunerAgent, an agent-based recursive self-improvement (RSI) framework for symbolic pruning-metric discovery. PrunerAgent nests two loops: an inner GP loop evolves symbolic trees over weights, activations, and gradients, and an outer meta-agent monitors population statistics and adapts the GP hyperparameters, compute budget, seeding strategy, grammar constraints, and validation thresholds of the inner loop. The search process itself, not only its output, is therefore improved during the run. Two further components reduce cost: an agent-managed symbolic search space that canonicalizes expressions and prunes theoretically defective subexpressions before evolution, and a six-stage validation pipeline that combines symbolic reasoning with progressively more expensive proxy evaluations so that only promising candidates reach full GPU evaluation. On LLaMA-1/2/3 and Mistral models at 50–80% unstructured and 4:8/2:4 semi-structured sparsity, consistently outperforms Pruner-Zero and hand-crafted baselines while reducing search cost by 62%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Model compression, network pruning, symbolic pruning strategy, recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CGadOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
2
Presentation
1(−1)
Soundness
2
Read review
Paper summary

This paper addresses the limitation of hand-crafted and static automated pruning metrics for Large Language Models (LLMs). The authors propose PrunerAgent, an agent-based Recursive Self-Improvement (RSI) framework that dynamically discovers symbolic importance metrics. PrunerAgent features a nested two-loop structure: an inner Genetic Programming (GP) loop that evolves expression trees and an outer meta-agent that monitors population statistics and adjusts hyperparameters, compute budget, and constraints online.

Critical strengths

Nesting a meta-agent outer loop over an inner symbolic GP loop conceptually moves beyond standard self-refinement by optimizing the search algorithm itself online.

Critical weaknesses
  1. Although the meta-agent utilizes LLMs (GPT-5.4) for reasoning, its action proposals are strictly bounded and overridden by a heuristic rule layer. This introduces human inductive bias into the outer loop, which somewhat weakens the claim of "pure recursive self-improvement."

  2. Stage 4 uses a very small calibration slice (16 sequences) to filter candidates, which may cause a risk that non-monotonic or architecture-sensitive metrics might be falsely rejected early.

Questions for authors
  1. In Section 4.5, how sensitive is the meta-agent to the choice of rule thresholds?

2.Could the authors provide a breakdown of the wall-clock time and financial API cost incurred by GPT-5.4 and GPT-5.4-mini calls during the 350-trial search process? How does this monetary/latency overhead compare against the saved GPU compute cost?

  1. Monotonicity (MonoMono) and Alignment (AlignAlign) explicitly penalize metrics that diverge from magnitude and loss-perturbation heuristics. Does this screening risk biasing the search toward variants of Wanda/SparseGPT and preventing the discovery of radically novel pruning paradigms?
Reviewer_LsrROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is a problematic paper that should clearly be rejected.

  1. Figure 1 is obviously AI-generated and contains garbled words. For example, "softmax" in Figure 1(a) is not rendered correctly.
  2. The abstract appears to have been submitted without careful checking, as it still contains the unrendered LaTeX macro "\method{}".
  3. There is a hallucinated citation. The reference arXiv:2311.09755 is titled "On the Impact of Calibration Data in Post-training Quantization and Pruning", but the reference list gives a different title.
  4. On page 21, line 1131, the text claims that σ\sigma is used in the equation, but the preceding equations do not use this notation.
  5. On page 13, line 700, the paper states that the OPT architecture has been evaluated, but no OPT results are reported.
Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

Reviewer_qM3yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PrunerAgent, a framework for discovering automatic model pruning formula based on recursive self-improvement. Its core contribution is a Meta-Agent driven adaptive search framework, in which an outer Meta-Agent uses feedback from previous iterations to dynamically optimize the search strategy of an underlying genetic programming algorithm responsible for discovering symbolic pruning importance metrics. Additionally, the authors combine mechanisms such as symbolic reasoning, experience replay, and six stages candidate validation to improve search efficiency. The authors’ experiments demonstrate that this method achieves better performance than existing pruning methods on multiple models, while reducing GPU search time.

Critical strengths

This paper proposes a self-adaptive framework, in which a Meta-Agent dynamically adjusts the genetic programming search strategy based on historical search results, achieving good experimental performance on model pruning tasks. Additionally, the authors designed a symbolic reasoning mechanism and a multi-stage candidate validation mechanism that can identify and filter out low quality candidate solutions at an early stage, effectively reducing computational overhead and improving search efficiency.

Critical weaknesses

My main concern is that the paper does not sufficiently establish the novelty and generalizability of the proposed framework. The method only combines established techniques, and the experiments are limited to a specific pruning task and architecturally similar models. Although the reported improvements are promising, the current evidence is insufficient to demonstrate a generalizable recursive self-improvement mechanism or a substantial methodological advance. My specific concerns are outlined below.

  1. This paper presents recursive self-improvement as its core contribution, but the current framework and experiments primarily focus pruning formula searching based on genetic programming. Could authors supplement the paper with experiments on other automatic algorithm discovery tasks to validate the generalizability of the proposed framework?

  2. This paper primarily uses LLaMA-1/2/3 and the early Mistral 7B series models, which were released some time ago and having similar structures. Given the significant advancements in current language models in terms of architectural design and model capabilities, we suggest that the authors supplement their work with experiments on more recent models, like Qwen3.5-9B-Base, which having a quite different structure comparing with models used currently in this paper. Would the authors consider evaluating the effectiveness of the proposed method on such models?

  3. This paper achieves good performance and efficiency by dynamically adjusting search parameters via Meta-Agent. However, the current method still relies heavily on predefined rules and manually defined search constraints. Could the authors further analyze what general patterns the proposed method has identified?

Questions for authors

Please see the above part

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.