- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TraceNAS, a training-free proxy for neural architecture search in LLM pruning. It uses gradient-trace correlation to estimate how well a pruned architecture preserves the optimization behavior of the original model, then searches candidate architectures before applying recovery training. Experiments on several LLM families evaluate the proxy, the quality of selected architectures, and the search cost. The main claim is that TraceNAS can reduce the cost of architecture search while maintaining competitive post-recovery performance.
Critical strengths
-
The paper addresses an important computational bottleneck. Training-aware architecture search can be expensive when each candidate requires substantial recovery training. A proxy that can rank candidates without training each one is potentially useful.
-
The proposed proxy is evaluated rather than simply assumed to work. The paper studies its correlation with recovered-model performance and includes comparisons with alternative proxy choices. This provides some empirical support for the central idea.
-
The evaluation covers multiple LLM families and search-cost measures. The experiments report model quality as well as search time, token usage, and memory consumption. This offers a more complete picture than evaluating final accuracy alone.
Critical weaknesses
-
The proxy validation does not fully establish reliability in the actual search setting. Some correlation results are obtained in a different architecture space or under different recovery settings from the main search experiments. The paper should more clearly demonstrate that the proxy ranks candidates reliably under the same conditions in which it is used.
-
The efficiency comparisons are not fully matched. The reported search-time and resource savings involve different baselines and hardware configurations. Wall-clock time, total GPU-hours, and token usage should be reported separately under clearly specified settings. Otherwise, the size of the practical efficiency advantage is difficult to assess.
-
The final quality gains are not sufficiently supported by variability analysis, and some textual claims do not match the tables. For example, the discussion of the LLaMA-3.1 comparison says that DarwinLM is worse only on MMLU, while the reported results also show lower scores on PIQA and HellaSwag. In addition, the relatively small differences in some final accuracy results need stronger evidence that they are robust across independent recovery runs.
Questions for authors
-
Can you validate the gradient-trace correlation proxy within the same architecture search space and recovery setup used in the main experiments, and report how stable the candidate rankings are across calibration samples?
-
Can you provide a clearer efficiency comparison that reports wall-clock time and total GPU-hours separately, using the same hardware and accounting for the full search and recovery costs?
-
Can you report variability across independent recovery runs for the main quality comparisons and correct the discussion of the LLaMA-3.1 results so that it matches the reported table?