- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ELAX, an LLM-guided evolutionary NAS framework targeting heterogeneous edge deployment. The pipeline consists of two primary phases:
- Elastic Supernet Construction: A pretrained backbone (e.g., ResNet, ViT, Swin, DeiT) is converted into a weight-sharing supernet using a two-stage training scheme involving feature-level knowledge transfer and task-level fine-tuning across sampled subnets.
- Hardware-Aware Subnet Extraction: The supernet parameters are frozen, and a multi-objective evolutionary search (NSGA-II) extracts subnets that optimize validation accuracy against device-specific latency constraints.
Rather than generating complete architectures from scratch, ELAX uses a local LLM (Llama 3.1 8B) as a constrained mutation controller. The LLM proposes local structural edits conditioned on parent encodings, search history, target hardware profiles, and latency bounds. Every proposal passes through a deterministic validator to ensure search-space compliance, falling back to random mutation if malformed.
The authors evaluate ELAX on elastic CNNs and Transformers across CIFAR-10, CIFAR-100, and ImageNet-1K, alongside NAS-Bench-201 and HW-NAS-Bench across five hardware profiles (edgegpu, raspi4, pixel3, eyeriss, fpga).
Critical strengths
-
Restricting the LLM to local mutation proposals validated by a deterministic schema solver solves the hallucination and search-space violation issues common in LLM-NAS. Combining this with an elastic supernet allows LLM-guided candidate edits to be evaluated instantly without retraining.
-
Decoupling the one-time supernet training from the hardware-specific Pareto search allows a single trained supernet to be reused across multiple edge targets and latency limits without repeating full training cycles.
Critical weaknesses
-
The hardware-aware evaluation relies on static lookup tables (HW-NAS-Bench) that sum isolated operator latencies. Real edge execution is heavily influenced by compiler graph fusions, SRAM/DRAM transfer costs, cache behavior, and memory bandwidth bottlenecks. Static LUTs fail to capture these dynamic harware behaviors.
-
The evaluation focuses entirely on static vision backbones (CNNs/ViTs) on image classification datasets. Modern edge AI has pivoted heavily toward small language models and generative decoders where KV-cache management and autoregressive memory bandwidth dominate execution. Also, omitting low-precision quantization choices (INT8/INT4) leaves out a primary dimension of real-world edge hardware co-design.
-
The paper omits total search wall-clock time and LLM inference overhead. While candidate scoring on the frozen supernet is fast, querying a local Llama 3.1 8B model via Ollama for 20% of the evolutionary offspring introduces sequential generation latency. Without reporting end-to-end wall-clock search time and the exact fraction spent waiting on LLM calls, it is unproven whether ELAX provides a net time savings over standard multi-objective Bayesian optimization baselines (e.g., qLogEHVI).
Questions for authors
-
How does ELAX perform when evaluated against direct on-device profiling or compiler-in-the-loop latency measurements rather than static lookup tables?
-
Can the LLM mutation controller and template menu be expanded to co-design layer-wise quantization precision (e.g., INT8 vs INT4), and how does that impact the validator rejection rate?
-
What is the total wall-clock runtime for a complete 500-evaluation search run, and what fraction of that time is spent strictly on LLM inference latency?
2 comments
Thank you for the responses and updates. They clarified a couple of my questions.
That said, two key limitations remain: (1) the newly extracted elastic subnets were not compiled or profiled across the physical edge targets, and major edge considerations like mixed-precision quantization and autoregressive decoders remain unaddressed, and (2) spending 557.1s out of 578.7s on local LLM calls makes the search ~20x slower than qLogEHVI.
Balancing the strengths and weaknesses of this paper, I will maintain my score of weak acceptance.
We thank the reviewer for recognizing the value of validated local edits and reusable elastic weights. The comments help us distinguish the hardware evidence available in the present study from deployment measurements that remain to be made. We have clarified the use of benchmark latency in §5.1 and Appendix A.8, stated the deployment and model-scope limitations in §6, and reported search-loop wall time in Appendix A.5.
Q1. How does ELAX perform with direct on-device or compiler-in-the-loop profiling?
We have not run the five-profile search with online compilation and profiling, so we cannot claim that its Pareto rankings or latency feasibility would be preserved under that protocol. The hardware comparison queries stored HW-NAS-Bench latency for NAS-Bench-201 architectures; it does not compile and measure each candidate during ELAX search (§5.1 and Appendix A.8). A distinction about those records is useful. For NAS-Bench-201, HW-NAS-Bench provides architecture-level latency values. Its Edge GPU, Raspberry Pi 4, and Pixel 3 latency data come from measurements of compiled networks on devices; its Eyeriss and FPGA values are estimates. The benchmark’s approximation formed by summing block costs is described for its FBNet search space, rather than the NAS-Bench-201 records used here. These distinctions follow the HW-NAS-Bench paper. The stored records support a reproducible, device-specific comparison of search methods under the same 500-evaluation budget. They do not establish the latency of newly compiled ELAX elastic subnets under different compilers, workloads, or memory conditions. The separate ResNet-50 study measures subnet latency on an RTX 4060 Ti (§5.2), but is not a deployment study across the five edge targets. We identify search and deployment of elastic subnets on multiple physical devices as future work (§6).
Q2. Can the controller include INT8/INT4 choices, and how would they affect validator rejection?
The constrained-edit approach could be extended to include device-supported precision choices in its legal menu. The validator would then need to check the proposed precision assignments against operator support, tensor and residual-path compatibility, and the target compilation constraints. Evaluating accepted candidates would also require quantized-accuracy and compiled-device latency measurements. This is a proposed extension, not an experiment reported here. The present edit menu does not include layer-wise bit-width choices, so we cannot report a quantization-specific rejection rate or an INT8/INT4 accuracy–latency result. The rate could change with the device and legal menu; it cannot be inferred from the current controller audit. Section 6 now identifies mixed-precision quantization and validation of expanded edits as future work. The current experiments also concern image-classification CNNs and vision transformers. Applying the approach to autoregressive decoders would require a suitable decoder search space and measurements of token-generation latency and KV-cache memory, including the effects of memory bandwidth. Section 6 identifies this extension without treating the vision results as evidence for decoder workloads.
Q3. What is the 500-evaluation wall time, and how much is spent on LLM calls?
We have added search-loop wall-time accounting in Appendix A.5. Across the 50 saved NAS-Bench-201 hardware-search runs, a complete 500-unique-evaluation ELAX run averaged 578.7 s, including 557.1 s in local LLM requests, or approximately 96.3% of the measured loop time. The matched random-ranking control averaged 22.4 s. For additional context, the saved qLogEHVI baseline runs averaged 28.6 s under the same evaluation budget. The present implementation therefore does not demonstrate a wall-time saving over qLogEHVI. The LLM request timing includes model execution and service and communication overhead; our logs do not isolate model computation. The matched 500-evaluation comparison remains useful for assessing the Pareto quality obtained from the same number of distinct architecture evaluations and the same benchmark records across methods. We now report that result separately from elapsed time, so the paper’s efficiency claim does not imply that the LLM controller makes the search faster. These timings cover the NAS-Bench-201 search loop, which queries stored accuracy and latency records. They do not include elastic-supernet training or deployment. The trained supernet can be reused across target profiles, but the present experiments do not measure the total cost of that reuse. We state this boundary in the limitations and identify full training-plus-search measurement and physical-device profiling as future work (section 6). Reducing controller overhead, for example through fewer or batched requests, is also a useful direction for future evaluation.