- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to accumulate and reuse deployment experience to improve the efficiency of searching for LLM parallelization configurations when real training measurements are scarce. The authors propose Est2Exp, which combines a performance estimator with an experience bank. It first calibrates a mixed-granularity DAG estimator using limited real profiling data, then distills estimated results into experience entries with applicability conditions and supporting evidence through controlled single-factor comparisons and FunSearch-based interpretable policy search. The resulting experience constrains the search space and initializes subsequent searches, while additional measurements resolve conflicts between the estimator and stored experience.
The main contribution is a bidirectional feedback framework between the estimator and the experience bank, together with mechanisms for latency calibration, experience extraction, and refinement. The evaluation uses real training traces from Ascend clusters to validate estimation accuracy and a heterogeneous simulation environment to construct the experience bank and evaluate search transfer from 128 to 256 devices. The authors report DAG latency prediction accuracy of 99.3%–99.9%. Experience-based warm starts increase search success from 73.3% to 98.3% and reduce candidate evaluations from 12 to 4 among successful runs.
Critical strengths
-
Combining low-cost evaluation with experience reuse across tasks has practical value. Experience entries with applicability conditions and supporting evidence provide a more structured way to organize deployment knowledge than storing only a best configuration. Measurement-driven conflict resolution also accounts for the possibility that experience may become invalid.
-
The estimator is validated using real large-scale training traces. The evaluation covers models with 30B–505B parameters and clusters of 128–16K NPUs, including extrapolation to large-scale settings not used for calibration. This provides valuable empirical support for the estimator.
Critical weaknesses
-
The added value of experience extraction over ordinary warm starts has not been established. Table 4 compares only cold starts with policy warm starts, without a baseline that directly reuses previously identified high-performing configurations. It therefore does not distinguish the benefit of initialization with historical information from that of the proposed experience extraction and policy representation.
-
The quality of the selected configurations has not been validated on real deployments. The latency extrapolation in Section 6.2 and conflict resolution in Section 6.4 have measurement support, but the ranking of multiple candidate configurations within the same new scenario remains unverified. Experience generation and search evaluation depend on the same estimator, so shared biases may prevent simulation-based search success from translating into real deployment gains.
Questions for authors
-
Regarding Weakness 1: Under the same search budget, how much additional benefit does policy-based experience provide over warm-starting directly from previously identified high-performing configurations?
-
Regarding Weakness 2: Could you measure multiple candidate configurations in the same new deployment scenario to validate the estimator's ranking and the quality of the configuration selected by experience-guided search?
6 comments
Response to Reviewer nDJN (Part 2/2)
W2 / Q2. Hardware validation.
Thank you for raising this point. We agree that measuring multiple configurations under identical conditions in a new scenario would provide an independent check of both the estimator’s ranking and the quality of the configuration selected by experience-guided search.
Our estimator-based evaluation covers 100 heterogeneous 256-GPU scenarios. Even a focused follow-up on 256 NPUs would require coordinating cluster access with our industry partner, reproducing and verifying the device slowdowns and network bandwidth limits, and testing several candidate configurations under identical conditions in each scenario. This setup and measurement effort requires more time than the rebuttal window allows.
For follow-up validation, we plan to test the configuration recommended using transferred experience alongside several strong alternatives in a few representative new scenarios. We will compare the hardware results for memory feasibility, training-step latency, and configuration ranking with the deployment findings distilled from the estimator’s outputs. If discrepancies arise, we can use these measurements in the existing Loop 3 (Figure 1) to determine whether to recalibrate the estimator or refine the stored guidance.
These experiments would extend the hardware validation of the existing Est2Exp architecture. The core contribution is the automated workflow already implemented in Est2Exp. It distills deployment findings from estimator outputs, and reuses them to guide configuration search in new scenarios. This gives engineers a practical way to reuse deployment knowledge across settings, start new configuration searches from prior findings, and reduce the need for costly hardware trials.
Thank you for the additional comparisons and clarification. The results provide further evidence for the benefits of warm starts, but the added value of policy extraction over direct reuse of strong historical configurations under a matched budget remains unclear.
Response to Reviewer nDJN (Part 1/2)
W1 & Q1: Added Value of Policy-Based Experience Extraction and Reuse
Thank you for this insightful suggestion. We agree that reusing a previously identified high-performing configuration is an important baseline for assessing the value of extracting reusable policies. In the revised manuscript, we compare these two forms of reuse by examining the quality of their initial configurations and how quickly each search improves them.
(1) Evaluation metrics. We use two metrics, defined in Section 6.1 (Metrics, page 8) of the revised manuscript.
Optimality Gap (OG) measures the relative latency difference between the best parallelism configuration identified through experience-guided search and the optimal configuration. Lower OG indicates better configuration quality.
Search Iterations measures search efficiency by recording the number of iterations required to first identify the reference-optimal configuration.
(2) Experimental setup. We evaluate 100 heterogeneous 256-GPU deployment scenarios using three random seeds per scenario. The new Figure 3 (Section 6.3, page 9) compares the following four settings .
(3) Results and implications. Figure 3 tracks the optimality gap as the iteration budget increases, with each curve reaching 0% once all 300 runs for that setting have found the reference optimum (100 scenarios with three random seeds each). Table 5 (page 10) reports the maximum and average numbers of iterations that runs needed to first reach a 0% gap. Since an iteration involves different operations in the two search methods, we compare each experience-guided variant with its own cold-start baseline.
Policy-based search. In Figure 3, the experience-guided variant (orange) starts with a smaller optimality gap and reaches 0% sooner than the cold-start variant (blue). As reported in Table 5, experience reuse reduces the maximum iteration count from 19 to 10 (about 47% fewer) and the average from 7.0 to 1.7 (about 76% fewer).
Simulated annealing. The experience-guided variant (green) reaches small optimality gaps sooner than the cold-start variant (purple). A few runs take longer to reach 0%, extending the tail of the green curve. The average iteration count drops from 14.3 to 2.9, a reduction of 11.4 iterations (about 80%).
Both methods reach the reference optimum in fewer iterations on average when initialized from prior experience. Policy-based search starts from a previously learned scoring policy, and simulated annealing starts from a configuration derived from the experience bank.
Compared with simulated annealing, which returns the best configuration it finds for the target scenario, Est2Exp’s policy-based search yields both a configuration and a reusable scoring rule. Est2Exp first distills deployment experience from the scoring rule, then stores the resulting findings together with their applicability conditions in the experience bank. The stored experience can guide candidate ranking in similar scenarios and warm-start the search for new policies. The advantage of Est2Exp lies in automatically turning the results of individual searches into reusable deployment knowledge, reducing the need to rediscover effective strategies across scenarios.