- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PROMA searches for configurations of distributed training to achieve higher model FLOPs utilization (MFU). The paper first gives three properties that remain stable across cluster sizes: infeasible regions, high-priority search axes, and estimator bias. PROMA uses a small-scale pilot to record failures, estimator bias, and important configuration axes. At the target scale, agents propose configurations, and two analytical estimators, MemEst and MfuEst, drop candidates predicted to run out of memory or to have clearly worse MFU before a short training run. On dense and mixture-of-experts models from 6B to 236B and on 8 to 256 GPUs, PROMA achieves higher MFU than expert-tuned configurations and OptiCo. The authors also report lower estimator error than llm-analysis and Calculon, and ablate the pilot and both estimators.
Critical strengths
This paper studies a practical problem: finding a distributed training configuration with high MFU incurs high costs, and most of these costs repeat when the scale changes. It gives three properties that remain stable across cluster sizes: infeasible regions, high-priority search axes, and estimator bias. The evaluation includes dense and mixture-of-experts models ranging from 6B to 236B parameters, 8 to 256 GPUs, and three hardware platforms. This work describes MemEst and MfuEst in detail and compares them with llm-analysis and Calculon in terms of prediction error and out-of-memory threshold. The ablation study is relatively comprehensive. The search curves plot MFU under a fixed GPU-hour budget, instead of only showing the final value.
Critical weaknesses
-
Insufficient clarity of the metric: Throughout this article, "MFU" is treated as a known metric from start to finish, with no definition provided and no citation of the PaLM paper that proposed it.
-
The experiments are insufficient to support the conclusion: The article only uses OptiCo as the comparison baseline besides "expert". The original OptiCo paper's own conclusion is also not very strong: on more than 25 configurations, it only outperforms the expert average by 1.84% (0.08%–8.65%). Moreover, OptiCo's Limitations section states that experiments were only conducted on 8-GPU and 16-GPU setups. PROMA refers to it as "SOTA agentic search", which lacks sufficient supporting evidence. There are also three flaws in the comparison presented in Table 1: (a) The OptiCo figures marked with are copied from the original paper and were not rerun on the same patched Megatron setup. However, Section 4.1 states that both methods were run within the same framework and on the same search space, which contradicts the table caption. (b) At large scales, OptiCo falls noticeably below the expert baseline: for DeepSeek-V2 236B with 128 GPUs, OptiCo achieves 10.1% versus the expert's 17.6% (-42.6%); for Llama3-70B with 128 GPUs, it reaches 40.7% versus 55.8% for the expert (-27.1%). Outperforming an already ineffective method at scales not validated in the original work does not prove that the search algorithm is superior. (c) The 25.9% figure in the abstract is the arithmetic mean of relative gains across all configurations, skewed by two outliers: OptiCo's drop to 10.1% on the 236B model, and the jump from 13.7% to 25.2% on Qwen3-6B. The absolute gap of 6.3 percentage points (shown in Sec. 4.2) is a more representative metric than the 25.9% value. Moreover, there are many methods that are introduced in the Related Work section but not included in the experiments.
-
The introduction claims that three properties of the search space stay stable across cluster sizes: infeasible regions, the relative importance of search axes, and the sign of estimator bias. However, the paper does not report these measurements. I think the three properties of the search space staying stable across cluster sizes are very important to the conclusion of the article, as they lay the foundation of the proposed method for experience transfer, but the article does not provide enough evidence to support this claim.
-
The code implementation is inconsistent with the paper: Section 3.2 describes the pilot as two LLM agents: "The planner agent (Aplan) identifies coverage gaps and schedules experiments, and the writer agent (Aw) turns raw measurements into transferable priors." The released code defaults to PILOT_STUDY_ACQUIRE=coverage, and run.py labels that path "coverage checklist (pure Python)" with "coverage: auto_write". The README states that "the ICLR main table uses the coverage-eval Et" and that this arm is "pure Python, no LLM." The LLM path (plan.md, write_e.md) exists only when PILOT_STUDY_ACQUIRE=pilot_study.
-
The description of the method is not clear enough: The method has two stages, but Sections 3.2 and 3.3 describe only the first, the pilot study. The second stage, the target-scale search, is only briefly mentioned in Section 3.1.
Questions for authors
-
Could you show some examples of the three properties of the search space that stay stable across cluster sizes? For example, at both the pilot scale and the target scale, the overlap of failure types, the ranking of axis importance, and whether estimator residuals keep the same sign?
-
Which path on earth produced Table 1? If it was coverage, why does Section 3.2 present the planner and writer as LLM agents?
-
Could the authors describe the second stage with the same level of detail?
Thank you for these questions.
The revision adds the definition of MFU and cites the PaLM paper.
For reproducible agentic search, we compare with OptiCo, and every row also compares with Expert. Alpa, Galvatron, Calculon, and llm-analysis are cost models. They do not produce a training configuration, and they are not set up for this large-scale training. PROMPTS has no implementation that we can rerun. In its own paper, OptiCo is already above automatic parallelism.
We are sorry that ten OptiCo rows in the submitted Table 1 were taken from the OptiCo paper. We have remeasured these ten rows with the same API, the same software stack, and the same Megatron. On these rows, OptiCo is 2.5 pp above Expert on average, a relative gain of 6.6%.
These ten rows are the 8-GPU and 16-GPU settings. On the large models, OptiCo is still below Expert, and PROMA is above Expert. Table 1 reports both baselines. The model is larger and the GPU count is higher, so more parallel combinations are legal and the search space is larger. OptiCo searches from scratch at the target scale. Its limited number of trials cannot cover that space. At 8 and 16 GPUs the space is smaller, and OptiCo can score above Expert. PROMA uses the constraints recorded at small scale to narrow the search before the first large-scale run.
On all 17 rows, the average gain over OptiCo is 5.2 pp. The arithmetic mean of the relative gains is 22.5%, and the geometric mean is 19.8%.
The small-scale study records three kinds of results. The target scale uses each kind in its own way. The first kind is infeasible regions. We exclude them before a run starts. The second kind is search axes that affect MFU a lot. The target scale keeps trying along these axes. The third kind is estimator bias. We record only whether the estimate is high or low.
Software or operator incompatibilities go into the infeasible regions. On GPU-3, the small-scale study first found that one communication overlap switch cannot run, because Transformer Engine does not support it. The DS-16B and DS-236B searches at 128 GPUs each have 20 measured runs. Neither search tried this switch again. Other overlap switches that already ran at small scale stay on in these two searches. On GPU-1, the small-scale study first found that TP=2 is incompatible with the attention implementation. In the later 20 measured runs at 256 GPUs on GPU-1, TP=2 was never used.
The small-scale study compares how much each search axis affects MFU. Take activation recomputation as an example. At small scale, changing it changes MFU by more than changing tensor parallelism or pipeline parallelism. This comparison still holds at large scale. For Llama3-8B on GPU-2 at 16 GPUs, the average MFU of the three recomputation choices differs by up to about 9 pp. Changing tensor parallelism or pipeline parallelism changes it by less than 2 pp.
The small-scale study records whether the estimate is above or below the measurement. This bias comes from how the analytical model computes that layout. It does not come from the GPU count. When the layout stays the same and the GPU count grows, the direction of the bias stays the same. For memory, take Llama3-8B on GPU-2. Tensor parallelism is 1, pipeline parallelism is 2, and recomputation is off. At small scale, the estimate is about 7% above the measured peak. At 16 GPUs, the same layout has 4 successful runs. The estimate is still above the measurement, by about 11% on average. For MFU, take Qwen3 on GPU-2. Tensor parallelism is 1, pipeline parallelism is 1, recomputation is selective, and expert parallelism is 8. At small scale, the estimate is about 8.7%, and the measurement is about 17% to 18%. The estimate is below the measurement. At 16 GPUs, the same layout has 6 successful runs. The estimate averages about 16%, and the measurement averages about 20%. The estimate is still lower.
The coverage path is the setting used in the ablation. It is not the main table. The main table uses PILOT_STUDY_ACQUIRE=pilot_study, which calls the planner and the writer. The released README did not separate these two settings clearly. That is corrected now.
The revision adds Section 3.3 to the paper. It describes the second stage at the same level of detail as the pilot study.