acceptodds
Under review as a conference paper at ICLR 2027

PROMA: Progressive Training Efficiency Optimization via Multi-Agent Planning

Abstract

Optimizing large-scale training efficiency remains heavily reliant on human expertise. Existing automated methods, whether relying on profiling, prior knowledge, or iterative search, fail to remove this burden. They still incur the full tuning cost at every new setting, whenever the model, cluster, or scale changes, making large-scale training optimization prohibitively expensive. We propose PROMA, a PRogressive training efficiency Optimization framework via Multi-Agent planning. PROMA begins with a small-scale pilot study, where a team of agents identifies the experience that remains valid across settings. In the target large-scale setting, PROMA guides an iterative search with this experience, refining proposals as training results accumulate. Two estimators filter out infeasible and clearly inferior proposals. Across different model architectures, training scales from 8 to 256 GPUs, and multiple hardware platforms, PROMA improves MFU by 25.9% over SOTA agentic search methods and 17.4% over expert-designed strategies, while reducing tuning GPU hours by 34.5% on average. The source code is available at https://anonymous.4open.science/r/PROMA.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM, Multi-Agent Systems, Distributed Training Optimization, Parallel Strategy Configuration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1gQ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PROMA searches for configurations of distributed training to achieve higher model FLOPs utilization (MFU). The paper first gives three properties that remain stable across cluster sizes: infeasible regions, high-priority search axes, and estimator bias. PROMA uses a small-scale pilot to record failures, estimator bias, and important configuration axes. At the target scale, agents propose configurations, and two analytical estimators, MemEst and MfuEst, drop candidates predicted to run out of memory or to have clearly worse MFU before a short training run. On dense and mixture-of-experts models from 6B to 236B and on 8 to 256 GPUs, PROMA achieves higher MFU than expert-tuned configurations and OptiCo. The authors also report lower estimator error than llm-analysis and Calculon, and ablate the pilot and both estimators.

Critical strengths

This paper studies a practical problem: finding a distributed training configuration with high MFU incurs high costs, and most of these costs repeat when the scale changes. It gives three properties that remain stable across cluster sizes: infeasible regions, high-priority search axes, and estimator bias. The evaluation includes dense and mixture-of-experts models ranging from 6B to 236B parameters, 8 to 256 GPUs, and three hardware platforms. This work describes MemEst and MfuEst in detail and compares them with llm-analysis and Calculon in terms of prediction error and out-of-memory threshold. The ablation study is relatively comprehensive. The search curves plot MFU under a fixed GPU-hour budget, instead of only showing the final value.

Critical weaknesses
  1. Insufficient clarity of the metric: Throughout this article, "MFU" is treated as a known metric from start to finish, with no definition provided and no citation of the PaLM paper that proposed it.

  2. The experiments are insufficient to support the conclusion: The article only uses OptiCo as the comparison baseline besides "expert". The original OptiCo paper's own conclusion is also not very strong: on more than 25 configurations, it only outperforms the expert average by 1.84% (0.08%–8.65%). Moreover, OptiCo's Limitations section states that experiments were only conducted on 8-GPU and 16-GPU setups. PROMA refers to it as "SOTA agentic search", which lacks sufficient supporting evidence. There are also three flaws in the comparison presented in Table 1: (a) The OptiCo figures marked with †\dagger are copied from the original paper and were not rerun on the same patched Megatron setup. However, Section 4.1 states that both methods were run within the same framework and on the same search space, which contradicts the table caption. (b) At large scales, OptiCo falls noticeably below the expert baseline: for DeepSeek-V2 236B with 128 GPUs, OptiCo achieves 10.1% versus the expert's 17.6% (-42.6%); for Llama3-70B with 128 GPUs, it reaches 40.7% versus 55.8% for the expert (-27.1%). Outperforming an already ineffective method at scales not validated in the original work does not prove that the search algorithm is superior. (c) The 25.9% figure in the abstract is the arithmetic mean of relative gains across all configurations, skewed by two outliers: OptiCo's drop to 10.1% on the 236B model, and the jump from 13.7% to 25.2% on Qwen3-6B. The absolute gap of 6.3 percentage points (shown in Sec. 4.2) is a more representative metric than the 25.9% value. Moreover, there are many methods that are introduced in the Related Work section but not included in the experiments.

  3. The introduction claims that three properties of the search space stay stable across cluster sizes: infeasible regions, the relative importance of search axes, and the sign of estimator bias. However, the paper does not report these measurements. I think the three properties of the search space staying stable across cluster sizes are very important to the conclusion of the article, as they lay the foundation of the proposed method for experience transfer, but the article does not provide enough evidence to support this claim.

  4. The code implementation is inconsistent with the paper: Section 3.2 describes the pilot as two LLM agents: "The planner agent (Aplan) identifies coverage gaps and schedules experiments, and the writer agent (Aw) turns raw measurements into transferable priors." The released code defaults to PILOT_STUDY_ACQUIRE=coverage, and run.py labels that path "coverage checklist (pure Python)" with "coverage: auto_write". The README states that "the ICLR main table uses the coverage-eval Et" and that this arm is "pure Python, no LLM." The LLM path (plan.md, write_e.md) exists only when PILOT_STUDY_ACQUIRE=pilot_study.

  5. The description of the method is not clear enough: The method has two stages, but Sections 3.2 and 3.3 describe only the first, the pilot study. The second stage, the target-scale search, is only briefly mentioned in Section 3.1.

Questions for authors
  1. Could you show some examples of the three properties of the search space that stay stable across cluster sizes? For example, at both the pilot scale and the target scale, the overlap of failure types, the ranking of axis importance, and whether estimator residuals keep the same sign?

  2. Which path on earth produced Table 1? If it was coverage, why does Section 3.2 present the planner and writer as LLM agents?

  3. Could the authors describe the second stage with the same level of detail?

1 comment

Thank you for these questions.

The revision adds the definition of MFU and cites the PaLM paper.

For reproducible agentic search, we compare with OptiCo, and every row also compares with Expert. Alpa, Galvatron, Calculon, and llm-analysis are cost models. They do not produce a training configuration, and they are not set up for this large-scale training. PROMPTS has no implementation that we can rerun. In its own paper, OptiCo is already above automatic parallelism.

We are sorry that ten OptiCo rows in the submitted Table 1 were taken from the OptiCo paper. We have remeasured these ten rows with the same API, the same software stack, and the same Megatron. On these rows, OptiCo is 2.5 pp above Expert on average, a relative gain of 6.6%.

These ten rows are the 8-GPU and 16-GPU settings. On the large models, OptiCo is still below Expert, and PROMA is above Expert. Table 1 reports both baselines. The model is larger and the GPU count is higher, so more parallel combinations are legal and the search space is larger. OptiCo searches from scratch at the target scale. Its limited number of trials cannot cover that space. At 8 and 16 GPUs the space is smaller, and OptiCo can score above Expert. PROMA uses the constraints recorded at small scale to narrow the search before the first large-scale run.

On all 17 rows, the average gain over OptiCo is 5.2 pp. The arithmetic mean of the relative gains is 22.5%, and the geometric mean is 19.8%.

The small-scale study records three kinds of results. The target scale uses each kind in its own way. The first kind is infeasible regions. We exclude them before a run starts. The second kind is search axes that affect MFU a lot. The target scale keeps trying along these axes. The third kind is estimator bias. We record only whether the estimate is high or low.

Software or operator incompatibilities go into the infeasible regions. On GPU-3, the small-scale study first found that one communication overlap switch cannot run, because Transformer Engine does not support it. The DS-16B and DS-236B searches at 128 GPUs each have 20 measured runs. Neither search tried this switch again. Other overlap switches that already ran at small scale stay on in these two searches. On GPU-1, the small-scale study first found that TP=2 is incompatible with the attention implementation. In the later 20 measured runs at 256 GPUs on GPU-1, TP=2 was never used.

The small-scale study compares how much each search axis affects MFU. Take activation recomputation as an example. At small scale, changing it changes MFU by more than changing tensor parallelism or pipeline parallelism. This comparison still holds at large scale. For Llama3-8B on GPU-2 at 16 GPUs, the average MFU of the three recomputation choices differs by up to about 9 pp. Changing tensor parallelism or pipeline parallelism changes it by less than 2 pp.

The small-scale study records whether the estimate is above or below the measurement. This bias comes from how the analytical model computes that layout. It does not come from the GPU count. When the layout stays the same and the GPU count grows, the direction of the bias stays the same. For memory, take Llama3-8B on GPU-2. Tensor parallelism is 1, pipeline parallelism is 2, and recomputation is off. At small scale, the estimate is about 7% above the measured peak. At 16 GPUs, the same layout has 4 successful runs. The estimate is still above the measurement, by about 11% on average. For MFU, take Qwen3 on GPU-2. Tensor parallelism is 1, pipeline parallelism is 1, recomputation is selective, and expert parallelism is 8. At small scale, the estimate is about 8.7%, and the measurement is about 17% to 18%. The estimate is below the measurement. At 16 GPUs, the same layout has 6 successful runs. The estimate averages about 16%, and the measurement averages about 20%. The estimate is still lower.

The coverage path is the setting used in the ablation. It is not the main table. The main table uses PILOT_STUDY_ACQUIRE=pilot_study, which calls the planner and the writer. The released README did not separate these two settings clearly. That is corrected now.

The revision adds Section 3.3 to the paper. It describes the second stage at the same level of detail as the pilot study.

Reviewer_HdMbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PROMA, a multi-agent framework for optimizing large-scale distributed training configurations. The authors introduce a progressive optimization strategy that extracts transferable knowledge from small-scale pilot experiments to guide configuration search at the target scale. PROMA employs multiple agents to plan experiments, generate configurations, and refine search strategies based on training feedback. Two estimators, MemEst and MfuEst, are developed to predict memory usage and training efficiency, filtering out infeasible or unpromising configurations. Experiments across different model architectures, GPU scales, and hardware platforms show that PROMA achieves higher MFU than existing agent-based methods and expert-designed configurations while reducing tuning costs.

Critical strengths
  1. The paper addresses an important practical problem in large-scale distributed training: finding efficient configurations without spending excessive GPU resources on tuning. The idea of transferring knowledge from small-scale pilot experiments to guide large-scale configuration search is interesting and helps reduce repeated exploration across different training settings.

  2. The experimental evaluation is extensive, covering different model architectures, GPU scales, and hardware platforms. PROMA achieves consistent improvements in MFU over the baselines, and the ablation studies provide useful evidence for the effectiveness of the pilot study and the two proposed estimators.

Critical weaknesses
  1. The pilot study still relies on prior expert knowledge to identify which properties should be tested, as acknowledged in Section 5. It is unclear how well the framework would work when such knowledge is limited, especially for new model architectures or hardware platforms.

  2. PROMA uses hard constraints extracted from the pilot study to reduce the search space. However, if some of these constraints are incorrect, potentially good configurations may be excluded before the search even starts. The paper does not discuss how PROMA handles this issue.

Questions for authors
  1. How much does PROMA depend on expert knowledge in the pilot study? If less expert knowledge is provided, will it affect the final MFU?

  2. In Section 3.2, PROMA uses hard constraints to reduce the search space. What if some of these constraints are wrong? Is it possible that some better configurations are excluded?

  3. In Table 2, there is a large accuracy gap between the structural and fully calibrated estimators. How much calibration data is needed to get this accuracy? What happens if less calibration data is used?

1 comment

Thank you for these questions.

About the expert note in the pilot study. The small scale and the target scale use the same expert note. The note gives the parameter description and which parts tend to become bottlenecks. Dense and MoE models, every scale, and all three GPUs use this one note. The small-scale study checks this note.

About hard constraints. Before a run starts, we remove only software or operators that the small-scale study has already shown cannot run. On GPU-3, the small-scale study first found that one communication overlap switch cannot run because Transformer Engine does not support it. The DS-16B and DS-236B searches at 128 GPUs each have 20 measured runs. Neither search tried this switch again. On GPU-1, the small-scale study first found that TP=2 is incompatible with the attention implementation. The 20 measured runs at 256 GPUs never used TP=2. A removal stays on the software stack where it was observed. It is not applied to another GPU or another software stack. A discontinuity is not a removal. A small-scale out of memory result does not become this kind of removal.

A small-scale out of memory result does not become this kind of removal. Neither does whichever recomputation choice was better at small scale. Llama3-70B on GPU-3 ran out of memory on 8 nodes with recomputation off. We did not remove recomputation off. The best configuration at 128 GPUs is recomputation off, and its MFU is 56.1%. If that case had been written as a removal, this configuration would not have been run.

In Table 2, the gap between the structural row and the Ours row comes from the runtime constants. We determined these constants from 93 configurations with measured MFU and 62 configurations with measured peak memory. The measured set has 601 MFU configurations and 421 memory configurations. So this group is about 15% of the measured set. After we remove these 93 MFU configurations and 62 memory configurations, 508 MFU configurations and 359 memory configurations remain. The MFU error is 1.85 pp for dense models (1.8 in Table 2) and 2.14 pp for MoE models (2.1 in Table 2). The memory error is 2.53% for dense models (2.5 in Table 2) and 2.98% for MoE models (2.9 in Table 2). The error with all of these runtime constants removed is the Ours (structural) row in Table 2. For both dense and MoE models, this row is lower than llm-analysis and Calculon.

Reviewer_nosoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents PROMA, a system of LLM agents that searches for good settings for large-scale model training, such as how to split the model across GPUs, batch sizes, and ways to save memory. The aim is to get more useful work out of the hardware while spending fewer GPU-hours on the search itself. PROMA first runs short trials on small clusters to learn things that should still hold at full scale, for example, which settings fail and which ones are most relevant. It then uses this knowledge to guide the search on the target cluster. Two estimators, one for memory use and one for training speed, filter out settings that would crash or clearly perform badly before any real run is started. The authors then test PROMA on dense and MoE models from 6B to 236B parameters, on 8 to 256 GPUs and three hardware types. They compare it with expert-tuned settings and with OptiCo, an earlier agent-based method. They report that their method substantially improves over the baselines with far fewer GPU-hours spent on tuning. They also show that their estimators are much more accurate than existing tools, and an ablation suggests that each of the three main components contributes to the result.

Critical strengths

S1. The two estimators (MemEst and MfuEst) are much more accurate than existing tools. They predict memory use and training speed far better than llm-analysis and Calculon, on both dense and MoE models. Table 3 shows this most clearly. When MemEst says a configuration will run out of memory, it almost always does, while the other tools wrongly reject most configurations that would have run fine. Even the simpler version of the estimators, without any fitting, is better than either of those tools. I would argue that this part of the paper is the strongest in terms of being backed up by experiments. The ablations also show that it accounts for most of the gain. Even without the agent framework, these estimators could be useful.

S2. I think that the problem the authors tackle is an important one, and the experiments they run use realistic scales. Finding good training settings is expensive, and even if it is a small speed-up, it can save a lot of compute on large runs. The paper tests on up to 256 GPUs, MoE models with up to 236B parameters, and three types of hardware. Section 4.3 also has a useful negative finding. Reusing the best setting from a small run, or the full log of small runs, works worse than starting from nothing.

Critical weaknesses

W1. The baseline comparison is flawed. Table 1 sets out to display the superiority of PROMA compared to OptiCo and expert tuning. Section 4.1 says that PROMA and OptiCo were run on the same software and search space. This seems, however, not to be the case for 10/17 rows in Table 1. On those rows, both the OptiCo and the Expert numbers are copied from the OptiCo paper directly, which used a different LLM and possibly a different Megatron version and training setup. So those numbers are not comparable. On the remaining seven rows where the authors ran OptiCo themselves, it performs worse than the expert tuning baseline 5/7 times, in three of them by more than 25%. I would argue this is a poor reimplementation, and there aren't many details in the paper about this either. This performance comparison in Table 1 is the core of what should show the utility of PROMA, but it is far from doing that job.

W2. The main idea of the paper is that properties of the search space (e.g., failing configurations, estimator bias) that are found in small runs would still be valid at even a large scale, but this is only indirectly shown. The authors never check how many of the rules learnt in the pilot study actually held at the target scale, or how many were wrong. The only evidence we see is the final MFU. The ablation also suggests that the estimators (MemEst, MfuEst) matter at least as much as the pilot study. Removing the MfuEst reduces the gain over Expert from 17.4% to 3.3%, while removing the pilot study reduces it to 8.6%. The accuracy of these estimators primarily comes from coefficients that are "fitted" per platform, which bring the MFU error down by 4x. It was not clear to me what data they are "fitted" on and whether it overlaps with the test configurations. Each result also seems to come from a single run of an LLM-driven search whose outcome varies between runs. The small performance difference in Section 4.3 between PROMA and the empty-history search may just as well be noise. The gains seem to mainly come from the estimators.

Questions for authors

Q1. How many times was each search run?
Q2. What data are the platform-level estimator coefficients fitted on?

1 comment

Thank you for these questions.

We are sorry that ten OptiCo rows in the submitted Table 1 were taken from the OptiCo paper. We have remeasured these ten rows with the same API, the same software stack, and the same Megatron. On these rows, OptiCo is 2.5 pp above Expert on average, a relative gain of 6.6%.

These ten rows are the 8-GPU and 16-GPU settings. On the large models, OptiCo is still below Expert. The model is larger and the GPU count is higher, so more parallel combinations are legal and the search space is larger. OptiCo searches from scratch at the target scale. Its limited number of trials cannot cover that space. At 8 and 16 GPUs the space is smaller, and OptiCo can score above Expert. PROMA uses the constraints recorded at small scale to narrow the search before the first large-scale run.

Experience found at small scale agrees with experience at large scale. Take activation recomputation as an example. At small scale, changing it changes MFU by more than changing tensor parallelism or pipeline parallelism. For Llama3-8B on GPU-2 at 16 GPUs, the average MFU of the three recomputation choices differs by up to about 9 pp. Changing tensor parallelism or pipeline parallelism changes it by less than 2 pp.

The estimates show the same agreement. For Llama3-8B on GPU-2, tensor parallelism is 1, pipeline parallelism is 2, and recomputation is off. At small scale, the memory estimate is about 7% high. At 16 GPUs, the same layout has 4 successful runs, and the estimate is still about 11% high. For Qwen3 on GPU-2, tensor parallelism is 1, pipeline parallelism is 1, recomputation is selective, and expert parallelism is 8. At small scale, the MFU estimate is about 8.7%, and the measurement is about 17% to 18%. At 16 GPUs, the same layout has 6 successful runs. The estimate averages about 16%, and the measurement averages about 20%. The estimate is still lower.

Configurations that cannot run follow the same pattern. The small-scale study finds that they cannot run, and they do not appear again in the large-scale search. On GPU-3, one communication overlap switch cannot run because Transformer Engine does not support it. The DS-16B and DS-236B searches at 128 GPUs each have 20 measured runs. The use count is 0. On GPU-1, TP=2 is incompatible with the attention implementation. In the 20 measured runs at 256 GPUs, the use count is 0.

The two estimators and the pilot study do different jobs. MfuEst chooses the direction toward high MFU. MemEst removes configurations that do not fit in memory. The pilot study brings the small-scale experience to the target scale and narrows the search before the first run. In Table 4, removing MfuEst drops peak MFU the most. When the pilot study is removed, both estimators remain, but the search returns to the full space. Convergence is slower, and the gain over Expert is also smaller.

The platform coefficients were not fit on the full measured set. They include hardware quantities and runtime constants. The hardware quantities are communication bandwidth, startup latency, HBM, and GEMM throughput. We measured them before tuning started, on 16 GPUs across two machines, with 8 GPUs on each machine. Bandwidth was measured with 1GB messages for allreduce, point to point, and alltoall. Each one has its own startup latency, and the latency does not depend on message size. These hardware quantities were not fit to training MFU.

The runtime constants were fit on 93 configurations with measured MFU and 62 configurations with measured peak memory. The models are Llama3-8B, Llama2-7B, Qwen2.5-7B, GPT3-7B, DS-16B, DS-236B, and Llama3-70B. Table 2 uses 601 configurations with measured MFU and 421 configurations with measured peak memory, on GPUs with 80GB and 96GB of memory. The fit uses about 15% of that set. After we remove these 93 MFU configurations and 62 memory configurations, 508 MFU configurations and 359 memory configurations remain. The MFU error is 1.85 pp for dense models (1.8 in Table 2) and 2.14 pp for MoE models (2.1 in Table 2). The memory error is 2.53% for dense models (2.5 in Table 2) and 2.98% for MoE models (2.9 in Table 2).

In Table 1, PROMA is above Expert for every model, every GPU count, and all three GPUs. Against OptiCo, Llama2-7B at 8 GPUs is 68.4% for both, and PROMA is higher on the other rows. The average gain over OptiCo is 5.2 pp. This direction is not one lucky setting. Each full search was run once. An LLM search can vary, but that variation does not explain the same direction across the whole table.

Comments on the paper

2 comments

For all reviewers: as the discussion period is approaching its end, please join the discussion and put acknowledge on authors' rebuttal, thank you!

AuthorsOpenReview ↗

General response

Thank you for the reviews. Every change in the revised PDF is marked in yellow.

We are sorry that ten OptiCo rows in the submitted Table 1 were taken from the OptiCo paper. We have remeasured these ten rows with the same API, the same software stack, and the same Megatron. The tables and figures now use these measurements. On these rows, OptiCo is 2.5 pp above Expert on average, a relative gain of 6.6%. These rows are the 8-GPU and 16-GPU settings. On the large models, OptiCo is still below Expert, because the search space is larger and OptiCo searches it from scratch. PROMA uses the constraints recorded at small scale to narrow the search before the first large-scale run, and PROMA is above Expert on every row. On all 17 rows, the average gain over OptiCo is 5.2 pp. The arithmetic mean of the relative gains is 22.5%, and the geometric mean is 19.8%. The gain over Expert is unchanged, at 17.4% (4.7 pp).

The platform coefficients were not fit on the full measured set. Hardware quantities were measured on each GPU before tuning started. The runtime constants were fit on 93 configurations with measured MFU and 62 configurations with measured peak memory, about 15% of the measured set. The measured set has 601 MFU configurations and 421 memory configurations. After these 93 MFU configurations and 62 memory configurations are removed, 508 MFU configurations and 359 memory configurations remain. The MFU error is 1.85 pp for dense models and 2.14 pp for MoE models, compared with 1.8 pp and 2.1 pp in Table 2. The memory error is 2.53% for dense models and 2.98% for MoE models, compared with 2.5% and 2.9% in Table 2. The difference is in the second decimal place. Section 4.5 states where the coefficients come from.

The revision adds two sections.

Section 3.3 describes the target-scale search at the same level of detail as the pilot study. In each round, the proposal agent (Ap) proposes one configuration, and a short training job measures its MFU or its failure. The proposal uses the model, the device, the pilot records, and the results already measured in this search. The first round has no measurement, so it uses the pilot records alone. The records are used in three ways. The validator agent (Av) excludes infeasible regions before a run. An axis that changes MFU more than the others is still varied at the target scale. MemEst and MfuEst make a new estimate, and Ap reads it with the high or low direction recorded by the pilot.

The case study gives the three properties that stay stable across cluster sizes. They are infeasible regions, the relative importance of search axes, and the sign of estimator bias. A configuration is removed before a run only when the small-scale study has shown that a software stack or an operator cannot run, and that removal stays on the software stack where it was observed. An out-of-memory result is not a removal, and neither is the recomputation choice that was better at small scale. The pilot uses one expert note at every scale. The note gives the parameter description and which parts tend to become bottlenecks. The note is in the released code.

The introduction adds the definition of MFU and cites the PaLM paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.