acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training

Abstract

Training multimodal large language models (MLLMs) is challenged by both model and data heterogeneity. Existing systems redesign the training pipeline to address these challenges, but remain bound by a Pareto frontier between compute and memory efficiency, improving one only at the expense of the other. We present BigMac, a new training pipeline for multimodal LLMs. The core idea of BigMac is to elegantly nest the encoder and generator computation into the original LLM pipeline, forming a dependency-safe nested pipeline structure. With this design, BigMac reduces the activation memory complexity of the encoder and generator to while keeping the activation memory complexity of the LLM unchanged. At the same time, it achieves the same computational efficiency as the idealized setting with unlimited memory. As a result, BigMac breaks the Pareto frontier between computational efficiency and memory usage, enabling simultaneous optimization of both computation and memory in MLLM training. We evaluate BigMac on multiple MLLMs and training workloads. Experimental results show that BigMac achieves a 1.08-1.9 training speedup over baseline systems while maintaining stable memory usage as batch size increases.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
multimodal large language models, pipeline parallelism, distributed training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aYwpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training system for multimodal LLMs that aims to improve computational efficiency while limiting modality activation memory. It preserves the LLM's internal operator order and inserts encoder and generator computations at dependency-safe points. A bounded encoder lookahead window and early backward execution shorten activation lifetimes. The system also implements the cross-module communication needed to execute the schedule and supports context parallelism and FSDP. The claimed constant-memory bound concerns modality activations as the number of microbatches increases at a fixed parallel configuration, rather than total training memory.

Critical strengths
  1. The scheduling approach addresses a useful practical problem. BigMac combines limited encoder lookahead with early backward execution to shorten activation lifetimes while preserving the LLM's internal operator order.
  2. The design extends beyond encoder-only scheduling. The paper handles both encoders and generators and describes the cross-module communication, context-parallel conversion, and FSDP execution needed to support the schedule.
Critical weaknesses
  1. The incremental contribution over prior scheduling systems is not sufficiently clear. MegaScale-Omni already describes on-demand encoder insertion and early encoder backward execution to reduce activation memory. The paper does not adequately explain how BigMac differs from this design on understanding-only workloads, and no direct comparison is provided. Its characterization of the compute–memory trade-off in existing systems also needs more qualification: Optimus includes fine-grained scheduling of encoder operations into LLM bubbles, while LongCat-Flash-Omni explicitly uses encoder activation recomputation. Neither observation establishes that these systems achieve BigMac's claimed trade-off, but both are relevant to the motivation and comparison. Generator support and the broader scheduling framework may be meaningful extensions; the current presentation does not clearly separate those contributions from the encoder-side ideas already available.

  2. The analysis and evaluation do not yet establish the full claimed compute–memory advantage. Section 4.1 describes a sequential Optimus execution, leaving unclear whether the original fine-grained bubble scheduler and parallel-plan search are retained, and how the baseline is extended to generation. A matched comparison with encoder activation recomputation is also missing. Since Optimus runs out of memory at every tested generation batch size, those results demonstrate a memory advantage over the tested configuration but leave the throughput comparison against a compute-efficient alternative unresolved. On the analytical side, Section 3.1 provides a brief argument for the three-unit lookahead under interleaved 1F1B with at least two virtual chunks, but does not sufficiently derive the ordering property or explain its boundary conditions. Preserving operator order alone does not establish unchanged efficiency once execution-time variation and communication are included. The CP and FSDP experiments are useful, but do not isolate these central scheduling claims. Overall, the evidence is more convincing about the schedule's practical usefulness than about matching the idealized compute-efficient execution.

Questions for authors
  1. Does the Optimus baseline retain the original fine-grained encoder-in-bubble scheduler and parallel-plan search? If the tested schedule is sequential, was it selected by the optimization procedure or imposed by the implementation? How was Optimus extended to handle the generator? Please provide the per-module parallel configurations and recomputation settings used for the main comparisons, along with the code version or implementation details identifying the Megatron-DistTrain baseline.

  2. With encoder activation recomputation enabled under otherwise matched settings, how do Optimus or a LongCat-style baseline compare with BigMac in iteration time and peak memory? This comparison would help establish how much of the advantage comes from the proposed schedule rather than the absence of recomputation in the tested baseline.

Reviewer_V8iiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new training framework for multimodal large language models, called BigMAC. In particular, the proposed method introduces a nested pipeline scheduling strategy to improve both computational and memory efficiency during MLLM training. By integrating the modality encoder and generator into the original LLM pipeline while preserving their dependencies, BigMAC aims to reduce activation memory without introducing additional pipeline bubbles. Besides, this work designs a pipeline executor to support different parallelism strategies, including context parallelism and FSDP. Finally, experiments on MLLM understanding and generation workloads are conducted to demonstrate the effectiveness of the proposed framework, achieving up to 1.9× training speedup. In general, I believe this work is interesting and addresses an important problem in efficient multimodal training. But I still have some questions for the authors for clarification.

Critical strengths
  1. The motivation is intuitive and interesting. This work considers both computational and memory efficiency, rather than optimizing only one of them. I believe this is a meaningful direction for large-scale multimodal model training.

  2. The proposed nested pipeline design is reasonable. By considering the dependencies among the encoder, LLM, and generator, this work provides a systematic solution to reduce memory consumption while maintaining computational efficiency.

  3. The experimental results seem promising. The evaluation on 128 GPUs and the production deployment on 1,536 GPUs help demonstrate the scalability and practical value of the proposed framework. Besides, the structure and presentation are relatively clear.

Critical weaknesses
  1. This paper claims to break the Pareto frontier between computational and memory efficiency. However, I wonder whether the current experiments fully support this conclusion. Based on Figures 9 and 10, BigMAC achieves better efficiency than the selected baselines, but does this necessarily mean that the Pareto frontier is broken? Could the authors provide a more comprehensive comparison or analysis to support this important claim?

  2. Based on Section 3.1, BigMAC reduces encoder activation memory to O(1) by maintaining a constant number of encoder units. However, each unit contains P microbatches, where P is the pipeline parallelism degree. Does this mean that the memory still depends on P? Besides, the analysis mainly considers interleaved 1F1B with vpp size ≥ 2. Can this conclusion generalize to other pipeline schedules? I think clarifying the conditions of the theoretical analysis is important.

  3. The experimental evaluation could be more comprehensive. The authors mainly use one LLM backbone with two training workloads. I wonder whether the proposed method can achieve similar improvements on different model architectures and pipeline configurations. Besides, Figure 11 shows that the improvement from the one-sided pull mechanism is only up to 1.03×. Is this improvement significant enough to justify the additional implementation complexity? A more detailed analysis would be helpful.

  4. There are some additional questions about the experimental settings. First, the comparison mainly focuses on Optimus and Megatron-DistTrain. Could the authors include more recent or optimized pipeline scheduling baselines? Second, the production results in Figure 12 mainly show training loss and gradient norm, but do not directly demonstrate computational or memory efficiency at 1,536 GPUs. Could the authors provide throughput, GPU utilization, or memory measurements for this large-scale deployment? I think these results would further strengthen the practical contribution of this work.

Questions for authors
  1. What is the main evidence that BigMAC breaks the compute-memory Pareto frontier, rather than simply outperforming the selected baselines?

  2. The O(1) activation memory analysis assumes a constant number of encoder units. Does this conclusion still hold when the pipeline parallelism degree or virtual pipeline size increases?

  3. Can BigMAC achieve similar improvements under other pipeline schedules, such as standard 1F1B or zero-bubble scheduling?

  4. Based on Figure 11(b), the one-sided pull mechanism only provides up to 1.03× speedup. Is this improvement sufficient to justify its additional implementation complexity?

  5. Could the authors provide throughput and peak GPU memory measurements for the 1,536-GPU production deployment, rather than only training loss and gradient norm?

Reviewer_mjg1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents BigMac, a new training framework that improves both computational and activation memory efficiency in multimodal large language model (MLLM) training. A key observation motivating this work is the compute-memory trade-off exhibited by existing approaches. That is, compute-efficient approaches decouple modality encoders and generators from the LLM pipeline, which helps reduce pipeline bubbles that lead to wasted compute time. However, such approaches incur high activation memory overhead because intermediate activations must be retained until the corresponding backward passes. In contrast, memory-efficient approaches integrate all components into a single pipeline, which suffers from pipeline bubbles due to stage imbalance, further aggravated by data heterogeneity.

BigMac is proposed to address the tension between compute- and memory-efficient approaches. BigMac interleaves encoder and generator operations with an existing LLM pipeline schedule while preserving execution dependencies. It also supports heterogeneous context parallelism and FSDP.

The evaluation was conducted using 128 NVIDIA Hopper GPUs. The paper reports 1.08–1.9× speedups over the baselines, with stable peak memory usage as batch size increases. The paper also reports a production deployment involving a 345B-parameter MLLM trained on 1,536 GPUs.

Critical strengths
  1. Tackles an important systems optimization problem in large-scale MLLM training.

  2. Provides useful observations on the interplay between modality-specific components (encoders and generators) and the LLM backbone.

  3. Presents a system-level implementation and reports its deployment in a real-world production training system.

  4. Reports considerable speedups over the evaluated baselines.

Critical weaknesses
  1. The paper makes overly broad claims about the compute-memory trade-off and the generality of the proposed approach.

  2. Several assumptions and theoretical claims require further justification and clarification.

  3. The evaluation raises questions about the fairness of the baseline comparisons and the interpretation of the reported performance results.

  4. Important experimental configuration details are missing, limiting reproducibility.

  5. The evaluation could be extended to cover more diverse model and workload configurations to better demonstrate the generalizability of the proposed approach.

Questions for authors
  1. Overgeneralization of claims and the compute-memory trade-off.

    The paper could improve its writing by more clearly defining the scope of its claims and providing stronger justifications for its observations. For example, it categorizes existing methods into compute-efficient and memory-efficient approaches and uses this distinction to motivate a compute-memory Pareto frontier. However, the existence and generality of this Pareto frontier require further justification and supporting evidence. When pipeline stages are well-balanced, computation overlap can provide latency benefits for coupled pipelines. Conversely, decoupled execution may sacrifice such overlap and introduce additional overhead.

    The evaluation demonstrates improvements over the selected baselines, supporting the efficacy of BigMac under the evaluated configurations. However, these results do not necessarily establish that a general compute-memory Pareto frontier has been broken. The limited scope of the evaluation and its implications for generalizability should be more clearly acknowledged. Could the authors clarify how the Pareto frontier is defined and what evidence supports its existence? Under what workload and system configurations does the claimed trade-off hold?

  2. Assumptions regarding pipeline parallelism.

    The paper appears to assume that pipeline parallelism is generally necessary for large-scale MLLM training. For example, Section 3.1 states that the LLM backbone "usually requires pipeline parallelism." However, PP is only one of several distributed training approaches. Alternatives, including FSDP, tensor parallelism, and expert parallelism, may enable training without PP, depending on the model architecture and system configuration.

    Focusing on PP-based training is reasonable, but the paper should clearly define its scope rather than presenting PP-dependent challenges as general properties of MLLM training.

Could the authors justify the assumption that PP is generally required and clarify the conditions under which PP is necessary or preferable? How does this assumption affect the generality of the proposed approach?

  1. Justification of the scheduling guarantee.

    Section 3.1 claims that three encoder units are sufficient to preserve the interleaved 1F1B schedule without additional stalls. However, the justification for this guarantee could be elaborated further. The fact that encoder gradients become available before a subsequent LLM forward operation does not necessarily guarantee that the required encoder backward and subsequent forward computations can finish before the LLM needs the next encoder output. The actual execution latency and communication overhead may affect this guarantee.

    Furthermore, the analysis focuses on interleaved 1F1B with a virtual pipeline degree of at least two, so the applicability of the guarantee to other pipeline scheduling strategies is not fully established.

Could the authors clarify the assumptions regarding computation and communication latency required for the no-stall guarantee? Does the guarantee extend to other pipeline schedules, and if so, under what conditions?

  1. Fairness and interpretation of iteration-time comparisons.

    Figures 9 and 10 report iteration time under different per-GPU batch sizes. Although iteration time is an appropriate performance metric for distributed training, it is unclear whether the compared systems process identical effective global batch sizes per iteration.

Could the authors clarify whether the reported iteration times correspond to equivalent amounts of training work, including identical effective global batch sizes and optimizer updates, across the evaluated systems?

  1. Missing experimental configuration details.

    Some important experimental configuration details, including the PP, VPP, DP, and gradient accumulation configurations of each system, should be clarified, as these settings can considerably affect both computational efficiency and memory usage.

Could the authors provide the detailed parallelism configurations and relevant optimization settings for BigMac and each baseline? Were the baseline configurations independently tuned for the evaluated workloads?

  1. Additional performance metrics.

    Reporting samples/sec or tokens/sec, in addition to iteration time, could provide better insights into the reported performance improvements and make comparisons across different system configurations more interpretable.

Could the authors report training throughput in addition to iteration time? A breakdown of pipeline bubbles, computation time, and communication overhead would also help explain the sources of the reported speedups.

  1. Generalizability of the evaluation.

    The main evaluation uses the same LLM backbone and vision encoder across both workloads. Additional experiments varying model configurations, degrees of workload heterogeneity, and pipeline parallelism settings would strengthen the generalizability of the findings.

How sensitive is BigMac's performance to different model architectures, pipeline configurations, and degrees of workload imbalance? In particular, how does its performance change as the relative execution times of the encoder, LLM, and generator vary?

  1. Production-scale performance evaluation.

    The production deployment is a significant result, but Section 4.4 reports only training loss and gradient norm. These measurements provide evidence of stable training behavior but do not directly demonstrate the claimed compute-memory efficiency benefits at production scale.

Could the authors report throughput, peak memory usage, or pipeline efficiency measurements for the production deployment? A comparison against the previously used training configuration would be particularly helpful in demonstrating the practical benefits at this scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.