- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training system for multimodal LLMs that aims to improve computational efficiency while limiting modality activation memory. It preserves the LLM's internal operator order and inserts encoder and generator computations at dependency-safe points. A bounded encoder lookahead window and early backward execution shorten activation lifetimes. The system also implements the cross-module communication needed to execute the schedule and supports context parallelism and FSDP. The claimed constant-memory bound concerns modality activations as the number of microbatches increases at a fixed parallel configuration, rather than total training memory.
Critical strengths
- The scheduling approach addresses a useful practical problem. BigMac combines limited encoder lookahead with early backward execution to shorten activation lifetimes while preserving the LLM's internal operator order.
- The design extends beyond encoder-only scheduling. The paper handles both encoders and generators and describes the cross-module communication, context-parallel conversion, and FSDP execution needed to support the schedule.
Critical weaknesses
-
The incremental contribution over prior scheduling systems is not sufficiently clear. MegaScale-Omni already describes on-demand encoder insertion and early encoder backward execution to reduce activation memory. The paper does not adequately explain how BigMac differs from this design on understanding-only workloads, and no direct comparison is provided. Its characterization of the compute–memory trade-off in existing systems also needs more qualification: Optimus includes fine-grained scheduling of encoder operations into LLM bubbles, while LongCat-Flash-Omni explicitly uses encoder activation recomputation. Neither observation establishes that these systems achieve BigMac's claimed trade-off, but both are relevant to the motivation and comparison. Generator support and the broader scheduling framework may be meaningful extensions; the current presentation does not clearly separate those contributions from the encoder-side ideas already available.
-
The analysis and evaluation do not yet establish the full claimed compute–memory advantage. Section 4.1 describes a sequential Optimus execution, leaving unclear whether the original fine-grained bubble scheduler and parallel-plan search are retained, and how the baseline is extended to generation. A matched comparison with encoder activation recomputation is also missing. Since Optimus runs out of memory at every tested generation batch size, those results demonstrate a memory advantage over the tested configuration but leave the throughput comparison against a compute-efficient alternative unresolved. On the analytical side, Section 3.1 provides a brief argument for the three-unit lookahead under interleaved 1F1B with at least two virtual chunks, but does not sufficiently derive the ordering property or explain its boundary conditions. Preserving operator order alone does not establish unchanged efficiency once execution-time variation and communication are included. The CP and FSDP experiments are useful, but do not isolate these central scheduling claims. Overall, the evidence is more convincing about the schedule's practical usefulness than about matching the idealized compute-efficient execution.
Questions for authors
-
Does the Optimus baseline retain the original fine-grained encoder-in-bubble scheduler and parallel-plan search? If the tested schedule is sequential, was it selected by the optimization procedure or imposed by the implementation? How was Optimus extended to handle the generator? Please provide the per-module parallel configurations and recomputation settings used for the main comparisons, along with the code version or implementation details identifying the Megatron-DistTrain baseline.
-
With encoder activation recomputation enabled under otherwise matched settings, how do Optimus or a LongCat-style baseline compare with BigMac in iteration time and peak memory? This comparison would help establish how much of the advantage comes from the proposed schedule rather than the absence of recomputation in the tested baseline.