- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors introduce introduce SorcarCCL which is a research agent that extends related work (OverlayCCL) by proposing and evaluating optimizations on a modification of overlayccl's simulator. The authors indicate that their loop finds optimizations that overlay missed. They then show that SorcarCCL improves upon OverlayCCL's performance across multiple problems, with and end-to-end training of 9B+ models improved by 2.13× and 2.15× on a per-step basis.
Critical strengths
A few strengths:
- The authors have measured results on hardware (from the paper) "7 AWS trn1.32xlarge instances in a single placement group, with 32 NeuronCores per node and 224 ranks".
- The authors indicate that there are five xm.*collective primitives that are available.
- Amongst their contributions, the authors increase the fidelity of their simulator to capture pre-/post-processing time and use a single, predetermined set of operations as the calibration probe for the simulator.
- The programs come with ablations to help make the outputs more inspectable.
Critical weaknesses
There's just a LOT of this paper which reads AI-written which makes parsing many of the things presented just a tad onerous, e.g., "This oracle, not the wording of the prompt, defines correctness" -- which to be honest, I'd never assumed that the wording of the prompt would have defined correctness so this just feels like empty padding of pages? What exactly was validated?
-
The primary weakness from my perspective is that I'm not sure I understand the takeaway here? If forced, I'd say assume that the authors seem to trade-off an adaptive search, that uses a cheap proxy to do wider initial searches, and then uses an adaptive loop to get better results. This is good, but I don't think the authors have actually shown me this? I need to put in a lot of effort to even get that insight/intuition?
-
https://pytorch.org/blog/kernelagent-hardware-guided-gpu-kernel-optimization-via-multi-agent-orchestration/ seems similar but for kernel level optimizations? How might you be able to distinguish your results from this -- beyond simply saying kernel optimizations are different from collectives? I'm not sure it's apparent to me why this might be fundamentally different?
-
I'd also like details on baselines? How might the adaptive version of Overlay compare on the other problems that you're comparing against it on. My understanding was that the adaptive search in overlay didn't bring significant marginal utility vs. the fixed search budget, and hence was not recommended. But you do have an adaptive search -- is your method helpful due to adaptivity? Due to wider search? I don't see it yet?
-
How would SorcarCCL perform on the older OverlayCCL simulator? I see that you gave overlay the newer simulator, but I don't think that the older method might be well suited to the new sim? The replication work seems to be a tad iffy too?
"The isolated-amplified bolt-on avoids this: the 8 emitted family kernels run
repeatpasses over fixed native payloads in a graph that is disjoint from the model graph (xm.mark_step()everyinner_segpasses → one reused compiled graph whose count is independent ofrepeat). The model loss is then bit-identical across backends, while amplification (repeat=224) makes the family dominate the step so the overall step ratio tracks the per-call family ratio (~3×). A discarded global-scalar "sink" keeps the collectives alive against dead-code elimination without feeding the loss."
^ From the readme of the repo -- this makes it seem like the speedup is over an amplified extra set of elements? Additionally, it seems like the outputs here do not feed the loss? So I don't think the bit-identical part matters?
Questions for authors
See weaknesses?
4 comments
Thank you for your response. Unfortunately, I still don’t quite see what my main takeaway should be. The broader writing is still too opaque. My understanding is that the workflow is to:
- Use a cheap simulator to explore candidate rewrites.
- Have the LLM compare ideas before implementing them.
- Test code with adversarial inputs and use failures to repair it.
- Keep a record of what worked and failed, so the agent can use that history.
Although there are ablations, I still have difficulty in understanding, as an example -- how I should generalize this to my own workflows. What's the most impactful single step from your paper? I don't see it? Is it that everything is so tightly coupled that we can't generalize/decompose to it's composing units?
At present, my takeaway feels like “implement a good workflow” I don't see how this translates to a methodology contribution?
Thanks for the followup! We believe we can answer “What is the single most impactful step?” directly using data already in the paper.
The single most impactful step is proposing multiple ideas at once and comparing them before implementing any of them (Stage A).
Your 4-point summary of the workflow is accurate. And we'll say that point 2 is the key takeaway. The NoA baseline represents what a practitioner would normally do. It uses the standard generate -> evaluate -> refine workflow with the same search space, proposer model, and budget as OverlayCCL. Adding Stage A improves the results on 7 more (net) problems out of 40.
This gives us a direct comparison between a reasonable default agent workflow and our method. The key difference is simple. When testing all implementations of all possible ideas is expensive (which is true in distributed training), the LLM agent should first generate several ideas and compare them against one another and against previously observed failures. Only the most promising idea is then implemented and tested. This avoids spending time and resources on weak candidates.
Regarding whether our workflow "generalizes/decomposes to it's composing units", we believe the workflow is modular, and the ablation study shows this.
We remove each stage separately and measure how much performance drops. Every version still runs from beginning to end, which shows that the stages can be separated and studied independently.
The results also show that each stage contributes measurable value:
- Removing proposal batching and pairwise comparison costs 7 net wins.
- Removing selection and iterative refinement costs 7 net wins.
- Removing adversarial testing costs 6 net wins.
- Removing the convergence check costs 5 net wins.
Importantly, NoB still outperforms OverlayCCL on 33 of the 40 problems. Removing a stage makes the workflow less effective, but does not cause it to break down. This is evidence that the stages are modular, not too tightly coupled to separate.
The stages also work well together because they address different problems. One helps avoid wasting resources on weak ideas. Another helps recover candidates that are almost correct. Others help detect failures and avoid repeating unsuccessful attempts.
For a methodology contribution, we frame it as a set of four design principles that apply beyond this particular system.
The paper is not simply about putting together a better agent workflow. It identifies four practical principles for searching when testing candidates is expensive. Each principle addresses a specific problem, and the ablation study measures what happens when we remove it.
-
Focus on ranking candidates correctly, not predicting their exact performance. The simulator does not need to predict exact running times. It just needs to identify which candidates are better. Our simulator ablation shows this.
-
Spend reasoning effort before spending evaluation resources. Generate multiple candidates and compare them before implementing the most promising one. Removing this step costs 7 wins under a same budget.
-
Test and repair candidates instead of simply accepting or rejecting them. Some candidates are almost correct and can become successful after small fixes. A one-time correctness check would reject these candidates without giving them a chance to improve.
-
Keep track of what has already worked and failed. Record validated results and known failure cases so the agent can reuse them instead of repeating the same unsuccessful experiments.
These principles are not specific to collective optimization. They can also apply to other tasks where the system being optimized is difficult to understand, a simulator or other cheap evaluation method is available, and final validation is expensive. For example, this workflow can be extended to measure compilation time on AWS Trainium. Other tasks in other hardware could also benefit from the workflow, like kernel autotuning and query rewriting.
We would also like to highlight one final observation from the ablation. On most easy problems, all versions of the workflow eventually find the same solution. The biggest benefits appear on the harder problems, where simply generating and testing one idea at a time often fails.
Thanks, I'll raise my score in response to this.
We thank the reviewer for the direct questions.
The takeaway. We apologize for not underscoring the key takeaways in an explicit way. The high-level takeaway is indeed a trade-off: favoring an adaptive search strategy that uses a cheaper proxy to explore a broader set of candidates initially, then iteratively refines promising candidates through an adaptive feedback loop. At a finer level, our ablations reveal 3 additional insights in the "adaptive feedback loop": (a) Pairwise judging of ideas helps reduce search costs by filtering out unpromising candidates early, as in the "noA" ablation. (b) LLM-proposed adversarial testing is essential for "saving" promising strategies with minor bugs or missing edge cases. Without it, these candidates would be rejected by the one-shot correctness gate. (c) The global ledger, which tracks validated improvements and records failure modes, is essential for avoiding redundant exploration and reducing search costs.
Relation to KernelAgent. We've added it to Related Work in the revised paper, together with the boundary distinction below and a row in the new comparison table (Table 2). The distinction is not simply the topic label of "kernels vs. collectives," but rather what the vendor boundary permits and where performance gains come from. KernelAgent authors new GPU kernels under an open programming model that provides low-level control. KernelAgent could indeed be ported to computational kernels on Trainium, as Trainium supports programmable custom kernels through the Neuron Kernel Interface (NKI). However, it cannot be trivially extended to collective communication on Trainium because the underlying collective implementations are exposed as black boxes. Nothing below the five
xm.*collective APIs is writable, so authoring collective kernels on Trainium is unavailable by construction.Our line of optimization is a combination of the surrounding computation (where we have greater control) and the sequence of collective calls (which must be treated as black boxes). Finding optimization opportunities requires both modeling black-box collective operations and learning from a faithful simulator to understand how we better pre-process/post-process our data around the calls to these black boxes.
Adaptive OverlayCCL as the baseline. Adaptivity vs. wider search. Adaptivity and wider search are not the sole differentiator. Removing selection and iteration from SorcarCCL (in our "NoB" ablation) still shows 33/40 wins over OverlayCCL. Pairwise judging of ideas, LLM-proposed adversarial testing, and the global ledger are all crucial parts of the loop.
SorcarCCL on the older OverlayCCL simulator; fairness of giving OverlayCCL the newer one. Note that our extension only adds new cost terms. It does not change the terms inherited from OverlayCCL's released Phase-1 simulator. OverlayCCL's candidates receive the same scores as before in the new simulator. Using both (old and new) versions of the simulator, SorcarCCL reproduces OverlayCCL's emitted code (logically equivalent) on Overlay's own eight problems (Section 4.1), providing a replication check.
The amplified e2e setup and problem that outputs do not feed the loss".
We did perform an experiment where the collective outputs directly into the model's computation and loss. Putting the emitted collective inside the transformer layer (as part of the model's own tensor-parallel communication) does preserve the about 2× performance ratio (as documented in the artifact README in the
/e2efolder). However, it breaks bit-identical loss parity: when Sorcar and Overlay collectives are integrated into the same compiled model execution, the Neuron compiler changes the execution order of the model's own BF16 reductions. Because floating-point addition is sensitive to operation order, this introduces small numerical differences that accumulate over a few training steps, causing the loss to drift (~0.1% to at most 3%). This is a consequence of the non-associativity of floating point numbers, not a bug of the code generated by SorcarCCL.We chose to report isolated-amplified bolt-on numbers to enforce loss parity, following the established approach in this line of research (e.g., MSCCL, TACCL, OverlayCCL). We claim this setup satisfies our main purpose:
Validate execution under real training conditions. The emitted rewrites should compile and run in a real 224-rank training process, exercising the actual compiler, runtime, memory pressure, and EFA network. This captures effects that isolated per-problem benchmarks cannot fully account for, such as compilation overhead and hardware-level computation–communication overlap.
Verify that isolated speedups survive co-location. The per-call family speedup (2.94~2.96×) matches the isolated per-call measurements, showing that the gains persist when multiple collective strategies run alongside one single training step.