- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents ESUN-Bench, a workload-aware benchmarking framework for Ethernet-based scale-up AI fabrics. It addresses the observation that communication delays may be hidden by concurrent computation or extend iteration time through execution dependencies, making packet latency and throughput insufficient to directly characterize application benefits.
The framework combines four levels of workload abstraction, packet-level simulation, and execution-DAG analysis to associate network events with communication completion and quantify exposed iteration overhead (EIO). The authors provide 102 benchmark cases. The experiments report that, compared with summing communication-operation duration changes, the EIO Predictor reduces iteration-overhead prediction MAE from 0.264 μs to 0.099 μs, and illustrate the distinction between mechanism activation, communication improvement, and iteration-level benefit.
Critical strengths
S1. The evaluation problem is practically relevant. Packet latency and throughput alone do not adequately characterize the value of network optimizations for AI execution. Focusing on communication completion and iteration time helps clarify the relationship between network metrics and application benefits.
S2. The cross-layer association design is reasonably clear. The framework preserves mappings among packets, transfers, and communication operations, and distinguishes participant-local completion from global synchronization. Table 4 further separates mechanism activation, communication improvement, and iteration-level benefit, providing useful observables for mechanism analysis.
S3. The evaluation covers a broad range of scenarios and includes preliminary cross-validation. The benchmark cases cover protocol overhead, congestion, collectives, topology, scheduling, and reliability. The appendix documents mechanism configurations and reports timing agreement with ns-3 for selected packet-level and communication-operation-level metrics.
Critical weaknesses
W1. The incremental value of ESUN-specific evaluation is not sufficiently demonstrated. The paper positions itself as a complement to existing ML–network co-simulation frameworks. However, Themis (ISCA 2022), Khan et al. (HOTI 2022), and ATLAHS (SC 2025) have already investigated, from different perspectives, the relationship among communication behavior, computation–communication overlap, and end-to-end performance. EIO is also conceptually close to prior analyses of exposed communication time. The incremental value should therefore be demonstrated more clearly through the additional analytical capabilities and findings enabled by ESUN-specific mechanism modeling. The current results primarily confirm known effects of computation overlap, message granularity, and traffic hotspots. Although Table 4 jointly reports mechanism events, communication-operation durations, and iteration-time changes, the analysis of how specific mechanisms affect communication completion and the execution critical path remains limited. I suggest presenting concrete cross-layer attribution cases that demonstrate what additional explanations or application performance bottlenecks ESUN-Bench can reveal compared with existing evaluation approaches.
W2. The representativeness of the AI execution templates and application-level validation remain insufficient. Table 1 lists three execution-template families comprising nine cases, whereas Table 6 provides six synthetic workloads and simplified DAG descriptions. Their correspondence, complete dependency structures, and phase durations remain unclear. The computation and communication parameters lack supporting real execution traces or measurements. Although the experimental setup reports an RTX 3090, its role within the simulation program is not explained—for example, whether it was used to collect actual GPU operator or NCCL communication timings. The current validation relies primarily on synthetic workloads and simulation results. Whether these simplified templates represent communication dependencies in real training and inference, and whether EIO reflects actual application performance, remain insufficiently supported.
W3. The fixed computation baseline and predefined overlap parameters limit the applicability of the conclusions. Table 6 uses a uniform 40 ms computation baseline and assigns a fixed overlap ratio to each workload. This is useful for isolating network effects, but the resulting conclusions depend on the assumption that computation costs do not change with communication contention. Actual communication kernels may compete for GPU execution and memory resources. NVIDIA also notes that increasing NCCL communication CTAs may improve communication microbenchmarks while degrading end-to-end workload performance. I suggest clarifying how the overlap parameter jointly determines execution timing with DAG dependencies and resource constraints, and conducting sensitivity analyses over computation-to-communication ratios and overlap levels to assess the stability of the main conclusions and the relative benefits of network mechanisms.
W4. The evidence for the EIO Predictor’s accuracy is not sufficiently independent. Section 4.2 computes the target ΔO from simulation-recorded iteration times and generates predictions using communication timings, the DAG, and overlap parameters from the same simulations. Section 3.5 also defines iteration time as the longest-path duration in the augmented DAG. If the target and prediction share the same timing-propagation rules, the low error may primarily reflect internal model consistency. The paper should clarify their respective information sources and computation procedures, and explain why the current validation is not simply a recomputation of results from the same execution model.
W5. The validity of the approximate network-mechanism models and the controlled comparisons require further support. Appendix A.2 discloses the local-pause approximation for PFC, fixed-rate credit replenishment for CBFC, and ECN without a sender response. However, the effects of these approximations on communication completion and application benefits have not been sufficiently validated. Moreover, the comparison between Ethernet and ESUN-Header changes both header overhead and the Flow Label used for path selection, making their individual contributions difficult to isolate. ECN only performs marking, so its zero benefit cannot establish the effectiveness of a complete closed-loop congestion-control mechanism. I suggest adding controlled comparisons and explaining the applicability of each simplified model and its impact on the main conclusions.
W6. The dynamic coupling between network simulation and AI execution remains unclear. Section 3.3 specifies that communication operations are released by their DAG predecessors, but Appendix A.2 describes projecting reconstructed communication timings onto an iteration DAG. It remains unclear whether the implementation provides bidirectional coupling during simulation. For example, delayed MoE Dispatch can postpone expert computation and subsequent Combine communication, thereby changing network contention and queueing. The paper should distinguish between computing iteration time from previously generated network timings and dynamically changing subsequent traffic according to execution dependencies. A case involving multiple dependent communication stages would help validate the impact of this distinction on communication completion and EIO.
Questions for authors
Q1. Could the authors provide the complete DAG, phase durations, and execution timeline for one MoE case, explaining the role of overlap and how delayed Dispatch changes the initiation of subsequent Combine communication and network contention?
Q2. Do the ΔO target and EIO Predictor use the same DAG timing-propagation logic? What information does each use, and why does the prediction error constitute independent evidence of predictive accuracy?
Q3. What communication and iteration-level benefits arise when only header overhead is changed with routing held fixed, and when only the Flow Label is changed with header overhead held fixed? How is the validity of the approximate flow-control models established?