acceptodds
Under review as a conference paper at ICLR 2027

ESUN-Bench: A Workload-Aware Benchmarking Framework for Ethernet-Based Scale-Up AI Fabrics

Abstract

Ethernet fabric metrics do not by themselves determine AI execution performance. The same communication delay can be hidden by concurrent computation or exposed at a synchronization boundary. Existing machine learning network co-simulators provide foundations for coupling communication with application execution. However, Ethernet scale-up benchmarking additionally requires mechanism-specific modeling and observability to interpret these interactions. We present ESUN-Bench, a benchmarking framework that connects packet-level simulation with communication operation timing and an iteration directed acyclic graph. In addition, we formalize the interfaces required for this mapping, including packet-to-operation association, communication completion semantics, and dependency-aware timing propagation. Moreover, we define exposed iteration overhead as the modeled iteration-time difference between matched network configurations, retaining workload dependencies that aggregate packet metrics discard. Evaluation across 102 benchmark cases investigates what communication timing reveals about iteration overhead, which workload conditions shape that timing, and how network-mechanism gains translate into application benefits. The results show that preserving communication semantics and execution dependencies improves iteration-overhead estimation over aggregate network metrics. They further identify message granularity, congestion pressure, and spatial contention as workload factors that shape communication service. Mechanism comparisons demonstrate that activation, communication improvement, and iteration benefit are distinct outcomes: the application value of a network optimization depends on the completion times it changes and the execution dependencies those changes affect. As a lightweight simulation framework, ESUN-Bench fills the gap in existing Ethernet-oriented evaluation methodologies and encourages more researchers to engage in ESUN-related studies.

Primary area:
datasets and benchmarks
Keywords:
AI Scale-Up Systems, Ethernet Fabrics, Workload-Aware Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_N1HFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ESUN-Bench, a workload-aware benchmarking framework for Ethernet-based scale-up AI fabrics. It addresses the observation that communication delays may be hidden by concurrent computation or extend iteration time through execution dependencies, making packet latency and throughput insufficient to directly characterize application benefits.

The framework combines four levels of workload abstraction, packet-level simulation, and execution-DAG analysis to associate network events with communication completion and quantify exposed iteration overhead (EIO). The authors provide 102 benchmark cases. The experiments report that, compared with summing communication-operation duration changes, the EIO Predictor reduces iteration-overhead prediction MAE from 0.264 μs to 0.099 μs, and illustrate the distinction between mechanism activation, communication improvement, and iteration-level benefit.

Critical strengths

S1. The evaluation problem is practically relevant. Packet latency and throughput alone do not adequately characterize the value of network optimizations for AI execution. Focusing on communication completion and iteration time helps clarify the relationship between network metrics and application benefits.

S2. The cross-layer association design is reasonably clear. The framework preserves mappings among packets, transfers, and communication operations, and distinguishes participant-local completion from global synchronization. Table 4 further separates mechanism activation, communication improvement, and iteration-level benefit, providing useful observables for mechanism analysis.

S3. The evaluation covers a broad range of scenarios and includes preliminary cross-validation. The benchmark cases cover protocol overhead, congestion, collectives, topology, scheduling, and reliability. The appendix documents mechanism configurations and reports timing agreement with ns-3 for selected packet-level and communication-operation-level metrics.

Critical weaknesses

W1. The incremental value of ESUN-specific evaluation is not sufficiently demonstrated. The paper positions itself as a complement to existing ML–network co-simulation frameworks. However, Themis (ISCA 2022), Khan et al. (HOTI 2022), and ATLAHS (SC 2025) have already investigated, from different perspectives, the relationship among communication behavior, computation–communication overlap, and end-to-end performance. EIO is also conceptually close to prior analyses of exposed communication time. The incremental value should therefore be demonstrated more clearly through the additional analytical capabilities and findings enabled by ESUN-specific mechanism modeling. The current results primarily confirm known effects of computation overlap, message granularity, and traffic hotspots. Although Table 4 jointly reports mechanism events, communication-operation durations, and iteration-time changes, the analysis of how specific mechanisms affect communication completion and the execution critical path remains limited. I suggest presenting concrete cross-layer attribution cases that demonstrate what additional explanations or application performance bottlenecks ESUN-Bench can reveal compared with existing evaluation approaches.

W2. The representativeness of the AI execution templates and application-level validation remain insufficient. Table 1 lists three execution-template families comprising nine cases, whereas Table 6 provides six synthetic workloads and simplified DAG descriptions. Their correspondence, complete dependency structures, and phase durations remain unclear. The computation and communication parameters lack supporting real execution traces or measurements. Although the experimental setup reports an RTX 3090, its role within the simulation program is not explained—for example, whether it was used to collect actual GPU operator or NCCL communication timings. The current validation relies primarily on synthetic workloads and simulation results. Whether these simplified templates represent communication dependencies in real training and inference, and whether EIO reflects actual application performance, remain insufficiently supported.

W3. The fixed computation baseline and predefined overlap parameters limit the applicability of the conclusions. Table 6 uses a uniform 40 ms computation baseline and assigns a fixed overlap ratio to each workload. This is useful for isolating network effects, but the resulting conclusions depend on the assumption that computation costs do not change with communication contention. Actual communication kernels may compete for GPU execution and memory resources. NVIDIA also notes that increasing NCCL communication CTAs may improve communication microbenchmarks while degrading end-to-end workload performance. I suggest clarifying how the overlap parameter jointly determines execution timing with DAG dependencies and resource constraints, and conducting sensitivity analyses over computation-to-communication ratios and overlap levels to assess the stability of the main conclusions and the relative benefits of network mechanisms.

W4. The evidence for the EIO Predictor’s accuracy is not sufficiently independent. Section 4.2 computes the target ΔO from simulation-recorded iteration times and generates predictions using communication timings, the DAG, and overlap parameters from the same simulations. Section 3.5 also defines iteration time as the longest-path duration in the augmented DAG. If the target and prediction share the same timing-propagation rules, the low error may primarily reflect internal model consistency. The paper should clarify their respective information sources and computation procedures, and explain why the current validation is not simply a recomputation of results from the same execution model.

W5. The validity of the approximate network-mechanism models and the controlled comparisons require further support. Appendix A.2 discloses the local-pause approximation for PFC, fixed-rate credit replenishment for CBFC, and ECN without a sender response. However, the effects of these approximations on communication completion and application benefits have not been sufficiently validated. Moreover, the comparison between Ethernet and ESUN-Header changes both header overhead and the Flow Label used for path selection, making their individual contributions difficult to isolate. ECN only performs marking, so its zero benefit cannot establish the effectiveness of a complete closed-loop congestion-control mechanism. I suggest adding controlled comparisons and explaining the applicability of each simplified model and its impact on the main conclusions.

W6. The dynamic coupling between network simulation and AI execution remains unclear. Section 3.3 specifies that communication operations are released by their DAG predecessors, but Appendix A.2 describes projecting reconstructed communication timings onto an iteration DAG. It remains unclear whether the implementation provides bidirectional coupling during simulation. For example, delayed MoE Dispatch can postpone expert computation and subsequent Combine communication, thereby changing network contention and queueing. The paper should distinguish between computing iteration time from previously generated network timings and dynamically changing subsequent traffic according to execution dependencies. A case involving multiple dependent communication stages would help validate the impact of this distinction on communication completion and EIO.

Questions for authors

Q1. Could the authors provide the complete DAG, phase durations, and execution timeline for one MoE case, explaining the role of overlap and how delayed Dispatch changes the initiation of subsequent Combine communication and network contention?

Q2. Do the ΔO target and EIO Predictor use the same DAG timing-propagation logic? What information does each use, and why does the prediction error constitute independent evidence of predictive accuracy?

Q3. What communication and iteration-level benefits arise when only header overhead is changed with routing held fixed, and when only the Flow Label is changed with header overhead held fixed? How is the validity of the approximate flow-control models established?

Reviewer_6mrSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

ESUN-Bench proposes a framework for evaluating Ethernet scale-up performance under AI workloads. It provides 102 benchmark cases for network mechanisms and AI communication workloads. It models selected ESUN mechanisms using simplified packet-level simulation. A dependency-based timing model was formulated using simulated communication completion times to estimate the final changes in AI iteration time. This model produces lower estimation errors than predictors based on aggregating network metrics. Selected packet and communication-operation timings are also compared against ns-3.

Critical strengths
  1. The paper addresses the problem of how a change in network configuration can result in the final application-level runtime. By taking operation dependency and computation-communication overlap into account, the communication timing exposed through EIO gives more insight than packet-level statistics alone.

  2. The controlled scenario suite and explicit mechanism-event reporting could be useful for diagnosing network behavior. The ns-3 comparisons in Appendix A.1 provide an initial consistency check for selected packet and communication-operation timings.

Critical weaknesses
  1. The application-level conclusions lack sufficient validation and workload grounding.

Section 4.2 reports a mean absolute error of 0.099 μs for the EIO predictor, but how the target values were constructed remains unclear. It seems that the values predicted by the EIO predictor and the target values given by the simulation come from the same machinery. The agreement between them could only demonstrate internal consistency instead of accuracy.

Appendix A.1 compares selected packet P99 and operation-completion timings against ns-3, but not end-to-end AI iteration times against real hardware. These comparisons therefore do not validate the compute model, the overlap behavior, or the predicted changes in iteration time.

The workload construction compounds this gap. Table 6 describes synthetic generators built on a common 40 ms computation baseline with configured overlap ratios, yet it offers no specification on how exactly these traces are generated and the exact computation pattern for each node. The synthetic computation-communication pattern described in the paper could be far from the real computation-communication pattern that will be displayed in a real AI training trace. For example, framework and compiler choices - such as operation fusion and communication grouping - can change the DAG that is actually executed. The simulator does not have to implement such a compiler internally, but it should state its abstraction level and the assumptions it relies on. This deserves particular attention since these assumptions can directly determine EIO.

  1. The framework's incremental contribution and mechanism-evaluation scope are insufficiently established.

ASTRA-sim already provides dependency-aware execution graphs, resource-aware scheduling, collective decomposition, and a packet-level ns-3 backend; ATLAHS likewise offers application-centric network simulation, explicit trace construction, and validation against measured runtime [1, 2]. This paper does not clearly state which capabilities its method adds beyond Ethernet-specific models and benchmark configurations. Also, the claimed lightweight character is not supported by comparative runtime measurements.

The ECN experiment exposes the limits of the mechanism evaluation. Appendix A.2 and Table 7 state explicitly that the evaluated ECN configurations lack a closed-loop sender-rate response. As a result, the reported 14,660 marks with zero timing improvement do not exercise an operational ECN-based congestion controller: marking alone cannot be expected to improve service when it never changes transmission behavior. The narrow conclusion that activation does not establish benefit is valid, but the experiment largely reflects the chosen model rather than showing how congestion-control improvements propagate to application execution.

Questions for authors
  1. Predictor and target: Please provide the exact algorithms or pseudocode that generate the EIO prediction and the reference target Delta O in Section 4.2. Which inputs and computations do they share, what produces the nonzero prediction error, and which Table 3 predictors receive linear calibration? What reference configuration is used in Table 2? Does predicting EIO for a new network configuration require a completed packet-level simulation of that configuration?

  2. Workload provenance and construction: For the six generators in Table 6 and the three templates in Table 2, how are compute nodes, per-node durations, dependencies, and device/resource assignments generated? What supports the 40 ms compute baseline and the specified overlap ratios? Is overlap an emergent property of the execution schedule or an independently applied adjustment? Which model, batch/sequence dimensions, parallelism settings, and execution stack do the named AI templates represent? Are these execution assumptions grounded in measured traces, and what validation supports their fidelity?

  3. Distinction from prior frameworks: Which specific capabilities are unavailable in ASTRA-sim with its ns-3 backend or in application-centric toolchains such as ATLAHS? Please distinguish new simulation methodology from new mechanism implementations, event instrumentation, and benchmark cases. What evidence supports the claimed lightweight character?

Reviewer_w61ROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ESUN-Bench, a simulation-based framework for evaluating Ethernet scale-up AI fabrics. It maps packet events to transfer and communication-operation completion, and then propagates these timings through an iteration DAG. The paper defines exposed iteration overhead (EIO) as the modeled iteration-time difference between paired network configurations. Evaluation includes 102 synthetic cases, comparisons with aggregate network metrics, mechanism studies, and eight packet/operation-level comparisons against ns-3.

Critical strengths

The cross-layer packet-to-operation-to-DAG formulation addresses an important problem: a network improvement matters only when it affects communication on the application’s critical path.

The framework provides reasonably broad controlled coverage and multi-level instrumentation, which could be useful for diagnosing where network effects arise.

Critical weaknesses

Table 3 appears circular as written. The target is ΔO=1000[(Titerv−Tcomp)−(Titerb−Tcomp)]=1000ΔTiter\Delta O=1000[(T_{\mathrm{iter}}^{v}-T_{\mathrm{comp}})-(T_{\mathrm{iter}}^{b}-T_{\mathrm{comp}})]=1000\Delta T_{\mathrm{iter}}, because compute time is shared within each pair. The final predictor is also ΔTiter\Delta T_{\mathrm{iter}}, constructed from the same simulated communication timings and DAG. Therefore, the reported accuracy appears to measure self-consistency rather than prediction against independent iteration-level ground truth. The fact that the correlation is not exactly one is also unexplained.

The evaluation does not establish realistic application-level fidelity. All workloads are artificial templates with fixed compute times and hand-selected overlap parameters. The ns-3 validation covers only eight selected packet/operation-level cases under a substantially different configuration, with no hardware or independent iteration-level validation. Moreover, mechanisms are simplified. For example, ECN has no sender-response loop. So some reported findings follow directly from the modeling assumptions.

Questions for authors
  • Is the target TiterT_{\mathrm{iter}} in Table 3 produced by an execution model independent of Eq. 3? If so, please describe the two pipelines and explain why ΔO\Delta O and ΔTiter\Delta T_{\mathrm{iter}} are not algebraically identical.

  • Can the authors provide independent iteration-level validation using hardware measurements or a trusted full-system simulator, measured AI traces, and closed-loop mechanism models? Such evidence would materially affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.