- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TROPOS, a risk-aware scheduler for distributed training that minimizes the CVaR of stochastic iteration completion times rather than relying on deterministic duration estimates. It combines differentiable critical-path evaluation, contention-aware duration modeling, importance sampling, and a heterogeneous graph attention network. Experiments on Qwen3 models report substantial reductions in P99 latency compared with RoundPipe, along with promising transfer results on an unseen MoE workload.
Critical strengths
-
The paper introduces a risk-aware perspective on distributed training scheduling, supported by theoretical analysis of the limitations of deterministic objectives.
-
The experimental evaluation is reasonably comprehensive, including comparisons with relevant schedulers, controlled ablations, and transfer experiments. The reported improvements in P99 latency are substantial, particularly for larger models.
Critical weaknesses
-
The mathematical validity of the importance sampling formulation requires clarification. The exponentially tilted distribution in Eq. (12) may not be normalizable under the unbounded Log-normal–Pareto model in Eq. (11). The authors should clarify whether truncation or other approximations are used and justify their effects.
-
The advantages of CVaR optimization are workload-dependent. Table 3 shows that the mean-based variant achieves better P99 latency on smaller models. Further analysis would help establish when risk-aware optimization provides consistent benefits.
-
The zero-shot transfer mechanism is insufficiently explained. The scheduler uses node-type-specific projections but is trained only on dense graphs. It remains unclear how the model handles previously unseen MoE-specific node types, which weakens the evidence supporting its claimed generalization capability.
Questions for authors
- Sampling Validity (Eqs. 11–13): The exponential tilting in Eq. (12) may not be normalizable under the unbounded Log-normal–Pareto distribution in Eq. (11). Could the authors clarify how this proposal distribution is implemented and sampled in practice? If truncation or approximation is applied, how does it affect the validity of the CVaR estimator?
- Acyclicity of Relaxed Scheduling (Eq. 10): The soft priority matrix allows weighted precedence relationships, while the max-plus recursion assumes an acyclic execution graph. How is acyclicity maintained during differentiable training, and how are potentially conflicting precedence relationships handled?
- In Section 4.3, operations sharing GPU, memory, or communication resources may experience correlated delays. Does the proposed duration model explicitly capture these dependencies, or are residuals sampled independently? How might the modeling assumptions affect the accuracy of tail-risk estimation?
- In Section 5.4, the scheduler uses node-type-specific projections but is trained only on dense graphs. How are the parameters associated with previously unseen MoE-specific node types handled during zero-shot transfer? Are these representations shared, initialized separately, or mapped to existing node types?
- In Section 5 and Appendix B, does the author use a portion of the traces to fit the model and then use new, independent traces to validate the results? Additionally, since P99 latency is a key metric, how many iterations are measured per run, and how are the results aggregated across the three runs?
- Effectiveness of CVaR Optimization: Table 3 shows that CVaR optimization outperforms mean-based optimization on larger models but not on smaller ones. Could the authors clarify what factors determine this performance difference? Additionally, since the sensitivity analysis of α is conducted only on Qwen3-1.7B, how was α = 0.2 selected for the larger models?
- In Figure 2, the theoretical reference is expressed in terms of the number of near-critical paths (n(z)), whereas Proposition 3 derives the bound using the total number of paths (N). Could the authors clarify the relationship between these quantities in the experimental construction?
- Minor concerns for presentation: (1) Text format for figure 2 is a little bit small, (2) The sign convention for ΔP99 differs between Tables 1 and 3, which may cause confusion. A consistent convention or an exlicit definition in the table captions would improve clarity.
We answer the eight questions in order, each with the measurement that settles it; all are incorporated in the revision. Changes are marked in blue in the new PDF.
Q1 (and W1) — tilted proposal. The reviewer is right that Eq. (11) admits no exponential moment: for every η>0 and κ>2, E[eηϵ]≥eηxπ(xm/x)κ→∞. Section 3.1 now bounds the residual, 0≤ϵv≤Ev, so dv(z)=cv(z)+ϵv is bounded for every schedule. Then 1≤Zη≤eηM, and the proposal is sampled exactly by rejection on ϵ with acceptance probability eη(S(ϵ)−M) (Proposition 4, Algorithm 2). Truncation costs at most α−1E[T1ϵ≤E] in CVaR; Pr(ϵv>Ev)=3.3×10−4 per operation at n=3000. On a fixed schedule at K=32, bias is 1.0% and variance 38% lower than naive sampling (Table 15).
Q2 — acyclicity. The soft matrix never builds the graph. In the forward pass, list scheduling on (sv) yields a topological order of (V,E) for any scores, and Eord links consecutive operations of each resource along it; all edges point forward in one total order, so Gz is acyclic for every parameter value, not merely at convergence. Conflicting precedences are excluded by construction: an operation is not eligible before its data predecessors, so a data edge overrides a contradicting priority, and scalar scores are transitive, so two priorities cannot conflict ([Πz]vu+[Πz]uv=Av⊤Au). Πz carries only the gradient, via oˉvu=ovu+[Πz]vu−sg([Πz]vu).
Q3 — correlated delays. Residuals are drawn independently given the schedule, so dependence between resource-sharing operations enters only through the contention term. We measure what this costs (Table 14, third row of each block): drawing the residuals of resource-sharing operations jointly at unchanged marginals lowers the P99 shortfall to 6.6% (1.7B) and 9.1% (14B). The sign is not uniform — dependence along a path widens the tail of T, dependence across parallel paths narrows the tail of their maximum — and the first dominates here. Every policy is trained under the same independence, so the bias is common to TROPOS and RP+Var, and CVaR's worst-case reweighting by 1/α (ρα=supEQ[T]:dQ/dP≤1/α) hedges part of the residual tail mass.
Q4 (and W3) — unseen MoE node types. There are none. Gate and expert operations are typed compute and all-to-all dispatch/combine communication, the two types learned on dense graphs; routing features (expert identity, expected load, ψ(h)) are zero on dense graphs and masked to zero at zero-shot. No parameter is new or re-mapped — what is new is connectivity (dispatch → experts → combine), which message passing handles. The 50 adaptation steps of Table 2 unmask the routing features.
Q5 — independent traces and aggregation. Yes. Per configuration the duration model is fitted on one 3000-iteration run; every reported prediction error is on a second, independent run (Tables 13–14). An end-to-end run has 5 warm-up and 3000 measured iterations; P99 is computed within each run, and the reported value is the mean over three runs, with a 95% interval over those three runs. Per-run logs in Table 6.
Q6 (and W2) — when CVaR pays, and how α was chosen. Table 3 confounds model size with contention; Table 21 does not. With model, batch and budget fixed and only PCIe load varied, the risk objective's P99 gain over the mean objective is −2.3 at 26.8% utilisation (95% interval [−5.7), 10.6% at 45.6% and 13.4% at 67.2% on 1.7B, and 6.8%, 15.0%, 16.2% on 14B. The crossover is the low-contention end of one curve, not a property of small models: at 45.6% the 1.7B model gains 10.6%, more than the 14B model's 6.8% with no extra load. α=0.2 was selected on 1.7B alone (Figure 5a) and then fixed for every other workload and platform, including ones it never helped to tune, so the large-model gains are not from per-model tuning.
Q7 — n(z) versus N. In the construction of Figure 2 all N paths have equal expected length, hence all are near-critical and n(z)=N; the figure now uses N. In general the relevant count is the slack-weighted Nλ=∑pe−λgp∈[1,N] of Proposition 3, which equals N exactly in this construction and degenerates to 1 when one path is slack.
Q8 — presentation. Figure 2 is regenerated with larger fonts and the N axis. Every latency table defines ΔP99 in its caption, and Tables 1, 3, 19, 20 and 23 share one convention: Δ=(row−reference)/reference, negative meaning lower than the reference; Table 21 reports a reduction and says so.