acceptodds
Withdrawn from ICLR 2027

TROPOS: Risk-aware Scheduling for Distributed Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Computation–communication overlap schedulers rank candidate execution plans under point-estimated operation durations, then run the chosen plan for thousands of iterations in which durations vary with contention. Because durations add along each dependency path and iteration time is the maximum over competing paths, the objective is convex in the durations, and its point-estimate value under-predicts the expected iteration time by a Jensen gap that grows with the number of near-critical paths and with the tail weight of the durations. We formulate overlap scheduling as minimizing the conditional value-at-risk (CVaR) of a stochastic max-plus completion time under memory constraints and show that the deterministic objectives of RoundPipe, LAER-MoE, FCP, HCMS, and NEST are its point-mass special case. TROPOS makes this objective trainable with a log-sum-exp relaxation of the max-plus recursion whose error is bounded and one-sided, a contention-aware duration model with a log-normal–Pareto residual fitted from traces, and tail-focused importance sampling; a heterogeneous graph attention network trained on the resulting loss emits a schedule in one forward pass of 1.24 ms. On Qwen3 dense models from 1.7B to 32B parameters on eight RTX 4090 GPUs, TROPOS lowers P99 iteration latency by 21.0–35.7% and raises throughput by 14–48% relative to RoundPipe, and by a further 8–17% at P99 relative to RoundPipe given the same fitted duration model. Trained only on dense graphs, it lowers P99 on an unseen Qwen3-235B-A22B mixture-of-experts workload by 13.7% relative to LAER-MoE zero-shot and by 20.5% after 50 adaptation steps. The code is available at https://anonymous.4open.science/r/TROPOS.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Communication-Computation Overlap, Tail-Latency Optimization, Risk-Aware Scheduling, Amortized Optimization, Distributed Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z9qJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TROPOS, a risk-aware scheduler for distributed training that minimizes the CVaR of stochastic iteration completion times rather than relying on deterministic duration estimates. It combines differentiable critical-path evaluation, contention-aware duration modeling, importance sampling, and a heterogeneous graph attention network. Experiments on Qwen3 models report substantial reductions in P99 latency compared with RoundPipe, along with promising transfer results on an unseen MoE workload.

Critical strengths
  1. The paper introduces a risk-aware perspective on distributed training scheduling, supported by theoretical analysis of the limitations of deterministic objectives.

  2. The experimental evaluation is reasonably comprehensive, including comparisons with relevant schedulers, controlled ablations, and transfer experiments. The reported improvements in P99 latency are substantial, particularly for larger models.

Critical weaknesses
  1. The mathematical validity of the importance sampling formulation requires clarification. The exponentially tilted distribution in Eq. (12) may not be normalizable under the unbounded Log-normal–Pareto model in Eq. (11). The authors should clarify whether truncation or other approximations are used and justify their effects.

  2. The advantages of CVaR optimization are workload-dependent. Table 3 shows that the mean-based variant achieves better P99 latency on smaller models. Further analysis would help establish when risk-aware optimization provides consistent benefits.

  3. The zero-shot transfer mechanism is insufficiently explained. The scheduler uses node-type-specific projections but is trained only on dense graphs. It remains unclear how the model handles previously unseen MoE-specific node types, which weakens the evidence supporting its claimed generalization capability.

Questions for authors
  1. Sampling Validity (Eqs. 11–13): The exponential tilting in Eq. (12) may not be normalizable under the unbounded Log-normal–Pareto distribution in Eq. (11). Could the authors clarify how this proposal distribution is implemented and sampled in practice? If truncation or approximation is applied, how does it affect the validity of the CVaR estimator?
  2. Acyclicity of Relaxed Scheduling (Eq. 10): The soft priority matrix allows weighted precedence relationships, while the max-plus recursion assumes an acyclic execution graph. How is acyclicity maintained during differentiable training, and how are potentially conflicting precedence relationships handled?
  3. In Section 4.3, operations sharing GPU, memory, or communication resources may experience correlated delays. Does the proposed duration model explicitly capture these dependencies, or are residuals sampled independently? How might the modeling assumptions affect the accuracy of tail-risk estimation?
  4. In Section 5.4, the scheduler uses node-type-specific projections but is trained only on dense graphs. How are the parameters associated with previously unseen MoE-specific node types handled during zero-shot transfer? Are these representations shared, initialized separately, or mapped to existing node types?
  5. In Section 5 and Appendix B, does the author use a portion of the traces to fit the model and then use new, independent traces to validate the results? Additionally, since P99 latency is a key metric, how many iterations are measured per run, and how are the results aggregated across the three runs?
  6. Effectiveness of CVaR Optimization: Table 3 shows that CVaR optimization outperforms mean-based optimization on larger models but not on smaller ones. Could the authors clarify what factors determine this performance difference? Additionally, since the sensitivity analysis of α is conducted only on Qwen3-1.7B, how was α = 0.2 selected for the larger models?
  7. In Figure 2, the theoretical reference is expressed in terms of the number of near-critical paths (n(z)), whereas Proposition 3 derives the bound using the total number of paths (N). Could the authors clarify the relationship between these quantities in the experimental construction?
  8. Minor concerns for presentation: (1) Text format for figure 2 is a little bit small, (2) The sign convention for ΔP99 differs between Tables 1 and 3, which may cause confusion. A consistent convention or an exlicit definition in the table captions would improve clarity.
1 comment

We answer the eight questions in order, each with the measurement that settles it; all are incorporated in the revision. Changes are marked in blue in the new PDF.

Q1 (and W1) — tilted proposal. The reviewer is right that Eq. (11) admits no exponential moment: for every η>0\eta>0 and κ>2\kappa>2, E[eηϵ]≥eηxπ(xm/x)κ→∞\mathbb{E}[e^{\eta\epsilon}]\ge e^{\eta x}\pi(x_m/x)^{\kappa}\to\infty. Section 3.1 now bounds the residual, 0≤ϵv≤Ev0\le\epsilon_v\le E_v, so dv(z)=cv(z)+ϵvd_v(z)=c_v(z)+\epsilon_v is bounded for every schedule. Then 1≤Zη≤eηM1\le Z_\eta\le e^{\eta M}, and the proposal is sampled exactly by rejection on ϵ\epsilon with acceptance probability eη(S(ϵ)−M)e^{\eta(S(\epsilon)-M)} (Proposition 4, Algorithm 2). Truncation costs at most α−1E[T1ϵ≰E]\alpha^{-1}\mathbb{E}[T\mathbf{1}\\{\epsilon\not\le E\\}] in CVaR; Pr⁡(ϵv>Ev)=3.3×10−4\Pr(\epsilon_v>E_v)=3.3\times10^{-4} per operation at n=3000n=3000. On a fixed schedule at K=32K=32, bias is 1.0% and variance 38% lower than naive sampling (Table 15).

Q2 — acyclicity. The soft matrix never builds the graph. In the forward pass, list scheduling on (sv)(s_v) yields a topological order of (V,E)(V,E) for any scores, and EordE_{\mathrm{ord}} links consecutive operations of each resource along it; all edges point forward in one total order, so GzG_z is acyclic for every parameter value, not merely at convergence. Conflicting precedences are excluded by construction: an operation is not eligible before its data predecessors, so a data edge overrides a contradicting priority, and scalar scores are transitive, so two priorities cannot conflict ([Πz]vu+[Πz]uv=Av⊤Au[\Pi_z]_{vu}+[\Pi_z]_{uv}=A_v^{\top}A_u). Πz\Pi_z carries only the gradient, via oˉvu=ovu+[Πz]vu−sg([Πz]vu)\bar o_{vu}=o_{vu}+[\Pi_z]_{vu}-\mathrm{sg}([\Pi_z]_{vu}).

Q3 — correlated delays. Residuals are drawn independently given the schedule, so dependence between resource-sharing operations enters only through the contention term. We measure what this costs (Table 14, third row of each block): drawing the residuals of resource-sharing operations jointly at unchanged marginals lowers the P99 shortfall to 6.6% (1.7B) and 9.1% (14B). The sign is not uniform — dependence along a path widens the tail of TT, dependence across parallel paths narrows the tail of their maximum — and the first dominates here. Every policy is trained under the same independence, so the bias is common to TROPOS and RP+Var, and CVaR's worst-case reweighting by 1/α1/\alpha (ρα=sup⁡EQ[T]:dQ/dP≤1/α\rho_\alpha=\sup\\{\mathbb{E}_Q[T]:\mathrm{d}Q/\mathrm{d}P\le1/\alpha\\}) hedges part of the residual tail mass.

Q4 (and W3) — unseen MoE node types. There are none. Gate and expert operations are typed compute and all-to-all dispatch/combine communication, the two types learned on dense graphs; routing features (expert identity, expected load, ψ(h)\psi(h)) are zero on dense graphs and masked to zero at zero-shot. No parameter is new or re-mapped — what is new is connectivity (dispatch →\to experts →\to combine), which message passing handles. The 50 adaptation steps of Table 2 unmask the routing features.

Q5 — independent traces and aggregation. Yes. Per configuration the duration model is fitted on one 3000-iteration run; every reported prediction error is on a second, independent run (Tables 13–14). An end-to-end run has 5 warm-up and 3000 measured iterations; P99 is computed within each run, and the reported value is the mean over three runs, with a 95% interval over those three runs. Per-run logs in Table 6.

Q6 (and W2) — when CVaR pays, and how α\alpha was chosen. Table 3 confounds model size with contention; Table 21 does not. With model, batch and budget fixed and only PCIe load varied, the risk objective's P99 gain over the mean objective is −2.3-2.3\\% at 26.8% utilisation (95% interval [−5.7[-5.7\\%,1.1\\%]), 10.6% at 45.6% and 13.4% at 67.2% on 1.7B, and 6.8%, 15.0%, 16.2% on 14B. The crossover is the low-contention end of one curve, not a property of small models: at 45.6% the 1.7B model gains 10.6%, more than the 14B model's 6.8% with no extra load. α=0.2\alpha=0.2 was selected on 1.7B alone (Figure 5a) and then fixed for every other workload and platform, including ones it never helped to tune, so the large-model gains are not from per-model tuning.

Q7 — n(z)n(z) versus NN. In the construction of Figure 2 all NN paths have equal expected length, hence all are near-critical and n(z)=Nn(z)=N; the figure now uses NN. In general the relevant count is the slack-weighted Nλ=∑pe−λgp∈[1,N]N_\lambda=\sum_p e^{-\lambda g_p}\in[1,N] of Proposition 3, which equals NN exactly in this construction and degenerates to 1 when one path is slack.

Q8 — presentation. Figure 2 is regenerated with larger fonts and the NN axis. Every latency table defines Δ\DeltaP99 in its caption, and Tables 1, 3, 19, 20 and 23 share one convention: Δ=(row−reference)/reference\Delta=(\text{row}-\text{reference})/\text{reference}, negative meaning lower than the reference; Table 21 reports a reduction and says so.

Reviewer_A8MwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TROPOS is a scheduler for distributed training that optimizes the CVaR of iteration time to reduce tail latency. It combines contention modeling, heavy-tailed operation durations, a smoothed critical-path calculation, and a graph neural network to generate schedules. The authors evaluate it on several Qwen3 models using eight RTX 4090 GPUs, reporting improvements in latency and throughput, as well as transfer to an MoE workload.

Critical strengths

The paper addresses a practical problem: variation in iteration time during distributed training. Comparing scheduling policies within the same execution stack and including TROPOS-Mean are sensible choices for testing whether the risk objective itself provides a benefit.

Critical weaknesses
  1. The importance sampling distribution is not valid as defined. Equation (11) uses an untruncated mixture of log-normal and Pareto residuals, while Eq. (12) applies a positive exponential tilt to favor slower samples. Neither distribution has finite positive exponential moments. Since the completion time is at least as large as any individual operation duration, the normalizing constant in Eq. (12) is infinite. Setting the Pareto shape parameter to (\kappa>2) ensures finite variance, but does not fix this problem. Self-normalized weights do not resolve it either. The issue is not that the normalizing constant is difficult to compute, but that it is infinite. Neither the main text nor the appendix describes truncation or an alternative proposal distribution. It is therefore unclear how Eq. (13), Algorithm 1, and the experiments in Table 11 were actually implemented.

  2. Key experimental numbers do not agree with the stated definitions. Table 10 provides the clearest example. The authors state that Point estimate assigns one fixed duration to each operation and underestimates mean iteration time and P99 by 8.7% and 18.9%, respectively. For a fixed schedule, this deterministic model produces a single completion time, so its predicted mean and P99 should be identical. Yet the reported numbers imply a mean prediction of (21.000\times(1-8.7%)=19.173) s and a P99 prediction of (29.862\times(1-18.9%)=24.218) s. These are substantially different. If the errors use different samples or aggregation procedures, this needs to be explained. Table 1 raises a related question. For Qwen3-8B, multiplying mean latency by throughput gives approximately 975,000 tokens per step for RP and 1,042,000 for TROPOS, a difference of about 6.9%. If both methods process the same number of tokens per step and throughput is calculated as total tokens divided by total elapsed time, these values should agree. The paper does not provide enough measurement detail to explain the difference, so the reported throughput gain is also difficult to verify.

Questions for authors
  1. How does the implementation sample from the distribution in Eq. (12)? Does it use truncation or a different proposal distribution? Please provide the actual sampling procedure and weight formula, and clarify which implementation was used for Table 11.
  2. What are the original predicted values for Point estimate in Table 10? Why do the reported mean and P99 errors imply two different predictions?
  3. How is throughput in Table 1 calculated? Please specify the number of tokens per step, the timing scope, and how results are aggregated across runs, and explain the discrepancy between mean latency and throughput for Qwen3-8B.
2 comments

We agree with all three findings; each was a real defect. The support of the duration model was implicit, Table 10 mixed two protocols, and Table 1 divided tokens by two timing windows (revision: Tables 14, 15, 6). Changes are marked in blue in the new PDF.

Q1 — the proposal is normalisable and is sampled exactly. The reviewer is right: under Eq. (11), for every η>0\eta>0 and κ>2\kappa>2, E[eηϵ]≥eηxπ(xm/x)κ→∞\mathbb{E}[e^{\eta\epsilon}]\ge e^{\eta x}\pi(x_m/x)^{\kappa}\to\infty, so Zη=∞Z_\eta=\infty and Eq. (12) is not a distribution. Section 3.1 now bounds the residual: 0≤ϵv≤Ev0\le\epsilon_v\le E_v, so dv(z)=cv(z)+ϵv≤cv(z)+Evd_v(z)=c_v(z)+\epsilon_v\le c_v(z)+E_v for every schedule. This is the fix: a bound on dvd_v would have to move with cv(z)c_v(z), and for cv(z)≥Dvc_v(z)\ge D_v the conditioning event has probability zero. With S(ϵ)=Ez∼πθ[T~ε(z,c(z)+ϵ)]S(\epsilon)=\mathbb{E}_{z\sim\pi_\theta}[\widetilde T^\varepsilon(z,c(z)+\epsilon)], 1≤Zη≤eηM1\le Z_\eta\le e^{\eta M} via 0≤S(ϵ)≤S(E)=M0\le S(\epsilon)\le S(E)=M (Proposition 4). Sampling is exact rejection in residual space: accept ϵ∼pϕ\epsilon\sim p_\phi with probability eη(S(ϵ)−M)e^{\eta(S(\epsilon)-M)} and compose d=c(z)+ϵd=c(z)+\epsilon; the rate Zηe−ηM∈[e−ηM,1]Z_\eta e^{-\eta M}\in[e^{-\eta M},1] is bounded away from zero, and ωk=Ke−ηS(ϵ(k))/∑je−ηS(ϵ(j))\omega_k=Ke^{-\eta S(\epsilon^{(k)})}/\sum_j e^{-\eta S(\epsilon^{(j)})} since ZηZ_\eta cancels. Truncation costs at most α−1E[T,1ϵ≰E]\alpha^{-1}\mathbb{E}[T\\,\mathbf{1}\\{\epsilon\not\le E\\}] in CVaR. Per op., Pr⁡(ϵv>Ev)=3.3⋅10−4\Pr(\epsilon_v>E_v)=3.3\cdot10^{-4} and Pr⁡(ϵ≰E)≤∣V∣/(n+1)\Pr(\epsilon\not\le E)\le|V|/(n+1) by union bound: this bounds the expectation without making it negligible, so the revision states truncation as a modelling assumption. Algorithm 2 is the sampler used throughout; its earlier variance columns used changing schedules, so Table 15 fixes the schedule (reference 23.032 s):

Naive MC from pϕp_\phi

KKBiasVar.RMSE (s)
8−0.21%1.8271.352
32+0.09%0.4650.682
128−0.09%0.1200.346

Tilted qηq_\eta (Algorithm 2)

KKBiasVar.RMSE (s)ESS/KK
8+2.91%0.9281.1730.888
32−1.02%0.2860.5840.895
128−0.76%0.0940.3530.888

At K=32K=32: variance −38-38\\%, RMSE −14-14\\%, bias 1.0%, ESS =89=89\\% of KK. Limits also hold: gradient variance is 8–10% higher, the RMSE gain vanishes at K=128K=128, and one estimate costs 1.7–4.2$\times$ more. Draws and weights are fixed in the backward pass, so gradients flow only through the reparameterised d(k)=c(zθ)+ϵ(k)d^{(k)}=c(z_\theta)+\epsilon^{(k)} and T~ε\widetilde T^\varepsilon; differentiating the weights would add a −η,Cov(fθ,∂θSθ)-\eta\\,\mathrm{Cov}(f_\theta,\partial_\theta S_\theta) term.

Q2 — the two errors of Table 10 describe two objects. Under a frozen schedule a point estimate gives one number, so predicted mean equals predicted P99: 21.308 s on 1.7B, 37.183 s on 14B. The 8.7%/18.9% are not that number: the policy re-plans every iteration on the refreshed ϕ\phi, so a run holds 3000 schedules and 3000 point predictions, and the two figures are the shortfalls of the mean and the 99th percentile of that sequence. Table 14 removes the confound (frozen schedule, fit on run 1, tested on an independent run):

ModelDuration modelPredicted Mean / P99 (s)Measured Mean / P99 (s)
1.7Bpoint estimate21.30823.119 / 25.697 (−7.8-7.8\\% / −17.1-17.1\\%)
1.7Bfull, independent residuals22.291 / 23.72923.119 / 25.697 (−3.6-3.6\\% / −7.7-7.7\\%)
1.7Bfull, joint residuals22.388 / 23.99623.119 / 25.697 (−3.2-3.2\\% / −6.6-6.6\\%)
14Bpoint estimate37.18341.732 / 60.296 (−10.9-10.9\\% / −38.3-38.3\\%)
14Bfull, independent residuals38.032 / 51.95741.732 / 60.296 (−8.9-8.9\\% / −13.8-13.8\\%)
14Bfull, joint residuals39.137 / 54.79241.732 / 60.296 (−6.2-6.2\\% / −9.1-9.1\\%)

Joint draws reproduce dependence among resource-sharing operations at unchanged marginals and cut the P99 shortfall to 6.6% and 9.1%.

Q3 — Table 1 was inconsistent, and we corrected it. Mean latency averaged per-iteration times, while throughput divided tokens by wall clock that included host stalls (concentrated in the 8B RP runs): exactly the 975k-versus-1,042k the reviewer computed. The revision uses one window, with throughput pooled over runs, Thr=∑rNrB/∑rTr=B/Mean‾\text{Thr}=\sum_r N_rB/\sum_r T_r=B/\overline{\text{Mean}}. For Qwen3-8B (B=1,048,576B=1{,}048{,}576): RP 75.73175.731 s, 13,84613{,}846 tok/s; RP+Var 63.61063.610, 16,48516{,}485; TROPOS-Mean 58.65758.657, 17,87617{,}876; TROPOS 59.83059.830, 17,52617{,}526. RP's 8B throughput rises from 12,874 to 13,846 tok/s, so the 8B gain falls from 35.3% to 26.6%; P99 is unchanged, and over five models the gain over RP is 16.3–42.9%.

Reviewer_A8MwOpenReview ↗

I thank the authors for acknowledging the three defects and for providing corrections. However, these responses do not resolve my concerns for the version under review.

I thank the authors for acknowledging the three defects and for providing corrections. However, these responses do not resolve my concerns for the version under review.

Q1: The authors now agree that Eq. (12) was not a distribution under Eq. (11) and introduce a bounded residual plus rejection sampling. This is a substantive post hoc change to the modelling assumption, not a clarification of what was originally implemented. The original paper did not describe truncation or an alternative proposal, so the validity of Eq. (13), Algorithm 1, and the Table 11 results as reported remains unclear. The new bounds and sampler need to be tied to the actual experiments; if the original runs did not use this corrected procedure, the affected results must be rerun rather than reinterpreted.

Q2: The authors concede that Table 10 mixed two protocols. That concession confirms my concern: the reported 8.7%/18.9% errors cannot be interpreted under the definitions given in the text. The new Table 14 does not retroactively validate the original Table 10; it shows that under a frozen schedule the point-estimate prediction is a single number, which is precisely why the original two different implied predictions were inconsistent. The revised numbers may be useful, but they do not repair the reliability of the originally reported experimental claims.

Q3: The authors also concede that Table 1 divided tokens by two different timing windows and that the Qwen3-8B throughput gain falls from 35.3% to 26.6% after correction. This is an acknowledgment that a key quantitative result in the paper was inconsistent. Correcting it is necessary, but it materially changes the reported gains and raises questions about the other timing-based numbers.

Overall, the authors have identified and acknowledged real defects, but the corrections are largely new material that does not establish the soundness of the reviewed submission. I therefore retain my original assessment.

Reviewer_jjS4OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces TROPOS, a risk-aware computation–communication overlap scheduler for distributed training. The key idea is to treat iteration time as a stochastic max-plus completion time and optimize a tail risk statistic (CVaR) rather than a deterministic point estimate, thereby addressing the underestimation inherent in prior deterministic objectives. TROPOS pairs a theoretically grounded formulation with a practical amortized solver: a log-sum-exp relaxation of the max-plus recursion with a one-sided error bound, a contention-aware heavy-tailed duration model, tail-focused importance sampling, and a heterogeneous graph attention policy that emits a schedule in a single forward pass. On 8 RTX 4090 for Qwen3 dense models (1.7B-32B) and one MoE workload, TROPOS reduces P99 latency by 21-36% and improves throughput by 14-48% versus RoundPipe, with additional gains over a strengthened RoundPipe baseline that uses the same fitted duration model.

Critical strengths

I think the real strength of this paper is not that it builds yet another scheduler. It explains the disease clearly: iteration time is the maximum over several competing paths, so ranking plans by average duration is optimistic by construction, and the more aggressive the overlap, the worse that bias gets. The unification of RoundPipe style methods as point estimate special cases gives the change of objective a proper basis. The method is also closed loop: smoothed max, heavy tailed residual duration model, tail focused sampling, and a graph policy that emits a schedule in one forward pass, not just one end to end number. The most convincing experiments are RP+Var and TROPOS-Mean. RP+Var shows the gain is not only from better duration modeling, and TROPOS-Mean shows a mean objective is fine on small models, while the CVaR objective only pays off around 14B and 32B.

Critical weaknesses
  1. The main limitation is hardware coverage. The paper only evaluates one eight GPU RTX 4090 setup, so the gains may be specific to this PCIe configuration. Other GPUs or interconnects could behave differently, but no such test is provided.
  2. The training setup is not clear enough to judge generality. The paper does not report which tasks were used for training, how many there were, or which tasks were held out. As a result, the gains may depend on the chosen tasks rather than reflect a general scheduling rule.
Questions for authors

See Weaknesses.

2 comments

W1: one 8×8\timesRTX 4090 platform; W2: training and held-out workloads unclear. We add a second platform without retraining, a held-out protocol, and a workload outside language modeling. The P99 gains survive all three: −20.9-20.9\\%/-37.7\\% on the A10 cluster, −38.1-38.1\\%/-38.8\\% on the two held-out dense models, −14.7-14.7\\% on image classification.

W1 — second platform, no retraining. Two nodes of four A10 GPUs (PCIe within a node, 32 Gbps network link between nodes) replace both the GPU model and the interconnect. The policy trained on the 4090 node is transferred with no parameter update; only the duration model is refitted, and it is given to every method that uses one, so the comparison is again within one execution stack.

ModelMethodMean (s)P99 (s)95% CI of P99Δ\DeltaP99
Qwen3-1.7BRP20.23125.322[22.051, 28.593]--
RP+Var19.03122.311[19.606, 25.016]−11.9-11.9\\%
TROPOS-Mean18.93223.727[20.657, 26.797]−6.3-6.3\\%
TROPOS18.02220.035[17.597, 22.473]−20.9-20.9\\%
Qwen3-14BRP70.25895.877[93.294, 98.460]--
RP+Var54.92370.231[67.751, 72.711]−26.7-26.7\\%
TROPOS-Mean52.29071.394[68.853, 73.935]−25.5-25.5\\%
TROPOS45.33059.752[57.239, 62.265]−37.7-37.7\\%

A10 cluster, 3 runs ×\times 3000 iterations. Δ\DeltaP99 =(P99row−P99RP)/P99RP=(\text{P99}_{\text{row}}-\text{P99}_{\text{RP}})/\text{P99}_{\text{RP}}. On the 4090 node the same two cells read −21.0-21.0\\% and −35.7-35.7\\% (Table 1).

TROPOS attains the lowest mean and P99 on both workloads; on 14B its interval is disjoint from RP, and on 1.7B the two overlap by 0.4 s. We do not claim the full ranking is identical, since TROPOS-Mean and RP+Var exchange the middle two places. The gains do not track a PCIe topology: −20.9-20.9\\% against −21.0-21.0\\% at 1.7B and −37.7-37.7\\% against −35.7-35.7\\% at 14B.

Why the mechanism transfers. The paper does not claim that PCIe-4.0 contention is special; it claims that point-estimate objectives are biased. Iteration time is a maximum of affine functions of the duration vector, so T(z,Ed)≤E[T(z,d)]T(z,\mathbb{E}d)\le\mathbb{E}[T(z,d)] by Jensen's inequality, and the gap grows with the perturbation scale σ\sigma and with the slack-weighted path count NλN_\lambda (Proposition 3). Any platform on which durations vary creates the gap. We chose the 4090 node because it maximises it (CPU offload over PCIe across two NUMA domains), not because the method requires it; the A10 cluster adds a network hop that the 4090 node does not have, and the gains do not shrink. Within one platform, Table 21 isolates the same effect: with the model, batch and budget fixed and only PCIe utilisation varied, the risk objective's P99 gain goes from −2.3-2.3\\% (27% utilisation) to +13.4+13.4\\% (67%) on 1.7B and from 6.8% to 16.2% on 14B.

W2 — what trains on what, and what is held out. The workload family is the model configuration, not a downstream task: with synthetic tokens, iteration time is fixed by tensor shapes and by the schedule, not by token values (Appendix B.3), so the unit of the study is the task graph. The protocol is summarised below and tabulated in Table 5.

QuantityData used
Policy θ\thetaTask graphs of Qwen3-1.7B, 4B and 8B only (sequence length 2048).
Duration model ϕ\phiFitted per workload and platform on one 3000-iteration profiling run; every prediction error reported is measured on a second, independent run of 3000 iterations.
α\alpha, KK, temperaturesSelected on Qwen3-1.7B only, then reused unchanged.
Held outQwen3-14B and 32B (independent test traces), Qwen3-235B-A22B MoE, the A10 platform, Qwen3-VL-8B on CIFAR-10.

On the held-out dense models the re-measured P99 reduction relative to RP is 38.1% (14B) and 38.8% (32B), with 95% intervals disjoint from those of RP and RP+Var (Table 19); neither model contributed to policy training or to hyperparameter selection. For a non-text modality, Qwen3-VL-8B on CIFAR-10 (60,000 images, 10 classes) lowers P99 from 72.620 s to 61.921 s (−14.7-14.7\\%) and the mean by 14.2% (Table 22).

Thanks for the detailed response and the new experiments.

On W1, the A10 zero-shot results (-20.9% and -37.7% P99 without retraining) and the PCIe utilization test you reported (where P99 gains grow from -2.3% to +13.4% on 1.7B and 6.8% to 16.2% on 14B) look solid. For W2, training only on 1.7B–8B graphs while holding out 14B/32B, MoE, and the A10 setup makes the split clear, and the extra CIFAR-10 test on Qwen3-VL-8B shows it also works outside text models.

My two main concerns are mostly resolved. However, given the issues raised by reviewers A8Mw and Z9qJ, it looks like your new experiments may also have a problem.

I have one quick question about the A10 cluster numbers: based on the official specs, the NVIDIA A10 (2021 https://www.nvidia.com/en-us/data-center/products/a10-gpu/) has much lower compute and memory bandwidth than the RTX 4090 (~31 vs ~83 FP32 TFLOPS, 600 GB/s vs 1 TB/s), and the 2-node setup also adds a 32 Gbps inter-node link. Yet for both RP and TROPOS, the absolute mean and P99 iteration times on the A10 cluster are actually lower than on 8x RTX 4090 (for example, 14B TROPOS P99 is 59.75s on A10 vs 64.93s on 4090). Could the authors clarify this? It seems odd that stronger GPUs with lower communication overhead run slower.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.