- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a method for switching parallelism based on the generation status, using the number of unfinished requests as the only online decision metric to determine whether to switch from DP-dominated to TP-dominated parallelism. It achieves speedups by adapting to substantial variation in rollout generation lengths and the resulting stragglers, with reported P99/P50 ratios of 2.74–4.27 after excluding truncated samples.
Critical strengths
- Switching parallelism during an ongoing rollout to adapt to changing concurrency is a useful systems idea for mitigating straggler-related inefficiency.
Critical weaknesses
-
Insufficient specification and empirical support for the switching policy. The paper does not adequately explain how the “profiled tail workload,” throughput measurements, and switching costs are combined to obtain the unfinished-request threshold. In particular, it does not provide sufficient empirical evidence that the preferred configuration can be characterized by a single threshold after accounting for switching costs. This limits my ability to assess the justification and reproducibility of a central component of the method.
-
Insufficient comparison with request rebalancing and evaluation of broader parallelism choices. The paper lacks a comparison against a fixed DP8/TP1 baseline with request/KV-cache rebalancing, such as this implementation, making it difficult to assess the additional benefit of topology switching over improved load balancing. The evaluation is also limited to the predefined DP8/TP1 → DP2/TP4 switch and does not establish whether the proposed policy can select and switch among a broader set of parallelism configurations. This leaves the generality of the selection and switching policy insufficiently validated.
Questions for authors
Q1: Does denote the generated output length of still-active (unfinished) requests? I find the phrase “remaining output length” very confusing, since it would be difficult to measure, as argued in the paper. I would interpret it as the generated output length of the remaining (i.e., unfinished) requests.
Q2: Based on Q1, why is a threshold justified? What empirical evidence supports the monotonicity of the preferred parallelism configuration with respect to after accounting for switching costs? Please define “profiled tail workload” precisely and provide the corresponding workload information, switching costs, and throughput comparisons at different values of . Please show how these measurements lead to the threshold , and clarify how the resulting empirical boundary is consistent with the use of in the threshold definition and the runtime condition .
Q3: How are throughput and switching costs measured or estimated? Please specify the profiling protocol.
Q4: Please address the following two aspects of the evaluation:
-
Comparison with request rebalancing. Does the static DP8/TP1 baseline already include request/KV-cache rebalancing across DP replicas? If so, please describe the policy and configuration. Otherwise, please compare AdaRollout against a fixed DP8/TP1 baseline with such rebalancing enabled. Please use the same hardware and workload, keep other runtime optimizations comparable, and report total rollout completion time.
-
Selection and switching among more parallelism configurations. Does the proposed profiling- and cost-based policy support selection and switching among more than two parallelism configurations? Please evaluate this capability with a broader candidate set, for example DP8/TP1, DP4/TP2, DP2/TP4, and DP1/TP8 under the same device budget, and explain how the policy determines which configuration to use and when to switch. Please report the total rollout completion time, and compare the results with the reported two-configuration policy (DP8/TP1 → DP2/TP4). If the current method is restricted to two predefined configurations, please clarify this limitation.
Please upload anonymized profiling data so that I can assess the throughput and switching-cost measurements and reproduce the threshold-selection procedure, including the derivation of . These materials are necessary for me to assess the empirical justification and reproducibility of the proposed policy. Without adequate supporting evidence and clarification, I remain concerned about the transparency of the method and the reproducibility of its claimed benefits.