- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents FastCI, a framework that improves the efficiency of continuous integration (CI) for LLM training frameworks, where many tests are GPU-intensive end-to-end training or evaluation runs.
FastCI first samples runtime stack traces of previous CI runs to build a function-level runtime evidence graph and selects only the tests that executed the modified functions (with expansion rules when evidence is missing).
Then the context-aware test pruning in FastCI removes tests that execute the modified function in the same calling context.
During execution, FastCI runs tests with higher historical failure risk first; and reduces workload dimensions (e.g., model size, logging) that are outside the validation scope of each test.
On the CI workload of CompanyX, FastCI is reported to reduce CI latency by 77.5% and GPU resource usage by 63.9% compared with the currently deployed path-based test selection.
Critical strengths
-
This paper focuses on a valuable engineering problem in LLM infrastructure. Reducing the latency and GPU cost of such CI pipelines has clear practical value for the teams that build and maintain these frameworks.
-
The workflow design of FastCI is clear and well presented. Figure 2 gives a clean overview of the whole pipeline, and the technical sections are supported by necessary concrete cases (e.g., Figure 3 and Figure 4). These examples make the design of each module easy to follow.
Critical weaknesses
-
Scope mismatch with ICLR. FastCI is essentially a software engineering/systems contribution: it combines runtime-trace-based test selection, context-aware pruning, history-based test prioritization, and workload reduction into a CI workflow. It is more like an implementation and combination of existing techniques and does not propose a new algorithm, method, or theory. It would be judged more appropriately at a software engineering or systems venue.
-
Unverifiable and possibly non-generalizable evaluation. All results come from a single internal, unnamed LLM training framework (
CompanyX). The main text does not report the experimental environment, the number of evaluated MRs and tests, how the metrics are computed, or any variance across runs, and there is no component-wise ablation. As a result, the headline numbers (77.5% lower CI latency and 63.9% lower GPU usage) can neither be verified nor attributed to specific modules, and it is unclear whether they transfer to other frameworks. This weakness directly limits the significance and reproducibility of this paper.
Questions for authors
- Scope Issue. CI is an important component of software development and maintenance, and improving CI is an active topic in software engineering, which is also reflected by the fact that most references of this paper come from software engineering journals and conferences (e.g., ICSE, ISSTA, FSE). However, the core idea of FastCI, i.e., collecting runtime evidence to select only the related tests and scheduling them based on execution history, is closer to an engineering workflow than to a new method or theory that the ICLR community focuses on. Learning is only marginally involved (e.g., the risk score in Appendix A is a hand-designed weighted failure rate). I suggest the authors clarify the relevance of this work to the ICLR scope; otherwise, a software engineering or systems conference (e.g., FSE/ICSE/ASE/ISSTA) may be a more suitable target.
- What is the internal training framework of
CompanyX, and how representative is it of mainstream frameworks such as DeepSpeed? How many MRs and tests are evaluated, on what GPU resource pool, and over how many runs? Are the reported reductions stable across runs? - Which module of FastCI addresses the second limitation stated in the introduction, i.e., that manually maintaining the file-to-test mapping is error-prone?
- How much of the improvement of FastCI over FastCI-select (18.3% lower GPU usage and 28.6% lower CI latency) comes from risk-aware scheduling, and how much from workload-optimized execution?
- Suggestion: Evaluate FastCI on open-source LLM training frameworks (e.g., DeepSpeed or Megatron-LM) with public CI histories, so that the results can be reproduced and compared by others.
5 comments
Thanks for authors' detailed responses and the additional experimental details/results. My earlier concerns are mainly about the scope and the evaluation details, and the clarifications and new experiments address most of them. I have several remaining questions:
- Compared with existing runtime tracing methods, what is the essential technical novelty of FastCI? Please do not restate the functions of each module. Here, the authors should point out which specific techniques cannot be achieved by existing work, or what new concept FastCI introduces. As mentioned in my review, this paper currently reads like a combination of existing techniques, which is an incremental engineering contribution.
- What is the
Codex Plannerbaseline in the Megatron-LM experiment? I could not find this term in the main text of the paper.
In addition, since the authors can revise the paper during the ICLR discussion period, I suggest adding these clarifications, the experimental details, and the new results to the paper (even if some of the results are placed in the appendix). Please also indicate the specific line numbers of these revisions in the response. This will help the reviewers reconsider their scores.
Thanks for your reply. Here are our answers for your remaining questions and we have provided the revised manuscript now.
Q1: What is new beyond existing runtime tracing and test selection?
FastCI contributes an end-to-end CI design that uses incomplete runtime evidence to select and prune distributed training jobs for each code change and optimizes their execution through risk-aware scheduling and workload reduction. To our knowledge, FastCI is the first system to demonstrate end-to-end optimization of GPU-intensive CI in production for a large-scale distributed LLM training framework.
Despite extensive prior work on runtime tracing and regression test selection, many GPU CI workflows in practice still rely on predefined test scopes or file-level rules. Megatron-LM uses a predefined L0 tier in its default PR workflow, vLLM has used declared source-file dependencies to select CI jobs, and our previous production CI relied on manually maintained file-to-test mappings. Even NameRTS, a state-of-the-art regression test selection method, retains less modified-code coverage at higher GPU cost than FastCI-select in our Megatron-LM replay (Appendix E). Megatron-LM PR workflow · vLLM CI pipeline generator
FastCI extends prior CI techniques with new designs grounded in observations from production training workloads. For example, sampling can miss short-lived functions, so we expand through related methods and callers to recover candidate tests. With pruning enabled, expansion was triggered in 85.1% of MRs and added tests in 46.8% (Appendix C.4). In scheduling, change-specific failure history can be sparse, and recent instability captures recurring failures. A simple score combining both signals reduced the median GPU cost of the ordered test prefix through the first failing test from 12.6 to 2.3 GPU-hours compared with COLEMAN (§5.3).
Together, these designs form a workflow that optimizes which tests run, when they run, and how much GPU work they do. FastCI has demonstrated its effectiveness at production scale by saving about 724,000 GPU-hours (Appendix D.1).
Q2: What is Codex Planner in the Megatron-LM evaluation?
“Codex Planner” is the LLM reasoning baseline. For each commit, GPT-6.1 Sol receives the changed paths, a patch excerpt, and the IDs and YAML definitions of the 47 candidate functional tests. It returns the IDs of tests likely to exercise the changed behavior. The model receives no runtime traces, coverage data, or test outcomes. The CPU replay scores the saved selections without another model call (Appendices E and F).
Thanks for your feedback and questions. Here is our response for each question:
Q1: Scope and fit for ICLR
FastCI fits ICLR’s scope on infrastructure for large-scale learning. In distributed LLM training, precision and parallelism determine which numerical and communication behaviors a test must validate. TorchTitan checks loss convergence across parallelism configurations, and SpareTrain makes reliability checks during training cheaper. FastCI addresses the cost of repeating these kinds of checks as the training framework changes. The Megatron-LM replay shows that this approach also works on public training CI (Q5). ICLR 2027 Call for Papers, TorchTitan (ICLR 2025, Appendix B.10), SpareTrain (ICLR 2026)
FastCI contributes a change-specific method for selecting and pruning tests that share training code. It recovers candidate tests through related methods and callers when sampling misses a modified function, then uses call contexts and validation metadata to identify redundant checks. Workload optimization reduces GPU work while preserving the training settings each test validates. Appendix A’s risk score is a hand-designed but effective in production. In the scheduling replay (Section 5.3), FastCI brings the median first failure forward from 9 under COLEMAN, a learning-based prioritizer, to 4.
Q2: Framework, evaluation scale, and stability
Our internal distributed LLM training stack resembles Megatron-LM. Its CI checks parallelism, checkpoint and resume, and numerical consistency across training configurations. We assess transfer on 100 public commits from Megatron-LM using its functional test suite (Q5).
Figures 5 and 6 use two days of shadow CI mirroring online MRs, with a common budget of 16 machine slots and 16 GPUs per slot. Results include sampling overhead and exclude graph construction, which takes minutes on CPU. The selection replay covers 100 production MRs and 2,482 distinct modified entities.
Gains vary across MRs and frameworks. Figure 6 shows consistent latency reductions relative to path-based CI from typical to slower runs. The internal and Megatron-LM replays show substantial GPU savings while preserving most modified-code coverage (Q5).
Q3: Which module removes manual mapping
Evidence-guided test selection uses the runtime evidence graph to automatically build and update mappings from Python functions to the tests that execute them, replacing manual maintenance.
Manual mappings remain for native code and configuration files, with negligible effort against maintaining mappings for a large Python codebase.
Q4: Source of gains over FastCI-select
Both gains in Figure 5 come from workload-optimized execution. The end-to-end experiment schedules the same selected tests in different orders for FastCI and FastCI-select, and all remaining tests run after a failure. Early cancellation could yield further gains over FastCI-select.
Section 5.3 evaluates risk-aware scheduling separately. Figure 7 shows that FastCI reduces median first-failure rank by 63.6% and median GPU use before that failure by 89.2% relative to FIFO.
Q5: Evaluation on public frameworks
We evaluate FastCI on Megatron-LM's public functional CI by performing test selection on 100 consecutive commits from its main branch. We score selections against a fixed line-coverage reference from 47 functional tests. Recall and retention summarize the commits with affected tests, and GPU cost estimates use measured test runtimes. In this experiment, the GPU consumption of FastCI includes the sampling overhead.
FastCI-select retains 98.2% of modified-code coverage and achieves 64.1% affected-test recall with context-aware pruning enabled. Without pruning, these metrics rise to 99.2% and 82.7%, respectively. Based on measured test runtimes, FastCI can achieve an estimated 74.0% GPU saving relative to full CI.
The open-source release includes a native Megatron-LM adapter and a CPU replay that recomputes the table from the archived selection decisions. The repository provides the public commit list, functional test configurations, per-commit selections and line coverage, and measured test runtimes used in the cost estimates. Megatron-LM adapter and CPU replay