acceptodds
Under review as a conference paper at ICLR 2027

FastCI: Efficient GPU-Intensive CI for LLM Training Frameworks

Abstract

As large language models (LLMs) keep growing in size and complexity, their training frameworks evolve at a rapid pace as well. Therefore, continuous integration (CI) is critical for maintaining the quality and stability of these frameworks. However, unlike traditional software, CI for LLM training frameworks relies on GPU-intensive tests, which usually involve complete model training or evaluation. This leads CI itself to become a new bottleneck for fast-paced development. In this paper, we introduce FastCI, a framework that improves the efficiency of CI for LLM training frameworks. FastCI leverages runtime evidence to select affected tests and prune tests that execute changed code in equivalent contexts. Then FastCI prioritizes high-risk tests to expose potential failures earlier, and optimizes test workloads along dimensions outside the intended validation scope of each test. Evaluated on the CI workload of our LLM training framework, FastCI reduces the CI latency by 77.5% and the GPU resource usage by 63.9%, while improving the modified code coverage retention by 3.2%, compared with the currently deployed CI pipelines. FastCI has now been integrated into the CI pipelines of our LLM training framework at CompanyX.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
continuous integration, LLM training, software testing

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_z9SROpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FastCI, a framework that improves the efficiency of continuous integration (CI) for LLM training frameworks, where many tests are GPU-intensive end-to-end training or evaluation runs. FastCI first samples runtime stack traces of previous CI runs to build a function-level runtime evidence graph and selects only the tests that executed the modified functions (with expansion rules when evidence is missing). Then the context-aware test pruning in FastCI removes tests that execute the modified function in the same calling context. During execution, FastCI runs tests with higher historical failure risk first; and reduces workload dimensions (e.g., model size, logging) that are outside the validation scope of each test. On the CI workload of CompanyX, FastCI is reported to reduce CI latency by 77.5% and GPU resource usage by 63.9% compared with the currently deployed path-based test selection.

Critical strengths
  • This paper focuses on a valuable engineering problem in LLM infrastructure. Reducing the latency and GPU cost of such CI pipelines has clear practical value for the teams that build and maintain these frameworks.

  • The workflow design of FastCI is clear and well presented. Figure 2 gives a clean overview of the whole pipeline, and the technical sections are supported by necessary concrete cases (e.g., Figure 3 and Figure 4). These examples make the design of each module easy to follow.

Critical weaknesses
  • Scope mismatch with ICLR. FastCI is essentially a software engineering/systems contribution: it combines runtime-trace-based test selection, context-aware pruning, history-based test prioritization, and workload reduction into a CI workflow. It is more like an implementation and combination of existing techniques and does not propose a new algorithm, method, or theory. It would be judged more appropriately at a software engineering or systems venue.

  • Unverifiable and possibly non-generalizable evaluation. All results come from a single internal, unnamed LLM training framework (CompanyX). The main text does not report the experimental environment, the number of evaluated MRs and tests, how the metrics are computed, or any variance across runs, and there is no component-wise ablation. As a result, the headline numbers (77.5% lower CI latency and 63.9% lower GPU usage) can neither be verified nor attributed to specific modules, and it is unclear whether they transfer to other frameworks. This weakness directly limits the significance and reproducibility of this paper.

Questions for authors
  1. Scope Issue. CI is an important component of software development and maintenance, and improving CI is an active topic in software engineering, which is also reflected by the fact that most references of this paper come from software engineering journals and conferences (e.g., ICSE, ISSTA, FSE). However, the core idea of FastCI, i.e., collecting runtime evidence to select only the related tests and scheduling them based on execution history, is closer to an engineering workflow than to a new method or theory that the ICLR community focuses on. Learning is only marginally involved (e.g., the risk score in Appendix A is a hand-designed weighted failure rate). I suggest the authors clarify the relevance of this work to the ICLR scope; otherwise, a software engineering or systems conference (e.g., FSE/ICSE/ASE/ISSTA) may be a more suitable target.
  2. What is the internal training framework of CompanyX, and how representative is it of mainstream frameworks such as DeepSpeed? How many MRs and tests are evaluated, on what GPU resource pool, and over how many runs? Are the reported reductions stable across runs?
  3. Which module of FastCI addresses the second limitation stated in the introduction, i.e., that manually maintaining the file-to-test mapping is error-prone?
  4. How much of the improvement of FastCI over FastCI-select (18.3% lower GPU usage and 28.6% lower CI latency) comes from risk-aware scheduling, and how much from workload-optimized execution?
  5. Suggestion: Evaluate FastCI on open-source LLM training frameworks (e.g., DeepSpeed or Megatron-LM) with public CI histories, so that the results can be reproduced and compared by others.
5 comments

Thanks for your feedback and questions. Here is our response for each question:

Q1: Scope and fit for ICLR

FastCI fits ICLR’s scope on infrastructure for large-scale learning. In distributed LLM training, precision and parallelism determine which numerical and communication behaviors a test must validate. TorchTitan checks loss convergence across parallelism configurations, and SpareTrain makes reliability checks during training cheaper. FastCI addresses the cost of repeating these kinds of checks as the training framework changes. The Megatron-LM replay shows that this approach also works on public training CI (Q5). ICLR 2027 Call for Papers, TorchTitan (ICLR 2025, Appendix B.10), SpareTrain (ICLR 2026)

FastCI contributes a change-specific method for selecting and pruning tests that share training code. It recovers candidate tests through related methods and callers when sampling misses a modified function, then uses call contexts and validation metadata to identify redundant checks. Workload optimization reduces GPU work while preserving the training settings each test validates. Appendix A’s risk score is a hand-designed but effective in production. In the scheduling replay (Section 5.3), FastCI brings the median first failure forward from 9 under COLEMAN, a learning-based prioritizer, to 4.

Q2: Framework, evaluation scale, and stability

Our internal distributed LLM training stack resembles Megatron-LM. Its CI checks parallelism, checkpoint and resume, and numerical consistency across training configurations. We assess transfer on 100 public commits from Megatron-LM using its functional test suite (Q5).

Figures 5 and 6 use two days of shadow CI mirroring online MRs, with a common budget of 16 machine slots and 16 GPUs per slot. Results include sampling overhead and exclude graph construction, which takes minutes on CPU. The selection replay covers 100 production MRs and 2,482 distinct modified entities.

Gains vary across MRs and frameworks. Figure 6 shows consistent latency reductions relative to path-based CI from typical to slower runs. The internal and Megatron-LM replays show substantial GPU savings while preserving most modified-code coverage (Q5).

Q3: Which module removes manual mapping

Evidence-guided test selection uses the runtime evidence graph to automatically build and update mappings from Python functions to the tests that execute them, replacing manual maintenance.

Manual mappings remain for native code and configuration files, with negligible effort against maintaining mappings for a large Python codebase.

Q4: Source of gains over FastCI-select

Both gains in Figure 5 come from workload-optimized execution. The end-to-end experiment schedules the same selected tests in different orders for FastCI and FastCI-select, and all remaining tests run after a failure. Early cancellation could yield further gains over FastCI-select.

Section 5.3 evaluates risk-aware scheduling separately. Figure 7 shows that FastCI reduces median first-failure rank by 63.6% and median GPU use before that failure by 89.2% relative to FIFO.

Q5: Evaluation on public frameworks

We evaluate FastCI on Megatron-LM's public functional CI by performing test selection on 100 consecutive commits from its main branch. We score selections against a fixed line-coverage reference from 47 functional tests. Recall and retention summarize the commits with affected tests, and GPU cost estimates use measured test runtimes. In this experiment, the GPU consumption of FastCI includes the sampling overhead.

FastCI-select retains 98.2% of modified-code coverage and achieves 64.1% affected-test recall with context-aware pruning enabled. Without pruning, these metrics rise to 99.2% and 82.7%, respectively. Based on measured test runtimes, FastCI can achieve an estimated 74.0% GPU saving relative to full CI.

MethodAffected-test recallModified code coverage retentionGPU savings
Full CI100.0%100.0%0.0%
Megatron upstream standard test selection (L0)5.2%72.7%94.0%
Codex Planner53.6%74.9%83.1%
NameRTS (Megatron-LM adapter)87.3%91.1%33.0%
FastCI-select without context-aware pruning82.7%99.2%62.6%
FastCI-select64.1%98.2%74.0%

The open-source release includes a native Megatron-LM adapter and a CPU replay that recomputes the table from the archived selection decisions. The repository provides the public commit list, functional test configurations, per-commit selections and line coverage, and measured test runtimes used in the cost estimates. Megatron-LM adapter and CPU replay

Thanks for authors' detailed responses and the additional experimental details/results. My earlier concerns are mainly about the scope and the evaluation details, and the clarifications and new experiments address most of them. I have several remaining questions:

  1. Compared with existing runtime tracing methods, what is the essential technical novelty of FastCI? Please do not restate the functions of each module. Here, the authors should point out which specific techniques cannot be achieved by existing work, or what new concept FastCI introduces. As mentioned in my review, this paper currently reads like a combination of existing techniques, which is an incremental engineering contribution.
  2. What is the Codex Planner baseline in the Megatron-LM experiment? I could not find this term in the main text of the paper.    

In addition, since the authors can revise the paper during the ICLR discussion period, I suggest adding these clarifications, the experimental details, and the new results to the paper (even if some of the results are placed in the appendix). Please also indicate the specific line numbers of these revisions in the response. This will help the reviewers reconsider their scores.

Thanks for your reply. Here are our answers for your remaining questions and we have provided the revised manuscript now.

Q1: What is new beyond existing runtime tracing and test selection?

FastCI contributes an end-to-end CI design that uses incomplete runtime evidence to select and prune distributed training jobs for each code change and optimizes their execution through risk-aware scheduling and workload reduction. To our knowledge, FastCI is the first system to demonstrate end-to-end optimization of GPU-intensive CI in production for a large-scale distributed LLM training framework.

Despite extensive prior work on runtime tracing and regression test selection, many GPU CI workflows in practice still rely on predefined test scopes or file-level rules. Megatron-LM uses a predefined L0 tier in its default PR workflow, vLLM has used declared source-file dependencies to select CI jobs, and our previous production CI relied on manually maintained file-to-test mappings. Even NameRTS, a state-of-the-art regression test selection method, retains less modified-code coverage at higher GPU cost than FastCI-select in our Megatron-LM replay (Appendix E). Megatron-LM PR workflow · vLLM CI pipeline generator

FastCI extends prior CI techniques with new designs grounded in observations from production training workloads. For example, sampling can miss short-lived functions, so we expand through related methods and callers to recover candidate tests. With pruning enabled, expansion was triggered in 85.1% of MRs and added tests in 46.8% (Appendix C.4). In scheduling, change-specific failure history can be sparse, and recent instability captures recurring failures. A simple score combining both signals reduced the median GPU cost of the ordered test prefix through the first failing test from 12.6 to 2.3 GPU-hours compared with COLEMAN (§5.3).

Together, these designs form a workflow that optimizes which tests run, when they run, and how much GPU work they do. FastCI has demonstrated its effectiveness at production scale by saving about 724,000 GPU-hours (Appendix D.1).

Q2: What is Codex Planner in the Megatron-LM evaluation?

“Codex Planner” is the LLM reasoning baseline. For each commit, GPT-6.1 Sol receives the changed paths, a patch excerpt, and the IDs and YAML definitions of the 47 candidate functional tests. It returns the IDs of tests likely to exercise the changed behavior. The model receives no runtime traces, coverage data, or test outcomes. The CPU replay scores the saved selections without another model call (Appendices E and F).

Reviewer_DhmuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the issue where Continuous Integration (CI) becomes a development bottleneck in LLM training frameworks due to high reliance on GPU-intensive end-to-end tests. FastCI builds a runtime evidence graph based on runtime-sampled traces to perform evidence-guided test selection, and prunes tests that execute changed code in equivalent contexts via context-aware test pruning. It then performs risk-aware test scheduling based on historical change–failure associations and recent test instability, and applies workload-optimized test execution that reduces test workloads along dimensions outside their intended validation scope. By combining these techniques to optimize both test selection and test execution, FastCI reduces CI latency by 77.5% and GPU resource usage by 63.9% compared to the previously deployed path-based baseline, and reduces average CI latency by 58.4% in production deployment at CompanyX.

Critical strengths

This paper's main strength lies in identifying and solving a concrete, real-world production problem. FastCI successfully uncovers inefficiencies in both test selection and test execution within a large-scale CI environment and proposes methods to address them. It significantly reduces CI latency and GPU resource usage compared to the deployed path-based baseline while improving modified code coverage retention. The evaluation was conducted using the CI workloads of an actual production LLM training framework rather than relying solely on synthetic workloads. This further demonstrates the practical applicability of the proposed approach.

Critical weaknesses

Because FastCI relies on lightweight sampling rather than exhaustive tracing for runtime evidence collection, it has a structural limitation where full coverage and zero false-negatives cannot be strictly guaranteed (e.g., short-lived functions may be missed). While the paper introduces an expansion mechanism (within-class and caller expansion) to mitigate incomplete evidence, the evaluation lacks an ablation study to independently quantify its effectiveness. Providing a clearer analysis of the remaining false-negative risk and quantitative evidence for the expansion mechanism would significantly strengthen the reliability claims of evidence-guided test selection.

Questions for authors
  1. Is the GPU consumption incurred by the periodic re-validation performed in Section 4.4 included in the reported overall GPU resource reduction of 63.9%?
  2. The four core modules of FastCI (Sections 4.1–4.4) appear to be optimizations generally applicable to GPU-intensive LLM/ML systems (such as inference frameworks) beyond LLM training frameworks. Is there a technical reason or domain-specific challenge that specifically restricts the target scope of the paper to "LLM Training Frameworks"? If there are no unique differentiators exclusive to LLM training, is it acceptable to understand the scope of this paper more generally as "GPU-intensive LLM Systems CI"?
  3. Since FastCI relies on lightweight sampling, I am curious whether the risk of omitting affected tests can be further analyzed or bounded beyond the empirical metrics of "recall of affected tests" and "modified code coverage retention." Additionally, Section 4.1 introduces within-class expansion and caller-function expansion to mitigate the issue of incomplete runtime evidence. How effective are these expansion mechanisms in practice?
4 comments

Thanks for your feedback and questions. Here is our response for each question:

Q1: Revalidation cost in the 63.9% saving

The reported 63.9% GPU resource saving excludes periodic workload revalidation because the evaluation measures GPU usage for MR-triggered CI. In our framework, validated reductions remain stable while the test itself stays unchanged, so we revalidate them less frequently in nightly CI. Scheduling these runs overnight when MR-triggered CI is idle keeps their impact on developers’ CI jobs small.

Q2: Applicability beyond training frameworks

FastCI can also apply to GPU-intensive inference CI when two conditions hold. First, individual tests must consume enough GPU time to make runtime sampling a small fraction of their cost. Second, the test suite must contain enough repeated validation across configurations for test selection to remove substantial work. Risk-aware scheduling and workload reduction can help generally, with more gains when these two conditions are satisfied.

The paper focuses on LLM training framework CI. LLM training framework CI meets these two conditions and can be well optimized from our internal to Megatron-LM. The title of this paper stems from this, but we believe FastCI can be a general solution for optimizing GPU-intensive CI.

Q3: Bounding omission risk and expansion effects

Bug-detection capability is difficult to measure directly, so we use two measurable proxies, affected-test recall and modified-code coverage retention. To measure fault detection more directly, as the reviewer suggests, we replay test selection on 30 historical production bugs. Both FastCI variants select the critical test that can expose the fault in each of the 30 bugs, and these results further support the fault-detection capability of FastCI.

Both expansion mechanisms improve affected-test recall and modified-code coverage retention with and without context-aware pruning, with within-class expansion contributing most of the gain. We vary the two mechanisms independently on Table 1. Within-class expansion alone raises modified-code coverage retention from 93.2% to 99.1% with context-aware pruning and from 94.3% to 99.8% without context-aware pruning. Caller-function expansion also improves coverage retention in both settings. Together, the two mechanisms achieve the highest recall and coverage retention in both settings.

Context-aware pruningExpansionNumber of selected testsAffected-test recallModified Python coverage retainedGPU resource cost
OnNone23.8%57.8%93.2%26.7%
OnWithin-class expansion only25.4%65.7%99.1%28.3%
OnCaller-function expansion only24.0%58.3%93.3%26.9%
OnBoth25.7%66.2%99.2%28.6%
OffNone34.6%76.1%94.3%38.4%
OffWithin-class expansion only35.7%83.8%99.8%39.2%
OffCaller-function expansion only34.9%76.9%94.3%38.6%
OffBoth35.9%84.5%99.8%39.4%

In this evaluation, with context-aware pruning enabled, at least one expansion is triggered in 85.1% of MRs and at least one test is added by expansion in 46.8% of MRs, indicating the expansion mechanisms can take effect in many cases.

We have provided a revised manuscript and additional results in response to your concerns, and we would appreciate your thoughts on these updates. If you have any further concerns or questions, we would be happy to clarify and discuss them.

Thank you for the additional experiments. The expansion ablation and the finding that FastCI selected the relevant test in all 30 historical bug cases give me more confidence in its test selection. I also appreciate the clarification that the reported 63.9% GPU savings exclude periodic revalidation. I am keeping my rating.

Thank you for your feedback and for taking the time to consider our responses.

Reviewer_HQWuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FastCI, a framework designed to improve the efficiency of GPU-intensive continuous integration (CI) pipelines for large language model (LLM) training frameworks.

Critical strengths

1. Addresses a practical and increasingly important infrastructure problem.

The paper identifies a meaningful challenge in maintaining large-scale LLM training frameworks, where end-to-end regression tests require substantial GPU resources. As training frameworks grow in complexity, reducing CI costs without compromising reliability is practically important. The paper is well motivated by a real production environment and presents convincing evidence that conventional path-based test selection can be inefficient.

2. Demonstrates substantial efficiency improvements in a production-oriented setting.

The reported reductions in CI latency and GPU resource consumption are significant. The evaluation considers several baselines. The comparison across test-selection efficiency, resource usage, and scheduling effectiveness provides useful empirical evidence of FastCI's benefits within the evaluated environment.

3. Presents a practical and modular system design.

The combination of runtime sampling, dependency-based selection, context-aware pruning, and workload adaptation is operationally appealing. In particular, the use of lightweight runtime sampling and the integration with existing CI infrastructure demonstrate attention to deployment constraints. The production rollout and associated operational lessons strengthen the paper's practical contribution.

Critical weaknesses

1. Limited methodological novelty and unclear contribution to the ICLR community.

My primary concern is that the paper's technical contributions are predominantly in software engineering and CI infrastructure rather than machine learning research.

The core techniques—runtime-based regression test selection, redundancy elimination, history-based test prioritization, and workload reduction—are closely related to established ideas in software testing and systems optimization. Although the paper combines these techniques effectively, it is unclear which component constitutes a sufficiently novel methodological contribution beyond their integration and adaptation to a particular production environment.

Moreover, the proposed framework does not directly introduce new learning algorithms, training optimization techniques, or methods that exploit the underlying mathematical properties of LLM training. The connection to LLMs primarily arises from the computationally expensive nature of the tests.

2. Insufficient evidence that test selection and pruning preserve regression detection capability.

A central claim is that FastCI can substantially reduce the number of executed tests while maintaining adequate regression validation. However, the evaluation relies heavily on modified-code coverage retention, which is not necessarily a reliable proxy for fault-detection effectiveness.

According to Table 1, FastCI-select retains 99.2% of modified-code coverage but achieves only 66.2% recall of affected tests, compared with 76.1% for path-based selection and 94.0% for NameRTS.

This distinction is important because different tests can execute identical code lines while exercising different inputs, configurations, or assertions. Therefore, high line coverage does not establish that the retained tests can detect the same regressions as the original test suite.

Experiments involving fault injection, mutation testing, or historical regression replay would provide more direct evidence that the proposed optimizations do not compromise CI reliability.

3. Limited generalizability and reproducibility of the empirical evaluation.

The experiments are conducted on a single proprietary LLM training framework. Although this provides realistic production workloads, it makes it difficult to determine whether the reported improvements generalize to other frameworks with different architectures, test granularities, and execution patterns.

Moreover, the workload characteristics, GPU cluster configuration, and CI resource contention may significantly influence the reported latency reductions. Broader evaluations or additional sensitivity analyses would strengthen the generalizability of the conclusions.

Questions for authors

Q1. How does FastCI preserve actual fault-detection capability?

Table 1 shows that FastCI-select reduces affected-test recall to 66.2% while retaining 99.2% modified-code coverage. Can the authors provide direct measurements of regression-detection recall using historical bugs, fault injection, or mutation testing? In particular, how often does FastCI prune a test that uniquely detects a regression missed by the retained tests?

Q2. How does the framework handle incomplete tracing of distributed and native execution?

Given that runtime evidence is primarily collected from rank-0 Python processes, how does FastCI account for changes affecting non-root ranks, C++/CUDA kernels, or execution paths that are systematically missed by sampling? Additionally, how does the fallback mechanism protect against missed test-function relationships when a modified function already appears in the graph but its execution by another test was not observed?

Q3. How is the safety of workload optimization established?

Section 4.4 states that workload dimensions unrelated to test validation are reduced, with periodic comparisons against original workloads. How frequently are these comparisons performed, and how are validation-relevant dimensions determined? Can the authors quantify the probability of missing scale-dependent failures, such as numerical instability, or distributed synchronization errors, after reducing model sizes or other workload dimensions?

Q4. Can the authors provide stronger evidence of generalizability and component-level effectiveness?

How well does FastCI transfer to other LLM training frameworks or publicly available CI workloads? Can the authors also provide controlled end-to-end ablations that independently enable test selection, context-aware pruning, risk-aware scheduling, and workload optimization, while reporting both wall-clock failure-feedback latency and regression-detection effectiveness?

5 comments

Thanks for your feedback. Here is our response for each question:

Q1: Measuring fault-detection capability

We use two measurable proxies for regression-detection capability, affected-test recall and modified-code coverage retention. Context-aware pruning deliberately omits affected tests when a retained test exercises the same modified function in the same observed call context and has matching validation metadata. This explains why affected-test recall is 84.5% for FastCI without pruning and 66.2% for FastCI-select, a difference of 18.3 percentage points in our evaluation. But coverage retention differs by only 0.6 points. Because FastCI's techniques are modular, context-aware pruning can be disabled when CI tests are not comprehensive or GPU pressure is low.

To check whether FastCI removes tests needed to expose faults, we replay test selection on 30 historical production bugs. In all cases, both FastCI variants retain the critical test that can expose the fault, while NameRTS and LLM reasoning with GPT-6 Sol each cover 29 of 30, and pruning introduces no additional misses in this replay.

Q2: Tracing gaps in distributed and native execution

FastCI samples one representative per group of ranks with symmetric execution behavior. In many CI jobs, all ranks are symmetric, so sampling rank 0 alone suffices. In pretraining jobs with pipeline parallelism, different stages execute different code. Post-training jobs can have ranks with different roles. In these jobs, FastCI samples one representative rank per group. We have clarified this rank sampling policy in Appendix B of the revised manuscript.

The runtime evidence graph records only Python execution, so production FastCI uses the existing path-based selector for native code changes outside Python. In this paper's test selection experiments, FastCI uses the same rules for those changes and for configuration changes. Native code, however, changes infrequently: our 90-day audits find such changes in 4.3% of commits in our internal framework and 6.0% in Megatron-LM.

When a modified function is already in the graph but sampling misses its execution by an affected test, FastCI can miss affected tests. Omitted tests, however, need not indicate missing bugs: in the bug replay in Q1, both FastCI variants catch 30 of 30 historical production bugs, outperforming NameRTS and LLM reasoning with GPT-6 Sol.

Q3: Safety of workload optimization

We validate each workload reduction by comparing the original and reduced test outcomes over multiple paired runs before enabling it. Developers already reduce workloads manually to speed up CI. FastCI automates these practices with rules derived from experience, matching test metadata, such as names and descriptions. The rules preserve dimensions that define the validation target, such as parallelism degree for distributed execution or sequence length for long-context training.

In our framework, the original and reduced workloads continue to agree once a reduction is validated and the test itself remains unchanged. We therefore compare them periodically in nightly CI, which has little impact on daily CI, since CI runs are always triggered in daytime. We conduct FastCI's workload reduction on 331 distinct CI task definitions, and we observe no numerical instability or distributed synchronization errors attributable to workload reduction.

Q4: Generalization and component ablations

FastCI's selection and pruning transfer to Megatron-LM's public functional CI. We adapt the open-source implementation and replay 100 public commits. With context-aware pruning enabled, FastCI-select retains 98.2% of modified-code coverage and achieves an estimated GPU saving of 74.0% relative to full CI, including sampling overhead. Without pruning, it retains 99.2% and saves 62.6%. See Overall for the baseline comparison and cost model. FastCI implementation

Test selection and context-aware pruning. See Section 5.2 (Table 1) for the comparison with path-based selection and with pruning enabled or disabled. Section 5.1 (Figures 5 and 6) shows the combined effect of selection and pruning on end-to-end CI.

Risk-aware scheduling. See Section 5.3 (Figure 7) for the comparison of execution orders with the tests and outcomes held fixed. It reports first-failure rank and GPU use before the first failure.

Workload reduction. Section 5.4 compares original and reduced workloads. Section 5.1 covers end-to-end performance. As we do not perform early cancellation in the end-to-end experiments, increase from FastCI-select to FastCI stems from workload reduction.

Q1 reports whether selected tests can expose known faults in the bug replay. Both expansion mechanisms improve recall in the supplemental ablation (Overall response). Within-class expansion recovers most of the coverage, and combining both achieves the highest recall and coverage retention with context-aware pruning on and off.

For your concern about FastCI's novelty and relevance to ICLR, here is our replay:

FastCI’s methodological contribution is to use a sampled runtime graph both to find affected tests and to identify redundant checks for each code change. End-to-end LLM training tests often share an entry point and core training loop across precision and parallelism configurations. Evidence-guided selection recovers candidate tests through related methods and callers when sampling misses a modified function. Context-aware pruning compares call contexts and validation metadata at that function, keeping one low-cost representative for repeated checks and retaining tests that validate distinct training behavior. Risk-aware scheduling combines change-specific failure history with recent test instability to surface failures earlier. Workload optimization reduces GPU work while preserving the training settings each test validates.

FastCI fits ICLR’s scope on infrastructure for large-scale learning. TorchTitan uses loss-convergence checks to validate different parallelism configurations, and SpareTrain lowers the cost of reliability checks during training. FastCI addresses the related cost of checking for regressions as training frameworks evolve. The Megatron-LM experiment shows that the selection and pruning design transfers to public training CI (Q4). TorchTitan (ICLR 2025, Appendix B.10), SpareTrain (ICLR 2026)

We have provided a revised manuscript and additional results in response to your concerns, and we would appreciate your thoughts on these updates. If you have any further concerns or questions, we would be happy to clarify and discuss them.

Reviewer_nEdSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FastCI reduces GPU-intensive CI workloads through sampled call traces, context-aware test selection, risk-aware scheduling, and workload reduction. The paper evaluates these components individually and in shadow pipelines, then reports a three-week production rollout covering 784 runs. Reported latency savings are 77.5% in the shadow study and 58.4% in production.

Critical strengths
  1. The selection granularity fits the workload. File-level selection can over-select when many end-to-end tests import shared training modules. Function and call-context information offers a more discriminating signal; Table 1 reports both fewer selected tests and higher affected-test recall.
  2. The evaluation includes meaningful operational evidence. Component studies help attribute the gains, while the production rollout tests the system under real workloads. Reporting shadow and production results separately is useful because the realized benefits differ substantially.
Critical weaknesses
  1. The safety evidence does not fully address lost fault detection. Coverage retention is measured over Python code and excludes native C++/CUDA execution. This leaves an important part of a training framework outside the reported guarantee. Moreover, preserving pass/fail on observed executions does not establish that a trimmed test retains its ability to detect future faults. Omitted-test failures, reverts, and withdrawn reductions would provide more direct evidence than coverage retention or three weeks without an attributed regression.
  2. The scope of the headline savings is difficult to reconstruct. Table 1 and Figure 5 report different path-based baseline selection and GPU-use rates. This may reflect different evaluation populations, but the populations should be stated alongside the metrics. The end-to-end accounting should also explicitly include graph construction, maintenance, and query overhead; asymptotic arguments do not establish their practical cost.
  3. Independent verification and generalization remain limited. Results come from one proprietary codebase, with no released artifact and some internally implemented baselines. The gains may depend strongly on its import graph, test redundancy, and failure distribution. A public workload, anonymized trace representation, or reproducible synthetic suite would help establish which properties make the approach effective.
Questions for authors

How are native-code changes covered by selection and rollback? How many omitted-test failures or workload-reduction withdrawals occurred? Please also identify the evaluation population and included overhead for each headline result, and clarify what artifacts can be released.

2 comments

Thanks for your feedback and questions. Here is our response for each question:

Q1: Native-code selection and rollback

FastCI covers native-code changes with the existing path-based selector, and rollback for these changes is unchanged. Its runtime evidence graph records only Python execution, so selection for native code changes and for configuration modifications outside Python reuses the path-based rules that production CI applies, and our test selection experiments apply the same rules to those changes. We have clarified the path-based handling of these changes in Section 4.1 of the revised manuscript.

Native code is modified infrequently. Counting Triton kernel changes as native code changes, over the 90 days ending November 9, 2026, 67 of the 1,549 commits on our internal framework's master branch (4.3%) contain such changes, and over the 90 days ending November 12, 2026, 58 of the 968 commits we examine on Megatron-LM's main branch (6.0%) contain such changes. Counting each changed file once per commit, native code accounted for 3.2% of file changes in our internal framework and 1.4% in Megatron-LM.

Q2: Omitted-test failures and workload withdrawals

Neither omitted-test failures nor workload-reduction withdrawals have been reported. Since FastCI entered production, we received no reports of regressions due to FastCI. In our framework, once a workload reduction is validated, test outcomes remain consistent between the original and reduced workloads as long as the test itself remains unchanged. We can compare the two workloads periodically in nightly CI and restore the original workload if their outcomes disagree.

To assess whether omitting tests weakens regression detection, we replay test selection on 30 historical production bugs. For each bug, we check whether the selected suite includes the critical test that can expose the fault. FastCI with and without context-aware pruning and path-based CI each cover 30 of 30 cases, while NameRTS and LLM reasoning with GPT-6 Sol each cover 29 of 30, and pruning introduces no additional misses in this replay.

Q3: Evaluation population and overhead of each result

Figures 5 and 6 use the results from two days of shadow CI that mirror online MRs, with a common budget of 16 machine slots and 16 GPUs per slot. Relative to path-based CI, FastCI reduces average CI latency by 77.5% and GPU resource usage by 63.9%. These results include the sampling overhead but exclude the graph construction overhead, since graph construction completes on CPU within minutes.

The separate test selection replay covers 100 production MRs and 2,482 distinct modified entities. The results of Table 1 differ from Figure 5 because the two evaluations use different MR cohorts.

The reported end-to-end savings exclude the cost of running the original and reduced workloads for comparison. In production, these comparisons can run in nightly CI. These comparisons can run less frequently than MR CI, so we expect their amortized GPU cost to stay small.

Q4: Releasable artifacts and verification

We release the FastCI implementation, including an adapter for Megatron-LM's functional test suite. In an offline replay of 100 public Megatron-LM commits, FastCI-select with context-aware pruning enabled retains 98.2% of modified code coverage and achieves 64.1% affected-test recall, and with context-aware pruning disabled it reaches 99.2% modified code coverage retention and 82.7% affected-test recall. Based on measured test runtimes, we estimate a net GPU saving of 74.0% relative to full CI, including sampling overhead. FastCI implementation

The public repository includes the CPU demo script, the commit list and the benchmark data used to reproduce the reported results of Megatron-LM. CompanyX's confidentiality requirements prevent us from releasing the internal MR CI workload.

We have provided a revised manuscript and additional results in response to your concerns, and we would appreciate your thoughts on these updates. If you have any further concerns or questions, we would be happy to clarify and discuss them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Key results of FastCI's response

We thank the reviewers for their feedback on FastCI's contribution, bug detection, and evaluation. We report new results from production, release an open-source implementation, and add experiments on both our internal training framework and Megatron-LM. We have provided a revised manuscript alongside this response.

  • Production deployment. FastCI has been in production for over 11 weeks. The production archive contains 4,496 CI runs across 1,226 source branches, with 503,366 CI task instances across 331 distinct CI tasks. Based on the online statistics, we estimate that FastCI saves approximately 724,000 GPU-hours compared with the previous path-based CI. Since deploying FastCI in production, we received no reports of regressions due to FastCI.

  • Open-source implementation. We release FastCI with a runnable CPU demo, the Megatron-LM adapter, and a CPU replay with the data needed to reproduce the results on Megatron-LM. FastCI implementation

  • Historical bug replay. In an offline replay of test selection for 30 historical production bugs, FastCI with and without pruning and path-based CI, which is the selector previously used in production, each cover 30/30 bugs. NameRTS and LLM reasoning with GPT-6 Sol each cover 29/30. Context-aware pruning introduces no additional misses in the replay.

  • Megatron-LM experiment. We adapt FastCI to Megatron-LM's public functional CI and replay test selection on 100 consecutive commits from its main branch using a fixed coverage reference from 47 Megatron's functional tests. With context-aware pruning enabled, FastCI-select retains 98.2% of modified-code coverage and achieves 64.1% affected-test recall. These metrics increase to 99.2% and 82.7% with context-aware pruning disabled. Based on measured test runtimes, it achieves an estimated GPU saving of 74.0% relative to full CI, including sampling overhead.

MethodAffected-test recallModified code coverage retentionGPU savings
Full CI100.0%100.0%0.0%
Megatron upstream standard test selection (L0)5.2%72.7%94.0%
Codex Planner53.6%74.9%83.1%
NameRTS (Megatron-LM adapter)87.3%91.1%33.0%
FastCI-select without context-aware pruning82.7%99.2%62.6%
FastCI-select64.1%98.2%74.0%
  • Fallback ablation. We conduct supplemental ablation on the same workload of 100 MRs and 2,482 distinct modified entities as Table 1. In this ablation, we independently toggle within-class and caller-function expansion with context-aware pruning on and off. Within-class expansion improves affected-test recall and raises modified-code coverage retention from 93.2% to 99.1% and from 94.3% to 99.8%, with and without context-aware pruning. Caller-function expansion also improves recall and adds coverage gains. Combining the two achieves the highest recall and coverage retention in both settings.
Context-aware pruningExpansionNumber of selected testsAffected-test recallModified Python coverage retainedGPU resource cost
OnNone23.8%57.8%93.2%26.7%
OnWithin-class expansion only25.4%65.7%99.1%28.3%
OnCaller-function expansion only24.0%58.3%93.3%26.9%
OnBoth25.7%66.2%99.2%28.6%
OffNone34.6%76.1%94.3%38.4%
OffWithin-class expansion only35.7%83.8%99.8%39.2%
OffCaller-function expansion only34.9%76.9%94.3%38.6%
OffBoth35.9%84.5%99.8%39.4%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.