acceptodds
Under review as a conference paper at ICLR 2027

TrainTailor:A Coding Agent for Synthesizing Model-Specific Training Engines

Abstract

Large language models (LLMs) are released at a rapid pace, requiring training frameworks to continually adapt to new architectures. However, adapting general-purpose training frameworks to these architectures requires substantial engineering effort, while their shared abstractions can introduce runtime overhead and constrain model-specific optimization. In this paper, we explore an alternative enabled by recent advances in coding agents: building a dedicated training engine from scratch for each model. We instantiate this idea as TrainTailor, an agent-driven system that synthesizes and optimizes a standalone training engine tailored to a given model, training task, and hardware environment. Specifically, TrainTailor first builds a functional training engine with an initial execution configuration for the target hardware and validates its training behavior against a trusted reference. It then iteratively refines the engine implementation and execution configuration together, guided by profiling and bottleneck analysis, to improve training efficiency. Freed from the need to support diverse architectures, the generated engine can eliminate unnecessary abstractions and enable model-specific optimizations, such as architecture-aware operator fusion. Experiments show that the engines generated by TrainTailor satisfy the prescribed numerical tolerances over 30 steps, while improving model FLOPs utilization (MFU) by approximately 30% on average.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
training engines, code agents, code generation, performance optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eUfcOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

TrainTailor uses a coding agent to build a model-specific training engine from the model’s source code, a fixed training task, and target hardware. It first generates and verifies an engine, then uses profiling to optimize its implementation and execution configuration together. Experiments cover Qwen3.5-0.8B and a reduced Qwen3.8-Flash-Next-15B model, measuring MFU and checking numerical consistency against reference loss trajectories over 30 steps.

Critical strengths

The paper reports an efficiency gain on Qwen3.5-0.8B: 26.94% MFU, versus 20.65% for LlamaFactory and 20.16% for Megatron-Bridge (Table 1). The workflow checks input consistency, runs unit/system tests, and compares loss trajectories against a reference during optimization (§3.2). These provide useful safeguards when an agent modifies training code.

Critical weaknesses
  1. The motivation is not convincing. Why write a training engine from scratch instead of building on a mature framework? Would adapting an existing framework be more efficient and reliable? The paper does not clearly justify this choice.
  2. Many important technical details are missing, making it hard to trust the results. The paper reads like an unfinished draft actually: Training data: Which datasets were used, how were they prepared, and what sequence lengths and image inputs were used? Coding agent: Which agent and model/version were used? What prompts and implementation skills guided generation and optimization? Experimental setup: The full training hyperparameters, baseline configurations, and timing protocol for measuring MFU are not clearly reported.
  3. There is no external performance baseline for Qwen3.8-Flash-Next-15B. The paper attributes this to the existing frameworks lacking native support, but its method describes building a PyTorch reference in such cases. Why not report that reference’s performance? Comparing only with its own initial engine does not show how competitive the final engine is.
  4. There is no direct comparison with closely related work, especially ForgeTrain, which also uses agents to build and optimize training engines from scratch.
  5. The paper does not analyze the cost of the method. How many agent calls, tokens, GPU hours, and failed attempts are needed to obtain the final engine? How much human intervention is involved? The final MFU alone does not show whether the method saves time or money overall. These costs should be compared with adapting an existing framework and with related methods.
Questions for authors

See the above section content.

3 comments
AuthorsOpenReview ↗

Response to Reviewer eUfc (1/2)

Thank you for recognizing the strengths of our work and for your constructive feedback. We have carefully considered each comment and respond point by point below. We acknowledge that the original manuscript lacked clarity and have extensively revised the writing throughout. Key revisions are highlighted in blue in the revised PDF. During the remaining discussion period, we would appreciate it if you could revisit our responses and share any remaining questions or concerns. We would be happy to provide further clarification before the discussion closes.

W1: Motivation for generating a specialized training engine

W1: The motivation is not convincing. Why write a training engine from scratch instead of building on a mature framework? Would adapting an existing framework be more efficient and reliable? The paper does not clearly justify this choice.

Response:

As discussed in Sections 2.1–2.2, adapting a mature framework to a new model requires coordinated changes across interdependent components. Framework reuse also requires correctness validation: Qwen3.5-VL support in Megatron-LM involved fixes for checkpoint loading, shared-expert initialization, and token leakage across packed sequences (Section 2.1). PithTrain (Lai et al., 2026) shows that large codebases and indirect component specifications increase the effort coding agents spend locating implementations, tracing dependencies, and checking whether changes are complete. These difficulties are especially relevant when a new architecture lacks native framework support.

This motivates generating an engine directly for the target model, task, and hardware, while reusing PyTorch operators and distributed primitives. Such specialization allows the agent to coordinate implementation and execution choices around the target and offers opportunities for performance gains even when framework support exists (Section 2.2). Our Qwen3.5-0.8B results demonstrate this opportunity, achieving 26.94% MFU compared with 20.65% for LlamaFactory and 20.16% for Megatron-Bridge on the same task and hardware (Section 4.3, Table 2).

W2: Experimental and implementation details

W2: Many important technical details are missing, making it hard to trust the results. The paper reads like an unfinished draft actually: Training data: Which datasets were used, how were they prepared, and what sequence lengths and image inputs were used? Coding agent: Which agent and model/version were used? What prompts and implementation skills guided generation and optimization? Experimental setup: The full training hyperparameters, baseline configurations, and timing protocol for measuring MFU are not clearly reported.

Response:

We have added the datasets, sample selection, image preprocessing, and training hyperparameters to Appendix B.2, and the baseline configurations to Appendix B.3. The workloads use CORD-v2 for Qwen3.5-0.8B and a mixture of LLaVA image-instruction and Tulu-3 long-text examples for Qwen3.8-Flash-Next-15B (Section 4.1).

TrainTailor uses Cursor with Grok 4.6, as specified in Section 4.1. Sections 3.2–3.3 now describe the generation and optimization prompts and their associated skills, including the training-engine contract, test requirements, and guidance for optimizing operators, data layouts, memory use, and communication.

Section 4.1 also specifies the MFU timing protocol. We measure 25 training steps after five warm-up steps, taking the maximum synchronized step time across ranks for each step and averaging over the measurement window.

AuthorsOpenReview ↗

Response to Reviewer eUfc (2/2)

W3: External baseline for the 15B model

W3: There is no external performance baseline for Qwen3.8-Flash-Next-15B. The paper attributes this to the existing frameworks lacking native support, but its method describes building a PyTorch reference in such cases. Why not report that reference’s performance? Comparing only with its own initial engine does not show how competitive the final engine is.

Response:

We now report the PyTorch FSDP2 reference, based on the original model implementation, as a performance baseline for Qwen3.8-Flash-Next-15B. It achieves 2.29% MFU, compared with 10.15% for TrainTailor. The results are included in Section 4.3 (Table 2), with baseline configurations detailed in Appendix B.3.

W4: Comparison with ForgeTrain

W4: There is no direct comparison with closely related work, especially ForgeTrain, which also uses agents to build and optimize training engines from scratch.

Response:

ForgeTrain has released generated training engines, while its generation and optimization harness is not yet available. We therefore cannot apply ForgeTrain directly to the models and tasks in our study. Comparing against its released engines would require establishing a separate matched workload and validating training consistency, which we were unable to complete within the rebuttal period. We will complete this comparison and include the results in a future revision.

W5: Development costs and comparison with framework adaptation

W5: The paper does not analyze the cost of the method. How many agent calls, tokens, GPU hours, and failed attempts are needed to obtain the final engine? How much human intervention is involved? The final MFU alone does not show whether the method saves time or money overall. These costs should be compared with adapting an existing framework and with related methods.

Response:

We now report development costs in Section 4.3. For the Qwen3.5-0.8B case study, the complete pipeline took approximately 22.5 hours of wall-clock time, with an estimated resource usage of 11 GPU-hours and approximately 300 million LLM tokens. Human involvement primarily consists of preparing the model source, training task, and hardware configuration. Build required one repair round (Section 4.4), and Appendix B.5 reports the 20-round optimization trajectory, including rounds with no retained changes.

For new models without native framework support, which TrainTailor primarily targets, adapting existing frameworks still entails substantial development effort, as illustrated by PithTrain (Lai et al., 2026) and VibeServe (Kamahori et al., 2026). Given the time required for adaptation, verification, and performance optimization, we could not complete a matched comparison within the rebuttal period. We will add this comparison in a future revision, reporting wall-clock time, token usage, GPU-hours, and final MFU. We will report the same metrics in the planned comparison with ForgeTrain.

Reviewer_eUfcOpenReview ↗

Reviewer Reply

Thanks for the updates. The added experiments, illustration, and the 15B baseline are helpful. I’m raising Presentation and Soundness to 2 and changing my overall rating to Weak Reject.

Reviewer_d5RrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author introduced TrainTailor, a coding agent that implement a training engine from scratch for one model on a specific hw, testing its correctness and loss, and then improve performance by profling and tune code and configs together.

Experiments on Qwen3.5-0.8B shows 26.94% MFU, which is 30% higher than LlamaFactory 20.65%, and Megatron-Bridge 20.16%. It also runs a reduced 15B Qwen3.8-Flash-Next model that these frameworks do not support.

Critical strengths
  • The setting is realistic and it is a painpoint in real world training setting. "The process does not require an existing optimized training implementation of the target model", and it works on a model where "the evaluated frameworks lack native training support".

  • The paper demonstrated the optimization story well. Table 2 lists the changes for every round, and the MFU difference, so readers can see where the speedup comes from clearly.

Critical weaknesses
  • The current experiments is not fully isolating the methods own contribution. For example, the ablation fixes the microbatch at 2, but the full pipeline is 4. So it only shows that a fixed configuration is hurt, not that joint search beats normal tuning.
  • The paper also does not provide the explicit name of the agent being used.
  • The correctness check is not shown to catch bugs. The way it compares losses against references is only over a fixed window for 30 steps training, but the tests are written from the same agent. It might have risk to have a buggy system without other agent's validation.
Questions for authors
  • could you please report the agent used and the approximated budget?
  • Could you report the MFU variance across multiple runs if possible?
  • For ablation, could you compare against the same agent using direct generation followed by normal configuration tuning?
  • If you inject known bugs into a correct engine, what is the percentage of bugs that the verification is able to catch? and which check catches which one?
2 comments
AuthorsOpenReview ↗

Response to Reviewer d5Rr (1/2)

Thank you for recognizing the strengths of our work and for your constructive feedback. We have carefully considered each comment and respond point by point below. We acknowledge that the original manuscript lacked clarity and have extensively revised the writing throughout. Key revisions are highlighted in blue in the revised PDF. During the remaining discussion period, we would appreciate it if you could revisit our responses and share any remaining questions or concerns. We would be happy to provide further clarification before the discussion closes.

W1 / Q3: Joint optimization and conventional configuration tuning

W1: The current experiments is not fully isolating the methods own contribution. For example, the ablation fixes the microbatch at 2, but the full pipeline is 4. So it only shows that a fixed configuration is hurt, not that joint search beats normal tuning.

Q3: For ablation, could you compare against the same agent using direct generation followed by normal configuration tuning?

Response:

We have clarified the ablation setup in Section 4.1 and report the results in Section 4.4. The ablated variant retains the full Build pipeline and allows both configuration and implementation changes, but removes TrainTailor's explicit coordination between them. For Qwen3.5-0.8B, the agent initially explored the execution configuration and selected a microbatch size of 2, but did not revisit this choice as the implementation evolved. The resulting engine achieves 13.21% MFU, compared with 26.94% at a microbatch size of 4 for TrainTailor. As discussed in Section 2.3, implementation optimizations can make new configurations feasible, while exploiting them can require further implementation changes.

TrainTailor revisited a microbatch size of 8 near the end of the optimization budget, but it offered little benefit at that stage. Subsequent refinements to gradient staging and communication reduced execution overhead. A supplementary experiment on the final engine achieved 29.42% MFU with a microbatch size of 8, compared with the 26.94% reported in the paper with a microbatch size of 4.

Our results demonstrate the benefits of joint configuration and implementation optimization, and the supplementary experiment indicates further performance headroom. We will rerun the pipeline with more optimization rounds to pursue additional gains through continued joint refinement.

We will also add a comparison with the same agent using direct generation followed by conventional configuration tuning under a matched budget.

W2 / Q1: Coding agent and optimization budget

W2: The paper also does not provide the explicit name of the agent being used.

Q1: could you please report the agent used and the approximated budget?

Response:

TrainTailor uses Cursor with Grok 4.6. We budgeted 20 optimization rounds for Qwen3.5-0.8B and 40 for Qwen3.8-Flash-Next-15B. We have added the agent, model, and optimization budgets to Section 4.1.

AuthorsOpenReview ↗

Response to Reviewer d5Rr (2/2)

W3 / Q4: Independent review and fault detection

W3: The correctness check is not shown to catch bugs. The way it compares losses against references is only over a fixed window for 30 steps training, but the tests are written from the same agent. It might have risk to have a buggy system without other agent's validation.

Q4: If you inject known bugs into a correct engine, what is the percentage of bugs that the verification is able to catch? and which check catches which one?

Response:

We injected faults based on public reports from three widely used repositories—Transformers, ms-swift, and Megatron-LM—into our Qwen3.5-0.8B engine. The loss gate rejected 2/3 of these faults within 30 steps, and existing unit tests detected the remaining mRoPE fault. Together, these checks detected all three cases.

Injected faultLoss gateExisting UT/ST
Missing packed GDN boundaries (Transformers #45034)Rejected at step 1Detected
Missing GDN Q/K normalization (ms-swift #10567)Rejected at step 1 (NaN)Not detected
Incorrect mRoPE layout (Megatron-LM #4755)Not rejectedDetected

As described in Section 3.2, TrainTailor checks input and supervision consistency, tests both component behavior and the integrated training path, and compares loss trajectories against a reference. A separate agent reviews the engine and its tests, while verification failures trigger diagnosis, repair, and re-verification before the engine is accepted.

During optimization of Qwen3.5-0.8B, a unit test caught an error in packed-window attention by comparing its output with a reference that processes each window separately. The agent corrected the implementation and reran the test, which then passed.

Q2: Variance across independent runs

Q2: Could you report the MFU variance across multiple runs if possible?

Response:

We have clarified in Section 4.1 that we ran the complete pipeline once per model, so the current results do not provide an estimate of run-to-run variance in final MFU. Given the time and compute required for each end-to-end run, completing several independent runs within the rebuttal period is difficult. We will repeat the complete pipeline on one model after the rebuttal period and report the final MFU of each run and the sample variance across runs.

Reviewer_KWqrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses a coding agent to generate a training engine for a given model, training task, and hardware target. Build extracts model structure and behavioral knowledge, generates an engine, and verifies its behavior. Optimize iteratively refines the implementation and configuration using profiling feedback.

Critical strengths

The paper integrates verification with three parts for correctness during Optimize, which is a rigorous design.

Critical weaknesses
  1. Appendix A checks that seven framework configurations satisfy the numerical gate, but does not test whether faulty engines are rejected. The gate’s sensitivity to silent training errors remains untested.
  2. The paper reports no seeds, repeated syntheses, success rates, or uncertainty estimates. It is unclear whether 26.94% MFU is representative or selected from multiple attempts, and how reliably Build produces a verifiable engine.
  3. The approximately 30% average improvement rests on two baselines for one 0.8B workload. The 15B model has no external performance baseline. The extent to which configuration differences explain the gains remains unclear.
  4. No comparison against another external coding agent workflow is provided. The direct generation ablation removes Decompose/Extract within TrainTailor’s own pipeline.
  5. The coupling ablation fixes microbatch size at 2, whereas the full pipeline uses 4. The 13.21% versus 26.94% MFU gap therefore conflates joint refinement with access to configuration tuning.
  6. Experiments use only two and four GPUs, and the detailed Qwen3.5 case requires neither tensor nor pipeline parallelism. The evaluation does not establish effectiveness for multi-node training or more complex parallelism.
  7. The coding agent model, optimization budget, token and GPU costs, and extent of human intervention are unspecified. The total search effort behind the 18 selected optimization rounds is also unclear.
  8. Numerical validation covers only 30 training steps. No convergence runs, held-out evaluations, or final model quality comparisons are reported. Short term loss agreement does not establish that the generated engines preserve longer term training outcomes.
Questions for authors
  1. What fraction of injected training faults does the numerical gate reject within 30 steps?
  2. How many independent synthesis runs were performed per model? Please report successful and failed Build attempts, the distribution of final MFU, and uncertainty estimates.
  3. Were baseline text packing, input prefetching, and communication precision comparable to TrainTailor? Do the gains persist under matched settings?
  4. How does TrainTailor compare with a comparable coding agent under matched budgets? Does Decompose/Extract improve verification success and total development cost when its own overhead is included?
  5. Does joint refinement outperform implementation optimization with an independently tuned static configuration, or simple alternating code tuning? Please compare these alternatives under equal budgets and across repeated runs.
  6. Can you provide an external performance baseline for the 15B model?
  7. Which coding agent model, prompts, and optimization budget were used? How do the 18 selected rounds relate to the complete optimization history?
  8. Can you report longer training runs against a trusted reference? Do the generated engines preserve these outcomes beyond the validation window of 30 steps?
5 comments
AuthorsOpenReview ↗

Response to Reviewer KWqr (1/3)

Thank you for recognizing the strengths of our work and for your constructive feedback. We have carefully considered each comment and respond point by point below. We acknowledge that the original manuscript lacked clarity and have extensively revised the writing throughout. Key revisions are highlighted in blue in the revised PDF. During the remaining discussion period, we would appreciate it if you could revisit our responses and share any remaining questions or concerns. We would be happy to provide further clarification before the discussion closes.

W1 / Q1: Sensitivity of the correctness checks

W1: Appendix A checks that seven framework configurations satisfy the numerical gate, but does not test whether faulty engines are rejected. The gate’s sensitivity to silent training errors remains untested.

Q1: What fraction of injected training faults does the numerical gate reject within 30 steps?

Response:

Our per-step tolerance parameters (atol=0.01, rtol=0.05) match those used in Liger-Kernel’s 32-step BF16 mini-Llama3 tests, and we additionally apply a stricter initial-loss check. As reported in Appendix A, the tightest per-step comparison across the baseline configurations has a loss difference of 0.010689 against a tolerance bound of 0.013374.

We injected faults based on public reports from three widely used repositories—Transformers, ms-swift, and Megatron-LM—into our Qwen3.5-0.8B engine. The loss gate rejected 2/3 of these faults within 30 steps, both at step 1: one loss mismatch and one nonfinite loss.

Injected faultLoss gateExisting UT/ST
Missing packed GDN boundaries (Transformers #45034)Rejected at step 1Detected
Missing GDN Q/K normalization (ms-swift #10567)Rejected at step 1 (NaN)Not detected
Incorrect mRoPE layout (Megatron-LM #4755)Not rejectedDetected

As described in Section 3.2, TrainTailor checks input and supervision consistency, tests both component behavior and the integrated training path, and compares loss trajectories against a reference. A separate agent reviews the engine and its tests, while verification failures trigger diagnosis, repair, and re-verification before the engine is accepted. During an actual Qwen3.5 Build run, the loss gate also detected a visual-position interpolation error, which the agent subsequently corrected.

W2 / Q2: Independent runs and reproducibility

W2: The paper reports no seeds, repeated syntheses, success rates, or uncertainty estimates. It is unclear whether 26.94% MFU is representative or selected from multiple attempts, and how reliably Build produces a verifiable engine.

Q2: How many independent synthesis runs were performed per model? Please report successful and failed Build attempts, the distribution of final MFU, and uncertainty estimates.

Response:

We conducted three independent Build runs for Qwen3.5-0.8B, all of which produced engines that passed verification. We have clarified in Section 4.1 that we ran the complete pipeline once for each model and training task. The reported final MFU comes from that run.

Given the substantial time and computational costs, completing repeated end-to-end experiments within the rebuttal period is challenging. We will conduct repeated complete pipeline runs on one model afterward and report their outcomes, including the distribution and uncertainty of final MFU.

W3 / Q6: External baseline for the 15B model

W3: The approximately 30% average improvement rests on two baselines for one 0.8B workload. The 15B model has no external performance baseline. The extent to which configuration differences explain the gains remains unclear.

Q6: Can you provide an external performance baseline for the 15B model?

Response:

We now report the PyTorch FSDP2 reference, based on the original model implementation, as a performance baseline for Qwen3.8-Flash-Next-15B. It achieves 2.29% MFU, compared with 10.15% for TrainTailor. The results are included in Section 4.3 (Table 2), with baseline configurations detailed in Appendix B.3.

AuthorsOpenReview ↗

Response to Reviewer KWqr (2/3)

Q3: Baseline configuration comparability

Q3: Were baseline text packing, input prefetching, and communication precision comparable to TrainTailor? Do the gains persist under matched settings?

Response:

Using the baseline frameworks’ native options, we aligned FP32 gradient communication and enabled input prefetching. We also set the final TrainTailor engine to the same microbatch size of 8 as the baselines, achieving 29.42% MFU, compared with 21.67% for LlamaFactory and 20.77% for Megatron-Bridge. As discussed in our response to W5/Q5, the final implementation refinements enabled further gains from the larger microbatch size. These results confirm that the performance gains persist after configuration alignment.

W4 / Q4: Coding-agent comparison and Decompose/Extract

W4: No comparison against another external coding agent workflow is provided. The direct generation ablation removes Decompose/Extract within TrainTailor’s own pipeline.

Q4: How does TrainTailor compare with a comparable coding agent under matched budgets? Does Decompose/Extract improve verification success and total development cost when its own overhead is included?

Response:

Section 4.4 compares runs with and without the TrainTailor harness using the same agent, model, Qwen3.5-0.8B task, hardware, and evaluation protocol. The task explicitly requested engine generation and optimization, with configuration tuning allowed. Without the harness, the agent set the microbatch size to 2, then became stuck in operator-level optimization without further configuration tuning. After making no further gains, it reported convergence and stopped after approximately 10 hours at 8.37% MFU. TrainTailor achieved 26.94% MFU through the complete pipeline in approximately 22.5 hours.

In the Decompose/Extract ablation, both variants retained Verify and diagnostic repair. The full Build pipeline required one repair round to pass Verify, with the Qwen3.5-0.8B Build taking 106.0 minutes, including 13.5 minutes for Decompose and Extract. In comparison, direct generation required four repair rounds and recorded 225.3 minutes of cumulative preparation and execution time. We now report these results in Section 4.4.

W5 / Q5: Joint configuration and implementation optimization

W5: The coupling ablation fixes microbatch size at 2, whereas the full pipeline uses 4. The 13.21% versus 26.94% MFU gap therefore conflates joint refinement with access to configuration tuning.

Q5: Does joint refinement outperform implementation optimization with an independently tuned static configuration, or simple alternating code tuning? Please compare these alternatives under equal budgets and across repeated runs.

Response:

We have clarified the ablation setup in Section 4.1 and report the results in Section 4.4. The ablated variant allows both configuration and implementation changes but removes TrainTailor's explicit coordination between them. For Qwen3.5-0.8B, the agent initially explored the execution configuration and selected a microbatch size of 2, but did not revisit this choice as the implementation evolved. The resulting engine achieves 13.21% MFU, compared with 26.94% at a microbatch size of 4 for TrainTailor. As discussed in Section 2.3, implementation optimizations can make new configurations feasible, while exploiting them can require further implementation changes.

TrainTailor revisited a microbatch size of 8 near the end of the optimization budget, but it offered little benefit at that stage. Subsequent refinements to gradient staging and communication reduced execution overhead. A supplementary experiment on the final engine achieved 29.42% MFU with a microbatch size of 8, compared with the 26.94% reported in the paper with a microbatch size of 4.

Our results demonstrate the benefits of joint configuration and implementation optimization, and the supplementary experiment indicates further performance headroom. We will rerun the pipeline with more optimization rounds to pursue additional gains through continued joint refinement.

AuthorsOpenReview ↗

Response to Reviewer KWqr (3/3)

W6: Parallelism and larger-scale evaluation

W6: Experiments use only two and four GPUs, and the detailed Qwen3.5 case requires neither tensor nor pipeline parallelism. The evaluation does not establish effectiveness for multi-node training or more complex parallelism.

Response:

During the four-GPU Qwen3.8-Flash-Next-15B run, TrainTailor explored FSDP, expert parallelism combined with FSDP, and two-stage and four-stage pipeline parallelism. The two-stage pipeline also combined expert parallelism and FSDP within each stage.

We will extend the evaluation to eight GPUs to assess how TrainTailor jointly adapts parallel configurations and engine implementations to training at a larger scale.

W7 / Q7: Agent, prompts, budget, and optimization history

W7: The coding agent model, optimization budget, token and GPU costs, and extent of human intervention are unspecified. The total search effort behind the 18 selected optimization rounds is also unclear.

Q7: Which coding agent model, prompts, and optimization budget were used? How do the 18 selected rounds relate to the complete optimization history?

Response:

We have added the coding-agent setup and optimization budgets to Section 4.1, development costs to Section 4.3, and the optimization history to Section 4.5 and Appendix B.5. TrainTailor uses Cursor with Grok 4.6. We budgeted 20 optimization rounds for Qwen3.5-0.8B and 40 for Qwen3.8-Flash-Next-15B. We now describe the prompts in Sections 3.2 and 3.3. The Generate prompt directs the agent to the generation skill, model source, training task, hardware specification, blueprint, and knowledge cards. Each optimization prompt identifies the optimization skill, the current candidate, and the best verified engine, and includes verification and MFU results from earlier rounds when available. Human involvement primarily consists of preparing the model source, training task, and hardware configuration.

For the Qwen3.5-0.8B case study, the complete pipeline took approximately 22.5 hours of wall-clock time, with an estimated resource usage of 11 GPU-hours and approximately 300 million LLM tokens.

The original figure showed 18 of the 20 optimization points in the case study. We omitted the other two because the final engine did not retain their changes, and we have now included both points in Figure 5 (Section 4.5) and Table 4 (Appendix B.5).

W8 / Q8: Extended training and final model quality

W8: Numerical validation covers only 30 training steps. No convergence runs, held-out evaluations, or final model quality comparisons are reported. Short term loss agreement does not establish that the generated engines preserve longer term training outcomes.

Q8: Can you report longer training runs against a trusted reference? Do the generated engines preserve these outcomes beyond the validation window of 30 steps?

Response:

To further evaluate the training quality of the generated engine, we compared TrainTailor with LlamaFactory and Megatron-Bridge by fine-tuning Qwen3.5-0.8B on CORD-v2 for five epochs (250 optimizer updates).

ImplementationValidation loss ↓Field micro-F1 (%) ↑TED accuracy (%) ↑
TrainTailor0.0309289.3994.90
LlamaFactory0.0293589.7695.43
Megatron-Bridge0.0305089.1494.17

The results show comparable convergence and final task quality across the three frameworks. In Section 4.2, Figure 4 shows the training and validation loss curves, and Table 1 reports final validation loss, field micro-F1, and TED accuracy. Appendix B.4 describes the training and evaluation protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.