- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
TrainTailor uses a coding agent to build a model-specific training engine from the model’s source code, a fixed training task, and target hardware. It first generates and verifies an engine, then uses profiling to optimize its implementation and execution configuration together. Experiments cover Qwen3.5-0.8B and a reduced Qwen3.8-Flash-Next-15B model, measuring MFU and checking numerical consistency against reference loss trajectories over 30 steps.
Critical strengths
The paper reports an efficiency gain on Qwen3.5-0.8B: 26.94% MFU, versus 20.65% for LlamaFactory and 20.16% for Megatron-Bridge (Table 1). The workflow checks input consistency, runs unit/system tests, and compares loss trajectories against a reference during optimization (§3.2). These provide useful safeguards when an agent modifies training code.
Critical weaknesses
- The motivation is not convincing. Why write a training engine from scratch instead of building on a mature framework? Would adapting an existing framework be more efficient and reliable? The paper does not clearly justify this choice.
- Many important technical details are missing, making it hard to trust the results. The paper reads like an unfinished draft actually: Training data: Which datasets were used, how were they prepared, and what sequence lengths and image inputs were used? Coding agent: Which agent and model/version were used? What prompts and implementation skills guided generation and optimization? Experimental setup: The full training hyperparameters, baseline configurations, and timing protocol for measuring MFU are not clearly reported.
- There is no external performance baseline for Qwen3.8-Flash-Next-15B. The paper attributes this to the existing frameworks lacking native support, but its method describes building a PyTorch reference in such cases. Why not report that reference’s performance? Comparing only with its own initial engine does not show how competitive the final engine is.
- There is no direct comparison with closely related work, especially ForgeTrain, which also uses agents to build and optimize training engines from scratch.
- The paper does not analyze the cost of the method. How many agent calls, tokens, GPU hours, and failed attempts are needed to obtain the final engine? How much human intervention is involved? The final MFU alone does not show whether the method saves time or money overall. These costs should be compared with adapting an existing framework and with related methods.
Questions for authors
See the above section content.
3 comments
Response to Reviewer eUfc (2/2)
W3: External baseline for the 15B model
W3: There is no external performance baseline for Qwen3.8-Flash-Next-15B. The paper attributes this to the existing frameworks lacking native support, but its method describes building a PyTorch reference in such cases. Why not report that reference’s performance? Comparing only with its own initial engine does not show how competitive the final engine is.
Response:
We now report the PyTorch FSDP2 reference, based on the original model implementation, as a performance baseline for Qwen3.8-Flash-Next-15B. It achieves 2.29% MFU, compared with 10.15% for TrainTailor. The results are included in Section 4.3 (Table 2), with baseline configurations detailed in Appendix B.3.
W4: Comparison with ForgeTrain
W4: There is no direct comparison with closely related work, especially ForgeTrain, which also uses agents to build and optimize training engines from scratch.
Response:
ForgeTrain has released generated training engines, while its generation and optimization harness is not yet available. We therefore cannot apply ForgeTrain directly to the models and tasks in our study. Comparing against its released engines would require establishing a separate matched workload and validating training consistency, which we were unable to complete within the rebuttal period. We will complete this comparison and include the results in a future revision.
W5: Development costs and comparison with framework adaptation
W5: The paper does not analyze the cost of the method. How many agent calls, tokens, GPU hours, and failed attempts are needed to obtain the final engine? How much human intervention is involved? The final MFU alone does not show whether the method saves time or money overall. These costs should be compared with adapting an existing framework and with related methods.
Response:
We now report development costs in Section 4.3. For the Qwen3.5-0.8B case study, the complete pipeline took approximately 22.5 hours of wall-clock time, with an estimated resource usage of 11 GPU-hours and approximately 300 million LLM tokens. Human involvement primarily consists of preparing the model source, training task, and hardware configuration. Build required one repair round (Section 4.4), and Appendix B.5 reports the 20-round optimization trajectory, including rounds with no retained changes.
For new models without native framework support, which TrainTailor primarily targets, adapting existing frameworks still entails substantial development effort, as illustrated by PithTrain (Lai et al., 2026) and VibeServe (Kamahori et al., 2026). Given the time required for adaptation, verification, and performance optimization, we could not complete a matched comparison within the rebuttal period. We will add this comparison in a future revision, reporting wall-clock time, token usage, GPU-hours, and final MFU. We will report the same metrics in the planned comparison with ForgeTrain.
Reviewer Reply
Thanks for the updates. The added experiments, illustration, and the 15B baseline are helpful. I’m raising Presentation and Soundness to 2 and changing my overall rating to Weak Reject.
Response to Reviewer eUfc (1/2)
Thank you for recognizing the strengths of our work and for your constructive feedback. We have carefully considered each comment and respond point by point below. We acknowledge that the original manuscript lacked clarity and have extensively revised the writing throughout. Key revisions are highlighted in blue in the revised PDF. During the remaining discussion period, we would appreciate it if you could revisit our responses and share any remaining questions or concerns. We would be happy to provide further clarification before the discussion closes.
W1: Motivation for generating a specialized training engine
Response:
As discussed in Sections 2.1–2.2, adapting a mature framework to a new model requires coordinated changes across interdependent components. Framework reuse also requires correctness validation: Qwen3.5-VL support in Megatron-LM involved fixes for checkpoint loading, shared-expert initialization, and token leakage across packed sequences (Section 2.1). PithTrain (Lai et al., 2026) shows that large codebases and indirect component specifications increase the effort coding agents spend locating implementations, tracing dependencies, and checking whether changes are complete. These difficulties are especially relevant when a new architecture lacks native framework support.
This motivates generating an engine directly for the target model, task, and hardware, while reusing PyTorch operators and distributed primitives. Such specialization allows the agent to coordinate implementation and execution choices around the target and offers opportunities for performance gains even when framework support exists (Section 2.2). Our Qwen3.5-0.8B results demonstrate this opportunity, achieving 26.94% MFU compared with 20.65% for LlamaFactory and 20.16% for Megatron-Bridge on the same task and hardware (Section 4.3, Table 2).
W2: Experimental and implementation details
Response:
We have added the datasets, sample selection, image preprocessing, and training hyperparameters to Appendix B.2, and the baseline configurations to Appendix B.3. The workloads use CORD-v2 for Qwen3.5-0.8B and a mixture of LLaVA image-instruction and Tulu-3 long-text examples for Qwen3.8-Flash-Next-15B (Section 4.1).
TrainTailor uses Cursor with Grok 4.6, as specified in Section 4.1. Sections 3.2–3.3 now describe the generation and optimization prompts and their associated skills, including the training-engine contract, test requirements, and guidance for optimizing operators, data layouts, memory use, and communication.
Section 4.1 also specifies the MFU timing protocol. We measure 25 training steps after five warm-up steps, taking the maximum synchronized step time across ranks for each step and averaging over the measurement window.