- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AutoSFT, a multi-agent system for improving training data for supervised fine-tuning a large language model. The system contains a meta-agent, an executor, analysis agents, and a synthesis agent. The authors run experiments using Qwen3-1.7B-Base on IFEval and IFBench (instruction following) and MBPP+ (code generation). The results show that analysis agents improve the best IFEval score by 6.5 points within the first 30 rounds; in a separate comparison, adding a synthesis agent improves the best IFEval and IFBench scores by 3.43 and 2.93 points, respectively; on MBPP+, analysis improves the best score by 3.7 points; the executor reduces the main agent’s context compactions from 10 to 5 during the process. The paper mainly argues that automated SFT data research benefits from a structure that supports a broader range of data engineering actions.
Critical strengths
The authors carry out a substantial amount of experimental work, including many rounds of model training and evaluation across different configurations. The component comparisons and analysis of the agents’ data modifications provide useful insights into how the system behaves in practice.
Critical weaknesses
-
The experiments do not clearly separate the contribution of the proposed role structure from those of additional analysis instructions and inference computation. It remains unclear whether a general agent prompted to delegate subtasks and perform the same analyses could achieve similar results.
-
As acknowledged in the limitations, each configuration has only one complete search run, and the experiments use only one base model across a limited range of tasks. The stability and generalization of the reported gains are uncertain.
Questions for authors
Could the authors provide more detail on whether the single-agent baseline and AutoSFT have comparable overall computational or monetary costs, including subagent analysis and data generation?
4 comments
We just wanted to kindly check in as the discussion period is approaching its end. If you have any remaining concerns or questions regarding our response, we would be happy to address them promptly. Thank you again for your time and consideration.
Thank you for the detailed response. The rebuttal largely addresses my concerns, and I will raise my overall score from 2 to 3
Thank you for your positive feedback and for raising your score. We are glad that our response addressed your concerns, and we sincerely appreciate your time and careful consideration.
Response to Reviewer qyxw
Thank you for this comment. [Global Response 1] reports the comparison. One agent receives the four analysis angles in a single prompt, with the same tools, execution isolation, base checkpoint, pool and per-round token cap, and it is free to delegate subtasks of its own. Over the 18 rounds AutoSFT ran it reaches 63.57 and 62.17 against 69.52, and given 25 rounds to hold total time comparable its best is 63.78.
Thank you for raising both points.
Stability. [Global Response 3] separates evaluation, training and search noise and reports each one. There are now two complete independent searches per configuration, and the two observed ranges are disjoint.
Generalisation. Also in [Global Response 3], on a second backbone Qwen3-4B-Base over 20 rounds, the analysis configuration reaches a best IFEval of 70.06 against 63.71. The effect is larger at this scale than at 1.7B. Beyond the code generation already in the submission, SciCode and AIME26 are the domains we plan to add next, since a single complete search consumes on the order of 320 to 680M agent tokens and 25 to 39 node-hours, where a node-hour is one 8×H800 node running for one hour.
Thank you for this question. [Global Response 4] reports the accounting, and it includes subagent analysis and data generation. The headline is cost to reach a score rather than cost per round. On MBPP+, exceeding the best score the single agent reaches anywhere in 25 rounds takes 37% of its GPU node-hours, 16% of its agent tokens and 22% of its cost, and the single agent spent six times the agent tokens of the configuration that beat it.
Rebuttal Summary to qyxw
Thank you for your reviews.
Separating role structure from analysis instructions and inference compute W1. We ran the condition that holds the analyses, tools, isolation and budget fixed, leaving the agent free to delegate subtasks of its own. It reaches 63.57 and 62.17 over two independent runs against 69.52, and 63.78 when given 25 rounds against 18 to match total time.
Stability and generalisation W2. We treat these as two claims. For stability, two complete independent searches per configuration with disjoint ranges, plus newly measured evaluation and training noise. For generalisation, a second backbone, Qwen3-4B-Base, where the analysis configuration reaches 70.06 against 63.71. SciCode and AIME26 are the domains we plan to add next.
Cost including subagents and generation Q1. Reported per configuration and, more usefully, as cost to reach a score: 37% of the node-hours, 16% of the agent tokens and 22% of the cost to exceed what the single agent reaches in 25 rounds.
We would be glad to know whether this addresses what you had in mind.