- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents an agent-driven framework for closed-loop post-training, called DAPF, which is motivated by connecting training feedback to actionable interventions. Specifically, DAPF contains 6 modules in a closed-loop workflow from hypothesis generation, design, execution to diagnosis and intervention selection. DAPF is evalauted under seven benchmarks using Qwen3-1.7B for text tasks and Qwen2.5-VL-3B for multimodal reasoning, outperforming baselines.
Critical strengths
- The experiments cover 7 benchmarks
- The reported tested scores contain standard deviations
- The idea of this paper is clear and straightforward
Critical weaknesses
- The introduction is too concise. The motivation and the final results are not well-connected. A solid description for this manuscript can lead the readers to have a better understanding of how and what the authors would like to solve, especially the current length of the introduction is almost similar to the abstract. Not saying that authors are required to use the whole page limit for the main manuscript. However, at least should be informative enough. The writing and paper organization seems not ready and requires polish.
- Since the goal of this work aims to demonstrate a more complicated workflow integration can improve the performance against the baselines, it is necessary to demonstrate that those baselines are well-defined and setup, e.g., how do the authors select final hyper-parameters and what are the final hyper-parameters. Also, it is not clear what FT-agent means, where the authors have not mentioned any abbreviations.
- In FT-agent and DAPF, do authors conduct training without SFT, if yes, a clear ablation studies or reasons why not doing so should be provided.
- It is not too surprised that integrating a more informative workflow can outperform those vanilla baselines. However, more advanced baselines or recent similar works' performance can make the conclusion stronger, especially that the current tested scores' margins are limited
Questions for authors
- One of the motivations in the introduction is that automation alone does not explain why a training run improves. Can authors show the results how DAPF can help explanation?
- In equation 3, the training is limited by the budget. what will be the performance of DAPF with different budget limits?
- In 4.2 BASELINES AND ATTRIBUTION ABLATION, the authors mention that they use the same maximum compute budgets. Could authors describe more on what the compute budgets refer to here given the fact that the 4.5 EFFICIENCY ANALYSIS shows the GPU runtime comparison?