- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper transforms the actual engineering records into training tasks for agents. The author first summarizes the original requirements, and then uses these requirements to constrain the generation of the tasks, the environment, and the verifier. After generation, the tasks will be actually run for checking, and the tasks will be modified based on the failure logs. Finally, these tasks are used for SFT and RL, and the training effects are compared.
Critical strengths
S1. Having the tasks, environment and verifier follow the same set of requirements is a relatively straightforward design. When generating separately, it is prone to situations where the tasks have requirements but the environment does not support them, or the verifier does not check them.
S2. The checks are not only based on whether the reference solution can complete the task. The author also checked the scores when doing nothing and when only addressing superficial issues. After failure, modifying the task based on the logs can also help filter out problematic tasks.
S3. The generated data shows training benefits across multiple model scales. Both the SFT and the subsequent RL stages can see improvements.
S4. The author constructed over ten thousand executable tasks and collected over twenty thousand execution trajectories. The tasks are equipped with a runtime environment and a verifier, which can support SFT and RL.
Critical weaknesses
W1. The ablation experiment did not specify how much data each group used, nor did it provide the specific training budget. This makes it impossible to determine whether the performance decline is due to a deterioration in the quality of the tasks or a reduction in the training volume.
W2. "TerminalWorld" and "Terminal-Universe" already have very similar task construction and execution check processes. This paper emphasizes explicit evidence constraints, but does not directly compare how much it reduces the number of tasks that deviate from the original requirements relative to existing methods. The existing experiments show that the generated data is useful. It is not clear how much value the new design brings.
W3. The main experiment did not clearly state whether the SFT trajectories of each group were generated by the same teacher model. If the teachers are different, the improvement may also come from better demonstrations.
Questions for authors
Q1. Can you provide the actual data volume and training budget used by each ablation group? If these were not matched, would there still be similar benefits after matching them?
Q2. Can you use the same batch of original records and a similar generation budget to compare the construction processes of this paper and TerminalWorld? I would like to see how much the new evidence constraints can reduce cases where the original requirements are omitted or changed.
Q3. Can you explain whether the SFT trajectories of each group were generated by the same teacher model? If not, can you provide a control group using the same teacher?
Good job, and thanks for the extensive experiments. They addressed some of my concerns, so I will increase my score to 3.