- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
StateForge generates executable environments and tasks defined by target state changes. It produces direct, progressive, and recovery trajectories, then uses state-based checks for supervised training and reinforcement learning. Experiments show improvements on several agent benchmarks.
Critical strengths
1)Checking final state changes allows different valid action sequences. The same outcome-based framework supports task generation, trajectory filtering, and reward design. 2)Results across two model sizes and comparisons using the same backbone support the value of the generated data beyond the synthetic environments.
Critical weaknesses
1)Training corpora and trajectory mixtures differ in size and token exposure. Equal training epochs do not ensure equal training budgets, so the contribution of the state-based design remains partly unclear. 2)Exact state checks depend on correct generated task specifications. The paper provides limited independent evidence about specification errors, and some synthesis and reinforcement learning settings need clearer reporting.
Questions for authors
1)Do the gains remain when supervised tokens and training updates are matched across data sources? 2)What error rate does an independent audit find in the generated tasks and target states?
2 comments
Dear Reviewer,
Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
We sincerely look forward to your feedback.
part 1/1
Thank you for recognizing outcome-based verification and the results across two backbone sizes. We have added a comparison and ablation studies with matched training-set sizes.
W1 / Q1: Matching data size and training exposure
We added two comparisons to examine whether the reported gains persist after controlling training-set size and, for the interaction-mixture ablation, training updates. These comparisons partially address your concern.
First, we control training-set size across data sources. Using the same Qwen3.5-4B backbone and shared SFT settings, we compare StateForge and EnvFactory with 29,463 training records each.
StateForge retains an advantage on both τ² Macro and Shopping. For example, Shopping accuracy increases from 11.67% to 42.50%. This shows that the cross-corpus advantage persists at equal record counts.
Second, we control both record counts and training updates in the interaction-mixture ablation. Each condition uses 10,000 training records and the same number of optimizer updates.
Adding Progressive improves all three aggregate benchmark scores, and adding Recovery further improves them. Thus, the benefits of mixing interaction modes persist when record counts and update counts are held constant.
These results address the effects of training-set size and, within the mixture ablation, update count.
W2: Task-specification errors and reporting of synthesis and RL settings
We agree that the effectiveness of exact state checks depends on the correctness of the tasks themselves. Task construction includes two layers of verification: reference execution establishes a reachable target from the initial state; a bidirectional task–target audit then checks whether the requested effects are fully covered, the target effects are justified, the entities are unambiguous, and the required information is accessible. Exact trajectory verification subsequently compares the actual outcome against this accepted specification. These checks are used to control task-specification quality.
Regarding the request for clearer reporting of synthesis and RL settings, we will consider open-sourcing the task and trajectory synthesis code, as well as the RL training code, after the paper is accepted.
Q2: Independent audit of generated tasks and target states
We have not yet conducted a dedicated statistical assessment of this error rate. Based on a rough estimate of the synthesis process, more than 35% of tasks exhibit incomplete alignment among the task requirements, evaluation criteria, and target states during the auditing stage and are consequently discarded.