- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper uses code agents to build stateful tool environments and deterministic programs to generate and verify training tasks. It then selects and revises tasks based on student performance. The generated data improves tool-use performance, with the strongest evidence coming from the 4B results.
Critical strengths
-
The paper addresses an important bottleneck in tool-use RL. It connects environment code, task generation, reference trajectories, and rewards in a practical workflow that could be useful to other researchers.
-
The 4B results are a strong part of the paper. Gains across six external evaluations, supported by task-level statistical analysis, show that the generated data has value beyond the training environments.
-
The failure analysis is useful. Showing that weak models can improve benchmark scores by avoiding actions highlights an evaluation issue that matters beyond this framework.
Critical weaknesses
-
The chosen difficulty band is not fully justified. Tasks above 50% success can still produce mixed rewards, while the main downstream selection comparison also varies corpus composition and training budget. The results demonstrate the recipe’s usefulness, but provide less evidence that this particular band explains the gains.
-
Verification is closely tied to a specific reference solution. The checker matches the gold write sequence as well as the final state, which may exclude valid alternatives when independent writes can be reordered. The paper does not clearly establish when this restriction follows from the task’s requirements.
Questions for authors
Please see the weakness part.
2 comments
Dear Reviewer eM4621,
As the rebuttal deadline approaches, we would be grateful for your feedback on our responses. We are happy to address any remaining questions.
If you feel that we have addressed your concerns, we would appreciate it if you could consider updating your assessment and score accordingly.
Best regards,
The authors
Thank you for highlighting the 4B transfer results and the failure analysis. Your concerns about the difficulty band and verification restrictions identify two places where our description needs to be more precise.
W1. The difficulty band and its downstream justification
You are correct that tasks above 50% success can still provide group-relative reward signal. For eight independent binary outcomes with success probability p, the probability of a mixed group is
P(mixed) = 1 − p⁸ − (1 − p)⁸.
It is approximately 90.0% at both p=0.25 and p=0.75, and 99.2% at p=0.5. Thus, the upper threshold is an empirical curriculum choice. Our observed band corresponds to one through four successes out of eight; five through seven successes can also be informative. We will correct the overly categorical explanation.
Our training logs also show how quickly this signal can disappear. In the main 4B run, the batch at update 15 already had 61 successful rollouts out of 64 (95.31%). With eight groups of eight, at least five groups therefore had 8/8 successes and zero group-relative reward signal. These observations motivate leaving room for learning below the saturation region. A related empirical choice appears in FrogNano, Section 3.1: its policy-calibrated coding-task pipeline targets p̂=0.5 in iterations 1–4 and 0<p̂≤0.5 in the revised fifth iteration. This supports targeting challenging but learnable tasks, without establishing that our exact band is universally optimal.
We also agree that the original aggregate comparison mixes corpus composition and training budget. We added three random controls, each containing 129 tasks with the same per-domain counts across the same 17 domains, sampled from the 1,354-candidate pool before in-band selection. Initialization, optimization recipe, training seed and number of updates were held fixed.
W2. Valid alternative trajectories and write ordering
The action matcher checks required calls and arguments against unused calls in the trajectory; it does not impose a global call order. Read-only exploration need not reproduce the reference retrieval path. In a real fleet-dispatch task, for example, listing all available vehicles and then selecting class-D vehicles is accepted just as the reference class-filtered lookup is. Both executions receive reward 1.
However, checking the complete reward revealed a genuine limitation. Reversing the five independent maintenance writes passes action/argument matching and produces identical vehicle states, but changes the order of entries in an append-only maintenance log. The exact database hash therefore rejects that trajectory:
We will correct the phrase “in order” and distinguish order-insensitive action matching from order sensitivity that may remain in state serialization. When log order is not required by the task, rejecting an otherwise valid reordering is a limitation of our current verifier implementation. A suitable remedy is schema-declared order-insensitive comparison for fields where ordering is irrelevant, while retaining genuine workflow dependencies. Alternative write APIs are accepted only when the applicable contract permits them, not automatically because they reach a similar state.