- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
JIT-Agent is a trained 27B model which writes the software controlling another agent for each new task. It generates modules for memory, planning, actions, and tool selection. Training teaches it to customize this software, repair failures, and improve on earlier designs. At inference time, it generates three candidates, selects one using the model's own likelihood score, and executes it after checks and any needed repair. A streaming version saves useful designs for later tasks. Across nine benchmarks, it improves all 18 main comparisons against the same task model using ReAct. Comparisons with established harnesses also show competitive performance and lower reported execution cost.
Critical strengths
The improvements are broad and sometimes substantial. Both main task models improve on all nine benchmarks, including large planning gains. Only DeepSeek-V4-Flash supplies execution feedback during training, so the gains with GLM, Qwen, and Mimo task models support transfer beyond that training executor (Table 2; Appendix A.2; Figure 5).
The generated software is usually usable after repair. Table 4 reports 98.3% validity after at most one repair for Stage II and 97.9% for the final Stage III model. Against established harnesses, the full system wins four of six performance comparisons and has the lowest reported execution API cost in all six (Table 3). It does not win every performance comparison, but it is competitive beyond a basic ReAct baseline.
Critical weaknesses
The benefit of training is not fully separated from the supplied examples and design rules. Table 4 shows improvements across training stages, which is useful evidence. However, it starts after supervised training. The paper does not compare with the base generator or teacher given the same protocol and reference harnesses, or with simply selecting from the 13 seed harnesses. Those comparisons would show what the learned generator adds. The production-harness comparison also needs clearer details about versions, tool access, and adapters to establish equivalent conditions.
The cost comparison leaves out part of the system's work. The main tables count task-execution API cost. Appendix C.2 does measure generation time: roughly 52–58 seconds per rollout, or 9–10% of execution time, on two benchmarks. The remaining question is the generator's full monetary or compute cost, including its inputs and repairs. The streaming version also checks candidates against an incumbent and replays both on five earlier tasks. It is unclear whether the streaming cost curves include all of that work. This could affect some close cost comparisons.
Questions for authors
-
How does the trained model compare with the base generator, the teacher, and selection from the seed bank when they receive the same information and comparable budgets?
-
What is the complete cost of generating, repairing, and running the selected harness? Do the streaming cost curves include incumbent checks and all historical replays?
-
How stable are the main comparisons across independent runs? Do the streaming gains hold under different task orders? This is especially relevant for the smaller test sets.
-
Which versions and tool adapters were used for the established harnesses? Did they receive equivalent tools, information, and execution limits?