acceptodds
Under review as a conference paper at ICLR 2027

JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution

Abstract

Agent capability is not determined by the model alone. The agent harness, encompassing memory management, planning strategy, action protocol, and tool/skill orchestration, can dominate the contribution of the underlying foundation model. Yet harness design remains manual, task-specific, and fundamentally unscalable. We present JIT-Agent, a harness intelligence model trained to synthesize task-adaptive agent harnesses on the fly for arbitrary off-the-shelf agentic LLMs. We formalize the agent harness as a composable, machine-generatable artifact governed by a fixed four-module protocol, and train JIT-Agent to customize harnesses for a given task at hand, repair harnesses for stable and reliable execution, and self-evolve by distilling performance signals from an expanding archive of prior harness configurations. Equipped with JIT-Agent as a harness helper, DeepSeek-V4-Flash surpasses GPT-5.6 on DeepSearchQA () and OdysseyBench (), while the already strong GLM-5.2 gains up to points. Across controlled evaluations, JIT-Agent-generated harnesses are performance-competitive with mature agent runtimes such as OpenCode and Claude Code and consistently improve multi-scale model families of DeepSeek V4, Mimo-V2.5, and Qwen3.6. To our knowledge, JIT-Agent is the first model purpose-built for just-in-time harness generation, establishing harness intelligence as a trainable, transferable, and compounding dimension of agent capability orthogonal to model scaling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Harness, LLM Agent

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XkMsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JIT-Agent is a trained 27B model which writes the software controlling another agent for each new task. It generates modules for memory, planning, actions, and tool selection. Training teaches it to customize this software, repair failures, and improve on earlier designs. At inference time, it generates three candidates, selects one using the model's own likelihood score, and executes it after checks and any needed repair. A streaming version saves useful designs for later tasks. Across nine benchmarks, it improves all 18 main comparisons against the same task model using ReAct. Comparisons with established harnesses also show competitive performance and lower reported execution cost.

Critical strengths

The improvements are broad and sometimes substantial. Both main task models improve on all nine benchmarks, including large planning gains. Only DeepSeek-V4-Flash supplies execution feedback during training, so the gains with GLM, Qwen, and Mimo task models support transfer beyond that training executor (Table 2; Appendix A.2; Figure 5).

The generated software is usually usable after repair. Table 4 reports 98.3% validity after at most one repair for Stage II and 97.9% for the final Stage III model. Against established harnesses, the full system wins four of six performance comparisons and has the lowest reported execution API cost in all six (Table 3). It does not win every performance comparison, but it is competitive beyond a basic ReAct baseline.

Critical weaknesses

The benefit of training is not fully separated from the supplied examples and design rules. Table 4 shows improvements across training stages, which is useful evidence. However, it starts after supervised training. The paper does not compare with the base generator or teacher given the same protocol and reference harnesses, or with simply selecting from the 13 seed harnesses. Those comparisons would show what the learned generator adds. The production-harness comparison also needs clearer details about versions, tool access, and adapters to establish equivalent conditions.

The cost comparison leaves out part of the system's work. The main tables count task-execution API cost. Appendix C.2 does measure generation time: roughly 52–58 seconds per rollout, or 9–10% of execution time, on two benchmarks. The remaining question is the generator's full monetary or compute cost, including its inputs and repairs. The streaming version also checks candidates against an incumbent and replays both on five earlier tasks. It is unclear whether the streaming cost curves include all of that work. This could affect some close cost comparisons.

Questions for authors
  1. How does the trained model compare with the base generator, the teacher, and selection from the seed bank when they receive the same information and comparable budgets?

  2. What is the complete cost of generating, repairing, and running the selected harness? Do the streaming cost curves include incumbent checks and all historical replays?

  3. How stable are the main comparisons across independent runs? Do the streaming gains hold under different task orders? This is especially relevant for the smaller test sets.

  4. Which versions and tool adapters were used for the established harnesses? Did they receive equivalent tools, information, and execution limits?

Reviewer_BgEnOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a just-in-time agent to construct harness for tasks. The JIT agent operates in the space of memory, planning, action, and capability orchestration. The paper further presents the training pipeline to cover customize, repair, and evolve harness stages

The paper demonstrates the performance improvements over LLM models and AOT harness. Furthermore, the detailed ablation study shows the incremental value from stop training at different stages.

Critical strengths

The paper clearly establish the motivation and articulates the sounding solution with sufficient technique details.

Critical weaknesses

The paper carefully isolates the training data to avoid leakage. However, in reality, the in-distribution data can be used to finetune LLM. So ablation the source of training data can be valuable.

Questions for authors

No

Reviewer_2oidOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JIT-Agent trains a dedicated model to generate task-adaptive agent harnesses for off-the-shelf language-model agents. Its Qwen3.6-27B generator produces executable code for memory, planning, action, and tool/skill orchestration under a fixed four-module protocol. Training combines SFT/DPO, repair imitation, and Evo-GDPO, with generator weights frozen during deployment. The evaluation reports improvements in all 18 same-backbone comparisons. Against five established runtimes, it leads on performance in four of six settings and has the lowest execution API cost in all six.

Critical strengths
  1. The executable protocol and diagnostic-driven repair training provide a concrete, testable reliability contribution. The four-module interface specifies state, tool exposure, and execution interactions. Stage II converts failed code and diagnostics into repair supervision with up to two repair rounds. In Table 4, repair training increases Valid@1 from 90.2% to 98.3%, an 8.1 percentage-point gain. This measures protocol and execution validity, rather than task correctness.

  2. Same-backbone comparisons across executors demonstrate practical transfer value. All 18 comparisons in Table 2 and all 24 transfer comparisons in Figure 5 improve.

Critical weaknesses
  1. The incremental benefit of JIT generation over reuse from the same archive remains unquantified. Table 3 and the task-specific control flows in Appendix D already support the system's practical value. The missing comparison is matched reuse with legitimate parameter binding under the same protocol, tools, archive, selection rule, and execution budget. Without it, the contribution of instance-specific synthesis remains difficult to separate from the learned, reusable harness prior. This does not imply that per-instance generation must universally outperform reuse.

  2. The independent contribution of the Evo-GDPO bonus remains unidentified; this is a supplementary mechanism-analysis suggestion. From Stage II to III, DSQA/AgentIF scores improve from 83.4/61.1 to 85.1/63.8, and all three streams in Figure 6 finish higher. These results support the combined training stage and experience retention, but do not isolate the bonus term. I do not treat a separate bonus ablation as an additional acceptance condition or as evidence against the demonstrated joint-system gains.

Questions for authors
  1. Please clarify the initial reference archive and reset policy for Tables 2–4. Section 4.2 and Appendix A.3 explain inference and pre-execution candidate selection; A.4 reserves terminal feedback for subsequent tasks. One configuration row per table would suffice: inference mode, initial archive source/snapshot, whether it contains only the 13 seeds or also training-retained designs, and reset granularity between tasks. This is a reporting clarification and requires no new experiment.

  2. Please define "execution attempts" in Table 9 and specify the corresponding baseline recovery rules. A single scored rollout in A.1 can contain internal recovery, so the reported 1.07/1.20 attempts and accumulated execution time are not inherently inconsistent. An existing multi-attempt timeline and both configurations could identify the attempt level, triggers, limits, failure handling, and final scoring rule. This is a protocol clarification; the current evidence does not establish asymmetric opportunities or best-of-attempt scoring.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.