acceptodds
Under review as a conference paper at ICLR 2027

JET: Judge-Guided Evolution at Test-Time for Agent Harnesses

Abstract

An agent’s executable program governs how it uses tools, processes observations, and responds to failures. Evolving this program at test time can help adaptation, but deciding which changes to retain is difficult when true rewards are unavailable. Execution traces provide evidence of agent behavior, yet interpreting that evidence requires a judge that remains useful as tasks and candidate programs change. We introduce Judge-Guided Evolution at Test Time (JET), which evolves an executable judge on labeled source trajectories, then freezes and transfers it to guide target-side program evolution. The judge supplies scores and diagnostic feedback without target evaluator access or model-weight updates. On unseen WebShop tasks, JET achieves approximately 13% higher mean reward than fixed-rubric guidance when evolution begins from an unevolved program (cold start) and 4% higher when it begins from one already optimized on source tasks (warm start), with a 36% relative improvement in cold-start exact success. An exact-judge control on PushT, where the judge reconstructs the scoring rule from observations, shows that without judge error, program search becomes the bottleneck. Analyses identify useful reward-prediction logic in the evolved code and show that better final selection alone cannot explain the gains. These results support executable judge transfer for program adaptation under evaluator-preserving task shifts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, test-time adaptation, agent harness optimization, evaluator transfer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CXr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose JET, a two-stage method for improving an agent’s program when true task rewards are unavailable. First, an LLM develops a judge using examples with known rewards. The judge is then frozen and provides scores and feedback to guide changes to the agent’s program on new tasks. The main experiment transfers from clothing to non-clothing shopping tasks in WebShop. On unseen tasks, the authors report about 13% higher reward than a fixed-rubric judge from an un-evolved starting program, and 4% higher from a previously optimized program. PushT provides a control where the judge can calculate the reward exactly.

Critical strengths
  1. The paper addresses a useful problem, improving an agent when its true performance can't be directly scored during deployment. The experimental protocol keeps adaptation separate from final evaluation. Reward information is removed from the interaction records, and a separate set of unseen tasks is held out. The authors also acknowledge that limited replication makes the smaller gains uncertain.

  2. The main cold-start result is promising. JET reaches 0.7915 on unseen tasks, compared with 0.7008 for the fixed-rubric judge. Even perfect final selection among the fixed-rubric judge’s candidates would close only about 9.7% of the adaptation gap on the adaptation tasks, which suggests that the guidance affects which programs are generated. Recalibrating the fixed rubric doesn't match the evolved judge’s prediction accuracy, and evolving the judge’s code beats changing only its prompt.

Critical weaknesses
  1. My main concern is that the gap between the main experiments and the ablations is not explained. Appendix F says it keeps the same starting agent, task sets, and search protocol, but the full JET condition reaches only 0.6984 on adaptation tasks and 0.6869 on unseen tasks, compared with 0.7797 and 0.7915 in Tables 1-2. It's not clear whether Appendix F used a different judge or if the gap reflects variation between searches. Without the fixed-rubric baseline in that series we can't tell if JET’s advantage holds there. The paper also reports no confidence intervals or tests of differences between methods. This is important mostly for the smaller warm-start gain and for textual feedback, whose benefit the conclusion calls uncertain, while Section 5.4 is more positive. It's also it's not clear how many judges were developed, and repeating searches with one frozen judge wouldn't show how reliably the first stage produces a useful judge.

  2. The judge comparisons and the evidence for transfer could both be stronger. The seed judge differs from the fixed-rubric baseline and has lower source-holdout error than the fixed rubric, but guides search poorly in Appendix F. This raises the question of how well source prediction error, the criterion used to select the judge, predicts usefulness during search. Evolving the fixed-rubric judge would be a more direct comparison, and the LLM-free version of the evolved judge, which has slightly lower prediction error than the full version, is not tested as an online guide. On transfer, PushT is an exact-judge control, leaving WebShop as the main test of an imperfect judge, and both environments keep the same scoring rule across source and target tasks. This is a legitimate setting, but broader claims would need more evidence.

Related work that is worth reading: Eureka (Ma et al., ICLR 2024) and Text2Reward (Xie et al., ICLR 2024) use LLMs to generate executable reward code. VeRO (Ursekar et al., ICML 2026) studies coding agents that optimize agent harnesses, and HarnessOpt-Bench (Ursekar et al., 2026) evaluates harness optimization on a held-out split hidden during search and finds that gains vary substantially across tasks and seed regimes. A brief comparison would help clarify JET’s contribution.

Questions for authors
  1. Does Table 7 use the same frozen judge as Tables 1-2, and what explains its lower results? Please add the fixed-rubric baseline to that series and say how many judges were independently developed.

  2. Could you report uncertainty for the differences between methods, especially for warm starts and textual feedback? Also it would be helpful to confirm the warm-start held-out value of 0.6951 in Table 2, which matches the adaptation mean in Table 1.

  3. Could the proposer access the environments’ reward code in either stage? Can you provide the seed judge, evolved judge, and fixed rubric, or describe their scoring rules?

  4. How does the LLM-free evolved judge perform as an online guide, and would evolving the fixed-rubric baseline give similar gains?

  5. Are the adaptation and held-out tasks disjoint in products as well as session IDs, and if not, how much do they overlap?

1 comment

We thank reviewer CXr9 for their time and detailed reading. We address the five questions individually below.

Q1. Does Table 7 use the same judge and protocol? What about the fixed-rubric baseline and independent judge developments?

Thank you for identifying this inconsistency. Table 7 uses a different frozen judge and a smaller protocol, so the values are not matched replications. Appendix F incorrectly describes these protocols as matching. We will correct the description and reconcile the table's means and SDs with the contributing campaigns.

Q2. Uncertainty for differences between methods, and confirmation of the warm held-out value

Thank you for this suggestion. We will additionally report confidence intervals for between-method reward differences, using matched independent evolution campaigns, and revise the corresponding conclusions accordingly.

Q3. Could the proposer access reward code in either stage? What do the judges compute?

Thank you for asking about reward access and the judges' implementation. In the new WebShop experiments, the proposer cannot read the benchmark reward code in either stage. Judge development uses labeled source examples; solver evolution uses target trajectories with reward information removed.

The three WebShop judges work as follows:

  • Seed judge: asks an LLM to assess the trajectory using general judging instructions.
  • Fixed-rubric judge: asks an LLM to assess it using shopping-specific instructions. This differs from the original seed judge.
  • Evolved judge: adds code that extracts the purchased item, selected options, and price, checks them against the request, and combines these features with the LLM assessment through source-evolved scoring and calibration rules.

We will add representative code excerpts and judging prompts for the seed, fixed-rubric, and evolved judges to the appendix, illustrating how each maps trajectory evidence to its predicted score.

Q4. How does the deterministic evaluator perform online, and would evolving the fixed rubric give similar gains?

Using a deterministic evaluator. We will extend this comparison to online solver evolution. Our completed offline evaluation shows that the judge without new LLM calls retains similar source-holdout prediction accuracy to the full judge (MAE 0.1153 versus 0.1168). The online comparison will test whether this accuracy translates into effective guidance during solver evolution.

Evolving the fixed-rubric judge. We tested this directly. Starting from the fixed rubric, executable evolution reduces observed mean target MAE from 0.2879 to 0.1675, compared with 0.2147 when only the prompt is optimized. The two learned conditions use the same starting judge, source data, and development budget. Thus, the improvement in prediction accuracy also appears when the fixed-rubric baseline itself is the starting point.

Q5. Are adaptation and held-out tasks disjoint in products as well as session IDs?

The historical main cohorts are session-disjoint but not fully requested-product-disjoint. The 500 adaptation tasks contain 482 distinct requested product identifiers, and the 500 reporting tasks contain 483. There are 23 shared requested-product identifiers, affecting 28 reporting tasks, or 5.6%. Thus, 94.4% of reporting tasks request products absent from the adaptation goals.

Thank you again for your thoughtful questions and constructive suggestions, which will help us strengthen the analysis and presentation of our work.

Reviewer_Lzn3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Judge-Guided Evolution at Test Time (JET), a framework for improving agent programs in settings where ground-truth rewards are unavailable during test-time adaptation. JET adopts a two-stage approach. First, it evolves an executable judge on labeled source trajectories, optimizing its reward prediction accuracy. The resulting judge is then frozen and transferred to target tasks, where it provides numerical scores and textual diagnostics to guide iterative agent program evolution without access to target ground-truth rewards or model-weight updates.

The authors evaluate JET on WebShop and PushT under source-to-target distribution shifts, comparing it against fixed-rubric judges and other baselines. The results show that JET improves agent performance in both cold-start and warm-start settings, with gains also observed on held-out target tasks. Additional analyses examine judge prediction accuracy, candidate selection versus generation quality, and the contributions of executable judge evolution, textual diagnostics, and score adjustments.

Critical strengths
  1. Comprehensive experimental evaluation: The paper provides a relatively thorough experimental evaluation across two benchmarks, WebShop and PushT, covering both cold-start and warm-start settings. The authors also evaluate the evolved programs on held-out target tasks and conduct multiple independent evolution runs, which helps assess the effectiveness and generalization of the proposed approach within the studied settings.

  2. Detailed analysis and ablation studies: Beyond reporting final task performance, the authors provide extensive analyses of judge accuracy, selection regret, and candidate generation quality. In particular, the analysis distinguishing improvements in candidate selection from improvements in candidate generation is informative. The ablation studies further investigate the contributions of executable judge evolution, textual diagnostics, and numerical score adjustments, providing useful insights into the effectiveness of individual components.

Critical weaknesses

The paper is motivated by the difficulty of obtaining ground-truth rewards in real-world agent adaptation and proposes transferring a judge evolved on labeled source trajectories to reward-unavailable target tasks. However, I have two major concerns regarding whether the experimental setting adequately supports this motivation and whether the proposed approach fully addresses the underlying problem.

  1. First, the experimental settings do not convincingly reflect the challenges described in the motivation. Both WebShop and PushT provide well-defined, automatically computable rewards through their benchmark evaluators, whose outputs are artificially withheld during adaptation. It is therefore unclear whether these benchmarks adequately represent scenarios where obtaining reliable evaluation signals is genuinely difficult. Moreover, the source-to-target distribution shifts appear relatively limited from the perspective of judge generalization. For instance, WebShop transfers across product categories while preserving the same underlying evaluator, and PushT retains an identical geometric evaluation rule that can be exactly reconstructed from observable states. While these shifts may introduce challenges for agent adaptation, they do not necessarily present comparably challenging generalization requirements for the judge. Consequently, the experiments provide limited evidence of the proposed approach's effectiveness under more challenging reward-unavailable settings or substantial shifts in the distribution of behaviors that the judge must evaluate.

  2. Second, the paper does not sufficiently justify why the evolved judge must remain frozen during target-side adaptation. The proposed framework already has access to a collection of target tasks and their execution trajectories. Even without ground-truth rewards, such information could potentially help the judge. However, the authors exclusively evolve the judge on source trajectories and do not investigate whether incorporating unlabeled target-side information could further improve evaluation quality and subsequent agent evolution. This raises the question of whether transferring a completely frozen judge is the most appropriate solution to the stated problem, rather than an unnecessarily restrictive design choice.

Questions for authors
  1. Could the authors consider evaluating JET in more realistic scenarios that better align with the paper's motivation, particularly settings where ground-truth rewards are genuinely difficult or impractical to obtain? Such experiments would help establish the practical relevance of the proposed approach and provide stronger evidence of judge generalization under more challenging distribution shifts.

  2. Could the authors explore a reasonable approach to further evolving or adapting the judge using unlabeled target-task trajectories, without access to target ground-truth rewards? Since the framework already collects target-side execution trajectories during agent program evolution, it would be valuable to investigate whether this information could improve the judge's understanding of target tasks. A comparison with the current frozen-judge approach would help clarify the benefits and limitations of source-only judge evolution.

1 comment

We thank reviewer Lzn3 for their time and for examining the deployment motivation and the decision to freeze the evaluator.

Q1. Evaluation in more realistic reward-unavailable settings

The setting studied by JET is adaptation without access to target reward labels, rather than an environment in which an external evaluator cannot compute rewards. Withholding benchmark rewards provides a controlled way to evaluate transferred supervision while retaining ground truth for independent post-hoc measurement.

Keeping the underlying objective fixed isolates a particular transfer question: can evaluation learned from source behaviors remain useful for new tasks and behaviors under the same objective? WebShop tests imperfect evaluation across task categories. PushT serves a different role: its observable geometry permits exact reward reconstruction, making it a control for the distinction between accurate evaluation and successful program search.

We also quantified requested-product overlap. In the historical 500-task reporting cohort, 472 tasks, or 94.4%, request products absent from the adaptation goals.

We thank the reviewer for highlighting the importance of evaluation in settings where reliable rewards are genuinely difficult to obtain. We will incorporate this perspective into the revised discussion, clarifying that the current experiments study controlled transfer under withheld benchmark rewards, and identifying deployment studies with independent human assessment as an important next step.

Q2. Adapting the evaluator using unlabeled target trajectories

We thank the reviewer for this valuable suggestion. Unlabeled target trajectories could help the judge adapt to target-specific formats and behaviors. Our current design freezes the judge to maintain a stable evaluation objective during solver evolution: candidate improvements are measured against the same evaluator, without conflating solver changes with changes in the evaluation rule. Freezing also isolates the contribution of source-to-target evaluator transfer. Additionally, we will clarify this rationale and discuss target-aware judge adaptation as a promising extension, rather than imply that a frozen judge is necessarily optimal.

We appreciate your careful reading and valuable feedback, which have helped us identify opportunities to clarify and strengthen the paper.

Reviewer_ZNbeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JET, a framework that evolves an executable judge using labeled source trajectories and transfers it to guide test-time agent program evolution without access to target rewards. Experiments on WebShop and PushT demonstrate improvements over fixed-rubric judges, supported by analyses of judge accuracy and program selection.

Critical strengths
  1. Well-motivated problem. The paper addresses an important challenge in test-time agent evolution, where ground-truth rewards may be unavailable and reliable evaluation is needed to guide program optimization.
  2. Detailed empirical analysis. The paper includes cold-start and warm-start experiments, held-out evaluations, and useful ablations that help distinguish improvements in judge accuracy, candidate selection, and program generation.
Critical weaknesses
  1. Insufficient comparisons with learned reward models. The primary fixed-rubric baseline does not benefit from source-label optimization, unlike JET. Although the authors include prompt optimization and calibration controls, comparisons with stronger source-trained reward models under matched supervision would help establish the benefits of executable judge evolution.

  2. Unclear experimental stability. The experiments use only five independent campaigns, and some improvements exhibit considerable variance. Moreover, the cold-start evolved judge achieves 0.7797 in Table 1 but only 0.6984 in the separate ablation study (Table 7), despite apparently comparable evaluation settings. Could the authors clarify what accounts for this discrepancy and provide stronger evidence that the improvements are robust across independently evolved judges?

Questions for authors

See the weakness

1 comment

We thank reviewer ZNbe for their time, careful reading, and thoughtful questions about source-supervised comparisons and experimental stability. Below, we address your questions:

Q1. Comparison with learned reward models under matched source supervision

Thank you for suggesting this comparison. We conducted additional experiments comprising completed independent JET developments, prompt-only developments and reward-model training. We report mean ± sample SD across all completed developments on identical evaluation inputs and matched source labels.

JET achieves the lowest observed target prediction error among the tested evaluators, with mean MAE approximately 17% below the source-trained reward model and 22% below prompt-only optimization:

EvaluatorSource-holdout MAE ↓Target MAE ↓
Fixed-rubric judge0.22470.2879
Source-trained reward model0.1587 ± 0.00720.2018 ± 0.0249
Prompt-only optimization0.1568 ± 0.00540.2147 ± 0.0121
JET0.1480 ± 0.00040.1675 ± 0.0024

Comparison with a source-trained neural reward model. Following your suggestion, we extended the comparison to a neural reward model based on Qwen3-4B, fine-tuned with LoRA and a scalar reward head to predict trajectory rewards from the same source-labeled data. The neural baseline learns reward prediction through weight training, whereas JET develops its evaluator through program evolution. On the saved target trajectories, JET achieves approximately 17% lower observed mean MAE (0.1675 versus 0.2018). Both evaluators are selected using source validation and frozen before target scoring. This provides evidence that executable judge evolution is an effective alternative for transferring reward prediction learned from source supervision.

Q2. Experimental stability and the 0.7797 versus 0.6984 discrepancy

Thank you for identifying this inconsistency. The values are not matched replications. Appendix F incorrectly describes these protocols as matching. We will correct the description and reconcile the table's means and SDs with the contributing campaigns.

Once again, thank you for your thoughtful questions and constructive suggestions, which will help us strengthen the analysis and presentation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.