- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose JET, a two-stage method for improving an agent’s program when true task rewards are unavailable. First, an LLM develops a judge using examples with known rewards. The judge is then frozen and provides scores and feedback to guide changes to the agent’s program on new tasks. The main experiment transfers from clothing to non-clothing shopping tasks in WebShop. On unseen tasks, the authors report about 13% higher reward than a fixed-rubric judge from an un-evolved starting program, and 4% higher from a previously optimized program. PushT provides a control where the judge can calculate the reward exactly.
Critical strengths
-
The paper addresses a useful problem, improving an agent when its true performance can't be directly scored during deployment. The experimental protocol keeps adaptation separate from final evaluation. Reward information is removed from the interaction records, and a separate set of unseen tasks is held out. The authors also acknowledge that limited replication makes the smaller gains uncertain.
-
The main cold-start result is promising. JET reaches 0.7915 on unseen tasks, compared with 0.7008 for the fixed-rubric judge. Even perfect final selection among the fixed-rubric judge’s candidates would close only about 9.7% of the adaptation gap on the adaptation tasks, which suggests that the guidance affects which programs are generated. Recalibrating the fixed rubric doesn't match the evolved judge’s prediction accuracy, and evolving the judge’s code beats changing only its prompt.
Critical weaknesses
-
My main concern is that the gap between the main experiments and the ablations is not explained. Appendix F says it keeps the same starting agent, task sets, and search protocol, but the full JET condition reaches only 0.6984 on adaptation tasks and 0.6869 on unseen tasks, compared with 0.7797 and 0.7915 in Tables 1-2. It's not clear whether Appendix F used a different judge or if the gap reflects variation between searches. Without the fixed-rubric baseline in that series we can't tell if JET’s advantage holds there. The paper also reports no confidence intervals or tests of differences between methods. This is important mostly for the smaller warm-start gain and for textual feedback, whose benefit the conclusion calls uncertain, while Section 5.4 is more positive. It's also it's not clear how many judges were developed, and repeating searches with one frozen judge wouldn't show how reliably the first stage produces a useful judge.
-
The judge comparisons and the evidence for transfer could both be stronger. The seed judge differs from the fixed-rubric baseline and has lower source-holdout error than the fixed rubric, but guides search poorly in Appendix F. This raises the question of how well source prediction error, the criterion used to select the judge, predicts usefulness during search. Evolving the fixed-rubric judge would be a more direct comparison, and the LLM-free version of the evolved judge, which has slightly lower prediction error than the full version, is not tested as an online guide. On transfer, PushT is an exact-judge control, leaving WebShop as the main test of an imperfect judge, and both environments keep the same scoring rule across source and target tasks. This is a legitimate setting, but broader claims would need more evidence.
Related work that is worth reading: Eureka (Ma et al., ICLR 2024) and Text2Reward (Xie et al., ICLR 2024) use LLMs to generate executable reward code. VeRO (Ursekar et al., ICML 2026) studies coding agents that optimize agent harnesses, and HarnessOpt-Bench (Ursekar et al., 2026) evaluates harness optimization on a held-out split hidden during search and finds that gains vary substantially across tasks and seed regimes. A brief comparison would help clarify JET’s contribution.
Questions for authors
-
Does Table 7 use the same frozen judge as Tables 1-2, and what explains its lower results? Please add the fixed-rubric baseline to that series and say how many judges were independently developed.
-
Could you report uncertainty for the differences between methods, especially for warm starts and textual feedback? Also it would be helpful to confirm the warm-start held-out value of 0.6951 in Table 2, which matches the adaptation mean in Table 1.
-
Could the proposer access the environments’ reward code in either stage? Can you provide the seed judge, evolved judge, and fixed rubric, or describe their scoring rules?
-
How does the LLM-free evolved judge perform as an online guide, and would evolving the fixed-rubric baseline give similar gains?
-
Are the adaptation and held-out tasks disjoint in products as well as session IDs, and if not, how much do they overlap?
We thank reviewer CXr9 for their time and detailed reading. We address the five questions individually below.
Q1. Does Table 7 use the same judge and protocol? What about the fixed-rubric baseline and independent judge developments?
Thank you for identifying this inconsistency. Table 7 uses a different frozen judge and a smaller protocol, so the values are not matched replications. Appendix F incorrectly describes these protocols as matching. We will correct the description and reconcile the table's means and SDs with the contributing campaigns.
Q2. Uncertainty for differences between methods, and confirmation of the warm held-out value
Thank you for this suggestion. We will additionally report confidence intervals for between-method reward differences, using matched independent evolution campaigns, and revise the corresponding conclusions accordingly.
Q3. Could the proposer access reward code in either stage? What do the judges compute?
Thank you for asking about reward access and the judges' implementation. In the new WebShop experiments, the proposer cannot read the benchmark reward code in either stage. Judge development uses labeled source examples; solver evolution uses target trajectories with reward information removed.
The three WebShop judges work as follows:
We will add representative code excerpts and judging prompts for the seed, fixed-rubric, and evolved judges to the appendix, illustrating how each maps trajectory evidence to its predicted score.
Q4. How does the deterministic evaluator perform online, and would evolving the fixed rubric give similar gains?
Using a deterministic evaluator. We will extend this comparison to online solver evolution. Our completed offline evaluation shows that the judge without new LLM calls retains similar source-holdout prediction accuracy to the full judge (MAE 0.1153 versus 0.1168). The online comparison will test whether this accuracy translates into effective guidance during solver evolution.
Evolving the fixed-rubric judge. We tested this directly. Starting from the fixed rubric, executable evolution reduces observed mean target MAE from 0.2879 to 0.1675, compared with 0.2147 when only the prompt is optimized. The two learned conditions use the same starting judge, source data, and development budget. Thus, the improvement in prediction accuracy also appears when the fixed-rubric baseline itself is the starting point.
Q5. Are adaptation and held-out tasks disjoint in products as well as session IDs?
The historical main cohorts are session-disjoint but not fully requested-product-disjoint. The 500 adaptation tasks contain 482 distinct requested product identifiers, and the 500 reporting tasks contain 483. There are 23 shared requested-product identifiers, affecting 28 reporting tasks, or 5.6%. Thus, 94.4% of reporting tasks request products absent from the adaptation goals.
Thank you again for your thoughtful questions and constructive suggestions, which will help us strengthen the analysis and presentation of our work.