- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors present DuetHarness, which jointly trains the policy and the harness during RL. They perform alternating training phases, finding a Nash equilibrium of a common-payoff game with their method. The method is evaluated on ALFWorld, WebShop, and DBBench. The method achieves the best (or tied for the best) results in nearly every category.
Critical strengths
- The paper is well-written and clearly explains the idea with good logical flow.
- The diagrams are very clear, especially Fig 1 and Fig 2.
- The results are good. The method outperforms baselines on a variety of datasets.
- The paper focuses on an important problem - improving harnesses and RL optimization through a joint optimization.
- The method is simple enough to be easily understood while representing a novel contribution.
- The paper includes a lot of detail, as well as code for reproducibility.
Critical weaknesses
None
Questions for authors
Can you add a couple of concrete examples of what the harness editor learned/changed to help build intuition of what it did to improve?
3 comments
Thank you for the information and addition to the appendix. I will maintain my score.
Thank you for your positive assessment
Dear Reviewer qWLa,
Thank you for the positive assessment and thoughtful feedback. We are glad that the additional examples helped clarify the learned harness edits and their impact. We appreciate your time and consideration throughout the review process.
Best regards,
The Authors
Response to Reviewer qWLa: Case studies of learned harness edits
We thank the reviewer for the positive assessment and for this helpful suggestion. Below, we give three concrete examples, one per benchmark, from the harnesses retained for testing.
Q1. Concrete examples of what the harness editor learned.
Example 1 (WebShop). Fixing clicks that do not match any button. In WebShop, a click only works if its text exactly matches a button on the page. Otherwise it is ignored but still costs a step. Under h0, the agent often clicks text that is not a button, such as
click[size]or a full product title, and many episodes run out of steps before buying. The editor learned a hook that checks each click before it is sent. If the click is close to a listed button, the hook changes it to that button. If not, the hook blocks it and tells the agent which buttons are available. For example, in one test task the agent needs a scented candle in the "voyager" option. The agent clicks the long product titleclick[Mer Sea Luxury Scented Candle in Tin - Voyager, 25 Hour Burn], and the hook changes it toclick[voyager], so the correct option is selected and the purchase succeeds. Together with the policy trained to use this hook, episodes that end without a purchase drop from 128 to 2 out of 500 (Table 11 (now Table 13)).Example 2 (ALFWorld). Going to the right place before acting. In ALFWorld, an action such as putting an object on a countertop only works when the agent is standing at that countertop. Under h0, the agent often tries to act from the wrong place, so the action fails and a step is wasted. For example, in the task "clean some knife and put it in countertop", the agent under h0 keeps issuing
move pan 1 to countertop 1from elsewhere and the episode ends after repeated failed actions. The editor learned a hook that catches such actions and turns them into the missing step. When the agent issuesmove knife 1 to countertop 1away from the countertop, the hook changes it togo to countertop 1, and the agent then completes the task. Together with the policy trained to use this hook, these failedmoveactions drop from 285 to 37, and success on the unseen split rises from 41.8% to 94.8% (Table 1).Example 3 (DBBench). Checking a change once before submitting. In DBBench, a task that modifies the database (INSERT, UPDATE, or DELETE) is judged by the final content of the table. Under h0, the agent often loses track of whether its change has already been applied. For example, in a task that asks to insert one record into the table
Athlete Performances, the agent under h0 issues the sameINSERTfour times, which leaves duplicate rows, and the task fails. The editor learned two hooks for such tasks. After a modification, the first hook asks the agent to check the change with aSELECTbefore submitting. Once thisSELECTreturns the new row, the second hook tells the agent that the change has been applied and that it should submit without repeating it. With these hooks, the agent inserts the record once, checks it with oneSELECT, and submits. Together with the policy trained to use them, these hooks are involved in 25 of the 34 test tasks that DuetHarness solves but the vanilla agent does not.These hooks were not written in one step. On WebShop, for example, the editor first added guards that blocked clicks on option names, but these guards also blocked some valid "Buy Now" clicks and hurt performance. In a later round, the editor deleted them and wrote the hook in Example 1 instead. This shows that the editor learns from the results of its own edits, and can remove a harmful hook rather than only adding new ones.
We have added these examples to Appendix K.