- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper deals with agents in a cooperative setting and tries to answer the question of when an agent should try to shape another agent's future behavior. At each step, the focal agent (Main) predicts whether a shaping action changes the teammate's (Helper's) belief and therefore Helper's behavior. It evaluates whether Helper picks the correct complementary task, and the team completion time, under the shaping action, and compares this to task-direct behavior. It then decides whether to shape. Based on Helper's realized response, Main replans the next step. This approach is shown to be more effective when compared to belief-based cooperative planning (BSCP) baselines. The WSP framework is evaluated on 400 GridWorld scenes and 87 household instances derived from PARTNR, both with scripted Bayesian teammates. WSP outperforms the baselines of open and closed loop BSCP: it reaches 97.0% complementary-task accuracy in GridWorld (vs. 95.5% for Open-Loop and 96.25% for Closed-Loop BSCP) and 62.1% in Household (vs. 36.8% and 46.0%).
Critical strengths
- Just knowing or influencing a change in belief without influencing a change in action does not help us in completing the task. Therefore the premise of the paper, influencing the teammate's belief only when it leads to a change in action, makes it clearly motivated.
- The 2x2 comparison of Open-Loop BSCP, Closed-Loop BSCP, Timed Open-Loop BSCP and WSP lays out transparently the separation of contribution of timing from contribution of replanning.
Critical weaknesses
- The paper uses a fixed, scripted Bayesian teammate (Helper). A Bayesian teammate needs a prior, and its change in behavior follows directly from its belief update by design. If an LLM-based agent (or a human) acted as the teammate, it would provide some interesting insights on whether the approach generalizes beyond scripted teammates
- Both the open and closed BSCP are designed by the authors as baselines. The paper does not compare against legibility-based planners or other prior work.
- In the Household task, most of WSP's gain comes from choosing when to shape: Timed Open-Loop, which does not replan, already reaches 59.8%. Replanning only adds 2.3 points (about 2 of 87 instances), and no confidence interval is reported for this difference. The evidence for closed loop replanning being beneficial is weak.
- The evaluation size in the PARTNR-derived Household setting is small (87 instances from 30 episodes), and some confidence intervals are wide (e.g., [7.79, 27.16] for the 16.09-point gain over Closed-Loop BSCP).
Questions for authors
- Have any experiments been run with a teammate other than the assumed model (e.g., an LLM agent or a human study)?
- Have any experiments been run with a legibility-maximizing baseline or a published planner as a baseline?
- Are results on more Household episodes available?
- What is the confidence interval for the accuracy difference between WSP and Timed Open-Loop?