- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PIPA, a framework for personalized proactive agents. It extracts interaction skills from user histories, maintains Bayesian beliefs about user preferences, and uses profile-conditioned simulation to decide when and how to intervene. Experiments across three LLM backbones and several benchmarks show improvements in downstream agent performance.
Critical strengths
-
The problem is interesting and practically relevant. Modeling not only user preferences but also uncertainty about those preferences is important for proactive agents.
-
The experiments are fairly comprehensive, covering multiple backbones, benchmarks, and agent harnesses. The profile-shuffling and uncertainty ablations provide useful evidence for the proposed approach.
Critical weaknesses
-
The offline evaluation protocol needs clarification. PIPA updates its beliefs based on user feedback after an intervention. However, the experiments rely on logged interactions. It is unclear how feedback is obtained when the selected intervention differs from the recorded action. This makes it difficult to assess whether the claimed online adaptation is validated.
-
The compute comparison is questionable. The paper claims compute-matched uncertainty ablations, but Appendix C indicates that PIPA-Mean uses one simulator call per action, while full PIPA uses 16 posterior samples per action. This difference makes it difficult to attribute the improvement solely to uncertainty modeling. The evaluation also involves relatively few users, which limits confidence in the generality of the results.
Questions for authors
-
How is user feedback handled when PIPA selects an action different from the one recorded in the offline interaction history?
-
How is PIPA-Mean compute-matched to full PIPA given the different numbers of simulator calls?