- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work is a methodology paper that proposes Behavioral Quotient Policy Optimization (BQPO). It addresses how states should be grouped when computing relative advantages for RL in interactive agents. BQPO introduces two components. Firstly, the behavioral quotient, which groups observations with similar observed action-transition structures. Second, the exact-routing, which dynamically chooses whether advantages should be normalised within exact-state groups or larger behavioral-equivalence groups.
Critical strengths
- The paper identifies an important problem in group-relative RL, which is to determine which experiences should serve as reference for advantage normalisation.
- The results comprise different environments and model sizes, which I appreciate, and there are several structural diagnostics that demonstrate that meaningful grouping occurs.
- I also appreciate that the runs were done over 3 seeds which would lead to more stable estimates for the RL results.
Critical weaknesses
- This paper is pretty hard to read and understand, in terms of the methodology. It is not really clear, intuitively why the method leads to better results.
- I think that the improvements are rather small (around 2-3%) for both ALFWorld and Webshop, where the variance isn't really reported. Without variance reporting It's hard to see whether the improvements manage yield genuine gains.
- The ablations are rather limited, and not within the main text. The ablations mainly consider random-partition which is intuitively pretty weak. I would ask the author to include more informative baselines as whether there may be simpler grouping heuristics.
- Beyond the ablations, although there is a mechanism analysis, this seems to be more descriptive of the training dynamics. The paper seems to lack convincing analyses and experiments to show that the method's mechanisms are working as intended, and are the cause of the gains. I.E. What is it about behavioral quotient/ exact-routing that is responsible for the gains? and can these be demonstrated.
Questions for authors
Questions include the weaknesses above. Besides those, I have a few additional here:
- Can the authors provide more naive baselines for the ablations (i.e. simpler grouping heuristics) to compare BQPO?
- Can the authors also improve and simplify the presentation so that it reads as more intuitive?