- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents APOLLO, a method for offline-to-online preference-based RL. The agent starts from offline data with preference labels. During online training it gets no reward and can only ask a limited number of questions of the form "which of these two clips is better?". The main contribution is the rule for choosing which pairs to ask about. It prefers pairs on which an ensemble of reward models is uncertain, and adds a Fisher-information term that favours pairs that are informative for the reward model and unlike earlier queries. The rule is combined with an existing flow-based, action-chunked policy. The paper also gives a bound on the reward model's preference error that holds when queries are chosen adaptively.
Critical strengths
-
The problem is well motivated and the idea is simple. When there is no reward online, deciding which pairs to label is the main lever, and the rule only needs a differentiable reward ensemble, so it can be combined with other policy learners. With the flow-based actor, the full system improves from offline to online on seven of eight sparse, long-horizon tasks, while the baselines stay much lower (Table 1).
-
The theory allows each query to depend on earlier labels. This is what happens in active querying, and the usual bounds that assume independent samples do not cover it. The paper handles this with a martingale argument, and it is clear that its comparison with random querying is only between upper bounds.
Critical weaknesses
-
It is not yet clear that this query rule is better than simpler ones. Without DOUBLE-TASK1, the full rule is about the same as No-IG and Disagreement, and there is no U^IG-only variant. Theorem 1 holds for any query rule, so the theory does not settle this either.
-
The setting relies on strong assumptions. All tasks are OGBench cube manipulation, labels are simulated, and the method needs 10,000 offline preference labels, about ten times the online budget, to start well (Fig. 5, Appendix D.12). It is unclear how it would work in a new domain or without a cheap, good offline start.
Questions for authors
- Could you run Random, Disagreement, No-IG and the full rule on the other four tasks, and add a rule that uses only U^IG? If the full rule still helps there, I would raise my score.
- On DOUBLE-TASK1, Random collapses with the flow policy but not with the Gaussian one. Appendix D.13 also suggests that the policy exploits errors in the learned reward during offline training. Does the same happen online, and does the query rule reduce it? Tracking the reward model's accuracy on new pairs during training would show this.