- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new approach to reward in the context of PbRL. The proposed method, BRM, uses the first two layers of the critic in a soft-actor critic (SAC) policy as input features to the reward model learned from preference labels over pairs of trajectories. In the experiments section, BRM is compared against several PbRL baselines and the various design decisions are evaluated to understand their value. There results suggest including features from the critic improve the performance of both the policy trained on the learned reward and the learned reward on the preference labeling task.
Critical strengths
The paper is well grounded in its writing and narrative arc (e.g. how the work is motivated and how readers are introduced to the problem). The experiments run dig deeper into what about including critic layers as input improves reward model performance.
Critical weaknesses
- There are key experiments results missing from the experiments section to truly understand the impact of the proposed method and to appropriately contextualize the key learning:
- The most important one is, there is no comparison to REED, which the authors call out as the most related of prior work to what they propose. The lack of a comparison to REED leaves the ablation in Section 5.4 rather weak. The conclusion from Section 5.4 is that extra capacity in the form of extra reward model parameters, random features, or actor features lead to similar gains as the critic features. However, evaluating the benefits of a learned representation via something like a reconstruction loss is not explored. Therefore, it is not clear if the gain is from having a representation that is learned from a somewhat related objective (i.e. the critic) or having a representation learned with some objective. The temporal consistency objective from REED would have been a reasonable comparison point to help fill out this ablation and understand the benefits of critic features.
- There are no experiments to understand if the critic features are beneficial simply because they are exposed to more data than the reward model is, or because the something about the critic objective is beneficial for learning more generalizable features. A comparison to REED would have helped here or an experiment where the number of preference features or the frequency of soliciting preference feedback is increased to see how much preference data is needed to match the performance of BRM.
- SAC performance on each of the tasks is not provided, so it is not clear how far BRM moves towards recovering performance achieved from the ground truth reward function.
- The extent to which BRM improves the reward model on truly OOD samples is not clear as the reward model is only evaluated on samples from SAC's replay buffer, which is where the preference samples the reward model is trained on are also sampled from. It is not possible to tell how similar the samples used in the "Does the reward model generalize beyond the preference distribution?" experiments are to the samples the reward model is trained on. This makes it difficult to know how much the critic features help with generalization.
- The authors claim that the skip connection allows the reward model to selectively pick to use the critic features based on whether they are mature enough. However, there are no experiments to test if this is actually the case. A simple experiment to look at this would have been to look at how much influence the critic features have at different training phases, e.g. m's.
- Not a critical weakness per se, but something that would be beneficial to address. There are awkward and unconventional turns of phrase, word usages, and sentence construction. The authors include using a LLM to help with paper refinement, so I am guessing these come from the LLM. However, it would help the reader to clean these up. Some examples, (1) over use of ":" that seem to have been swapped in where LLMs are prone to use m-dashes and (2) in "...the reward model is refit..." and "...the reward model is refit...", the word "refit" doesn't really have a meaning,
Questions for authors
- Why does combining BRM with other PbRL methods drop performance relative to BRM alone?
- What parts of PEBBLE are not used by BRM? You claim PEBBLE is the base algorithm, but then have an experiment that is PEBBLE+BRM, which is confusing.
- In PEBBLE the reward model quickly achieved 100% accuracy on the preference data and the authors had to hardcode a 97% accuracy training cut off. Does the BRM reward model exhibit similar behaviors?
- What is meant by "dual-critic ablation" on line 279? It would be helpful to have a little more detail so the reader can better understand the claim, "...no significant effect of decoupling the two."
- It seems odd that later layer critic features correlate more with the reward, but are worse as reward features. What is your intuition behind this tension?
4 comments
Thank you for your timely feedback. Please update your PDF to incorporate the information you have provided in this rebuttal (highlighted in a color of your choice). Additionally, please review the overall writing quality and clean up awkward and unclear constructions.
Let me know when you have made these changes, and then I will reconsider my score.
We thank the reviewer for their reply and for their constructive feedback throughout the discussion period.
We have updated the PDF, with the changes discussed in the rebuttal highlighted in red, including the REED comparison, SAC trained with the true reward as a reference, and the clarifications on the evaluation sets and the dual-critic ablation. We also revised parts of the text for clarity, including the examples the reviewer pointed out.
We will add a more detailed global comment listing the changes at the end of the discussion phase. We hope the revised version addresses the reviewer's concerns, and we would be grateful if they could reconsider their score. Thank you!
Thank you for your responses and updated draft. I am changing my score from 2 -> 3
We thank the reviewer for their detailed comments and concrete suggestions, which we address below.
W1 (REED comparison): We thank the reviewer for this important point. We agree that REED is an important comparison for this work. Per-task results on the 11 tasks used for the IQMs are in our answer to Reviewer quTf.
In aggregate, BRM reaches an IQM of 0.86 against 0.68 for REED, and the two 95% confidence intervals do not overlap. REED improves over QPA, so we believe a learned representation already helps the reward model. BRM remains clearly ahead of REED, which suggests that the critic's objective brings more than the self-predictive representation objective used by REED. We will add REED to Table 1.
W2 (more data vs. critic objective): We thank the reviewer for this constructive suggestion. REED gives a direct way to separate the two, since its representation is trained on the same replay buffer as the critic but with a different objective. As it remains clearly below BRM (W1), we believe the amount of data alone does not explain the gain.
W3 (SAC with ground-truth reward): We thank the reviewer for this suggestion. We will add SAC trained on the true reward as a reference line to the learning curves.
W4 (truly OOD evaluation): We thank the reviewer for this point. The SAC set does not come from the agent being trained, but from a separate SAC agent trained on the true reward, whose transitions the reward model never sees. Both methods use the same SAC buffer for each seed. The shift is visible in the accuracies in Table 3.
W5 (Skip connection): The skip connection is meant to let the reward model rely on the raw input when the features are less useful, and the design ablation suggests that it helps (IQM 0.79 without it, 0.87 with it, Table 9), though we agree this experiment does not isolate when each input is used. We thank the reviewer for this suggestion.
Q1 (combining BRM with other PbRL methods): We thank the reviewer for this question. BRM is not a standalone algorithm, as it changes the reward model's input and runs on top of a PbRL method. In Table 1, BRM runs on QPA, so "BRM alone" is QPA+BRM in Table 2. PEBBLE+BRM and RUNE+BRM are the same change on the PEBBLE and RUNE backbones, and each improves over its own base (six of seven and seven of seven tasks). The differences between the three combined versions follow the backbone rather than BRM, as QPA+BRM is higher on most DMC tasks, while RUNE+BRM is higher on the three MetaWorld tasks and walker-run.
Q2 (what parts of PEBBLE BRM uses): We thank the reviewer for pointing out this confusion. As explained in Q1, BRM's default backbone is QPA (Table 1). QPA and RUNE are themselves built on PEBBLE, so PEBBLE+BRM applies the same change directly to PEBBLE. We will clarify this in Section 5.1.
Q3 (reward model reaching 100% training accuracy): BRM keeps the same reward-training rule as its backbone, including PEBBLE's accuracy cutoff, so its reward model stops training at the same point as the baselines.
Q4 (dual-critic ablation): We thank the reviewer for asking for more detail, and we agree that our phrasing could be improved for clarity. Sharing features with the reward can also improve the Q-function, as in DeepMDP (Gelada et al., 2019), so the gain could come from a better critic rather than a better reward model. BRM-DC (Appendix D.2) separates the two by adding a critic whose only goal is to provide features for the reward model, but does not feed SAC's own critic and policy. This does not change performance (IQM 0.86 vs. 0.87 for BRM), which suggests the gain comes mainly from the reward model. The feature critic is still trained on the reward model's output, so "decoupling the two" refers only to SAC's critic. We will rephrase this in Section 4.2 and Appendix B.2.
Q5 (later layers more correlated, worse as features): We thank the reviewer for this question, and first clarify that BRM uses only the first layer of the critic. In Figure 7, the second layer often appears more correlated with the true reward, but its correlation rises and dips during feedback on door-unlock and drawer-open. In PbRL, such a dip can be critical, since a worse reward model can break the policy, and new queries then come from worse behavior, which makes it hard to recover. The critic's targets also change with each new reward model, and the second layer, closer to the output, follows these changes more than the first. The first layer then gives a more stable input across rounds, in line with Yosinski et al. (2014), where earlier features are more general and transfer better.
We look forward to hearing from you and providing any other clarifications during the discussion period. Thank you!