- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes BDGxRL, a novel framework that leverages Diffusion Schrödinger Bridge (DSB) to align source transitions with target-domain dynamics encoded in offline demonstrations. The method first transforms transtions collected from source domain to target domain, and then uses a reward estimator to calculate reward from transitions. The author claims that proposed method solves two problems: 1. the target domain data are usually difficult to collect and 2. the data do not need reward annotation.
Critical strengths
- This paper studies a setting where target domain data and reward is hard to collect, which is practical under real-world case.
- The simulation results show the proposed method outperforms the baselines, which proves the effectiveness of the proposed method. The ablation experiments show the different components contribute to the final reported results.
Critical weaknesses
- The problem formulation relies on unjustified assumption about reward model and state representations. The reward estimator takes the form of , which is not common in RL research where reward model is usually set to $R(s, a, s`)R_{max}s_t$, but the corresponding transitions are considered collected from target domain. This implies that only transition function is different from source domain and target domain, while the state distribution is the same. This hypothesis does not hold in most applications. For example, if the length of the robot body is different, the height of head can vary, which may probably affect the state representation.
- The motivation of applying DSB for distribution adaptation is not clearly claimed. DSB can be a good tool for distribution matching, but the paper does not explain its advantages over other alternatives like MLP, transformer or diffusion model. Futhermore, if the performance improvements simply come from DSB over other technics, the contribution of this paper is not sufficiently proved.
Questions for authors
- Can the author justify why in the same transition, the previous state comes from source domain while the next output state comes from target domain? Is there implicit assumption that the source domain and target domain shares the same state distribution and only differs in transition model?
- What exactly is and is it a bound on the true target reward, the source reward, or the learned reward model? Please explain how action-dependent reward terms are handled and add the reward-model error to the value bound if the learned reward differs from the target reward.
- Can the author provide results on other environments rather than Walker2d and HalfCheetah? And it would be better if the method can be adapted to zero-shot generalization of unknown domain.