- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper explores whether Q-functions learned separately for different tasks can be combined to recover a shared dynamics model and plan for new tasks. Q-Map Fusion tackles this via Bellman inversion, using frozen Q-functions and known task definitions without revisiting the training data. The paper also analyzes when these constraints are sufficient for useful predictions (as a straightforward bounded analysis for the limits of the proposed methods!), even if the dynamics cannot be fully identified. Experiments in four environments examine the benefits of combining different tasks and how Q-value errors affect the results.
Critical strengths
-
This is my first time reading this topic in the RL field, and I found it pretty interesting. The intuition is straightforward to understand. The resulting conclusion/solution is also somehow "expected": the usefulness of this kind of Q-fusion for new tasks depends on a related, rank-like compositional feature, namely how well the source-task measurements can represent the predictions needed for the new tasks. I like that the paper makes this intuition more concrete and distinguishes useful prediction from recovering the full dynamics. The theory part is also easy to follow.
-
The experiments are complete regarding the things the authors want to prove. They cover both finite-state and continuous-state settings, and examine model recovery as well as performance on new tasks. The matched-budget comparisons and Q-value perturbation experiments are particularly helpful for understanding when different source tasks provide useful additional information.
Critical weaknesses
My two main concerns are as follows. Both come from a somewhat mundane practical perspective, but I find them quite serious honestly.
-
The assumption about the available source Q-functions seems unrealistic to me. In most practical cases, we simply do not have these Q-functions. A less demanding starting point would be to assume access to a good, or even idealized optimal, policy from a source MDP. For example, one could make such an assumption about a policy obtained through imitation learning and play some math upon it (e.g. Residual Q Learning/ Residual-MPPI). But having sufficiently accurate Q-functions for several different MDPs sharing the same dynamics already sounds like a rather demanding setup. In what practical scenario would we naturally have this collection of Q-functions and then want to perform this kind of fusion? Why would it not be more straightforward to take the new MDP and simply run RL on it?
-
More fundamentally, why do we need this? In a simulated MDP, the transition mechanism is already available and is somehow "free." Even for a more complex physics simulation, given the speed of modern simulators, we could often use the simulator itself as the transition model. What practical advantage would recovering that model from Q-functions provide? In the real world, I would expect obtaining reliable Q-functions for multiple tasks to be substantially more expensive than learning the dynamics directly. This brings me back to the first concern: it is difficult to see a natural setting in which the required Q-functions are available, the dynamics are not, and Q-fusion is the most useful next step.
Despite the mundane concerns, I do like the math games in the RL fields, so I still lean towards acceptance. It would be much better if you can provide some practical cases to convince me that it is not only interesting but also useful.
Questions for authors
See Weakness.