acceptodds
Withdrawn from ICLR 2027

Q-Map Fusion: Recovering Shared Dynamics from Separately Learned Values

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Value functions in reinforcement learning are tied to particular rewards, yet tasks in the same environment share the dynamics that give rise to those values. We study how knowledge acquired separately across tasks can be combined after training to support new decisions. Building on Bellman inversion, Q-Map Fusion jointly fits a shared transition model to the constraints supplied by multiple Q-functions, then plans under new rewards and termination rules. Recovery uses the source values and known task definitions, requiring no training trajectories, no updates to the Q-functions, and no additional environment interaction. The usefulness of an additional task depends on the ambiguities its values resolve and the accuracy with which they are learned. For finite-state models, our analysis characterizes when the joint constraints control specified one-step predictions even when the full transition model remains unidentified. Experiments in FourRooms, MountainCar, Pendulum, and Reacher demonstrate effective model recovery and planning for held-out tasks across the evaluated tabular and continuous-control settings. Separate matched-budget and perturbation studies show that the benefit of additional values depends on both complementary task information and robustness to value errors. We hope this perspective encourages the reuse of learned values as shared environment knowledge for decisions beyond their original rewards.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning,Q-Map functions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q2ebOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores whether Q-functions learned separately for different tasks can be combined to recover a shared dynamics model and plan for new tasks. Q-Map Fusion tackles this via Bellman inversion, using frozen Q-functions and known task definitions without revisiting the training data. The paper also analyzes when these constraints are sufficient for useful predictions (as a straightforward bounded analysis for the limits of the proposed methods!), even if the dynamics cannot be fully identified. Experiments in four environments examine the benefits of combining different tasks and how Q-value errors affect the results.

Critical strengths
  1. This is my first time reading this topic in the RL field, and I found it pretty interesting. The intuition is straightforward to understand. The resulting conclusion/solution is also somehow "expected": the usefulness of this kind of Q-fusion for new tasks depends on a related, rank-like compositional feature, namely how well the source-task measurements can represent the predictions needed for the new tasks. I like that the paper makes this intuition more concrete and distinguishes useful prediction from recovering the full dynamics. The theory part is also easy to follow.

  2. The experiments are complete regarding the things the authors want to prove. They cover both finite-state and continuous-state settings, and examine model recovery as well as performance on new tasks. The matched-budget comparisons and Q-value perturbation experiments are particularly helpful for understanding when different source tasks provide useful additional information.

Critical weaknesses

My two main concerns are as follows. Both come from a somewhat mundane practical perspective, but I find them quite serious honestly.

  1. The assumption about the available source Q-functions seems unrealistic to me. In most practical cases, we simply do not have these Q-functions. A less demanding starting point would be to assume access to a good, or even idealized optimal, policy from a source MDP. For example, one could make such an assumption about a policy obtained through imitation learning and play some math upon it (e.g. Residual Q Learning/ Residual-MPPI). But having sufficiently accurate Q-functions for several different MDPs sharing the same dynamics already sounds like a rather demanding setup. In what practical scenario would we naturally have this collection of Q-functions and then want to perform this kind of fusion? Why would it not be more straightforward to take the new MDP and simply run RL on it?

  2. More fundamentally, why do we need this? In a simulated MDP, the transition mechanism is already available and is somehow "free." Even for a more complex physics simulation, given the speed of modern simulators, we could often use the simulator itself as the transition model. What practical advantage would recovering that model from Q-functions provide? In the real world, I would expect obtaining reliable Q-functions for multiple tasks to be substantially more expensive than learning the dynamics directly. This brings me back to the first concern: it is difficult to see a natural setting in which the required Q-functions are available, the dynamics are not, and Q-fusion is the most useful next step.

Despite the mundane concerns, I do like the math games in the RL fields, so I still lean towards acceptance. It would be much better if you can provide some practical cases to convince me that it is not only interesting but also useful.

Questions for authors

See Weakness.

Reviewer_pzxpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Although tasks in the same environment share the same dynamics, their value functions and rewards are task-specific. This paper combines multiple Q-functions from different tasks in the same environment to recover a transition model, building on Bellman inversion. One transition model is fitted to all Q-functions: by a linear program in finite-state environments, and as a neural network that predicts the next state in continuous environments. The recovered model is then kept frozen and used to plan for new tasks where no Q-function is available. The approach is evaluated on four environments, and the results show that fusing Q-functions from more, and more different, tasks improves the transition model and achieves high returns on new tasks.

Critical strengths
  • Recovering a shared model from separately trained Q-functions to solve new tasks, without new interaction, is an interesting idea.
  • Results are reported transparently, including failed seeds and negative differences.
Critical weaknesses
  • The paper is very hard to follow and poorly written, and the writing quality is not acceptable in its current form. Many sentences are vague and unclear, and the same caveats are repeated instead of properly explaining the method. The core idea is simple but buried under vague terms and heavy notation.
  • The paper builds directly on Letcher et al. (2026), as also stated in Appendix A. However, the contribution over it, and also over Adamczyk (2025), is unclear and questionable. Please clearly state the contributions over these works and provide direct comparisons.
  • The simple example in Appendix B.3 explains the idea much better than all of Section 3 and should be in the main text.
  • Training the source Q-functions requires a large amount of environment interaction (20M steps per agent, 40M in total for the two-bank model). With this much data, a dynamics model or an offline RL policy (goal-conditioned or not) could be trained directly. Please compare the proposed approach against these alternatives, since the usefulness of the idea is questionable in its current form.
  • Only toy environments with small state-action spaces are used in the experiments, so it is unclear whether the proposed idea scales at all. Please evaluate the method on more complex tasks with high-dimensional state spaces.
  • The practical usefulness of the approach is unclear. How does it compare to training an RL agent on the new task in terms of compute and time? Could the recovered model be used in model-based RL to improve performance?
  • There are no comparisons with external baselines. Please compare with a dynamics model trained directly on the transitions, offline RL, P-learning (Letcher et al., 2026), and successor-feature-based approaches (Barreto et al., 2017).
  • Several citations do not support the claims they are attached to (for example, PPO and DQN are cited for Bellman inversion). Please verify all citations.
Questions for authors

See Weaknesses.

Reviewer_ZF2nOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Motivated by the Bellman inversion equation, this paper studies how to infer shared dynamics from given optimal value functions. The inferred shared transition model is then used to adapt to tasks with different rewards and termination rules. Experiments on FourRooms, MountainCar, Pendulum, and Reacher demonstrate effectiveness on test tasks across the evaluated tabular and continuous-control settings.

Critical strengths

The example in Fig.1 on tasks with different goals clearly illustrates the methodology, and the pseudocode details the specific implementation. Besides, this paper validates the effectiveness of the proposed method on different tabular and continuous-control benchmarks.

Critical weaknesses
  1. My major concern lies in the novelty of this paper. Prior related work, such as [1–2], also studies the problem of inferring dynamics models directly from the value function. Specifically, [2] also investigates how to infer shared dynamics, so that only the reward is changed to adapt to tasks with different goals. Moreover, [2] also evaluates its proposed method on the MountainCar and Reacher benchmarks employed in this paper. However, this paper does not compare against any baselines, especially the relevant [1] and [2], which makes it difficult to evaluate the effectiveness of the proposed method.

[1] Inferring Transition Dynamics from Value Functions. (AAAI-25 8th Workshop on Generalization in Planning)

[2] Inverting the Bellman Equation: From Q-Values to World Models. (NeurIPS 2026)

  1. I think the implementation of the proposed methodology in Eq.3 is unreasonable. Q^(x)\hat{Q}(x) denotes the standard state-action value, whereas ϕ^(x,fθ(x))\hat{\phi}(x, f_\theta(x)) represents the value of the transition state, but they are directly subatrcted. This also contradicts Eq.2, where a vector pp appears between them. Moreover, the Bellman inversion equation that forms the foundation of this paper is not well explained in Eq.1.
Questions for authors
  1. In this paper, Q^(x)\hat{Q}(x) denotes the standard state-action value, whereas ϕ^(x,fθ(x))\hat{\phi}(x, f_\theta(x)) represents the value of the transition state. Since these two terms appear to different meanings, why is ϕ^(x,fθ(x))\hat{\phi}(x, f_\theta(x)) subtracted directly from Q^(x)\hat{Q}(x) in Eq.3?

  2. What specifically do anchor, supplement, and query refer to in the reinforcement learning task of this paper?

  3. What conclusion can be drawn from the bound in Eq.5? Does it constitute a tighter upper bound?

  4. What elements is prescribed row class Dx\mathcal{D}_x is composed of, and what do the probability simplex and vertex set specifically denote in Line 188?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.