acceptodds
Under review as a conference paper at ICLR 2027

Data-Anchored Dynamic Routing for One-Step Offline Reinforcement Learning

Abstract

One-step actors keep inference cheap in offline reinforcement learning, but still have to improve under a critic without moving too far from the data. In teacher–student methods such as FQL, the same student output is asked to do both jobs: move toward higher Q and stay near its paired teacher endpoint. When these directions disagree, the update is a compromise on that sample, even if another output could represent the target. We propose DROL, which replaces this teacher–student pairing with dynamic routing and trains the actor directly from offline data. For each state–action pair, the actor samples candidates and updates only the nearest one with behavior cloning and critic guidance. As the actor changes, another candidate can take over the reconstruction target, allowing different latent regions to specialize without fixing their assignments in advance. Multiple candidates are needed only during training; inference uses a single actor pass. Our local analysis explains this assignment flexibility, and diagnostics track it in trained policies. On OGBench and D4RL, DROL is competitive with one-step baselines, with multi-candidate training improving performance across manipulation and puzzle task families.

Primary area:
reinforcement learning
Keywords:
Offline RL, One-Step Policy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sg4GOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper replaces the teacher-student setting proposed in FQL by a single one-step actor. For each state-action pair (s,a)(s, a), the approach consist of sampling KK actions (using KK latents samples) with a one-step actor and use only the action that is closest (L2L^2 distance) to aa to update the policy using a regularized actor-critic loss. The author evaluate the approach on OGbench benchmark along with some ablations studies and theoretical justification.

Critical strengths

The main idea and the motivation of the paper is easy to understand and well presented. Empirical results are competitive with the baselines specially when the hyperparameter KK (number of action candidat is tuned per task).

Critical weaknesses
  1. Except the illustrative examples in Figure 1 and Figure 2 (Figure 2 is not well explain and is difficult to interpreted, how was it produced, the abbreviations etc), there is no a clear theoretical justification why the proposed framework mitigate the tension between the two objectives (maximize the Q-function and remain near the dataset support) better than previous previous approach.

  2. It is not clear how the proposed approach capture a multi-modal distribution

  3. In term of mitigating the tension the tension between maximizing the Q-function and remain near the dataset support (which is the main problem the paper attend to address), recent methods like GFP [1] have being proposed. How does DROL compare to these methods.

  4. Even though the main idea is easy to understand the paper remain difficult to follow, the writing is not precise e.g in row 39 and 40 what do you clearly mean by 'the loss resolves the conflict by compromising on that sample', such kind of claim can found throughout the paper.

  5. Limited related work

  6. The theory provide in the paper are general facts and does not justify the proposed method.

    • How is section 3.3 relevent?
    • Proposition 4.1 is just a general fact.
    • Except the change in term of the gradient of the objectif, Proposition 4.2 does not establish and improvement in term of the resulting policy.
    • Proposition 4.3 is general fact with strong unrealistic assumption e.g supposing that a value function is m-strongly convexe even locally. The framing of the proposition itself is questionable e.g what do you mean by "Let U∈RdaU \in \mathbb{R}^{d_{a}} be a closed convex local analysis neighborhood" ? etc
  7. As the author point out in section 5.3 (Conflict in a pointwise update .....) the experiment does not show DROL reduces the conflict or than the policy performance is related to the conflict.

  8. Can the authors also clarify how they defines and mesure the change of responsibility in their experimental setup (section 5.4)

  9. In term of experiment, how does DROL perform on noisy dataset from OGBench which can be seen as a good proxy multi-modal dataset? Or can the illustrate the approach on a true multi-modal example?

[1] [Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning](https://arxiv.org/abs/2512.03973

Questions for authors

see above...

Reviewer_NMVUOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DROL, a one-step offline RL policy trained through data-anchored candidate routing. For each dataset state–action pair, the actor generates multiple candidates, selects the candidate nearest to the dataset action, and applies both behavior cloning and critic-guided improvement to that winner. The paper provides local analysis, benchmark evaluations, and diagnostics of candidate specialization and responsibility changes.

Critical strengths

The method is simple and retains inexpensive single-pass inference without requiring a separate generative teacher. The experiments demonstrate competitive performance across OGBench and D4RL. In particular, the complete-family comparison between (K=1) and (K=16) provides useful evidence that multi-candidate training improves several manipulation and puzzle task families. I also appreciate the authors’ explicit distinction between observing responsibility changes and establishing their causal effect on performance.

Critical weaknesses
  1. The independent value of the proposed BC/Q coupling is not isolated. Changing K simultaneously changes BC matching quality and which actions receive Q updates. A control retaining winner-only BC while applying Q updates to all candidates would establish whether coupling both updates to the same winner drives the gains.
  2. The empirical advantages depend strongly on task-specific tuning. Fixed-K=16 DROL underperforms FQL on cube-double (20 versus 29). The large improvement on antmaze-giant uses K=1, which does not demonstrate a benefit from multiple candidates. Matched tuning-budget comparisons are needed to establish a consistent advantage.
Questions for authors
  1. Can the authors provide the above comparison between winner-only Q updates and Q updates averaged over all candidates, while keeping the nearest-candidate BC objective unchanged?
  2. Does DROL consistently outperform this control on the task families where increasing (K) helps?
Reviewer_274HOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DROL, an offline reinforcement learning method for training expressive policies with single-pass inference. The method generates multiple latent-conditioned action candidates and selects the one closest to the dataset action, routing both behavior-cloning and Q-guided updates to this candidate while using a single randomly sampled latent at inference.

Critical strengths

DROL removes the iterative teacher and distillation pipeline while retaining single-pass inference, simplifying the training of expressive offline policies. Matched ablations show substantial gains from multi-candidate training across manipulation and puzzle task families under the same single-sample inference protocol.

Critical weaknesses
  1. Nearest-candidate reconstruction is established in IMLE and IMLE Policy, but the K ablation changes both matching and Q-update recipients, leaving the benefit of winner-only Q guidance insufficiently isolated.
  2. The diagnostics quantify persistent and transient responsibility changes, but do not establish whether reassignment contributes to return gains.
Questions for authors
  1. With K>1 fixed and nearest-candidate BC unchanged, could the authors compare Q guidance applied to the nearest winner, one random candidate, and all candidates (averaged), alongside a BC-only variant? Matched training settings and reported compute costs would help isolate the benefit of winner-only Q guidance.
  2. How does distance-only routing compare with assignment based on the combined BC/Q loss? Excluding Q may reduce sensitivity to critic error but overlook higher-value, behavior-supported candidates. Could the authors provide a matched comparison and clarify when each approach is preferable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.