- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VRQL, a modification of DriftQL where the critic's values are folded into the repulsion weights between particles. The original repulsion softmax only looks at distances. VRQL adds a normalized Q term to the logits so that neighbors with lower Q push harder, which moves particles toward regions the critic likes. The Q values are divided by the within state standard deviation over the batch to keep the scale under control. Experiments cover 50 OGBench tasks and the 6 D4RL AntMaze tasks.
Critical strengths
- No extra network is needed, so the idea can be bolted onto an existing one-step policy fairly easily.
- The evaluation covers a lot of tasks, and the numbers are consistently good.
Critical weaknesses
- Table 1 shows VRQL outperforming DriftQL, which is really its baseline, almost everywhere. But once I looked at the code in the supplementary material, I found the per-task tuning so heavy that I cannot credit this to value-guided repulsion. Appendix A.2 says every run uses the Laplace kernel, yet train.sh switches to the Gaussian kernel on 25 OGBench tasks and 4 D4RL tasks. Appendix A.1 says the Bellman target takes the min over critics, but 35 OGBench tasks are run with q_agg=mean, which replaces the min with the mean. Table 2 lists N = 32 and lr = 3e-4, but some tasks use N = 4 and lr = 1e-4 or 5e-5, and goes as low as 0.005 and as high as 16, both outside the stated [0.05, 10] range. and the discount also change per task and are never reported. Even within tasks 1–5 of the same environment, differs by two or three orders of magnitude. Practically speaking, the 50 tasks are 50 separate configurations.
- I do not see much of a contribution here. The method amounts to one extra term in the repulsion logits of DriftQL, and the "adaptive value calibration" is just dividing by the batch standard deviation. There is also hardly any theoretical justification for why this should work.
Questions for authors
- Given how much the hyperparameters vary from task to task, I am not sure how to separate the effect of the method from the effect of tuning. How were the hyperparameters chosen, and did the baselines, DriftQL in particular, get the same treatment?
- I would like to see the ablations on more tasks. Figs. 2, 3, and 4 are each on single tasks right now. Running them over each full group of five tasks would make the picture a lot clearer.
- The method comes with no theory. (i) Value-guided repulsion pushes a particle away from its low- neighbors, that is, toward higher , which is exactly what the direct -maximization term in Eq. 14 already does. The direction you get from comparing neighbors' values is basically a particle estimate of , so it looks like the same force is being added twice. What makes this different from just turning up the coefficient on the direct term? (ii) The original drifting procedure converges to the data distribution. Once the term is in, what does it converge to? (iii) If particles keep getting pushed toward higher , can attraction alone keep them inside the data support?
- There is only one data action per state (Eq. 8). How does the multimodal-behavior motivation from the introduction fit this setup?
- The method relies on only two critics, and the conclusion itself acknowledges that wrong value orderings from the critic can misguide the repulsion. Did you try a larger ensemble or any other way of making the guidance more robust to critic error?
6 comments
Q3 · Relationship to Direct Q Gradients, Convergence, and Behavioral Support
(i) Distinction from Increasing the Direct -Loss Coefficient
We conducted a sweep over the direct -loss multiplier for DriftQL, with as the default.
| Task | DriftQL (, default) | DriftQL () | DriftQL () | DriftQL () | VRQL (Ours) |
|---|---|---|---|---|---|
| antmaze-giant-navigate-task1 | 32.00 | 12.67 | 0.67 | 0.00 | 56.00 |
| antmaze-giant-navigate-task2 | 79.00 | 28.67 | 0.00 | 0.00 | 69.00 |
| antmaze-giant-navigate-task3 | 43.00 | 30.67 | 0.00 | 0.00 | 67.00 |
| antmaze-giant-navigate-task4 | 64.00 | 5.00 | 0.00 | 0.00 | 87.00 |
| antmaze-giant-navigate-task5 | 85.00 | 56.33 | 8.67 | 0.00 | 89.00 |
As shown above, increasing the direct -loss weight from to , , or causes severe performance degradation and eventual collapse to 0% across all tasks.
This contrast highlights why value-guided repulsion inside is fundamentally distinct from scaling the direct loss. VRQL operates inside the behavioral regularizer . In datasets dominated by exploratory or suboptimal trajectories with sparse successful demonstrations, our objective is to make the behavioral constraint itself value-aware—preserving strong alignment toward high-value regions while pushing generated particles away from low-value alternatives so that policy improvement is not counteracted by a value-blind geometric anchor.A natural alternative would be to apply value weighting to positive dataset actions. However, each transition in a continuous-state offline dataset provides only a single ground-truth action anchor (and our preliminary experiments fitting an auxiliary flow behavior model to sample multiple synthetic positive anchors introduced two-stage approximation errors and underperformed). VRQL circumvents this bottleneck by keeping the single ground-truth anchor intact and injecting relative value guidance into the generated negative particles that define repulsion.
(ii) What Does the Modified Procedure Converge To?
In VRQL, value information shapes policy learning at two complementary levels. Target critic evaluations modulate the repulsive weights to shift the stopped behavioral target inside , while the online critic ensemble provides direct policy improvement gradients.
In practical training, the actor minimizes
Fixing the current critic and a batch of generated particles, and treating the stopped target as constant during differentiation, an interior stationary point (where the coordinatewise box projection is inactive) satisfies the first-order force-balance equation
where
Thus, a stationary point balances behavioral attraction, value-guided repulsion, and the direct critic gradient. Standard geometric drifting (DriftQL) leaves completely value-blind, forcing to pull against an unguided geometric barycenter. VRQL assigns larger repulsive weights to lower-valued neighbors at equal distances. This shifts the repulsive barycenter toward lower-valued particles and moves the stopped target away from suboptimal actions inside the regularizer itself.
(iii) Can Attraction Alone Keep Particles Inside the Data Support?
As formulated in Section 2.1, VRQL designs the drifting term as a soft behavioral regularizer. In offline RL benchmarks containing heterogeneous or suboptimal trajectories, soft regularization permits controlled local movement and trajectory stitching while discouraging unsupported actions. VRQL bounds and stabilizes particle updates through the attraction force.The attraction vector acts as a first-order linear restoring force anchored at the observed dataset action , weighted by in the overall drift regression loss . As any particle moves further from , the attractive pull grows linearly with the displacement , while the projection operator strictly confines all actions and targets within the valid action box .
Q4 · Single Positive Anchors and Multimodal Behavior
In a continuous-state offline RL dataset , each specific sampled state appears almost only once in the dataset, and thus can only provide a single action sample . However, because the parameterized policy network possesses smooth generalization capabilities over the continuous state space, different modal actions from distinct trajectories at nearby states jointly shape the conditional distribution during training.
If there is only the single-sample attractive term (i.e., setting the repulsive force ), minimizing would indeed degenerate into standard Behavior Cloning (BC), causing all noise to collapse to the conditional mean and failing to represent multimodality.
However, VRQL simultaneously samples independent noise particles for each state and computes the pairwise kernel repulsive force between them, . Pairwise repulsion forces particles mapped from different latent variables to push away from each other, maintaining the spread and multimodal diversity of the generated distribution in the action space. When different behavioral modal anchors at nearby states are sampled across different training steps, the localized kernel repulsion (with bandwidth ) allows different clusters of particles to occupy distinct action modes. VRQL's value-guidance mechanism strengthens the repulsive force on particles in low-value modes and weakens it on those in high-value modes, thereby guiding the particle population to preferentially concentrate toward high-return behavioral modes while preserving multimodal expressiveness.
We evaluated behavioral trajectory diversity on puzzle-3x3-play-task4 across 64 evaluation rollouts.
| Metric | DriftQL | VRQL |
|---|---|---|
| Successful rollouts | 11 / 64 | 56 / 64 |
| Distinct successful trajectories | 100.0% | 64.7% |
| Distinct trajectories overall | 92.9% | 81.3% |
VRQL increases the number of successful rollouts more than fivefold (from to ) while retaining substantial trajectory diversity (81.3% distinct trajectories overall and 64.7% distinct solutions among successful rollouts).
Q5 · Effect of the Critic Ensemble Size
We evaluated critic ensembles of size on antmaze-giant-navigate and puzzle-3x3-play. Each evaluation covered all five tasks in the family, with one million training steps per task. The table reports five-task mean success rates (%) and sample standard deviations across seeds.
| Task family | |||
|---|---|---|---|
| antmaze-giant-navigate (5 tasks) | 56.00 ± 14.14 | 73.00 ± 2.36 | 59.00 ± 8.01 |
| puzzle-3x3-play (5 tasks) | 47.00 ± 7.07 | 43.33 ± 4.71 | 46.00 ± 0.94 |
These results show that increasing the critic ensemble size can improve action-ranking reliability and reduce variance on navigation tasks (e.g., increasing from to raises the success rate from 56.00% to 73.00% on antmaze-giant-navigate), albeit at the cost of additional training compute. However, on extremely long-horizon tasks, single-step TD learning relies on bootstrapping, where target estimation bias compounds over the horizon. As analyzed in Park et al. (2025), Horizon Reduction Makes RL Scalable, merely scaling the critic ensemble size or dataset coverage does not resolve value-ranking degradation caused by the curse of horizon—which also explains why both VRQL and prior baselines achieve low success rates on humanoidmaze-large-navigate. Reliable critic rankings on long-horizon tasks remain a limitation of VRQL, and horizon reduction is outside the scope of this work.
Q1 · Hyperparameter Selection
Our experimental protocol follows the publicly released task configurations of DriftQL. Across the 50 single-task configurations in OGBench, 43 tasks use base hyperparameters identical to the reference DriftQL configurations once VRQL-specific guidance parameters are excluded. The only 7 exceptions are humanoidmaze-large-navigate-task3, cube-double-play-task1–4, and puzzle-4x4-play-task1,3. We will list the per-task settings in the appendix.
Q2 · Expanded Repulsion Assignment Ablations Across Full Task Families
We expanded the repulsion weight assignment ablation from single representative tasks to all 5 tasks in each of the antmaze-giant-navigate, humanoidmaze-medium-navigate, and puzzle-3x3-play families.
Across all three task families, Learned (assigning stronger repulsion to lower-value neighbors) consistently achieves the highest mean success rate, outperforming both pure geometric repulsion (None) and randomly shuffled value biases (Permuted), while reversing the value assignment (Inverse) causes severe performance degradation across every family.