acceptodds
Under review as a conference paper at ICLR 2027

Value-Guided Repulsion for Offline Reinforcement Learning

Abstract

Drifting models have emerged as a highly efficient and stable paradigm for policy design in offline reinforcement learning (RL). However, standard drift policies determine repulsion solely from geometric relationships among generated actions and therefore cannot distinguish them by expected value. To address this limitation, we introduce Value-Guided Repulsion Q-Learning (VRQL) to integrate action value estimates into the particle interaction field. For each state, VRQL combines geometric kernel distance with a normalized Q value scaling. This mechanism assigns greater repulsive influence to particles with lower values while strictly retaining attraction toward behavior action. Consequently, this value-guided interaction field aligns behavior regularization with policy improvement without adding an auxiliary behavior model stage. On 50 OGBench task configurations and 6 D4RL AntMaze configurations, VRQL demonstrates highly competitive performance against established baselines, suggesting that value-guided particle interaction is an effective method for offline RL.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning、Drifting Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4LcKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VRQL, a modification of DriftQL where the critic's values are folded into the repulsion weights between particles. The original repulsion softmax only looks at distances. VRQL adds a normalized Q term to the logits so that neighbors with lower Q push harder, which moves particles toward regions the critic likes. The Q values are divided by the within state standard deviation over the batch to keep the scale under control. Experiments cover 50 OGBench tasks and the 6 D4RL AntMaze tasks.

Critical strengths
  • No extra network is needed, so the idea can be bolted onto an existing one-step policy fairly easily.
  • The evaluation covers a lot of tasks, and the numbers are consistently good.
Critical weaknesses
  • Table 1 shows VRQL outperforming DriftQL, which is really its baseline, almost everywhere. But once I looked at the code in the supplementary material, I found the per-task tuning so heavy that I cannot credit this to value-guided repulsion. Appendix A.2 says every run uses the Laplace kernel, yet train.sh switches to the Gaussian kernel on 25 OGBench tasks and 4 D4RL tasks. Appendix A.1 says the Bellman target takes the min over critics, but 35 OGBench tasks are run with q_agg=mean, which replaces the min with the mean. Table 2 lists N = 32 and lr = 3e-4, but some tasks use N = 4 and lr = 1e-4 or 5e-5, and β\beta goes as low as 0.005 and as high as 16, both outside the stated [0.05, 10] range. α\alpha and the discount also change per task and are never reported. Even within tasks 1–5 of the same environment, β\beta differs by two or three orders of magnitude. Practically speaking, the 50 tasks are 50 separate configurations.
  • I do not see much of a contribution here. The method amounts to one extra −Q−Q term in the repulsion logits of DriftQL, and the "adaptive value calibration" is just dividing by the batch standard deviation. There is also hardly any theoretical justification for why this should work.
Questions for authors
  1. Given how much the hyperparameters vary from task to task, I am not sure how to separate the effect of the method from the effect of tuning. How were the hyperparameters chosen, and did the baselines, DriftQL in particular, get the same treatment?
  2. I would like to see the ablations on more tasks. Figs. 2, 3, and 4 are each on single tasks right now. Running them over each full group of five tasks would make the picture a lot clearer.
  3. The method comes with no theory. (i) Value-guided repulsion pushes a particle away from its low-QQ neighbors, that is, toward higher QQ, which is exactly what the direct QQ-maximization term in Eq. 14 already does. The direction you get from comparing neighbors' QQ values is basically a particle estimate of ∇aQ\nabla_a Q, so it looks like the same force is being added twice. What makes this different from just turning up the coefficient on the direct QQ term? (ii) The original drifting procedure converges to the data distribution. Once the QQ term is in, what does it converge to? (iii) If particles keep getting pushed toward higher QQ, can attraction alone keep them inside the data support?
  4. There is only one data action per state (Eq. 8). How does the multimodal-behavior motivation from the introduction fit this setup?
  5. The method relies on only two critics, and the conclusion itself acknowledges that wrong value orderings from the critic can misguide the repulsion. Did you try a larger ensemble or any other way of making the guidance more robust to critic error?
6 comments

Q1 · Hyperparameter Selection

Our experimental protocol follows the publicly released task configurations of DriftQL. Across the 50 single-task configurations in OGBench, 43 tasks use base hyperparameters identical to the reference DriftQL configurations once VRQL-specific guidance parameters are excluded. The only 7 exceptions are humanoidmaze-large-navigate-task3, cube-double-play-task1–4, and puzzle-4x4-play-task1,3. We will list the per-task settings in the appendix.

Q2 · Expanded Repulsion Assignment Ablations Across Full Task Families

We expanded the repulsion weight assignment ablation from single representative tasks to all 5 tasks in each of the antmaze-giant-navigate, humanoidmaze-medium-navigate, and puzzle-3x3-play families.

Task familyLearned (VRQL)None (DriftQL)PermutedInverse
antmaze-giant-navigate (5 tasks)74607144
humanoidmaze-medium-navigate (5 tasks)88626051
puzzle-3x3-play (5 tasks)54353418

Across all three task families, Learned (assigning stronger repulsion to lower-value neighbors) consistently achieves the highest mean success rate, outperforming both pure geometric repulsion (None) and randomly shuffled value biases (Permuted), while reversing the value assignment (Inverse) causes severe performance degradation across every family.

Q3 · Relationship to Direct Q Gradients, Convergence, and Behavioral Support

(i) Distinction from Increasing the Direct QQ-Loss Coefficient

We conducted a sweep over the direct QQ-loss multiplier λ∈1,2,4,8\lambda \in \\{1, 2, 4, 8\\} for DriftQL, with λ=1\lambda = 1 as the default.

TaskDriftQL (λ=1\lambda = 1, default)DriftQL (λ=2\lambda = 2)DriftQL (λ=4\lambda = 4)DriftQL (λ=8\lambda = 8)VRQL (Ours)
antmaze-giant-navigate-task132.0012.670.670.0056.00
antmaze-giant-navigate-task279.0028.670.000.0069.00
antmaze-giant-navigate-task343.0030.670.000.0067.00
antmaze-giant-navigate-task464.005.000.000.0087.00
antmaze-giant-navigate-task585.0056.338.670.0089.00

As shown above, increasing the direct QQ-loss weight λ\lambda from 11 to 22, 44, or 88 causes severe performance degradation and eventual collapse to 0% across all tasks.

This contrast highlights why value-guided repulsion inside Ldrift\mathcal{L}_{\text{drift}} is fundamentally distinct from scaling the direct QQ loss. VRQL operates inside the behavioral regularizer Ldrift\mathcal{L}_{\text{drift}}. In datasets dominated by exploratory or suboptimal trajectories with sparse successful demonstrations, our objective is to make the behavioral constraint itself value-aware—preserving strong alignment toward high-value regions while pushing generated particles away from low-value alternatives so that policy improvement is not counteracted by a value-blind geometric anchor.A natural alternative would be to apply value weighting to positive dataset actions. However, each transition in a continuous-state offline dataset provides only a single ground-truth action anchor ab+a_b^+ (and our preliminary experiments fitting an auxiliary flow behavior model to sample multiple synthetic positive anchors introduced two-stage approximation errors and underperformed). VRQL circumvents this bottleneck by keeping the single ground-truth anchor intact and injecting relative value guidance into the NN generated negative particles that define repulsion.

(ii) What Does the Modified Procedure Converge To?

In VRQL, value information shapes policy learning at two complementary levels. Target critic evaluations modulate the repulsive weights to shift the stopped behavioral target tbit_b^i inside Ldrift\mathcal{L}_{\text{drift}}, while the online critic ensemble provides direct policy improvement gradients.

In practical training, the actor minimizes

Lactor(θ)=αLdrift(θ)−Es,z[Qϕ(s,xθ(s,z))].\mathcal{L}_{\text{actor}}(\theta) = \alpha \mathcal{L}_{\text{drift}}(\theta) - \mathbb{E}_{s,z}[Q_\phi(s, x_\theta(s,z))].

Fixing the current critic and a batch of generated particles, and treating the stopped target tbi=sg[ΠA(xbi+ηVbi)]t_b^i = \text{sg}[\Pi_{\mathcal{A}}(x_b^i + \eta V_b^i)] as constant during differentiation, an interior stationary point (where the coordinatewise box projection ΠA\Pi_{\mathcal{A}} is inactive) satisfies the first-order force-balance equation

2αηdVbi+∇aQϕ(sb,xbi)=0,\frac{2\alpha\eta}{d} V_b^i + \nabla_a Q_\phi(s_b, x_b^i) = 0,

where

Vbi=(ab+−xbi)−∑j≠iwbi,j(xbj−xbi).V_b^i = (a_b^+ - x_b^i) - \sum_{j \neq i} w_b^{i,j}(x_b^j - x_b^i).

Thus, a stationary point balances behavioral attraction, value-guided repulsion, and the direct critic gradient. Standard geometric drifting (DriftQL) leaves VbiV_b^i completely value-blind, forcing ∇aQϕ\nabla_a Q_\phi to pull against an unguided geometric barycenter. VRQL assigns larger repulsive weights to lower-valued neighbors at equal distances. This shifts the repulsive barycenter ∑j≠iwbi,jxbj\sum_{j \neq i} w_b^{i,j} x_b^j toward lower-valued particles and moves the stopped target tbit_b^i away from suboptimal actions inside the regularizer itself.

(iii) Can Attraction Alone Keep Particles Inside the Data Support?

As formulated in Section 2.1, VRQL designs the drifting term as a soft behavioral regularizer. In offline RL benchmarks containing heterogeneous or suboptimal trajectories, soft regularization permits controlled local movement and trajectory stitching while discouraging unsupported actions. VRQL bounds and stabilizes particle updates through the attraction force.The attraction vector Abi=ab+−xbiA_b^i = a_b^+ - x_b^i acts as a first-order linear restoring force anchored at the observed dataset action ab+a_b^+, weighted by α\alpha in the overall drift regression loss Ldrift\mathcal{L}_{\text{drift}}. As any particle xbix_b^i moves further from ab+a_b^+, the attractive pull grows linearly with the displacement ∣ab+−xbi∣2\\|a_b^+ - x_b^i\\|_2, while the projection operator ΠA\Pi_{\mathcal{A}} strictly confines all actions and targets within the valid action box A\mathcal{A}.

Q4 · Single Positive Anchors and Multimodal Behavior

In a continuous-state offline RL dataset D=(sb,ab,… )\mathcal{D} = \\{(s_b, a_b, \dots)\\}, each specific sampled state sbs_b appears almost only once in the dataset, and thus can only provide a single action sample ab+a_b^+. However, because the parameterized policy network xθ(s,z)x_\theta(s, z) possesses smooth generalization capabilities over the continuous state space, different modal actions ab+a_b^+ from distinct trajectories at nearby states s≈s′s \approx s' jointly shape the conditional distribution πθ(⋅∣s)\pi_\theta(\cdot \mid s) during training.

If there is only the single-sample attractive term Abi=ab+−xbiA_b^i = a_b^+ - x_b^i (i.e., setting the repulsive force Rbi=0R_b^i = 0), minimizing E[∥xθ(s,z)−a+∥22]\mathbb{E}[\Vert{}x_\theta(s, z) - a^+\Vert{}_2^2] would indeed degenerate into standard Behavior Cloning (BC), causing all noise zz to collapse to the conditional mean E[a+∣s]\mathbb{E}[a^+ \mid s] and failing to represent multimodality.

However, VRQL simultaneously samples NN independent noise particles xbii=1N\\{x_b^i\\}_{i=1}^N for each state sbs_b and computes the pairwise kernel repulsive force between them, Rbi=∑j≠iwbi,j(xbj−xbi)R_b^i = \sum_{j \neq i} w_b^{i,j}(x_b^j - x_b^i). Pairwise repulsion forces particles mapped from different latent variables zbiz_b^i to push away from each other, maintaining the spread and multimodal diversity of the generated distribution in the action space. When different behavioral modal anchors ab+a_b^+ at nearby states are sampled across different training steps, the localized kernel repulsion (with bandwidth τ\tau) allows different clusters of particles to occupy distinct action modes. VRQL's value-guidance mechanism strengthens the repulsive force on particles in low-value modes and weakens it on those in high-value modes, thereby guiding the particle population to preferentially concentrate toward high-return behavioral modes while preserving multimodal expressiveness.

We evaluated behavioral trajectory diversity on puzzle-3x3-play-task4 across 64 evaluation rollouts.

MetricDriftQLVRQL
Successful rollouts11 / 6456 / 64
Distinct successful trajectories100.0%64.7%
Distinct trajectories overall92.9%81.3%

VRQL increases the number of successful rollouts more than fivefold (from 11/6411/64 to 56/6456/64) while retaining substantial trajectory diversity (81.3% distinct trajectories overall and 64.7% distinct solutions among successful rollouts).

Q5 · Effect of the Critic Ensemble Size

We evaluated critic ensembles of size H∈2,5,7H \in \\{2, 5, 7\\} on antmaze-giant-navigate and puzzle-3x3-play. Each evaluation covered all five tasks in the family, with one million training steps per task. The table reports five-task mean success rates (%) and sample standard deviations across seeds.

Task familyH=2H = 2H=5H = 5H=7H = 7
antmaze-giant-navigate (5 tasks)56.00 ± 14.1473.00 ± 2.3659.00 ± 8.01
puzzle-3x3-play (5 tasks)47.00 ± 7.0743.33 ± 4.7146.00 ± 0.94

These results show that increasing the critic ensemble size can improve action-ranking reliability and reduce variance on navigation tasks (e.g., increasing HH from 22 to 55 raises the success rate from 56.00% to 73.00% on antmaze-giant-navigate), albeit at the cost of additional training compute. However, on extremely long-horizon tasks, single-step TD learning relies on bootstrapping, where target estimation bias compounds over the horizon. As analyzed in Park et al. (2025), Horizon Reduction Makes RL Scalable, merely scaling the critic ensemble size or dataset coverage does not resolve value-ranking degradation caused by the curse of horizon—which also explains why both VRQL and prior baselines achieve low success rates on humanoidmaze-large-navigate. Reliable critic rankings on long-horizon tasks remain a limitation of VRQL, and horizon reduction is outside the scope of this work.

Reviewer_BtEvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Value-Guided Repulsion Q-Learning (VRQL), an offline RL method built on DriftQL’s one-step drifting policy. For each state, the policy generates multiple action particles. VRQL evaluates these particles using a target critic ensemble, centers their Q values within the state, and scales them by an average within-state standard deviation. The resulting value signal is combined with a geometric kernel in the softmax used to weight pairwise repulsion, so that lower-valued neighbors exert stronger repulsion. Behavior-action attraction remains unchanged, and the actor is trained with a stopped drift regression target plus a direct Q-maximization objective. The empirical evaluation covers 50 OGBench single-task configurations and 6 D4RL AntMaze configurations, with 14 baseline algorithms and five seeds per task. The paper reports category-level improvements over DriftQL and other baselines, and presents ablations on repulsion-weight assignment, value normalization, guidance temperature, and particle count.

Critical strengths

The empirical evaluation is broad and, at the category level, shows substantial improvements over DriftQL. The paper evaluates 50 OGBench tasks and 6 D4RL AntMaze tasks with 14 baselines and five seeds, and provides per-task results in the appendix. The reported gains on, for example, antmaze-giant, cube-double, and D4RL AntMaze are meaningful if they can be clearly attributed to the proposed mechanism. The learned/permuted/inverse ablation is also a useful sanity check that the direction of the Q assignment matters.

Critical weaknesses

The central contribution is not yet clearly separated from the existing value objective. The manuscript motivates VRQL by arguing that geometric drift repulsion cannot distinguish actions by expected value. However, DriftQL already includes an explicit Q-maximization loss; geometric repulsion can reasonably be viewed as the behavioral/diversity regularizer rather than the component responsible for value ranking. The fact that repulsion is value-agnostic does not by itself imply that the policy is value-blind or that policy improvement is hindered. The proposed method adds a second, Q-dependent pathway inside the regularizer, but the experiments do not isolate its independent effect. In particular, all reported VRQL variants retain the direct Q loss, and there is no ablation with Q-guided repulsion but no direct Q loss. Without such evidence, and without a comparison to a properly tuned DriftQL with adjusted Q and regularization weights, it is difficult to determine whether the new term provides a distinct benefit or functions as an additional adaptive regularization mechanism. This directly affects the paper’s novelty claim.

Questions for authors

1.Given that DriftQL already contains a direct Q-maximization objective, could the authors clarify what specific deficiency of geometry-only repulsion remains after accounting for that loss? Would they be willing to provide a quantitative comparison against a properly tuned DriftQL, such as a Q–diversity–support Pareto-frontier analysis, to establish that Q-guided repulsion provides a non-redundant benefit? 2.Could the authors provide an ablation in which the direct Q loss is removed while Q-guided repulsion is retained, compared with a geometry-only repulsion baseline under matched tuning and compute? Does Q-guided repulsion alone improve return, critic Q, or diversity/support retention?

1 comment

Q1 · What Problem Does Value-Guided Repulsion Solve When the Direct Q Loss Is Already Present?

Thank you for raising this insightful question. To test whether value-guided repulsion can be replaced simply by increasing the weight on the direct QQ-maximization term in the actor objective, we conducted a sweep over the direct QQ-loss multiplier λ∈1,2,4,8\lambda \in \\{1, 2, 4, 8\\}, with λ=1\lambda = 1 as the default, for DriftQL on all 5 tasks of antmaze-giant-navigate.

TaskDriftQL (λ=1\lambda = 1, default)DriftQL (λ=2\lambda = 2)DriftQL (λ=4\lambda = 4)DriftQL (λ=8\lambda = 8)VRQL (Ours)
antmaze-giant-navigate-task132.0012.670.670.0056.00
antmaze-giant-navigate-task279.0028.670.000.0069.00
antmaze-giant-navigate-task343.0030.670.000.0067.00
antmaze-giant-navigate-task464.005.000.000.0087.00
antmaze-giant-navigate-task585.0056.338.670.0089.00

As shown above, increasing the direct QQ-loss weight λ\lambda from 11 to 22, 44, or 88 causes severe performance degradation and eventual collapse to 0% across all tasks.

This contrast highlights why value-guided repulsion inside Ldrift\mathcal{L}_{\text{drift}} is fundamentally distinct from scaling the direct QQ loss. VRQL operates inside the behavioral regularizer Ldrift\mathcal{L}_{\text{drift}}. In datasets dominated by exploratory or suboptimal trajectories with sparse successful demonstrations, our objective is to make the behavioral constraint itself value-aware—preserving strong alignment toward high-value regions while pushing generated particles away from low-value alternatives so that policy improvement is not counteracted by a value-blind geometric anchor.A natural alternative would be to apply value weighting to positive dataset actions. However, each transition in a continuous-state offline dataset provides only a single ground-truth action anchor ab+a_b^+ (and our preliminary experiments fitting an auxiliary flow behavior model to sample multiple synthetic positive anchors introduced two-stage approximation errors and underperformed). VRQL circumvents this bottleneck by keeping the single ground-truth anchor intact and injecting relative value guidance into the NN generated negative particles that define repulsion.

Furthermore, we evaluated behavioral trajectory diversity on puzzle-3x3-play-task4 across 64 evaluation rollouts.

MetricDriftQLVRQL
Successful rollouts11 / 6456 / 64
Distinct successful trajectories100.0%64.7%
Distinct trajectories overall92.9%81.3%

VRQL increases the number of successful rollouts more than fivefold (from 11/6411/64 to 56/6456/64) while retaining substantial trajectory diversity (81.3% distinct trajectories overall and 64.7% distinct solutions among successful rollouts).

Q2 · Does Value Guidance Still Help When the Direct Q Loss Is Removed?

To isolate the standalone effect of value-guided repulsion, we evaluated both DriftQL and VRQL on puzzle-3x3-play with the direct QQ-maximization term completely removed (λ=0\lambda = 0).

TaskDriftQL (λ=0\lambda = 0)VRQL (λ=0\lambda = 0)
puzzle-3x3-play-task15.0021.67
puzzle-3x3-play-task23.335.00
puzzle-3x3-play-task31.670.00
puzzle-3x3-play-task41.673.33
puzzle-3x3-play-task50.000.00

Even without the direct QQ loss, value-guided repulsion yields clear gains on several tasks (e.g., improving task1 from 5.00% to 21.67%). At the same time, because Ldrift\mathcal{L}_{\text{drift}} serves as a behavioral regularizer anchored to dataset actions and optimal trajectories are scarce in offline datasets, the direct QQ-maximization term in Eq. (14) remains essential as the primary driver of policy improvement, working in tandem with our value-guided regularizer.

Reviewer_8TtAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes VRQL, an extension of a drifting offline-RL policy that introduces a negative, normalized target-Q bias into the softmax weights governing repulsion among generated actions. The behavioral attraction anchor and a separate actor Q-maximization term are retained. A minibatch-level scale based on within-state Q dispersion calibrates the guidance. The actor still produces an action in one forward pass at inference. Experiments report results on 50 OGBench configurations and six D4RL AntMaze configurations, together with assignment, normalization, temperature, and particle-count ablations.

Critical strengths
  • A precise and inexpensive intervention. Equations 9–14 specify where value information enters and where gradients stop. The method changes training-time particle interactions while preserving direct action generation, making the proposal straightforward to understand and potentially useful in existing drifting-policy implementations.

  • Meaningful gains with mechanism-oriented ablations. The reported improvement over DriftQL is substantial in categories such as cube-double and antmaze-giant. The none/permuted/inverse comparisons are useful controls: they test whether the association between neighbor value and repulsive weight matters, rather than merely adding another source of interaction noise.

Critical weaknesses
  1. The central mechanism relies on critic rankings whose reliability is not established. VRQL uses learned values both to improve actions and to reshape the behavioral constraint. As the authors acknowledge, incorrect rankings can repel optimal actions. The assignment ablations demonstrate sensitivity to corrupted rankings, but do not establish reliability during normal offline training.

  2. The experiments do not establish a generalizable gain over DriftQL under comparable selection budgets. Task-specific tuning and imported baseline scores leave the mechanism’s contribution insufficiently separated from configuration and selection differences. Limited assignment controls do not resolve this across the benchmark; independently selected settings and a budget-matched comparison are needed.

Questions for authors
  1. When are the critic rankings reliable enough to guide repulsion? Can you measure ranking quality among generated actions during normal training and relate it to performance gains/regressions and data coverage?

  2. Do the gains generalize beyond task-specific tuning? How are β, τ, and α selected, using what signal and budget? Can you report common β/τ settings or settings selected on independent tasks/task families and then frozen?

  3. Does the advantage over DriftQL persist under a matched comparison? Can you align architectures, critic settings, datasets, gradient steps, evaluation checkpoints/aggregation, and search budgets, and report selection signals, trial counts, and computational cost?

1 comment

Q1 · When Is the Critic's Action Ranking Reliable?

We evaluated critic ensembles of size H∈2,5,7H \in \\{2, 5, 7\\} on antmaze-giant-navigate and puzzle-3x3-play. Each evaluation covered all five tasks in the family, with one million training steps per task. The table reports five-task mean success rates (%) and sample standard deviations across seeds.

Task familyH=2H = 2H=5H = 5H=7H = 7
antmaze-giant-navigate (5 tasks)56.00 ± 14.1473.00 ± 2.3659.00 ± 8.01
puzzle-3x3-play (5 tasks)47.00 ± 7.0743.33 ± 4.7146.00 ± 0.94

These results show that increasing the critic ensemble size can improve action-ranking reliability and reduce variance on navigation tasks (e.g., increasing HH from 22 to 55 raises the success rate from 56.00% to 73.00% on antmaze-giant-navigate), albeit at the cost of additional training compute. However, on extremely long-horizon tasks, single-step TD learning relies on bootstrapping, where target estimation bias compounds over the horizon. As analyzed in Park et al. (2025), Horizon Reduction Makes RL Scalable, merely scaling the critic ensemble size or dataset coverage does not resolve value-ranking degradation caused by the curse of horizon—which also explains why both VRQL and prior baselines achieve low success rates on humanoidmaze-large-navigate. Reliable critic rankings on long-horizon tasks remain a limitation of VRQL, and horizon reduction is outside the scope of this work.

Q2 & Q3 · Hyperparameter Selection, Unified Settings, and Matched Comparisons

Our experimental protocol follows the publicly released task configurations of DriftQL. Across the 50 single-task configurations in OGBench, 43 tasks use base hyperparameters identical to the reference DriftQL configurations once VRQL-specific guidance parameters are excluded. The only 7 exceptions are humanoidmaze-large-navigate-task3, cube-double-play-task1–4, and puzzle-4x4-play-task1,3. We will list the per-task settings in the appendix.

Unified Family-Level Hyperparameter Settings

Task familyGuidance temperature β\betaKernel temperature τ\tau
humanoidmaze-medium-navigate (5 tasks)1.00.5
humanoidmaze-large-navigate (5 tasks)0.10.2
antmaze-large-navigate (5 tasks)0.10.5
antmaze-giant-navigate (5 tasks)1.00.2
antsoccer-arena-navigate (5 tasks)0.10.5
cube-single-play (5 tasks)0.10.02
cube-double-play (5 tasks)0.050.05
scene-play (5 tasks)0.10.2
puzzle-3x3-play (5 tasks)0.10.5
puzzle-4x4-play (5 tasks)0.10.8
D4RL AntMaze (6 tasks)1.00.1

For individual tasks, the search over β\beta typically used 0.05,0.1,0.5,1,4\\{0.05, 0.1, 0.5, 1, 4\\}. We generally took τ\tau from DriftQL’s public task configurations. All network architectures, critic ensemble settings, offline datasets, gradient update steps, evaluation checkpoints, and aggregation rules strictly match DriftQL. We select β\beta according to the mean evaluation success rate and three internal training diagnostics.

  • repel_logit_term_ratio tracks the relative magnitude of the calibrated value bias versus the geometric kernel term.
  • repel_drift_displacement measures the shift in the drift vector field induced by value guidance.
  • repel_entropy_frac monitors the concentration (entropy) of the softmax repulsion weights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.