acceptodds
Under review as a conference paper at ICLR 2027

DAGA: Distributional Actor-Critic with Action-Gradient Alignment and Stability-Triggered Guidance

Abstract

Multi-critic actor–critic methods improve continuous-control learning by aggregating critic values or return distributions, yet deterministic actors are updated through critic action gradients. Critics can therefore agree in value while inducing different policy-update directions, a phenomenon we call value–gradient mismatch. We propose DAGA, a Gaussian multi-critic framework combining Stability-Triggered Guidance (STG), which adapts actor guidance using distribution-level reliability, with Action-Gradient Policy Alignment (AGPA), which selectively regularizes actor-relevant gradient directions when excessive dispersion coexists with a coherent consensus. We characterize value–gradient decoupling and both mechanisms locally. Experiments evaluate mismatch prevalence, component contributions, cross-task performance, and sensitivity to intervention strength and ensemble size.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Actor-Critic, Action-Gradient Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v6oYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DAGA studies disagreement between the action gradients of critics that may agree on their predicted values. It retains a Gaussian multi-critic precision-fusion backbone and adds two mechanisms: STG adjusts detached critic weights for actor guidance using distributional disagreement, and AGPA regularizes normalized critic action gradients only when dispersion exceeds a threshold but a sufficiently coherent consensus remains. The paper provides local mathematical characterizations, a component study on h1hand-hurdle-v0, and ten-seed comparisons on six additional continuous-control tasks. The empirical improvements over the fixed backbone are useful, although the evidence does not fully isolate the value of selective intervention or establish the comparative advantage over the closest gradient-aware alternatives.

The two additions act at different points. STG changes how critic means contribute to the actor while keeping its weighting variables detached; AGPA changes the critics themselves by differentiating an alignment penalty through their action gradients at detached actor actions. The Gaussian Bellman/precision-fusion backbone remains fixed. This separation is important because improved actor weighting and improved gradient geometry are different hypotheses, and agreement between critics is not equivalent to correctness of their shared direction (pp. 3–7, Eqs. 5–45).

Critical strengths
  1. The paper identifies the actor-facing quantity precisely. Equations 2–3 separate critic action gradients from their transformation by the policy Jacobian. Appendix A.1 correctly distinguishes pointwise agreement from neighborhood agreement: equal values at one action need not imply equal derivatives, whereas smooth local functional agreement constrains gradient differences. This avoids treating a simple pointwise counterexample as a general theorem that all critic disagreement is harmful (pp. 2–3, 11–12).

    The distinction also prevents an overly strong negative conclusion about value fitting. With smoothness and agreement on a neighborhood, Appendix A.1 bounds derivative disagreement; the paper is therefore not claiming that value information can never constrain gradients. This makes the diagnostic problem precise and gives the proposed intervention an appropriately local motivation.

  2. The construction is explicit about what is and is not optimized. STG weights are detached during actor optimization; AGPA retains the critic's derivative graph but detaches actor actions, guidance weights, and gate decisions. The separate moment-matched reference also avoids using the mechanically shrinking variance of precision fusion as an ensemble-disagreement reference. These distinctions make the proposed algorithm substantially more assessable than a generic “uncertainty-weighted” update (pp. 3–7, Eqs. 5–45).

    In particular, detaching the guidance weights in the alignment objective prevents the regularizer from being reduced merely by reassigning weights instead of changing directions. Detaching the actor action prevents the critic alignment loss from directly moving the actor. Those are concrete implementation decisions that help distinguish the proposed objective from an underspecified consensus penalty (p. 6).

  3. Reported performance is supported by ten independent training seeds. DAGA improves the mean final return over its Backbone on all six cross-task evaluations and has the highest mean in four. The strongest gaps over the best listed comparator are H1-Reach, 7029.1 versus 5848.4, approximately +20.2%, and Cheetah-Run, 1144.7 versus 1019.0, approximately +12.3%. The paper also correctly acknowledges that PPO and SimbaV2 lead on T1-Rough and H1-Basket, respectively. These are meaningful results without requiring that the method win every task (pp. 7–8, Section 4.2, Figure 2/Table 1).

    The gains over Backbone are also distributed across the task suite: for example, H1-Basket improves from 738.1 to 1001.7 even though it does not beat the strongest competitor. That is useful evidence for adding the mechanisms to this backbone. It should be distinguished from a claim that every nominal lead over every external baseline is statistically established.

  4. The primary-task study includes the relevant two-component combinations. Figure 3 compares Backbone, Backbone+STG, Backbone+AGPA, and full DAGA, and tracks both resultant length and gradient magnitude. Figure 4 additionally varies intervention strength, alignment weight, and ensemble size. The conclusion recognizes biased consensus and non-monotonic ensemble scaling as limitations rather than claiming correctness from agreement alone (pp. 7–9, Figures 3–4 and Section 5).

    The gradient-magnitude diagnostic is a useful anti-collapse check because higher agreement alone could be achieved by making critic derivatives uninformative. The reported comparable mean magnitudes do not prove derivative accuracy, but they do address one simple degenerate explanation. Similarly, the full-versus-single-component comparison is meaningful intervention evidence and should not be described as a purely correlational study.

Critical weaknesses
  1. The central benefit of selective alignment is not isolated from ordinary alignment. The 2x2 component study shows that adding STG and AGPA helps in the chosen diagnostic task; it does not show that the proposed dead zone and no-consensus abstention are what make AGPA effective. The sensitivity plot changes eta_max and lambda_A, but not the gate thresholds A0 and r0. At the defaults A0=0.2 and r0=0.8, intervention is restricted to 0.2 < A_omega <= 0.36, equivalently 0.8 <= R_omega < approximately 0.8944. The paper does not report how often this region is visited or compare against an otherwise identical ungated alignment penalty. This matters because selective preservation of useful diversity is a principal methodological distinction, not a peripheral implementation choice (pp. 6, 8–9, 15; Eqs. 38–42, Figure 4, Table 2).

    The narrow active region is not an algebraic contradiction: the thresholds satisfy the paper’s condition, and abstention under near cancellation is intentional. The missing evidence is whether those particular boundaries improve the useful accuracy/diversity trade-off. One matched ungated or simple-gate comparison, together with activation frequency, would test that claim more directly than extending the same strength sweep. This can be done on the existing diagnostic task rather than requiring all mechanisms to be rerun on all tasks.

  2. The diagnostics establish directional concentration more clearly than they establish improved update quality. Figure 3(c) uses within-run quantiles of distributional disagreement; its lowest quantile is relative to that run and is not itself an absolute test that critics agree closely in value or distribution. Increased resultant length and comparable gradient norms are consistent with the intended regularizer, but do not show that the resulting direction better improves the underlying return. In addition, Section 3.1 says propagation to actor-parameter updates is empirically examined, whereas the displayed diagnostics concern action-gradient geometry and magnitudes. The policy Jacobian can change these relationships, as the paper itself acknowledges. A limited diagnostic tying the action-space intervention to the actual parameter update or local realized improvement would strengthen the mechanism claim. This is a missing validation, not a claim that consensus must always be correct or that a global guarantee is required (pp. 2, 7–9, 13; Eq. 3, Figure 3, Appendix A.3).

    There is an additional interpretive distinction in the weighted resultant RwR_w: it may increase because gradient directions become more concentrated or because STG concentrates weights on already similar directions. The AGPA-only variant partly controls for this by retaining the original weighting. The full-method curve alone does not separate the effects. Reporting an unweighted geometry diagnostic alongside the actor-used weighting, or the promised parameter-space measurement, would clarify which changes actually occur. This is a request to interpret the present mechanism evidence, not an oracle requirement that every update improve true return.

  3. Comparison to the closest gradient-direction approach remains mainly conceptual. Section 2 appropriately cites vMFER, which already uses disagreement in critic gradient directions to influence actor-critic learning, and explains that DAGA intervenes through weighting/regularization rather than resampling. The listed experiments, however, compare primarily with general actor-critic algorithms and the fixed fusion Backbone, without a matched gradient-aware comparator. Thus the paper supports “these additions help this backbone” more strongly than “this is a better solution than existing gradient-direction interventions.” The concern is limited to comparative contribution; it is not an allegation that DAGA duplicates vMFER, and it does not require comparison to unpublished contemporaneous work (pp. 2, 7–9; Section 2 and Figures 2–4).

    A focused nearest-mechanism comparison would be more informative than adding many generic RL baselines. vMFER is a published extended abstract already discussed by the paper, and its replay-resampling intervention differs from DAGA’s weighting/regularization. The review therefore does not infer duplication or insufficient novelty from shared terminology. The question is whether the claimed advantage of the new intervention has been empirically distinguished from a relevant existing way of using the same gradient-direction information.

  4. The experimental protocol is not yet fully reconstructable, and the cost of AGPA is unquantified. Table 3 provides DAGA-specific settings, including batch size 8192, width-1024 critics, two updates per iteration, and “final” learning rates, but it does not specify the initial rates/schedule, complete environment collection protocol, baseline architectures/tuning budgets, or a common definition of update-to-data ratio across vectorized environments. This does not prove unfairness, but it limits interpretation of the cross-method comparison. AGPA also differentiates through action gradients with respect to critic parameters; this is a material additional operation, yet no time or memory comparison with the Backbone is reported. Environment-step curves alone cannot establish practical efficiency. A matched Backbone comparison and configuration details would resolve much of this concern (pp. 6–8, 15; Section 4.1, Eqs. 33/42–43 and Table 3).

The distinction between environment interactions, optimizer steps, and processed replay examples should be explicit because large batches and vectorized environments can produce very different workloads under the same plotted step label. Runtime reporting should quantify the additional mixed-derivative computation rather than assume that fewer environment steps imply lower cost. The paper does not claim a measured speedup, so the missing cost report limits the practical trade-off rather than refuting an asserted efficiency theorem.

Questions for authors
  1. What fraction of replay states activates AGPA across training, and does the stated selective gate improve the primary-task result relative to the same alignment loss with the gate fixed to Γ(s)=1\Gamma(s)=1 or replaced by a simple fixed gate? Can this be checked within the existing primary-task setup, without a new benchmark suite (Weakness 1)?

    The decisive evidence is the value of selective intervention, rather than whether any nonzero alignment coefficient can help. Please keep capacity, update budget, and tuning effort comparable between the alignment variants.

  2. Which result empirically measures propagation through the policy Jacobian, as promised in Section 3.1? If it was not measured, can that statement be narrowed and the current diagnostic interpreted as action-space concentration rather than validation of the actual actor-parameter direction? What are the absolute disagreement values in Figure 3(c)'s bins (Weakness 2)?

    If no parameter-space result exists, correcting the claim is acceptable. Please also distinguish a relatively low quantile from an absolute small-disagreement criterion when interpreting mismatch prevalence.

  3. How should the empirical contribution be distinguished from prior gradient-direction interventions such as vMFER? A bounded comparison on the primary diagnostic task, or a more carefully delimited claim explaining an incompatible experimental setting, would help assess the proposed mechanism's incremental value (Weakness 3).

    The response need not compare every cited method. One appropriate control, or a precise scope argument that avoids a broader comparative claim, would be enough to clarify this concern.

  4. Please specify initial learning rates and schedules, vectorized environment/step counting, baseline configuration and tuning provenance, and the measured time/memory increment of AGPA relative to the matched Backbone. Were the six-task results produced with one fixed DAGA configuration selected without using those tasks' test outcomes (Weakness 4)?

    Please state what “Backbone” holds fixed in the reported main comparison. This is more important to the recommendation than expanding the task suite or obtaining the highest score on every task.

1 comment
AuthorsOpenReview ↗

Gate Selectivity, Gradient Geometry, and Backbone

Thank you for separating the component evidence from the stronger questions of gate selectivity and update correctness.

Q1 — Selectivity versus unconditional alignment. We agree that the four-component comparison does not isolate the benefit of the two-sided gate. The preliminary gate-control runs used AGPA every critic update on 1,024 samples, whereas the submitted setting applies it every five updates on 128 samples. They cannot be combined with a submitted-setting full-DAGA result and interpreted as changing only the gate. We preserve those runs but do not claim a matched gate-isolation result for the submitted setting. Fig. 4(b) varies λA\lambda_A with the gate fixed; it is a strength-sensitivity analysis, not an ungated proxy. Multiplying λA\lambda_A cannot reproduce removing a state-dependent gate because it leaves the zero-support region unchanged. The revision therefore presents the dead zone and low-consensus abstention as design choices, without claiming established superiority in return or stability over unconditional alignment.

Q2 — Geometry, absolute disagreement and parameter updates. The Fig. 3(c) analysis groups logged probe-batch records pooled across runs, not individual states within each run. Its lowest-disagreement group spans δˉ∈[0.003,0.067]\bar\delta\in[0.003,0.067], with mean pairwise cosine approximately 0.42 and mean conflict ratio 0.085; these numbers are now stated in the mechanism section. This is a relative-low-disagreement group with an explicitly reported absolute range, not a universal value-agreement threshold. We also agree that concentration is not correctness. The revised problem setup no longer claims that the displayed action-gradient diagnostics validate propagation through the policy Jacobian to actor-parameter updates or realized returns. That connection requires a separate measurement and is not presented as completed here.

Q3 — Relation to vMFER. The related-work discussion distinguishes experience resampling from DAGA's critic-specific actor guidance and critic-side directional regularization. This is a distinction in intervention, not evidence of superiority over vMFER. We make no matched empirical-superiority claim without that comparison.

Q4 — Backbone and implementation details. The intended Backbone holds fixed the Gaussian critics, precision-fusion Bellman-target construction, distributional critic loss and TD3-style actor/target update mechanics. STG changes actor-guidance weights and AGPA adds a critic-side regularizer; these roles are now stated before the detailed equations. The training table now explicitly includes the 128-sample AGPA subset and five-critic-update execution interval. The ten-run component study and independent five-run sensitivity study use separate seed batches, so shared configurations need not have identical empirical means. We do not claim calibrated uncertainty, optimal critic consensus, a measured runtime advantage, or a task-selection/tuning procedure beyond the available experimental records.

Reviewer_9j5KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents DAGA which is a TD3 with N=3 Gaussian distributional critics combined by precision fusion. The motivation for this approach stems from "value-gradient mismatch". DAGA applies a gated critic regularizer to increase agreement when the gradients points towards similar directions. It also reweighs the N critics based on their distributional disagreement (STG).

Critical strengths
  1. The motivation ("value-gradient mismatch") is well-defined and informative. The analysis in Figure 3c shows that this is a phenomena (critics agreeing on value but disagreeing on gradient direction) happens in practice.
  2. DAGA shows a clear and consistent advantage over the basic (N=1) TD3.
Critical weaknesses
  1. The experimental setting specify small and non-static step budget per task, with a range of 45k to 250k steps which represent early training (longer training phases are needed to make solid claims regarding asymptotic performance and robustness).

  2. DAGA uses (large) batch size 8192, and 2 updates per iteration. It is unclear whether the baselines received similar capacity and update ratios. In prior papers I've seen claims that increased training effectiveness can be achieved via simply increasing the number of training (not environment) steps, see OT-DQN for example.

  3. Component ablations use one task and the 8 added hyperparameters were tuned on it.

  4. The paper does not report whether gate activation coincides with improved returns.

  5. The paper reports no wall-clock or per-update cost relative to the backbone and SimbaV2, so the tradeoffs between them cannot be assessed.

Questions for authors
  1. What are the per-task budgets in environment steps, and why do they differ? How do results look at standard budgets, for example 1M steps?
  2. What network sizes, batch sizes and update ratios do the baselines use, and were they tuned?
  3. Does mismatch predict instability, for example through correlations between conflict measures and return drops? How often is the AGPA gate active?
  4. What is the compute overhead of AGPA?
3 comments
AuthorsOpenReview ↗

Training Budget, Evaluation, and Computational Cost

Thank you for the concrete questions about budget, fairness, diagnostics and cost.

Q1 — Budget and longer training. We agree that finite-budget results do not establish asymptotic robustness. The Table 1 plotting protocol specifies 150,000 vectorized collector steps on CheetahRun and 100,000 on H1-Reach, with evaluation curves ending at 145,000 and 95,000, respectively. Separately, we trained the Gaussian Backbone and DAGA for 300,000 collector steps on each task, with one training seed (seed 0) per arm. At the last scheduled evaluation (295,000), returns were 939.42 versus 1019.38 on CheetahRun and 8898.43 versus 9398.23 on H1-Reach (Backbone versus DAGA), differences of +8.51% and +5.62%. The corresponding absolute gaps (about 80 and 500) are smaller than the Table 1 ten-seed summary gaps (about 274 and 1818); CheetahRun's single-seed DAGA return is also below its Table 1 summary of 1144.7±36.71144.7\pm36.7. Table 1 reports ten-seed final evaluation returns; the additional check reports one seed at collector step 295,000. This checkpoint follows the CheetahRun Table 1 curve endpoint at 145,000 and the H1-Reach endpoint at 95,000. It is a descriptive different-budget, different-seed contrast, not a paired estimate of how any Table 1 run changed. Six-checkpoint means over 270,000--295,000 are 937.63 versus 1017.51 and 8893.44 versus 9392.79, but checkpoints from one run are correlated, not independent seeds. These checkpoint values are the logged evaluation returns. Table 3 lists 10 evaluation episodes for the main setup; this single-seed check was not separately re-audited against that episode count. DAGA uses AGPA every five critic updates on 128 samples, with r0=0.8r_0=0.8 and λA=0.1\lambda_A=0.1; the different AGPA interval/subset fields in the H1-Reach Backbone are inactive because AGPA is disabled. One collector step uses 1024 parallel environments on CheetahRun and 128 on H1-Reach, not one transition. The smaller observed gaps do not establish stability of the Table 1 advantage or asymptotic performance, and no matched wall-clock/peak-memory result is available.

Q2 — Configurations and tuning. PPO, SAC, DDPG and REDQ were run by us, rather than copied from published score tables. TD3 and SimbaV2 are also included in Table 1; their per-method training configurations and result provenance should not be inferred from DAGA settings alone. The DAGA training table specifies a critic width of 1024 with two hidden blocks, actor width of 256 with one hidden block, batch size 8192, and two gradient updates per iteration. We have added the AGPA-specific subset size and execution interval. These DAGA settings do not by themselves prove that every external baseline has matched capacity, optimization work or tuning effort. We therefore do not use shared codebase membership as a substitute for per-method configuration evidence.

Q3 — Predictive instability and activation. The manuscript treats the gradient measurements as critic-level diagnostics. The current evidence does not establish a reliable prediction of future return drops, nor does an equal-weight resultant mean determine the activation fraction of a gate using weighted Rω(s)R_\omega(s). In particular, the activation rates from the every-update, 1,024-sample supplemental runs are not reported as rates for the submitted every-five-update, 128-sample protocol. The component and sensitivity results remain useful evidence, but do not close this measurement gap.

Q4 — Computational overhead. We do not yet have a matched wall-clock and peak-memory measurement to report. Each scheduled AGPA application computes critic action gradients and differentiates the alignment loss through those gradients with respect to critic parameters. The reported schedule—128 replay samples once every five critic updates—specifies the additional computation, not a measured runtime bound or a guarantee of negligible overhead. We make no wall-clock-efficiency claim without that measurement.

Ablation breadth and sensitivity. The mechanism study remains concentrated on one task. Fig. 4 varies intervention strength and ensemble size, but this does not establish robustness to every new hyperparameter or replace component ablations across tasks. We retain these limitations rather than interpreting a small study as universal robustness.

Reviewer_9j5KOpenReview ↗

Thank you for clarifying

Budget. Thank you for clarifying. This corrects my reading: the budgets are large in transitions, not early training. However, the paper describes these as environment steps, and parallelism differs across tasks. Please report the number of environment transitions and parallel environments per task, for DAGA and every baseline. The single-seed 300k check shows smaller gaps than Table 1 and cannot establish the trend.

Baselines. My question remains unanswered. Please report the batch size, network widths, updates per collector step and parallel environments for each baseline, and state which baselines were run by you and which were imported. DAGA’s setup closely resembles FastTD3 (Seo et al., 2025): massively parallel TD3, large batches, distributional critics and the same task suites. Is the TD3 baseline FastTD3 or standard TD3? If the baselines were trained in a different regime, Table 1 does not isolate DAGA’s contribution. FastTD3 should be cited and compared.

Gate and instability. I understand that activation rates are available only for a different configuration. I agree with Reviewer v6oY that reporting the existing ungated runs, clearly labeled with their configuration, would be more informative.

Compute. A matched wall-clock comparison with the backbone is needed to assess the trade-off.

Overall. The response clarifies the training scale but provides no new evidence on the main concerns. The baseline configurations, gate activation and compute cost remain unreported, and the benefit of the gate’s selectivity is now explicitly unsupported. I maintain my score.

AuthorsOpenReview ↗

Additional Protocol Details and Supplemental Gate-Control Results

Thank you for the follow-up. We have corrected the step terminology and strong-baseline identity in the revision, added the FastTD3 citation (Seo et al., 2025), and summarized the available gate controls below.

Training scale. For DAGA and its Gaussian Backbone, the configured budgets are vectorized collector steps; one such step collects one transition per parallel training environment. Evaluation interactions are excluded from the nominal transition counts:

TaskCollector stepsParallel environmentsNominal training transitions
T1-Rough100,0001,024102.4M
Leap-Reorient150,0001,024153.6M
H1-Reach100,00012812.8M
H1-Basket250,00012832.0M
Go1-Getup50,0001,02451.2M
CheetahRun150,0001,024153.6M

The DAGA/Backbone launcher specifies batch size 8,192, a 1,024-wide two-block critic, a 256-wide one-block actor, and two gradient updates per collector iteration. These are settings for those two arms, not the other baselines. The separate 300,000-step check has one training seed per arm and smaller gaps than Table 1; it cannot establish a late-training trend.

Baselines. PPO, SAC, DDPG, REDQ, and TD3 were independently run by us. The separate TD3 row uses standard TD3, not FastTD3. The strong-baseline row formerly labeled SimbaV2 actually used agent=td3_simbav2 and is now labeled FastTD3+SimbaV2, with both methods cited; it is not the original SAC-based SimbaV2 setup.

This identifies the systems compared, but the score exports do not provide a trustworthy per-baseline table of batch sizes, network widths, update ratios, and parallel environments. We cannot transfer the DAGA settings above to those methods or call Table 1 a matched-capacity comparison. The component ablations within the Gaussian Backbone address the narrower contribution of STG and AGPA under a shared setup.

Supplemental gate controls. On h1hand-hurdle-v0, the additional gate-mode runs use AGPA at every critic update on a 1,024-sample replay subset. For common training seeds 0 and 1, each run's score is its env_rewards mean over the final 10% of logged collector steps.

Gate activation is the fraction of eligible states with a nonzero gate, averaged within each run and then across the two seeds. Eligible states have at least two valid action-gradient directions.

Gate modeSeed 0 / seed 1Two-seed meanMean gate activation
Always (no selection conditions)0.901 / 0.8710.886100%
Dead-zone only0.779 / 0.8890.83412.2%
Abstention only0.482 / 0.7610.62198.8%
Full two-sided gateapproximately 0.86 / 0.830.849approximately 8%

These supplemental controls provide direct measurements of how the selection conditions affect activation. The full two-sided gate activates on approximately 8% of eligible states, compared with 100% for always alignment. Across these separately trained runs, dead-zone-only alignment activates much less frequently than abstention-only alignment.

Full gating achieves a two-seed mean training reward of 0.849, compared with 0.886 for always alignment. Thus, these runs document selective application; their two-seed mean rewards do not show an advantage for full gating over always alignment. These results concern the supplemental K=1K=1, subset-size-1,024 protocol; the activation fraction under the submitted K=5K=5, subset-size-128 protocol remains to be measured.

Compute. AGPA differentiates through critic action gradients. The submitted every-five-update, 128-sample schedule is specified, but no matched same-hardware wall-clock or peak-memory comparison with the Backbone has been measured; we make no efficiency claim.

Reviewer_sChsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DAGA, a distributional multi-critic actor-critic method for continuous control. The paper studies the value-gradient mismatch problem, where different critics may have similar Q-value estimates but different action-gradient directions. DAGA introduces two components to address this problem. Stability-Triggered Guidance (STG) adjusts the critic weights based on distributional disagreement, and Action-Gradient Policy Alignment (AGPA) regularizes the critic action gradients to encourage directional agreement. The paper also provides some theoretical analysis of value-gradient mismatch and the proposed methods. The experiments evaluate DAGA on six continuous-control tasks, and compare with several baselines including PPO, SAC, TD3, REDQ, and SimbaV2. DAGA achieves the highest mean return on four of the six tasks, and improves over its Gaussian multi-critic backbone on all six tasks. The paper also includes ablation and sensitivity experiments on one task, h1hand-hurdle-v0.

Critical strengths

None

Critical weaknesses
  1. The motivation of this paper is unclear to me. The paper argues that value-gradient mismatch is an important problem in multi-critic RL, but I don't think it clearly explains why disagreement among critic action gradients is necessarily harmful to policy learning. Such disagreement may simply reflect uncertainty among critics, and averaging the gradients could already help reduce the effect of conflicting directions. The experiments show that value-gradient mismatch exists and AGPA can increase gradient alignment, but I don't see sufficient evidence that this mismatch actually causes poor performance, or that better alignment leads to more accurate policy updates.

  2. I find this paper quite difficult to follow, especially Section 3. There are too many definitions, equations and intermediate variables introduced without enough explanation of the intuition behind them. I had to go back and forth between different subsections to understand how STG and AGPA work together. Also, I don't quite understand the motivation behind some design choices. For example, why does STG use distributional disagreement to adjust critic weights, while the main problem is about action-gradient disagreement? I think the paper should first give a clearer high-level explanation of the method before going into all the mathematical details. The lack of clarity makes it difficult to assess the technical contributions of the paper.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.