- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DAGA studies disagreement between the action gradients of critics that may agree on their predicted values. It retains a Gaussian multi-critic precision-fusion backbone and adds two mechanisms: STG adjusts detached critic weights for actor guidance using distributional disagreement, and AGPA regularizes normalized critic action gradients only when dispersion exceeds a threshold but a sufficiently coherent consensus remains. The paper provides local mathematical characterizations, a component study on h1hand-hurdle-v0, and ten-seed comparisons on six additional continuous-control tasks. The empirical improvements over the fixed backbone are useful, although the evidence does not fully isolate the value of selective intervention or establish the comparative advantage over the closest gradient-aware alternatives.
The two additions act at different points. STG changes how critic means contribute to the actor while keeping its weighting variables detached; AGPA changes the critics themselves by differentiating an alignment penalty through their action gradients at detached actor actions. The Gaussian Bellman/precision-fusion backbone remains fixed. This separation is important because improved actor weighting and improved gradient geometry are different hypotheses, and agreement between critics is not equivalent to correctness of their shared direction (pp. 3–7, Eqs. 5–45).
Critical strengths
-
The paper identifies the actor-facing quantity precisely. Equations 2–3 separate critic action gradients from their transformation by the policy Jacobian. Appendix A.1 correctly distinguishes pointwise agreement from neighborhood agreement: equal values at one action need not imply equal derivatives, whereas smooth local functional agreement constrains gradient differences. This avoids treating a simple pointwise counterexample as a general theorem that all critic disagreement is harmful (pp. 2–3, 11–12).
The distinction also prevents an overly strong negative conclusion about value fitting. With smoothness and agreement on a neighborhood, Appendix A.1 bounds derivative disagreement; the paper is therefore not claiming that value information can never constrain gradients. This makes the diagnostic problem precise and gives the proposed intervention an appropriately local motivation.
-
The construction is explicit about what is and is not optimized. STG weights are detached during actor optimization; AGPA retains the critic's derivative graph but detaches actor actions, guidance weights, and gate decisions. The separate moment-matched reference also avoids using the mechanically shrinking variance of precision fusion as an ensemble-disagreement reference. These distinctions make the proposed algorithm substantially more assessable than a generic “uncertainty-weighted” update (pp. 3–7, Eqs. 5–45).
In particular, detaching the guidance weights in the alignment objective prevents the regularizer from being reduced merely by reassigning weights instead of changing directions. Detaching the actor action prevents the critic alignment loss from directly moving the actor. Those are concrete implementation decisions that help distinguish the proposed objective from an underspecified consensus penalty (p. 6).
-
Reported performance is supported by ten independent training seeds. DAGA improves the mean final return over its Backbone on all six cross-task evaluations and has the highest mean in four. The strongest gaps over the best listed comparator are H1-Reach, 7029.1 versus 5848.4, approximately +20.2%, and Cheetah-Run, 1144.7 versus 1019.0, approximately +12.3%. The paper also correctly acknowledges that PPO and SimbaV2 lead on T1-Rough and H1-Basket, respectively. These are meaningful results without requiring that the method win every task (pp. 7–8, Section 4.2, Figure 2/Table 1).
The gains over Backbone are also distributed across the task suite: for example, H1-Basket improves from 738.1 to 1001.7 even though it does not beat the strongest competitor. That is useful evidence for adding the mechanisms to this backbone. It should be distinguished from a claim that every nominal lead over every external baseline is statistically established.
-
The primary-task study includes the relevant two-component combinations. Figure 3 compares Backbone, Backbone+STG, Backbone+AGPA, and full DAGA, and tracks both resultant length and gradient magnitude. Figure 4 additionally varies intervention strength, alignment weight, and ensemble size. The conclusion recognizes biased consensus and non-monotonic ensemble scaling as limitations rather than claiming correctness from agreement alone (pp. 7–9, Figures 3–4 and Section 5).
The gradient-magnitude diagnostic is a useful anti-collapse check because higher agreement alone could be achieved by making critic derivatives uninformative. The reported comparable mean magnitudes do not prove derivative accuracy, but they do address one simple degenerate explanation. Similarly, the full-versus-single-component comparison is meaningful intervention evidence and should not be described as a purely correlational study.
Critical weaknesses
-
The central benefit of selective alignment is not isolated from ordinary alignment. The 2x2 component study shows that adding STG and AGPA helps in the chosen diagnostic task; it does not show that the proposed dead zone and no-consensus abstention are what make AGPA effective. The sensitivity plot changes eta_max and lambda_A, but not the gate thresholds A0 and r0. At the defaults A0=0.2 and r0=0.8, intervention is restricted to 0.2 < A_omega <= 0.36, equivalently 0.8 <= R_omega < approximately 0.8944. The paper does not report how often this region is visited or compare against an otherwise identical ungated alignment penalty. This matters because selective preservation of useful diversity is a principal methodological distinction, not a peripheral implementation choice (pp. 6, 8–9, 15; Eqs. 38–42, Figure 4, Table 2).
The narrow active region is not an algebraic contradiction: the thresholds satisfy the paper’s condition, and abstention under near cancellation is intentional. The missing evidence is whether those particular boundaries improve the useful accuracy/diversity trade-off. One matched ungated or simple-gate comparison, together with activation frequency, would test that claim more directly than extending the same strength sweep. This can be done on the existing diagnostic task rather than requiring all mechanisms to be rerun on all tasks.
-
The diagnostics establish directional concentration more clearly than they establish improved update quality. Figure 3(c) uses within-run quantiles of distributional disagreement; its lowest quantile is relative to that run and is not itself an absolute test that critics agree closely in value or distribution. Increased resultant length and comparable gradient norms are consistent with the intended regularizer, but do not show that the resulting direction better improves the underlying return. In addition, Section 3.1 says propagation to actor-parameter updates is empirically examined, whereas the displayed diagnostics concern action-gradient geometry and magnitudes. The policy Jacobian can change these relationships, as the paper itself acknowledges. A limited diagnostic tying the action-space intervention to the actual parameter update or local realized improvement would strengthen the mechanism claim. This is a missing validation, not a claim that consensus must always be correct or that a global guarantee is required (pp. 2, 7–9, 13; Eq. 3, Figure 3, Appendix A.3).
There is an additional interpretive distinction in the weighted resultant : it may increase because gradient directions become more concentrated or because STG concentrates weights on already similar directions. The AGPA-only variant partly controls for this by retaining the original weighting. The full-method curve alone does not separate the effects. Reporting an unweighted geometry diagnostic alongside the actor-used weighting, or the promised parameter-space measurement, would clarify which changes actually occur. This is a request to interpret the present mechanism evidence, not an oracle requirement that every update improve true return.
-
Comparison to the closest gradient-direction approach remains mainly conceptual. Section 2 appropriately cites vMFER, which already uses disagreement in critic gradient directions to influence actor-critic learning, and explains that DAGA intervenes through weighting/regularization rather than resampling. The listed experiments, however, compare primarily with general actor-critic algorithms and the fixed fusion Backbone, without a matched gradient-aware comparator. Thus the paper supports “these additions help this backbone” more strongly than “this is a better solution than existing gradient-direction interventions.” The concern is limited to comparative contribution; it is not an allegation that DAGA duplicates vMFER, and it does not require comparison to unpublished contemporaneous work (pp. 2, 7–9; Section 2 and Figures 2–4).
A focused nearest-mechanism comparison would be more informative than adding many generic RL baselines. vMFER is a published extended abstract already discussed by the paper, and its replay-resampling intervention differs from DAGA’s weighting/regularization. The review therefore does not infer duplication or insufficient novelty from shared terminology. The question is whether the claimed advantage of the new intervention has been empirically distinguished from a relevant existing way of using the same gradient-direction information.
-
The experimental protocol is not yet fully reconstructable, and the cost of AGPA is unquantified. Table 3 provides DAGA-specific settings, including batch size 8192, width-1024 critics, two updates per iteration, and “final” learning rates, but it does not specify the initial rates/schedule, complete environment collection protocol, baseline architectures/tuning budgets, or a common definition of update-to-data ratio across vectorized environments. This does not prove unfairness, but it limits interpretation of the cross-method comparison. AGPA also differentiates through action gradients with respect to critic parameters; this is a material additional operation, yet no time or memory comparison with the Backbone is reported. Environment-step curves alone cannot establish practical efficiency. A matched Backbone comparison and configuration details would resolve much of this concern (pp. 6–8, 15; Section 4.1, Eqs. 33/42–43 and Table 3).
The distinction between environment interactions, optimizer steps, and processed replay examples should be explicit because large batches and vectorized environments can produce very different workloads under the same plotted step label. Runtime reporting should quantify the additional mixed-derivative computation rather than assume that fewer environment steps imply lower cost. The paper does not claim a measured speedup, so the missing cost report limits the practical trade-off rather than refuting an asserted efficiency theorem.
Questions for authors
-
What fraction of replay states activates AGPA across training, and does the stated selective gate improve the primary-task result relative to the same alignment loss with the gate fixed to or replaced by a simple fixed gate? Can this be checked within the existing primary-task setup, without a new benchmark suite (Weakness 1)?
The decisive evidence is the value of selective intervention, rather than whether any nonzero alignment coefficient can help. Please keep capacity, update budget, and tuning effort comparable between the alignment variants.
-
Which result empirically measures propagation through the policy Jacobian, as promised in Section 3.1? If it was not measured, can that statement be narrowed and the current diagnostic interpreted as action-space concentration rather than validation of the actual actor-parameter direction? What are the absolute disagreement values in Figure 3(c)'s bins (Weakness 2)?
If no parameter-space result exists, correcting the claim is acceptable. Please also distinguish a relatively low quantile from an absolute small-disagreement criterion when interpreting mismatch prevalence.
-
How should the empirical contribution be distinguished from prior gradient-direction interventions such as vMFER? A bounded comparison on the primary diagnostic task, or a more carefully delimited claim explaining an incompatible experimental setting, would help assess the proposed mechanism's incremental value (Weakness 3).
The response need not compare every cited method. One appropriate control, or a precise scope argument that avoids a broader comparative claim, would be enough to clarify this concern.
-
Please specify initial learning rates and schedules, vectorized environment/step counting, baseline configuration and tuning provenance, and the measured time/memory increment of AGPA relative to the matched Backbone. Were the six-task results produced with one fixed DAGA configuration selected without using those tasks' test outcomes (Weakness 4)?
Please state what “Backbone” holds fixed in the reported main comparison. This is more important to the recommendation than expanding the task suite or obtaining the highest score on every task.
Gate Selectivity, Gradient Geometry, and Backbone
Thank you for separating the component evidence from the stronger questions of gate selectivity and update correctness.
Q1 — Selectivity versus unconditional alignment. We agree that the four-component comparison does not isolate the benefit of the two-sided gate. The preliminary gate-control runs used AGPA every critic update on 1,024 samples, whereas the submitted setting applies it every five updates on 128 samples. They cannot be combined with a submitted-setting full-DAGA result and interpreted as changing only the gate. We preserve those runs but do not claim a matched gate-isolation result for the submitted setting. Fig. 4(b) varies λA with the gate fixed; it is a strength-sensitivity analysis, not an ungated proxy. Multiplying λA cannot reproduce removing a state-dependent gate because it leaves the zero-support region unchanged. The revision therefore presents the dead zone and low-consensus abstention as design choices, without claiming established superiority in return or stability over unconditional alignment.
Q2 — Geometry, absolute disagreement and parameter updates. The Fig. 3(c) analysis groups logged probe-batch records pooled across runs, not individual states within each run. Its lowest-disagreement group spans δˉ∈[0.003,0.067], with mean pairwise cosine approximately 0.42 and mean conflict ratio 0.085; these numbers are now stated in the mechanism section. This is a relative-low-disagreement group with an explicitly reported absolute range, not a universal value-agreement threshold. We also agree that concentration is not correctness. The revised problem setup no longer claims that the displayed action-gradient diagnostics validate propagation through the policy Jacobian to actor-parameter updates or realized returns. That connection requires a separate measurement and is not presented as completed here.
Q3 — Relation to vMFER. The related-work discussion distinguishes experience resampling from DAGA's critic-specific actor guidance and critic-side directional regularization. This is a distinction in intervention, not evidence of superiority over vMFER. We make no matched empirical-superiority claim without that comparison.
Q4 — Backbone and implementation details. The intended Backbone holds fixed the Gaussian critics, precision-fusion Bellman-target construction, distributional critic loss and TD3-style actor/target update mechanics. STG changes actor-guidance weights and AGPA adds a critic-side regularizer; these roles are now stated before the detailed equations. The training table now explicitly includes the 128-sample AGPA subset and five-critic-update execution interval. The ten-run component study and independent five-run sensitivity study use separate seed batches, so shared configurations need not have identical empirical means. We do not claim calibrated uncertainty, optimal critic consensus, a measured runtime advantage, or a task-selection/tuning procedure beyond the available experimental records.