- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper shows that the differentiability assumption underlying the deterministic policy gradient theorem is governed by a race between the discount and a separation rate that depends on the environment: below < 1 the action value is smooth in the action, beyond it only Hölder continuous with exponent , and on a hyperbolic attractor the action-gradient is absent at almost every state. Section 4 argues that what TD3 actually follows is the slope of an action value smoothed by the target noise and the critic's resolution. Experiments on TD3 agents measure the separation rate with a "two-copy" simulator test and validate the predicted exponent against direct action-value measurements.
Critical strengths
-
The observation that DPG's differentiability assumption is decided by a discount-separation race is right and important. It identifies a concrete failure mode of the foundational assumption of DDPG/TD3/DPG on the very benchmarks used to evaluate them. That result has value independently of the rest of the paper.
-
Casting the target noise and the critic's resolution as a deliberate smoothing mechanism whose state-averaged slope recovers the return gradient, turns implementation folklore into a potential theory with direct implications for choosing the exploration noise, network capacity, and minibatch size. If substantiated, it would explain why deterministic actor-critic still works where its own justification fails.
Critical weaknesses
-
The theory does not fully apply to the experiments that motivate it. For instance, Theorem 3 ("no gradient at almost every state") holds only under Assumption 2. However, it is not verified on any benchmark, and the assumptions look implausible for the four contact-rich tasks, where ground contact makes the dynamics non-smooth. The experiments measure the Hölder scaling , which cannot establish almost everywhere non-differentiability. Similarly Theorem 5 rely on a lot of unverified assumptions: the dynamics, reward, and policy be seven times differentiable; that every slightly perturbed policy yield a closed loop that is chaotic and yet forgets its initial state at one common rate; that the critic be trained on fresh data from the current policy, with exploration noise the same size as the target noise, whereas the experiments use 0.1 for one and 0.2 for the other (and replay old data); and it simply leaves out the minimum over the two critics, one of TD3's defining features. Finally, the theorem only covers critics that resolve finer detail than the width of the target noise, and the authors' own measurements show that for 13 of the 42 trained agents the critic is coarser than that noise, so the theorem cannot apply to them.
-
The claim that the smoothed slope "is why the method still works beyond the threshold" is not tested: no experiment compares the critic-supplied update direction against an independent estimate of , there are no DDPG = 0 agents, and no training-time sweep against the predicted slope scaling.
Questions for authors
- How do you reconcile Theorem 5's condition with the 13 runs of Table 4? Does the "unbiased up to " conclusion survive on those agents?
- Appendix C.3 assumes on-policy data whose action-offsets have the same width as the target noise, but the agents are trained with . What does the analysis predict in the actual configuration?
- The causal story (see second weakness above) is testable by training DDPG = 0 agents and TD3 at several , then comparing the trained critics' action-gradients against the predicted scaling and against a finite-difference estimate of , it would directly confirm the paper's central claim.
4 comments
Paper updates and clarifications
3. The direction of the update and the sweep over
"no experiment compares the critic-supplied update direction against an independent estimate of , there are no DDPG agents, and no training-time sweep"
Thank you, this was a great suggestion, and it gave the paper its most direct test. We removed the sentence and ran this experiment.
We trained TD3 at and DDPG with three seeds on each task. On the six tasks other than Pendulum, where is close to zero at the trained actors, we compared the update from each trained critic with estimated by central differences of the return. We call an update uphill or downhill when the 16th to 84th percentile interval of their cosine lies above or below zero (Appendix F).
| uphill | downhill | undecided | |
|---|---|---|---|
| TD3, | 31 | 18 | 19 |
| TD3, | 3 | 10 | 5 |
| TD3, , first critic only | 10 | 3 | 5 |
| DDPG | 8 | 2 | 7 |
The target noise turns the update of TD3 from mostly downhill to mostly uphill, from 23% to 63% of the decided runs (one-sided Fisher test, ), and this holds without InvertedDoublePendulum too, whose actors sit near a stationary point of ().
DDPG has no target noise either, yet it mostly points uphill. Since it differs from TD3 at in the minimum over the two critics and in the delayed actor update, we also trained TD3 without noise and with targets that use only its first critic (third row). It behaves like DDPG ( against TD3 with the minimum), so without noise it's the minimum that turns the update downhill, as you suspected.
To give a sense of scale, the median cosine is with target noise, while a random direction among the 67,000 to 95,000 parameters of an actor would give about .
"the predicted slope scaling"
Theorem 4 bounds the slope of by a constant times , so we measured that slope directly, from rollouts with the noise clipped as in TD3. On the paper's HalfCheetah agent (), its spread over states grows as , with a 95% interval of that contains the of the theorem.
Across three agents with a relu critic per task, the growth is no faster than this rate on all agents but one, and on every HalfCheetah and Ant agent it's compatible with . On InvertedDoublePendulum and Pendulum, where the state stays near a fixed point, the slope doesn't grow, which the theorem allows, since its lower bound needs the expanding attractor of Theorem 3.
The trained critics are a different story. Their slope changes little with , and at it's 8 to 11 times smaller than that of . This is within the bound of Proposition 5, but it means the critic doesn't follow the slope of the smoothed value, so we corrected the sentences that said it does. The agreement with above is about the update averaged over states, which is the quantity of Theorem 5.
Paper updates and clarifications
4. The coarse critics
"How do you reconcile Theorem 5's condition with the 13 runs of Table 4? Does the 'unbiased up to ' conclusion survive on those agents?"
Good catch. The condition was there to say when a trained critic resolves the smoothed value finely enough, but the proof doesn't use it, so we removed it. The conclusion does survive on those agents, because it's about and doesn't involve the critic at all.
5. The actual configuration
"What does the analysis predict in the actual configuration?"
Good question. Theorem 5 is not affected, because it is about the smoothed value and doesn't depend on how the critic is trained.
What changes is our analysis of the critic. Propositions 5 and 6 assume that the critic is trained on fresh data, with actions perturbed by noise of the same width as the target noise. In TD3 the exploration noise (0.1) is narrower than the target noise (0.2), the data come from a replay of older actors, the critic is a neural network, and the targets take the minimum of two critics. The propositions don't cover any of this, so in the actual configuration our analysis makes no prediction for the critic. Appendix C.3 now says so, and this is why we measured the update of the trained critic directly (point 3).
Paper updates and clarifications
Thank you again for these suggestions, which really shaped the new experiments. They support the claim on the averaged update and correct the one on the slope of the critic. We believe these revisions address the concerns you raised, and we hope they make the paper an accept in your view. If anything remains open, please let us know, we'd be very happy to discuss!
Paper updates and clarifications
Thank you so much for the thorough and constructive review! We're really glad that you find the race between discount and separation "right and important", and that you see in the target noise and the resolution of the critic a possible reason why deterministic actor-critic works even where its justification fails.
Your third question was especially helpful, and we ran the experiment you proposed. We have updated the paper with new experiments, most of them suggested in your review. They turned out to be very informative, and your suggestions made the paper even stronger. In summary, we have
The new experiments are in Appendix E, and Section 5 summarises them. We answer each of your points below.
1. Theorem 3 and Assumption 2
That's a fair point, and we agree. A measurement at finite displacements can't prove that a derivative is missing, and our agents don't satisfy Assumption 2, because of their relu actors and, on four tasks, because of contact. So Section 5 now says that the experiments test Theorems 1 and 2, and that what they find is consistent with Theorem 3. We also changed the abstract and the introduction, which now say that trained agents can violate the differentiability assumption.
Still, the measurements are hard to reconcile with a derivative. On every HalfCheetah and Ant agent the action value changes as hα with α≤0.06 down to h=10−5, while a derivative would give α=1.
2. The assumptions of Theorem 5
Thanks for going through them so carefully. You're right, and we don't claim that our agents satisfy them. Theorem 5 is a statement about the exact smoothed value Qπσ~, and seven derivatives with one common rate of forgetting are the standard conditions under which the transfer operator of a chaotic system has a spectral gap that survives small smooth changes (Gouëzel and Liverani, 2006, 2008). Relu actors and contact don't meet them, which is why we also tested its conclusion directly on trained agents (point 3).
We answer the points on the noise, the replay and the coarse critics in 4 and 5 below. And you're right about the minimum over the two critics. Our analysis leaves it out, and the new experiments show that it matters a lot (point 3).