acceptodds
Under review as a conference paper at ICLR 2027

Deterministic Actor–Critic Without an Action-Gradient

Abstract

Deterministic policy gradient methods such as DDPG and TD3 improve the actor by ascending the action-gradient of a learned critic, and their analysis assumes that the action value is differentiable in the action. That assumption fails on agents trained on commonly used continuous control benchmarks, and whether it fails is decided by the trained agent and not by the task alone. Differentiability in the action is governed by a competition between the discount factor and the rate at which the closed loop separates nearby states. We give the threshold below which the action-gradient exists, together with the stricter one below which the action-curvature does, and show that a loop that stretches fast enough on a hyperbolic attractor leaves no gradient at almost every state of its basin. The actor ascends the gradient of an action value smoothed by the noise TD3 adds to its targets and by the resolution of the critic. The width of the smoothing sets the size of that gradient: it is at most an inverse power of the width, and in the mean square near the attractor it grows as an inverse power when the noise is reduced. Averaged over the states the actor visits, this slope is the gradient of the return, up to a positive factor and the square of the noise, which is why the method still works beyond the threshold. On agents trained on standard benchmarks we measure the separation rate and the exponent it predicts, test the prediction against direct measurements of the action value, and give a diagnostic that detects, from the simulator alone, when a trained agent lies beyond the threshold.

Primary area:
reinforcement learning
Keywords:
deterministic policy gradient, actor-critic, DDPG, TD3, differentiability, action-gradient, chaotic dynamics, Lyapunov exponent, exploration noise, continuous control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hSCkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that the differentiability assumption underlying the deterministic policy gradient theorem is governed by a race between the discount γ\gamma and a separation rate Λ\Lambda that depends on the environment: below γΛ\gamma\Lambda < 1 the action value is smooth in the action, beyond it only Hölder continuous with exponent α\alpha, and on a hyperbolic attractor the action-gradient is absent at almost every state. Section 4 argues that what TD3 actually follows is the slope of an action value smoothed by the target noise σ~\tilde\sigma and the critic's resolution. Experiments on TD3 agents measure the separation rate with a "two-copy" simulator test and validate the predicted exponent against direct action-value measurements.

Critical strengths
  1. The observation that DPG's differentiability assumption is decided by a discount-separation race is right and important. It identifies a concrete failure mode of the foundational assumption of DDPG/TD3/DPG on the very benchmarks used to evaluate them. That result has value independently of the rest of the paper.

  2. Casting the target noise and the critic's resolution as a deliberate smoothing mechanism whose state-averaged slope recovers the return gradient, turns implementation folklore into a potential theory with direct implications for choosing the exploration noise, network capacity, and minibatch size. If substantiated, it would explain why deterministic actor-critic still works where its own justification fails.

Critical weaknesses
  1. The theory does not fully apply to the experiments that motivate it. For instance, Theorem 3 ("no gradient at almost every state") holds only under Assumption 2. However, it is not verified on any benchmark, and the assumptions look implausible for the four contact-rich tasks, where ground contact makes the dynamics non-smooth. The experiments measure the Hölder scaling α\alpha, which cannot establish almost everywhere non-differentiability. Similarly Theorem 5 rely on a lot of unverified assumptions: the dynamics, reward, and policy be seven times differentiable; that every slightly perturbed policy yield a closed loop that is chaotic and yet forgets its initial state at one common rate; that the critic be trained on fresh data from the current policy, with exploration noise the same size as the target noise, whereas the experiments use 0.1 for one and 0.2 for the other (and replay old data); and it simply leaves out the minimum over the two critics, one of TD3's defining features. Finally, the theorem only covers critics that resolve finer detail than the width of the target noise, and the authors' own measurements show that for 13 of the 42 trained agents the critic is coarser than that noise, so the theorem cannot apply to them.

  2. The claim that the smoothed slope "is why the method still works beyond the threshold" is not tested: no experiment compares the critic-supplied update direction against an independent estimate of ∇J\nabla J, there are no DDPG σ~\tilde\sigma = 0 agents, and no training-time σ~\tilde\sigma sweep against the predicted σ~α−1\tilde\sigma^{\alpha-1} slope scaling.

Questions for authors
  1. How do you reconcile Theorem 5's condition ls≤LFσ~l_s \leq L_F \tilde\sigma with the 13 runs of Table 4? Does the "unbiased up to σ~2\tilde\sigma^2" conclusion survive on those agents?
  2. Appendix C.3 assumes on-policy data whose action-offsets have the same width as the target noise, but the agents are trained with σ≠σ~\sigma \neq \tilde\sigma. What does the analysis predict in the actual configuration?
  3. The causal story (see second weakness above) is testable by training DDPG σ~\tilde\sigma = 0 agents and TD3 at several σ~\tilde\sigma, then comparing the trained critics' action-gradients against the predicted σ~α−1\tilde\sigma^{\alpha-1} scaling and against a finite-difference estimate of ∇J\nabla J, it would directly confirm the paper's central claim.
4 comments
AuthorsOpenReview ↗

Paper updates and clarifications

Thank you so much for the thorough and constructive review! We're really glad that you find the race between discount and separation "right and important", and that you see in the target noise and the resolution of the critic a possible reason why deterministic actor-critic works even where its justification fails.

Your third question was especially helpful, and we ran the experiment you proposed. We have updated the paper with new experiments, most of them suggested in your review. They turned out to be very informative, and your suggestions made the paper even stronger. In summary, we have

  1. run your experiment on all seven tasks, training TD3 at five target-noise widths and DDPG with three seeds each, and comparing the update of each trained critic with a finite-difference estimate of ∇θJ\nabla_\theta J,
  2. added an ablation without the minimum over the two critics, which shows why TD3 without target noise behaves differently from DDPG,
  3. measured the σ~α−1\tilde\sigma^{\alpha-1} scaling of Theorem 4 on all seven tasks,
  4. removed the condition ℓs≤LFσ~\ell_s\le L_F\tilde\sigma from Theorem 5, since its proof doesn't use it,
  5. made the scope of each result explicit in Section 5 and Appendix C.3, and removed "which is why the method still works beyond the threshold".

The new experiments are in Appendix E, and Section 5 summarises them. We answer each of your points below.

1. Theorem 3 and Assumption 2

"Theorem 3 ('no gradient at almost every state') holds only under Assumption 2. However, it is not verified on any benchmark"

That's a fair point, and we agree. A measurement at finite displacements can't prove that a derivative is missing, and our agents don't satisfy Assumption 2, because of their relu actors and, on four tasks, because of contact. So Section 5 now says that the experiments test Theorems 1 and 2, and that what they find is consistent with Theorem 3. We also changed the abstract and the introduction, which now say that trained agents can violate the differentiability assumption.

Still, the measurements are hard to reconcile with a derivative. On every HalfCheetah and Ant agent the action value changes as hαh^{\alpha} with α≤0.06\alpha\le0.06 down to h=10−5h=10^{-5}, while a derivative would give α=1\alpha=1.

2. The assumptions of Theorem 5

"Similarly Theorem 5 rely on a lot of unverified assumptions"

Thanks for going through them so carefully. You're right, and we don't claim that our agents satisfy them. Theorem 5 is a statement about the exact smoothed value Qπσ~Q^{\pi_{\tilde\sigma}}, and seven derivatives with one common rate of forgetting are the standard conditions under which the transfer operator of a chaotic system has a spectral gap that survives small smooth changes (Gouëzel and Liverani, 2006, 2008). Relu actors and contact don't meet them, which is why we also tested its conclusion directly on trained agents (point 3).

We answer the points on the noise, the replay and the coarse critics in 4 and 5 below. And you're right about the minimum over the two critics. Our analysis leaves it out, and the new experiments show that it matters a lot (point 3).

AuthorsOpenReview ↗

Paper updates and clarifications

3. The direction of the update and the sweep over σ~\tilde\sigma

"no experiment compares the critic-supplied update direction against an independent estimate of ∇J\nabla J, there are no DDPG σ~=0\tilde\sigma=0 agents, and no training-time σ~\tilde\sigma sweep"

Thank you, this was a great suggestion, and it gave the paper its most direct test. We removed the sentence and ran this experiment.

We trained TD3 at σ~∈{0,0.05,0.1,0.2,0.4}\tilde\sigma\in\{0,0.05,0.1,0.2,0.4\} and DDPG with three seeds on each task. On the six tasks other than Pendulum, where ∇θJ\nabla_\theta J is close to zero at the trained actors, we compared the update from each trained critic with (1−γ)∇θJ(1-\gamma)\nabla_\theta J estimated by central differences of the return. We call an update uphill or downhill when the 16th to 84th percentile interval of their cosine lies above or below zero (Appendix F).

uphilldownhillundecided
TD3, σ~>0\tilde\sigma>0311819
TD3, σ~=0\tilde\sigma=03105
TD3, σ~=0\tilde\sigma=0, first critic only1035
DDPG827

The target noise turns the update of TD3 from mostly downhill to mostly uphill, from 23% to 63% of the decided runs (one-sided Fisher test, p=0.01p=0.01), and this holds without InvertedDoublePendulum too, whose actors sit near a stationary point of JJ (p=0.03p=0.03).

DDPG has no target noise either, yet it mostly points uphill. Since it differs from TD3 at σ~=0\tilde\sigma=0 in the minimum over the two critics and in the delayed actor update, we also trained TD3 without noise and with targets that use only its first critic (third row). It behaves like DDPG (p=0.008p=0.008 against TD3 with the minimum), so without noise it's the minimum that turns the update downhill, as you suspected.

To give a sense of scale, the median cosine is +0.12+0.12 with target noise, while a random direction among the 67,000 to 95,000 parameters of an actor would give about 0.0040.004.

"the predicted σ~α−1\tilde\sigma^{\alpha-1} slope scaling"

Theorem 4 bounds the slope of Qπσ~Q^{\pi_{\tilde\sigma}} by a constant times σ~α−1\tilde\sigma^{\alpha-1}, so we measured that slope directly, from rollouts with the noise clipped as in TD3. On the paper's HalfCheetah agent (α^=0.004\hat\alpha=0.004), its spread over states grows as σ~−0.91\tilde\sigma^{-0.91}, with a 95% interval of [−1.35,−0.32][-1.35,-0.32] that contains the σ~−1.00\tilde\sigma^{-1.00} of the theorem.

Across three agents with a relu critic per task, the growth is no faster than this rate on all agents but one, and on every HalfCheetah and Ant agent it's compatible with σ~−1\tilde\sigma^{-1}. On InvertedDoublePendulum and Pendulum, where the state stays near a fixed point, the slope doesn't grow, which the theorem allows, since its lower bound needs the expanding attractor of Theorem 3.

The trained critics are a different story. Their slope changes little with σ~\tilde\sigma, and at σ~=0.05\tilde\sigma=0.05 it's 8 to 11 times smaller than that of Qπσ~Q^{\pi_{\tilde\sigma}}. This is within the bound of Proposition 5, but it means the critic doesn't follow the slope of the smoothed value, so we corrected the sentences that said it does. The agreement with ∇θJ\nabla_\theta J above is about the update averaged over states, which is the quantity of Theorem 5.

AuthorsOpenReview ↗

Paper updates and clarifications

4. The coarse critics

"How do you reconcile Theorem 5's condition ls≤LFσ~l_s \leq L_F \tilde\sigma with the 13 runs of Table 4? Does the 'unbiased up to σ~2\tilde\sigma^2' conclusion survive on those agents?"

Good catch. The condition was there to say when a trained critic resolves the smoothed value finely enough, but the proof doesn't use it, so we removed it. The conclusion does survive on those agents, because it's about Qπσ~Q^{\pi_{\tilde\sigma}} and doesn't involve the critic at all.

5. The actual configuration

"What does the analysis predict in the actual configuration?"

Good question. Theorem 5 is not affected, because it is about the smoothed value and doesn't depend on how the critic is trained.

What changes is our analysis of the critic. Propositions 5 and 6 assume that the critic is trained on fresh data, with actions perturbed by noise of the same width as the target noise. In TD3 the exploration noise (0.1) is narrower than the target noise (0.2), the data come from a replay of older actors, the critic is a neural network, and the targets take the minimum of two critics. The propositions don't cover any of this, so in the actual configuration our analysis makes no prediction for the critic. Appendix C.3 now says so, and this is why we measured the update of the trained critic directly (point 3).

AuthorsOpenReview ↗

Paper updates and clarifications

Thank you again for these suggestions, which really shaped the new experiments. They support the claim on the averaged update and correct the one on the slope of the critic. We believe these revisions address the concerns you raised, and we hope they make the paper an accept in your view. If anything remains open, please let us know, we'd be very happy to discuss!

Reviewer_cohFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The deterministic policy gradient theorem, on which DDPG and TD3 rest, assumes that the action value QμθQ^{\mu_\theta} is differentiable in the action. This paper argues that whether this assumption holds depends on a competition between the discount factor γ\gamma and the rate Λ\Lambda at which the closed loop produced by training separates nearby states. When γΛ<1\gamma\Lambda<1, the action-gradient exists (Theorem 1). When γΛ>1\gamma\Lambda>1, only a Hölder exponent α=log⁡(1/γ)/log⁡Λ<1\alpha=\log(1/\gamma)/\log\Lambda<1 is guaranteed (Theorem 2). Moreover, when the closed loop has a uniformly hyperbolic attractor and stretches fast enough on it, the action-gradient is absent at almost every state of its basin (Theorem 3). To explain why the method still works, the paper argues that the critic represents the action value smoothed by TD3's target noise σ~\tilde\sigma and by the resolution of the critic. The slope of this smoothed value, averaged over the states the actor visits, equals (1−γ)∇J+O(σ~2)(1-\gamma)\nabla J+O(\tilde\sigma^2) (Theorems 4–5).

The experiments use 42 agents trained with TD3 (7 tasks × 3 seeds × relu/softplus critic). Two copies of the simulator are started in the same state, and only the first action is displaced. From the distance between the two copies, the exponent is predicted without evaluating the action value. This prediction is compared with the exponent measured directly from the change in the discounted return. At γ=0.99\gamma=0.99, 33 of the 42 agents lie beyond the threshold, and the prediction and measurement differ by a median of 0.020. The paper also shows two further results. Reweighting the same rewards with a different γ\gamma crosses the threshold without retraining. Pendulum agents trained in the same setting fall on both sides of the threshold.

Critical strengths
  • The paper measures, without retraining, which side of the differentiability threshold a trained agent lies on The two-copy test reads the separation rate from the distances between trajectories alone, and its results agree well with the theory. At γ=0.5\gamma=0.5, the measured exponent is one on every task. The delay DD uses neither the reward nor γ\gamma. On the tasks whose copies part fastest, it still tracks the measured exponent as the discount varies (Table 1). Of the three Pendulum agents (relu critic) that differ only in their seed, only one lies beyond the threshold. Together, these results make a fairly convincing case that the trained closed loop decides the side. Bounds of this form are themselves known [5, 7, 8]. What is new is the empirical evidence that the differentiability assumption used in analyses of DPG-type methods can fail on agents trained in standard settings. This is the most valuable part of the paper.
Critical weaknesses
  • There is a gap between Theorems 4–5 and the trained agents Theorems 3, 4(b) and 5 treat the regime beyond the threshold, and all of them assume a uniformly hyperbolic attractor. Theorem 5 further assumes a spectral gap and linear response (Appendices B.4, C.2). The paper does not examine whether the agents in the experiments satisfy these assumptions. All actors are relu networks, and 4 of the 7 tasks involve contact, so the agents fall outside even Assumption 1. For Theorems 1–2, Section 5 confirms that the prediction matches the measurement even outside the assumptions, and the measured exponents are consistent with the conclusion of Theorem 3. However, the hypotheses of Theorems 3–5 are not checked, and there is no corresponding check for the conclusions of Theorems 4–5. In addition, g(θ)g(\theta) in Theorem 5 (Eq. 20) is written in terms of the exact Qπσ~Q^{\pi_{\tilde\sigma}}. The only quantitative approximation bound connecting this slope to that of a fitted critic is Proposition 6, which covers critics linear in their parameters. The authors themselves note at the end of Appendix C.3 that this bound is smaller than σ~2\tilde\sigma^2 only when ℓs≲σ~4−α\ell_s\lesssim\tilde\sigma^{4-\alpha}. This condition is considerably stronger than ℓs≤LFσ~\ell_s\le L_F\tilde\sigma in the statement of Theorem 5. The abstract's "which is why the method still works beyond the threshold" is therefore, at present, a claim about idealized systems.

  • No experiment directly tests Theorems 4–5 Section 5 tests the threshold and exponent of Theorems 1–2, and the curvature threshold. The smoothed slope in Figure 6 (Eq. 182) is the slope of QμθQ^{\mu_\theta} smoothed only in the first action, with the noiseless actor in control afterwards. This is a different quantity from Qπσ~Q^{\pi_{\tilde\sigma}}, in which noise enters at every step. The spread between states grows as the smoothing is refined, but this is what one would expect from the roughness already measured by α^\hat\alpha, and so adds little independent support for Theorem 4. The paper does not measure whether the actual update direction of the actor aligns with ∇J\nabla J, so the central claim of Theorem 5 rests on theory alone. The experiments therefore show that the premise fails, but not why the method still works.

Questions for authors
  • On the scope of Theorems 3–5 Is there any reason to believe that the closed loops of the trained agents satisfy the hypotheses of Theorems 3–5? If so, please state it. Section 4.2 attributes the differentiability of JJ to the existence of a density for dμθd^{\mu_\theta}, while Theorem 5 assumes a smooth initial density p0p_0. Moreover, the constant in O(σ~2)O(\tilde\sigma^2) depends on ∥p0∥Cp\|p_0\|_{C^p} (proof of Theorem 5). Please explain how this compares with the fractal objective of [8] cited in the introduction, and how large that constant is for the narrow resets of the MuJoCo tasks. If no such reason can be given, please narrow the claims in the abstract and the conclusion.

  • On a direct test of Theorem 5 Could you run the following measurement on some of the trained agents? For a fixed actor, fit critics at different target-noise widths σ~\tilde\sigma, and compare the update direction they supply (the average of ∇θμθ⊤∇aQϕ\nabla_\theta\mu_\theta^\top\nabla_aQ_\phi) with (1−γ)∇J(1-\gamma)\nabla J estimated by finite differences along directions in parameter space. Also, the condition ℓs≤LFσ~\ell_s\le L_F\tilde\sigma in Theorem 5 does not appear to be used in its proof (Appendix C.2). If that is correct, please either remove it or combine it with Proposition 6 and restate it as a claim about trained critics.

5 comments
AuthorsOpenReview ↗

Paper updates and clarifications

Thank you so much for the careful and constructive review! We are very glad that you found the two-copy test convincing and the results on trained agents valuable.

We are also grateful for the direct test of Theorem 5 that you proposed. It was extremely helpful, and together with your other comments it led us to two further measurements that, we believe, make the second half of the paper much stronger. In response to your comments, we

  1. measured the slope of Theorem 4 directly on all seven tasks, and found that it grows as the theorem allows, close to its largest rate on HalfCheetah and Ant,
  2. ran your test, and a larger version of it, on frozen trained actors and on new agents trained on all seven tasks, with TD3 at five target-noise widths, with DDPG and with TD3 without the minimum over its two critics. In TD3 the target noise turns most updates uphill on JJ,
  3. removed the condition ℓs≤LFσ~\ell_s\le L_F\tilde\sigma from Theorem 5, which, as you noted, its proof does not use.
  4. revised the abstract and the conclusion to keep what is proved apart from what is measured.
  5. now cite Blondé et al. (2022), Pfrommer et al. (2025), Young and Pugeault (2024) and Nachum et al. (2018).

The new measurements are in Appendix E, their protocol in Appendix F, and Section 5 summarises them. Below we answer each of your points in detail.

1. Known bounds

"Bounds of this form are themselves known [5, 7, 8]."

You are right that bounds of this kind are known, and we build on them. What Theorems 1 and 2 add is the form that the deterministic policy gradient needs. Below the threshold, Rachelson and Lagoudakis (2010) and Maran et al. (2023) show that the value is Lipschitz, which gives a gradient only at almost every action, possibly not at the action μθ(s)\mu_\theta(s) of the actor. Theorem 1 gives a continuous gradient everywhere, with thresholds that cannot be improved.

Beyond the threshold, Maran et al. and Pfrommer et al. (2025) reach only smaller exponents, and Wang et al. (2023) the right exponent only near one point. Theorem 2 proves it between any two actions, with one constant for all states and actions.

2. The assumptions of Theorems 3 to 5

"the agents fall outside even Assumption 1"

Thank you for raising this, it is an important point, and we agree. Theorems 3 to 5 assume a smooth closed loop with a hyperbolic attractor, the standard setting of the theory of chaotic systems (Ruelle, 1976; Gouëzel and Liverani, 2006).

Our trained agents do not satisfy these assumptions, because of their relu actors, contacts and resets, so the theorems do not apply to them directly. We therefore test whether their predictions hold empirically. Section 5 does this for Theorems 1 to 3, as you kindly note, and our new experiments do it for Theorems 4 and 5. Section 5 now also says which theorem each measurement tests, and that Theorems 3 to 5 hold under Assumptions 2 and 3.

3. Theorem 5 and the trained critic

"a claim about idealized systems"

We fully agree, thank you. We removed "which is why the method still works beyond the threshold", and the abstract now states Theorem 5 for the exact smoothed value Qπσ~Q^{\pi_{\tilde\sigma}}. For the trained critics, we now report direct measurements in our answer to your second question.

AuthorsOpenReview ↗

Paper updates and clarifications

4. Figure 6 and Theorem 4

"a different quantity from Qπσ~Q^{\pi_{\tilde\sigma}}"

Thank you for this careful observation, you are right. Figure 6 smooths QμθQ^{\mu_\theta} in the first coordinate of the first action only, and Section 5 now says so. As you say, it adds little beyond α^\hat\alpha. Your remark motivated a new measurement of the quantity of Theorem 4 itself. There the noise enters at every step, so nothing fixes the growth of the slope in advance, and Theorem 4 links it to α^\hat\alpha. We are happy to report that this link holds on the trained agents.

On the HalfCheetah agent with relu critic and seed 0 (α^=0.004\hat\alpha=0.004 at γ=0.99\gamma=0.99), at the 40 states of Figure 6 and with the noise clipped as in TD3, we obtain (estimator in Appendix F)

σ~\tilde\sigma0.050.10.20.4
spread of the slope over states27.717.69.54.1

The spread grows as σ~−0.91\tilde\sigma^{-0.91}, with a 95% interval of [−1.35,−0.32][-1.35,-0.32] over trajectories. This interval contains the rate σ~α−1=σ~−1.00\tilde\sigma^{\alpha-1}=\tilde\sigma^{-1.00} at which Theorem 4 bounds the slope, so the growth it describes is already present at the widths that TD3 uses.

We then repeated it on three agents with a relu critic per task (Appendix E). On all of them but one the growth is no faster than Theorem 4 allows, on every HalfCheetah and Ant agent it is compatible with σ~−1\tilde\sigma^{-1}, and on the two tasks whose state stays near a fixed point the slope does not grow.

We measure the last point you raise, whether the update of the actor follows ∇θJ\nabla_\theta J, in our answer to your second question.

AuthorsOpenReview ↗

Paper updates and clarifications

5. The scope of Theorems 3 to 5

"Is there any reason to believe that the closed loops of the trained agents satisfy the hypotheses of Theorems 3–5?"

Thank you for this question. We cannot verify these hypotheses on the trained agents, so the abstract and the conclusion now keep what is proved apart from what is measured. What we can show is that the predictions hold, for Theorem 3 through the measured exponents, as you note, and for Theorem 4 through the growth of its slope described above.

As for Section 4.2, you are right that the two statements need the same thing. The density of dμθd^{\mu_\theta} is the initial density p0p_0 carried forward by the closed loop, so it is smooth only if p0p_0 is. With a fixed initial state, as in Wang et al. (2023), p0p_0 is a point mass, and JJ is as rough as the value, which is the fractal objective they describe.

The constant is indeed large. It grows with the derivatives of p0p_0, and for the narrow resets of MuJoCo it is between 101010^{10} and 105110^{51} (Appendix C.2). On these tasks Theorem 5 explains the mechanism but does not give a useful bound, so we test its conclusion directly in our answer to your next question.

6. The direction of the update

"Could you run the following measurement on some of the trained agents?"

Yes, and thank you for proposing it! The update of each critic is Eq. 17 (Eq. 20 of the submission) with the critic in place of Qπσ~Q^{\pi_{\tilde\sigma}}. We compare it to (1−γ)∇θJ(1-\gamma)\nabla_\theta J estimated by central differences of the return. We report their cosine with a 16th to 84th percentile interval (Appendix F). An update points uphill when the interval lies above zero, downhill when it lies below, and is undecided otherwise.

We first ran your test. We froze 9 trained actors. The critic trained with each actor points uphill on 6 of them and downhill on none.

We also fit three critics to each actor, on its training replay, at every σ~∈{0,0.025,0.05,0.1,0.2,0.4}\tilde\sigma\in\{0,0.025,0.05,0.1,0.2,0.4\}. These refitted critics are less accurate, and point uphill and downhill about equally often. They are fit after training, mostly on transitions of earlier actors, and on HalfCheetah they are coarser than the critic trained with the actor, with a resolution of 0.15 to 0.44 against 0.07. With errors of this size, the sweep over σ~\tilde\sigma cannot resolve how the O(σ~2)O(\tilde\sigma^2) term of Theorem 5 grows.

We then trained new agents, TD3 at σ~∈{0,0.05,0.1,0.2,0.4}\tilde\sigma\in\{0,0.05,0.1,0.2,0.4\} and DDPG, with three seeds on each of the seven tasks, and measured the direction on the six tasks other than Pendulum, whose actors sit where ∇θJ\nabla_\theta J is close to zero. The table counts the runs of each kind.

uphilldownhillundecided
TD3, σ~>0\tilde\sigma>0311819
TD3, σ~=0\tilde\sigma=03105
TD3, σ~=0\tilde\sigma=0, first critic alone1035
DDPG827

The target noise turns the update of TD3 from mostly downhill to mostly uphill, from 23% to 63% of the decided runs (one-sided Fisher test, p=0.01p=0.01), and the effect remains without InvertedDoublePendulum, whose actors are close to a stationary point of JJ (30% to 69%, p=0.03p=0.03). The cosine varies between seeds and tasks (Appendix E).

DDPG, which has neither the target noise nor the minimum over two critics, points uphill on 8 of its 10 decided runs, consistent with the smoothing by its own network (Section 4.1). TD3 without noise differs from DDPG in the minimum, which our analysis leaves out, and in the delayed actor update. To tell the two apart, we also trained TD3 without noise and with targets that use its first critic alone (third row). It behaves like DDPG, uphill on 10 of 13 decided runs (p=0.008p=0.008 against TD3 with the minimum), so without noise it is the minimum that turns the update downhill.

7. The condition in Theorem 5

"does not appear to be used in its proof"

Good catch, you are right. The condition was there to say when a trained critic resolves the smoothed value finely enough, but the proof does not use it, so we removed it. Theorem 5 is about Qπσ~Q^{\pi_{\tilde\sigma}} and does not involve the critic at all.

We did not combine it with Proposition 6, because the two average over different distributions. Proposition 6 bounds the error of the critic on the data it is trained on, while Theorem 5 averages over the states the actor visits, and linking the two would need an assumption we cannot check on our agents. For the trained critics we report direct measurements above.

Reviewer_VdRpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I am putting my entire review here, as a sort of protest against the review form, which just makes it harder to be clear, in my honest opinion.

Paper 58333: Deterministic Actor–Critic Without an Action-Gradient

My summary is that this paper is another emission from the flailings of LLMs. It does not really have an overall idea that gives the paper a clear shape---and that makes it hard for me to give a meaningful summary of the paper. However, the paper does start with a clear idea: the observation that deterministic actor-critic RL methods assume in their theory that the true action-value function is differentiable, whereas in fact it is in many cases not. The simplest cases of this is when the dynamics is not differentiable, but this, the reader eventually realizes, is not the case that the paper is about. Rather, the paper is about the case in which the dynamics smoothly amplifies disturbances and as a result the derivative is infinite for sufficiently large values of the discount-rate parameter. Treating this case, the paper then goes on to analyze this effect and what various algorithms do in this case. All these observations might be considered small contributions, but they are poorly presented by the LLM prose. Small contributions like this might make for an acceptable paper if they were well presented, but here they are not well presented.

Below I describe and give examples of the paper's poor presentation.

Figure 1 is an example of a function that has no derivative because it is chaotic. This is problematic presentationally, because the paper is not about this case, and because there are simpler examples---just put a discontinuity in the dynamics, as commonly occurs in control problems where there are bounds, borders, or hard objects; there is no need, and it is off the point, to bring up fractal dynamics. Why is this done in a figure in the Introduction?

The style of the LLM writing is very imprecise, wordy, and repetitive. The part of this that I find most jarring is the loose mismatching of subject and verb. The paper says that "assumptions fail", that "closed loops act", that "rewards change", that "trajectories collect rewards", that "TD3 reads". Here are some example sentences: "The same closed loop can be below the threshold at one state and beyond it at another." and "A loop that carries a bounded region of states into itself draws the trajectories that start there onto the set the region shrinks to, the attractor of the loop." Remember that there is no clear idea of a loop here, and now it is below states and doing things. Or consider the sentence that begins the experiment section: "A trained agent carries its own separation rate, so the threshold of Section 3 and the exponent of Theorem 2 can be measured on it." This is poor presentation.

The paper's final sentence is a doozy: "Reading the threshold off two copies carries to a dynamics smooth away from the contacts, and to a neural critic, for which the resolutions measured here say what a bound should give." That is some classic gobbledygook there.

Critical strengths

I don't find this part of the review form helpful.

Critical weaknesses

see review above.

Questions for authors

see review above.

1 comment
AuthorsOpenReview ↗

Response to the review

Thank you for taking the time to review our paper. We'll be honest that the tone of the review surprised us, but we'd like to start with the technical points, since that is what matters most.

The overall idea. We do think the paper has one clear idea, and your summary actually states it well. DDPG and TD3 move the actor along the slope of the critic, and their theory assumes that the true action value has a slope too. We asked when it really does. A small change of the action changes the trajectory that follows, and in many systems the difference grows from step to step, while the discount makes later rewards count less and less. When the discount wins, the slope exists. When the growth wins, it can disappear, and this is what you describe as dynamics that "smoothly amplifies disturbances". We find that this happens on most trained agents.

The paper then asks what the actor follows in that case. TD3 trains its critic on targets smoothed by noise, so the critic is fitted to a smoothed action value, which does have a slope. We show that, under the assumptions of Theorem 5, the slope of this smoothed value, averaged over the states the actor visits, points along the gradient of the return up to an error that shrinks with the noise. The experiments we added for the other reviewers check this on trained agents.

The writing. Thank you for the comments on the writing, they were useful. We went through the paper to make it clearer, and we rewrote the sentences you quoted. The last sentence of the paper was meant to mark where our results stop and where they could be taken next. In the revision the conclusion sticks to what we prove and measure.

Figure 1. As your summary notes, the paper is not about dynamics that fail to be differentiable, but about dynamics that smoothly amplify disturbances, and Figure 1 shows that case. Nothing jumps, yet the effect of a small change of the action keeps growing along the trajectory, so the action value gets wrinkles at every scale, which is the gray curve.

A discontinuity in the dynamics would give a simpler example, already well known in control, but of the case that, as you say, the paper is not about. This is why the figure opens the paper, and we added a sentence to the caption to make it explicit.

On the comments about LLMs. Comments on style have always been part of peer review, long before language models existed, so an awkward sentence is no evidence that a language model wrote it. As ICLR asks, our paper states how we used language models, and the ideas, the theorems, the proofs and the experiments are our own.

Describing the whole paper as "another emission from the flailings of LLMs" is a very heavy accusation. It comes with no evidence, and it rests on the style of some sentences, not on an error in a theorem, a proof or an experiment. The other two reviewers rate the presentation higher and don't raise these concerns. We really did find your comments on style helpful, and we thank you for them, but a review should also assess the science.

On the review itself. We have to say that this reads more like a polemic than a review. It opens as a protest against the ICLR form, calls the strengths section unhelpful, sends the weaknesses and questions back to "see review above", and reaches its verdict in the first lines. Words like "gobbledygook" don't show much respect for the work of other researchers.

How language models are changing research is an important topic, and one our community should discuss openly. But a review of a single paper is not the place for that debate, and not the place to dismiss someone's work without evidence and in this tone. That said, we did our best to use your comments to improve the paper. We ran a set of new experiments, described in our answers to the other reviewers, and if you have any technical concern about a theorem, a proof or an experiment, we'd be glad to answer it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of paper updates

Thank you to all the reviewers for your comments! Your suggestions led us to new experiments, which turned out to be very informative and made the paper even stronger.

In summary, we have

  1. measured the slope of the smoothed action value of Theorem 4 on all seven tasks, and found that it grows as the theorem allows,
  2. compared the update of the trained critic with the gradient of the return, on frozen actors and on new agents trained with TD3 at five target-noise widths and with DDPG, and found that in TD3 the target noise turns most updates uphill,
  3. added an ablation without the minimum over the two critics, which shows why TD3 without target noise behaves differently from DDPG,
  4. removed a condition from Theorem 5 that its proof does not use,
  5. made the scope of each result explicit in the abstract, Section 5 and Appendix C.3,
  6. revised the writing throughout, including every sentence quoted in the reviews.

The new experiments are in Appendix E and their protocol in Appendix F.

Thank you again for the time and effort you put into reviewing our paper. Please let us know if you have any additional concerns, we are happy to discuss!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.