acceptodds
Under review as a conference paper at ICLR 2027

Differential-Policy Double Deep Q-Network with Decoupled Architecture for Attitude Tracking of Quadrotor UAVs

Abstract

This paper proposes a Differential-Policy Double Deep Q-Network (DP-DDQN) algorithm to overcome the challenge of applying value-based reinforcement learning to continuous control tasks. The algorithm introduces a differential policy to control variations in the actuator output, whereby discrete decision actions are accumulated into bounded continuous-valued execution signals and the previous actuator command is included in the observation. Under bounded rewards and exact value representation, the ideal channel-wise clipped differential Bellman operator is proved to be contractive based on the contraction mapping theorem. Furthermore, to avoid the curse of dimensionality caused by generating multidimensional actions with a single network, a decoupled network architecture is designed to implement the multi-axis control policy. The DP-DDQN is deployed on a quadrotor Unmanned Aerial Vehicle (UAV) system to execute attitude tracking. Numerical simulations and hardware-in-the-loop (HITL) experiments validate the effectiveness and real-time feasibility of the algorithm. DP-DDQN succeeds in all 200 regulation runs and all 40 tilted-gate runs, and reduces the random-attitude recovery root mean square error from to relative to the PX4 proportional–integral–derivative controller without missing the 10-ms control deadline.

Primary area:
reinforcement learning
Keywords:
Decoupled architecture, differential policy, discrete incremental control, reinforcement learning, value-based control.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_J9SXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a differential-policy double deep Q-network. In particular, it focuses on the quadrotor attitude tracking, which falls within the community of robotics and automation. Thus, its topic is not suitable for ICLR. In addition, the main contribution is simply adding an inner loop for the action space. The actual action is the torque, while the agent output the torque difference. The novelty is limited. Thus, the reviewer suggests a rejection.

Critical strengths

This paper claims that, the hardware-in-the-loop experiments are conducted.

Critical weaknesses
  1. This paper actually focuses on a robotics topic, which does not fall in the interest of ICLR.
  2. The so-called “differential-policy” is a simple inner-loop action design, which lacks the novelty.
  3. this paper adopts discrete torque levels to manipulate the quadrotor, which is different from the common-used continuous RL baselines, such as PPO, DDPG, or SAC.
  4. This paper only adopts PID, MPC, PPO, and TD3 for comparison. No SOTA algorithms for attitude tracking are considered.
Questions for authors

The Euler angle of the quadrotor is strongly coupled state space. If three independent networks are adopted for generating torque along each axis, how can the stability be guaranteed?

Reviewer_e4eLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a RL controller for quadrotor attitude tracking. The controller selects discrete torque increments and accumulates them into torque commands. Three separate Q-networks control roll, pitch, and yaw. Each network receives the attitude errors, angular velocities, and previous torque commands, allowing it to observe motion across all three axes. The authors evaluate the controller through simulated attitude regulation, tracking through tilted gates, and hardware-in-the-loop recovery from different initial attitudes. The reported results show successful regulation and gate crossing in all tested cases, faster recovery than PX4 PID, and inference times within the 10-ms control period on a flight-control processor.

Critical strengths
  1. The controller is simple enough to run on a flight-control processor. The combination of discrete torque increments and small axis-specific networks is practical. Including the previous command in the observation also makes sense because the applied torque depends on accumulated actions.

  2. The reported control results are quite good. The method succeeds in all 200 regulation trials and all 40 gate-crossing trials. In the hardware-in-the-loop experiments, it also achieves lower average tracking error and faster recovery than PX4.

Critical weaknesses
  • The comparison with existing RL flight controllers is incomplete. My main concern is that the paper compares against its own torque-output implementations of PPO and TD3, without establishing how competitive these implementations are. Prior work has already demonstrated strong quadrotor control using RL policies that output collective thrust and body rates. [1] specifically studied the choice of action space and demonstrated a strong PPO-based controller in real flight. These works are directly relevant to assessing the practical value of the proposed design. I would like to see a comparison with a well-tuned CTBR-based RL controller adapted to the same tracking tasks. Since CTBR policies retain a body-rate controller, the comparison should account for the complete control stack and clearly report differences in observations, control frequencies, and actuator limits. The current results, particularly TD3 failing to reach the gate region in every setting, leave me uncertain whether the reported advantage reflects the proposed method or weaknesses in the baseline training. The novelty also needs to be separated from the effects of the action representation and reward design. Comparisons with incremental-action PPO or TD3, an existing decoupled Q-learning method, and the PD-like reference controller used in the reward would help establish where the gains come from.

  • Real-flight validation is important given the limited methodological novelty. The method combines several established ideas, but do not establish stability or robustness of the deployed neural controller. Much of the paper's contribution therefore rests on its practical control performance. The hardware-in-the-loop experiments demonstrate that the implementation can execute on a flight-control processor, but the vehicle dynamics remain simulated. They leave open whether the controller works with actual motor dynamics, imperfect thrust allocation, estimation errors, vibration, and battery-dependent changes in actuator response. Given this balance of contributions, I would need real-flight evidence to be convinced of the practical significance. Repeated attitude-tracking or trajectory-following experiments against a tuned classical controller and a strong RL baseline would be particularly valuable. Without such evidence, the results support embedded execution and performance in the tested simulator, while the benefit on a physical quadrotor remains unclear.

[1] A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight

Questions for authors
  • Additional comparison: Could you evaluate a well-tuned RL controller using collective thrust and body-rate commands on the same tracking tasks? Does the advantage of DP-DDQN remain against this type of controller?
  • Real-flight evidence: Are any physical flight results available? Given the limited methodological novelty, repeated real-flight tracking experiments would materially affect my assessment. Can the reported improvements over PID and RL baselines be reproduced when the controller drives actual motors and receives estimated vehicle states?
  • Baseline quality: Why does TD3 fail to reach the gate region in every tested setting, and why does PPO deteriorate so strongly as gate tilt increases? Could you provide tracking performance during training, results across independent seeds, and evidence that these policies were sufficiently trained and tuned? Episode survival alone does not establish that a useful tracking policy was learned.
  • Training resources: As described, DP-DDQN appears to use approximately 21 million one-step model evaluations per policy, compared with around 500,000 transitions for PPO and TD3. Is this interpretation correct? How do the methods compare when reporting both simulator calls and training time, and when allowing the baselines sufficient training to reach stable performance?
  • Action representation: How much of the improvement comes from predicting torque increments? Could PPO or TD3 be trained with the same incremental action representation and previous-command observation? An absolute-action DDQN comparison would also help distinguish the effect of the action representation from the learning algorithm.
Reviewer_ZKJTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the use of value-based reinforcement learning for quadrotor attitude control. The authors propose DP-DDQN, which replaces direct torque selection with discrete torque increments. These increments are accumulated and clipped to produce actuator commands, while the previous command is included in the network observation. The controller uses three separate Q-networks for roll, pitch, and yaw, with each network observing the full rotational state.

The theoretical part establishes contraction of the ideal channel-wise clipped Bellman operators and bounded changes in the sampled control commands. The experimental part includes attitude regulation, tilted-gate tracking, and hardware-in-the-loop (HITL) recovery tests. The method achieves 200/200 successful regulation runs and 40/40 successful gate crossings. In HITL experiments, it improves average tracking RMSE and response time compared with PX4 PID, while completing network inference within the 10-ms control period.

The paper is primarily an effort to make discrete-action Q-learning practical for a continuous, real-time control problem.

Critical strengths

1. The incremental action design is a sensible choice for this particular control problem.

I find the motivation for using torque increments convincing. For attitude control, selecting among a small number of absolute torque values can be restrictive, while using a very fine discrete action grid quickly becomes expensive in multiple dimensions. The proposed controller avoids directly enumerating all joint actions and allows commands to evolve through accumulated increments.

The resulting architecture is relatively lightweight: three seven-action networks instead of a joint action-value output containing 343 entries. Although this reduction does not by itself prove better computational efficiency than every alternative architecture, it is a reasonable engineering design for a controller operating at 100 Hz.

2. The evaluation includes a useful level of implementation detail and goes beyond an isolated tracking simulation.

I appreciate that the authors report more than just attitude-error curves. The gate-crossing experiment considers the rotor envelope and clearance constraints, and the HITL test places the controller on an STM32F765 flight-control processor rather than evaluating network inference only on a desktop computer.

The reported results are encouraging, especially the successful crossings at 70° gate tilt and the improved recovery response time. The maximum reported inference time of 7.543 ms is also relevant evidence for the claimed real-time implementation. These experiments make the paper more interesting than a purely algorithmic demonstration on a simplified control benchmark.

Critical weaknesses

1. It is difficult to separate the contribution of the learned policy from the control knowledge already built into the reward.

My main concern comes from the reward design in Appendix C.1. Equation (22) includes a penalty for deviating from a reference torque uk,jdu^d_{k,j}, and this reference is computed from the attitude error and angular velocity using natural-frequency and damping-ratio parameters. In other words, the learning objective already contains a fairly explicit PD-like control target. The reference configuration also assigns a non-negligible weight to this term (wt=2w_t=2).

I do not consider using control knowledge in the reward to be a problem in itself. In fact, this may be one reason why the method works well. However, it changes how I interpret the experimental results. Is DP-DDQN learning a substantially better control law, or is it mainly learning to approximate and adjust a reasonably good analytical controller?

The paper compares against PID and MPC, but it does not isolate the torque-reference penalty or evaluate the reference controller itself under the same conditions. Without such a comparison, I find it difficult to attribute the performance gain specifically to the proposed differential Q-learning formulation.

This also affects the novelty assessment. Incremental actions and dimension-wise Q-learning are already established ideas. The paper would be considerably more convincing if it showed which part of their combination produces the reported improvement.

2. The theoretical analysis and the actual training/evaluation setup leave two important gaps.

The channel-wise contraction result appears mathematically reasonable under the stated assumptions, but I am not convinced that it explains the behavior of the deployed controller. During training, a candidate action for one axis is evaluated while the other torque commands remain fixed. During execution, however, all three networks select and apply their increments simultaneously. The resulting joint transition is not the same transition used to construct the individual channel-wise Bellman operators.

Thus, the contraction result applies to the idealized channel-wise operators, not necessarily to the value functions learned by the neural networks or to the stability of the joint closed-loop policy. The distinction matters because the method is motivated partly by handling coupled rotational dynamics.

I also have concerns about how the learning-based comparisons should be interpreted. Appendix C.2 specifies 20 Bellman iterations, 50,000 sampled states per iteration for each axis, and evaluation of seven candidate actions per state. This suggests approximately 21 million single-step model-action evaluations across the three axes. By contrast, Appendix D reports approximately 500,000 training transitions for PPO and TD3. These are not directly equivalent computational units, but the difference is large enough that training cost and access to the dynamics model should be discussed explicitly.

As presented, the comparisons establish that the proposed training-and-control pipeline works well in the tested setting. They do not yet establish that it is a more efficient or generally preferable reinforcement learning method.

Questions for authors

Q1. How much of the improvement comes from the torque-reference term in the reward?

In Eq. (22), the reward explicitly encourages the selected torque to stay close to uk,jdu^d_{k,j}, which is computed from a PD-like feedback law. Could the authors report performance after removing this term (wt=0w_t=0)? I would also be interested in the performance of directly applying uk,jdu^d_{k,j} as the controller. This would help distinguish the benefit of reinforcement learning from that of the embedded control prior.

Q2. What is the actual training cost of DP-DDQN relative to PPO and TD3?

According to Appendix C.2, each axis uses 20 iterations and 50,000 sampled states per iteration, with seven candidate actions evaluated at each state. If I understand the algorithm correctly, this amounts to approximately 21 million one-step model-action evaluations across the three axes. Is this interpretation correct? How does the total number of model evaluations and the training wall-clock time compare with the PPO and TD3 baselines? I would find a comparison under more closely matched computational budgets informative.

Q3. How important is the mismatch between channel-wise training and joint execution?

In Section 4.1, the candidate transition for one channel holds the other stored torques fixed, whereas deployment changes all three torque commands simultaneously. Could the authors quantify the effect of this approximation? For example, does performance deteriorate when the rotational coupling becomes stronger? It would also be helpful to clarify how the nine-dimensional observation satisfies the augmented Markov-state assumption for time-varying gate-tracking references, given the discussion in Section 2.1.

Q4. Why does DP-DDQN exhibit substantially larger reverse overshoot in the HITL tests?

Table 2 reports a mean reverse overshoot of 2.176° for DP-DDQN, compared with 0.384° for PX4 PID, despite improvements in RMSE and response time. Is this mainly caused by the reward design, the use of relatively large torque increments, or the emphasis on rapid recovery? Can this trade-off be adjusted without losing the reported advantages? I consider this relevant because a faster response is not automatically preferable if it comes with significantly greater reversal in the attitude error.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.