A Recipe for Drone Racing with Reinforcement Learning

This article serves as the conclusion to a three-part series focused on quadcopter simulation. Having previously explored the derivation and simulation of motion equations in both 2D and 3D, we now shift our focus to Reinforcement Learning (RL). The goal is to train a policy that can first maintain a stable hover and eventually navigate a series of racing gates.

Unlike the previous tutorial-style entries, this post is structured as a recipe.

Note: This is a collection of empirical tricks and techniques. There are no rigorous proofs here—only methods that have been proven to work through trial and error.

Because this is a recipe, I will only highlight the critical code segments. For the complete implementation, visit the GitHub repository: mrandri19/quadcopter-racing.

---

🗺️ The Incremental Strategy

Attempting to build the simulation, the RL environment, the reward function, and the model simultaneously is a recipe for failure. When the policy underperforms, it becomes nearly impossible to debug whether the issue lies in the physics, the reward, or the hyperparameters.

To mitigate this, I utilized a three-stage pipeline:

1. The "Hello World" of RL: Solving the inverted pendulum (cartpole) problem using PPO and vectorized environments.
2. Custom Quadcopter Env: Implementing a MuJoCo-based environment, defining observation/action spaces, and creating a hovering reward.
3. Racing Expansion: Transitioning from hovering to gate-racing with updated observations and randomized initialization.

---

🛠️ Stage 1: Solving the Inverted Pendulum

To ensure the RL pipeline is functional, we start with a basic vectorized problem using stable-baselines3. We aim for the policy to reach the maximum episode length in the InvertedPendulum-v5 environment.

from stable_baselines3 import PPO 
from stable_baselines3.common.env_util import make_vec_env 
from stable_baselines3.common.policies import ActorCriticPolicy 

def main() -> None: 
    # Vectorizing 128 environments for efficiency
    env = make_vec_env("InvertedPendulum-v5", n_envs=128) 
    
    model = PPO(
        policy=ActorCriticPolicy, 
        env=env, 
        learning_rate=3e-3, # Scaled: 3e-4 * sqrt(128) ≈ 3e-3
        n_steps=512,        # Reduced from 2048 due to high n_envs
        batch_size=1024,    # Increased from 64 for better stability
        n_epochs=5,         # Reduced from 10 to accelerate training
        verbose=1, 
    ) 
    model.learn(total_timesteps=750_000)

Hyperparameter Logic

Results: Since each step provides a $+1$ reward, the reward equals the episode length. The policy typically hits the 1,000-step ceiling by the 600k timestep mark.

• Validate PPO configuration.
• Confirm hyperparameter stability.
• Separate algorithm bugs from environment bugs.

---

🚁 Stage 2: Learning to Hover

With the RL pipeline validated, we move to the custom quadcopter environment.

Action Design: CTBR + P-Controller

The policy does not output raw motor speeds. Instead, it utilizes CTBR (Collective Thrust and Body Rates). The policy outputs:

• Mass-normalized thrust
• Desired roll, pitch, and yaw rates

This architecture is common in high-level research (e.g., Deep Drone Acrobats). A low-level Proportional (P) Controller then translates these into motor commands using the current angular velocity ($\omega_{body}$) as feedback.

def _controller(self, omega_body: NDArray, denormalized_actions: NDArray) -> NDArray: 
    f_des = denormalized_actions[:, 0] 
    w_des = denormalized_actions[:, 1 : 4] 
    w = omega_body 
    Jw = self._max_thrust / 4 
    
    # P-controller logic
    u_w = w_des - w 
    u_p, u_q, u_r = u_w[:, 0], u_w[:, 1], u_w[:, 2] 
    
    motor = np.zeros((len(f_des), 4), dtype=np.float32) 
    motor[:, 0] = f_des - u_p - u_q - u_r 
    motor[:, 1] = f_des + u_p - u_q + u_r 
    motor[:, 2] = f_des - u_p + u_q + u_r 
    motor[:, 3] = f_des + u_p + u_q - u_r 
    
    return np.clip(motor, 0, 1)

Why CTBR?

1. Simplicity: The RL policy focuses on where to go and how to rotate, while the P-controller handles the high-frequency attitude dynamics.
2. Deployability: Real-world flight stacks like Betaflight natively accept CTBR-style inputs.

To understand the physics, we look at Euler's rotation equation for body-frame angular velocity $\omega$:

$J \dot{\omega} = \mathbf{\tau} - \omega \times (J\omega)$

Figure 1: Conceptual flow from RL Policy $\rightarrow$ CTBR $\rightarrow$ P-Controller $\rightarrow$ Motors.