Safe and efficient urban unmanned aerial vehicle navigation optimization via deep reinforcement learning

Safe and efficient urban unmanned aerial vehicle navigation optimization via deep reinforcement learning

Junle Liu, Yunsen Xing, Yanyu Ke, Guandi Wang, K.T. Tse

1. Overview

We train a continuous control Proximal Policy Optimization (PPO) agent to navigate a UAV through a complex urban wind field that two tandem buildings generate. We keep the agent fully reactive: a single launch phase auxiliary waypoint clears the obstacle cluster near the start, and the agent then reaches the target using only local velocity observations, nine onboard range sensors, and a physics embedded dense reward. The agent needs no global map and no precomputed global path.

This code reproduces the following key features:

• The policy uses a streamlined feedforward PPO network : an MLP of widths (256, 128, 64) with roughly 66k parameters.
• The environment defines a 21 dimensional observation and a 2 dimensional continuous action.
• The environment integrates flow aware RK4 dynamics and enforces a two layer (soft and hard) obstacle safety margin.
• We define four navigation scenarios on the same flow field and train each one with three random seeds.
• The policy achieves a 100% deterministic success rate across all scenarios and produces smooth control commands that suit sim to real transfer.
• An ablation shows that the single launch phase waypoint bootstraps exploration past the initial obstacle cluster.

2. Navigation scenarios

All four scenarios share the same flow field, geometry, reward, and dynamics, and only the start and target placement varies, which changes the geometric difficulty. We report coordinates in metres.

• Scenario 1 (headwind): the start sits at (−20, 0) and the target at (185, 0); the UAV passes both buildings.

• Scenario 2 (headwind): the start sits at (−20, 0) and the target at (155, 0); the UAV passes the first building.

• Scenario 3 (tailwind): the start sits at (185, 0) and the target at (15, 0); the UAV passes the second building.

• Scenario 4 (tailwind): the start sits at (185, 0) and the target at (−15, 0); the UAV passes both buildings.

Every scenario places a single auxiliary waypoint near the launch obstacle. Once the UAV reaches the arrival radius, the environment switches the active target to the final target, and the UAV then continues reactively.

3. Method summary

• Observation: the environment concatenates the sine and cosine of the heading, the unit vector to the active waypoint, the normalized distance to the waypoint, the UAV velocity and speed, the local flow velocity, the nine range sensor readings, and the previous action. Together these values form a 21 dimensional observation.

• Action: the policy outputs two continuous commands, the desired heading and the desired forward speed. It squashes each raw sample through a tanh function and then rescales it to the allowed range. The heading covers the full turning circle, and the speed runs from 0 up to a maximum of 4 metres per second.

• Dynamics: a fourth order Runge Kutta scheme advances the UAV motion. The model couples a first order velocity tracker with a second order heading tracker and adds the local flow velocity to the motion. After each integration substep, the environment caps the UAV speed at the maximum of 4 metres per second.

• Reward: the environment combines a dense progress term with a soft obstacle proximity penalty, a step and low speed cost, and an action smoothness penalty. It adds a large bonus when the UAV reaches the waypoint and another bonus when the UAV reaches the final target, and it applies a heavy penalty when the UAV collides or leaves the domain.

• Safety margins: the environment keeps two safety zones around every obstacle. A hard collision boundary at 0.5 metres ends the episode, and a wider soft buffer at 3.0 metres triggers the proximity penalty before the UAV reaches that boundary.

4. Citation

If you find this work helpful, please consider citing our paper.

@article{liu_uav_drl,
  title   = {Safe and efficient urban unmanned aerial vehicle navigation optimization via deep reinforcement learning},
  author  = {Liu, Junle and Xing, Yunsen and Ke, Yanyu and Wang, Guandi and Tse, K. T.},
  journal = {Preprint submitted to Elsevier},
  year    = {2026}
}