Find Your Way, but... Social Navigation via Constrained Reinforcement Learning

This document explains how to integrate constrained reinforcement learning (CRL) extensions into RL Games. The CRL layer plugs into the existing A2C/PPO pipeline (rollouts, dataset prep, logging) and adds cost‑aware critics and losses. For a general overview of RL Games, refer to the upstream library; this README focuses on the CRL-specific additions and how to use them.

These changes were developed for SocialNav, a social navigation environment. To train and evaluate policies in SocialNav, apply the accompanying CRL changes in that project’s repository (see its documentation). The RL Games modifications described here are environment-agnostic and can be reused with other tasks that expose a cost signal.

Below, we introduce RL Games, describe the modifications, and provide a guide for integrating them into the upstream library.

RL Games

RL Games is a high‑performance RL framework with fast PPO/A2C implementations, strong vectorized environment support, and end‑to‑end GPU pipelines. We chose it for the high performance in terms of parallelisation but also in terms of the results it achieved in the tasks we were interested in (compared to sb3 and omnisafe).

Learn more:

• Overview and quickstart: README.md
• Extending with custom envs/algos: HOW_TO_RL_GAMES.md

Modifications

At a high level, we extend PPO (A2CAgent) to form PPO-Lag. We modify the PPO objective by subtracting a Lagrangian penalty term that accounts for cost. The goal of PPO-Lag is to maximise reward while satisfying a cost constraint over expected cost per episode.

PPO objective: $$ \max_\theta \mathbb{E}_t!\left[\min!\left(r_t(\theta)\hat A_t,\ \operatorname{clip}!\left(r_t(\theta),1-\epsilon,1+\epsilon\right)\hat A_t\right)\right] $$

PPO-Lag objective: $$ \max_\theta \mathbb{E}_t!\left[\min!\left(r_t(\theta)\hat A_t^r,\ \operatorname{clip}!\left(r_t(\theta),1-\epsilon,1+\epsilon\right)\hat A_t^r\right) -\lambda,\min!\left(r_t(\theta)\hat A_t^c,\ \operatorname{clip}!\left(r_t(\theta),1-\epsilon,1+\epsilon\right)\hat A_t^c\right)\right] $$

The high-level adaptations required:

• Provide a cost limit $d$ to the env for $\lambda$ updates.
• Log per-step costs $c_t$ alongside rewards during rollouts.
• Add a cost critic $V_c$ and compute cost advantages $\hat{A}_t^c$ via GAE (with cost discount).
• Modify the actor loss by subtracting $\lambda \hat{A}_t^c$ using the same PPO clipping as the reward term.
• Update $\lambda$ by projected gradient ascent: $\lambda \leftarrow \max (0, \lambda+\eta(\hat{c}-d))$.
• Expose CRL hyperparameters.

Files added

• rl_games/safe_rl/safe_experience.py - SafeExperienceBuffer class to handle cost tensors
• rl_games/safe_rl/safe_models.py - SafeModelA2CContinuousLogStd model with cost value outputs
• rl_games/safe_rl/safe_socialnav_humanpose.py - SafeSocialNavHumanPose network with cost value outputs
• rl_games/safe_rl/safe_a2c_continuous.py - SafeA2CAgent class with cost-related modifications
• rl_games/safe_rl/safe_player.py - SafePpoPlayerContinuous class with cost handling

Files modified

• rl_games/torch_runner.py - register SafeA2CAgent in the algo factory and SafePpoPlayerContinuous in the player factory
• rl_games/algos_torch/model_builder.py - register SafeModelA2CContinuousLogStd in the model factory
• envs/__init__.py - register SafeSocialNavHumanPose network

Documentation of Modifications

For a detailed understanding of the modifications, review the files individually. Each file begins with a brief description of its changes. The added and modified components are annotated with docstrings. Additionally, key modifications are annotated inline with comments using the prefix CRL:, for example: # CRL: add cost_value to return tuple.

Integration

To integrate the CRL layer into upstream RL Games:

1. Add safe_rl folder

Copy the safe_rlfolder, which contains this README, into the base rl_games directory. This directory also contains the subdirectories algos_torch, common, configs, envs, interfaces and networks.

2. Register builders in factories

filepath: rl_games/torch_runner.py

# ...existing code...
from rl_games.safe_rl.safe_a2c_continuous import SafeA2CAgent
from rl_games.safe_rl import safe_player

class Runner:
    def __init__(self, algo_observer=None):
        # ...existing code...
        self.algo_factory.register_builder('safe_a2c_continuous', lambda **kwargs: SafeA2CAgent(**kwargs))
        self.player_factory.register_builder('safe_a2c_continuous', lambda **kwargs: safe_player.SafePpoPlayerContinuous(**kwargs))
        # ...existing code...

filepath: rl_games/algos_torch/model_builder.py

# ...existing code...
from rl_games.safe_rl import safe_models

class ModelBuilder:
    def __init__(self):
        # ...existing code...
        self.model_factory.register_builder(
            'safe_continuous_a2c_logstd',
            lambda network, **kwargs: safe_models.SafeModelA2CContinuousLogStd(network)
        )
        # ...existing code...

filepath: rl_games/envs/__init__.py

# ...existing code...
from rl_games.safe_rl.safe_socialnav_humanpose import SafeSocialNavHumanPoseBuilder

def build_env(...):
    # ...existing code...

# Register the safe network
model_builder.register_network('SafeSocialNavHumanPose', SafeSocialNavHumanPoseBuilder)
# ...existing code...

3. Ensure environment API supports costs

   • Env.step(actions) must return: obs, rewards, costs, dones, infos
   • costs shape: (num_costs,) per env (float32). For vectorized envs: (num_envs, num_costs).
   • Emit cost signals for your constraints (e.g., collisions).

4. Configure training (YAML/JSON)

Point the runner to the safe agent, model, and (optionally) the safe network. Add CRL hyperparameters.

# example
params:
  algo:
    name: safe_a2c_continuous
  model:
    name: safe_continuous_a2c_logstd
  network:
    name: SafeSocialNavHumanPose # or your own network that outputs cost_value
  config:
    # standard PPO/A2C settings...
    crl_config:
      num_costs: 1
      cost_gamma: 0.99
      cost_limit: 0.1
      lagrangian_init: 0.0
      lagrangian_upper_bound: 100.0
      lambda_lr: 1e-3
      lambda_optimizer_type: adam  # or sgd
      normalize_cost_value: true
      normalize_advantage: true

Support

The isolated modifications folder is intended to facilitate compatibility with future versions of RL Games without the need to maintain a fork, although such compatibility is not guaranteed. Development initially targeted version 1.6.1, but was later updated to the latest release, 1.6.5, for which it is now optimised.

Citing

If you use this extension to RL Games in your research project please use the following citation:

@misc{navbut,
title = {Find Your Way, but... Social Navigation via Constrained Reinforcement Learning},
author = {Streuli, David},
month = {September},
year = {2025},
}