Detection VLM in ROS

This repository provides ROS wrappers for vision-language perception modules:

• Open-vocabulary 2D object detection with 3D LiDAR grounding.
• Semantic mapping, which accumulates detections into a persistent colored voxel map.
• Binary visual question-answering with confidence visualization.

The Humble branch is documented here. For ROS Noetic, see the noetic branch.

---

Table of Contents

• Setup
  • General Requirements
  • Building
  • Python Virtual Environment
• Repository Layout
• Configuration
• Usage
  • Detection VLM
  • Semantic Map
  • Q&A VLM
• License
• Contact

---

Setup

General Requirements

These instructions assume that ros-humble-desktop is installed on Ubuntu 22.04.

Building

Build the repository:

mkdir -p vlm_ws/src
cd vlm_ws/src

git clone git@github.com:ntnu-arl/detection_vlm.git detection_vlm
rosdep install --from-paths . --ignore-src -r -y

cd ..
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release
source install/setup.bash

Python Virtual Environment

It is highly recommended to set up a Python virtual environment for the ROS Python nodes:

cd vlm_ws/src/detection_vlm/detection_vlm_python
python3.10 -m venv --system-site-packages detection_vlm_env
source detection_vlm_env/bin/activate
pip install -U pip
pip install -r requirements.txt

The launch files expose a python_env argument. Point it to your virtualenv Python executable if it differs from the launch default:

ros2 launch detection_vlm_ros detection_vlm.launch.yaml \
  python_env:=/path/to/detection_vlm_env/bin/python3

For OpenAI-backed VLM modes, export an API key before launching:

export OPENAI_API_KEY=<Your OpenAI API key>

YOLOE-backed modes do not require an OpenAI key. The first YOLOE run may download model weights through Ultralytics if they are not already available locally.

---

Repository Layout

• detection_vlm_python: model wrappers, OpenAI client, shared config helpers, and common output dataclasses.
• detection_vlm_ros: ROS 2 nodes, launch files, prompts, and YAML examples.
• detection_vlm_msgs: ROS service definitions, currently SetPrompt.srv.

---

Configuration

The launch files load YAML configs from detection_vlm_ros/config and then pass runtime overrides through the config string parameter.

Provided examples:

• detection_yoloe.yaml: YOLOE detection with segmentation masks.
• detection_vlm.yaml: OpenAI detection VLM.
• semantic_mapping_yoloe.yaml: YOLOE-based semantic mapping.
• reasoning_vlm.yaml: OpenAI binary reasoning VLM.

Common parameters:

• vlm.type: yoloe or openai.
• prompt: detection or reasoning query. This can also be changed at runtime with the set_prompt service.
• worker.min_separation_s: minimum time between processed images.
• compressed_image: subscribe to sensor_msgs/msg/CompressedImage when true, otherwise sensor_msgs/msg/Image.
• target_frame, camera_frame, body_frame: frame names used for TF lookup.
• lidar_to_body_transform: static LiDAR-to-body transform kept in the YAML config for this ROS 2 branch.
• camera_intrinsics and distortion_coeffs: optional calibration values. If a CameraInfo message is received, it is used instead.
• classes_file: optional class list for YOLOE models that support setting custom classes. Prompt-free YOLOE models with pf in the model name ignore it.

---

Usage

Detection VLM

The detection node runs 2D object detection on the input image and projects LiDAR points into the camera frame to recover corresponding 3D object volumes. It can use either YOLOE or an OpenAI VLM, depending on the selected config.

Launch:

ros2 launch detection_vlm_ros detection_vlm.launch.yaml

Useful launch arguments:

ros2 launch detection_vlm_ros detection_vlm.launch.yaml \
  config_path:=$(ros2 pkg prefix detection_vlm_ros)/share/detection_vlm_ros/config/detection_yoloe.yaml \
  input_image_topic:=/cam_front/image_raw/compressed \
  input_pointcloud_topic:=/mimosa_node/lidar/manager/points_full_res \
  input_camera_info_topic:=/camera/color/camera_info \
  target_frame:=mimosa_world \
  camera_frame:=camera0 \
  body_frame:=mimosa_body

Inputs:

• /detection_vlm/input_image
• /detection_vlm/input_pointcloud
• /detection_vlm/input_camera_info
• TF from body_frame to target_frame, plus the configured lidar_to_body_transform.
• TF from target_frame to camera_frame.

Outputs:

• /detection_vlm/detections_image: annotated detection image.
• /detection_vlm/accumulated_pointcloud: accumulated point cloud in target_frame.
• /detection_vlm/detected_bboxes_3d: vision_msgs/msg/BoundingBox3DArray.
• /detection_vlm/visualization_3d_bboxes: RViz markers.
• /detection_vlm/debug_clusters_pointcloud: optional colored cluster debug cloud when publish_debug_clusters is enabled.

Service:

• /detection_vlm/set_prompt: update the detection prompt at runtime.

Important config options:

• association_method: dbscan or front_surface for selecting which projected LiDAR points belong to a 2D detection.
• use_masks_for_projection: use segmentation masks when available instead of only 2D boxes.
• voxel_size, eps_dbscan, min_points_per_cluster: point cloud filtering and clustering behavior.
• confidence_threshold: ROS 2 runtime parameter for detector confidence.
• publish_debug_clusters: publish the colored point clusters used for association.
• keep_boxes: keep previous RViz box markers for debugging.

Semantic Map

The semantic map node builds a persistent semantic voxel map from image detections and LiDAR geometry. Incoming point clouds are transformed through the configured lidar_to_body_transform and the body-to-target TF, then accumulated as geometry in target_frame. Each processed image updates per-voxel semantic scores and periodically publishes a colored semantic cloud and clustered object markers.

Launch:

ros2 launch detection_vlm_ros semantic_map.launch.yaml

Useful launch arguments:

ros2 launch detection_vlm_ros semantic_map.launch.yaml \
  config_path:=$(ros2 pkg prefix detection_vlm_ros)/share/detection_vlm_ros/config/semantic_mapping_yoloe.yaml \
  classes_file:=$(ros2 pkg prefix detection_vlm_ros)/share/detection_vlm_ros/config/classes.txt \
  input_image_topic:=/cam_front/image_raw/compressed \
  input_pointcloud_topic:=/mimosa_node/lidar/manager/points_full_res \
  input_camera_info_topic:=/camera/color/camera_info \
  target_frame:=mimosa_world \
  camera_frame:=camera0 \
  body_frame:=mimosa_body

Inputs:

• /semantic_map/input_image
• /semantic_map/input_pointcloud
• /semantic_map/input_camera_info
• TF from body_frame to target_frame, plus the configured lidar_to_body_transform.
• TF from target_frame to camera_frame.

Outputs:

• /semantic_map/detections_image: annotated 2D detections used to update the map.
• /semantic_map/semantic_map: transient-local sensor_msgs/msg/PointCloud2 containing active semantic voxels colored by label.
• /semantic_map/semantic_objects: transient-local visualization_msgs/msg/MarkerArray containing clustered semantic objects.

Service:

• /semantic_map/set_prompt: update the detection prompt at runtime.

Important config options:

• association_method: dbscan, front_surface, or none for selecting which projected LiDAR points should update each detected label.
• semantic_update_method: counter for additive votes or bayes for Bayesian score updates.
• semantic_publish_min_observations and semantic_publish_min_confidence: thresholds before a voxel is published as semantic.
• semantic_publish_period_s: publish period for the semantic point cloud.
• publish_semantic_objects and semantic_objects_period_s: enable and rate limit semantic object marker publishing.
• semantic_cluster_eps, semantic_cluster_min_points, semantic_object_min_voxels, semantic_object_min_confidence: object clustering and filtering parameters.
• active_window_enabled: update only a local window around body_frame. active_window_apply_to_updates limits computation during map updates.
• semantic_visualization_voxel_size and semantic_visualization_max_points: downsample the published map for RViz performance.
• max_active_update_points: cap the number of map points projected into each image update.

Q&A VLM

The reasoning node sends the input image and a binary question to an OpenAI VLM. It publishes a visualization image with the answer probability and explanation.

Launch:

export OPENAI_API_KEY=<Your OpenAI API key>
ros2 launch detection_vlm_ros reasoning_vlm.launch.yaml

Inputs:

• /reasoning_vlm/input_image

Outputs:

• /reasoning_vlm/reasoning_image: reasoning visualization.

Service:

• /reasoning_vlm/set_prompt: update the question at runtime.

Useful config options:

• overlay: enable or disable the probability color overlay.
• overlay_alpha: overlay opacity.
• footer_height: text footer height.
• runtime_logging_enabled: write per-query runtime rows to CSV.

---

License

Released under BSD-3-Clause.

---

Contact

For questions or support, reach out via GitHub Issues or contact the authors directly:

• Albert Gassol Puigjaner
• Kostas Alexis