ToF camera

This page describes the integration of PMD Flexx2 ToF camera with NavQPlus

1. What is a ToF camera

2. PMD Flexx camera specification

3. Install Royale SDK

4. Connect the camera to a PC

   1. Using Linux machine

   2. Using Windows machine

5. Connect the camera to the NavQP

   1. Dispaly via TCP server

   2. Display via RVIZ and ROS2

6. References

What is a ToF camera

A Time-of-Flight camera (ToF) is a range imaging sensor that produces a depth image, each pixel of which encodes the distance to the corresponding point in the scene [1].

In addition to the camera that produces 2D images, the ToF sensor includes an IR transmitter. By measuring the phase difference between the radiated (red) and reflected (blue) IR waves, we can calculate the distance to the object as depicted in the figure below.

The principle of ToF camera [1]

PMD Flexx2 camera specification

The PMD Flexx2 camera intended use are: augmented Reality / Virtual Reality and SLAM. It has the following specifications:

• Depth Measurements from 10 cm to 7 m

• VCSEL with 940 nm Wavelength

• 38,000 3D Pixels with up to 60 FPS

• Independent of External Light Source

• 224 x 172 Resolution

• Multiple built-in User Modes

• 56 x 44 Degree Field-of-View

• USB 3.0 Type-C Port

• 71.9 x 19.2 x 10.6 mm compact size

• Software Suite "Royale" and API

Install Royale SDK

Download the royale SDK from the official website: https://pmdtec.com/en/download-sdk/

You need to fill in the form then you will get an email with a download link available for 24 hours. Use the password that comes with the camera.

You need to download two .zip files: one for your PC where you will display the camera data connected to the PC or the NavQP, and one for the NavQP.

The recent SDK for Flexx2 camera does not have the ROS2 driver. But the previous SDK does. Then, as recommendation, download the zip file libroyale-4.24.0.1201-LINUX-arm-64Bit-ub2004.zip for NavQP.

• Linux PC: libroyale-<ROYALE-SDK-VERSION>-LINUX-x86-64Bit.zip

• Windows PC: libroyale-<ROYALE-SDK-VERSION>-WINDOWS-x86-64Bit.exe.zip

• NavQP: libroyale-4.24.0.1201-LINUX-arm-64Bit-ub2004.zip

Connect the camera to a PC

Connect the camera to the PC using a USB-A to USB-C cable.

Using Linux machine

If you are using a Linux machine, download the zip file libroyale-<ROYALE-SDK-VERSION>-LINUX-x86-64Bit.zip from the download page, add the udev rules then run royaleviewer after reboot:

$ unzip ~/libroyale-<ROYALE-SDK-VERSION>-LINUX-x86-64Bit.zip
$ cd ~/libroyale-<ROYALE-SDK-VERSION>-LINUX-x86-64Bit/
$ sudo cp driver/udev/10-royale-ubuntu.rules /etc/udev/rules.d/
$ reboot
$ cd ~/libroyale-<ROYALE-SDK-VERSION>-LINUX-x86-64Bit/bin
$ ./royaleviewer

A GUI will start, click on start button on the bottom right corner to start the display:

You can use the TOOLS button to change some parameters. For example you can change the color range, or the type of data displayed (Gray image, depth image), etc

There is also a LOG button on the top right corner of the GUI where you can find useful debug or error logs.

Using Windows machine

If you are using a windows machine, install the zip file libroyale-<ROYALE-SDK-VERSION>-WINDOWS-x86-64Bit.exe. Unzip the file then run the executable to install the Royale Viewer.

Once installed you can find the executable with the name royaleviewer-royale_<ROYALE-SDK-VERSION> under C:\Users\<USER>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\royale_<ROYALE-SDK-VERSION> (Win64)

Once the program is launched, the steps are similar to those described in Linux sub-section.

Connect the camera to the NavQPlus

Connect the camera to the NavQPlus with an USB-C to USB-C cable (see picture below).

Connecting the camera to the NavQPlus uses a USB-C to USB-C cable. A USB-A to USB-C cable will not work and the camera will not seen by the board. The USB port is configured correctly as HOST.

Before using the camera, you need to add the udev rules available in the SDK zip file that you downloaded from PMD download link.

$ unzip ~/libroyale-4.24.0.1201-LINUX-arm-64Bit-ub2004.zip
$ cd ~/libroyale-4.24.0.1201-LINUX-arm-64Bit-ub2004/
$ sudo cp driver/udev/10-royale-ubuntu.rules /etc/udev/rules.d/
$ reboot

Display via TCP server

Under the bin/ directory, run the binary tcpserver:

$ cd  ~/libroyale-4.24.0.1201-LINUX-arm-64Bit-ub2004/bin
$ ./tcpserver

Display via RVIZ and ROS2

The Royale SDK provides some sample codes available under samples/cpp/ directory. You should find a directory sampleROS2, which contains code for ROS2 driver.

You need to adjust this code to be able to compile in ROS2 Humble. First create a ROS2 workspace and copy the ROS2 driver there:

$ mkdir -p ws_tof/src
$ cp -rf ~/libroyale-4.24.0.1201-LINUX-arm-64Bit-ub2004/samples/cpp/sampleROS2 ~/ws_tof/src

Then change the CMakeLists.txt file and the package.xml file to add the package pluginlib dependency:

package.xml file update

CMakeLists.txt file update

Then compile the ROS2 workspace using the following command:

$ colcon build --cmake-args "-DCMAKE_PREFIX_PATH=~/libroyale-4.24.0.1201-LINUX-arm-64Bit-ub2004/share"

Change the path to Royale SDK depending on your environment, and make sure to reference to share/ directory in this command.

Run the ROS2 node:

$ source install/setup.bash
$ ros2 run royale_in_ros2 royale_in_ros2

Make sure that mlan0 interface in listed in the list of interfaces in CycloneDDSConfig.xml file, then run RVIZ in a PC:

References:

[1] Hansard, Miles, Seungkyu Lee, Ouk Choi, and Radu Horaud. Time-of-Flight Cameras: Principles, Methods and Applications. SpringerBriefs in Computer Science. London: Springer London, 2013. https://doi.org/10.1007/978-1-4471-4658-2.

[2] https://3d.pmdtec.com/en/3d-cameras/flexx2/

[3] SDK download link: https://pmdtec.com/en/download-sdk/