Intel® RealSense™ D400 series depth cameras use stereo-based algorithms to calculate depth. One key advantage of stereo depth systems is the ability to use as many cameras as you want to within a specific scene. In this post, we are going to cover creating a unified point cloud with multiple cameras using ROS.
For the initial demonstration, we set up two Intel® RealSense™ D435 cameras looking at the same area from two different points of view.
The cameras themselves have no data regarding their relative position. In Step 3, we’ll use a 3rd party program to set this up.
Install the Intel RealSense SDK 2.0 (librealsense) for Linux* following the instructions here. Install the ROS (not ROS 2) wrapper for librealsense from here. It is recommended to follow this set of instructions for the installation.
If you already know each camera’s serial number you can skip this step.
1. Open terminal and change directory to catkin_ws
2. Make sure only cam_1 is connected and start the realsense2_camera wrapper:
Note the serial number it finds in the following log line:
The serial number in this case is 728312070349. Record this somewhere and terminate the node using CTRL+C. Repeat the step with now only cam_2 connected.
Open 2 terminal windows (and change directory to catkin_ws directory).
In terminal 1 enter the following (don’t forget to replace 728312070349 with the cam_1 serial number you recorded in Step 1):
In terminal 2 enter the following (again, fill in the correct serial number for cam_2):
We now have two cameras running with their own point clouds.
We estimate the translation of cam_2 from cam_1 at 70(cm) on X-axis and 60(cm) on Y-axis. We also estimate the yaw angle of cam_2 relative to cam_1 as 90(degrees) clockwise – this based on our knowledge of the setup in the image above. To simplify things, the coordinate system of cam_1 will serves as the coordinate system for the whole scene. These are the initial parameters we set for the transformation between the 2 cameras.
The following script calculates the transformation between the 2 cameras from the given parameters and publish the frame transformation.
Open a third terminal window – terminal 3 – and enter the following:
Open a fourth terminal – terminal 4 – and run RViz. RViz is a 3D visualizer for displaying sensor data and state information from ROS:
In the RViz window, do the following:
Now both point clouds are shown together. If you looked closely it’s easy to see that the clouds are not exactly in position. Some features are duplicated. Now it’s time for fine calibration.
Switch to terminal 3, where set_cams_transforms.py is still running. Use it to fine-calibrate cam_2 relative to cam_1. The instructions are printed by the program:
Below that a constantly updating line is printed, showing the current mode, value and step size after every key stroke.
Notice, that the program prints the path of the current configuration file. It saves its last configuration automatically, all the time, to be used on the next run. If you have been running the program for a while, and want to back up the configuration file, just copy it aside. Read the rviz manual, by running without any parameters to learn more about changing the configuration filename.
There is no question that it would be much better to find the precise matrix between the cameras automatically. In the meanwhile, after a lot of fiddling around, I was relatively satisfied with the following numbers:
The result is a more complete cover of the scene:
There are a couple of reasons why the number of cameras connected to a single computer is limited: CPU usage, USB bandwidth, power supply and also, the length of the USB cable. These are all covered in length in the Multiple camera configuration white paper. In order to demonstrate the scenario we created the following array:
Both cam_2 and cam_3 are visible in cam_1’s point cloud. That way, it’s easier, though not trivial, to use set_cams_transforms.py to fix their location in cam_1’s coordinate system.
While using multiple computers we have to specifically set the roscore host. In this demo, the first command in each terminal is:
In this demo, the roscore, cam_1 and cam_2 run on perclnx319 and cam_3 is on perclnx217. Also, in this demo all the terminals are being opened on roscore’s computer, perclnx319. The commands to run are as follows – but remember to replace the serial numbers with your own.
cam_1 is rotated about 30 degrees compared to the left side of the imaginary square created by the 3 cameras. This camera is our coordinate system. We estimate the translation of cam_2 from cam_1 at 90(cm) on X-axis and 30(cm) on Y-axis. We also estimate the yaw angle of cam_2 at 120(degrees) clockwise, compared to cam_1. These are the initial parameters we set for the transformation between the 2 cameras. As for cam_3, we estimate the translation on X-axis at 160(cm) and on Y-axis at -20(cm). We estimate the yaw at 170(degrees) counter-clockwise. The following script calculates the transformation from the given parameters and publish the frame transform.
Press Q to Quit.
Now set transformation for cam_3:
Since set_cams_transforms.py is not a ROS-package and you can’t run 2 nodes with the same name, we run one copy at a time.
From now on, the last values for the relative transformations are saved in the specified files. If you want to modify cam_2 extrinsics, type:
Notice: Use 2 different file names for saving the results for two different sets of cameras.
Enter the following command: ‘ rviz ‘ In RViz, do the following:
Now three point clouds are shown together. It’s easy to see that the clouds are not exactly in position. Some features are duplicated. Now it’s time for fine calibration: Switching to terminal 5, where set_cams_transforms.py is still running, we use it to fine calibrate cam_2 and cam_3 relative to cam_1. The instruction are printed by the program:
Without an automatic calibration tool it can be quite challenging to tune the cameras. Here are a couple of tips for calibration:
Eventually, I was relatively satisfied with the following numbers.
No doubt, a SLAM based solution is better then the manual labor I’ve been through to get this calibration, both in time and results. Nevertheless, the final 3D-model I got looked like that:
Subscribe here to get blog and news updates.
3D scanning There are many different applications for 3D scanning, and many different approaches and solutions, depending on things like
What is 3D Scanning? 3D scanning has come a long way since the first ever object to be ‘scanned’ using students,
We're here to help
Let’s talk about how Intel RealSense depth sensing cameras can enhance your solution.
We'll be in touch soon.
You are about to leave our website and switch to intel.com. Click here to proceed
Keep up to date with Intel® RealSense™ Technology: product updates, upcoming announcements, and events.