With the introduction of the Intel® RealSense™ Depth Camera D435i and Tracking Camera T265, the librealsense SDK has been enhanced to support and effectively utilize the functionality provided by these new sensors.
The D435i and T265 models sport a Bosch BMI055 6-axis inertial sensor. Inertial Measurement Units (IMU) are sensors which allow measurement of both directional movement and rotation. The sensor is comprised of triaxial 12bit linear acceleration and triaxial 16bit angular velocity gyro sensors.
The D435i depth camera generates and transmits the gyro and accelerometer samples independently, as the inertial sensors exhibit different FPS rates (200/400Hz for gyro, 63/250Hz for accelerometer). Each IMU data packet is timestamped using the depth sensor hardware clock to allow temporal synchronization between gyro, accel and depth frames.
The T265 tracking camera utilizes the same IMU sensor as the D435i. However, unlike the D435i, which delivers the raw IMU data directly to the host PC, the T265 redirects IMU readings into an Intel® Movidius™ Myriad™ 2 Vision Processing Unit (VPU). The inertial sensor data is also complemented by video from two fisheye monochrome sensors that are fed into the VPU as well.
The VPU employed with T265 is capable of fusing inertial and video sensors to produce a reliable and persistent six degrees of freedom (6DOF) pose, offloading the CPU-intensive calculations from the host PC.
The following table summarizes the features and capabilities of the two devices (T265 vs D435i):
The IMU sensor location and orientation relative to the depth sensors are conveniently embedded into the sensor’s extrinsic data. In order to align and them with SDK-wide established orientation convention using the depth sensor as CS origin, each IMU sample is multiplied internally by the extrinsic matrix.
The resulted orientation angles and acceleration vectors share the coordinate system with the depth sensor.
1. The positive x-axis points to the right. 2. The positive y-axis points down. 3. The positive z-axis points forward
The angles are also compatible with OpenCV convention for pinhole camera model.
To aid AR/VR integration, the TM265 tracking device uses the defacto VR framework standard coordinate system instead of the SDK standard:
1. Positive X direction is towards right imager 2. Positive Y direction is upwards toward the top of the device 3. Positive Z direction is inwards toward the back of the device
The center of tracking corresponds to the center location between the right and left monochrome imagers on the device.
The IMU sensor does not include internal calibration, which may manifest itself with non-zero angular velocities produced at idle mode and the gravity (accel) force measured being not equal to 9.80665 m/s2. In order to rectify those inaccuracies, the T265’s IMU sensor is calibrated in the production line.
For D435i the IMU sensor is not calibrated, hence a free calibration tool is available as as part of the SDK. Running the calibration routine will calculate IMU intrinsic covariance and zero offsets, and then store them on device’s NVRAM for later use in SDK. The depth<->IMU sensor extrinsic (rigid body transformation) is precalculated based on mechanical drawings and cannot be modified. When initialized, the SDK will query the device for the existence of the IMU calibration data, and if present – apply it to the raw IMU samples produced by the device.
The following librealsense tools and demos are IMU and tracking-ready:
– rs-capture – 2D Visualization.
– rs-enumerate-devices – list the IMU and tracking profiles (FPS rates and formats).
– rs-data-collect – Store and serialize IMU and Tracking (pose) data in Excel-friendly csv format. The tool uses low-level sensor API to minimize software-imposed latencies. Useful for performance profiling.
– realsense-viewer – Provides 2D visualization of IMU and Tracking data. 3D visualization is available for Pose samples:
The IMU and Tracking data streams are fully compatible with SDK’s embedded recorder utility.
The IMU and tracking sensors are treated by the SDK like any other supported sensor. Therefore the sensor access and invocation API calls are similar to those of the depth/rgb sensors of D400 and SR300:
while (app) // Application still alive?
rs2::frameset frameset = pipe.wait_for_frames();
// Find and retrieve IMU and/or tracking data
if (rs2::motion_frame accel_frame = frameset.first_or_default(RS2_STREAM_ACCEL))
rs2_vector accel_sample = accel_frame.get_motion_data();
//std::cout << "Accel:" << accel_sample.x << ", " << accel_sample.y << ", " << accel_sample.z << std::endl;
if (rs2::motion_frame gyro_frame = frameset.first_or_default(RS2_STREAM_GYRO))
rs2_vector gyro_sample = gyro_frame.get_motion_data();
//std::cout << "Gyro:" << gyro_sample.x << ", " << gyro_sample.y << ", " << gyro_sample.z << std::endl;
if (rs2::pose_frame pose_frame = frameset.first_or_default(RS2_STREAM_POSE))
rs2_pose pose_sample = pose_frame.get_pose_data();
//std::cout << "Pose:" << pose_sample.translation.x << ", " << pose_sample.translation.y << ", " << pose_sample.translation.z << std::endl;
See our GitHub for more details.
The IMU and tracking sensors are currently supported on Windows and Linux* OS. Linux OS requires the kernel patches to applied either manually or via the librealsense2-dkms package. macOS* and Android* support is planned but there is still no firm timeline.
Subscribe here to get blog and news updates.
In this How-To, we are going to share a simple way to 3D scan an environment that you can then
3D scanning There are many different applications for 3D scanning, and many different approaches and solutions, depending on things like
Let’s talk about how Intel RealSense depth sensing cameras can enhance your solution.
We'll be in touch soon.
Keep up to date with Intel® RealSense™ Technology: product updates, upcoming announcements, and events.
You were successfully subscribed.
You are about to leave our website and switch to intel.com. Click here to proceed