Depth from Stereo

Using this tutorial you will learn the basics of stereoscopic vision, including block-matching, calibration and rectification, depth from stereo using opencv, passive vs. active stereo and relation to structured light.

Why Depth?

Regular consumer web-cams offer streams of RGB data within the visible spectrum. This data can be used for object recognition and tracking, as well as some basic scene understanding.

Even with machine learning grasping the exact dimensions of physical objects is a very hard problem

This is where depth cameras come-in. The goal of a depth camera is to add a brand-new channel of information, with distance to every pixel. This new channel can be used just like the rest (for training and image processing) but also for measurement and scene reconstruction.

Stereoscopic Vision

Depth from Stereo is a classic computer vision algorithm inspired by human binocular vision system. It relies on two parallel view-ports and calculates depth by estimating disparities between matching key-points in the left and right images:

Most naive implementation of this idea is the SSD (Sum of Squared Differences) block-matching algorithm:

import numpy

fx = 942.8        # lense focal length
baseline = 54.8   # distance in mm between the two cameras
disparities = 64  # num of disparities to consider
block = 15        # block size to match
units = 0.001     # depth units

for i in xrange(block, left.shape[0] - block - 1):
    for j in xrange(block + disparities, left.shape[1] - block - 1):
        ssd = numpy.empty([disparities, 1])

        # calc SSD at all possible disparities
        l = left[(i - block):(i + block), (j - block):(j + block)]
        for d in xrange(0, disparities):
            r = right[(i - block):(i + block), (j - d - block):(j - d + block)]
            ssd[d] = numpy.sum((l[:,:]-r[:,:])**2)

        # select the best match
        disparity[i, j] = numpy.argmin(ssd)

# Convert disparity to depth
depth = np.zeros(shape=left.shape).astype(float)
depth[disparity > 0] = (fx * baseline) / (units * disparity[disparity > 0])

There are several challenges that any actual product has to overcome:

1. Ensuring that the images are in fact coming from two parallel views
2. Filtering out bad pixels where matching failed due to occlusion
3. Expanding the range of generated disparities from fixed set of integers to achieve sub-pixel accuracy

Calibration and Rectification

In reality having two exactly parallel view-ports is challenging. While it is possible to generalize the algorithm to any two calibrated cameras (by matching along epipolar lines), the more common approach is image rectification. During this step left and right images are reprojected to a common virtual plane:

Software Stereo

opencv library has everything you need to get started with depth:

1. calibrateCamera can be used to generate extrinsic calibration between any two arbitrary view-ports
2. stereorectify will help you rectify the two images prior to depth generation
3. stereobm and stereosgbm can be used for disparity calculation
4. reprojectimageto3d to project disparity image to 3D space

import numpy
import cv2

fx = 942.8          # lense focal length
baseline = 54.8     # distance in mm between the two cameras
disparities = 128   # num of disparities to consider
block = 31          # block size to match
units = 0.001       # depth units

sbm = cv2.StereoBM_create(numDisparities=disparities,
                          blockSize=block)

disparity = sbm.compute(left, right)

depth = np.zeros(shape=left.shape).astype(float)
depth[disparity > 0] = (fx * baseline) / (units * disparity[disparity > 0])

The average running time of stereobm on an Intel(R) Core(TM) i5-6600K CPU is around 110 ms offering effective 9 FPS (frames per second).

Get the full source code here

Passive vs Active Stereo

The quality of the results you will get with this algorithm depends primarily on the density of visually distinguishable points (features) for the algorithm to match. Any source of texture, natural or artificial will improve the accuracy significantly.

That's why it is extremely useful to have an optional texture projector (usually adding details outside of the visible spectrum). As an added benefit, such projector can be used as an artificial source of light at night or in the dark.

Remark on Structured-Light

Structured-Light is an alternative approach to depth from stereo. It relies on recognizing a specific projected pattern in a single image.

Having certain benefits, structured-light solutions are known to be fragile since any external interference (from either sunlight or another structured-light device) will prevent you from getting any depth.

In addition, since laser projector has to illuminate the entire scene, power consumption grows with range, often requiring a dedicated power source.

Depth from stereo on the other hand, only benefits from multi-camera setup and can be used with or without projector.

The D400 Series

<i><b>D400</b> RealSense cameras</i></p>

D400 RealSense cameras offer the following basic features:

1. The device comes fully calibrated producing hardware-rectified pairs of images
2. All depth calculations is done by the camera at up-to 90 FPS
3. The device offers sub-pixel accuracy and high fill-rate
4. There is an on-board texture projector for tough lighting conditions
5. The device runs of standard USB 5V power-source drawing around 1-1.5 W

This product was designed from the ground up to address conditions critical to robotics / drones developers and to overcome the limitations of structured light.

Just like opencv, RealSense technology offers open-source and cross-platform set of APIs for getting depth data.

1. electronicdesign - New RealSense Cameras Deliver Low-Cost 3D Solutions
2. Augmented World Expo - Help Your Embedded Systems See, Navigate, and Understand
3. List of whitepapers covering D400 RealSense technology
4. Where to buy Intel RealSense cameras