Depthmapping with FFMpeg and OpenCV

Depthmapping with FFMpeg and OpenCV

Depthmaps are typically greyscale images that represent how far an object is by the degree of shade. They can be white-near, black-near, or some other color scale such as ironbar. Depthmaps are useful for creating renders or systems that have a distance sensing need.

In this post I’ll be using footage that I shot using 2 Innovv cameras. The rig was not precise, so I also post processed it to correct distortion and alignment issues. The post processing topic was covered in last weeks post.

What we’ll do is take a segment from the left and right cameras and use that for our depthmapping. Here are the commands to get the segments.

ffmpeg -ss 00:00:00 -i "/home/local/Desktop/STP 2017/Anaglyph/L0000004-Right-Defish-Rotate-Trim.MOV" -t 00:00:15 -acodec copy -vcodec copy -async 1 "/home/local/Desktop/DepthMap/Right/Right_Seg.MOV"
ffmpeg -ss 00:00:00 -i "/home/local/Desktop/STP 2017/Anaglyph/L0000004-Left-Defish-Trim.MOV" -t 00:00:15 -acodec copy -vcodec copy -async 1 "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV"

The OpenCV script is based off of the work demonstrated here, http://docs.opencv.org/3.1.0/dd/d53/tutorial_py_depthmap.html

The python script only works with single images. I have modified it so it will process a sequence of image files that have incremental numbering in the file name. First I’ll convert our video segments to images using this command.

ffmpeg -i "/home/local/Desktop/DepthMap/Right/Right_Seg.MOV" -vf fps=30 "/home/local/Desktop/DepthMap/Right/Right_Seg%d.png"
ffmpeg -i "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV" -vf fps=30 "/home/local/Desktop/DepthMap/Left/Left_Seg%d.png"

Now that I have the images, the python script can be run to process the images. Here is the python code.

==== DepthMap.py

import numpy as np
import cv2
from matplotlib import pyplot as plt

count = 0
Rimg = '/home/local/Desktop/DepthMap/Right/Right_Seg'
Limg = '/home/local/Desktop/DepthMap/Left/Left_Seg'
Dimg = '/home/local/Desktop/DepthMap/Depth/Depth_Seg'
Ext = '.png'

# The values in range(x,y,z) are x=start, y=end, z=step
for N in range(1,453,1):
count = count + 1

imgR = cv2.imread(Rimg + str(count + 4) + Ext, 0) # Frames not alligned, this compinsates
imgL = cv2.imread(Limg + str(count) + Ext, 0)

stereo = cv2.StereoBM(1, 16, 15)
disparity = stereo.compute(imgL,imgR)

plt.imshow(disparity,'gray')
plt.savefig(Dimg + str(count) + Ext)
====

One thing that I noticed with my version was that the python script tanked after 142 images. To continue on, I changed the value for the count variable to 142 and re-ran the python script. Eventually it finished. I just want to make that clear for anyone reproduction the effort.

This python script will create the file list needed for the image to video conversion.

==== FileList.py

count = 0
Dimg = 'file \'/home/local/Desktop/DepthMap/Depth/Depth_Seg'
Ext = '.png\''
text_file = open("/home/local/Desktop/DepthMap/Depth/Output.txt", "w")
# The values in range(x,y,z) are x=start, y=end, z=step
for N in range(1,453,1):
count = count + 1
text_file.write (Dimg + str(count) + Ext)
text_file.write ('\n')
text_file.close()
====

Now we can convert the image sequence into a video using this command.

ffmpeg -y -r 30 -f concat -safe 0 -i "/home/local/Desktop/DepthMap/Depth/Output.txt" -c:v libx264 -vf "fps=30,format=yuv420p" "/home/local/Desktop/DepthMap/Depth/Depth_Seg.mov"

The OpenCV process introduced a scale background which I would like to remove. After finding the cropping locations using GIMP, I ran this command.

ffmpeg -i "/home/local/Desktop/DepthMap/Depth/Depth_Seg.mov" -filter:v "crop=621:351:100:125" -c:a copy "/home/local/Desktop/DepthMap/Depth_Seg_Crop.mov"

To give a better presentation I decided to inlay the depthmap on top of the original left camera footage. I desaturated the original video to draw more attention to the depthmap inlay.

ffmpeg -i "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV" -vf "eq=saturation=0" "/home/local/Desktop/DepthMap/De-Saturate.mov"

Then I inserted the depthmap as a picture in picture.

ffmpeg -i "/home/local/Desktop/DepthMap/De-Saturate.mov" -vf "movie=/home/local/Desktop/DepthMap/Depth_Seg_Crop.mov, scale=621:351 [vid2]; [in][vid2] overlay=main_w-overlay_w-20:main_h-overlay_h-20" "/home/local/Desktop/DepthMap/De-Saturate_PicNPic.mov"

Finally, I drew a red boarder around the depthmap.

ffmpeg -i "/home/local/Desktop/DepthMap/De-Saturate_PicNPic.mov" -vf drawbox=x=1197:y=663:w=621:h=351:color=red@1 "/home/local/Desktop/DepthMap/De-Saturate_PicNPic_Final.mov"

That completes the process for creating depthmaps using FFMpeg and OpenCV. The results are far from ideal. This is mainly a proof of concept using existing footage. The efforts of creating depthmaps are still maturing. One effort by University College London is producing some striking results.

 

Comments are closed.