![Stereo “Binocular” Vision
5
[from Wikipedia: Binocular vision]
Vision made with TWO “EYES”
It gives a “creature” a spare eye in case one is damaged
It gives a wider field of view
It can give stereopsis in which binocular disparity (or
parallax) provided by the two eyes' different positions on
the head gives precise depth perception
It allows a creature to see more of, or all of, an object
behind an obstacle
It gives binocular summation in which the ability to detect
faint objects is enhanced
WalterLucetti–myzhar@robot-home.it
21/05/2016Officine Robotiche 2016](https://image.slidesharecdn.com/robotsthatseelikehumans-160524075352/85/Robots-that-see-like-humans-5-320.jpg)
Robots that see like humans
- 1. Robots that see likeRobots that see like humanshumans Real Time Stereo Vision With ZED Camera on Nvidia Jetson TX1 Walter Lucetti [email protected]
- 2. Summary 21/05/2016Officine Robotiche 2016 2 WalterLucetti–[email protected] Live Demo Conclusions Introduction Stereo “Binocular” Vision Robotic Stereo Vision “Simplified” Stereo Vision Pin Hole Camera model Stereo Vision camera model Stereo Vision process A Robotic Stereo Vision System Nvidia Jetson TX1 Stereolabs ZED camera Stereolabs ZED SDK Stereolabs ZED engine Simple Code example
- 4. Introduction 4 WalterLucetti–[email protected] Can a robot replicate the complicated human binocular vision system? 21/05/2016Officine Robotiche 2016
- 5. Stereo “Binocular” Vision 5 [from Wikipedia: Binocular vision] Vision made with TWO “EYES” It gives a “creature” a spare eye in case one is damaged It gives a wider field of view It can give stereopsis in which binocular disparity (or parallax) provided by the two eyes' different positions on the head gives precise depth perception It allows a creature to see more of, or all of, an object behind an obstacle It gives binocular summation in which the ability to detect faint objects is enhanced WalterLucetti–[email protected] 21/05/2016Officine Robotiche 2016
- 6. Robotic Stereo Vision 6 How to replicate Human Stereo Vision? Two synchronized camera sensors A solid mechanical structure A powerful elaboration system and if we want to be more precise, but we want to complicate the system: Two motors to replicate vergence capability WalterLucetti–[email protected] 21/05/2016Officine Robotiche 2016
- 8. “Simplified” Stereo Vision 8 WalterLucetti–[email protected] A complete Stereo Vision system has vergence capability, but is too complicated We will study a Simplified Stereo Vision system where cameras cannot rotate 21/05/2016Officine Robotiche 2016
- 9. 9 WalterLucetti–[email protected] Pin hole camera model Camera Plane Optical Axis Image coordinates World coordinates World to image Image to world Without “Z” we cannot estimate the real dimension of an object using a single camera! “f” can be estimated calibrating the camera system This is a simplified camera model that does not take care of Optical Distorsion and Sensor Imperfections 21/05/2016Officine Robotiche 2016 u= f ∗x z v= f ∗y z x= u∗z f y= v∗z f
- 10. 10 WalterLucetti–[email protected] Stereo Vision model Very important parameter: ● Baseline: b From triangles similarity: Very important value: ● Disparity: D = ul - ur From disparity we can calculate how much a point is far from a camera: THE “Z” VALUE!!! 21/05/2016Officine Robotiche 2016 f Z P = ul X P f Z P = −ur b−X P ZP= b∗ f ul−ur
- 11. Calculate “Disparity” (D) for each point of the image 11 WalterLucetti–[email protected] Stereo Vision process Calculate the depth (Z) of each point of the image Calculate X and Y of each point of the image Algorithm MATCHING REQUIRES A LOT OF COMPUTATIONAL POWER D = ul - ur 21/05/2016Officine Robotiche 2016
- 12. 12 WalterLucetti–[email protected] Stereo Vision summary Allows to calculate real world coordinate of an object Requires a solid mechanical structure Requires a precise calibration of the optical system Requires a lot of computational power … Is more complicated than what seen until now 21/05/2016Officine Robotiche 2016
- 13. A Robotic Stereo Vision system13
- 14. 14 WalterLucetti–[email protected] A Robotic Stereo Vision System Nvidia Jetson TX1 www.nvidia.com/object/jetson-embedded- systems.html Stereolabs ZED www.stereolabs.com 21/05/2016Officine Robotiche 2016
- 15. 15 WalterLucetti–[email protected] Nvidia Jetson TK1 Main features Nvidia Tegra X1 SOC NVIDIA Maxwell GPU with 256 CUDA cores NVIDIA 4-Plus-1™ 2.3 Ghz quad- core ARM® Cortex™-A15 CPU 4 GB LPDDR4 Memory 16 GB eMMC 5.1 Flash Storage 1 full-size SD/MMC connector 1 full-size HDMI port 1 USB 2.0 port, micro AB 1 USB 3.0 port, A 1 RTL8111GS Realtek GigE LAN GPIO, SPI, I2C, UART, CAN WiFi & Bluetooth connection ... Native OpenCV support with GPU module for Real Time computer vision ROS enabled Ubuntu powered and more... Android ... 21/05/2016Officine Robotiche 2016
- 16. 16 WalterLucetti–[email protected] Stereolabs ZED camera Main features Auto calibrating and auto sync USB 3.0 High Res at high FPS 2208x1242 15 Hz 1920x1080 30 Hz 1280x720 60 Hz 640x480 120 Hz Simple and well documented C++ SDK Automatic Depth Map generation 3D Range: 1 m to 20 m (12 cm baseline) Outdoor and indoor Jetson TX1 driver ROS driver OpenCV support 21/05/2016Officine Robotiche 2016
- 17. 17 WalterLucetti–[email protected] Stereolabs ZED SDK C++ Classes and APIs for Linux and Windows 2 main classes and 5 support classes Main classes: Camera: the main class to use the ZED camera Mat: the class to store image data in GPU and CPU memory spaces Support classes: CamParameters: calibration params for the two sensors StereoParameters: calibration params for stereo processing resolution: dimensions of the images float3Struct: 3D point uchar3Struct: RGB pixel Current version: 0.9.3-beta for Linux and Windows, 0.9.3-alpha for Jetson TX1 21/05/2016Officine Robotiche 2016
- 18. 18 WalterLucetti–[email protected] Stereolabs ZED engine Stereo results are available in GPU memory Direct processing by CUDA or download to CPU memory for CPU processing 21/05/2016Officine Robotiche 2016
- 19. 19 WalterLucetti–[email protected] Stereolabs ZED tools ZED Explorer ZED Depth Viewer 21/05/2016Officine Robotiche 2016
- 20. 20 WalterLucetti–[email protected] Simple code example //ZED Includes #include <zed/Camera.hpp> int main(int argc, char **argv) { sl::zed::SENSING_MODE dm_type = sl::zed::RAW; sl::zed::Camera* zed; // Camera object creation zed = new sl::zed::Camera(sl::zed::HD2K); // We can use different resolutions: // HD2K HD1080 HD720 VGA // Sizes of the images int width = zed->getImageSize().width; int height = zed->getImageSize().height; // Initialization of the camera sl::zed::ERRCODE err = zed->init( sl::zed::MODE::PERFORMANCE ); if (err != sl::zed::SUCCESS) { delete zed; return 1; } // Stereo processing zed->grab(dm_type); // CPU Data sl::zed::Mat depth_cpu; sl::zed::Mat left_cpu; sl::zed::Mat right_cpu; // Data acquiring on CPU memory depth_cpu = zed->retrieveMeasure(sl::MEASURE::DEPTH); left_cpu = zed->retrieveImage(sl::SIDE::LEFT); right_cpu = zed->retrieveImage(sl::SIDE::RIGHT); // GPU Data sl::zed::Mat depth_gpu; sl::zed::Mat left_gpu; sl::zed::Mat right_gpu; // Data acquiring on GPU memory depth_cpu = zed->retrieveMeasure_gpu(sl::MEASURE::DEPTH); left_cpu = zed->retrieveImage_gpu(sl::SIDE::LEFT); right_cpu = zed->retrieveImage_gpu(sl::SIDE::RIGHT); return 0; } 21/05/2016Officine Robotiche 2016
- 21. Live demo21
- 22. Conclusions22
- 23. 23 WalterLucetti–[email protected] Conclusions With Stereo Vision A robot can see obstacles A robot can measure obstacles A robot can localize obstacles Stereo Vision requires a lot of mathematical elaborations a lot of computational power a solid mechanical structure so Nvidia Jetson TX1 with Stereolabs ZED is a really amazing solution for Robotic Stereo Vision! 21/05/2016Officine Robotiche 2016
- 24. The End! OR THANK YOU FOR YOUR ATTENTION! Walter Lucetti http://myzharbot.robot-home.it GitHub @myzhar Youtube Myzhar Twitter @myzhar Facebook robothome 24 21/05/2016Officine Robotiche 2016