How Google’s ARCore Depth API Tackles Occlusion with a Single Camera



Google has taken augmented reality to a whole new level for perceiving 3D space and providing optimal AR experiences without bringing in the use of dedicated hardware like depth sensing cameras. Google’s ARCore[1] Depth API is capable of creating depth maps with just a single camera. For those who don’t know, ARCore is Google’s platform for augmented reality runtime and an SDK for Android and iOS. With this, developers can easily do positional tracking, surface detection and various other estimations. When moving your phone with a single camera, Depth API [2] will take multiple pictures around you and then compare those images to estimate the distance from each pixel to create a 3-dimensional structure of your surroundings making it possible to project real-world instances of objects into the current frame.

ARCore visualizations

How the Magic Happens

Google says, “Depth API allows developers to use our depth-from-motion algorithms to create a depth map using a single RGB camera [3]. The depth map is created by taking multiple images from different angles and comparing them as you move your phone to estimate the distance to every pixel [5] .” A depth-map basically lets you decide which object is near to you in such a way that you can estimate if the virtual object placed in the same environment will overlap or not and therefore, solving the problem of AR experiences for the overlapping of virtual objects on top of the real objects in the same environment.

The picture below explains the main concept behind ARCore Depth API. On the left side, a cat (digital object) overlaps both the sofa and floor (real-world objects) which ideally is not possible in real scenarios. The right side of the image uses depth API to overcome this issue and gives a feel of actually having a digital object in the real world.

The projections with real objects

The main features that stand out in the latest implementation are:

  • It does not require any special hardware such as cameras and sensors,
  • It works for a wide range of devices that support ARCore,
  • Devices with special featured cameras and time of flight sensors are more likely to get better and more accurate results along with a richer experience.

Supported Devices


Depth API does not just support single camera devices but also anticipates that time of flight [4] (ToF) sensors will enable better, faster and dynamic occlusion [6] which is capable of hiding digital object parts behind real objects. Depth API works as mentioned before without using conventional depth sensors which is the key to unlock vast computer vision applications. Many of the latest generations iPhones have used the front-facing depth sensors. Following the advancement, few of the recent Samsung devices have started to add these sensors. Due to its really advance provision, many other companies are expected to follow the trend soon.

Practical Applications

Google says, “We will begin making occlusion available in Scene Viewer, the developer tool that powers AR in Search, to an initial set of over 200 million ARCore enabled Android devices today.” Occlusion is nothing but the ability of Digital world objects to define accurate space and blend around real-world objects i.e. virtual objects will appear behind real objects no matter if the real object is closer to the camera.

Houzz (a company that is based on home renovation and interior design) has used Google’s ARCore depth API for building an app that provides “View in My Room 3D” features to their customers [8]. Sally Huang (Visual Technologies Lead at Houzz) said “Using the ARCore Depth API, people can see a more realistic preview of the products they’re about to buy, visualizing our 3D models right next to the existing furniture in a room. Doing this gives our users much more confidence in their purchasing decisions.” Below is the sample gif image of the Houzz app where customers can virtually place furniture and visualize it before bringing it as a real-world object. Here, the person taps in the area where he wishes to keep the table and moves the camera for visualizing it at various different angles. We see that the table does not overlap the chair which comes with the power of ARCore depth API.

The Depth Map

The Endless Implementations

Depth data is powerful for enabling the occlusion feature. Besides this, 3D depth data serves a lot more help to scientific areas such as surface interaction, path algorithms, realistic physics, etc. As well as to gaming industries, depth API provides interactive game mechanics that helps player hide behind real-world objects. Bringing these applications collectively in one place, you can create many more experiences in which objects bounce and splash accurately across the given environment.

The ARCore depth API is just a basic outcome of what’s possible with the depth API and Google plans to bring it out in a more planned and high tech way. Developers who wish to have access to depth API and contribute their innovative ideas to this feature need to register to Google's collaborators form [9].

Learn More

If you'd like to learn how to develop your own AI apps in Computer Vision (AI-CV) check out our training called Ultimate AI-CV Practitioners PRO. This course will teach you how to become a PRO AI-CV developer with: Object Detection, Instance Segmentation, Pose Estimation, Android and Raspberry Pi Development! Click the link or image below to attend the FREE Webinar. - Click Here

References

  1. https://developers.google.com/ar/discover
  2. https://developers.google.com/depthmap-metadata
  3. https://en.wikipedia.org/wiki/Depth_map
  4. https://en.wikipedia.org/wiki/Time-of-flight_camera
  5. https://en.wikipedia.org/wiki/Ambient_occlusion
  6. https://developers.googleblog.com/2019/12/blending-realities-with-arcore-depth-api.html
  7. https://uploadvr.com/arcore-depth-api/
  8. https://www.androidauthority.com/arcore-depth-api-1064324/
  9. https://www.androidpolice.com/2019/12/09/arcore-depth-api/
  10. https://www.xda-developers.com/google-arcore-depth-api-create-depth-maps-using-single-camera/

Comments

Post a Comment

Popular posts from this blog

Movidius NCS (with Raspberry Pi) vs. Google Edge TPU (Coral) vs. Nvidia Jetson Nano – A Quick Comparison

Image
Lately, there has been a lot of talk regarding the possibility of machines learning to do what human beings do in factories, homes, and offices. With the advancement in artificial intelligence , there has been widespread fear and excitement about what AI, machine learning, and deep learning is capable of doing. What is really cool is that Deep Learning and AI models are making their way from the cloud and bulky desktops to smaller and lower powered hardware. In this article, we will help you understand the strengths and weaknesses about three of the most dominant deep learning AI hardware platforms out there. The Intel Movidius Neural Compute Stick (NCS) with Raspberry Pi Developed by Intel Corporation, the Movidius Neural Compute Stick can efficiently operate without an active internet connection. Its computing capabilities come from the Myriad 2 Vision Processing Unit (VPU) . It offers profiling, tuning, and compiling a Deep Neural Network (DNN) on a development computer with the ri...
Read more

DeepSORT - Deep Learning applied to Object Tracking

Image
Introduction So in this article, Im going to give to you a clear and simple explanation on how Deep SORT works and why its so amazing compared to other models like Tracktor++, Track-RCNN and JDE. But to understand how DeepSORT works, we first have to go back, waaay back and understand the fundamentals of object tracking and the key innovations that had to happen along the way, for DeepSORT to emerge. Before we get started, if you are interested in developing object tracking apps, then checkout my course in the link down below, where I show you how you can fuse the popular YOLOv4 with DeepSORT for robust and real-time applications. Okay so back to Object Tracking. Now lets imagine that you that you are working for Space X, and Mr. Musk has tasked you with ensuring that on launch, the ground camera is always pointing at the Falcon 9 as it thrust into the atmosphere. As much as you are excited to be personally chosen by Elon to work on this task you ask yourself, “How will I go about th...
Read more

OpenCV AI Kit - An Introduction to OAK-1 and OAK-D

Image
Introduction Okay so I just got my OpenCV AI Kit and I am really excited to play with. But even though Iv seen a bunch of features on Kickstarter, I don’t quite understand the true capabilities of this device and why I should even care . Well in this video we going take a deep dive into this device and find out exactly what its capable off, and at the end of the video I will share my thoughts on if this device is worth all the hype as well how it matches up to the competition. What is the OpenCV AI Kit So before we get into that, lets get into the first question of What exactly is the OpenCV AI Kit or OAK. According to the creators, it is a tiny, powerful, open source Spatial AI system. So you can think of it as if an embedded 4k camera and neural compute stick had a baby, make that two babies OAK-1 and OAK-D Difference between the OAK 1 – OAK D Now What sets these two devices apart is that the OAK-1 has automatic motion-based lossless digital zooming which means that the sensor has ...
Read more