Xivanki Share !!: Understanding the Visual World, one 3D reconstruction at a time

Understanding the Visual World, one 3D reconstruction at a time - Selamat datang di situs media global terbaru Xivanki, Pada halaman ini kami menyajikan informasi tentang Understanding the Visual World, one 3D reconstruction at a time !! Semoga tulisan dengan kategori 3D reconstruction !! annotation !! computer vision !! depth !! ICCV 2013 !! jianxiong xiao !! kinect !! labelme !! labelme3D !! MIT !! paper !! reconstruction !! sfm !! structure from motion !! ini bermanfaat bagi anda. Silahkan sebarluaskan postingan Understanding the Visual World, one 3D reconstruction at a time ini ke social media anda, Semoga rezeki berlimpah ikut dimudahkan Allah bagi anda, Lebih jelas infonya lansung dibawah -->

The first generation of datasets in the computer vision community were just plain old images --  simple arrays of pixels.  Seem like nothing fancy, but we must recall that there was a time where a single image could barely fit inside a computer's memory.  During this early time, researchers showcased their image processing algorithms on the infamous Lenna image.  But later we saw datasets like the Corel dataset, Caltech 101, LabelMe, SUN, James Hays' 6 million Flick images, PASCAL VOC, and Image Net.  These were impressive collections of images and for the first time computer vision researchers did not have to collect their own images.  As Jitendra Malik once said, large annotated datasets marked the end of the "Wild Wild West" in Computer Vision -- for the first time, large datasets allowed researchers to compare object recognition algorithms on the same sets of images! What is different about these datasets is that some come with annotations at the image level, some come with annotated polygons, and some come with nothing more than objects annotated at the bounding box level.  Images are captured by a camera and annotations are produced by a human annotation effort.  But these traditional vision datasets lack depth, 3D information, or anything of that sort.  LabelMe3D was an attempt at reconstructing depth from object annotations, but it would only work in a pop-up world kind of way.

The next generation of datasets is all about going into 3D.  But not just annotated depth images like the NYU2 Depth Dataset depicted the in following image:

What a 3D Environment dataset (or 3D place dataset) is all about is making 3D reconstructions the basic primitive of research.  This means that an actual 3D reconstruction algorithm has to first be ran to create dataset.  This is a fairly new idea in the Computer Vision community.  The paper which introduces such a dataset, SUN3D, was introduced at this year's ICCV 2013 conference.  I briefly outlined the paper in my ICCV 2013 summary blog post, but I felt that this topic is worthy of its own blog post.  For those interested, the paper link is below:

J. Xiao, A. Owens and A. Torralba SUN3D: A Database of Big Spaces Reconstructed using SfM and Object Labels Proceedings of 14th IEEE International Conference on Computer Vision (ICCV2013). paper link

Running a 3D reconstruction algorithm is no easy feat, so Xiao et al. found that some basic polygon-level annotations were sufficient for snapping Structure from Motion algorithms into place.  For those of you that don't what know a Structure from Motion (SfM) algorithm is, it is a process which reconstructs the 3D locations of points inside images (the structure) as well as the camera parameters (the motion) for a sequence of images.  Xiao et al.'s SfM algorithm uses the depth data from a Kinect sensor in addition to the manually provided object annotations. Checkout their video below:

Depth dataset vs SUN 3D dataset
The NYU2 Depth dataset is useful for studying object detection algorithms which operate on 2.5D images while MIT's SUN 3D dataset is useful for contextual reasoning and object-object relationships.  This is important because Kinect images do not give full 3D, they merely return a 2.5D "depth image" from which certain physical relationships cannot be easily inferred.

Startups doing 3D
It is also worthwhile pointing out that Matterport, a new startup, is creating their own sensors and algorithms for helping people create their own 3D reconstructions. Check out their vide below:


What this means for the rest of us
We should expect the next generation of smartphones to have their own 3D sensors.  In addition, we should expect the new generation of wearable devices such as Google Glass to give us more than 2D reasoning, they should be able to use this 3D data to make better visual inferences.  I'm glad to see 3D getting more and more popular as this allows researchers to work on new problems, new data structures, and push their creativity to the next level!