Author Archives: Naveen Anand

About Naveen Anand

I am currently a Masters student in the department of Electrical and Computer Engineering at University of Wisconsin-Madison. My concentrations are Computer Architecture and Operating Systems. I work with Dr.Kevin Ponto (https://blogs.discovery.wisc.edu/kponto/) in the Living Environments Lab, Wisconsin Institute for Discovery as a Project Assistant for the vizHOME project. Image processing and Virtual Reality have always been one of my major co-curricular interests since undergrad. This blog is going to be a chronological compilation of my independent study and future work with Dr.Ponto.

ReKinStruct: To Sum It Up

Posted on by


https://vimeo.com/95045553

What started as a project that would reconstruct point cloud data through Kinect Fusion SDK along with PCL had its limitations that led me to take a detour to getting Time Varying Datasets and finally landing in the Kinect Fusion SDK.

There was a very steep initial learning curve with respect to setting up the drivers and the software. My Macbook was not supporting the Kinect drivers as it had a low end graphics card and I had to use Dr.Ponto’s Alienware laptop with an NVIDIA GTX780 graphics card that was pretty fast. Compiling PCL and its dependencies, OpenNI and PrimeSense was the next step which had a few issues of their own while interacting with Windows drivers. These initial phase was very frustrating as I have not really coded much in the Windows OS and had to figure out how to setup the hardware, drivers and software. It was almost mid-March when I had the entire setup running without crashing in the middle.

Although it was a late start, once the drivers and the software were setup, it was all exciting and fast. I was able to obtain datasets automatically using the OpenNI Grabber interface. I had to just specify the time interval between successive captures and the program saved them as PCD files (Color and Depth). It wasn’t late until I was able to get 400 PCDs of a candle burning down with PCDs captured 1 second apart from each other which would give a realistic 3D rendering of the scenario. The viewer program was pretty similar that took in the number of PCD files and the time interval between the display as arguments and visualized these 3D datasets.

Further on, I also tried learning the Windows SDK that is provided along with the Kinect. The Kinect Fusion basics is a beautifully written piece of code that obtains PLYs when scanned with the Kinect. PCL offers options to convert these into PCDs which was the desired final format. I also tried running multiple Kinects simultaneously to obtain data that would fill in the shadow points of one Kinect but I was not able to debug an error while Windows SDK’s Multi Static Cameras option. Given more time or as a future work, I believe using multiple Kinects to obtain PCD files would be a good area to explore. Working on these obtained PCD files like hole filling and reconstruction would also be good topics to cover in the future.

Here is a comparison of the image that is used in pointclouds.org(the one I put up in my second post as a target) against the image that I have obtained. Both are screenshots of PLYs.

Comparison

Left: Mesh from pointclouds.org; Right: Mesh obtained by me

The datasets can be found at:

Candle: https://uwmadison.box.com/s/j1zheh8b46fbxjs079xs

Walking Person: https://uwmadison.box.com/s/lxkr7a7io5rbz84xy4uw

Office: https://uwmadison.box.com/s/8mfccacpewkptymicx67

All in all, I am happy with the progress of the work. If the drivers were not a big hindrance, I would have had a better start in the beginning of the semester. Nevertheless, it was a great learning experience and an interesting area of study.

ReKinStruct: Time-Varying Kinect Fusion

Posted on by

I have tried to combine Kinect Fusion and the Time-Varying Dataset concept into one by obtaining two time varying PLYs of a scenario and converting them into PCDs. The link to this small dataset could be found at

https://uwmadison.box.com/s/bgw63fi54y5ir5wkedav

Meanwhile, since I did not have OpenNI support in my machine as I have deleted it while installating Kinect SDKs, I could not try running the program to visualise it. Nevertheless, the comparison of the two PLYs looks so in MeshLab.

KinectFusion_Comaprison

The scans look intact and the Kinect’s depth sensors were actually working much better compared to last week since I moved the Kinect slowly across the space.

The codes to the capture and viewing program that I have worked with until now can be found at

https://github.com/nsubramaniam/rekinstruct

For some reason, using the Multi Static Kinects option in the Kinect SDKs which obtains points from two Kinects gives an error while opening the PLY file it saves. The error corresponds to something like “Header has an EoF” which I believe is erroneous metadata. I am looking into it. Will update once I get to know something.!

ReKinStruct: Kinect Fusion Basics

Posted on by

There has been a bit of a change in my approach to obtain PCDs. I have moved on to obtain PCDs(PLY, to be precise) using the Kinect Fusion Colour Basics. It scans the volume for 3D points and allows me to save in one of the three formats (OBJ, STD and PLY). I chose the PLY format as I can easily convert PLYs to PCDs using PCL.

Here is a video of me scanning around the room to obtain the mesh.

http://vimeo.com/93961128

The mesh finally looks like this in MeshLab.

Mesh

The next steps would be to take a few PCDs and probably use the Time Varying PCD program to visualise them like a movie.

Meanwhile, there happened to be a few issues with the datasets in the file locker. So, I have uploaded the files in UW-Madison Box. Here are the links to the datasets.

Candle Dataset 1 second : https://uwmadison.box.com/s/j1zheh8b46fbxjs079xs

Walking Person : https://uwmadison.box.com/s/lxkr7a7io5rbz84xy4uw

ReKinStruct : Shifting Gears

Posted on by

As I said in the last post, I am trying to obtain colour and depth stream from two kinects at once. Apparently, OpenNI is not the best way to do it. So I am trying to go back to Kinect SDKs. I have successfully installed the softwares. There were a lot of dependency interferences from the PrimeSense Kienct Sensors that I had already. Had a bunch of installing and uninstalling to do. Now, the Kinect works to obtain colour and depth images as in one of my first posts :

https://blogs.discovery.wisc.edu/projects/2014/02/09/rekinstruct-abstract/

I am learning how to obtain this through a program in selected intervals so I can make a time varying point cloud. Will keep you updated.!

A good and simple tutorial to install Kinect SDKs that I found was:

http://www.packtpub.com/article/getting-started-with-kinect-for-windows-sdk-programming

Meanwhile, I got the dataset for the candle burning with an interval of one second between consecutive PCDs. The following link has 400 PCDs of a candle burning spaced one second apart.

https://filelocker.discovery.wisc.edu/public_download?shareId=8ab2882502ec2aea65d711cfec4bbdd8

Password: ReKinStruct

ReKinStruct : Candle Dataset

Posted on by

I have uploaded the candle dataset on the file locker system. You can download it from here.

https://filelocker.discovery.wisc.edu/public_download?shareId=1d35394ba86c9251776af95cb2821f46

Password: ReKinStruct

These are 40 PCD files of snapshots taken of two candles with a 10 second interval between consecutive shots.

I have also uploaded a video of the ReKinStruct viewer for the candle dataset here. The video however is like a fast forward version of candles burning where the 40 PCD files are displayed back to back with a time interval of 1 second. This makes a close-to-reality rendering of candles burning but only 10x faster.

http://youtu.be/zwA9J8xv248

I believe I kept the candles too close to the Kinect and didn’t get it’s depth data perfectly. However, the shadows and the candle melting show how the time varying datasets look in 3D.

I have been trying to setup two Kinects to grab data simultaneously and have been getting the classic segmentation fault.

Two Kinects SegFlt

I wonder if it goes to back to the original issue where the OpenNI grabber was not sensing the Kinect. I have looked online for some tutorials and most people seem to have used two OpenNI grabbers simultaneously. Will dig in about this a little more and post progress.!

ReKinStruct: Datasets & Further Plans

Posted on by

I have tried thinking of other datasets that can be obtained with the Kinect Snapshot program. A few interesting ones are a candle burning down, a fast growing plant, melting ice cream, etc. I have uploaded a PCD dataset of a person walking. I will be uploading similar datasets in the days to come.

Link to DatasetA: https://filelocker.discovery.wisc.edu/public_download?shareId=af0daed307f9c66123e3843360f328c8

If it asks for a password, it is ReKinStruct

Meanwhile, I have been thinking about next steps to do with the Kinect and one of the ideas that I have in mind is using two Kinects to obtain these datasets. This way, a lot of points would get filled for the scenario where it is full of shadows now. The concept image looks something like this.

Kinect Stereo

This way I would even get the dark side of the object. There are a few limitations to using two Kinects like interference of the speckle pattern. I am going to hide one Kinect from the other to avoid these effects. Working on setting up two Kinects and will upload some datasets this week.!

ReKinStruct : First Time Varying PCD

Posted on by

So, like I said in the last post, I had got the Kinect to obtain and view point cloud data but only manually. I updated the code a bit to do the above without intervention. Here is the link to the video of a time varying PCD that I obtained.

YouTube link : http://youtu.be/T4IPKq0rGII

Yes, that is me walking like a zombie. Note that I have loaded the points as PointXYZ on purpose to give a feel of the pointcloud. Loading the points as PointXYZRGBA feels like a picture and not a pointcloud.

The idea behind this is 8 PCD files named output1.pcd, output2.pcd and so on. The ReKinStruct_Snapshot.cpp code would take a snapshot once every second and save as binary PCD files with names as listed above. The Binary PCD files are constant in size (4801 KB).

The ReKinStruct_Viewer.cpp loads these files and displays them in sequence with a time delay of 1 second. It uses two pcl::PointXYZ pointers. One loads the output<even>.pcd files and the other loads the output<odd>.pcd files. So when the even-pointer is displaying the point cloud in the viewer, the odd-pointer loads the next .pcd file in the background and vice versa thus abstracting the user from the latency of loading the files.

And for some reason, Visual Studio 2010 didn’t let me use Sleep() or wait() routines. So I had to write my own API as follows.

#include <time.h>

void wait(unsigned int seconds)
{
     clock_t timeToWait = clock () + seconds * CLOCKS_PER_SEC ;
     while (clock() < timeToWait);
}

Next steps would be to obtain a faster scenario and progress through the 3D viewer faster, like real-time motion.

Will keep you posted.!