Author Archives: Naveen Anand

About Naveen Anand

I am currently a Masters student in the department of Electrical and Computer Engineering at University of Wisconsin-Madison. My concentrations are Computer Architecture and Operating Systems. I work with Dr.Kevin Ponto (https://blogs.discovery.wisc.edu/kponto/) in the Living Environments Lab, Wisconsin Institute for Discovery as a Project Assistant for the vizHOME project. Image processing and Virtual Reality have always been one of my major co-curricular interests since undergrad. This blog is going to be a chronological compilation of my independent study and future work with Dr.Ponto.

ReKinStruct: Switching Between PCDs

Posted on by

This week, I tried to switch visualising between two PCD files to check if it was feasible in run-time and to know how long it takes for the process. The following were the two PCD file visualisations I was trying to switch between: couch with the box and without the box.

With Box

Without Box

I tried doing this by two methods.

Method1) Load both the PCD files initially into cloud pointers. Display one and switch to another on the click of a button.

Method2) Load one PCD file and display it. On the click of a button, clear the current pointer, load the other PCD file and display it.

I am attaching videos to show how fast the process was.

Simultaneous Load: http://youtu.be/tFJUoOFaGcY

Sequential Load: http://youtu.be/77-9w1rnUyA

I have loaded the Point Cloud Data without the colours (PointXYZ as opposed to PointXYZRGBA) on purpose to get a feel of the point cloud. Also please note that the time taken to switch between the point clouds is the time from when I click ‘s’ to the time it prints ‘Changing point clouds’ on the console. ‘Changing’ would have made more sense if it was ‘changed’. My apologies.

The main observations were:

1) Switching between PCD files that have already been loaded into memory was faster than loading it from the disk.

2) Loading more PCD files into memory will require a lot of RAM space. There was a mild increase in memory used in the Task Manager during Method1 than Method2 window because only two files were loaded now. I suppose there might be a scenario where we would need to switch between ten or more PCD files that might end up using a large chunk of the main memory.

The moire pattern on the wall that was far from Kinect was due to poor resolution of the Kinect with respect to distance. The coloured pictures on the top show the same PCD files without the Moire pattern as the display windows are small and hence the reduced resolution.

This week I am going to try getting more PCD files from an interesting scenario and try switching between them automatically. I hope the video looks interesting. Will keep you posted.!

Note: An interesting find was that the pcd_viewer_release.exe always loaded my PCD files in a rotated axis. I had to almost rotate the point of view by 180 degrees on the Z-axis to view the data. However, the pcd visualiser class loads the data as how the PCD was recorded ( in our case, the snapshot point cloud data). In cases where we need to rotate the Point Cloud Data while opening, the pcl::transformPointCloud() could be of use.

ReKinStruct: Obtaining Kintinuous PCD

Posted on by

Last week, I focussed on fixing the Kinect and started obtaining Point Cloud Data. This week I have obtained a continuous PCD using the SCENECT software. The SCENECT software is fairly easy to use and complements for not having KinFu. It obtains data from the Kinect and forms a 3D PCD by registering the frames as we move the Kinect.

Scenect Scan

The window on the right shows the frame that is being currently read from the Kinect and the window on the left shows the registered PCD. The yellow and green points on the right show the registration checkpoints for the frames. In my opinion, it is fairly good for registration and colour values. However, scanning and registration take a bit of time. For example, as you can see from the small window on the extreme left, it took around 2600 frames to register this small point cloud data. I have not worked with KinFu so I do not have anything to compare against but all in all, I think it is a good GUI to obtain data. It also offers a lot of post-processing options which I will try to figure out this week.

Below is the final PCD obtained from the scanning. Scenect Final

However, SCENECT does not readily allow us to export the scan points as a .pcd file. The easiest way to go around this is to save it as a .xyz file and write a program that reads every line containing XYZRGB values and write a .pcd. There are two ways to do this:

1. Based on the tutorial from http://pointclouds.org/documentation/tutorials/writing_pcd.php. For this method, you need to know the number of points in the cloud beforehand. This essentially means that you go throughout the .xyz file twice: first to know the number of points to create the cloud of the necessary size; second to read in the values from the .xyz file

2. The easier and the simpler solution is to just create a cloud pointer as

pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud;

and push_back the points as and when read from the file.

( I do not know why pointclouds.org has not written the tutorial based on the second method )

On a parallel note, I also tried playing around with the OpenNI grabber to obtain PCDs. The salient difference in using this method is that it saves a .pcd of the one frame that is being read from the Kinect at that instant of time. Thus, there is no possibility of registering frames and making a huge point cloud. For a start, I read the frame as

OpenNI Viewerand the PCD file that was saved appears something like

PCD Viewer

That is the couch and the air conditioner nearby saved as a point cloud and viewed using the pcd_viewer_release.exe given by PCL. Though this method of obtaining PCDs doesn’t have a  potential advantage over the SCENECT, the OpenNI grabber PCDs can be used to obtain time varying PCD frames. That will be my goal for the upcoming week. Try to obtain time varying point clouds (like a candle melting) and switch through them. The dream would be to switch through ‘n’ number of point clouds fast enough that it will appear like a 3D movie.! Sounds cool, right?

ReKinStruct- PCD: Check

Posted on by

After a bit of an initial struggle to set the Kinect up with the laptop, I have finally obtained Point Cloud Data.

First PCD

Yes, it is Alienware. Yes, I got my hands on the beast.

So, the visualisation window on the screen shows the frame that is being read from Kinect. The program as of now keeps adding XYZRGBA points to the temp file and saves it as a .pcd file when I press the ‘s’ key and continues to obtain the next dataset. This is just a basic program that I used to test if the Kinect was working.

However, this also gave me the idea to obtain and save PCDs periodically as discussed in the earlier posts. I have not yet tried exploiting all possibilities with the existing code like how big a file can it write, how long can I keep it running to obtain data, how good is the Kinect when it joins frames etc. The plan for the upcoming week would be to try exploring these areas.

Meanwhile, Dr.Ponto suggested the ‘SCENECT’ software which uses a GUI to obtain and modify the frames obtained from the Kinect. That would be a good idea to try along with the existing PCL visualisation code.

Will keep you posted.

Over and out.!

ReKinStruct: Installing PCL, OpenNI and other Kinect Dependencies on Windows (Tutorial)

Posted on by

Since I couldn’t find a simple and direct tutorial on the internet that helps compiling PCL, OpenNI and its related dependencies on a Windows machine, here goes one. Before this begins, a few clarifications.

  • Why use a Windows OS? Because I have a Kinect-for-Windows. If you are having a Kinect 360 that would work on other Operating Systems, I would suggest trying that in Linux before switching to OSX or Windows.
  • I tried installing KinFu too but had to quit as there were a lot of path errors in the CMakeLists. So, if you want KinFu specifically, I am afraid this post would not help you much. You could try installing KinFu with some help from http://pointclouds.org/documentation/tutorials/compiling_pcl_windows.php which is the official documentation and has the kinfu app extensions (See section Downloading PCL Source Code). Best of Luck!
  • If you want to try Kinect Fusion(Microsoft’s version of the same), this post is so tally not going to help you. Installing Kinect Fusion essentially means cutting off all ties with PCL and its dependencies. So, again, Best of Luck!
  • Stick to one architecture for all installations. Since most computers these days have a 64-bit architecture, we will use the 64-bit versions for all installation packages. *If you are having a computer with a 32-bit architecture, I think it is high time you get a time machine. You have so got to travel in time.*
  • You need a good graphics card. I used an Alienware laptop with a NVIDIA GeForce GT 750M graphics card.

Step 1: Basics

Get Microsoft Visual Studio 2010 from www.dreamspark.com (if you are a student, you get it for free) or get it online. It has been one of my favourite IDEs and I hope you will find it useful too. The setup installs only a 32-bit Visual Studio package. You can change it to 64-bit Debug/Release mode by choosing Build->Configuration Manager->Active Solution Platform and changing it to x64 from Win32.

Step 2: Installing PCL

Installing PCL should be fairly straightforward. You can download the setup executable  from http://pointclouds.org/downloads/windows.html. Download the Windows
MSVC 2010 (64bit) All in One Installer. During installation, the setup will ask for the 3rd party dependencies it needs to install. Select Boost, Eigen, FLANN, Qhull, VTK. Uncheck OpenNI. We will install OpenNI in the next step from a different source. Redirect your directories in Visual Studio to point to the PCL files locations(PCL and every 3rd party dependence have their own bin, include and lib files).

Step 3: Installing Kinect Drivers

Okay, this is where it gets tricky. You need one (and only one) type of driver for the Kinect. Since we are going to stick to OpenNI, Do not try installing Microsoft Kinect SDK or KinFu.

Install OpenNI-Win64 from http://www.openni.org/wp-content/uploads/2013/11/

Install SensorKinect-Win64 from https://github.com/avin2/SensorKinect/downloads

Install NITE-Win64 from http://www.openni.org/wp-content/uploads/2013/10/

Try installing the latest versions of these Drivers. After installation, you should be able to see Primesense in your Device Manager with the Kinect Hardware as shown below.

Device Manager Primesense

If it does not appear so, it means the Drivers did not sync with your hardware. Try going to an older version of the Drivers. I have OpenNI 1.5.7.10, SensorKinect v5.1.2.1 and NITE 1.5.2 which are not the latest versions but these are the ones that work on my computer.

Step 4: Verification

Connect the Kinect to your laptop. Select Start->OpenNI 64-bit->Samples->NiViewer64. If step3 was successful you must now be able to see your Kinect reading in data (both depth and colour). I guess you can have a sigh of relief at this point.

Step 5: PCL program to obtain a PCD

Compile and run the example program from http://pointclouds.org/documentation/tutorials/openni_grabber.php in Visual Studio. Again, make sure the library files and linkers are directed properly in Visual Studio. When the program runs, you must be able to see a visualisation window in which you can see the input data from the Kinect and you can save the frame as a PCD when you click ‘s’.

There you go.!

I hope the tutorial was helpful. I know it is not as simple as installing it on Linux or OSX. Reminded me of this meme through out.

sudo sandwich

Image Courtesy: http://imgs.xkcd.com/comics/sandwich.png

However, I hope this post makes it easy now. Have fun. Happy Kinect-ing.!

For further details, email me at nsubramania2@wisc.edu. I will try to help as much as I can.

LEL Project Blog : https://blogs.discovery.wisc.edu/projects/

My Blog : https://blogs.discovery.wisc.edu/projects/author/nsubramaniam/

ReKinStruct : Apple to Alienware

Posted on by

I mentioned in my last post that I was having some trouble with getting KinFu running on my laptop. My Macbook Pro came with an Intel graphics card while KinFu (and most graphics intense applications) run only on machines with NVIDIA graphics card. Thankfully, Dr.Kevin Ponto has an Alienware laptop with an NVIDIA graphics card and is offering me. It should enable me to run KinFu without any trouble. I should be getting the laptop sometime soon and the first of many things to do would be set the IDE and PCL up on the laptop (Do ReKinStruct: First Look all over again). I hope to get a PCD by this weekend.

Meanwhile, Dr.Ponto and I have been discussing about other cool stuff we could do with Kinects. One potential idea is to obtain the color+depth images of a swiftly changing scenario (like a candle burning down or an ice melting) from a single vantage point. By obtaining data periodically, we can play through it making it look like a 3D movie. The same can also be done by playing PCD sequentially. However, changing PCD Visualization at the click of a button would require some high performance RAM and Solid State Disks and most of all, a new technique.

Will keep you posted.

Apologies for this unexpected delay. Pictures coming up soon.

ReKinStruct : The Hardware-Software Battle

Posted on by

So, I was trying to obtain PCDs from the Kinect and in order to do so, I wanted to read something from the Kinect. Like I mentioned in the previous post, I had PCL and MSVC2010 installed. I had also obtained the latest version of OpenNI and SensorKinect to serve as drivers for the Kinect. And the following happened.

I had one line of code that initialises the Kinect and returns a pointer to grab data. It kept crashing at run-time and there was no stack trace. A little bit of googling led me to believe that OpenNI does not comply with my laptop’s underlying hardware.

Okay, *deep breaths* that is not a dead-end. There had to be something else. So I got Kinect Fusion installed from Microsoft site. And from the tutorials, all I needed to do was type one command line code and my Kinect would (magically?) start obtaining a PCD. Although what happened was a different story.

There was something wrong with my Graphics card. I tried installing the latest CUDA drivers from NVIDIA but that did not help too. I have an Intel 4000HD Graphics card on my laptop and I wonder if KinFu works only with NVIDIA Graphics cards which would mean I would have to start using another machine.

So, the plan for this week is to check if the Kinect works on another machine and start proceeding from there. I also have a bad ( and a good ) feeling that I might have missed installing something on my laptop and so I can get the Kinect working on my machine too. Will keep you posted.

All in all, the classic conclusion, Hardware and Software do not like each other.

ReKinStruct – PCL : Check

Posted on by

So, the aim was to get PCDs from Kinect as soon as possible so I can start working with the point cloud data. On the way, I stumbled into getting the hang of how Kinect works, some basic programs that came along with the Kinect like obtaining a color+depth image, skeletal viewer etc.

Apologies if it looks distorted, had to both be in the image and take a screenshot (And taking a screenshot on Windows with a Mac keyboard requires clicking keys on all corners).

There were few more installations before even getting the Kinect running. Point Cloud Library with its 3rd party dependencies and OpenNI framework are installed and running. Compiled a simple program that reads data from Kinect and displays it on the screen. Stumbling into runtime errors and code halts. Memory usage is already up to 3GB which is three-fourths of the total. Let’s hope it doesn’t crash any soon.

Essentially, would like to get something like this in the next few weeks.

Image Courtesy: pointclouds.org

The target for this week is to get the program working on the IDE and start reading some data from the Kinect and store them as PCDs.

ReKinStruct – Abstract & First Look

Posted on by

Kinects are generally used to obtain depth maps of an environment using their speckle pattern reflection of infrared light and a color pattern similar to a video camera. This data is used to analyze the position and movement of human body thus giving a virtual reality gaming environment. Contrary to the above traditional use of Kinects, they can be used to obtain 3D Point Clouds similar to the usage of a LiDaR scanner. The striking difference between the Kinect and LiDaR scanning would be that a Kinect could be moved in any direction and still continues to obtain the Point Cloud Data while the LiDaR scanner has to be kept stationary during its operation. The Point Cloud Data obtained can be simultaneously or subsequently processed to enhance, reconstruct missing points and texturize the data obtained and can also serve as a map for SLAM (Simultaneous Localization and Mapping). The Point Cloud Library(PCL) and C++ are to be used almost for the entire process. OpenCV, OpenNI are the next likely frameworks.

And to get this project kicked off, I got Windows, Visual Studio and Kinect SDK installed on my Mac (It took much longer than I thought). With a little help from the Kinect SDK apps, I was able to obtain the color and depth maps of my laptop inside my laptop inside my laptop inside. . .

Voila.!

From here, the next step would be obtain a continuous point cloud from the Kinect which would get us something like a 3D image. And from there, it all looks exciting.

Naveen