My final project integrates both the leap motion and the zSpace into the world builder scene. The zSpace is used for displaying the world builder scene and it also is able to draw objects with the stylus. The Leap is used to rotate and zoom in and out of the object. The combination of these two controllers allows the user to more efficiently interact with the object that is drawn.

Looking through my old posts was encouraging because it showed the progress that was made through out the semester. Overall I am happy with the results of my project and have learned a lot this semester.

One of the largest obstacles that I had was getting the tracking to work correctly with the Z Space stylus. The first route I went was to programmatically fix this problem. I needed to find a way to calculate the rotation of the stylus so that it would update correctly on each frame. I turned to quaternions and soon found out that this approach was too complicated and would take some time to complete. Luckily, VRPN supports zSpace so this fixed the problem with the stylus tracking. The next problem was to find out why the stylus was not drawing on the screen. This was fixed by trial and error within the finona config file.

Another obstacle that I faced was to have the world builder scene run independently from the skeletal viewer. The reason for this was because the skeletal viewer needed to start before the world builder scene and running these two applications every time an edit was made took some time. The Leap Motion was also dependent upon the Skeletal Viewer which was annoying because I had to select the skeletal viewer before I would see my hands on the screen. To fix this problem I moved all of the code for the leap motion into the world builder scene. This had to be done carefully because much of the code is scattered across the code base and it caused several files to be edited in order for the conversion to work.

My most recent obstacle was getting the rotation set up for the leap motion. My goal was to get the leap motion to detect the angle of my hand and to rotate the world accordingly. This required some knowledge on how the FinonaLib orients the camera when using the wand. Basically I spoofed this idea and made it work with the leap motion. Currently you activate gestures by placing 5 fingers above the leap motion. Next you can either close your hand and pull in or out to simulate zooming, use one finger to rotate around the x-axis, and use two fingers to rotate around the y-axis. You can stop all of these gestures by placing fingers above the leap motion again.

If I had more time I would make a gesture to reset the camera to its original position. I believe this would help when the user makes a mistake when rotating the object. I would also like to be able to change the colors of the 3D object either by voice or by some type of gesture.

As the semester is coming to an end I would like to try to accomplish getting the world builder scene to rotate. From my last video, I really like how the scene can be adjusted by using the leap motion to draw the scene up, down, into, and out of the screen.

Next I would like to be able to rotate the scene along the x axis. So when the hand is closed we would save the position of the hand in the (x,y,z) plane and then rotate about the x axis so that you could see the backside of the figure that you are drawing.

I am hoping to do this action is one finger. When the rotation would be based up the change of x and the finger moves left to right. To accomplish this I will be using the GLrotatef(angle, x,y,z) function where x = change of x and the y and z are both constant. I am having trouble finding a way to calculate the angle for this function. I believe I will need to find the angle between the start point when he hand grabs the world and the ending point when the hand releases the world.

I will be making another video to demonstrate this effect tomorrow.

As a starting point for gestures I am now able to use the transition from an open hand to a close hand to move the camera position within the world builder scene. Heres a quick video demonstrating this:

For next week I would like to find out how I can grab the object and rotate it.

This week I will be working on implementing hand gestures in the world builder application.

The gesture that I would like to work with would allow the user to select and hold and object. The first gesture would be putting your hand near the object, like this:

The second gesture would be to close the hand into a fist:

During this gesture I would use the palmPostion() and palmVelocity() functions to both determine the position of the object and palm and to also rotate the object given its change in its velocity vector.

Sorry for the late update this week. Exams and projects have preoccupied my time primarily.

Last week I was able to get the skeletal viewer code for the leap motion put into the worldbuilder application. This actually was a pretty painless process except now I the leap motion is not drawing the hand. I’ll be working on getting this problem resolved hopefully today or this weekend.

After that bug is fixed, I will work on getting gesture movements implemented like how I explained in my last post.

Since I dont have a picture or video to provide this week heres a link of some cool leap motions apps that just came out recently:

http://blog.leapmotion.com/post/81491756746/vimeo-at-your-fingertips-4-more-new-apps

Update: Right now it seems that the leap motion and the Zspace dont get along. When I unplug the zspace the leap motion goes into the onFrame function (a function that is called every frame). Currently working on resolving this strange bug.

Update: Its working! Just had to plug the usb for the zspace into a different port. now the worldbuilder with leap motion integration are completely independent from the skeletalviewer!

The video posted below demonstrates what how the leap motion looks in the world builder application. My next step will to be to start adding gestures to manipulate the object that the user makes. I expect it to be a challenge to have the leap motion correctly detect all five fingers, so gestures will need to use 1-3 fingers. (Video to come in the next hour, free Vimeo accounts make you wait to “convert” the file).

The first gesture I want to implement is this:

And heres the documentation: https://developer.leapmotion.com/documentation/python/api/Leap.CircleGesture.html

I think this would be one of the easier gestures to test because it has a starting point with one finger and an ending point with one finger. By just using one finger it will be easier to test implement as opposed to using multiple fingers for a gesture.

I will be updating this post through out the week with my progress.

And heres a cool video I found on dynamic shape display just for fun: https://www.youtube.com/watch?v=5EkkTV51Pg0

Last week was quite busy for me, but I did begin sketching out ideas for the UI with when everything is attached to the world builder (stylus & leap motion).

I wanted to keep things simple so instead of using two hands with the leap motion, I believe using one hand to rotate (left hand) and one hand to draw/color (right hand) makes the most sense. The left hand would be used to rotate the object on the xyz plane. The right hand, which holds the stylus, is used to pick colors and draw shapes.

Something I see potentialy being a problem is implementing the select object function. The Leap Motion is pretty insensitive and does not always map the fingers properly. I believe a two finger pinching motion (similar to zoom function on smart phones) and then rotating the wrist which subsequently rotates the object would be the best approach for this function.

Edit 4/3/13

Was able to get world builder to detect the leap motion so I am making progress in that area. In order to get the leap motion to detect, I had to use skeletal viewer once again. Now world builder seems to halt all of its functionality (drawing, head tracking, etc) but a valid hand is being tracked by the leap. This will be the next problem to tackle.

Edit 4/6/13

Hands are now showing in world builder from the leap motion. Currently the leap motion will only show hands when the skeletalViewer is selected. This week I will be working on adding the leap motion functions into the world builder application so that it is not dependent upon the skeletalViewer.

Over break, Kevin & Ross helped me out tremendously by getting the stylus to now sculpt a model in the world builder application.

I believe the IPD may need to be slightly adjusted so that you cant see two images. This should be an easy fix, though.

My next steps are now to use the leap motion to grab and rotate the object on screen.

I will add more to this post this week as I work into integrating the leap motion into the world builder application.

Last week I was focused on making a function that would allow track the orientation of the stylus with position matrix. But while digging through the ZSpace SDK I noticed that VRPN supports ZSpace. This is really good news because now the stylus should work right away and we don’t need to worry about implementing a function that calculations the orientation of the wand. This week I will be attempting to tie all of the loose ends together and get the stylus fully functioning in the worldbuilder application.

A problem that I’ve come across is within the config file for Fiona. Since the Zspace is tilted at ~45 degree angle from the table, I need to calcuate where the x,y,z points are with respect to the screen and the angle in which it sits. The problems lies with in the config file, the format is: wall x y z width_x width_y width_z height_x height_y height_zThe wall is referring to the screen and how it is position. But when I run this I get nothing but a blank screen. It could be that my calculations are off (hopefully) or otherwise it will take some more digging to find the reason for this problem.

I don’t have much for pictures this week considering most of the tests are mostly code driven and don’t show much for outputs. But the image below represents x,y,z axis’ of a titled ZSpace screen.

For this week I would like to focus on a couple of things. The first being getting the rotation fixed with the stylus. I found some c++ code for converting a rotational matrix into a quaternion. A sample of the code can be found here: http://www.euclideanspace.com/maths/geometry/rotations/conversions/matrixToQuaternion/ . This should solve the problem as to why the cursor is being display slightly to the right of the stylus.

After this is complete, I would like to start mapping the keys from the stylus into the world builder application. This is done in a similar fashion as to how the stylus was integrated. After that, I would like to make this application self reliant meaning it would no longer need to use skeletal viewer to function.

A Video of the stylus tracking is to come in 40 minutes once Vimeo converts my clip. The output is showing the position of the stylus relative to the Zspace screen.

Also, just found a neat little link on vrpn integration. This could be helpful.
https://support.zspace.com/entries/23780202-How-to-Setup-zSpace-VRPN