The motivation for this project was two-fold. We wanted to make a realistic enough virtual surgery simulator that people could practice with enough realism to help them in real surgery. Practicing now involves cadavers or real-live patients. It would be a lot cheaper and less dangerous to practice in real life. Easy mistakes can be avoided with repetition. For the second part of the project, we also wanted a way for the phone and Vive to interact in 3d space. This could allow Google Cardboard to interact in Vive apps. It’s a nice, cheap alternative to a Vive HMD, and lets friends or other people experience Vive apps without the cost of adding an additional headset and computer.
Josh – Josh did the networking between the Vive PC and the Android phone. See more about that in the “Problems Encountered” section. Josh also got the Vuforia tracking working. Josh also took care of making everything else on the Android app work. He had to make a subset of the room and change some shaders and stuff to keep the framerate. Haixiang helped with the simulator integration.
Dustin – Dustin made the surgery room and some of the instruments used in the simulation. He also helped with object interactions and movement. Using the VR toolkit was a big help.
Haixiang – Haixiang borrowed the simulation code from his lab mate. The code is well written and has a very clear interface. The simulation is computationally intensive, therefore we decided to run the simulation in a remote server and communicate with unity through networking. The major contribution is the networking protocol between the simulation server and unity. The secondary contribution of my part is the management of the interaction between the simulation mesh and the VR objects.
Our project is a simulation of a Z-plasty surgery. You can lift and separate the skin on the head of our patient as well as suture it back together. There’s a bunch of tools that accomplish these purposes. The surgery is done in VR, and it can be viewed from an Android phone.
We met our original description and goals with the exception of the cutting of the skin. We’re very happy with the results and this is a project we would enjoy perfecting if we had the time.
Problems Encountered/Design Decisions:
A little bit about how it works: basically a remote server does all the simulation of the skin. This server had to be remote because the simulation requires quite a bit of CPU power to run. The server has 32 cores, and 256 GB of RAM (the CPUs were the important part).
The simulator takes a position input in the object space, and barycentric coordinate and triangle ID as the input of the constraints to the simulator. The first challenge is to get those informations from unity. We used the transform.inverse_point/inverse_vector functions for the position conversion. And luckily the ray-casting in unity returns the barycentric coordinate and triangle ID for a mesh collider. So as long as Unity and the simulation engine has the same triangle orders, we can directly use the Unity functions.
The second challenge is that the simulation updates the mesh at about 2-3 frames/second, while VR require to be updated at 90 fps. Therefore a separate thread is need for requesting new mesh positions from the simulation server to keep the frame rate. But unity functions can only be called from the main thread, we create a synchronization protocol between the two threads for informing unity to update the skin mesh, recompute its normal(for rendering) and update its collision mesh(for ray casing) when a new mesh position is received.
Last, the simulation runs on a machine under the cs.wisc.edu domain, took us a while to figuring out that there is a vpn can connect off cs machines to it.
As for the Android app, the app uses Vuforia to put the surgery subject’s head into Augmented Reality. Out of the box, Vuforia didn’t work too well. Some settings had to be tweaked to get it running smoothly on a not-so-new phone (Nexus 5 from 2013). The two settings that seemed to affect performance the most were the Camera Device Mode. This should be set to CAMERA_MODE_FAST. This setting changes if the phone goes for accuracy or speed while running Vuforia. The fast mode has the occasional twitch, but the twitches are quickly corrected. Also, the fast mode is quite a bit faster, so the framerate looks a lot nicer.
[This section is an edited version from another project..]
The process of design a tag is not for the feint of heart. The biggest barrier to designing the tag was using Adobe Illustrator. The program is designed for artists and graphic designers to use rather than engineers. There are a number of parts that each VuMark contains.
The algorithm Vuforia uses tracks enclosed shapes containing 5-20 sides. The shape edges are represented by areas of high contrast difference. The tags also require a ‘clear space’ on the inside of the shape. The shape can be rotationally symmetric, but when tags are used for camera pose tracking, they are required to not be rotationally symmetric.
The tag also has preset locations for white/dark markers used for information encoding. In my tag there are 31 markers which encode 127 unique values. On the image of the marker there are actually 32 markers, but one (the top, center one) is only used to make the marker pattern look better. The unique values didn’t really do anything here, but we thought we might put different stations or angles on different tags. Never got around to that. The markers we used are black and white, but they are only required to have high contrast as viewed in black and white. While designing the tag, there is lot of freedom to use different colors and shapes so developers can design an appealing tag that fits their purposes. The tag we used is shown above, and some example tags are also shown here to show the freedom of the tag design.
For the simulation stuff, we used almost the same code on the PC and Android to stream the skin meshes. We took out the initialization and everything worked fine.
For the syncing between the Vive and phone, we setup a relay on the server to relay packets from the Vive to the phone (both are connected anyway for the mesh streaming). We tried to use Unity’s networking to do this, but it was a BAD IDEA. Basically, there isn’t a very good way to use Unity’s built-in networking while using two different scenes. We ended up writing our own networking functions using TCP connections. This worked a lot better. To keep things simple, we also only streamed the tool selection, Vive HMD, and Vive controller positions and rotations (there was a small bug where the hook would sometimes be rotated 90 degrees… but it worked really well besides that).
To sync the Vive and the Phone in real-life coordinates, we just put the Vive controller on top of the Vuforia tag and used a button push to trigger the change. This then moved the head to the position of the controller. Then we just had to make sure the tag was lined up with the vive simulation (since we didn’t do any rotations). This worked pretty well. It was a little off because it was hard to get the scaling right for the Vuforia stuff. To set this once and for all, we just put the vive controllers on either side of the head, and kept scaling the Android stuff up until we got the controllers in the same spot on both images. There’s probably a more deterministic way to do this, but it worked as a quick fix. We also didn’t have time to find models for the HMD and Controllers the Vive uses, so they were replaced by a block and a squashed sphere.
It would be nice to have some interaction you could do on the phone. We wanted to make it so the phone could choose to load different types of surgeries into the simulator. Then the instructor could test a bunch of different ones right there. We could have instructions for each one displayed on the Vive, so it’s easier for the students to remember what to do.
We also wish we could have added real-time cutting. We wanted to, but those types of algorithms are complicated, and there wasn’t time. It would have been nice to have the cutting in there. Then we wouldn’t have even had to pre-load the models!
Another thing is currently the mesh streaming and constraint change(hook/suture) are running in the same thread on the server. The unity client has to keep sending frame update request to the server and wait the server sending the updated mesh. Ideally we should use one port for constraint change (upstream from Unity to simulator) and another used for pushing mesh to Unity(downstream from simulator to Unity).
Here’s the video. Sorry for the weird blurriness on the phone part.. We only had a camera with manual focus, and Josh had to hold the phone and camera at the same time! Also, the phone died part way through, but we got everything demoed before it died!
Here is the link to the video. It is 70 MB so we can’t upload it directly to the webpage.