TypeSafe is a hand-worn device designed to improve the wearer’s computer usage habits by discouraging uncomfortable wrist postures and encouraging  regular periods of rest during prolonged keyboard typing.  The idea is to help the wearer avoid long-term health problems that can result from excessive and unhealthy computer usage, such as Carpal tunnel syndrome.

The device consists of a finger-worn inertial measurement unit (IMU) and a wristband containing an Arduino microcontroller, vibrating component, and battery. The IMU tracks the orientation of the wearer’s hand. The Arduino program analyzes the orientation data and infers the wearer’s hand posture and typing activity. When the hand posture is inferred to be”bad” (uncomfortable), the device alerts the wearer with vibration. Moreover, the device keeps track of how long the wearer has been typing and produces vibration to let the wearer know when they should take a break.

Both hand posture and activity inference are accomplished by performing analysis of the orientation signal. Current posture is classified as bad when orientation significantly departs from the neutral (healthy) hand posture, while typing activity is inferred from high-frequency content of the orientation signal.

The device currently exists as a fairly crude prototype. Although most of the planned features of the device are supported in the prototype, their implementation is still very basic and lacking in robustness. The original project plan foresaw several iterations on the prototype, during which I would have performed more principled collection and analysis of IMU data during everyday computer usage and development of more sophisticated inference machinery for postures and activities. However, these steps never occurred due to lack of time.

A technical hurdle which proved quite costly in terms of time was getting the IMU to work with the chosen microcontroller (Arduino LilyPad) and obtaining reliable orientation data.  The solutions to this hurdle were relatively simple – changing the configuration of the microcontroller’s SDA and SCL pins, and making sure connections between the microcontroller and IMU were properly soldered. However, diagnosing and fixing these issues took long enough that less than two weeks remained for other steps on the project – data analysis, development of inference models, and user testing.

Given more time, I would make several improvements to the device:

– Use better sensors for posture tracking. IMU seems to be quite prone to drift, which makes it difficult to reliably track absolute posture. Flex sensors (https://www.sparkfun.com/products/10264) may be a better – and cheaper – alternative for this purpose.

– Introduce additional sensors for tracking postures of the elbows, shoulders, and the back, as these are also prone to repetitive strain during computer usage.

– Collect and analyze sensor data during real computer usage and develop trained models for posture and activity classification. This poses an interesting challenge, because these models must be fast and compact enough to run on a microcontroller with limited memory capacity and computing power.

This week’s activities:

– Got a nice sweatband and used it to build the first, rough prototype of my device. This involved quite a  bit of stitching, soldering, connecting things with wires, and taping them to other things. Here is what the prototype looks like:

– I’ve also written a good chunk of the Arduino code, though the meat – activity and posture inference – is still missing.

Problems encountered:

– My first attempt at prototype didn’t work at all. IMU was unreliable, due to wiring issues, and I also had a few shorts due to stitching issues (stitching  on this wristband is not as easy as on a rectangular piece of fabric).

Successes:

– All of the hardware problems that plagued me before seem to have been resolved.

I’m behind my original schedule, but reasonably confident that I’ll have a working prototype in time for the showcase. It just won’t be as full-featured or as pretty as what I originally envisioned.

Plan for next week:

– Implement the activity and posture inference in software.

– Demonstrate a working prototype of the device.

– Make a poster.

What I have been up to this week:

1) Got the IMU to work with my LilyPad, was able to get good Euler angle readings. Now we’re cookin’ with gas!

2) I started putting together a physical prototype of a wristband and IMU ring. You can see a physical mockup in the picture. Instead of using a flimsy piece of fabric for the wristband, tomorrow I plan to buy a sweatband and stitch my board and other components into it.

3) I’ve been experimenting with a neat little program called Gobetwino, which offers several useful features for communicating between an Arduino and a PC. I plan to use it for writing sensor data to a spreadsheet and analyzing the output, in order to derive activity and posture classification heuristics.

Current concerns:

1) I’m worried about wiring on the IMU. Specifically, I’m worried that connections might be too flimsy, which could cause the IMU to freeze periodically. I can’t think of a reliable solution, short of soldering the wires to the IMU, which I don’t want to do this early in development.

2) I’m also worried about wires being too thick and rigid, which could constrain the hand movement such that we get badly skewed sensor data for typing motion and posture. Anyone got any nicer, more flexible wires?

FInally got to do a bit of work this week.

1) I managed to get the Flora board working on my lab computer, though it is still quite unreliable and needs to be reset all the damn time.

2) To establish I2C communication with my IMU, I had to first make a small modification to the FreeSixIMU library. Apparently in order to get sensor readings from the accelerometer, it is necessary to disable internal pullup resistors on the SDA and SCL pins. FreeSixIMU library did not do this properly for 32u4, which is what Flora uses.

3) Good news end here: data I’m getting from the IMU seems to be junk. While initial readings from the IMU are not always the same, they don’t change at all as I move the IMU around. Also, the device seems to freeze frequently. Assuming the IMU isn’t defective, I suspect wiring issues, some problem with my modifications to FreeSixIMU, or maybe even an issue with serial communication (possibly caused by Flora driver issues.)

My next steps:

1) I plan to switch back to using the LilyPad for my device. Upon further inspection, I have determined there is no reason whatsoever why either LilyPad and LilyPad Simple should lack support for I2C. Moreover, LilyPad doesn’t use the 32u4 chip and therefore doesn’t suffer from Flora’s annoying driver issues. I suspect the reason why I couldn’t get the IMU working on the LilyPad previously was because of the internal pullup resistor thing, and possibly wiring issues (the IMU is very sensitive to those apparently.)

2) Once I get the IMU working properly with the LilyPad, I will be able to start getting sensor data and implementing my prototype device.

This week involved a lot of futile effort to get some data out of my IMU. I’m using a library called FreeSixIMU to get 6DOF tracking data.  The library in turn uses Wire library to communicate with the hardware using I2C. However, when trying to do this on my LilyPad simple board, Wire library fails to communicate with the IMU, possibly because the board does not support the SCL and SDA pins needed for I2C. I then tried to achieve the same on a second microcontroller, Adafruit Flora (which exposes the required pins). I barely managed to get Flora working on my machine (barely – Arduino likes to freeze a *lot* when using it), but still had no luck getting any kind of meaningful response from the IMU.

I plan to take the following steps to resolve or work around these issues:

1) Do a bit more testing with the Flora – I need to check whether I can get a simple example working on it, just to ensure that basic features like serial communication work properly.

2) Try the IMU on somebody else’s board/machine to make sure the thing isn’t defective.

3) I’m planning to reformat/reinstall the OS on my laptop, for unrelated reasons.  If the problem is on the software side (quite possible, given all the trouble I had installing Flora), maybe it’ll go away.

I don’t have any tangible results to show for this first week, so I’ll instead post a picture of a cat that looks like Hitler:

He’s actually a very sweet cat and not racist at all. I met him on the grounds of Dolmabahce palace in Besiktas, Istanbul. There are many stray cats like him around here and they are all very docile. However, he’s the first one I found that resembles a genocidal dictator. I call him “Fluffy the Führer.”