Rosetta Fist was my undergraduate senior design project, working alongside Justin Furiga and Frank Hiller. Attempting to create something interesting and novel, we decided on an interactive American Sign Language (ASL) tutoring system.

Rosetta Fist made use of the infrared camera in a Nintendo Wiimote. We constructed an infrared LED array, which projected infrared light onto the user’s hand. The light bounced off of reflective points on the user’s glove and was captured by the camera. The Wiimote transmits the spatial positions of the reflected light to WiiMoteLib via bluetooth. WiiMoteLib provides the individual point locations to Rosetta Fist, which attempts to interpret them as a letter pattern.

Rosetta Fist was written in C#, due to our familiarity with the language and because WiiMoteLib is also written in C#. The individual letter patterns were stored in XML. We included two learning modes: a “lesson mode” which gives the user a series of letters to sign in order, and a “free form” mode which allows a user to choose any letter to practice at his/her own pace. The lesson mode also allows for users to create their own lessons.

We performed two usability assessments for Rosetta Fist. The first was in-class, and covered topics such as simplicity, satisfaction with the system, error handling, and organization. On a scale of 1-7, Rosetta Fist scored an average of 5.53 across all questions and all users. Our second usability test was performed at Gannon’s Deaf Awareness Event, which provided both a wider demographic user base as well as introducing expert users to the system. The users were encouraged to be very critical, and the questions this time focused on the content of the system. In general, the users were quite pleased with the difficulty of the lessons, however some felt that the patterns were too restrictive.