About five years ago, electrical engineers at Cornell and the University of Washington (UW) began developing software for video compression to enable hearing-and speech-impaired individuals to converse by smart phones using American Sign Language (ASL). The resulting phone app, MobileASL, was recently put to the test when 11 students in a UW summer program for the deaf and hard-of-hearing were issued phones and were urged to use them. Early results of the field testing have been quite favorable. UW professor Eve Riskin, principal investigator of the project, told us that she and other members of the research team hope the new software will be widely available within a year or two.
It may be tempting for most of us to assume that those in the deaf community would find text messages and email to be adequate for mobile communications. But in fact, they are just as limiting as they are for the hearing world. The goal for the research team was to create a sign-language-capable phone that would enable hearing impaired individuals to engage in real-time communications with friends and family in their native language, with full expression of emotion – including the underlying subtleties of facial expression. A MobileASL phone could also be used to easily connect with an existing video relay service (VRS). A VRS provides translators to videophone users and is offered by telecommunications carriers to serve the needs of the deaf, hard of hearing, and speech-impaired when they want to phone a person who does not know sign language. VRS and TDD/TTY services (teletype, or modem-based, communications) are all supported by a small portion of the federal, FCC-imposed taxes on telecommunications services.
The MobileASL would allow those in the deaf community to converse in the easiest way as well as the way they find most natural: using sign language. In a news account in the IEEE Spectrum, UW professor Richard Ladner, whose parents were deaf, made his point by saying, "Today's world is more connected by cell phones than by any other device." In short, the researchers regard the sign-language-capable phone as a quest for equal access to the best mobile communication technology.
In fact, one successful emergency use of MobileASL during the field test is documented in a UW News account: one participant in the study got lost on the wrong Seattle city bus. By phoning one of the project's teaching assistants, the student was able to describe, in real time, what he was seeing as the bus traveled its route, and was soon "unlost" and assisted. The fastest thumb-typing and texting would not likely have been able to solve that dilemma as adroitly as the student's sign language conversation in real time. While it may have been stressful at the time, the bus incident offers quite an apt demonstration of the great potential of MobileASL for hearing– and speech-impaired users.
In developing MobileASL, the research team – two electrical engineering professors a computer science professor, and a professor in the UW Information School (and numerous graduate and undergraduate assistants) – overcame several obstacles inherent with conventional smart phone video conferencing technology. In particular, Cornell professor Sheila Hemani addressed the problem of sufficient image clarity that would allow a person to see the other person signing well enough for comprehension. She succeeded by introducing an ability to distinguish which areas of the video image are the signer's hands and face in the compression software. As a result of this metric, the hands and face remain in high resolution while the signer's torso and background are transmitted in low resolution.
Another problematic aspect of image clarity was frame speed – typically just 10 frames per second for standard video compression and transmission. Professor Riskin and her lab team at UW in Seattle found they could nearly quadruple the speed of the standard compression process by sampling only one-fourth of the pixels in each frame. On the receiving end, the interpolation feature of the Microsoft Windows Mobile operating system expands the video to full screen size without significant loss of image quality.
For now, the MobileASL succeeds in using the relatively limited bandwidth of U.S. wireless networks, as its encoders are compatible with the H.264/AVC compression standard. This is important: the UW press office notes that some new high-end phones that offer video conferencing (such as the iPhone4 and HTC Evo) are regarded as bandwidth hogs and are being blocked or restricted to higher-priced data plans.
One other crucial problem the engineers tackled was preserving battery life, considering the power-draining video compression and decompression functionality inherent in using a MobileASL phone. The engineering team accomplished this by innovative programming that recognizes whether the user is actively signing or watching, and increases and decreases the frame compression and transmission rate accordingly.
With early results looking so promising, many are anxious for the broader introduction of this innovative technology to the deaf community. The MobileASL research was funded primarily by the National Science Foundation but was supplemented by grants from Sprint Nextel Corporation, Microsoft Corporation, and Sorenson Communications. While Sprint Nextel and Microsoft hardly need an introduction, Sorenson Communications is the leading manufacturer of videophones customized for the deaf and hard of hearing and is also a leading video relay service provider. The company is based in Salt Lake City, Utah.
Photo courtesy of University of Washington.
