Student Project Forwards Future of Hand-Gesture Controlled Apps

Forget your computer mouse and touch screen. Soon you’ll control applications with a wave of your finger or hand.

Harvey Mudd College seniors Tiffany Lim, Jeb Brooks and Jasper (Zheng) Duan are part of a Global Clinic team working with Intel, Mozilla and the National University of Singapore (NUS) to develop a comprehensive software library needed to support advanced computer vision applications for web browsers.

The applications will analyze streaming video images captured by a webcam, interpret them and then translate the images into actions. “That might involve being able to recognize a hand is there in the first place, plus how many fingers there are, their orientation and their direction of movement,” said computer science Professor Bob Keller, faculty advisor for the Intel Global Clinic team. “And it needs to do that in real time, as you move your hand, so it has to be reasonably high performance.”

Sponsored by Intel, the Global Clinic project revolves around Intel’s new high-performance multi-core processors, codenamed “Haswell.” Mozilla is developing a parallel version of JavaScript that will drive Haswell, allocating processes simultaneously across the cores for improved performance. The three Harvey Mudd seniors are collaborating with the NUS student team to develop the library based on the Parallel JavaScript language. The culmination of the Global Clinic team’s work will include a demonstration of the library in action.

Coordinating the two student teams has stretched the students’ skills as much as the computer science aspects of their work. “It takes a lot of flexibility to juggle different people with different needs and skills while also making sure the project moves forward in the best way possible,” said Lim, project manager for the Harvey Mudd team.

To address the different time zones and tasks, the team has relied on weekly conference calls, emails and an online wiki to stay connected and apprised of each other’s challenges, insights and progress. They’ve also traveled to each other’s schools—the NUS team came to Harvey Mudd last fall and the Harvey Mudd team traveled to Singapore over winter break.

For most of the team members, the experience is their first exposure to working with a real-world client and their first time working with an international team.  “Working with an international team has been fun and eye-opening,” Lim said. “Our teams get along brilliantly and we have had the opportunity to pick [the NUS team’s] brains about their computer science experiences, educational systems and standards and their culture in general. It’s been a valuable experience for us all to see how similar and different students outside our own countries can be.”

Modeled after the College’s renowned domestic Clinic Program, Global Clinic provides long-term, sponsored engineering and science projects in which teams of Harvey Mudd students work with student teams from partnering schools in Singapore, India, Israel, Japan and Iceland with plans to expand into other countries. Company sponsors and a generous, $1-million endowment from the Robert and Joan Vickery family support the program.

Intel and NUS have been key partners in the Harvey Mudd Global Clinic program, which launched in 2005. The National University of Singapore has served as an academic partner since 2007. Intel has sponsored 10 projects (eight Clinics and two Global Clinics) since 2011, including a Global Clinic project in Israel.

This and 41 other projects will be presented May 6 during Projects Day, a special public event when the Harvey Mudd community gathers to support the Clinic teams as they unveil their project findings.