Have you ever had a question while shopping, ordering dinner, banking, or visiting the doctor but you couldn’t find someone to help you – or a way to help yourself – in your own native language?
Unfortunately, situations like this are a fact of life for more than 430 million deaf and hearing-impaired individuals around the globe who struggle with communications accessibility and language barriers in public spaces every day.
In Brazil, where more than 2.3 million people are profoundly hearing impaired, two distinctly different sign languages are in use today, on top of numerous regional dialects. This challenge adds to the hurdles many Brazilians face when needing to communicate with others using either the official Brazilian Sign Language (Libras) or Portuguese Sign Language (Língua Gestual Portuguesa or LGP).
During an internal team discussion at Lenovo in 2019, a software developer who is fluent in Libras and serves as a volunteer interpreter in her spare time, pointed out many of these accessibility issues and challenged Lenovo to do more to improve independence and quality of life for the deaf community.
“She said, ‘Lenovo promises smarter technology for all, but how can we include deaf people as well? All means all,’” recalls Hildebrando Lima, Lenovo’s R&D Director in Brazil and that statement from the developer caught his attention.
Inspired by this question, Lenovo began thinking about developing a solution: a real-time Portuguese to Brazilian Sign Language translation chat tool that lets hearing-impaired people sign to a device’s camera while an algorithm performs the simultaneous translation to another person, on the other end, in Portuguese text.
However, achieving the ideal vision of real-time video capture and simultaneous translation between languages is no small feat due to the sheer number of variables involved – not the least of which are the individual gestures for each word and the syntax of each sentence.
Understanding this, Lenovo turned to CESAR’s A.I. team to begin studying the problem, capturing data, and laying the foundation for future product development.
“There are so many hurdles involved with the video capture alone – including the person’s skin color, background color, lighting, clothing, the speed of a signer’s gestures, and hand positions relative to the body – to name just a few. On top of that, not every camera has the same level of depth perception. We knew that CESAR had a formidable A.I. team with the expertise and processing power we needed to capture and analyze these images in real time. That was the start of our journey together almost five years ago,” said Lima.
Since then, Lenovo and CESAR have created a dataset of thousands of Brazilian Sign Language videos and developed new, patent-pending technology to teach A.I. to visually identify and contextualize individual gestures. The technology can also be harnessed to teach hearing people sign languages by showing them how to make gestures using computational imaging to track the accuracy of those gestures against the custom-built database being constructed.
Rather than attempting to translate sign by sign, the A.I. and database being developed are learning
to recognize hand positions by extrapolating data from the contours of the hands and, most importantly, the digital articulation points of the signer’s bones. Once the algorithm can recognize and process these movements and gestures quickly and accurately, it’s possible to recognize the flow of a sentence and translate it into text.
So far, all testing and related algorithm training has taken place within Lenovo and CESAR, and numerous hearing-impaired employees from both teams have contributed to the project as trainers, testers, and focus-group participants. For privacy reasons, the identities of all volunteers involved in training the A.I. system are kept anonymous; only the digital gesticulation points are stored as reference points in the resulting database.
The next step is to scale the project beyond internal testing. Billions more data points will be needed to achieve the desired goal of rolling out a real-time sign language translation interface. However, CESAR and Lenovo have successfully created the building blocks necessary for a live translation model that will work not only with Brazilian Sign Language, but that can be extended to other sign languages around the world.
In an effort to scale faster and develop a minimum viable product (MVP), Lenovo is now tailoring these new translation solutions for specific verticals, including banking and retail, with the goal of implementing them in the real world in 2024. Limiting the work to specific contexts is intentional, making it easier for the A.I. to become fluent and provide a good user experience.
For Lenovo, partnering with CESAR on this long-term software development project wasn’t just about enhancing its own products – it’s about living the global technology leader’s mission and having a greater impact on driving inclusivity worldwide.
“The purpose of this project goes way beyond the business case. When we say, ‘Smarter technology for all’ at Lenovo, it can’t just be a slogan. This is part of our mission, and it was something we knew we needed to do for the benefit of the hearing-impaired community, no matter what,” said Lima.
CESAR Innovation Center in Recife, Brazil is one of Latin America’s most renowned institutions for world-class research, design, and innovation on demand. For more than 25 years, CESAR has been a trusted partner for some of the best-known brands seeking to co-develop breakthrough innovations that up-level their competitiveness and productivity.