Sign language services make the information world without obstacles

Content source: Huawei Developer Conference 2021 HMS Core 6 App Services Technical Forum, the keynote speech "Sign Language Services, Help You Build Information Accessible Applications".

Speaker: Zheng Xuan, Technical Director of Sign Language Service, Huawei Consumer Cloud Service, Professor of Beijing Normal University

Hello everyone. Today, I would like to share with you why HMS Core sign language service , the story behind it and the progress.

First of all, let’s take a look at the people we serve. According to a survey conducted by the world department, there are currently 460 million hearing-impaired people in my country, while there are at least 27 million hearing-impaired people in our country. 5% of the hearing impaired use hearing aids and other auxiliary devices. Therefore, due to the lack of information, our hearing impaired friends have some difficulties in daily learning and communication, which is why the channel of sign language in public places is very urgent.

sign language? What does it mean for the hearing impaired? What does it mean for healthy groups? We invited Professor Zheng Xuan from Beijing Normal University to share with you .

Hello, everyone, I am Zheng Xuan from Beijing Normal University. I am very happy to talk with you face to face.
In fact, I am deaf. You may not see that I wear a hearing aid in my left ear, but my spoken English is still good. My sign language interpreter is a hearing person, but she can use sign language fluently. I think at this moment, deaf The boundary between human and listening is not so clear, thanks to the advancement of science and technology.

want to share with you today is "Unlocking Sign Language Password and Entering the Silent World" .

When it comes to sign language, you may feel very mysterious. From a linguistic perspective, sign language is a language. When I was a PhD student, majoring in linguistics and applied linguistics in the Chinese Department of Fudan University, what I did was research from the perspective of linguistics. What kind of language is sign language? Sign language is the hand shape, position, movement, palm orientation, facial expression and body posture of the hand, as well as the necessary mouth shape, a communicative tool to express a specific meaning. It is a real language . Why do you say that? Because sign language is a language with a complete and complex system, with its pronunciation, vocabulary, and grammar, if the child’s parents are deaf, sign language can be learned as the first language. At this time, the development path and spoken language are basically synchronized. And we can prove from the perspective of brain science that when deaf people use sign language, the active area in the brain is basically the language area of the left brain.

There have been mature legal regulations regarding the status of sign language. For example, the 2006 UN Convention on the Rights of Persons with Disabilities and the UNESCO Yuelu Declaration that was later signed in China emphasized that sign language and spoken language are equal, and they are two languages. Every country is obliged to protect and promote the use of sign language. According to data from the World Federation of the Deaf, 66 countries and regions in the world have passed legislation to recognize the status of sign language.

Our country has also made great progress in the promotion of universal sign language. For example, 2015 and 2021 are the starting points of the "13th Five-Year Plan" and "14th Five-Year Plan" respectively. We have released two issues of the National Sign Language and Braille Standardization Action Plan. In 2018, we released the "National Sign Language Vocabulary", which has risen to the level of national language standards. In 2019, we published the "National Sign Language Dictionary". If you are interested, you can directly download it from Download the APP on the Huawei App Market.

The picture above is an actor from the Chinese Disabled People’s Art Troupe demonstrating the "China" sign language play. The "China" sign language play is to extend the fingers from the left shoulder to the right shoulder, and then imitate the style of traditional clothes in Hanfu. To imitate the concept of China.

sign language mean to deaf friends?

I am a deaf person, so I deeply feel the importance of sign language to us. You may not know that more than 95% of deaf children have their parents who are hearing-sound, which means that they cannot be at home. Communicate with parents as smoothly as a normal child. There is a word abroad to describe: table syndrome, even at the table, we and our parents are the most familiar strangers. Therefore, the intervention of sign language can change this situation.

When they grow up and enter school, sign language can be used as the language of instruction to connect children with the world and encourage them to learn knowledge; when they enter society, they can establish a connection with ordinary people through sign language. When everyone in the society can sign language and is deaf People and friends will feel equality and respect; when he gets old, sign language will create a spiritual home for him. Deaf friends are very emotional in sign language. There was a poem describing their feelings in sign language, "If you If you chop off my hands, I will use both arms to sign language. If you chop off my arms, I will use both shoulders to sign language."

What does sign language mean for a hearing friend?

I have taught sign language classes in colleges and universities many times. Students asked me: "Teacher, how do you signify that I love you?" His intelligence, when he enters the society, if he can sign language, it will actually help build an information barrier-free society, let him know that everyone can accept the diversity of the world and make this world a better place.
In addition, sign language can be used to communicate in special times, such as when the environment is noisy, when communicating secretly, sign language is very useful. When you get old, it has the meaning of helping others and yourself. In the past, it was him who helped others, but everyone He will get old, his hearing will decline, and sign language can be used as a visual language to support his communication.

I myself have many years of experience in sign language teaching advocacy and scientific research. I personally appreciate the sentence "If you accept someone, then you will accept his language. On the other hand, if you reject someone's language, it is Rejected him". I used to teach American deaf children to learn Chinese sign language and Chinese culture at a Confucius Institute in the United States. I used to teach flight attendant students in a class at Chongqing Normal University to learn sign language. I used to film the sign language song "Me and My Motherland". I have also worked with the teachers and classmates of Nanjing School for the Deaf to create our book on Chinese Sign Language.

When doing these things, I have a deep feeling that sign language is not actually a minority language. It is not far from everyone. We say that sign language is a bridge that can connect every isolated island in the world. Sign language is a resource. Let us Understand the unique beauty of each language. It is also the inheritance of life experience, a bridge for unity and struggle, and a bond for the deaf and hearing.

I participate in such a project as a consultant. Our vision is: Sign language is not far away and within reach. In fact, everyone will become a so-called "handicapped person" after getting old, after falling ill, and under certain special conditions.

Sign language is not mysterious. In fact, everyone can learn sign language. For example, this gesture means that a person is coming from far to near, and that person is going from near to far. Two people meet, separate, and chase. You see, we can learn a bunch of sign language words at once. It really doesn’t. difficult.

Sign language is not narrow and open to all rivers. We don't want deaf friends to have signs and oral groups torn apart. I sincerely hope that we can communicate together, build our homes together, and share the convenience of modern life together.

Sign language is not even a niche language. We can open our minds and create a sign language ecosystem and a new economy. From the glass wall to the wall breaker, Huawei has taken a crucial step.
In the process of participating in this project, I actually experienced a lot of ups and downs, and I am personally honored to be involved. I am a deaf person, and I am a researcher of hand linguistics. For us, this product can help us. But maybe not all people can understand it now, but we are very happy that through a period of research, we finally have a staged result. I think that darkness has given us black eyes, but I use it to find light. The eyes that have walked through the darkness are the brightest eyes in the world.

Finally, I want to teach you two sign languages.

The first is the meaning of welcome and cheers, we cheer this great era.

The second one is thank you. Thank you everyone for giving us new hope in the silent world.

thank you all.

It can be seen from Mr. Zheng's sharing that sign language is very important for both the hearing impaired and the hearing person. The country is also aware of this problem and has promulgated corresponding legal provisions. The regulations of the Shenzhen Special Economic Zone were just implemented on September 1 this year, as well as the regulations of Beijing on October 1, and the “14th Five-Year Plan” issued by the State Administration of Radio, Film and Television on October 22, all of which promote and encourage the use of sign language.

We have summarized three national legal provisions.

First, arrange national sign language interpreters in major party, state and local events.

Second, TV stations above the municipal level should actively create conditions to broadcast programs with sign language or subtitles, open sign language columns, and promote and use national sign language.

Third, public transportation such as airplanes, railways, and subways should provide national sign language services for the hearing-impaired.

Huawei’s sign language service did not start from scratch. In 2018, we launched the Storysign project with the European Association of the Deaf and the World Association of the Deaf. The mobile phone is aimed at children reading books, and the sign language animation can be automatically generated through artificial intelligence technology.

On the basis of the Storysign project, we have developed a sign language service, which is no longer oriented to special age groups. What we developed is oriented to open crowds, open scenes, and intelligent sign language actions.

Sign language includes not only hand movements, but also the entire expression of lip movements and facial expressions, because sign language is a visual language, which is not the same as spoken language. It has its own grammar, such as "I don't have an ID card", which is not in sign language. Typed word by word, the entire sign language is based on the visual language, with the words "ID", "I", "Take" and "No" first, followed by whether the sign language movements are coherent and natural, and then whether it comes with facial expressions.

Huawei Sign Language Service mainly has three modules.

first module of 16196f72095a3a, first converts speech into text , which is the ability to convert speech to text often in daily life. This is a relatively mature technology.

second module of 16196f72095a5a, from the text to the sign language expression , here is a key technology of sign language translation, using Huawei’s self-developed language model, combined with sign language linguistic data, we are different from other technologies The same is that we have a large training model, which is trained in a specific semantic environment, so the effect is very good.

third module of 16196f72095a7e, the text generates 3D action , this is a very difficult technical problem, we use the neural network multi-modal model to achieve. Of course, in this process, we not only have artificial intelligence algorithms and data, but also deeply integrate the two modules of sign language linguistics, sign language translation, and sign language generation, and output 3D actions, which are finally presented through 3D virtual humans. To drive a 3D virtual human, it is necessary to drive the limb movement through skeletal motion, and then perform rendering and presentation of the 3D virtual human.

The current language only supports Chinese sign language, and there are plans for English in the future. The vocabulary is also very important. Just now, Teacher Zheng also said that after the National General Dictionary was released in 2019, it actually contains 8214 words, but there are actually many words that have not been input. Together with Teacher Zheng, we have In daily life, the words that are not included in the national general dictionary are expanded, currently covering more than 10,000 words, of course, will continue to increase. It is estimated that it will reach 20,000 words next year, which can basically cover 90%-95% of daily vocabulary. .

If you want to apply sign language services to your business scenarios, 3D virtual person customization is also supported, such as changing the shape of the 3D lady in red.

The above is the key technology sign language service and . In what scenarios is the sign language service capability of

In the To C application scenario, for ordinary consumers, which can reduce the difficulty of communication and information acquisition , and realize the barrier-free communication .

First, daily life. Daily life can be used in scenes such as communicating with family and friends.

Second, education and learning. Sign language also has accents and local standards. The country has been promoting the national common sign language, but very few people really understand or are very proficient in professional sign language, so the Huawei sign language service based on the national common sign language standard is applied to deaf friends The education is very good.

Third, it is used to obtain social information. Such as travel information, dining, shopping and other scenes.

sign language service also played a big role in some To B scenes. For example, the state encourages or promotes the promotion and use of the national common sign language in some industries.

First, the scenes of broadcast and television, live broadcast and relay.

Second, it is used in government affairs, banking, transportation and other scenarios.

Third, it is used in scenes such as large-scale conferences and live events.

The product launch of sign language services is not our end, but our starting point. We hope that our innovation can promote the continuous progress of the accessibility of information in the entire society.

The above is my sharing, thank you.

for more details>>

Visit Huawei Developers Alliance official website
Obtain the development guide document
Huawei mobile service open source warehouse address: GitHub , Gitee

and learn about the latest technical information of HMS Core for the first time~