How’s AI Changing Multilingual Captions and Dubbing in Live Events?

Imagine watching a live event in a language you don’t understand. Frustrating, right? That’s where multilanguage captions and audio dubbing come in. When it comes to edtech learning, multiple language captioning and audio dubbing in live streaming can redefine how we deliver an immersive experience in classroom fashion.

Below is a thought-provoking discussion from our podcast episode between Rishi from Magic EdTech and Chris Zhang, Senior Solutions Architect from AWS Edge Elemental. With a background in network programming and media technologies, Chris is passionate about enhancing live event accessibility, particularly in providing multilanguage captions and audio dubbing for diverse audiences.

Their discussion provides insights into the current and historical challenges involved with multi-language captioning and audio dubbing in live events. Deep dive into the conversation to find the latest technologies that make live events more accessible today.

 

The Applications of Multi-Language Captions and Audio Dubbing in EdTech

In a virtual classroom, learners come from diverse backgrounds where English is a second language. However, instruction is often delivered primarily in English. This brings up the question – how do we help students have an immersive classroom experience? Also, how do we ensure that the students do not have to spend extra time doing the heavy lifting to understand what is being taught?

Multi-language captioning helps students to access captions in their preferred language, right at their fingertips. They may prefer to watch course material in their native language or English with the option to enable captions in another language, such as Chinese. This allows them to pick up unfamiliar or interesting words, improving their learning experience.

On the other hand, audio dubbing gives them a choice to either watch lectures with captions or select the appropriate audio dubbing feature.

Together, these technologies can expand the reach of the classroom, making education more inclusive and accessible to students around the globe.

 

Challenges in Live Captions

Live captions present unique challenges compared to captions for video-on-demand. In live streaming, the traditional methods for providing multilingual captions are particularly challenging.

The Language Divide

Many individuals are using live streaming in the traditional classroom fashion. This reflects the growing familiarity with remote learning especially after the pandemic. However, the challenge with such a setup is the diverse student backgrounds.

For example, in the United States, universities have students from Korea, Japan, Asia, and other European countries whose native language may not be English. To enhance their understanding, students may rely on applications that automatically transcribe the teacher’s speech. However, this adds an extra layer of complexity as students must manage the transcription process while seeking additional tutoring to fully grasp the material.

The language divide thus creates a significant burden on learners, impacting their overall educational experience.

The High Cost and Complexity of Traditional Captioning

In traditional live-streaming, providing multilingual captions often involves hiring stenographers or captionists—individuals fluent in English who actively listen, translate, and type in different languages. However, this process becomes complex and costly, especially during long events, lasting for several hours or a full day. Multiple captionists are required to take turns to ensure they get breaks, which adds layers of complexity in system setup, logistics, and cost.

Limitations of Current Technology

Today’s audio-based speech recognition engines (ASR) can automatically transcribe speech, typically into English, and translate it into various languages. However, the limitations of current captioning encoders and protocols, such as 608 and 708, are significant. These protocols only support seven Latin-American-based languages, including English and Spanish, leaving languages like Korean, Japanese, or Russian unsupported. Additionally, the 708 protocol lacks end-to-end support and cannot handle multiple languages simultaneously, posing a considerable barrier to delivering accessible content in diverse languages.

Challenges With Integration

Integrating ASR technology into the live-streaming workflow introduces further challenges. While the end-to-end process—from capturing video to transcoding, CDN distribution, and delivery—is relatively straightforward in a cloud environment, merging it with the captioning workflow of the existing live streaming workflow complicates the process.

Traditionally a captioner is hired to embed captions into the live stream before the transcoding process. This method relies on hardware-based solutions that require on-premises equipment at the event location, adding logistical complexity and cost. Furthermore, as mentioned earlier, these hardware encoders, which use the 608 and 708 protocols, are limited in their language support, primarily catering to Latin-American-based languages.

The challenges can be broken down into two phases:

1. Signal Generation and Contribution to the Cloud: The initial phase involves generating the video signal and contributing it to the cloud. Integrating captions at this stage can be difficult due to the need for specialized equipment and the limitations of existing protocols.

2. Signal Distribution to Viewers: The second phase involves distributing the signal to viewers. While most live shows are delivered using HTTP Live Streaming (HLS), which supports VTT sidecar files for captions, the challenge lies in synchronizing these captions with the video and ensuring compatibility across different CDNs and viewers.

Issues with Accuracy and Reliability

In live streaming, leveraging AI and ASR for automatic transcription of a teacher’s voice presents challenges in accuracy and reliability. ASR engines may misinterpret certain words, especially if the teacher pronounces them incorrectly or uses non-standard language. This can result in inaccuracies in the transcriptions, impacting the quality and understandability of the captions provided to students.

 

Technologies that Make Live Events More Accessible

Modern technologies are mature in terms of replacing the heavy-lifting. By leveraging them, users will be able to achieve their goals of delivering accessible content in live events. This includes:

Leveraging Cloud-based ASR Engines and a Generative AI-based Technology

Cloud-based ASR engines and generative AI significantly simplify the integration of captions into live-streaming workflows. By eliminating the need for on-premises hardware, these technologies allow for the generation, synchronization, and delivery of captions directly within the cloud. This modern approach broadens language support and improves overall efficiency, addressing many challenges associated with traditional captioning methods.

Late Binding for Captioning

Modern streaming protocols such as HTTP Live Streaming (HLS) support VTT sidecar files for captions. This capability enables the late binding of captions, meaning captions can be added and synchronized with the video stream after initial production. This facilitates flexible and accessible delivery of multi-language captions.

CDN and Player Agnostic Solutions

Utilizing cloud-based ASR and generative AI in combination with modern streaming protocols ensures that captions are compatible with various CDNs and viewer players. By generating and delivering captions in VTT format, this approach maintains broad compatibility, making the live stream CDN-agnostic and ensuring that it works seamlessly across different player platforms. This design improves the accessibility and versatility of live-streamed content.

 

Adapting to AI-Driven Language Tools

When innovating with AI-driven language tools, it’s essential to approach the process with an open mind. Ask yourself:

– What outcome is intended to be achieved?

– What customer experience should be delivered?

– What features will enhance the customer experience and make a significant impact?

 

Conclusion

Chris suggests starting by identifying the key objectives and then working backward to determine how to achieve them using current technologies. He emphasizes the importance of assessing available solutions and integration points. By starting with desired customer outcomes, a clearer understanding of where to focus and how to address problems can be gained. This can help avoid the constraints of older technologies.

As AI and cloud-based technologies continue to advance, Chris notes that costs are expected to decrease significantly. Exploring live captioning solutions can expand content reach to global audiences. This need is driven not only by strategic business goals but also by regulatory requirements that may dictate how content must be delivered.

By utilizing these technologies, edtech companies can reduce costs, eliminate manual processes, and ensure the delivery of accessible content to consumers.