Volume 12, Number 03, February 2022

An Application to Provide Translated Subtitles and Pictures for Youth English Learners
using Speech-to-Text and Nlp Techniques

  Authors

Harry Cao¹, Yu Sun² and Ariel Jiang³, ¹SuZou Industry Park Foreign Language School, China, ²California State Polytechnic University, USA, ³Cornell Tech, USA

  Abstract

Currently, thousands of free K-12 educational videos exist online with the aim of trying to help young students learn outside of the typical scholastic environment. However, most of these videos are in English, so without subtitles it may be difficult for non-native English-speaking students to fully understand them. These students may need to spend time searching for translations and understanding content, which can distract them from grasping the important concepts within the videos. The state-ofthe- art of speech-to-text and NLP techniques might help this group digest the content of instructional videos more effectively. This paper proposes an application that uses speech-to-text, machine translation, and NLP techniques to generate translated subtitles and visual learning aids for viewers of instructional videos. This video application supports more than 20 languages. We applied our application to some popular online educational videos and conducted a qualitative evaluation of its approach and effectiveness. The results demonstrated that the application could successfully translate the English of the videos into the viewers’ native language(s), detect keywords, and display relevant images to further facilitate contextual understanding.

  Keywords

Educational videos, mobile applications, language translation applications.