In the previous blog, we introduced PinyinTube, its purpose, and its mechanism to help SwapBrain users enjoy not only the best Chinese movies and videos experience, but also act as a useful Mandarin learning tool. In this post, we will go deeper inside the technology of PinyinTube and each role of PinyinTube’s task set.
1. A good listener
First, let’s dive into how PinyinTube records the actors’ voices and other audio snippets. Unlike Google Live Caption, which requires an inherent transcript set for each video, PinyinTube can access the microphone of each user’s computer and do the recording from microphone input. When the users press the microphone button, their voice will be recorded and stored in the software storage. This way, the users can playback their voice and compare with the actor’s voice. This can done multiple times, allowing the users to improve their speaking skills. PinyinTube’s live caption is built based on a JavaScript API called Web Speech API. Once installed into the JavaScript code snippet of the extension, this API will enable PinyinTube to collect the live audio the users.
2. A good translator
Now we have partially understood how PinyinTube retrieves the audio snippets and convert them to letter form. Yet, how can those words be converted to other languages like English to Pinyin or Mandarin to English? Interestingly, this all starts with how human beings pick up a new language.
Learning a new language demands time and effort. The first few months will prepare learners so that they can be involved in primary or intermediate conversations. With further learning and relentless practice, after a year or more, fluent daily communication with native people should not be a problem. This incredible progress is thanks to our neural system’s training and learning development. As we maintain frequent language exposure, our brain begins to “adapt” to new words, expressions, and phonetics. This way, hence, has been adopted to design one of the highest technological platforms, the Deep Neural Network.
Deep Neural Network is a subset of the Machine Learning field inspired by the human brain’s biological operation via trillions of neuron cells. Thousands of companies have recently applied Machine Learning to their tech product and SwapBrain is no exception. As an AI/Automation consulting company, we focus not only on AI services but also on Deep Neural Network application development. PinyinTube will be our firstborn using Machine Learning technology to be a great translating tool. To do that, the extension will process the caption input, push it through multiple layers consisting of cells in the Neural Network, and calculate the gradient before printing the output to the screen. This is the whole process for PinyinTube to translate Chinese/English subtitles into the user’s preferred language.
3. A good supporter
With the capability of a live translator, PinyinTube also has some other notable features that SwapBrain customers, can take advantage of at no cost. Once installed and activated, the extension will automatically translate Chinese or English audio into your desired language. The caption will be provided right above the video’s subtitle, along with multiple interactive popup buttons that serve your needs. As you may not know, PinyinTube is uniquely built to link each audio sentence with its live caption. Thus, you can both hear a sentence spoken by the actor and read the translation script simultaneously. Additionally, you can click the forward/backward buttons to see the next/previous sentence. PinyinTube will receive the order to roll over on any sentence you choose and also jump to the corresponding fragment. For users who wish to utilize the extension as a learning tool, the playback speed button is adjustable so as to match your reading and listening level.
However, more advanced items can only be accessed through a pro subscription such as “Anywhere Captions” and “Voice Comparison”. In particular, there is an extra charge for the extension to be used on other streaming websites (except Youtube and Netflix). With the latter, PinyinTube can also record your own audio to get tested and corrected against the actor’s voice. This will help you compare your phonetics with the standard actor’s voice in the video so that you can improve your language skills. Although PinyinTube is accessible to everyone, we believe spending a little extra would be largely beneficial for specific customers who are interested in learning Chinese or English to enrich your multilingual base.
4. What in your thought?
Now you have comprehended a set of stunning works we have been doing to bring you, our precious customers, the best version of a live translator and learning assistant. In the next few months, we are going to publish PinyinTube to the Chrome Web Store and serve you the first product that we have been doing our best on it. If you want to experience our MVP, please enter your email at the bottom of this page. If you have any feedback or suggestion abourt features that the extension is having, please let us know by commenting down below or send us a message via our contact page. Any comment will be a great contribution to the improvement and growth of SwapBrain and PinyinTube.
Let’s keep blogging and cheers!
