This Classroom Utilizes AI And VR For Mandarin Teaching

The process of learning new languages is a time-consuming and hectic one. Learners are required to invest in a lot of time as well as effort. More often than not, the process can be a little bit boring since foreign words are hard to understand and memorize. So, in order to make the learning process more interactive, New York-based Rensselaer Polytechnic Institute has set up a classroom equipped with AI and VR for Mandarin teaching.

The AI and VR used is a result of a partnership with IBM Research. Inside the classroom, there is a 360-degree virtual environment which is based on Beijing street. Students get to learn the language by interacting with street vendors and ordering food.

The people who work on the project think that the easiest and best way to study a new language is to being exposed to the environment where the language is natively spoken. And because it is not feasible for language learners to move to another country just to study a new language, they will get a similar experience with this classroom.

With AI, student queries are answered in real-time. The institute’s two faculty members were the one behind the project’s idea. They once used role-playing games so that students can understand the conversation better.

The classroom uses several sensors to adjust to the students’ behaviors to offer a better learning experience.

The image above shows how a student is interacting with the system. A camera is used to track the body movements and a mic for voice input.

It’s simple to interact with the environment. When students need to know more about a specific object, they just need to point at it and ask “What is this?” After that, relevant and suitable answers will be provided by AI.

Besides giving students the chance to interact with the environment, the platform also improves their pronunciation. This can be done via audio along with visual feedback given to students so that they can pronounce the words correctly. The platform’s algorithms also make a comparison between the student’s speech with one of the native speakers. Also, students can know how their pronunciation differs from native speakers.

At this moment, the classroom is still in its testing phase. It will be implemented later this year in a six-week course at the institute. The virtual environment will be deployed for half the duration while the other half will be for traditional classes. However, this will depend on the efficiency and effectiveness of the virtual environment.