Projects

Location Based Audio Game

In our new project, we are experimenting with the potential of location based audio games as a new approach to Japanese language and culture.

We are also in discussion with Prof. Ikumi Waragai from Keio University about a potential collaboration in this context.

Team
Fabian Russ (composer for orchestronic • creative director)

Armin Becker (Leipzig University)

Martin Roth (Leipzig University)

Hiroyuki Horie (Leipzig University)

Claudia Kluge (Leipzig University)

Overview

This project, initiated by Jun. Prof. Dr. Martin Roth and Fabian Russ within the department of Japanese studies, aims at enhancing listening comprehension skills of Japanese language students, by researching, if gamification and an audio-augmented reality can have positive influence on the process of understanding naturally spoken Japanese.

The approach involves the creation of the above-mentioned audio-augmented Japanese version of their hometown. Within this new city layer the students will be invited to go on a scavenger hunt, guided by Japanese audio clues. Instead of sitting in a classroom the students will walk around in the city, discovering Japanese stories in a virtual layer build on top of the familiarity of their hometown (e.g. mapping the Tokyo fish market to the market square of the town or opening portals to the world of yokai).

We saw how Pokémon Go made people rediscover their environments. Our approach involves tapping into the same processes of gamification to create a study environment, which potentially benefits the learning process by making the experience more immersive.

Project goals and concepts

The primary objective is enhancing the listening comprehension skills of Japanese language students. For this we want to investigate features and abilities of new media and will try to combine those with the process of learning.

A lot of textbook listening materials used in classes mostly consist of staged conversations with purposefully clear voiced dialogs and already known vocabulary. Facing real-life Japanese will often become a challenge, for which the students are not properly prepared. Unlike listening comprehension dialogs, in a real-life conversation natural speech will differ in many ways (e.g. gender, age and other characteristics strongly influence how people speak “naturally”) and encountering and dealing with unknown vocabulary is more common as well. Furthermore, the environment, in which the students are listening to the dialogs, will mostly consist of their classroom and their self-study locations.

  • Audio-Augmented Reality: Echoes
  • Engaging students: Gamification
  • Natural Japanese
Audio-Augmented Reality: Echoes

Pokémon Go and other augmented reality games managed to put another layer on top of the environment their users are living in. They created a virtual space, which enabled people to experience their daily environment in a new and interesting way. We aim to tap into this aspect of augmented reality to create a learning environment, which implements a fictional victual space into familiar, which adds an interesting layer, ready to explore.

Fortunately, there already exist free applications, with which one can map sound- and audio files to geolocations called Echoes. This allows for the creation of a broader spectrum of stories. Creative ideas, such as non-linear stories, interweaving distinct features of the local into practical Japanese practice and much more.

The free application eventually does not meet all our expectations, but with a creative mind, fruitful content can be created within the technical boundaries. But since limitataions exceeded usability, we had to come up with our own software solution.

Going another step further we want to make the audio experience as immersive as possible. Drawing from the expertise of Fabian Russ the ambience background will be created in a manner, which aims at invoking the feeling of being in japan (at least with your ears).

Engaging students: Gamification

We aim to create a captivating study-environment. Our approach includes an explorable world set in and connected to Japan and the hometown of the students. But it is not limited to this alone.

Applying methods of gamification, we will try to keep the whole experience interesting and captivating. Will a murder case in Japan in the 70s, where students must gather clues over a series of episodes keep them on the hunt? Or will they be allured by a modern feud between two samurai clans, fighting for control of the city (thinking of the modern Romeo and Juliette)? The number of possible scenarios is limited by the creativity of the ones creating them only. Unlocking parts of a story, finding secrets in the city, getting to know the inhabitants. Just like a good bock, audio training could have the potential to enchant the listeners as well.

Furthermore, the open format allows for active student participation: They can create stories as well. This can ensure that the offered stories meet the interest of current Japanese students, since they will be the ones creating them.

Natural Japanese

One priority is having our audio walks recorded by Japanese native speakers, who will use their natural way of speaking. The reasoning behind this decision involves the many facets of natural Japanese, which seldom is covered by textbooks but nevertheless encountered in every day speech. When talking to native speakers, it is not unlikely to encounter many dialects and speaking habits. And the one learning the language will have to adapt to this reality. We believe, listening comprehension training should as well be a place, where you can train for these occasions.

Another aspect is the vocabulary encountered in conversations. There will be a gap between the vocabulary learned by students, and the one used when talking to Japanese, resulting in not understanding every part of given dialogs. This is a natural phenomenon and students should learn, how to deal with these gaps in knowledge. One does not have to understand every single word, to grasp the concept of the story.

  • The Second Prototype - Environment: Swalk (self-written)
  • The First Prototype - Environment: Echoes
The Second Prototype - Environment: Swalk (self-written)

Developing the environment:

Thanks to the first prototype, we came to the conclusion, that development of the idea into a tangible experience is possible. But we encountered several issues, especially of technical nature.

This led to the conclusion, that we needed our own software. That would as well enable us to use features we deem important. After a few months of work we came up with another prototype, this time created by ourselves, and another story.

The First Prototype - Environment: Echoes

Testing the environment:

We have built a prototype-walk to test the technical environment of the audio-walks. It is available for free for everyone to test (Download the echoes app for iOs or Android and look up Leipzig).

This audio-walk takes the listeners to the southern parts of Leipzig downtown, where they accompany a fictional character, looking for a rumored statue.

Creating the Prototype


Story

We based the language used in the audio-walk on the textbook used in Japanese language classes at the university of Leipzig. Based on the new grammar and vocabulary the listeners will encounter many words they already know and can test their grammar, if they are studying with the textbook as well.

Of course, one is not limited to textbooks while creating audio walks. But we found it helps giving a framework regarding the organization of content. Since we aim at active student participation, we created a style sheet which will lower the threshold for creating original content.

The story told in this prototype introduces characters and content, which aim to mimic a Japanese setting, with content one could experience in Japan as well. The setting is the above-mentioned Japanese version of Leipzig.

Japanese

The dialog has been written by a german student. It was revised and improved by a native Japanese speaker currently living in Leipzig.

Audio

For an immersive experience we plan to record Japanese settings and implement the recordings into the audio-walks. Train stations, temples, festivals, pachinko halls and much more.

Original and natural background sounds are essential to an immersive experience, which we hope will make the Japanese learning more captivating.

Up until now we do not have these recordings and must use stock-sounds, which do set the mood, but lack the quality we want to achieve.

Technical Aspects

The application we are using gives us some freedom to experience. Although the features are limited and need some creativity to use, the principle we want to test can be put into action.

Creating so called echo-locations is easy. You encircle an area and add the specific audio file to it. For our test run we have disabled visibility of the areas, as to not give a clue, where the participant must go next.

After creating all the echo-locations, you safe the file and open it for public use. The created audio-walk can then be downloaded by everyone.

Prototype test run


We then tried the prototype to see how our ideas would survive the on-road test.

After downloading the audio-walk, the participants found the starting point in the description. After arrival they started the prototype and were then guided by the audio alone.

Since it is a test run, the prototype only consists of 5 geolocations. One is visible, encircles the test area and gives and audio hint, whenever the participant leaves the play zone.

After listening to the first audio, the participants had to find the next geolocation. For this, the dialog contains clues, which include specific points of interest in the city of Leipzig. We hope the inclusion of Leipzig helps connecting to the story.

The following series of pictures paints a picture of how the experience unfolds.

(Our gratitude towards Antonia Deseniß, a student of Japanese studies at the university of Leipzig, for participating and granting us the rights to present her experience)

Duration: 10-15 min

Antonia starts at the train station “Wilhelm-Leuschner Platz” and listens to the introduction. The application shows the area and plays the according sound. Her destination is a certain statue, which is supposed to be located somewhere around this area. She heads off to the next location, which the audio clues indicate will be the next train station. But the trains departing here can go to two different train stations.

On the way to the next train station she apparently went the wrong way and as soon as the GPS hits the boundaries of the play area, the audio hint plays, telling her to turn back and to look for another way.

Being back on track she found her way to the next audio clue. The little story snippets conclude and leave her with the hint to head to the park in the north. Since there is only one park to the north, finding it should be a piece of cake.

She found the park with ease. Now she only needs to find the statue.

But there are several statues scattered around. The first statue she encounters is not the one in question. Which means she must go on further.

Will she find right statue and unravel the secret behind it? Well, our suggestion is to try it on your own and find it out yourself.

Software


The features we added were:

  • Triggering audio bubbles:
    To keep a story consistent the audios should be triggered in order. Furthermore we can activate or deactivate audio in regards to player progression throughout the walk
  • Overlapping audios:
    BGM, Soundeffekts and speech can be played at the same time, constructing a virtual space, which (hopefully) feels more immersive
  • Moving audio bubbles:
    To simulate people walking around through the city
  • Spatial (3D) audio:
    Audio in a simulated 3D audio environment sounds as if the source of the audio actually is standing on the left or right and thus (hopefully) makes the experience more immersive

Story


The story for this prototype is about a little girl in a feudal japanese town. The townsfolk themselves are split into factions and the little girls sister went missing. The player will have to help her find her little stister. Story building techniques we will test include:

  • Hidden lore (compare the lore in the Dark Souls series):
    The characters will not explain the background of the story, but give (subbtle) hints. Also not all bubbles are neccessary for story progression. Some just include lore. We hope, that players will want to find out more about the underlying story and thus feel more emerged into it
  • Player involvement:
    Depending on player choices the story will progress in different ways, having different outcomes.
  • Natural Japanese:
    Same idea as with the first prototype, but this time there will be several speakers, each speaking in a more natural way

Testrun and Evaluation


The testing will be caried out by several students of the japanese study department of the University of Leipzig, as well as by japanese learning people outside of the university context.

This is a unique website which will require a more modern browser to work!

Please upgrade today!