Almost everyone is now familiar with the emerging media technology of virtual reality. Unlike 3D cinema or other immersive viewing experiences virtual reality involves the user wearing a headset through which they view the virtual world. The headset tracks and reacts to their head movements so that if they look up they see the ceiling or the sky or an incoming aircraft, and when they look behind them they see the door they have just walked through.
The headsets also track motion so that the views seen by the user change as they move through the virtual world being presented to them. This quite amazing technology not only has the potential to transform entertainment, gaming and the creative arts but also has applications in areas such as training, education, healthcare and many others.
One of the problems up until now, however, has been the lack of a realistic immersive sound experience to go with the visuals. In some ways we were still in the silent movie era when it came to virtual reality.
Now, thanks to the work of a team in Trinity College Dublin, we may well have entered into the talkies era. The issue has been that no audio system existed that could mirror the visual experience offered by the headsets – what users saw changed with their head and body movements but the soundtrack remained the same.
The challenge faced by the TCD team led by Prof Frank Boland was to attempt to recreate, through a normal pair of stereo headphones, what happens to sound as we experience it in real life. If we hear a sound from behind us and turn around to face the direction in which it came from, we then hear it from in front of us. Similarly, sounds change as we move towards and away from them.
Virtual reality experience
However, when wearing a pair of in-ear headphones as part of a virtual reality experience, the sounds effectively remain “in our heads”. No matter how we move or where we go or how clever the sound effects employed they do not match what we are doing. This detracts greatly from the overall experience.
This is not an entirely new problem for the entertainment industry. Attempts to recreate a true cinema sound experience in the home and 3D-type sound effects for video gamers go back a long way. “The industry hoped that Audio 5.1 would give that effect with three speakers in front of you and two speakers behind you,” says Boland.
While that did offer satisfactory results to a certain extent with the viewer or gamer hearing sounds coming from around and behind them as well as in front of them, it was necessarily limited. Users were, after all, sitting down looking at a static screen and weren’t moving around the room. A static soundscape was more than sufficient.
The advent of the Oculus Rift and other virtual reality headsets on the market proved to be a pivotal moment in the research. These headsets enabled accurate tracking of head movements and allowed the TCD team to work on the development of the Thrive 3D audio system, which takes that movement information and uses it to inform soundfield reproduction, generating a truly immersive audio experience that matches, and reacts to, the user’s movements in the virtual environment.
That makes it sound a lot simpler than it is, of course. The physical version of the Thrive 3D audio system involves 16 high-quality speakers located at different points around a room with the listener in the middle. The sounds, which have either been recorded using a specially designed 3D microphone or generated digitally on a computer, are broken down into a whole range of digital signal components which vary in accordance with the location of the listener and the position of their head.
“A sound coming from a 45 degree angle to a listener will come into their right ear differently than into their left ear and we have to be able to reproduce this difference through the headphones,” Boland explains.
That process has to be repeated for every single sound and sound component and for every possible position of the listener’s head and every change in their location.
Having achieved that, it is little wonder that in 2015 TCD was awarded an Irish Software Innovation Award in the category of Outstanding Academic Achievement for the Thrive project. A further validation of the achievement and the quality of the innovation came that same year when Google acquired the technology.
The audio signal processing research which underpins Thrive has been funded through a combination of State and industry grants and it continues with funding from the Science Foundation Ireland Investigators Programme. According to Boland, the impact of Thrive and virtual reality technologies generally are only beginning to be felt.
“As virtual reality becomes a part of mainstream entertainment and education, the immersive experience provided by 3D audio will enable a much richer experience,” he says. “We will ultimately see this apply to television with live virtual reality broadcasts becoming the norm. Movies will offer consumers the opportunity to immerse themselves in the action and we have already seen how rapid advancements in graphics have led to an ultra-realistic visual experience for gamers and that will only be enhanced by matching audio.”
