Written by Alexandre Saba, Senior Sound Designer at Funcom.

I had the opportunity to attend GDC this year and to meet some of the people behind the games I play and consider references. I got to pick at their brains and get details about their approach.

One thing they all shared was the need to push the boundaries, refine the processes, or further advance their toolsets. The conversations usually involved a technical challenge, followed by some sort of solution. Some of the solutions were temporary while others were more permanent. The process of constant iteration and improving the pipelines was to achieve the goal of creating a better product and to create a more immersive experience. But to the end user, what does that all mean?

What happens under the hood is a mystery. Even between departments it is sometimes difficult to fully understand what’s going on. Of course, players might understand the concepts of reverb, asset variations, and interactive music, but what really matters to them is what it sounds like.

That’s why some games just work. It’s because they’re designed for the player. They’re not created to be played exclusively by other developers and lost in a jargon of technical approaches. They’re intuitive. They’re challenging, but easy to understand, making the player’s progress fun. That comparison can be seen everywhere, and a good (but extreme) example is architectural functionalism. The core concept is this: Does the design serve it’s purpose?

Who do you design for? The designing process involves me thinking about the player. How will he or she interact, or interpret what they hear; is it clear? There is a part of me that also thinks of how my colleagues will hear it. To create something my peers can enjoy and analyze often helps push the product even further. However, when taken too literally, this can lead to the demise of the design.

When we think next-gen, we think more versatile tools, a more dynamic mix, more variations, more access to gameplay elements. Well, maybe, and I assume this list won’t harm the experience, but what truly makes a great sounding game for the player? I wish I had the answer. Hell, what it sounds like to me depends on so much, time of day, my mood, the listening environment… I don’t think there is just one answer, but i’ll throw one out there and see how it sticks.

Transparency. We prototype, design, and master, constantly refining our toolsets, all while compromising to fit within our technical limitations. All of that just to make things work, but the real challenge is in making it seamless, unnoticeable… as in nobody noticed it. This brings me back to the title question: Who are we catering to? Let’s not forget that.

Name:
Naila Burney

Company (or Freelance):
Freelance

Did you attend school for an audio-related degree?  If so, what school and degree?
School: Universidad Javeriana (Bogota, Colombia)
Degree: BA in Music with Emphasis on Sound Engineering

School:  Leeds Metropolitan University (Leeds, UK)
Degree:  MSc in Sound and Music for Interactive Games

What inspired you to work with sound?

I first wanted to do music, and when I was in high school I was the guitarist of an all-girl band for several years. However, I felt that it was not exactly what I wanted to do for a living, although I knew I wanted to be involved with art regardless of the medium (I tried sculpture, but it wasn’t my thing either!) Fortunately, I found the major at Javeriana University that merged music with sound engineering, which ended up being a perfect balance.

How old were you when you found out sound is what you wanted to do for a living?

At 18 I knew I wanted to do something related with sound and music. At 23, I knew I wanted to do sound design for film and television. At 28 I knew I wanted to do sound design for interactive games.

Was a school degree the first thing on your mind, or do everything self-taught?

It was the first thing on my mind. I had many gaps regarding music and null knowledge regarding sound. So for me it was essential to have a formal education. Besides, in that time the mentality or habit of searching for everything on the Internet wasn’t very common. Now, with tons of more information, audio communities, and social media, it is a lot easier to be self-taught.

Then, after five years of undergrad studies, I worked at a TV production company owned by Sony, where I learned tons of things that never even crossed my mind while I was in school. It actually was like another school for me, but with more practice and less theory. Things started getting too static after a while, though, so I decided to do the MSc.

I chose an on-campus MSc course instead of learning on my own or doing a virtual program because even with the massive information on the Internet nowadays, it is still crucial to meet people in real life. Also, the experience of going to a different country to learn is invaluable. It pushes you to work harder because you have to adapt to a different rhythm of life, culture, language, and educational scheme, and at the same time you have to learn about sound and respond with results. At the end, the whole experience broadens the scope of opportunities for you.

Bass Boost

What is your specialty/preference of the sound fields (sound design, music, recording, audio programming, implementation, etc)?  What do you like most about it?

Sound design is my specialty, although I would love to explore more audio programming and music. Learning about game audio has been very revealing because the fact of not knowing in advance the events in a visual sequence changes every traditional scheme of making music and sound.

What I like most of all of them is that you can be extremely creative. Not only can you express yourself, but you can also influence the audience’s emotions through sound.

What sound tools did you learn in your school curriculum?

Assuming that “sound tools” is not just software and hardware:

Javeriana: Recording techniques, mixing, editing, MIDI, Protools, Reason, field recording, live sound (although I never really got into it), operation and installation of sound devices such as mixers and managing studios, and music fundamentals (theory, sight reading, basic piano, ear training, etc).

Leeds Met: Game audio principles and implementation, in-depth sound design, and building interactive interfaces. At a master’s level you choose the software tools to make your own projects, so I mostly used UDK for implementation and Max/Msp and Processing for prototyping interactive applications.

What kind of projects did you have in your classes?

In Leeds Met I made an audiovisual application programmed in processing that involves generative music and allows compositional interactions by the player. The design of the app is based on synaesthesia effects (colors, moods, and sounds).

For my dissertation I made a fictional language dialogue system in Max/Msp that intends to generate expressive creature dialogue lines. Phrases are created by concatenating and recombining phonemes and non-vocal sounds, producing the illusion of a language. The system uses different probability settings of pitch, pace, pauses, volume, and layers of sounds, to express specific moods.

At Javeriana University I did many recordings of all kinds of music, including salsa and Caribbean folk. My graduation project was the music production of 40 short children-folk songs from different regions of the country where several autochthonous instruments outstand (drums from the Atlantic and Pacific coasts, Andean bandolas, charangos, marimbas, tiples, etc).

Were your teachers audio professionals?  Anybody the audience would know?

Most of them were audio professionals! They are all very well known depending on the audience. Probably the English speaking community wouldn’t know my teachers from my undergrad, but in the Latin American scene they are very well known, and the same happens the other way around:

Teachers at Leeds Met University, in the UK game audio scene:

Richard Stevens
Dave Raybould

Teachers at Javeriana University in other audio-related scenes (in Latin America):

Ricardo Escallón
Eduardo de Narváez
Andrés Cabrera
Jorge Díaz
Felipe López
Oliver Camargo
Luis Fernando Beltrán

External Mentors that I met along the way, virtually or in person (game audio):

Kenny Young
Andrew Quinn
Mojen Jenkins

Plug-Ins

Did you do any side projects during school?  If so, what were they like?

I did several small side projects to gain experience. I remember recording various ensembles and choirs.

I also worked as an editor at the Javeriana Estereo radio station, and then, before I graduated from my BA, I started working at a TV production company owned by Sony.

How many of your side projects were published?  Any of them turn profitable?

They were small private projects, so none of them turned profitable. The TV shows I started working on by the end of my major were the only ones published and profitable.

Echo

How large was your graduating class?  Were you all close?

In the BSc we were about forty people when we started, but only a few finished and graduated with me (around six).

In the MSc, we were eight at the beginning. Some retired, others switched to another audio field, others referred, and only two of us finished the course of Sound and Music for Interactive Games. However, we had many classes in common with students of other similar audio areas.

How often do you work with your old classmates today?

We don’t work together but we keep in touch and whenever we need feedback or something, we help each other.

Any old classmates you want to mention?  The more the merrier with the audio community!

From the MSc:

Stefan Rutherford – Sound Designer, specialist in Game Audio
Marc Weber – Sound Designer & Music Composer
Ben Reibel – Audio Engineer & Production Manager

From the BA:

Juliana Velásquez – Sound Designer & Events Manager
Sebastián García – Audio Engineer & Musician

Amplify

Do you feel more prepared for the sound industry than if you had not graduated from your program?

Definitely.

Do you have a website for your portfolio?  How often do you blog on it?

I recently started a blog in Spanish about game audio. I’m blogging once a week (and hope to keep it up!) The blog is gameaudiostuff.com.

Some of my portfolio works are available at be.net/nailaburney

Do you use social networking?  How often, and what communities?

Yes, I often use:

Twitter: @NailaAshi
Linkedin: linkedin.com/in/nailaburney
Facebook: Naila Ashi Burney

Fade Out

Any last words for future audio people looking to carve their education and career paths?

The more you study, the more you realize how little you know!

About Sonic Backgrounds

The sound industry is an ever growing field, ranging from linear sound design in film and TV, to interactive audio in games, and from live theatrical sound design to field recording for the creation of custom libraries.  It is only recently however, that school programs have begun to offer degrees in the sound-specific variety.  Graduates of these new programs are now coming into the industry, and it provokes the interesting question of how these new, specific programs are preparing individuals for the sound world, as opposed to the older approaches of entry, such as pure passion, musical talent, a film degree, or a computer science degree.

“Sonic Backgrounds” is an interview series focused on interviewing recent graduates of these educational sound programs around the globe, to see what exactly they are providing, and how they are shaping the new “academic”-based sound artist.

I’d like to address that last one. It’s the easiest one to answer of the bunch, really. The first two, those are the tough ones. No need to stir up controversy with those first two.

I should mention that I don’t have scientific evidence to answer this question from an analysis of hard data. But I have observed some stuff over the years. Also, I can say that I work at a place where audio developers are involved from the beginning, or, we are at least invited and encouraged to be involved from the beginning. It didn’t come without some effort on everyone’s part.

I think the first step towards figuring out why audio isn’t involved from the start is to ask ourselves why other people wouldn’t want to involve us. We have to be honest with ourselves when we ask this question. And we need to be fair to people who we may not understand.

It’s easy to point fingers at others. Some people don’t see past the audio label to see the whole human being with an entire life of experiences. Some see us coming in at the end of a project and can’t fathom how we’d function at the beginning. And there are some who just couldn’t give two craps about audio at all.

Bad news time, though. All of that stuff points right back to us. And we’re not going to make any progress until we can acknowledge that this is our responsibility. The issues above that seem like they’re about other people are really about us.

The easy part is explaining how we can contribute audio early on. We can talk to our colleagues and the leaders of our companies about ideas we have for early phases of development. Things like providing music for the team to work to that fits the style of the game, or placing evocative music to key pieces of concept art, or creating prototypes of new audio systems, or filling test levels with sound and music, or recording temporary voice for mission prototypes, and so on forever and ever for as long as we’re willing to experiment. We know this, and we need everyone else to know it, too.

Showing is better than talking, though. We can’t just talk. We can’t expect other disciplines to provide us with a bunch of work so early in the project. We don’t need them to, either. That’s exactly the perception that we’re fighting. We need to stand on our own two feet and show how we can add to the project from the very start. We dig for information and then we start doing stuff.

We need to have skills outside of audio. Often the absolute beginning of a project brings a scarcity of support. We can’t always expect lots of help in proving out our ideas. We need the ability to do it ourselves, which might involve level design, gameplay scripting, interactive fiction, interpretive dance, woodworking, honestly whatever tools we can use to express our ideas without needing other peoples’ time. Self sufficiency can mean the difference between being involved or not, since it’s just one person being added early on and not a more costly cadre of people.

We need to show results quickly, too. Things change quickly early in the project. Things can become irrelevant over the course of 48 hours as the teams explore possibilities. If we’re going to keep the pace, we need to be adept at quick sound design that gets the point across.

We also have to prepare to waste a bunch of work. Wasted work is a pretty dreaded thing during production and post-production. It’s going to happen in the early phases. Like, a lot. And that’s okay. Most of what everyone else is doing is going in the dustbin, too. Work in parallel with artists and designers and don’t wait for their stuff to be final. If timing or some other variable changes and it’s worth fixing the sound, then you’ll fix the sound.

I realize that sometimes we are understaffed in audio personnel. We’re already sweaty and overworked and in desperate need of some family time, and it can seem impossible to find time to involve ourselves in another project this early when we already have our hands full. But let me just suggest that one unexpected, meaningful contribution early in the project can be all it takes to convince the team that they need us. This might be good justification to hire.

My soapbox is caving in. There’s more to say but you’ve probably already finished your morning coffee and are ready to move on. If you have any other ideas or anything to say on the matter, I’d love to hear from you. You can leave a comment or contact me directly.

The group finally gets a chance to sit down and speak for the first time together about their experiences and adventures from GDC 2013. Warning: discussion of hugs and warm fuzzies throughout the podcast.

Relevant Links:

Game Developers Conference
Game Audio Podcast
IASIG
G.A.N.G.
Wwise
FMOD
Starseed Pilgrim
Bad Hotel
140
Ariel Gross’s 2012 GDC Talk: “80,000 Lines, Three Lessons Learned”

Theme song by Ariel Gross
Edited by Ariel Gross & George Hufnagl

#define

What is your background and education?

I’m originally from the North-East of France (more precisely from the region where they make the Champagne) but I studied in Paris where I obtained a diploma from a computer engineering school with a specialization in artificial intelligence, digital signal processing and automation. My final project was actually sound-related as it was the simulation of the singing voice. It was using different methods to analyze and synthesize vocal signals. The main system I built was called DiVA (short for Digital Vocal Artist). It was using a DSP chip from the Motorola 56000 family for the sound synthesis and was connected to a microcontroller responsible for parameter input, display, and MIDI interfacing. A few knobs were controlling the formants frequencies and bandwidths, the amount of vibrato, etc. Of course at the time that looked a bit more impressive than it would now, as resources about this kind of things were pretty much nonexistent.

How long have you been involved in audio programming? What games have you worked on?

I have been involved in audio programming professionally for more than 20 years, although it started many years before that as a hobby (I guess I’m showing my age here ). The first commercial products I worked on were sample editors, small MIDI utilities, audio file converters etc… They were developed on Amiga and everything was coded in assembler and blazing fast! Then I started working on commercial PC software around the time of Windows 3.1 and focused more on sound design-related applications. My main project at the time was a modular software synthesizer called Virtual Waves. It was pretty good at generating really weird sound effects so many clients started to come from the game industry and I worked closely with some of them.

I made the jump to the game industry itself when I took a job at Factor 5 in California. There, I worked on the Star Wars: Rogue Squadron series as well as on the MusyX audio middleware and on some DSP for the GameCube itself, as Factor 5 was working closely with Nintendo on the audio system of that console. I remember that on “Rogue Squadron: Rebel Strike” I managed to convince the game designers to sit down with them and to add a second – simplified – geometry of the levels for the audio in the editor, tagging it with the appropriate materials. The reverb I developed for MusyX was a variation of the Schroeder reverberator model (not enough cycles for a convolution reverb!), so including both comb filters for the early reflections and allpass filters for the dense reverberation. In the code, I was performing a very crude audio ray casting using that simplified geometry and the material absorption coefficients. Then I calculated the parameters of the filters at run-time from that. At the time, it was really awesome as an audio programmer to be able to do this kind of stuff! The people at Factor 5 took the audio very seriously which was great.

After Factor 5, I moved to Konami in Honolulu where I worked on all kinds of games from “Frogger” to “Dance Dance Revolution” and on many platforms. The next stop was EA in Burnaby, and finally Sony in London. In both cases I was working in central services, so your stuff ends up being used in many games and by many teams around the world, sometimes without you even knowing it. For example, I recently was told that I had been credited on the last “WipeOut” for Vita and that was quite a nice surprise because I just wrote a small audio analysis tool for them. But it definitely made my day as I have always been a huge “WipeOut fan”!

What inspired you to work in the audio field, and specifically programming?

I started programming on the Sinclair ZX81, like a lot of French kids from that period I suspect. At the time I was 11 and I was very much into electric organs and synthesizers. My dad was teaching electronics and was very interested in personal computers so he thought it would be great to order the ZX81. We ended up having a sound card for it and I spent a lot of time playing with that. That’s probably how it all started.

While I moved to other computers such as the ZX Spectrum, Apple IIc, etc., I continued to play with the audio, but I was also interested in many other things, especially artificial intelligence. However, when I switched to the Amiga with the famous Paula chip and its four channels of digital audio with modulations, things became more serious. There were all these trackers and other audio programs. At the same time, the electronic music scene was really booming in France so what started as a hobby slowly evolved into a passion and a career.

#include

What programming languages would you say are essential to have? Do they vary from project to project?

Indeed, it depends on the platforms you are targeting, but if you want to work in the game industry on AAA titles on platforms from the big three, C++ is pretty much a given. Then of course something like C# can be useful to quickly write tools. Knowledge of at least a scripting language such Lua and Python is helpful as well. Whatever language you use, it’s important to understand its strengths and its limitations, and to know how they will impact your project.

That being said, I’m not really a huge advocate of a language in particular. Like everything else, you should just use what works best for your project and within your constraints. Sometimes, people who are very knowledgeable about one language tend to use any little trick in the book, which makes it pretty hard to read and to maintain for the rest of the team and ultimately costs more to the company. In most cases, I believe it’s more important for an engineer to have good basics in mathematics, physics, computer architecture and algorithms and to be a creative problem-solver than to be a guru in a given language.

Provide a brief definition of each of these functions:

Oh, I can think of about a gazillion of serious things that should go in there, but since you asked me to be brief…

int main()

{

Do you mostly work with proprietary tools or third-party middleware (like FMOD, Wwise, Miles, etc.)? What are some benefits and challenges of each?

In my career I have exclusively developed and worked with proprietary tools and engines. This is due to the fact that I often ended up working in central services which were big enough to be able to develop their own technology or on R&D departments. That is also the kind of work I was looking for. But of course I have also often examined the features of the available middleware such as Wwise or Fmod. Also a notable exception is actually now since at tsugi we are – among other things – assuring the technical support for AudioKinetic in Japan, so obviously we have engineers who know the ins and outs of the Wwise API and how to use the authoring tool in detail.

One of the major advantages of having a proprietary audio tool and engine is to be able to drive its development in the direction you want, which is adapted to the type of games you are doing and on a schedule that is compatible with your studio’s releases. It’s also easier to interface it closely with the other tools and run-time components of your studio such as the physics engine or the animation system. Finally, when something is not working, it’s easy to just jump into the code and debug, without having to wait for a fix from the middleware provider (although these days you can sometimes get access to the source code, depending on the type of license you have).

However, nowadays developing your own audio engine and associated tool is a big investment. It can be very demanding in terms of human resources and time, not only to develop it but also to maintain, document and update it. Game audio has evolved and it’s not only about playing back a sample anymore, you need to offer a way to script sound effects, to manage 3D emitters and occlusion, to do real-time processing, to have a mixing system, reactive music and so many other things… Also, tools often take a long time to develop and to mature into something that has a usable workflow and is stable. In the game industry where the end product is the run-time, people often underestimate the amount of time required for the development and maintenance of good tools.

So unless you are doing something very specific audio-wise, or you are a big studio with already an audio system in place, it’s probably safer at this point to pick an audio middleware than to start developing your own. Actually, I even know several big companies which had internal R&D audio teams in the past and have recently started using commercial middleware, or at least have given their teams the choice between the in-house tools/engines and a 3^rd party offering.

You’re a strong proponent of procedural audio. What can this do to help advance game audio forward, and what role do audio programmers have in it?

Basically, procedural audio is to sample playback what 3D is to sprites. It allows you to deal with that one single asset that will be played differently depending on the context. If you take a 3D object in a game, it will appear differently depending on the rotation, the scale, the lighting, etc.  If you wanted to do the same with sprites you would have to draw many of them – from different angles, with different sizes, and colors. And of course you would not be able to recreate the infinity of possible views of that object; you would have to limit yourself to a subset of them.

That’s what happens with samples. Let’s say you want to play an impact sound, you will have to record many of them, with different materials, on different surfaces, with different forces etc. With procedural audio you can just create a model of that impact and synthesize the corresponding sound at run-time after selecting the right parameters. Moreover, if it’s well connected to your physics engine, you will probably be able to generate many contact sounds as well with the duration and intensity you want without having to record anything.

Procedural audio allows the sound to be more reactive and better integrated with the other subsystems of the game. Because the sound is generated at run-time depending on the context, you can create an infinity of variations. Therefore procedural audio helps fighting the repetitiveness due to sample playback and of course also saves memory as you would need to record a lot of samples otherwise to provide even a limited number of variations. That being said, it’s not a silver bullet that will solve all your audio problems. There are a lot of interesting things you can do with other methods to get a rich and interactive audio experience, and it does not apply necessarily to all situations either.

As for the role of audio programmers, since it’s a more technical approach than recording samples, they are responsible for building new, adapted tools and workflows, and for educating sound designers about synthesis. I don’t think they should be solely responsible for creating the models though.

What are some of the challenges in using more procedural audio into video games, and what are some ways we can overcome them?

Procedural audio is still in its infancy. Making the creation of the procedural models easier and accessible to the sound designers is primordial. Focusing more on the final sound than on the production mechanisms will allow us to build higher quality models. Until now, most of the models have been created using a bottom-up approach. Although this makes sense in an academic or research context, it is ill-suited for creative work, especially in a production environment because it requires more knowledge from the sound designer, more time spent building the model, and without offering any guarantee that the resulting sounds will correspond to what the sound designer wanted in the first place.  Although they generate recognizable sounds, most of the models created that way are nowhere close to the quality you would expect in a game. That’s why I have been advocating a top-down approach that takes existing sounds, analyze them and create a model from them. That model can then re-synthesize the original sound the designer wanted as well as many variations. However, this is clearly a more complicated approach to develop.

Generally, procedural audio is progressively gaining traction. I think we will soon be in an interesting position when the newest generation of sound designers and programmers – more aware of procedural audio (and certainly more excited about it from my discussions!) – will accede to more senior positions; and when more researchers will have joined the game industry who understand the context in which procedural audio has to work, better tools and models will then start appearing.

You’ve also spoken, and written about, dynamic mixing. How will this make audio engines “smarter”, and thus provide a better sound experience for players?

While at Sony I spent some time thinking about spectrally-informed audio engines: basically, making the audio engine aware of what it is playing frequency-wise and allowing it to make the best decisions for the game at a given moment. The idea is to analyze the audio assets in the frequency domain either at run-time or before-hand on the tool side, and then have the engine keep track of what it is playing and where in the listening space. For any given frame, you end up with a frequency/location map, with a level in dB for each point.

This makes dynamic mixing possible: you can define mix profiles or targets and then have the engine slightly adjust the mix based on what is actually happening in the game. Other interesting features can be implemented such as perceptual voice limiting or audio shaders, for example.

Do you feel there is a mutually creative link between audio programmers and sound designers/ composers? What can we do to help the creative process of sound designers, and what can we learn from them that we can apply to our efforts?

Yes, definitely. It’s often by bouncing ideas between audio programmers and sound designers / composers that you come up with great new features for the game or the technology. Audio programmers often act as the link between these creatives and the other programmers on the game team, especially when other systems are involved.  One of their roles is indeed to facilitate the implementation of the sound effects / music (and in addition, they will need a good dose of diplomacy to negotiate these few extra cycles or kilobytes…).

But they also should be here to educate the sound designers about what is possible to achieve with DSP, sound analysis and synthesis, or even just in terms of interface / features for the sound tools. You can’t expect a sound designer to ask for a new feature if he or she does not know what is achievable technically. On the other hand, sitting with sound designers, examining their workflow, and having them describe the issues and bottlenecks is probably the best way for an programmer to understand what the tool should do and how it should do it.

Can you think of any other audio features or technologies that have yet to be implemented in games that can add to the player experience? Or perhaps existing ones that can be improved upon?

We talked about a few of them, like procedural audio for which we need better models and tool suites, and dynamic mixing. Better simulation of the acoustics of closed spaces also comes to mind. At tsugi (www.tsugi-studio.com) this is part of our daily work: imagining how game technology (and especially audio) will evolve and what will be needed. For example, we are already thinking about what will happen after procedural audio or more precisely once most of your assets are patches or models instead of sample files. This brings some interesting challenges and requires new concepts. We also do audio pipeline audits. We visit clients, sit down with them and examine the workflow of the sound designers or the architecture of the audio engine with the programmers. Then we make recommendations about optimizations in the engine, or new features for the tool and we can also implement these changes ourselves if needed.

Although I can’t really go into the details of our projects here, there are a few obvious trends. One of them is that with the ever-growing amount of assets used in games, and the size of the sound databases, we need new ways to browse, to visualize and to select data, as well as to perform quality assurance once the assets are in the game. For example, one of our research projects, tsugi DataSpace, uses audio features extraction and self-organizing maps to visualize thousands of audio assets in a 3D world. It also supports gesture controllers such as the Leap to make the experience even more immersive. Because similar sounding assets are classified together it opens new possibilities, like finding appropriate sounds to layer together for example.

In the same area, i.e. dealing with numerous assets, the availability of full-featured, stable, batch processors is very limited and none of them boasts features dedicated to game audio development, so that’s why we built AudioBot. It offers all the functions of a regular batch processor but can also automatically create files compatible with game audio middleware such as Wwise work units or FMOD projects. It supports proprietary importers, exporters and processors through plug-ins so you can use your studio’s technology, you can edit chains of VST or VAMP plug-ins, it has a command line version to integrate with your audio pipeline etc…  It has really been made with game audio people in mind.

But to come back to your question, in general, I believe that any technology which makes either the tool or the engine a bit smarter and therefore improve the workflow of the sound designer or the quality of the run-time would be a welcome addition, thus the importance of audio analysis.

How do you see the demand for audio programmers evolving in the industry? It seems most positions are in mid-size to larger studios. Can smaller companies benefit from an audio programmer on the team?

This is closely related to both the evolution of game audio technology and the definition of the audio programmer. On one hand, as we talked about it earlier, it’s becoming harder for a studio to justify developing an in-house audio solution, because of the complexity and the costs involved. So it would seem to confirm your feeling, that only mid-size or larger studios will be able to do this and actually hire audio programmers. Smaller studios would usually use audio middleware and therefore allocate a generalist programmer to load banks, trigger the sound effects etc.

This brings us to the second point which is the definition of an audio programmer. In a lot of cases, we see job offers for audio programmers which involve very little more than just hooking sounds in the game. But in my opinion this has very little to do with game audio. You could be loading and triggering something entirely different than audio assets. An audio programmer for me is someone who can design audio systems, offer audio-centric solutions to problems, propose audio-related gameplay to the team and so on. So if you take that definition, then yes an audio programmer can definitely add something to any game, even a small one. But of course it’s a question of priorities and most likely the finances will go towards hiring another graphics engineer anyway .

But from talking with recruiters and other people in the industry, it seems audio programmers are still very much in demand, also because there are not so many of them to start with.

class FFT

{

What are some of your favorite DSP effects to use and/or design?

There are always new parameters to add or new tricks to use, even for effects as simple as a flanger for example. In that sense, I guess I don’t really have a favorite effect to design because they all offer countless hours of fun! However, I am very interested in effects which first analyze the sound – either by using a FFT, calculating MFCCs or any other feature extraction method – and then process it differently based on its characteristics; basically some kind of audio shaders.

List some of your favorite sound effects that have been procedurally generated/synthesized:

I usually mention the work of Staccato Systems in the early 2000s. They had really nice procedural footsteps and car engines, a long time before everybody. More recently, I worked on impacts / contacts sounds for Sony and got some nice results. Also I have been told that AudioKinetic’s SoundSeed Air has been used in quite a few titles and I like their presets. However, sounds such as impacts / contacts, wind, rain etc., are still relatively easy to simulate.

I have to admit that until now, there hasn’t really be a sound effect for which I learned after the fact that it was generated procedurally and told myself “wow, how the heck did they manage to model that?” I’m really looking forward to this moment though! One of the projects we are currently developing for a client at tsugi could definitely belong to that category… if we ever manage to make it work of course!

};

return 0;

What have you learned about audio as an audio programmer?

After all the mathematics have been studied, the filters designed, and the anti-aliasing programmed, there is still a space to experiment, break the rules, and where you can find this ridiculous little trick that will make your engine , plug-in or tool sound awesome. There is still this magic part into it and it’s great. That, and a plug-in with a great GUI will always sound better .

}

Name:
John Born

Company:
Born Audio

Position:
Owner / Operator

Did you attend school for an audio-related degree?  If so, what school and degree?
School:  Vancouver Film School
Degree:  Sound for Visual Media

School:  University of New Brunswick
Degree:  Multimedia Studies; Diploma in University Teaching

What inspired you to work with sound?
My love of music quickly became a love for all things relating to sound. I bought my first Peavey microphone and Tascam 4-track recorder in 1991 and I’ve been in love with recording and manipulating audio ever since. 

How old were you when you found out sound is what you wanted to do for a living?
16 years old. I work in sound, the Education field in Instructional Technology and as a Multimedia Instructor, but I’ve been lucky enough to combine all of these passions together.

Was a school degree the first thing on your mind, or do everything self-taught?
I’ve always wanted to do a combination of study and work on my own. Over time, I’ve realized that new areas of your art can open up when you start to collaborate with others in your discipline. My first degree was in Multimedia Studies at the University of New Brunswick and I did a diploma in University Teaching before I started the sound program at Vancouver Film School (VFS). Since I graduated from VFS, I’ve been finishing up my Masters in Education, so for me learning is a continuous journey.

Bass Boost

What is your specialty/preference of the sound fields (sound design, music, recording, audio programming, implementation, etc)?  What do you like most about it?
I’ve been in love with field recording and production audio for film lately. Since I’ve been back in Vancouver, I’ve had a chance to work on some great films with directors really interested in getting the best production audio possible. My passion for training and teaching others about sound will never go away. I’m as excited about working with others to develop their passion for sound, as I am my own. I recently gave a live audio webinar to the Educational Technology Users Group here in Vancouver. The presentation was called “Sound in Education,” about the application of sound to enhance online learning.

What sound tools did you learn in your school curriculum?
The heart of the school is based around Pro Tools. I had experience with Pro Tools before I enrolled at VFS, but my instructors helped me take it to a whole new level.  The hardware interfaces connected to Pro Tools that I worked with included the MBox, 003, Control 24s, and a duel ICON D-Control ES in my senior term. Also, I’ve worked with Reason, KOMPLETE, programming tools like Max/MSP, and game audio tools like WWise and FMOD.

What kind of projects did you have in your classes?
The projects varied from term-to-term, but were primarily based around post production for TV and film.  We spent a lot of time as editors working on building all aspects of post sound, like Backgrounds, Foley, Sfx, Spfx, Music and Dialogue. The teachers are coming right from industry, so they are bringing you projects to work on that directly apply to the type of work you would be doing as a job out in the field. We were also encouraged to take advantage of all the recording equipment available to start creating our own sound library. VFS has an extensive sound effects library, but it’s a great feeling going out and building your own library for projects. I had a chance to work on some great game audio projects, and as you progress in the program, you start doing more project mixing on the Control 24s in the 5.1 mix labs and with the duel ICON D-Control ES.

Were your teachers audio professionals?  Anybody the audience would know?
All the instructors are audio professionals with extensive backgrounds in film and TV. I had some amazing teachers at VFS:  Shane Rees, Steve Smith, Gary Bourgeois, Chris McIntosh, Brad Hillman, Anke Bakker, Curtis Wright, Tim McGuinness and Jonathan Fish.  VFS has a lot of the top industry professionals come in to run workshops and provide training. I had a chance to spend a week training in production sound with Ray Beckett and the Ear Candy crew out of LA. Sound designer, Craig Berkey, came in and out of the school for presentations and workshops. Craig was on my Industry panel for my final project and he gave me feedback on how to improve my sound design and mix.

Plug-Ins

Did you do any side projects during school?  If so, what were they like?
I did as many side projects as I possibly could. I took on a few extra audio projects for a web series, animations, and I worked on a game design project called Ginko. Ginko ended up being nominated at the “2011 Canadian Videogame Awards” for Best Student Game.

How many of your side projects were published?  Any of them turn profitable?
My final sound design project was published on the VFS YouTube channel and Ginko received a lot of attention at the Canadian Videogame Awards.

Echo

How large was your graduating class?  Were you all close?
There was eleven of us including myself. We were all very close. You live and breathe sound together for a full year in close quarters, so you have your up and downs, but there is a friendship, appreciation and respect that grows from going through a challenging program together. Each one of us had our strengths and weaknesses in the program, so you would connect with classmates that could help you in your weak areas.  One of the great things about the VFS program is that there are classes behind you and ahead of you that are easily accessible. Most of the time you could grab someone in a senior class and discuss how they approached assignments, but someone just coming into the program could easily have skills that could help you develop, so when I refer to classmates, I’m thinking of all the great relationships I made with classmates during my time at VFS. 

How often do you work with your old classmates today?
I’ve built some great working relationships with my classmates.  Regardless of where we are in the world, we try to connect on projects and keep helping each other out.  Recently, a classmate from a senior term helped get me a job as a Foley editor on my first feature length film. At the time, we were on opposite ends of the country, but we stayed connected through Skype and Dropbox to make the project work. A short film I’m working on now has classmates that are in the UK, Mexico and here in Vancouver. We can all connect through Facebook, Skype and Dropbox to share edit sessions.

Any old classmates you want to mention?  The more the merrier with the audio community!
Always a big hello to my class SD44 which was my main crew, but also to Scott Schrum who has been helping me out with some great set sound and post production work here in Vancouver.

Amplify

Do you feel more prepared for the sound industry than if you had not graduated from your program?
Absolutely! I have been self-taught and working on projects for years, but there is nothing like setting up a situation where your only work is your chosen craft. There is something about waking up every day and knowing that you’re working with 11 other sound designers, who want nothing more than to develop their skills and become better at what they do. There is often a debate about whether taking a sound program, or just starting to freelance is better than the other, but I think it all boils down to the individual. You only get out of a program what you put into it, If you work hard in a program, set some personal goals to achieve, network and build relationships, I think the sky’s the limit.  I worked in sound for 19 years before I decided to take the VFS sound program. For me it was an incredible experience that helped fill in a lot of the gaps that I needed in my work. 

Do you have a website for your portfolio?  How often do you blog on it?
I split my work up across several different sites, but my central site is my Word Press: http://bornthinkers.wordpress.com/ I try to blog on it as much as I can, but I find using Twitter and Facebook is a quicker and easier way for sharing.

Do you use social networking?  How often, and what communities?
I use as much social media as I can to promote myself.  I use:

Linkedin: http://www.linkedin.com/in/johnborn
My Space:  http://www.myspace.com/johnbornca
Vimeo:  https://vimeo.com/johnborn
Twitter:  https://twitter.com/bornaudio

Fade Out

Any last words for future audio people looking to carve their education and career paths?
In most cases when you enter in a sound program, you have to go in with an entrepreneurial mindset. Most of the people I know in the industry are working freelance and wearing as many different hats as possible. Do your best to stay sharp on a wide range of tools and techniques and always keep developing your skills. The great thing about working in sound is you have a lot of different areas that you can focus on. I try not to keep things locked down to just one aspect of sound. I’ve always been interested in production and post-sound, but I’m just as interested in providing training services for individuals and companies interested in incorporating sound into their business and brand. Persistence and a positive attitude go a long way working in sound. Things aren’t going to happen overnight, but if you keep at it and stay positive you can bet some great things will be heading your way. Lastly, it’s incredibly important to network with as many people as possible. Most of the work I’ve been getting has been through referrals by classmates and people I’ve met along the way. I always make it a habit to introduce myself to as many people as I can in the industry.

About Sonic Backgrounds

The sound industry is an ever growing field, ranging from linear sound design in film and TV, to interactive audio in games, and from live theatrical sound design to field recording for the creation of custom libraries.  It is only recently however, that school programs have begun to offer degrees in the sound-specific variety.  Graduates of these new programs are now coming into the industry, and it provokes the interesting question of how these new, specific programs are preparing individuals for the sound world, as opposed to the older approaches of entry, such as pure passion, musical talent, a film degree, or a computer science degree.

“Sonic Backgrounds” is an interview series focused on interviewing recent graduates of these educational sound programs around the globe, to see what exactly they are providing, and how they are shaping the new “academic”-based sound artist.

It’s that time again…

PlanetSide 2 - But what about the music?

Joystiq had an interview with composer for PlanetSide 2. Find out more about Jeff’s inspirations and the challenges of scoring an MMO. If you missed the full PlanetSide 2 game audio run-down last week, be sure to check out this post as well.

And then he SPAD all over the living room floor

Charles Verron  posted a free  downloadable Spatial Additive Synthesizer (SPAD). The video below shows SPAD in action procedurally generating environment ambiance and footsteps with 3D positioning. Inch by inch we crawl out of these sample-based ways.

Electrostatic Graphene Loudspeaker 

So what is the deal with Graphene? Well, this flat monolayer of carbon atoms, tightly packed into a two-dimensional honeycomb lattice, has excellent transductive properties. Some research types at MIT are sandwiching this material between silicone and creating speakers with solid frequency responses. All signs point to a future with smaller speakers that have less crappy sound quality.

Sound and Water Experiments

Watch this video to find out what happens to a falling stream of water when hit by high frequency sound waves. This is better than that rubber pencil trick I insisted on showing everyone at work.

Floisan-modulation

CreatingSound’s very own Christian Floisand released a ring modulator VST  named Dyner. Christian is a descendant of an ancient line of  VST craftsmen. They hammered the zeros and ones into stone back in the day, or at least I imagine sometimes. Seriously though, take Dyner for a spin, it’s free and PC/MAC compatible.

Preview

I absolutely adore collaborations. Be they with game developers, animators, fellow audiophiles or creatives of any stripe, I love the social nature of the experience. Brainstorming, exchanging ideas, supporting one another, learning about a new field, critiquing to improve the project; not only can it be a great relationship builder, but you can learn more about your own discipline and how others perceive it. Growing up, I participated in a variety of  team-centric activities – baseball, marching band, jazz combo playing Chuck Mangione tunes – so either my spirit of camaraderie has been cultivated over time or perhaps it’s in my nature. Or both. Either way, I think I’ve made my point. George loves collabs.

In 2009, I began shifting my focus from writing music for the concert stage to sound design with a primary focus on game development, so naturally, I looked forward to working with other creatives.The shift was in the cards for awhile and there was no better time than now (that is, the “then” now and not the “now” now… stay with me here). Of course, never having worked in this medium before, my initial experiences were pretty awkward. I was learning all kinds of new tech and nomenclature, virtual teams formed and dissolved by the month, and I had no context for what the sound designer’s role was in the creative process. I knew that, in the most basic sense, s/he is responsible for the creation and delivery of audio assets, but there were many questions beyond that:

• Who holds creative authority over the project?
• Do I get a say in the game design’s aesthetics?
• Am I able to offer design ideas?

Depending on scope of the project and team size, this inquiry can be quite extensive. For me, then, this was one of the most anxiety-producing points in a collaboration, especially when it’s someone with whom I’ve never worked. One never knows if the person will be open to your ideas or if it will be a situation like in “School of Rock”:

Dewey: “But Ned, no power plays, man. I’ve got vision up the butt, so just go with it.”

Fast forward to “now” now. Having had many wonderful, collaborative experiences since those first wobbly steps, I am much more comfortable with myself and the development process. It’s different every time at least in small ways, of course, but I did wish, though, that I had had some knowledge of how a collaboration might go or, at the very least, some solidarity in that awkwardness. Therefore, I felt it would be beneficial to share a slice of my experience with a real-life example. For those of you who are either venturing out into this area for the first time or who like to see how others work, I salute you.

I’ve chosen to highlight an app I worked on recently with Jacob Van Order of SushiGrass called PowerUp, A Retro 8-bit Video Game Camera.

This example works well because the audio asset list was small, the collaboration had a quick turnaround time and it can be demonstrated how the collaboration fared with some detail. First, Jacob describes the point at which he and I were introduced.

I had taken off work 3 whole months in order to work on PowerUp full-time with the product being an app that was 100% my vision from concept to execution. I spent weeks detailing the visual assets that went into the app. I obsessed over fine details and animations. I tested, retested and beta tested until there were no known ways I could crash the app. All this time went into the craft of the app because I wanted to put 100% effort into one of my products. But some of the feedback I got consistently was that, considering how playful it was, the app was “too quiet”. And I had to agree. After all, sure, I spent numerous hours as a kid staring at the screen while playing my favorite games but I spent just as much time having the songs of Mega Man 2, Super Mario Bros, Duck Tails and other games stuck in my head.

At this point, I tried to go at it alone. After having an app that was conceived, designed, programmed and constructed by myself, the last thing I wanted to do was lose control of the sound effects. After some recommendations, I tried a programmed called “CFXR” to try to eek out what I thought would represent the sounds of the era my app represented. And it worked… sorta. Slowly, I saw myself doing something that I used to disparage — I was assuming that because I had a tool, I was instantly a craftsman. Coming from a job in graphic design, people would approach me with slipshod representations of what they were trying to communicate with something they “designed” in PowerPoint. There were no considerations of the 7 elements of design, no years of experience backing up the decisions and no considerations of craft to produce a seamless product. I recoiled in horror as I realized that I had become the very monster that used to hand me a print out and tell me to “make it look pretty”.

Again, on another recommendation (shoutout to Benedict Fritz!), I was introduced to George. I sheepishly handed over my crayon-drawing equivalent of sound files and gave him examples of something similar to what I wanted. Again, coming from a design background, I knew how important it was to communicate your intent upfront but letting the subject-matter-expert take it from there. And though this was tough as the app was, up until this point, my 100% vision, I am so glad I did. George listened intently, worked quickly and efficiently to come up with sounds that blew me away. He had the knowledge and foresight to produce sounds that went way beyond what I had envisioned but conveyed what I wanted to communicate within the app better than my crude facsimiles. The result was something that felt right in place with a controller graphic that took 3 days to complete and code that was endlessly tweaked. This was my first time working with a sound designer but definitely not my last.

*Raises Hand* Sound designer’s honor that I did not ask Jacob to pen those very flattering words. You can tell, though, that we had a very pleasant experience and have developed a mutual respect for each other. After all, when I first opened the app I, too, was impressed with how well Jacob had thought through the presentation and flow. When you can operate something easily without instruction, then you know someone took their time in development. How did it go so well? Here is how we got there, together.

When Jacob sent the initial build of the app to me, it sounded like this:

As Jacob does with his own discipline, I find it always helps to encourage people to provide sound references when they have an idea of what they want their project to sound like. Wanting to get a sense of what sounds seemed nostalgic to him and appropriate for the app, I asked him to send a list of sound effects in addition to what he implemented above. So, after spending a goodly amount of time playing with the app, figuring out its logical flow as the user might experience it, and listening to the audio both in the app and on the reference list, I sent an email to him with my thoughts:

I listened to the sounds and, while I think you have some great ideas to represent the nostalgic appeal of the app, I was wondering how you’d feel if we layered in some acoustic elements to help give them shape. For instance, on the shutter, I think it might be helpful to have an initial click, like a standard film camera, but then dovetail into something like you have as the reference. I only say that because the touch UI might benefit from a tactile response in the sound. I could surely work to represent that in the core sounds, but bringing in some [non-synthetic] sounds could help give them a little more weight. Let me know what you think.

Jacob’s response:

Thanks for sending me the idea. I suppose I’m in a funny position. I guess I’ve always assumed that it would have a strictly synthesized sound as I have an affinity towards that sound.

But then again, visually, the app is, a mix of the skeuomorphic (bottom) and 8-bit (top) visuals by design. If I can mix those two styles in a way that works (in my opinion), then why not the sound, right?

I suppose that because the shutter is such a crucial part of this app, we could do this: why not try your approach for just the shutter. If you think the explosion sound is wrong, try something else. I will say that it will be an uphill struggle for you to change my perception but I will keep an open mind. If I like it, you can do the other 12 sounds in that style. If not, then I’d prefer a strictly synthesized palette.

This was a perfect response from Jacob. While I wanted to still honor his original plans for the app, his openness to my own creative contributions allowed me room to experiment and really take ownership of the sound design. It was the second paragraph in his response regarding the combination of nostalgic/contemporary design that resonated with me and suggested that we could do something similar with the sound design. After all, since the app’s design and camera filters do not reference just a single point in time, why couldn’t the sound design follow suit? We had the time and there was wiggle room in the budget, so off we went!

With only about 5 days to finish the project, we rapidly exchanged audio and builds, tested and tweaked, critiqued and encouraged, making sure that we were both satisfied with how it sounded. We even added new sounds that weren’t even on the original list!

Here is a sample of what it sounds like now:

(Motion design by Dan Fitzgerald.)

I’ve thought about why this process went so well and why we had so much fun:

1. We each presented well-reasoned, creative ideas. 
2. We held a mutual respect for each other’s abilities. 
3. We listened carefully and responded in kind to one another’s opinions.

And BOOM, you have a great combination of elements to make it happen. While this project briefly demonstrates how two people might work together, it serves as a great example of what I get out of doing this work. In other words, while the first two points above will get projects done, No. 3 builds relationships. I hear it over and over again – “it’s all about the people.” In a creative field like this, that’s what keeps me going and reminds me of how great collaborations can be. 

Creating Sound: Rob! What do you think the futuristic cyber-audio designer’s responsibilities will be in five years? Ten? Ten thousand?

Rob Bridgett: I think your responsibilities, and whom you report to, the kind of work you do, all depend on the culture you work in. I’m thinking how differently organized EA sports is from say, Media Molecule or Double Fine and I’m wondering how different things are from five years ago in our industry. Not that one is any more or less ‘creative’ or ‘technical’ than the other, just that their structural differences are necessary to the end product. I’d say in some cases, there is no difference at all, even going back 10 – 15 years, and I’m thinking that time might not be as useful a reference point as we might think. I think all the significant changes have come from the people working in the industry, and the types of games being made, and if you looked at those games outside of release date, you might think them from a previous generation. Games like Limbo, Soundshapes and Journey have changed things so much more than the technology has; they are new art forms, new business models, and herald new ways of designing and integrating sound into new experiences that the hardware manufacturers couldn’t have dreamed of. There are so many wonderful and innovative games and ideas coming through now, that they feel like creative opportunities exploiting the cracks between technologies.

Digging in a little on the audio technology side, I think the enrichment of middleware will continue and in doing so set a lot of new ideas free. This I see as incredibly positive, accessibility of tools and simple, solid interfaces that allow deep experimentation and customization were one of the fundamental ‘missing parts’ of the audio development landscape. There is also so much tutorial content and shared knowledge in this area, that it is now so quick and easy to get interactive content up and running. In the past I’d have struggled to give advice to anyone from a linear sound background as to how to best get a feel for game sound, now you can just point them to download Wwise and the tutorials or Fmod TV on YouTube and they’ll not only be able to hear and read about it, but build their own projects with their own content, hands-on, for free! You can’t even do that with Pro Tools or Nuendo, so we are ahead of the pack there in terms of accessibility. Again, I think this accessibility to enriched interactive tools opens up the craft and technology to people with very new and different ideas and uses for these engines.

To your first point, do you think that a studio’s culture influences the final product?

It is the biggest influence. The team and the culture operate on all levels of production, from attitudes to overtime to innovation. The Kixeye recruitment video really sums these things up nicely and puts a bow on it. It will make you laugh in recognition at this notion of the ‘old ways’ of game business culture. However, if you want to take this kind of thing to its logical conclusion, I recommend looking again at the Charlie Brooker / Chris Morris TV show, ‘Nathan Barley’. This is why I think it is important for audio to be represented at the cultural level in a studio, if it isn’t, then of course it will be neglected. Sound needs a seat at the table.

Going back to your second point about audio technology, with the barrier of entry into game audio getting lower and lower, what new qualifications will emerging for the audio designer?

So, with the access to the technology getting lower, accessibility to publishing platforms is easier if you have a small savvy team of coders and designers with a really great idea, they can make that happen at a relatively low cost. They’ll still need some business and production savvy, but they don’t need to go through the publisher sign-off and pitching process, which is good if the idea is focused and clean, but there are new challenges with that model, in that you can just put out an underdeveloped, buggy product. Sure, feedback and milestones can sometimes take the ‘edge’ off a product, but they also give it a more polished feel, and there are lessons to be learned in that, seeking feedback, setting regular milestones, that’s all a healthy part of production – of course some teams can’t and don’t work like that, but that polish process has to happen in there somewhere. In terms of audio production, there are a whole range of ways that audio can be integrated into indie games – from outsourced lists of sfx and music that the developers hook-up themselves, to on-site implementation and audio design roles. The former is a model that triple-a development studios have slowly moved away from (on the whole). There is a whole ecosystem of studios and cultures with different ways of bringing audio into their games, and I think that diversity is one of the really interesting and challenging things about our industry.

Anyway, to your point about qualifications, I think personally any qualification in any discipline will give you the perspective of working on something and finishing it to a high enough standard, the other essential thing you’ll need is experience in collaborating with others on projects. I’m not necessarily talking about paid work for a developer, although that experience would be hugely valuable, I’m thinking of examples of projects you’ve worked on with other people, where you can identify your role, talk about the process and decision making on that project, how feedback was gathered and prioritized, and how the final ‘thing’ was completed as a mutually accountable team – that experience could come from anywhere, be it a mountain climbing expedition, or publishing a small journal. Speaking from my own perspective, I’m always very impressed, and inspired, by people who are professional, respectful, and have a good sense of humour – that goes so much further than having shipped 5+ AAA titles that have reviewed 80 or above on metacritic. Soft skills are in short supply and high demand, but again, it is a balance between those soft skills and experience, and understanding of your craft. I think Valve described seeking T-shaped people – meaning having broad overall experience, which is the top of the ‘T’ – coupled with a deep specialization, which is the vertical part of the ‘T’ – it is another interesting way to evaluate holistically for a role within a team and particular culture, rather than trying to specifically solve your needs for a ‘technical sound designer’ or an ‘audio programmer’. Damian Kastbauer has said ‘it’s all about the people’ – and he’s totally nailed it – ‘it’ being making games.

Let’s say someone is an experienced sound designer. They can make kick ass sounds and implement them into the game. What should be their next priority?

From that description, it sounds like a very accomplished technical and artistic person. My own advice would be to encourage and help someone like that, who was actively looking for ways to develop, to deepen their human relations skills and social skills, and try to balance all three of those skill areas equally in their work, so that every typical week would involve all of those areas in equal measure – So, rather than designing and implementing all week, they’d be spending a couple of days of that week sitting down with designers or artists and gathering feedback and opinion, testing out their sounds with other collaborators and reviewing them in context. Learning deeply about other disciplines is also a great place to develop both other perspectives, understanding or processes and dependencies and different ways of communicating, mentor with a designer or animator, it’s very interesting to find similarities as well as differences in how we work; we deal with a lot of the same issues – and no matter what our discipline, we are all trying to solve either design problems or production problems with our work and workflows. I’m personally obsessed with listening to and reading about graphic design and architecture, as I find so many easily applicable ideas and concepts there. Bringing it back to sound, Randy Thom said, in an interview with Designing Sound, that “I sometimes think of the work as being about 33% art, 33% craft and 33% human relations.” – I think this is an utterly brilliant observation, and, for me, represents the perfect balance to strive towards for an individual working in any collaborative job, but especially sound. I also think that this formula is a great way to balance a team with various strengths, for example a gifted technical or implementation expert, working with someone who is a very creative sound effects designer, both working with someone who can explain and communicate with the rest of the team etc. It remains loyal to that formula. I find this quote from Randy is ‘a world within a grain of sand’ in terms of how it can be applied at various levels to sound teams.

Is there an area where you think audio designers are coming up short? Aside from our literal height. It feels like we’re at least an inch shorter than producers and artists on average, depending on the heels.

This isn’t just audio, but I’d love to see developer diaries or focus features where several disciplines get together and talk about the game or the thinking and process, rather than sitting down to single out sound or music, or dissect a single element of the production like the camera – it is a very one dimensional approach, very flat, it promotes segregation of the arts – I think presenting that viewpoint misses the point of how these experiences are really put together and built, and it creates a false impression that each discipline is somehow working in isolation towards the success of the game – Media Molecule have an interesting approach to podcasts, as do 2k on their Cult of Rapture podcasts, again, around this idea of a group of developers sitting down and you, as a listener, become a fly on the wall of their process and team conversations. This feeds back into the idea of an interdisciplinary studio ‘culture’. I’d personally much rather hear about how a team came up with an idea for a feature together, how the sound influenced the visuals, how the visuals influenced the sound, and how the balancing and weight of the game design elements like spawning enemy amounts and stealth vs gun-play influenced the approach to the environments and the mix and so on, these are the kinds of examples of production I’d much rather see representing game development – I recall a great featurette on one of the many DVD re-issues of Apocalypse Now showing a production meeting between all the principle collaborators on the film, Murch and Coppola et al, and you get such a huge amount of inspiration from the way Walter represents his ideas and how Francis responds to those ideas in a group setting. This was one of the only ‘making of’ segments that completely changed the way I wanted to work in sound.

So, to sort of answer your question, I don’t think this is specifically something audio designers are falling short on; it is just the way that interviews with developers are being framed and set-up. Similarly, why not have a publicity picture of an audio designer, not wearing headphones or slumped provocatively over a hot a mixing desk or other ‘gear’, but in a team or collaborative context, a still-shot of someone working and discussing ideas with others, no more pictures in front of mixing desks! This is also one of my personal goals from now on; we’ve all been guilty of it, but it is a big part of how sound design is perceived – put ‘sound designer’ into google image search, and you’ll see what I mean – I mean, sure, it is what we do, but somehow, it isn’t really what we do at all.

These kinds of pictures are codified with meaning, to other people, I imagine they say things like…

‘I have nice gear’
‘I can work this big mixing desk’
‘I have headphones on, sorry what was that?’
‘I am in a soundproofed room, shhh!’
‘This is awkward’
‘I am relaxed. Honest.’

– They don’t say …

‘Sound is an active collaborative element of design’.

So, sound has an image problem. Once the images begin to change, perhaps production thinking will change quicker. All these things can help or hinder the ability of sound to become an active player in early development.

Thinking again about the future, you’ve spoken a good deal about technologies and processes that you think need to happen for audio to raise itself up. What do you think is the very next thing that we should be doing? What’s the most important and attainable move we can make today to help ourselves in the future?

I think we could all make time to gain some fresh perspectives. It’s an awesome industry we work in, but it is often stressful and high pressure and can all be taken too seriously at which point it can become something dark. I think Conan O’Brien’s Clueless Gamer reviews put our industry into a healthy perspective, at least from the pov of the end-consumer, and if we can laugh at ourselves then this is healthy. Collaborate on great games and great experiences, enjoy the process and enjoy working with teams, solving problems with the people you are with… most importantly, get out of the studio – in terms of valuable experiences, the real world is kind of hard to beat, and anything you do outside of the job, whether it is travelling, duck-hunting, bird-watching, baking, cross country skiing, watch-making or collecting hard-to-find Pez dispensers, having a hobby in which you can immerse yourself, lose yourself, is probably the most valuable thing you can do as an influence on your work. You’ll be fresher and sharper when you are at ‘work’ if you are refreshed, rested and content elsewhere in your life. You’ll also have a healthy perspective on the work you do. I find meeting experts in other fields, completely different fields, like short-wave radio or vintage typewriters, is amazingly inspiring. It’s not what they know that I find useful, but the enthusiasm and passion for their subject that is utterly infectious, and these are usually hobbyists who have the most passion. It also allows you to switch off from ‘development’ thinking and ‘sound’ thinking, and it is almost exactly when this happens that the best ideas will occur to you.

So, what’s your hobby right now?

I’m learning about coffee.

Cool. How has learning about coffee affected your work and perspective?

I’m not sure yet, but I enjoy the coffee.

Right on. Tangent. Lots of stuff going around about Google Glass. Do you think there is or will be an audio equivalent?

Glass is screaming out for audio integration to be at the forefront of its development. The visual clutter that we’ve seen in the demos has already received a fair amount of ridicule from a practicality and safety point of view. Positional audio cues, for example, seem like a plausible avenue for exploration here for waypoint location etc. – as does a dialogue-only navigation system, leaving the visual field uncluttered. Life doesn’t need a HUD. It is funny, because I envisage a whole other class of human being appearing with this technology, and one that will appear ‘insane’ to those that don’t use that technology. There will be people standing in the streets and store aisles looking into space as they attempt to navigate an interface invisible to others. They will be less present as a species in the spaces that their physical bodies are, and probably more vulnerable, it could well be an evolutionary cul-de-sac for wearable technology. There is also a risk of highly advanced schizophonia (separation of visual and aural worlds) in this class of human, and I honestly think that prolonged use of this kind of technology will seriously mess people up, we see it as cool right now, but this might be our generation’s LSD. A technology that separates the body and the mind and that actually puts your physical safety at risk in the real world. Put it this way, you wouldn’t last long in my local pub wearing your google glass. Well, that’s quite dystopian of me to say that, but I see opportunities for audio augmented reality as perhaps a ‘less risky’ avenue of development for wearable technology, and I have plenty of ideas where that could go. I’m not saying there should be no visual development in the field of view at all, but a heavier leaning on audio to provide information to the wearer, and not force them to have to stop and read stuff. A lot of dystopian futures have already been written and imagined in this area. There are good ways this could go, and there are problematic ways. As designers, we’ve got to think about more things than just entertainment, we’ve got to think about the contexts in which these experiences are being experienced.

I’ll probably wait for Google Bowtie. End tangent. Is there any misrepresentation or stereotype about the audio discipline that you’d like to dispel?

I don’t believe sound is 50% of the experience, I think that is a disservice to all the other disciplines that go into making either film or games. I don’t like the idea of separating sound out from all the other elements and contexts within which it is experienced. That phrase has shone a light on sound, but, like the Wilhelm Scream, it feels like it is time to put it back in the box.

Do you think we should nuke that mantra then?

I don’t think we necessarily need to kill it right away, it will go away eventually, if we stop repeating it and refrain from keeping it in the production lexicon. These kinds of phrases are crutches, when we no longer need them, we’ll throw them aside.

Is there a talent that you think audio designers have that is not being exploited to its fullest potential?

Listening. It’s an obvious one to mention, but all collaborative disciplines could benefit massively by going through some positive exercises to listen more, to each other, acknowledge ideas and what has just been said. To not step on people’s words. For those people talking, listening constantly and knowing when to finish, or when to get some input – Listen to the game, and to the world around you. I honestly believe that every problem in the world today, whether it is a small issue, or a massive world problem, has arisen because someone has not listened.

“When people talk, listen completely. Most people never listen.”
— Ernest Hemingway

“Courage is what it takes to stand up and speak; courage is also what it takes to sit down and listen.”
— Winston Churchill

Yeah! I be chewin’ on that wisdom from my main man Winston.

Back to it, another area I think people from an audio discipline excel is in experimentation. I think this comes from the nature of the work, working in the darkness, building things up, every project, team and collaboration being different, with a million possible solutions and avenues. There is also something gratifying about the way we make mistakes; you could argue that the process of sound design, or composition, is just repeatedly making mistakes until something works. This is the way game development should work, and in some cases does, it’s just that the slower iteration time and $ cost associated with getting something to show up on screen and be playable makes mistakes more onerous.

Do you think immersion is the right way to describe how audio contributes to the player’s experience of a game? How can audio make the greatest impact on our players?

Immersion is a word I associate more with virtual reality than in video games, I think the most we can hope for, realistically, is to have an audience be engaged in something compelling. It boils down to communicating the experience to the player, above everything else. As long as all the work of the team is focused on this, then the sound design will provide what it needs to. Every game, and every movie, has a unique requirement from its soundtrack. Oftentimes the sound can go above and beyond, perhaps delivering an experience that was never thought possible of that project, but any seasoned developer will already be expecting that ‘magical’ element from the soundtrack. Sound can be a glue that binds the many moving parts of a project together and fixes problems, so in that sense it needs to be smooth, consistent and polished. It will also deliver the tone of the experience in a direct physical way to the player; it can make something represented on screen ‘feel’ real so it is a physical and intimate experience with the player.

I’d say the biggest thing we can do in terms of being compelling is to design the sound for the person playing the game, not the person watching the game being played. We should focus all our efforts in this area, and the only way to really do this is by playing the game yourself and having reviewers and feedback provided by people who play the game. Someone watching a game being played will ask ‘what’s that bleeping sound?’ or ‘why didn’t that tiny piece of metal-piping have any collision sounds when it hit the ground?’ – Whereas the player is aware of their low-health or ammo warning sounds, is familiar enough to realize that they require immediate attention without it ruining the ‘experience’ in fact, it is the experience, they are not aware of the small piece of metal-piping without its collision effect cluttering the mix. To lean on a real-world example, it is virtually the same difference between someone driving a car and someone in the passenger seat – do they experience the road-trip the same way? With cinema, everyone is a passenger in that experience, looking through the window, perhaps thinking they could drive this car better, whereas with games; you have to do the driving – so this is a fundamental difference in the audience you are designing for, and a difference in approach for sound design that is essential to grasp if the player is going to feel engaged.

Is there anything else that you’d like to offer our dear readers?

I’d definitely like to take the chance to encourage people to give back, to contribute to and be a part of the wider game development community, there are a lot of audio specific organizations like GANG, IESD, GVAC, AES, VGM, IASIG, Designing Sound, Creating Sound (of course), etc, and less formally, but just as importantly through #gameaudio. A lot of people are giving a ton of their time and energy to the wider community through these channels, and if you step back and look at that as a whole, it is an amazing, living resource, especially for those wanting to get into the industry, or learn more about it.

Also, if you find yourself out of work for any period of time in this industry, then these organizations offer a way to stay involved and continue to contribute, even if not through actual game development, so I’d encourage everyone to continue to contribute to those channels and grow them, even start your own.

Finally, if you haven’t already, perhaps offer to do a talk at your local university or school and take any chance you can to contribute to and evangelize for the wider understanding of sound design within game development. I’m all for a total grass-roots political approach to promoting the understanding of how sound, design, technology and visuals work together in media at every level, and the cumulative effect of taking any small opportunity to do this will help, just as much as making an awesome sounding game will.

- Should anyone make it to the end of the interview, I wanted to say a huge thanks to Ariel for asking me to be involved in this thing, it has been nice to dust up a couple of ideas and present them here.

Thank you, Rob! You rule and we all look forward to having you back.

Written by Rob Bridgett.

As we work towards greater control over the mix of a video game, and by mix, I also especially refer to the installation and removal of DSP effects, routing and sends (not just volume changes), and as we gain easier access to high-level sound culling features like HDR mixing (hitting the mainstream soon via Wwise), there starts to emerge a clear split in technology and approach.

HDR, essentially an automatic (passive) mixing system, is built to do the majority of the interpretive work on what sounds should be granted permission to play. The feature itself is an ingenious blending of a sound priority system, a playback culling system, and a mixing engine that simulates a perceptual realism based on dB SPL. Compare this with the somewhat hard-to-manage, but powerful (active) mixing technology of state-based mixer snapshots, which puts all of the control and decision making in the hands of the mix implementer (also now prevalent in Wwise and FMOD Studio) and you start to see two distinct aesthetic methods, in fact two distinct approaches at work. The first is an aesthetic of extreme realism, of vérité, almost to the point of an entirely objective listener depending on the realism of the dB SPL tagging, where sounds are prioritized by loudness rather than emotional or narrative value*. The latter, a system which seems like a method of overriding reality, of controlling sound that is completely implementer-defined, requires manual (or at least programmatic) installation, but requires the designer to make all the decisions themselves of what is or is not important at that particular moment.

An HDR exclusive system is perhaps not a good choice for games that require even the most occasional subjectivity from the point of view of its characters. I would also say, having co-developed and wrestled with exclusively state-based snapshot systems, that these systems can be a nightmare of one-off cases where the rules you’ve created fail at some point down the line, of little use when it comes to complex simulations of ‘realism’, story, or player-choice (however you choose to define that).

Figure 1: The right system for the right game.

What is clear, is that these two systems can absolutely work together, hand in hand, passing over from one system to another, or, to think of it another-way, layering state-based unique-case narrative control over the top of generic HDR-controlled simulation sub-systems. What I think this hybrid approach brings sound designers is the ability to have complete control over a rich and complex interactive sound world – a passive system that allows an active system to grab control at moments that require additional control. It allows windows of extreme realism, and the ability to quickly change perspective to that of an objective point of view, focusing only on very specific sounds, filtering out other sounds, or subtly pushing and pulling reverb saturation to orchestrate a malaise, claustrophobia, or paranoia.

What I think is missing however (and I think we have this problem in all aspects of sound production, not just mixing), is the shorthand for us to communicate and play with these ideas in the context of a collaborative team. These sound techniques are of no use whatsoever unless coupled, orchestrated, and integrated as part of a narrative and visual experience. These are powerful concepts and technologies that we now have access to, so how best to employ them?

I don’t necessarily think we need a lexicon, or a new sound or mix-centric language, because every game developer and team is different and relying on these rules and the definitions tie us to our past. I think what is needed are passionate, enthusiastic, socially mobile people, who are able to communicate with people from top to bottom in a game team.

When watching a film, and thinking about the audio-visual mix of a film, the narrative and point of view of the characters in the film, I tend to think of the broad strokes. These are the marks of the director, and are usually something you can see reflected in the approach of the cinematography, whether that is through use of color palette, lenses, or focal depth. These are often clues to what is rippling underneath in the mix.

The hybrid mix approach allows us to bounce from simulation to point-of-view, to color the scene with effects, remove sounds, or groups of sounds, focus, filter, and mangle the entire sound track, or even just a single part of it. We can begin to think in broad strokes, about different lenses that the player (or main character) sees and hears things through. They may be called ‘modes’ by game design, but I think there is a way to de-clutter that technical thinking and apply it more to narrative.

These modes are opportunities to think about perspectives and treatments of perception. One of the easiest and most common ways of talking about this with producers and art directors, is to talk in terms of ‘sound lenses’. Examples that are often relayed to me by these other disciplines are often of ‘sound lenses’ too. I find the broad strokes of the sound lens are very easy to describe and demonstrate, easy to mock-up off-line in an audio only, low-cost context, and easy and quick to implement in run-time as a state-based layer of DSP effects over the top of an HDR system.

Two Types of Lens

Again, very broadly, just as there are broadly two categories of mixing technology, those of passive systems and active systems, there are two categories of sound lens, though these are aesthetic and not tied to either of the mix-type categories per se.

An ‘objective’ lens, for example of a hand held video camera, with compression, distortion, EQ – could give a cold, remorseless eye/ear on a scene.

Switching to a reverb bathed, low pass filter reduced soundscape applied to all but the musical score, allows on the other hand an entirely ‘subjective’ perspective on the same interactive environment. Apply a snapshot of convolution reverb using not an impulse recording, but a shard of ambient sound, onto all in-game environmentally sound, and you can produce a completely broken, disturbing perspective, perhaps emulating ears that no longer function at all, and then coordinate an instant switch back to ‘reality’.

Arguably, another element, or parameter, of the sound lens is the way you transition from one to another. This can be a straight ‘cut’ or a blend over time, though this language and type of articulation is already thoroughly discussed and written about in the practice of film editing, and of course still relies heavily on co-ordination with the other disciplines to create an effect.

Using all of these perspective changes together in concert with transition times gives the game an underlying structure, or drama. It allows the creative team to think about moments, about contexts, and about the meaning for the player of the other sub-systems and meta-game systems that are at work under-the-hood.

This culmination of technologies, of enriched runtime mixing systems, of state-based DSP rich processing gives such powerful control to the sound designer, that the real challenge remains to fully leverage the technology to the service of narrative and how the orchestration of these lenses can create a powerful effect. This can only be done by seeding it into game design, narrative design, and art direction, and by listening to and coordinating all these elements together into a single experience. All of these elements are at the service of the experience, and the technology underneath, while fascinating and exhilarating to discuss among other technically minded implementers, should always be invisible to the player.

The notion of a mix is starting to expand for me into other areas beyond audio, and when you begin to think about mixing and transitioning lenses and broad emotional states, there seem to be a lot more creative possibilities for collaboration on the actual sound mix.

*This is not to say that you cannot tag dB of each sound in a narrative way, rather than a realistic way.