AES Show Spring 2021, Part Four: Gaming Sound, Rupert Neve

As a result of COVID-19, the Audio Engineering Society’s AES Show Spring 2021, appropriately named “Global Resonance,” was conducted online from Europe. This afforded me the rare opportunity to view a number of the presentations, which would have been otherwise impossible.

This show focused more on the academic side of audio technology than the previous New York-based AES Show Fall 2020. In Part One of Copper’s AES Show Spring 2021 coverage (Issue 139), I looked at presentations on binaural audio, audio mixing for residential television viewing environments, and an analysis of differences between Western and Chinese hip-hop music. Part Two (Issue 140) focused on psychoacoustics and studies on emotional responses to sounds. Part Three (Issue 141) covered the technology side of audio/video transmission, and the efforts to achieve greater realism in sampled orchestras. This final installment will focus on sound for interactive video games, and a tribute to the late Rupert Neve.

Audio in Games and Interactive Media

As mentioned in Part One, audio for games has reached a stage where it demands as much attention and expertise as is needed for feature films. This especially holds true for virtual reality (VR) games, where the demand for binaural and 3D sonic realism has even prompted a resurgence in the use of dummy head recording platforms.

Audio in Games and Interactive Media was collaboratively presented by gaming and VR technology research scholar Katja Rogers (host), game music composer Winifred Phillips (Assassin’s Creed, God of War, The Sims), journalist Sarah Fartuun Heinze (author of several pieces on the intersection of video, art, music, and interactivity), and sound designer and composer Mathilde Hoffmann (the Unreal series). They discussed both technical and aesthetic challenges in music and sound design that are unique to video games, VR, and interactive media.

Winifred Phillips explained that composing music for interactive vs. “linear” media has significant differences. Having a beginning, middle and end, with defined moments and timing as in linear media doesn’t exist the same way in interactive media, which constantly has to change. Games give the players control over the sonic (and visual) environment, so the music has to respond to the players’ choices. Music thus has to be composed in a modular way so that it can be disassembled and reassembled dynamically depending on the players’ reactions.

She also explained about the selection of musical genres for a game project as being a key consideration for a composer. The characters and their actions in the game often will dictate the themes in the music score. However, an awareness of the psychology of the gamers also needs to be factored in, so as to successfully meet the players’ expectations while simultaneously avoiding predictability. For example, action-filled games will tend to have more aggressive music, often with a heavy metal-style score to reflect the high-energy visual content.

Mathilde Hoffmann spoke enthusiastically about the creative process in sound design and for fantasy-based games in particular. In order to create a brand new sonic world, she enjoys finding sound sources, manipulating them, and making choices between acoustic and synthesized sounds. Due to the fast pace of games, signature sounds that have maximum impact have to be found quickly, as budgets often do not allow for a slow compilation of customized, nuanced sounds. Using the creation of sounds for monsters, for example, there have to be variations in movement, footsteps and vocal sounds, depending on the mood of the game and what the monster and the players are reacting to.

Sarah Fartuun Heinze compared the different types of interactivity between attending the theater vs. playing games, with musical theater coming closer to the same functions and emotional arcs as experienced in games. She noted the genre of game theater, a style of gaming that uses elements of theater, game aesthetics, live-action role players (“larp”) and offers a mashup of all of these elements.

Hoffmann felt it was important not to let music and sound fall into predictable clichés, where the sounds almost dictate to the players what emotions they are supposed to experience, like a sitcom laugh track. Hyper-realism and subtlety work better, in her estimation, so that the sound design becomes almost transparent, and the player can fully immerse him or herself into the game world without feeling manipulated. In some cases, hearing the music and sound design of a game prior to seeing the game’s visuals can create an entirely different image in one’s mind.

She also explained how game designers use the Pavlovian familiarity response in their work. For example, high-pitched, repetitive sounds are recognized in many cultures as a warning signal, or at the very least, something that puts the listener on mental alert.

Phillips reminded the participants that communication within a game development team is crucial, because individual frames of reference can be so subjective. She insists that actual music references, rather than just verbal descriptions of music styles, be utilized  whenever possible so that there is clarity in understanding, in order for all of the members of the development team to be on the same page. Case in point: Megadeth and Black Sabbath are both categorized as metal bands, but sound very different.

Getting consensus on acoustics and sounds within the development team is a must, and whatever helps the process, whether reference tracks, demo sounds and so on are all useful tools for game audio design engineers.

Phillips also pointed out that some players might spend hundreds or even thousands of hours with a game, at which point they will experience a great deal of exposure to the music and sounds. (That said, there is a trend among players to cut out the volume of the music during some games, especially shooter-based games, in order to more closely focus their concentration.) Therefore, one of the composer’s jobs is to build in enough variation so that each playing experience is both unique, yet familiar.

Along that same notion was a concern about how music can potentially reduce the immersion aspect of game playing, especially if the player dislikes the music genre used, or if it seems too divorced from reality, as in VR games. (Note: “immersion” in the gaming context refers to the player’s engagement in the virtual game world and not to immersive 3D sound per se.) However, since games are entertainment and a controlled, guided journey hyper-reality experience, the music is a necessary identity-enhancing aspect.

The discussion concluded with a thank you to VR games, which serve as a driver for cutting-edge audio research and development. Gaming audio has become a much bigger challenge, and immersive-sound technologies like spatialized audio and binaural audio have made resurgences.

Harrison used Batman: Arkham Asylum as an example of an “open world” interactive game that suffers from narrative fragmentation. It has many plot layers and characters and is difficult for players who are unversed in all of the character and scenario details from the Batman canon to navigate to the different layers within the game without third-party guidance. The same problem occurs in immersive theater when there are too many narrative roads that can easily mislead audiences into different tangents, and thus miss key points and messages within the theater piece’s main theme.

At the same time, the sense of freedom and individual choice is a big appeal of  interactive media. It is also crucial not to deluge gamers or audiences with voluminous instructions or heavy hand-holding, which ruin the whole exploratory experience of interactive media.

Sound design is a key tool for bridging these obstacles, as it can subliminally serve as an instructional guide. Designing sounds to trigger different listening modes, for example, will alter how gamers assimilate the content and their ability to make individual choices in the game’s interactive engagement.

Given the importance of sound and music design in the interactive experience, nothing can be accidental, and everything that can be heard should be deliberately chosen for its function in communicating information to gamers and audiences. Retaining sounds that do not contribute in this way can lead to erroneous conclusions by the gamer and audience, or worse, destroy the verisimilitude of the game or other medium in question.

Audio can also be used as an interface by which, in the case of video games, a user can be informed of status (for example, in a game where one’s avatar is injured or dying, the quality of the sound can be altered, to simulate diminished capacity). It is also important to match the sounds of the real worlds and the virtual world of the game, in order to maintain the game’s sonic “reality.” As examples, these would include simulating the sound of open versus enclosed spaces (using delay and reverb as appropriate), footsteps on pavement as opposed to in a forest, and a host of other audio elements.

Maintaining a consistency of sound helps to serve as intuitive instruction, such as the “boing” sound for jumping used in Super Mario Bros. Warning sounds like sirens or beeps can signal to gamers that they are attempting actions that will not be usable within a game.

Semiotics within interactive games are sounds that take common sounds that are imbued with established meanings within Western culture, and are applied to the game’s virtual reality. For example, sounds like a “ding” are associated with something positive (this originated with pinball machines and later became used for text messages on phones). A musical interval of a rising fifth is interpreted as being “heroic”; for example, in John Williams’ Superman theme, while the more dissonant tritones, aka, “The Devil’s Interval,” are associated with evil, as used by Danny Elfman in the Batman theme and Hela’s theme by Mark Mothersbaugh in Thor: Ragnarok). The sliding down of a note by five half-steps or more connotes “failure,” often heard in cartoons.

Bass sounds are used to immerse the listener into the space and atmosphere of the created reality, while higher-frequency sounds give clues about the direction of movement, location, or proximity of other characters or objects.