Audio Innovation: FretWorks

This week I would like to discuss a couple of examples of recent games that have shown innovation in the ways in which they make use of audio within the gaming environment. More specifically I would like to look at games that make use of a players physical interactions, as part of the overall audio experience. With the development of consoles such as the Wii and more recently X-Box Kinnect, games designers have relished being able to make use of physical movement within their games. Nintendo have a history of using non-conventional control methods in their music games, demonstrated by titles such as Donkey Konga; however they took this idea further with their Wii Music game - where the movement of the player controls a number of factors, including the tempo, instrumentation and quality of the output music. Other consoles have also made the move to having controllers that resemble real world instruments and playing styles, a couple of which I will mention later.

So it is clear that a number of recent titles have ventured into new and exciting realms of audio and controller technology. One game I would like to look at in particular is FretNice. Developed by Temco and released in 2010 for X-Box 360 and PS3. The game attempts to build upon a traditional platform format, with the addition of player interactivity more akin to the likes of Guitar Hero and Rockband, and giving the user direct input into the creation of the soundtrack.

The game is designed for use with a Guitar Hero controller, and therefore incorporates physical actions of instrumental playing into the game experience. Simple controls include: Green/Yellow buttons - Left/Right movement; swing the controller up (or Red button) - Jump; all colour buttons (whilst jumped) - Attack. In addition to this the player is also required to strum in order to carry out the button’s functions.

As the player progresses through the level, enemies are destroyed by firing chords at them, and which chord is needed is related to the appearance of the creature, in terms of their facial attributes (eyes, mouths, noses, etc). So for example: a creature with two eyes needs a two note chord, one with three eyes requires a three note chord,  and so on. Therefore, as is mentioned in the tutorial level of the game, this involves players attacking enemies by how they think they would sound.

The audio environment is made up of two main layers: the background featuring simple looping pop music backing tracks, and the foreground that features all the sounds that are triggered as a result of player input and on-screen action. The music system is designed so that as you walk around and attack enemies, the resulting pick-up sounds and different attack sounds are all in key with the musical backing. The pick up sounds work in a similar way to a feature seen in ‘Super Mario Galaxy’. where the musical pattern is played through one note at a time, as a new pick-up is touched.  The footstep system works similarly, with a percussive loop being activated whilst the player is moving along; the bongo notes being roughly in time with the footsteps. On the other hand the attack system works in a different way, which creates a possibly unwanted side-effect: each of the five ‘attack’ buttons is assigned to play a note, from the key that the backing track is in; but although these sounds are individually consonant with the music, together and in different combinations they can become dissonant.

However the games creators may have been well aware of this and included it intentionally for a couple of reasons. Firstly it is loosely addressed in the ‘Story’ section of the website, which outlines the intentions of the game and includes phrases such as “unique style of music” and “new type of music”, which suggest that that the in game music is supposed to be different to traditional accepted consonant rock. [FretNice 2010] Secondly, there is the knock on effect that as gameplay becomes more intense and more ‘attack’ is needed, the music becomes more dissonant; so perhaps this was also designed as a way to provide audible feedback on how intense the players situation is.

The points raised above about dissonance, open up an interesting discussion about music perception: do these note combinations only sound harsh to Western ears, used to the harmony of Western pop music? From a scientific stand point the notes in question are two far apart to cause any ‘clash’ in physiological terms, so it is possible that ears from another culture with different tonal systems, may not describe the relationships as dissonant at all. However taken from a Western viewpoint and combined with the use of phrases such as “new…music”, the use of dissonance can be seen as an attempt to break from the norm and explore new devices of game audio.

Because of the nature of how the output audio track is compiled, it means that each time the game is played the resulting sound will be different: pick-up will be collected at different times and different combinations of 'attack' notes will be triggered.






REFERENCES

FretNice. 2010. FretNice: Story. Temco LTD. <http://www.fretnicegame.com/fretnicegame/index.html>[Accessed 15/11/2011].

Tom McShea. 2010. Fret Nice Review.Gamespot UK. <http://uk.gamespot.com/fretnice/reviews/6250267/fret-nice-review/platform/ps3>
[Accessed 15/11/2011].

Games-
Donkey Konga. 2004. Nintendo.
FretNice. 2010. Temco
Guitar Hero. 2005. Activision.
Rockband. 2007. Harmonix.
Wii Music. 2008. Nintendo.

Videos-

Fret Nice Gameplay. 2010. <http://www.youtube.com/watch?v=lYfIdaTZuHk> [Accessed 15/11/2011].

Wii / Music Orchestra. 2008. <http://www.youtube.com/watch?v=De0K8ufUxtg> [Accessed 15/11/2011].
Gameplay: Donkey Konga. 2010. <http://www.youtube.com/watch?v=nrFt6guz7A8&feature=fvst>  [Accessed 15/11/2011].http://www.fretnicegame.com/fretnicegame/index.html

Interactive Music

As the games industry develops and expands it’s horizons, there is a growing call for music that goes beyond being just “a detached backdrop to the action on screen”, and begins “adding emotional depth and soul to scenarios and [to] help maintain the suspension of disbelief that is so crucial for players.”1 In other words music that reacts directly to the player and therefore gives each player an individual experience.

There is however a slight contradiction in the two terms most commonly used to describe this style of music: ‘Adaptive’ and ‘Interactive’. By definition the two words do mean different things, the former suggesting music that changes without direct player command and the later suggesting music that directly, and with a players knowledge, responds to game input. So perhaps the term ‘interactive’ should be reserved for games where the user is knowingly engaging with the changes to the music. Guy Whitmore has noted that “‘Adaptive audio’ is a term used to describe audio and music that reacts appropriately to - and even anticipates - gameplay. The term ‘interactive audio’ is also used in this way, but it is used more broadly, and it has been misused and overused.”1 This perhaps suggests that the word ‘interactive’ should be used to refer to games like Guitar Hero, Singstar and Legend of Zelda: Ocarina of time , where the player controls the music with a direct and intended action. Hence for this discussion of game music, I will be using the term ‘adaptive’ and mean it to refer to music that is controlled by player and game actions, but not necessarily with the players conscious knowledge.

As outlined above, adaptive music involves a soundtrack that is able to be varied in time with how action unfolds on screen. However this also means having a large amount of available audio, that can be called up at the appropriate time; and with modern gamers expectations of music being more in line the latest blockbuster than the blip-blop of the traditional arcade, this can be very memory intensive. This of course then brings up the argument of quality vs. quantity, which in this case comes down to pre-mixed samples vs. in game rendering.

Of all the various techniques used to implement adaptive music, the most common method is by having music that is split into individual segments and triggered in ‘states’ or ‘modules’. This can work both horizontally and vertically. Horizontally being changes in structure: Intro, looping section, possible second looping section, outro. Vertically being changes to texture: different layers that make up the music as a whole, are combined in different ways to vary timbre. Audio software such as Fmod and Wwise make it easy to set up music to operate in this way, as can be seen in this example, and it is also just as simple to set up basic systems for doing this in game editors such as UDK “which can be effective but are also time consuming to implement one by one.”2

Due to the fact that it is time consuming to set up the trigger systems in game editors, by which adaptive music can be implemented, it is becoming more common for the change in music to respond to game stats. The added advantage of this is that it uses a system that is more global and able to react to a player anywhere in the game level, whereas a trigger method would be more location specific. A good example of where a system like this is used is Hellgate: London.3 The music system within that game reacts based on variables such as player health, number of nearby enemies, total remaining enemies and the distance the player has travelled within a set time. This results in music variance that reflects individual playing styles of the end users.

This brings me on to the example that I wish to discuss in this post: No One Lives Forever 2. For which the score was composed by Guy Whitmore, whom I mentioned earlier.

The game was released in 2002 by Monolith Productions and Sierra Entertainment, and highlights approaches and techniques that are still relevant and used today.

As the cinematic ends, the music continues to seamlessly move the user into live gameplay. You can hear that different layers from the musical cue of the cinematic are looped, fading in and out whilst the player explores the items in their inventory. Once the player moves under the wooden structure, the ‘end’ sequence of this section begins: an orchestral swell that moves into the next musical looping section when the player finds the first letter. The tension then increases with the addition of drums to the texture; over the top of which musical snippets that were present in the cinematic, continue to decorate. As the player continues up the path, finding further letters, you can hear that each time one is found another layer is added or removed.

As the player nears the village and the robotic bird says “Be Careful” and starts talking about stealth, the music again changes. The drums leave the texture, along with other layers, and the ones which remain melodically switch to softer staccato and pizzicato notes. There is again an orchestral swell as the player reaches the transition into another area of the level, and another musical section.

An electric bass is added to the texture once the player is inside the town, which will later to be seen as a characteristic of this location. However as the player is still ‘undetected’, the main theme of the music remains in the ‘stealth’ cue. As the player progresses through the village the musical layers continue to fade in and out, and we see the use of several micky mouseing swells that coincide with the moving of a body and when the player is approaching corners. Then an onscreen message displays “An Alarm has been sounded” and as the player emerges out into the open, the music changes. The bassline becomes more prominent, the tempo increases and the music overall becomes more funky; after a few bars we also hear the addition of a brass section. This can be said to be either a “Fight” or “Discovered” cue, but overall signifies a heightened danger. The funky music continues until the last visible NPC character is killed; which is also marked with a brass stab. We then return to the ‘stealth’ cue.

Conclusions -



References 
1 - Guy Whitmore, Gamasutra, <http://www.gamasutra.com/resource_guide/20030528/whitmore_01.shtml> 
2 - Alexander Brandon, IASIG Publications <http://www.iasig.org/pubs/interviews/guy_whitmore.shtml>
3 - David Steinwedel, Sound Design for Games, Films, TV & Media, The Music System for Hellgate:London <http://www.dsteinwedel.com/Articles/Articles.html> 

Non-Repetitive Design

Halo: Combat Evolved, the first release of the now renowned Halo series, came out in November 2001 as a launch title to accompany the release of the Xbox console and as a showcase of the platforms capabilities. This new console saw major developments in sound capability and was the first to allow for Dolby Surround during game play, not just for pre-prepared cinematic sections. To assist in my analysis of this title, I will at times make reference to the insights provided by the game’s composer Martin O’Donnell in his Gamasutra entry on the project.

It becomes clear, from reading O’Donnell’s article, that the sound team were well aware of the direct completion between video games and other visual arts in the media climate of the time. It is also evident that they knew the surround sound capabilities of the Xbox would allow them to engage in this struggle head-on, in terms of the audio. Overall their aim was to create a soundtrack that could rival  a Hollywood blockbuster, but that was both realistic and immersive for the end user.

O‘Donnell mentions how he went about implementing a sense of continuity in the menu and opening cinematic sequences of the game and says he prefers “a consistent audio experience where sound or music isn’t abruptly cut off when the user makes a menu choice, where there can be a sense of elegance and continuity between the opening movie, interface and loading screen.” - But I would say that this game goes beyond that and continues this cinematic and immersive experience into the game itself, providing a fully encapsulating game play environment.

The implementation of this non-repetitive soundscape is taken care of by the game’s audio engine. The game’s sound library is made up of a range of different sound files for each audio event, that are relevant to certain environmental and physical conditions. The playing of this sound files and how they are played, is then organised by a system of sound-tags. The method by which these are recalled can be summarised thus:

• An in game action calls for a sound event to happen, in the form of a sound tag.
• The audio engine, upon receiving the sound-tag request, then takes into account player state, environmental factors, time since last sound occurrence, etc, to then call up an appropriate sound file.
• All of which takes place within a fraction of a second.

The sound-tag may also result in multiple sound files being called up and played back; for example an environmental soundscape may be made up of several layered sounds.

One of the most detectable ways in which this system is used is though the dialogue of the NPCs within the game. As the in-game dialogue is made up of many short fragments of speech, these can be combined in almost infinite combinations. O’Donnell notes “there are more than six thousand lines of dialogue in the game and one of the elements I still find enjoyable is listening to new and unique combinations of real-time interaction between the marine and alien AI’s. I literally never know what they’re going to say next.” However, although the bank of available dialogue is vast, the recall is very controlled and systematically organised. This is done with a complex web of situations where dialogue is needed: 12 main categories than split down into multiple smaller categories, each of which has several permutations of speech that would be relevant. “For example, in the category ‘hurting people’ one of 4 main sound-tags, ‘damaged enemy’ contains between 15 to 20 permutations depending on the character.” This means that if the end user happens to repeat a situation or scenario within the game, the dialogue has the scope to vary every time.

Similarly the sound effects are designed to vary. In creating overall ambience for the different settings within the game, as well as the continuous sounds (such as wind) one-shot sounds are used, that can be randomised in terms of surround sound position, repeating interval, volume and pitch, all with specified parameters. Other sounds, whose playback is directly triggered by the player, were designed like the dialogue to call up and play from a range of permutations. However, it is more evident that these sounds are aware of player state: “the automatic rifle sound-tag is a collection of gunfire sound file permutations that are called in sync with the weapons rate of fire. Each call is interrupted by the subsequent call until the trigger is released and then the permutation is allowed to finish playing.” This also helps to stop the audio environment becoming cluttered with sounds. Other examples of how the audio adapts include the pitch and volume variation of engine sounds in relation to factors relating to the current state of the vehicle.

The music playback is also linked to game play, albeit in a somewhat more basic way. The premise behind this system is that the playback of each musical cue consists of a ‘start’, a ‘loop’ and an ‘end’ section, the playback of which and transition between is controlled by what is happening on screen and a what stage a play is at in terms of mission progress and level position. A more detailed explanation of this can be found in the Producing Content section of O’Donnell’s article.

VIDEO EXAMPLE (at top)

The video example I have chosen to include, highlights a number of the issues addressed within the main body of analysis above. The example comes from the very beginning of the games narrative and thus is the first experience a player would have.

As the cinematic of the space ship crash ends, we smoothly transition into game play without any noticeable change to the audio track and are introduced to the ‘voice in your head’ that accompanies you throughout the game and guides you with objectives. As the player explores the area outside the ship this voice reacts to what is seen and incoming NPCs. In this scene, the waterfall provides an example of a sound that varies depending on character position. There are also a number of bird sounds that occur at random as the player walks around the environment; this adds to the realism by using a sound that the player would recognise and associate with the outdoors.

As well as the voice that is heard, the footsteps are another good example of where you can hear that a range of samples have been used and that they are called back at random. Contrastingly however, despite all the attention to detail and the clever implementation behind it, the machine gun sound does at times when the player is holding down the trigger, sound a little bit like a looped sound; although it is still clear that the audio engine is still working to cut all but the last sound short, until the sound would naturally die away.

In terms of music, there appears to be two main cues in this clip. The first being the tension music used as the ‘drop ships’ are seen and the second being the ominous drone that fades in after the last enemy is killed at the base over the hill.

As a means of contrast, this video shows a different players experience of the same level.

Bad Game Audio

Resident Evil is a 1996 PlayStation title from the games company Capcom. It was originally developed in Japan under the name Biohazard. The game featured various character modes, in which the storyline differed according to who you played as; I have chosen to look at the timeline of the character Jill.

A video of the section being discussed can be found here: Jill's Story.

Based on the users very first encounter with this game, the standard of the audio looks promising, as the arcade style title menu and menu transition sounds are pleasing. However from the outset of the gameplay, it becomes clear that it may not be able to live up to the expectations it has created. 

A number of issues become clear as soon as we first see the characters in the entrance hall. There are no room tones, or any atmospheric sounds applied, which makes the setting seem very unnatural; and combined with the audible reverb used on sounds made by the characters, create odd pauses in the on screen action. It also quickly becomes clear that the voice acting is of a poor standard, to an almost laughable extent. The way things are phrased and the inflections used on certain words just seem unnatural to the context and, in places, to the way anyone would normally speak. Also, in places where the speech is almost acceptable, sounds are triggered late which in turn degrade the character of the vocal:

“But Chris is…” | PAUSE | Gunshot that should have interrupted the speech

It is evident that the footstep sounds are made up of single samples for each flooring variant, which are repeated as the characters move around. In places these are also out of time with the characters viable footsteps.

The first new sound we hear in the dinning room, a ticking clock, also presents areas for criticism. Firstly that the sound is at an equal volume from all of the camera angles, despite the clock having a fixed position (and therefore sound origin point) in the room. Secondly, the pitch of the sound seems a bit high considering the size of the clock that is seen on screen. The standard of dialogue and footsteps continue as the characters explore the room. In this sequence the characters are also attacked for the first time and we hear the first outing of the music that accompanies these ‘danger’ moments. The music itself isn’t bad, if but a little corny, but it’s application is questionable. The music begins suddenly, even before the zombie is seen on screen, and ends even more abruptly mid-phrase. During this attack it is also clear that the same gunshot sample is repeated without being altered, however this is covered slightly by the music. Once the attack is over, the footstep problem then again comes into play, as the two characters sounds overlap causing odd effects.

When the action returns to the entrance hall there is again an obvious absence of any room tones or ambience, but we hear a musical underscore that wasn’t there before. Although successful in creating some form of tension, it becomes clear after a while that it is only a short section of music and is looped, having an obvious repeat point.

As Jill goes off on her own, through a series of different rooms, the pattern of questionable sounds continues. Starting with a distinct cut of the music from the entrance hall and the beginning of a different musical figure, as Jill goes through the first set of doors. At one point two zombie dogs jump through a window and we hear a return of the rather cheesy ‘danger’ music, however this time the music stops when Jill has left the room, even though she has not killed her attackers. Also, these dogs make what must assumed be a growling sound in time with the music, but this sounds more like a Millstone grating than a canine noise. As she continues through the halls there are points where she clearly passes undead characters, but in these instances they make no sound and the ‘danger’ music does not return. Another instance of note is when Jill goes upstairs: the sound and action of going up the stairs is out of sync for the first half of the cinematic, and when we return to a wide camera angle the nature of the footstep seems not to fit with the manner in which Jill is moving. Continually throughout this section of the level we see the use of a repeated single footstep sample for each flooring type within the game. The absence of room tones is also further evident and there seems to be an inconsistency in the use of musical underscore, with some rooms having it and some not. As we emerge onto the upper landing of the entrance hall, we return to it’s original state of having no room tones or music. The characters then converse in the same laughable way.

As Jill goes off again on her own, we begin to hear a more atmospheric soundtrack as a low drone begins and remains until she returns to the entrance hall setting. Although the tension created by this unnerving drone is quickly dissipated when Jill doesn’t come across any danger during this sequence. It is also noticeable that whilst on the balcony, although only for a short time, the atmospheric sounds loop: 

Wind sound| Crow sound || Wind sound | Crow sound

However it is definitely the poor vocal and footstep audio that are most obvious in this game, as by nature they run throughout the storyline and are integral to the soundscape the user encounters.

SUGGESTED IMPROVEMENTS

Voice acting
Record more natural sounding acting. Use techniques for radio play and then chop up if necessary.

Footstep randomisation
A number of different left and right foot samples for each flooring. Played back randomly for variance.

Room tones
Add room tones to all areas.

Random atmospheric sounds
Random ambient sounds for each room. Eg. Creaking floors/walls, water dripping.

Give static zombies groans
Even if they are not ‘active’ in terms of user interaction yet, they can still make a sound.

Vary gunshot sample
Have 3 or 4 different gun samples that playback randomly for variance.

Consistent use of underscore and transition through rooms
Either use underscore all the time or at places where zombies are. If two adjacent rooms have different underscore, smother transition.

Make musical cues longer, or more loop-able
For areas with underscore, either make the cue long enough to stretch over the whole action or less noticeable when it loops.

Give the ‘danger music an ending figure to wind down tension
Rather than a sudden stop, the end of the danger music loop transitions into an ending figure to end the cue.

More appropriate use of drones to create tension
Use drones in areas where zombies are, or where danger is approaching. That way as the story progresses, the user will associate the sound with danger and tension can be created more readily.