Prototype: Exploring Reactive Music in Wwise
Circumstances outside of my work have meant that the past few weeks have been fairly choppy for me, health and productivity wise. I’m in a constant flux of being just on top of my work or being slightly behind but I have made much more of a pro-active effort to work on things and progress is being made.
I spent some quality time in Wwise this afternoon developing a quick music prototype. I’ve been wanting to explore game reactive music since the start of the semester, discovering that Wwise affords the opportunity to implement music that reacts dynamically to the game state (such as performing actions or being in certain areas for instance). I took some instrument tracks I rendered out in FL Studio last week and added them to my project. Scouring the documentation proved to be the most challenging aspect of the prototype, if I’ll be honest. It’s incredibly text dense, and often relies on a previous understanding of Wwise. (The chapters on the music system are further on in the documentation.) Nevertheless, I think I managed to create a good demonstration of the concept! There’s a couple issues I want to rectify, but I know myself… I know that I would end up sitting there fixing every minute detail. Thankfully, I managed to restrain myself. It’s functioning, and that’s what matters.
Snapshot: Carys
Captured, 17th October, 2021.
Went for a coffee with Carys at Blend and caught up about things going on in Glasgow. Ended up chatting for a good few hours till the place shut – mostly about the shenanigans of Carys’ flat – although it’s nearly been a couple of weeks since and the finer details slip my mind. With that said, the story about the hamster food is firmly cemented in my head.
Sorry, Chloe.
I brought my camera with me so I thought I’d get a selfie before I went home. Ended up taking a couple, so Carys had a copy too! It’d be interesting to see how hers’ came out. At any rate, I’m looking forward to heading down for Halloween and seeing everyone again!
Finding Ambience: Soundwalk of Dundee City Centre
Nearly a couple weeks ago now, my classmate Zhoro and I went on a soundwalk around Dundee to record ambient audio clips for our weekly university task. After waiting for our Zoom H4Ns to slowly power on – in Zhoro’s case with his 128gb SD card slotted into the H4N’s ten-year old hardware, the wait was painfully five minutes long – and we set off. It took us a minute to find where to go. My fingers were already cold from waiting for the gigabyte-beast to awaken; fumbling around trying to get my phone working with touchscreen gloves proved futile. Luckily Zhoro had the map saved from our university’s learning portal, and we started to follow it. Loosely.
My excuse for failing to follow the preset path was not out of any rebellion or spite, but because it was bloody freezing and my fight-or-flight instincts started to kick in and Zhoro walks faster than me.
The headphones I use to monitor audio are a little finicky, but once I have them untangled and in my ears, they provide a pretty honest representation of what I’m wanting to listen to. The only issue? They are incredibly sensitive. A motorbike revved its engine whilst I listening through my recorder’s audio inputs, and based off the ringing I heard for the subsequent ten or twenty minutes after, I was walking around with my levels all wrong. I think I passed away for a brief moment; that revving noise has perturbed the deepest depths of my dark, dark soul, etching an unceasingly intense reminder for myself to always check audio levels and playback volume prior to and throughout recording. Of course, this seems obvious, but clearly not obvious enough for whatever version of me walked past Nisa and shat a brick large enough to need planning permission. Thankfully, I hadn’t started to record by that point and promptly quashed my whoopsie.
In the end, we managed to pull a few good sound clips that could be used for some ambient Dundonian noise, local walla and all! While some of the clips were duds due to accidental peaking, Dundee’s consistent car droning, and illegible shouts of passers-by, I managed to get a good few sounds. I have since compiled them into a single track.
Snapshot: Stobswell Being Store
Captured, 3rd October, 2021.
Fujifilm Instax 40; 8.6cm X 5.4cm; ISO800
Creating Metropolitan Lair
One Covid scare and a case of tonsillitis later, I’m somewhat back on my feet. Suffice to say, going out to Dundee Dance Event Monday-before-last was not the wisest move. I seem to be about a week behind, give-or-take. Thankfully, I’m managing catching up on my work. However, in the midst of illness and distraction, I found time to do my lab work for my audio class last Thursday.
We were tasked with creating a soundscape using Reaper, composed of sound clips and music tracks from online sound libraries. Over the years, I have grown estranged to Reaper. I made a couple silly tracks in it during high school when I well and truly had no clue what I was doing, but hadn’t touched it until I started working on my soundscape last week. I’m more familiar with FL Studio, but I’ve found that once you know the basics of one DAW, you know the basics for most others. Certainly, working with Reaper didn’t feel alien!
Before you read the process in the sections below, listen to the soundscape (preferably with a pair of headphones) first! I find that telling folk the process of creative work implants ideas into their head about what they’re about to experience, and I’m always fascinated by unswayed personal interpretations! If you thought of something different, feel free to reply to this post with what you thought you were listening to!
Process
I decided to approach the task as a story, guiding the listener through the environment. Harking back to my blog post from a couple weeks ago, I wanted to create an urban environment where the protagonist travels through different scenarios before finally reaching their destination.
Once the core idea was in place, I needed a way to convert the abstractions I had flickering around my head into something more tangible. I began roughly sketching each idea I had down onto different post-it notes, which ended up comprising of compartmentalised settings and actions.
Now that I had a few ideas ham-fistedly etched onto a few post-its, I began storyboarding. I laid out the post-its on a sheet of paper and moved them around until I had a configuration I was happy with, adding or removing them as required. The idea now had structure. (I would go on to iterate upon this while working in Reaper.)
Storyboard Outline
- Street
- Garage (Parking)
- Turn around
- Enter door
- Down stairs
- Walk through corridor
(Pass through open rooms)- Arcade
- Underground club
- Reach end corridor
- Find the obelisk* (in outer space)
Representation
The story starts with the protagonist walking down a street and walking into an indoor car-park. Turning around, they walk through a door and head down a staircase. A long corridor sprawls out ahead. Traversing down it, you pass the beeping sounds of an arcade and the heavy basslines of an illegal rave just before reaching the end and finding an intergalactic obelisk*
*Unfortunately, the intergalactic obelisk never made the cut.
With my story set, it was time to put it into practice! We had the limitation that the soundscape was supposed to be around a minute in length. With this in mind, I found and separated the core sections of the story into three chunks:
- Outside
- Garage
- Underground
Using Soundsnap and the Free Music Archive, I began downloading some sounds, renaming them, and importing them into Reaper.
I absolutely cannot guarantee that I did this very efficiently. Much like a chimpanzee flinging its steaming hot faeces at a passing car, I approached Reaper the only way I know how: by chucking random .wav files at it and praying. I eventually got the hang of it. Taking my chimp brain into account, I discovered that you can create new tracks by placing audio clips or MIDI files onto the timeline which adds a new slider to the mixer along the bottom of the window. I then coloured these tracks by section.
Once I had the core pieces of a section in place, I added effects to help convey the sounds as belonging to the intended space. A clear example of this can be found by listening out for the hip hop track which plays in the garage section of the soundscape. I imagined someone playing music out of their phone speaker. The music needed to sound reverberant and tinny, so I added a high-pass filter and a reverb effect to the track’s effect rack to convey this.
Music track EQ effect. Only allows through high frequencies, like a high-pass filter. 
Music track reverb effect. Reciprocates the ambience of the parking garage setting.
I completed each section individually and combined them in sequence. Some sections featured developing sounds internally, whilst other sections had to transition into other tracks. Using automation, I was able to change volume over time, pan, emulate the sounds of a door opening and affecting the music behind, and probably other things.
Afterthoughts
Once I combined all the sections together, the soundscape was mostly complete, final mixing tweaks aside. However, I still had to figure out how it should end. While I originally placed “Obelisk (from outer space)” on the storyboard as a joke (because I couldn’t think of anything immediately more interesting), there had to be a more cohesive way to round off the story? Then, chimp brain chimed in again with one last thought: hidden underground lairs are cool, right?
If I’m to be one-hundred percent truthful, I didn’t bolt the ending on at the end of the production process at all… I had a rough idea that the protagonist was some kind of illicit or secretive person with images of criminal masterminds and superheroes swirling around my aforementioned chimp brain. The underground lair aspect was merely just a response to this. Besides, it makes far more sense than an obelisk. What’s an obelisk supposed to sound like anyway? I reckon it probably hums.
All in all, I am incredibly satisfied with the soundscape and its ending! I believe it leaves a lot of room for further discussion about its world and its dynamics.
Thinking retrospectively, I don’t believe this is my first soundscape. I edited the teaser trailer for my team’s DARE Academy submission. For the opening thirty seconds of the video, I had to do something very similar to what I did for this task. Its incredible how much it can improve an experience! Perhaps the in-engine footage we captured for the trailer was amateurish, but I feel the audio nicely tied it together. What do you think?
Snapshot: Berry Tree
Captured, 30th September, 2021.
Fujifilm Instax 40; 8.6cm X 5.4cm; ISO800
(An Early Attempt at) Conceptualising an Audio-Led Game
We’re being tasked to help think of an audio-led game for our AUD311 module, and I have an idea. Perhaps this idea could be taken through to my final submission, although I’ll want to prototype it when I next get some free time to see if it is in any way viable.
Digital soundscapes are collections of procedural and pre-recorded voices that evoke a sense of place. (At least, this is my understanding of them so far; it’s still early days.) This week, we were tasked to form an idea for an audio-led game. i.e. a game where audio is the primary design element. I was using the level greybox showcased last week as a basis, as this is the same level we will be using in our final submission. I wanted to question if I could create a soundscape using a single virtual instrument played in various different ways throughout the entirety of the environment instead of utilising a myriad of different recordings and procedural sounds.
Walking footsteps plays alternating bass keys, whilst throwing a harpoon into the air plays keys across a scale depending on the trajectory and duration of flight. Birds tweeting could be represented by high trills, and scattering them could throw the composition into a state of discordance.
The example of John Zorn’s Cobra we were shown earlier on in the week inspired this idea conceptually, although I don’t think it would be unfair of me to suggest Zorn’s works are the musical equivalent of Marmite. I imagine that a soundscape using my concept would lie somewhere in-between Piano Jazz and Free Jazz.
Thinking Interactively
This idea is very open-ended. Although, as it stands, it’s fairly read only in the sense that there is little the player can do with the concept other than explore the environment. There is no game! So, let’s sort that out.
There was a great video by Game Maker’s Toolkit about Jonathan Blow’s puzzle designs, which goes into detail about how Blow creates a simple mechanic and through the design process, uses that one mechanic and pushes it beyond breaking point. Jonathon Blow’s games, such as Braid and The Witness, and works extraneous to his such as Arvi Teikari’s Baba Is You are great examples of games where rules can be manipulated (or just mentally-reframed) throughout gameplay.
The level greybox used for the assignment offers few signifiers to the player on its own. There are different rooms, floor buttons that trigger parts of the environment, and that is where it more-or-less ends.
I think it would be interesting to explore a call-and-response mechanic throughout gameplay. Harking back to my audio-led game idea, every interaction has a distinct response with the piano instrument. Once the player enters a new room, it would be interesting to try and play a sequence of piano notes which correspond to actions in the game. Through iterative play, the player can develop an understanding of which sounds correspond to which actions and decipher the audio clip, mastering an understanding of the novel intraludic (game-centric) piano language. Essentially, Simon Says with a piano instead of with spoken language.
Thinking of Ideas for an Audio Submission
This post is just collection of early stage ponderings for my audio coursework. I want to capture ideas that lend themselves to musicality where, in reality, there would be none. (This might include ideas where musicality is found instead of being created by anyone in everyday reality. ‘Song birds’ are an example that comes to mind.) The clanging music found across the environment in Silent Hill (1999) may be an obtuse example, but it highlights that you can use diegetic sounds to build a musically-informed soundscape.
The soundscape must use the Unity level (pictured above) as a base, although (as far as I am aware) we can reimagine it so long as it fits in with our chosen idea. Below are some starting points for ideas that I could later pursue. There’s no reason I cannot mix and match them if I so wished!
Idea 1. / Dimensional Rooms
This level reminds me a lot of the tutorial for Portal, although I am weary of making a design too similar to it. I do feel going for the sterilised sci-fi route would be the most direct path (which is by no means a bad thing!) but I imagine there’ll also be a lot of submissions that go down this path. For this idea, each room will have a different physical property. I think it would be interesting if it was set in an underground car park / storage facility. Urban, industrial, perhaps even gritty(?) Some inspirations for this could include the games Control or something a bit grungier like Fallout or Half-Life. I watch a lot of television series, and I had the show Misfits in my mind when I thought of the idea.
I could be creative in my use of street sounds and foley, and living in Dundee should mean that capturing these noises shouldn’t be too much of a problem either! Although, this idea doesn’t spark as much inspiration for myself as Idea 2.
Idea 2. / Fragmented Memories
For this idea, each room is a memory or a portal to a different, thematically linked place. I’m thinking of an outdoor or a high-fantasy mystical setting, possibly inspired by folk music and cottagecore. I think this would evoke the greatest sense-of-place for myself, given I spent the majority of my life on Skye. This idea is not as fleshed out and I feel there is a possibility it might be difficult to write about, although I would have a massive emotional investment in the concept. It would be a more challenging concept to tackle, but I could draw from a little bit of research from my second year project on Alzheimer’s to inform how the soundscape could traverse ‘memories’.
Above are some photos some friends and I have taken from my recent trip to Lewis. I feel I could have a unique yet informed perspective for this idea, although I would need to discuss its viability further.
Idea 3. / Victorian Haunted Library
I’m wanting to turn the idea of haunted being scary on its head for this idea.
This idea is a little more fantastical, but flipping the idea of something scary into something friendly and welcoming would be fun to explore! If I fail to dive into this idea exclusively, I still want to think outside the box and flip cultural narratives on their heads, keeping this idea’s spirit alive with whatever I pick. Pun intended.
Listening to Soundscapes
Getting back into university again
There’s much excitement abound, especially with all the music and sound related work in-store this semester!
Things are off to a relatively calm start this year, which I’m definitely thankful for. Having taken part in DARE Academy as a member of Crabertay, this past summer has been more or less non-stop for myself. We (as in, Crabertay) are a small indie outfit of eight, so everyone ends up having to diverge from their specific role from time to time. This meant that despite being the producer, I got to try my hand at making some of the more intense music for the game.
Having a deadline and a goal for making some music helped me learn a lot about music production. Our little custom written music handler had some strict limitations for playing audio clips. We had multiple team meetings discussing the best way to approach it and we came to the conclusion that each song would be comprised of three clips: an intro section, a loopable mid-section (which is played back twice), and an outro section.
I am a firm believer that setting up boundaries, regardless of whether they’re arbitrary or not, can be a good way to compartmentalise learning and creativity. During the morning lecture last Monday, we were told to think of some goals that we want out of the modules. Much in the same way of our little music engine’s technical limitations, setting a clear goal will no doubt help further my learning.
Early targets
In my spare time, I am slowly building up my understanding of music theory. I would like this to supplement into my learning across both audio modules, pushing in towards the territory of becoming a composer. For my final assessment, my primary goal is to build up a musically coherent soundscape although at this stage I haven’t many ideas for what that could look like yet, although I think it would be fun to have various diegetic environment sounds form some kind of procedural song.
While I’m still a long way off from being truly confident in my audio design or music production capabilities, I’m determined to get a stronger grasp of the fields to the point where it could lead to a secondary career path outside of the games industry. At any rate, I am thankful for the learning opportunity this year’s audio module provides and look forward to attempting to make a great soundscape!
Analysing a soundscape
The lecture task for this week was to (as I had taken note of it) “Go and find a game I really like and try to analyse a soundscape.” I have chosen to take a look at a minute of gameplay from The Last of Us (2013), where Joel, Ellie, and Bill break into a high school, escaping a horde of zombies.
In the video below, I’m specifically referencing the time between 3:00:28 and 3:01:30.
Key auditory features
In this section, I’m taking note of the most important sounds I am able to identify from the soundscape in the time-frame previously mentioned.
Diegetic
- Metallic door and furniture clanging noises.
- Groans and moans from zombies attempting to push their way though the doors.
- High treble grating cricket/insect type ambience.
- Aggressive tones in voice tones.
- Footsteps become more present in the quieter walk through the school corridor.
- Room reverb.
Non-Diegetic
- Music from (or something that sounds like) a marimba.
- Constant linear rhythm.
- Droning constant lower note.
- Higher note being played inconsistently, forming a minor chord.
Informal analysis
The soundscape constructs an uneasy tension which helps to mould the upcoming combat with the zombie “runners” (at 3:01:30 in the embedded YouTube play-through above.) The sound design paints a highly uncomfortable and lethal environment. The grating metallic noises of the doors banging and the disturbed anger in the voices of the characters in the scene makes the situation seem highly volatile. This tension is strengthened further with the constant yet near-discordant clanging music quietly mixed into the background sound effects. The heightened presence of the characters’ footsteps in the ‘calm before the storm’ walking down the corridor, emphasised further by the highly reverberant ambience, makes the hall seem foreboding, as if the high school corridors are much bigger than they are.
The scene invokes this sense of angst and anxiety, almost telling players there might not be a zombie just around the corner, but don’t count your luck on it. Scenes such as this present throughout most of the game promote intense, high stakes soundscapes which are, I find, much in tune with the game’s narrative design and story writing. To me, the atmosphere aligns with thriller tropes, verging into horror. Some similar examples that come to mind include Silent Hill (1999) and Outlast (2013).
the seylordev webzone 
You must be logged in to post a comment.