Presentation: Serialism & Sonification in Mini Metro

I gave a talk at GDC 2018 as part of a session at the Artificial Intelligence Summit called 'Beyond Procedural Horizons'. I talk about how we combined data sonification and concepts taken from the musical approach known as Serialism to build a soundscape for Mini Metro.

Transcript:

Serialism & Sonification in Mini Metro

or, how we avoided using looping music tracks completely,by using sequential sets of data to generate music and sound.

Serialism

In music, there is a technique called Serialism that usessequential sets of data (known as series), set about ondifferent axes of sound (pitch, volume, rhythm, note duration, etc),working together to create music.

In Mini Metro, we apply this concept by using internal data from the game and externally authored data in tandem to generate the music.

You might have noticed that the game has a clock - the game is broken up into time increments that are represented as hours, days and weeks. (though of course faster) And before diving in, it’s important to know that we derive our music tempo from the duration of...

1 In-Game Hour = 0.8 secs = 1 beat @ 72 bpm = our master pulse, one in-game hour.

We use this as our standard unit of measurement for when to trigger sounds.In other words, most of the sounds in the game are triggered periodically, using fractional durations of 0.8 seconds.

In Mini Metro, the primary mode of authorship lies in drawing and modifying metro lines. They’re also the means in which everything is connected.they serve as the foundation upon which the soundscape of pitches and rhythms is designed. The lines are represented by a unique stream of music generated using data from different sources.

The simplest way to describe this system is that each metro line is represented by a musical sequence of pulses, triggered at a constant rate with a constant pitch. This rate and pitch is constant until they are shifted by a change in gameplay. Each metro station represents one pulse in that sequence. Each pulse has some unique properties, such as volume, timbre, and panning, and these are calculated using game data. Other properties still are inherited from lower-levels of abstraction, namely unique loadouts for each game level. Some levels tell the pulses to fade in gradually, other levels might tell the pulses to trigger using a swing groove instead of a constant rate, and the levels are differentiated in other ways as well.

[1, 2, 3, 4, 6]

All of this musical generation is done using sets of data. And referring back to Serialism, the data is quite often sequential, in series. These numbers actually represent multiples of time fragments. Or, to put it more simply, rhythms. In the case of rhythms and pitches, the data is authored, so we have more control over what kind of mood we’d like to evoke. This data is cycled through in different ways during gameplay to generate music.

So we’ve got some authored data generating musical sequences, but what about using game data? Ideally we could sonify it to give the player some useful feedback about what is happening.

Combinations of game and authored data are found throughout Mini Metro’s audio system. There’s lots of game data and authored data being fed into the system, working in tandem. Authored data is often used to steer things in a musical direction, while game data is used to more closely marry things to gameplay. In some cases, authored data even made its way into other areas of the game. Certain game behaviors like the passenger spawning were retrofitted to fire using rhythm assignments specified by the sound system.

You might ask why go through the trouble to do things this way? Well, it is really fun. But beyond that there are a variety of answers and I could go into a lot of depth about it, but I think the most important reasons are:

Immediacy & Embodiment

Immediate feedback is often reserved for sound effects and not music, and immediate feedback in music can feel forced if not handled correctly. This type of system allows us to bring this idea of immediacy into the music in a way that feels natural.

The granularity of the system allows the soundscape to respond to the game state instantaneously and evolve with the metro system as it grows, and a holistic system handles all of the gradation for you. When your metro is smaller and less busy, the sound is smaller and less busy. As your metro grows and gets more complex, so does the music and sound to reflect that. When you accelerate the simulation, the music and sound of your metro accelerates. When something new is introduced into your system, you’re not only notified up front, but also regularly over time as its sonic representation becomes a part of the ambient tapestry.

Embodiment

And this all (hopefully) ties into a sense of embodiment. Because all of these game objects have sounds that trigger in a musical way, and all use a shared rhythmic language that is cognizant of the game clock, and use game data to further tie them to what is actually happening in the game, things start to feel communal and unified.

It’s an ideal more than a guarantee, but if executed well, I think you can start to approach something akin to a holistic experience for the player.

Thanks!

Link: GDC Vault: 'Beyond Procedural Horizons'

Round-Up: Articles

2018

Native Instruments. We talk about creative process, tools, and what it was like to write for orchestra on Under the Silver Lake.

2017

Gameumentary. We talk about prog rock, composing for orchestra, personal philosophy, early musical memories, workflow, and social media.

Score It. We talk about silence, influence of nature, crossing mediums, temp score, motifs, and mixing genre.

2015

Vehlinggo. Aaron Vehling and I had a very thorough conversation in New York over dinner. We talked a lot about creativity and lifestyle.

Round-Up: Podcasts

From a Piece by Kilian Eng.

2017

Lost Beat 6. We talk about creative process, philosophy, career experiences, and peership.

Game Dev Loadout. We talk about authenticity, building meaningful relationships, maintaining consistency over long periods and not focusing on too many different projects.

Checkpoints. We talk about creative processes, procedural audio systems, chiptunes, my evergreen love of Ice Hockey games, being exposed to Zelda in utero, Cave Story and my early musical steps creating entrance music for electronic wrestlers.

Indie Insider. We speak about my background, FEZ, Hyper Light Drifter, the art of seizing an opportunity, maintaining motivation and inspiration in your work over time, and negotiating pay as a contractor/freelancer.

Super Marcato Bros Podcast. I spoke with the Marcato Bros. about my career, and more specifically about my sound, and the specifics of projects like FEZ, Hyper Light Drifter, It Follows, Monsters Ate My Birthday Cake, Gunhouse, and Mini Metro.

2016

Adventures in Success. I recall the untold story of 'Jack Brooze', my eWrestling alter ego as a teenager.

Biya Byte. We talk about Hyper Light Drifter and It Follows, and work process. We also talk about VR.

Butterfly Effect. I spoke with Tristan Ettleman about the draw of chiptunes, the doors Fez opened, and writing the score to It Follows in three weeks.

The Collective Podcast. Ash and I talk about breaking into the music industry, the business side of making music, the importance of inspiration and collaboration, and where things are headed in the future.

Dear Air. I speak with Gamespot's horror podcast about my career, ewrestling, Rescue the Beagles and horror, naturally.

How Was Work? I talk with my friend Randy about moving to Los Angeles, workspace tinkering, and Beasts of Balance.

2015

People I Think Are Cool. I talk about It Follows, FEZ, creative freedom and expectations, live performances, switching format/medium, Wikipedia, musical cross-pollination, horror, Fantastic Fest, interviews, blogging, social media, patronage, streaming, income, work history, reading, Jodorowsky, Adventure Time, Mini Metro, GTA, Katamari, Keita Takahashi and my minimalistic philosophy and how it seeps into everything I do.

The Damn Fine Podcast. We talk a lot about vinyl and soundtracks.

Train Station at 8. We talk about video game music, live performance, listening habits, Doom, LA Noire, Tom Francis, Zan-zan-zawa-veia, Sonic the Hedgehog, The Floor is Jelly, domain names, Hearthstone, Peter McConnell, tone, soundtrack lengths, retro music facsimiles, The Hero’s Journey, Hideki Naganuma, Jake Kaufman (virt), Kirby, April Fools, Shnabubula, brain training, Jeff Bridges Sleeping Tapes, music in dreams, Mini Metro, etc…

Waveform City. Dropped by the local synthesizer museum to talk synths, naturally!

2014

Composer Quest. I talk about January, generative music, music theory, programming, The Floor is Jelly, postmortem, Monkey Island, iMUSE, time management, FEZ, Chopin, FZ, remix albums, hair, Rise of the Obsidian Interstellar, my solo piano project, songwriting, lyrics, advice for composers, etc…

Inside Video Game Music. I talk about my musical background, music rights, contract terms, NEUTRALITE, collaborative process, time management, FEZ, synthesis, “Adventure”, musical analysis, note entry, quantization, tempo, looping, soundtrack variations, Monsters Ate My Birthday Cake, mastering, leveling, Leq, January, programming, my solo piano project, It Follows, temp love, name origins, using an alias, identity consolidation, listening habits, favorite games, games with great music, Jukio Kallio, Eirik Suhrke, etc.

Shutup Songwriters. Kyle and I talk about style & genre, entrepreneurship, expectations, creative process & the role of technology, instrumentation, process, old ideas, FEZ, “The Greatest Video Game Music, Vol. 2”, the internet, musical background, chiptunes, limitations/constraints, bitcrushing, tape emulation, sketching, switching from design to music, Berklee College of Music, going to school for music, navigating being an artist and a business, identity, etc.

Top Score. I talk about FEZ, musical narrative, “Fear”, “Glitch”, “Sync”, “Puzzle”, soundtrack variations, “Continuum”, Chopin, name origins, January, my solo piano project, etc.

2013

Bandcamp. Bandcamp did an episode about video game music and spoke to me and a lot of other independent game composers like Danny Baranowksy, Austin Wintory, Jim Guthrie, Laura Shigihara, Ben Prunty, and Jimmy Hinson.