A nice synopsis of my career up to late 2013, touches on my time at the MIT Gamelab, and a lot of my projects.

Link: Gamesauce

Preface

Using code in games to actually create music, note by note, on the fly, is an underutilized idea that has potential to add huge value to games that aren’t even focused on music, by offering an extra sense of authorship to the player’s experience. From the simplest implementation, like the subtle musical sound effects in the menus of Super Mario Galaxy, to the generative music systems woven throughout Spore, the potential is broad and exciting.

Back in 2009, as an excuse to get better at coding and to make something in the process, I started exploring a game idea that involved a man walking around and licking snowflakes that made music. I created a bunch of sound assets of different tones, and based on what sounded good, I’d choose what asset to trigger next. Over time, I refined what I thought was a game into more of a music tool, adding lots of features such as chords, changing keys, different scales, and so forth. The following ideas all come from that tool, which is called ‘January.’ Check it out here.

Generating Something

So (maybe) you want to generate some music! For simplicity’s sake, we’ll create a program that uses the notes associated with the white keys on the piano, starting with the middle note of the keyboard, C3, and going up a single C Major Scale: C3, D3, E3, F3, G3, A3, B3, ending with the octave, C4. We’ll get a series of new note choices by figuring out what the last note played was. The program then chooses one randomly, plays it, and stores it as the new last note played. This creates a process that can be repeated to randomly generate a curated musical sequence.

// For the purposes of these examples I'm using Flixel and AS3, because it's what I'm familiar with.

// The last note played, stored after each new note is played.
public static var lastNotePlayed: Class;  

// Play a note!
public static function playNote():void
{
    var noteToPlay: Class = getNote();

    // This is Flixel's function for playing sounds.
    FlxG.play(noteToPlay);

    lastNotePlayed = noteToPlay;
}

// Get a new series of note choices, based on what the previous note was.
private static function getNote():Class
{
    var noteChoices: Array = [];

         if (lastNotePlayed == C3) noteChoices = [D3, E3, F3, G3, A3, B3, C4];
    else if (lastNotePlayed == D3) noteChoices = [C3, E3, G3];
    else if (lastNotePlayed == E3) noteChoices = [F3, G3, C4];
    else if (lastNotePlayed == F3) noteChoices = [E3, G3];
    else if (lastNotePlayed == G3) noteChoices = [C3, D3, E3, F3, A3, B3, C4];
    else if (lastNotePlayed == A3) noteChoices = [G3, B3, C4];
    else if (lastNotePlayed == B3) noteChoices = [G3, A3, C4];
    else if (lastNotePlayed == C4) noteChoices = [C3, G3, B3];
    else noteChoices = [C3, G3, C4];

    var choice: int = Math.round(Math.random() * (noteChoices.length - 1));
    var note: Class = noteChoices[choice];

    return note;
}

Generating Anything

The approach above works if you intend to stay in one scale, playing over one chord, but what if you want to have the generated notes adapt to different chords, scales, and eventually keys? To accomplish these things, a more sophisticated approach will be necessary. Storing your note choices as strings instead of using the sound classes themselves is one way to do this that will work. By separating out the logic choices from the actual sounds, it will be easier to adapt your system to new musical contexts.

A musician playing along to different chords in a single key is more often than not utilizing the various 'musical modes’ of that key. A brief synopsis of what these modes are may be necessary if you’re not familiar with the concept. The key of C Major is comprised of 7 modes, and they all use the white keys on a piano. Each mode is essentially it’s own 7-note scale starting from a different note of the C Major Scale (C, D, E, etc.). By starting the scale from a different place, you get a different sound, and each of these are called modes. If you start on C, you’re in Ionian Mode: C D E F G A B. If you start on D, however, you’re still in the key of C, but you’re in Dorian Mode (D Dorian): D E F G A B C. This concept continues on up the C Major Scale for E (Phrygian), F (Lydian), G (Mixolydian), A (Aeolian aka Natural Minor) and B (Locrian).

In order to make different musical choices based on what mode we’re currently using, we’ll need to come up with a modular system. Directly referencing the sound classes is no longer the best option, because the various modes, while possessing the same notes, tend to sound best when they use the notes in their own unique ways. A 'C’ note is going to sound quite different depending on whether you’re playing it over a C chord or a D chord. In order to accommodate this fact, the code below creates a unique object variable for each mode. We store that mode’s position in the scale (ie. Ionian is the 1st mode, Dorian the 2nd, etc) and use a 2D array of strings called 'logic’ to represent our potential note choices as scale degrees. A scale degree is the numerical representation of a note in the scale. For instance, in C Ionian, the 4th degree is F. They let us talk about a note in a musical key or scale without saying what the note actually is (ie. C, D, etc). Here is what our old C Major Scale would look like in this new form (Ionian), plus a new mode, Dorian:

public static const IONIAN: Object = { position: 1 };

IONIAN.logic = [

            /* 00 one */ ['two', 'thr', 'for', 'fiv', 'six', 'sev', 'one'],
            /* 01 two */ ['one', 'thr', 'fiv'],
            /* 02 thr */ ['for', 'fiv', 'oct'],
            /* 03 for */ ['thr', 'fiv'],
            /* 04 fiv */ ['one', 'two', 'thr', 'for', 'six', 'sev', 'oct'],
            /* 05 six */ ['fiv', 'sev', 'oct'],
            /* 06 sev */ ['fiv', 'six', 'oct'],
            /* 07 oct */ ['one', 'fiv', 'sev'],
            /* 08 else */['one', 'fiv', 'oct']
];

public static const DORIAN: Object = { position: 2 };

DORIAN.logic = [

            /* 00 one */ ['thr', 'fiv', 'sev', 'oct'],
            /* 01 two */ ['fiv', 'six', 'sev'],
            /* 02 thr */ ['one', 'for', 'fiv', 'six', 'sev', 'oct'],
            /* 03 for */ ['fiv', 'six', 'sev', 'oct'],
            /* 04 fiv */ ['thr', 'six', 'sev', 'oct'],
            /* 05 six */ ['one', 'for', 'fiv', 'sev', 'oct'],
            /* 06 sev */ ['thr', 'fiv', 'six', 'oct'],
            /* 07 oct */ ['one', 'thr', 'for', 'fiv', 'sev'],
            /* 08 else */['one', 'fiv', 'oct']
];

This is all fine and dandy, but these strings need to be hooked up to the sound assets in order to work. As a way of dealing with different modes, we can create a for loop that will fill a 'loadout’ object with the proper notes, assigned to the proper scale degrees.

// An array of the scale degrees, used as a reference for populating the loadout object. 
public static const DEGREES: Array = ['one', 'two', 'thr', 'for', 'fiv', 'six', 'sev', 'oct'];

// An array of all the notes needed to potentially form an eight note scale in any mode (Ionian is C3 - C4, Dorian is D3 - D4, Phrygian's E3 - E4, etc.) 
public static const C_MAJOR: Array = [C3, D3, E3, F3, G3, A3, B3, C4, D4, E4, F4, G4, A4, B4];

// The loadout object, to be filled with the notes of the current mode/key.
public static var loadout: Object = {};

// The last note played, stored after each new note is played.
public static var lastNotePlayed: Class;

// For the sake of the example, let's say the current key is F# Major.
public static var currentMode: Object = DORIAN;

// Play a note!
public static function playNote():void
{
    var noteToPlay: Class = getNote(currentMode); // Pass the current mode into getNote(), whatever mode you want it to be!

    FlxG.play(noteToPlay);

    lastNotePlayed = noteToPlay;
}

// Get a new series of note choices, based on what the previous note was.
private static function getNote(modeToUse: Object):Class
{
    // Whether we've found the last note played in the current loadout.
    var found: Boolean = false;
    // The note choices that we'll use to decide the next note to get.
    var noteChoices: Array = [];

    /* Make sure the current scale we're using to make logic decisions starts on the first note of the current mode. 
     ie. in Ionian, loadout['one'] = C3, loadout['two'] = D3, etc.
         in Dorian, loadout['one'] = D3, loadout['two'] = E3, etc. */                   
    for (var i:int = 0; i <= DEGREES.length - 1; i++)
        loadout[DEGREES[i]] = C_MAJOR[i + modeToUse.position - 1]; // - 1 is to account for starting at Zero.

    // Iterate through the loadout, find the last note played & store the associated set of choices to noteChoices.
    loop: for (var j: int = 0; j < DEGREES.length - 1; j++)
    {               
        if (lastNotePlayed == loadout[DEGREES[j]])
        {                    
            noteChoices = modeToUse.logic[j];
            found = true;

            break loop;
        }
    }

    // If the last note played was not found in the loadout, use the 'else' choices from the logic array.
    if (found == false)
        noteChoices = modeToUse.logic[08]; // [08], or the last set of arguments in the logic array, used like an else statement.

    // The numerical choice from the array of choices that we will use.
    var choice: int = Math.round(Math.random() * (noteChoices.length - 1));
    // The note that we will return to the playNote() function and eventually play.
    var note: Class = loadout[noteChoices[choice]] as Class;

    return note;
}

The system above will work great for moving through various modes in a single key, but what if you want to move through the other 11 keys, too? The most straightforward solution would be to create arrays for all of the keys. You also would of course need to create assets for all the new notes that you are using.

// Arrays of all the notes needed to potentially form an eight note scale in any mode, in any key. s = #, ie. Cs3 = C#3

public static const C_MAJOR : Array = [C3, D3, E3, F3, G3, A3, B3, C4, D4, E4, F4, G4, A4, B4];
public static const Cs_MAJOR: Array = [Cs3, Ds3, F3, Fs3, Gs3, As3, C4, Cs4, Ds4, F4, Fs4, Gs4, As4, C5];
public static const D_MAJOR : Array = [D3, E3, Fs3, G3, A3, Cs4, D4, E4, Fs4, G4, A4, Cs5];
public static const Ds_MAJOR: Array = [Ds3, F3, G3, Gs3, As3, D4, Ds4, F4, G4, Gs4, As4, D5];
public static const E_MAJOR : Array = [E3, Fs3, Gs3, A3, B3, Ds4, E4, Fs4, Gs4, A4, B4, Ds5];
public static const F_MAJOR : Array = [F3, G3, A3, As3, C4, E4, F4, G4, A4, As4, C5, E5];
public static const Fs_MAJOR: Array = [Fs3, Gs3, As3, B3, Cs4, F4, Fs4, Gs4, As4, B4, Cs5, F5];
public static const G_MAJOR : Array = [G3, A3, B3, C4, D4, F4, G4, A4, B4, C5, D5, F5];
public static const Gs_MAJOR: Array = [Gs3, As3, C4, Cs4, Ds4, Fs4, Gs4, As4, C5, Cs5, Ds5, Fs5];
public static const A_MAJOR : Array = [A3, B3, Cs4, D4, E4, G4, A4, B4, Cs5, D5, E5, G5];
public static const As_MAJOR: Array = [As3, C4, D4, Ds4, F4, Gs4, As4, C5, D5, Ds5, F5, Gs5];
public static const B_MAJOR : Array = [B3, Cs4, Ds4, E4, Fs4, A4, B4, Cs5, Ds5, E5, Fs5, A5];

// For the sake of the example, let's say the current key is F# Major.
public static var currentKey: Array = Fs_MAJOR;

// Then, you would adjust the for loop that fills the loadout to check for the current key's scale:
for (var i:int = 0; i <= DEGREES.length - 1; i++)
    loadout[DEGREES[i]] = currentKey[i + modeToUse.position - 1];

And voila! With these systems in place, you now have access to all 7 modes in all 12 keys. That’s 84 different scales! Other applications, such as chords, can be generated in a similar fashion, using 2D arrays to store chord choices and the loadout system as before to make sure you’re in the right key and the right mode. You could also use a timer to fan out the playback of the chord notes, if you want to be fancy. The possibilities are endless!

Considerations

If you start to experiment and your generative music system gets more complex, you will probably run into some 'musical bugs’. Namely, you will find dissonances (notes that don’t sound good together) and other unwanted events creeping up from time to time. You’ll want to consider preventing certain sequences from happening, such as repeated notes, 'trills’ (hearing the same note twice with another note in between), and if you start changing keys and scales on the fly, you may want to limit what kind of notes play around the time that those events happen. These kinds of problems can often be solved with conditional checks, often against the lastNotePlayed. If you want to check for trills, you could create a second variable, secondToLastNotePlayed, and continuously log the last two notes. This could also open the possibility space for what kind of choices you make about which notes come next, if you’re up to the challenge.

At the end of the day, the most sophisticated music generating code won’t amount to much if you don’t have someone musical onboard, using their ears, making decisions about what sounds good. Maybe that’s you, or maybe not, but hopefully either way this will be a decent springboard for thinking about using code to generate music, instead of the other way around.

I gave a five minute micro-talk as part of the 'Indie Soapbox Rant', at the GDC 2013 Independent Games Summit.

Transcript

For most of human history, our experience of music has mirrored the fleeting nature of life. Each performance was unique and unpredictable, heard once, and then gone forever. This impermanence, which you can still hear if you go to a live show, has great potential to create meaning. If you're lucky, the events you are witnessing might even feel important.

The invention of recording technology has done a lot to change our relationship to music, and to introduce the possibility of repeating music in an effectively identical way. Many of us have become accustomed to listening to recordings, and in games we have become accustomed to listening to loops. Video game music is generally made up of permanent elements (ie. assets). How can we bring the sum of those elements together to mimic the impermanence of live music?

Imagine a really great game that takes maybe fifteen minutes to play. Every time you play, the interactions feel fresh, and you could play it over and over and not grow tired of it. Now imagine that same game also has a one minute piece of music that loops. If you play the game 100 times, the gameplay may be varied enough to keep your interest, but you're hearing the same one minute piece of music 1500 times...

...

Why would you do this to your player? Why would you even bother having music at all if you're going to invite them to hear something so much that it's rendered meaningless? And yet, repeated music has some strengths. It can do a strong job of communicating an idea to a player. When you hear the underground music in Mario, you know you're underground and have to look out for a different set of obstacles. Also, repeatable music has the benefit of allowing creators to finely tailor the intended listening experience. The musical outcome is known - it's predictable, which can be good sometimes. So how can we embrace the strengths of repeatable music while softening its ugly edges in a way that pays tribute to our long tradition of live performed music?

Here are some examples that I think are doing this successfully:

Kentucky Route Zero uses ambience first and foremost, triggering music sparsely and tastefully to underline important moments. In the first scene of the first act, after the protagonist returns from the basement of a gas station, an ambient piece of music communicates progression in the story. You only hear this piece once, and instead of looping it withers away. This approach gives the sense of passing time, and is present throughout the game.

As you move around the audiovisual environment of Proteus, the music reacts in a way that simultaneously feels rational and yet unpredictable, and in this way has a strong similarity to live music. You rarely feel like you are hearing the exactness of prerecorded music.

Instead of using structured, linear recordings of music, January is made up of small assets representing single musical notes. They are continually rearranged in musical sequences that never sound quite the same. The intended effect is to be similar to the act of a musician improvising, in terms of which notes to play, how and when.

...

Silence can give the player room to breathe, or evoke a particular emotion such as the eeriness of the film No Country for Old Men.

Quotes
"Music is the space between the notes" - Claude Debussy

"Don't play everything (or every time); let some things go by. Some music is just imagined. What you don't play can be more important than what you do." - Thelonious Monk

The fact that we all pass away gives meaning to our lives. If we can honor the tradition of letting music pass too, and other things in games, then maybe we can create more meaningful experiences.