Presentation: Abracadata!

I gave a microtalk at GDC 2018 as part of a session at the Artificial Intelligence Summit called 'Turing Tantrums: Devs Rant!'. I shared a thought experiment about exploring the possibility space of abstracted data relationships that cross disciplinary boundaries. Unlikely data marriages!

Transcript:

As a bit of an outsider I thought instead of ranting, it might be better to > share a thought experiment around an area of interest for me lately ...

AbracaDATA!

Games are a treasure trove of data.

A lot of what happens in games has something to do with numbers and math, and this stuff is great for creating and reinforcing the internal relationships in a game.

The most common relationship is player input, and how it drives just about everything. But let’s focus elsewhere.

Maybe your game has an enemy. A blue rectangle OH NO! It’s being shuffled around the world with some movement instructions. And you spruce it up with an animation, and maybe it feels good with some tweaking, but if physics and animation share their data, maybe they make even better decisions. Also that bearded man is now a giant dog thing.

Of course, you may not want all yours systems to share, and sometimes a hand-authored, isolated thing might be what you need. But tying camera movements to explosions or using text to drive a gibberish voiceover are examples of ways that tentacular relationships can improve the way a game feels.

Speaking of gibberish, I spend a lot of time thinking about sound, and lately I’ve been thinking about data sonification: using a data input to generate a sonic output. When a gameplay event occurs, like a footstep, we like to trigger a sound. This is a very useful and simple form of data sonification.

Another common practice is to map an object’s spatial position visually, to a relative spatial position, sonically. For instance, mapping the X position on screen to the stereo pan position of a sound, or mapping an object’s distance from the camera to a sound’s volume and brightness.

These mimic the way we hear things in the real world and are simple victories. But the examples I’ve given so far are well known and commonly employed. They’re perfect for clarifying, giving the player more coordinated feedback about what they’re interacting with.

But I want to talk about some of the less utilitarian places these relationships can go. Why not do more to re-contextualize this data instead? We could springboard ourselves into explorations of relationships that are weird, novel, counterintuitive and wonderfully asymmetric.

Here’s a silly one. There’s a game called ‘Sonic Dreams Collection’, where changing the size of the game window on the main menu changes the pitch and speed of the music. But what if it went beyond that? Suddenly you might care about window sizes in this strange new context, and it might elicit a reaction normally not reserved for the size of your window…

Or what if you tied the movement pattern of the ripples of a nearby river to the hair physics of your player, but only inject the data as you move away from it? What is this environment trying to evoke? Negative magnetism? (what does that even mean)

Finding meaning here can be a bit like trying to parse through a tarot card reading. You draw some random sources and try to map meaning onto their relationship.

Maybe the gag about window size didn’t inspire a deeper search for meaning, but what about a more opaque & esoteric data abstraction ? You might experience it as a kind of intelligence. And we could employ these relationships in subtle but cumulative ways.

Bees, for instance, perform a figure-8 called the ‘waggle dance’ that relays important locational info to other bees. You could create a dumb version of this cooperative relationship using abstracted data, and employ it within a system of similar objects. Maybe the relationship relies on distance between the objects, so that when they get close, they appear to share information with each other through sound or movement.

As worldbuilders, we could hint at a deeper ecology, through layers of data abstraction that might seem cooperative, adversarial, emergent, or mysterious and difficult to verbalize. We can suggest that with or without the player, the actors in this ecosystem are hopelessly entangled, and will carry on with their ebb and flow, just like we all do. Could be cool ...

The nice thing is that unlike the "waggle dance", you don’t need to prove it out with science. Maybe even the most arbitrary data relationship could feel like real intelligence if it’s been sufficiently abstracted. Players will conjure up their own interpretations, they like to do that. So you just need to convince them that they are experiencing something meaningful.

In other words, I think that by making different parts of a game communicate and share information in non-traditional ways, we can emulate the vitality we experience from real intelligence, and as a result it may be possible to manufacture a deeper sense of meaning and causality.

And the more liberal the different parts are in communicating with unlikely partners, the more things may start to get downright ecological.

And an interesting ecology of data relationships would probably have different kinds at play ... opaque ones, transparent ones, those that seem arbitrary, those that are rational, the esoteric (opaque+arbitrary), the absurd (transparent+arbitrary), the accessible (transparent+rational), the intelligent? (opaque+rational) (i totally made this up)

I think the abstraction and recontextualization of data can lead to all sorts of results. But if we sense there are meaningful relationships of cause and effect at play, that could lead us to suppose there is intelligence, and that could bring more depth to our experience.

So give it a shot! Things will definitely happen if you let your systems co-mingle.

You could let the volume level of a creature’s mating call drive the probability that other creatures respond in kind.

You could light a room using the average color of the last 60 frames.

You could take the wave propagation system used to drive visual wind FX and map it to the size of an NPC’s shoes.

But in any case,

AbracaDATA!

Or perhaps … Abraca...dada?(ism)

Link: GDC Vault: 'Turing Tantrums! AI Devs Rant'

Round-Up: Articles

2018

Native Instruments. We talk about creative process, tools, and what it was like to write for orchestra on Under the Silver Lake.

2017

Gameumentary. We talk about prog rock, composing for orchestra, personal philosophy, early musical memories, workflow, and social media.

Score It. We talk about silence, influence of nature, crossing mediums, temp score, motifs, and mixing genre.

2015

Vehlinggo. Aaron Vehling and I had a very thorough conversation in New York over dinner. We talked a lot about creativity and lifestyle.

Round-Up: Podcasts

From a Piece by Kilian Eng.

2018

The Game Music Podcast We talk about the early years of my career, procedural music in Mini Metro and Beasts of Balance, techniques used in Fez and Hyper Light Drifter, among other things.

2017

Lost Beat 6. We talk about creative process, philosophy, career experiences, and peership.

Game Dev Loadout. We talk about authenticity, building meaningful relationships, maintaining consistency over long periods and not focusing on too many different projects.

Checkpoints. We talk about creative processes, procedural audio systems, chiptunes, my evergreen love of Ice Hockey games, being exposed to Zelda in utero, Cave Story and my early musical steps creating entrance music for electronic wrestlers.

Indie Insider. We speak about my background, FEZ, Hyper Light Drifter, the art of seizing an opportunity, maintaining motivation and inspiration in your work over time, and negotiating pay as a contractor/freelancer.

Super Marcato Bros Podcast. I spoke with the Marcato Bros. about my career, and more specifically about my sound, and the specifics of projects like FEZ, Hyper Light Drifter, It Follows, Monsters Ate My Birthday Cake, Gunhouse, and Mini Metro.

2016

Adventures in Success. I recall the untold story of 'Jack Brooze', my eWrestling alter ego as a teenager.

Biya Byte. We talk about Hyper Light Drifter and It Follows, and work process. We also talk about VR.

Butterfly Effect. I spoke with Tristan Ettleman about the draw of chiptunes, the doors Fez opened, and writing the score to It Follows in three weeks.

The Collective Podcast. Ash and I talk about breaking into the music industry, the business side of making music, the importance of inspiration and collaboration, and where things are headed in the future.

Dear Air. I speak with Gamespot's horror podcast about my career, ewrestling, Rescue the Beagles and horror, naturally.

How Was Work? I talk with my friend Randy about moving to Los Angeles, workspace tinkering, and Beasts of Balance.

2015

People I Think Are Cool. I talk about It Follows, FEZ, creative freedom and expectations, live performances, switching format/medium, Wikipedia, musical cross-pollination, horror, Fantastic Fest, interviews, blogging, social media, patronage, streaming, income, work history, reading, Jodorowsky, Adventure Time, Mini Metro, GTA, Katamari, Keita Takahashi and my minimalistic philosophy and how it seeps into everything I do.

The Damn Fine Podcast. We talk a lot about vinyl and soundtracks.

Train Station at 8. We talk about video game music, live performance, listening habits, Doom, LA Noire, Tom Francis, Zan-zan-zawa-veia, Sonic the Hedgehog, The Floor is Jelly, domain names, Hearthstone, Peter McConnell, tone, soundtrack lengths, retro music facsimiles, The Hero’s Journey, Hideki Naganuma, Jake Kaufman (virt), Kirby, April Fools, Shnabubula, brain training, Jeff Bridges Sleeping Tapes, music in dreams, Mini Metro, etc…

Waveform City. Dropped by the local synthesizer museum to talk synths, naturally!

2014

Composer Quest. I talk about January, generative music, music theory, programming, The Floor is Jelly, postmortem, Monkey Island, iMUSE, time management, FEZ, Chopin, FZ, remix albums, hair, Rise of the Obsidian Interstellar, my solo piano project, songwriting, lyrics, advice for composers, etc…

Inside Video Game Music. I talk about my musical background, music rights, contract terms, NEUTRALITE, collaborative process, time management, FEZ, synthesis, “Adventure”, musical analysis, note entry, quantization, tempo, looping, soundtrack variations, Monsters Ate My Birthday Cake, mastering, leveling, Leq, January, programming, my solo piano project, It Follows, temp love, name origins, using an alias, identity consolidation, listening habits, favorite games, games with great music, Jukio Kallio, Eirik Suhrke, etc.

Shutup Songwriters. Kyle and I talk about style & genre, entrepreneurship, expectations, creative process & the role of technology, instrumentation, process, old ideas, FEZ, “The Greatest Video Game Music, Vol. 2”, the internet, musical background, chiptunes, limitations/constraints, bitcrushing, tape emulation, sketching, switching from design to music, Berklee College of Music, going to school for music, navigating being an artist and a business, identity, etc.

Top Score. I talk about FEZ, musical narrative, “Fear”, “Glitch”, “Sync”, “Puzzle”, soundtrack variations, “Continuum”, Chopin, name origins, January, my solo piano project, etc.

2013

Bandcamp. Bandcamp did an episode about video game music and spoke to me and a lot of other independent game composers like Danny Baranowksy, Austin Wintory, Jim Guthrie, Laura Shigihara, Ben Prunty, and Jimmy Hinson.