Alien Acoustics: The Marriage of Music and Machine Learning

This is an article I wrote for my school's yearly magazine back in 2021, painstakingly transcribed for you. As a music scholar, I remember my brief being write something about music, which is rather broad. I often try to blur the lines between my passions, and this article was no exception, addressing the state of the art in generative music.

I clearly tried to keep politics out of my discussion, keeping it a light review of the technology, but you can still see some barbs poke through.

For many years, scientists struggled to craft techniques to deal with big data – datasets with thousands or millions of entries – that would allow them to identify patterns and trends to use in their research. Machine learning (ML) was the discovery that turned days of repetitive work by humans, who struggle not only with pace but with objectivity and consistency, into algorithms that could run in minutes to return whatever result was desired.

There are many different forms of ML, each specialising in a different type of dataset or result, but they all follow the same essential steps of simplifying the data until they can make a good guess without the researcher having to lift a finger once the process has started. Since its inception, ML has been applied to all manner of fields, from simulating fluid dynamics to analysing social media posts, and now it's beginning to enter the world of music.

So how do you take a tool designed for finding patterns in large lists of numbers and uses it to make music? Unsurprisingly, there are many completely different approaches, each of which offers a different future for music production. One implementation, used mostly in live performance, takes an input – the temperature, a performer's position on a stage, the number of likes on a YouTube video, or any other stream of data – and uses that data to tweak the settings of a synthesiser or the relative volumes of concurrent loops to create a constantly shifting soundscape. While these performances can create wonderful effects, they aren't interesting from a scientific perspective: not only is the implementation easy with modern ML tools, a programmer could write a traditional algorithm to achieve the same effect. As such, the rest of this article will focus on the other two main uses of ML in music: generating entire pieces in the form of either sheet music or raw audio.

Google's Magenta project is a library of code that can be used by anyone to create their own ML applications to do with music or art. My favourite of these is called Transformer, which was designed to generate long musical performances of between one and five minutes. Their app was trained on hundreds of thousands of piano recordings on YouTube that had been automatically transcribed by another Magenta project: Onsets and Frames. It then created MIDI files (sheet music for computers) that are played by a virtual piano. due to the vast array styles available in dataset, the results vary wildly from Baroque-sounding pieces to experimental jazz. The researchers quickly identified an issue, however: the algorithm generated thousands of new pieces but had no way of judging whether they were any good. While a human composer rejects many of their ideas or continually edits them until they produce a few pieces they're happy with, Transformer's approach creates many mediocre works. Most of the pieces I listened to had some emotional content, but I felt that each had some element that held it back and meant I'd only ever listen to it as background music: something that fills the air but doesn't say so much as to distract you. Pinning down what I felt was missing was hard, however, and it took listening to another project to realise what it was.

OpenAI's Jukebox takes a musical style and lyrics as input and returns a full audio track of what you could convince someone was a fully produced song. After listening to some examples in the style of Alan Jackson, Elvis Presley, Katy Perry and many others, I was astounded at how well the songs' elements coexisted. The backing tracks were in the exact style requested and everything was in the correct key and following the same chord progression. There was a glaring issue, however: the vocals made me feel seasick. Despite accurately reproducing the sound of each vocalist the way they faded in and out and blended words together was something out of a fever dream. Despite great promise, this technology has some way to go before it is consumer-ready. It was this property that resolved my thought on Transformer: the reason the pieces sounded wrong, empty or random was that I was judging them as if they were human pieces – that's like judging one culture's music against the rules and expectations of another, in this case, it's even more extreme though, since ML algorithms aren't another human culture, they're alien.

This article is not concerned with AI overlords or robots ruling the world, but it is impossible to write about ML without acknowledging that it is not human. When an algorithm makes music, it isn't concerned with how much money it'll make or how well it expresses its inner feelings. It has neither an ideology it wants to push nor a government it seeks to satirise. This lack of motivation is often used as the main argument to say ML music is bad, unoriginal and Not Art. This stems from the pre-19th century Romantic shift that took expectations of art from imitation to expression; the idea of the artist or composer as someone with something important to say still plagues our society as the only true form of art. ML music, on the other hand, is far more analogous to the Baroque and Classical eras, when composers would write hundreds of pieces in a specific style, often using lyrics given to them (see Bach's chorales for example) and without requiring any emotional attachment to the music.

So where does this music fit into the current musical landscape? Given where the technology is now (a necessary caveat: the pace of technological advancements makes it near-impossible to predict), I'd suggest that ML techniques will proliferate in a variety of different ways. Artists will use them to push the boundaries of their sounds, using them to subvert their human expectations of timbre and structure. Producers will use ML to quickly create original backing tracks for singers, leading to faster production, less money spent on professional musicians to record them, and more certainty that the audience will enjoy it. Perhaps most importantly though, a whole new genre of music will emerge: ML background music. If you've ever put on some light jazz or listened to lofi as you worked, one day those will be replaced with live-composed ML music that constantly evolves and changes but is designed perfectly to maximise your work efficiency and not distract you, and that day may be sooner than you think.