How Flex Council's Josh Levy Uses AI to Make His Own Brand of Eerie, Ghostly Electronica
Josh Levy writes music for his project Flex Council using neural networks. And it sounds like nothing else we've heard. His compositions have the ghostly, otherworldly character that is hard to mimic. They also belong to a rare breed of music that instantly provokes the question "What are these sounds? How did he make this?". We've had the privilege to ask Josh these questions in person.
— Can you please introduce us to your working method? How do you build your songs and why did the preparation and learning process take you two years?
— I used OpenAI's Jukebox algorithm to improvise over loops in the style of various artists like Cocteau Twins and The Weeknd, any of about 4000 artists it was trained on. I was going for a nostalgic, holiday theme, so I looked for loops and algorithmic models which supported that theme. It turned out that the most striking elements produced by Jukebox were the vocals — they sounded like lost ghosts. I liked the emotional quality even though they were sung by a computer.

Jukebox has the capacity to surprise, which is strange and exciting to say about a non-human system. It can also produce a lot of junk, so a big part of the process was listening to all the renders and picking out the best bits. I was also lucky enough to discover the source-separation algorithm Spleeter right before I started this project. That helped to open up the mix and focus on the vocals. After I had all the computer-generated parts isolated and repaired, it was like making any other track in Ableton.

It took two years to figure out a good way to incorporate AI into my workflow. I started using MIDI AI transformations for a Grimes remix about a year ago and then I tried waveform AI transformations with a lot of different algorithms. After that I took a few months taking apart and rebuilding Magenta's DDSP algorithm, which helped me to understand how neural nets think. I also had to brush up on my linear algebra!
— Does this require a lot of special knowledge, skills and expertise? If so, what knowledge and skills does it require? In other words, if I go to OpenAI's Jukebox's website with an intention to download the code and generate experimental music to later rework it into something original, what exactly do I need to know?
— Knowing a little Python helps to get a better variety of results, but you could probably get by without it if you're good with computers. Some knowledge of Unix commands, digital audio, and remote computing is useful, too.
— You mentioned using "MIDI AI transformations and waveform AI transformations with a lot of different algorithms". Can you please explain to our readers how both types of transformations work (what transforms into what in both cases)? And what did the different algorithms do?
— The MIDI input is compressed into a higher-level representation of itself, called the latent space, which is thought to correspond to musical characteristics like style and genre but no one really knows for sure. The neural net has learned this latent space by analyzing large numbers of MIDI files, and once it knows how your input MIDI file fits in with what it has already learned, it can alter the MIDI in new ways. For example, it can naturalistically interpolate between two melodies or continue a melody (or drum part) in a way that makes sense musically.

Waveform transformations work on the same principle but they are a lot more difficult because digital audio has so much more information than MIDI. Some algorithms work by training on a big dataset and then rebuilding representative examples of that dataset sample by sample. Other algorithms take more of a synthesizer-based approach, using a smaller dataset and modeling learned DSP elements like sine wave generators, filters, and reverbs. All the algorithms have their distinctive sound, but the results depend mostly on the dataset audio.
— I've always had a feeling that some of the best melodies come to us by chance when our brain is fooled by the sounds from the environment – for example, when we hear a car honk or a siren sound and perceive at as a melodic fragment by mistake. From what you have told us it seems like neural networks can do this job for us. What is the process like for you – are you waiting for the system to really surprise you or do you manipulate it in a way to get the results you expect? And also – which tones and melodies generated by AI were the biggest surprises for you? On which tracks have you used them?
— I know exactly what you mean. I get the best melodies when I'm not even thinking about music, or in dreams. Or humming along with a TV show. It can be very difficult to find those magic moments while sitting in front of a computer. So I'm happy for AI to do some of the heavy melodic lifting for me. But I'm usually feeding it some kind of musical seed, like a loop or chord changes, or the dataset. Also, there's something very appealing about working with an entity which has distilled so much human music in a way we can't entirely understand.

The biggest surprises come from Jukebox. It'll often suddenly break into a different genre or tempo, and most of the time this isn't what I want. But every once in a while the algorithm has a moment of coherence, usually just enough to extract an interesting loop. It's kind of like a crazy person who suddenly says something profound.

Usually a track is built around 10-15 of these loops. I used this technique on "December Song" and the Low and Magdalena Bay remixes.
—The algorithms you use are capable of generating what sounds like human voices and conventional instruments. But as of now, most of the sounds they produce have this shaky, ghostly, eerie character. And it's exactly this quality that makes your music so special and evokes the feelings and images that little other music can evoke (Burial would be another example) – the otherwordly landscapes, encounters with the supernatural and the existential loneliness. As the technology advances, the AI-generated sounds may lose this quality. Are you going to stick to this emotional palette you have found by chance or are you going to move on with the technology?
— Ah, I love Burial! Such a unique sound. I think the ghostly effect you're talking about will probably go away with time, but some people will always want to use the older versions of the algorithm, like people want to use an old synth now. This version of Jukebox will always be there if I need it, but I will probably move on to newer versions and other algorithms, as well as non-AI music.

Right now I'm working on a Max for Live device which alters the MIDI in a clip as you perform. It looks a bit like cellular automata. I'll make that available for free when it's done. Also, I can't wait to get a set together and start performing once live music comes back.
—Thank you for the interview! Before we say goodbye, can you recommend your latest favourite song to our readers (written with or without the help of AI)?
— Awesome, thanks! I would have to say "LA Freeway" by Guy Clark because I can't stop playing it on guitar.