The Jam_bot, a Real-Time System for Collaborative Free Improvisation With Music Language Models

Tasked with the challenge of designing a Generative AI system that could improvise on stage with GRAMMY-winning keyboard virtuoso Anon Visiting Artist, we developed the "jam_bot", a real-time performance system, that could match his eclectic improvisational aesthetics. We debuted the jam_bot at a high-stakes sold-out concert to critical acclaim, realizing a series of virtuosic tightly-coupled Human-AI free improvisations in varying musical styles. Reflecting on our year-long collaboration with the artist, we summarize learnings for AI researchers and musicians on the adaptations needed to turn state-of-the-art symbolic music Language Models (LMs) into jam_bots and the engineering required to make them performance-ready. We focus on three aspects: First, to enable jam_bots to take on different musical roles, such as lead, accompany, or engage in call and response, we adapt music LMs to take on different interaction strategies by modifying the context and conditioning signals they take in. Second, for jam_bots to match the style needed for each piece, we describe how AnonArtist intentionally structures his improvisation in order to finetune music LMs to enable these strategies. Third, we show the optimizations needed to run music LMs in real-time and how to embed them in a low-latency multi-threaded system that listens, and prompts and schedules model generations seamlessly. We hope these insights enable more musician-AI symbiotic virtuosity.