Machine Learning for Musicians and Artists

Have you ever wanted to build a new musical instrument that responded to your gestures by making sound? Or create live visuals to accompany a dancer? Or create an interactive art installation that reacts to the movements or actions of an audience? If so, take this course!

In this course, students will learn fundamental machine learning techniques that can be used to make sense of human gesture, musical audio, and other real-time data. The focus will be on learning about algorithms, software tools, and best practices that can be immediately employed in creating new real-time systems in the arts.

Specific topics of discussion include:

• What is machine learning?

• Common types of machine learning for making sense of human actions and sensor data, with a focus on classification, regression, and segmentation

• The “machine learning pipeline”: understanding how signals, features, algorithms, and models fit together, and how to select and configure each part of this pipeline to get good analysis results

• Off-the-shelf tools for machine learning (e.g., Wekinator, Weka, GestureFollower)

• Feature extraction and analysis techniques that are well-suited for music, dance, gaming, and visual art, especially for human motion analysis and audio analysis

• How to connect your machine learning tools to common digital arts tools such as Max/MSP, PD, ChucK, Processing, Unity 3D, SuperCollider, OpenFrameworks

• Introduction to cheap & easy sensing technologies that can be used as inputs to machine learning systems (e.g., Kinect, computer vision, hardware sensors, gaming controllers)

• Introduction
• What is machine learning? And what is it good for? We’ll introduce a variety of artistic, musical, and interactive applications in which machine learning can help you create new things.
• Classification, Part II; Design Considerations
• In this session, we’ll take a deeper look at what it means to build a good classifier, and we’ll explore some common and powerful classification algorithms, including decision trees, Naive Bayes, AdaBoost, and support vector machines. We’ll also dig deeper into an exploration of how learning algorithms can be integrated into your own work most easily to achieve your desired outcomes. You’ll get a chance to explore these new algorithms and continue to work them into your own projects.
• Working with Time
• In this session, we’ll talk about algorithms that have been specifically designed to help you make sense of changes in data over time. Rebecca will dive into dynamic time warping, and guest lecturer Baptiste Caramiaux will discuss Gesture Variation Follower, an algorithm designed with the arts in mind. You’ll continue to get plenty of opportunities to apply temporal modeling algorithms to real-time data analysis.
• Classification, Part I
• In this session, we’ll cover the basics of classification, which can be used to make sense of complex data in a meaningful way. We’ll look at two classification algorithms: nearest-neighbor and decision stumps. You’ll be introduced to the Wekinator, a free software tool for using machine learning in real-time applications.
• Sensors and Features: Generating Useful Inputs for Machine Learning
• Machine learning makes it easier and more fun to work with all sorts of real-time sources of data, including real-time audio, video, game controllers, sensors, and more! We’ll talk about good strategies for making sense of the data you’ll get from different inputs, and for designing feature extractors that make machine learning easier. We’ll be encouraging students to develop their own feature extractors and share them with each other!
• Developing a Machine Learning Practice; Wrap-up
• Guest lecturer Laetitia Sonami will give a masterclass in which she discusses the way machine learning fits into her own work building new musical instruments, and Rebecca will discuss practical tools, boos, and resources you can access for furthering your work in this field.
• Regression
• We will discuss the fundamentals of regression, which can be used for creating continuous mapping and controls. We’ll explore the use of linear regression, polynomial regression, and neural networks to create new types of interactions. You’ll gain hands-on practice exploring regression algorithms and starting to apply them to build your own systems.