Work in Progress
Thursday 5 September, 12:20 - 15:15 - HKU IBB
The Stickatron, the Elastiphone and the Perceptron: three co-played entangled instruments
Steve Symons


Click here for the video presentation
Abstract:

Emerging research in the field of digital musical instruments (DMIs) has drawn on entanglement ideas from post-human theories as a way to discuss areas such as open mixer feedback musicianship [1, 2] and human-AI musical creativity [3]. Such research extends DMI research by moving from the idea of ‘playing’, as in controlling an instrument, to an experience where the human player and the system mutually engage. Little research, however, has considered applying entanglement to two or more players collaborating within a single musical system.

I propose to present a series of 3 instruments that are part of a wider practice of applying post-human ideas of entanglement [4] and apparatus [5], alongside participatory sense-making ([6]) to shared musical experiences. We can see elements of these paradigms appearing in several existing multi-player projects. Such as SensorBand’s SoundNet [7], where a network of interconnected climbing ropes (mapped and sonified via tension sensors), enmesh players’ individual movements. While Fels directly couples players with the Tooka [8]; here a constant input of air from two participants onto a shared pressure sensor ensures that any sound is co-produced. TONETABLE [9] entangles individual contribution by creating an interactive environment facilitated through a fluid dynamics simulation. Players don’t interact directly but through affecting the simulated water.

The Stickatron, the Elastiphone and the Perceptron are being developed as part of a series of studies that explores how entangled instruments can enable enjoyable, embodied experiences and intimate connections between people as they are played.

The Stickatron is played by players holding one handle each of the interface and, together, they lift then move the stick in space to explore a synthesised world. Pitch (vertical angle) and yaw (horizontal rotation) both have an effect, while the position of the stick’s mid-point in the instrument space (a 1.5m sided cube centered on the sensing GameTrak) acts as a ’3 dimensional playing cursor’.

The Elastiphone replaces control of the synthesis system with a virtual elastic band between the players hands. The same data is extracted as for the Stickatron, but with the addition of the separation between the players’ hands. Like the Stickatron, VCV Rack is used for audio synthesis.

The Perceptron uses machine learning (a multi-layer perceptron) to combine the players' movements and to generate control data for a second machine learning model to create audio in real time [10].


Strung Along
Jack Hardwick


Click here for the video presentation
Abstract:

Strung Along is a novel violin-based interactive music system which offers real-time generated chordal accompaniment to a monophonic melody played by a violinist. The system allows momentary control of timbre and chord voicing by modulating tracked bowing parameters. It is a combination of two sub-systems, each capable of functioning independently, but are integrated into a combined system: the chord generation sub-system; and the bow tracking sub-system.

The chord generation sub-system employs a novel machine learning-based approach for generating chords, in which chords are represented as chroma histograms. When a new melody note is performed, the LSTM model generates a chroma histogram representing the chord to accompany that melody note. The input sequence consists of the new melody note, previous melody notes, and previously generated chroma histograms. This results in a new chord that is influenced by prior musical context.

The model is implemented in a real-time system in which melody pitch is extracted from live violin performance, and one chroma histogram is generated per melody note performed. The chroma histograms are then algorithmically voiced as a set of MIDI notes which are sounded by a digital sound engine. Each chord is generated and voiced in under 40 milliseconds. Currently, this sub-system is effective for generating diatonic chords to diatonic melodies and displays frequent cadential figures between subsequent chords. However, the model performs poorly in modelling chord order in sequences of three or more chords, therefore long-term musical coherence is limited.

Secondly, the bow tracking sub-system adapts an existing approach by Pardue et al. (2013, 2015) for tracking bow position and bow force. This approach uses four distance sensors mounted to the underside of the bow stick facing towards the bow hair. As the bow is used during performance, the measured distance at each point between the bow hair and bow stick is different at each sensor location. A dataset consisting of sensor measurements (input), and measured bow position and bow force (output), is used to train a regression model which predicts bow position and bow force from the distance sensor measurements.


A reinforcement learning framework for live musical agents
Vincenzo Madaghiele


Click here for the video presentation
Abstract:

Artificial musical agents are machines that can automate one or more musical tasks in the realm of music creation. My PhD project proposes the study of autonomous musical agents in the context of musical improvisation through their iterative development and evaluation. As part of the project, I am working on a framework to train artificial musical agents using reinforcement learning. The framework has the objective of allowing users to train a musical agent for a custom musical setup and musical goals. The implementation focuses on small, curated datasets, fast training times and integration with creative coding platforms such as Pure Data, Supercollider and Max/MSP.

Reinforcement learning is an unsupervised machine learning technique in which an agent learns to solve a problem by repeatedly acting in a predefined environment. During training, the agent receives a positive reward when its action in the environment is considered successful towards the goal, and a negative reward when the action is considered not successful. The aim of the agent is therefore to develop a strategy that allows it to receive as much positive rewards as possible. The reinforcement learning model is well suited to describe the realm of musical improvisation, in which musicians continuously adapt their musical choices with respect to what other musicians are playing, and the audiences’ reactions.

In the envisioned use case of this project, the agent controls the synthesis parameters of a user-defined digital synthesizer, adaptively changing the values of the parameters to adjust to what a live musician is playing. Such a system is characterized by three main components: analysis, patterning/reasoning and synthesis. These components determine the agent’s behavior in a musical interaction. In the reinforcement learning framework I propose, the analysis is done through a collection of perceptual sound descriptors which analyze the incoming sound of the musician and the agent in real time, such that the agent is aware of the musician’s and its own sound, and it can respond appropriately.


Wirwarp
Görkem Arıkan, Joris Takken, Lucas Vonk, Roman Willems, Rahsaan Bleijs, Floortje Smehuijzen & Temo Gonzalez


Click here for the video presentation
Abstract:

Wirwarp is a new interface for musical expression that incorporates conductive rubber-cord stretch sensors. The cords create a dynamic mesh on a wooden frame, providing a touch canvas for sonic exploration through the manipulation of tension dynamics. Playing the instrument involves warping and tangling the mesh – pulling the cords, pushing the nodes, and adjusting the frame's angle with movable hinges.

The Wirwarp comprises three main components: the sensor frame, a MIDI-enabled microcontroller box, and a Raspberry Pi box. The interface can function as a custom MIDI controller by plugging into MIDI-enabled devices via a USB port or by sending OSC messages over a network. In SoundLAB workshops, Wirwarp connects to the Raspberry Pi, which runs our sound engine written in RNBO, MaxMSP.

Since the conductive rubber cords act as resistors, their contact alters each other's resistance, making the performer a part of the circuit. This introduces a desired chaos and unpredictability to the control data, enhancing the playing experience beyond a straightforward interaction. Additionally, the foldable frame allows for simultaneous adjustment of sensor tension, providing the instrument with a deeper range of control.

The Wirwarp is developed in collaboration with 4th-year Music and Technology students from HKU, and SoundLab, Muziekgebouw Aan ‘t IJ. The project is commissioned by Sonic Acts for the Touched by Sound project that takes up the legacy of the pioneering performer and instrument maker Michel Waisvisz (1949–2008). Together with Kristina Andersen and Sonic Acts, artist and composer Tarek Atoui explores Waisvisz's archives as a catalyst for new forms of experimentation.


Experimenting with a GenAI MIDI Module
Charles Martin


Click here for the video presentation
Abstract:

This submission describes work in progress towards building cheap, minimal, battery-powered generative-AI music systems that can interface with commercial synthesisers and other equipment. The idea is to facilitate exploration of human-AI collaboration and creation of new AI-integrated music. While machine generation of symbolic music and digital audio are hot topics there have been relatively few digital musical instruments that integrate generative AI, partly due to the complexity of training and deploying artist-centered generative AI systems. Our approach uses small artist-collected datasets that can be trained on a regular computer and an inexpensive single board computer (Raspberry Pi Zero 2 W). The module interfaces to synthesisers via MIDI.

The GenAI system uses a mixture density recurrent neural network (MDRNN) to model sequences of interactions with any (fixed) number of MIDI controls (notes or CCs). Our software is a Python program running on the Raspberry Pi that generates musical data and sends it via a serial connection to an external MIDI-enabled electronic musical instrument. Machine learning predictions are performed on-device and no internet connection is necessary. Our software allows the musician to configure the MIDI controls and devices used as input and output. This allows sophisticated cross-mappings and for experimentation with different configurations using the same trained model. In addition to MIDI over USB or direct serial connections, IO can use OSC over UDP or WebSockets.

For this work-in-progress presentation we will demonstrate our GenAI MIDI Module with a commercial synth such as the Roland S-1. This represents one of the most physically compact and lowest cost generative AI music systems presently available.


TouchingWires
Sandy Ma


Click here for the video presentation
Abstract:

Revolutionary textile artist Anni Albers said "We touch things to assure ourselves of reality". Touching Wires introduces a textile interface for musical expression that invites performer to investigate tactility and movement for embodied interactions. This is an undergraduate thesis project due for completion in October 2024.

Tactile interaction is a cornerstone in how humans build relationships in the world. Tactility within Human-Computer Interaction fields have long been defined by rigid materials. With the increasing presence of artificial intelligence in creative practices, critical reflection on the relationships and definitions of collaboration with AI agents is required. This project aims to investigate the role of soft textile interfaces in navigating relationships between human users and computer systems. The technological and digital have been defined by progression towards the indestructable device - cold and solid materiality. Hwang et al. examine the perceptions of artificially intelligent agents and found that users viewed AI as an entity within a device rather than an abstracted consciousness. Creating a textile based interface extends the investigation on device based agents and will aim to explore how softness and gesture may impact relationships with AI.

This system uses a Bela Mini interfacing with a Trill Craft to control sensors. Wiring connected to a unit of copper tape is connected underneath a quilted surface to provide a capacitative interface.