Machine Orchestra

Human-Computer Music Performances Use

System That Links Music & Musical Gestures 

Every musical sound comes from a specific way that an instrument is played. With modern technology such as sensors, signal processing, and sometimes machine learning algorithms, researchers can determine the precise musical gesture used to produce a particular sound on an instrument. The ability to recreate musical gestures from sounds can be used for interactive human-computer music performances, music transcription, and other innovative applications.

In a new study, researchers have developed a method for capturing musical gestures and mapping them to sounds that overcomes some of the disadvantages of previous methods. Adam Tindale, Ajay Kapur, and George Tzanetakis – all trained musicians and computer scientists working at the University of Victoria in Victoria, Canada – have described the new method in a study to be published in IEEE Transactions on Multimedia. The method grew out of the authors' experiences in developing instruments for interactive human-computer music performances. At the time, Tindale and Kapur were both completing their PhDs at the University of Victoria. Tindale now works at the Alberta College of Art and Design and Kapur works at the California Institute of the Arts and the New Zealand School of Music.

As the researchers explain in their study, there are two main approaches for capturing musical gestures. One approach is direct acquisition, which involves attaching permanent sensors to instruments to create “hyper-instruments.” However, this approach is often invasive to performers and requires modification of expensive instruments. The second approach is indirect acquisition, which involves using a microphone to capture the sound, as well as sophisticated signal processing and machine learning algorithms to extract gestures from the sounds, which requires large amounts of training.

The researchers' new method is somewhat of a hybrid of these two approaches. They temporarily attach sensors to an instrument to capture musical gestures and a microphone to capture sound. This data is analyzed, and the gesture-sound mappings are used to train machine learning models to extract gestures from sounds only. These machine models then create what the researchers call a “surrogate sensor,” which behaves like the original invasive sensor but is not attached to the instrument. The surrogate sensor can determine the musical gestures based only on the analyzed sound captured from the microphone. 

(For complete article, plus vids of the Machine Orchestra in action, see PhysOrg.Com)

 

-Maureen Lang

 

Comments

[Anna W.]

Lovely post. Thanks for forwarding the link to it. The machine orchestra sounds quite nice Maureen, but can it really beat out the Coinola Orchestrion we heard at the Arboretum show last summer?

[Robert Murray]

I'm looking forward to checking it out when I get home Maureen. Thanks for posting this.

Best,
Bobby

[Robert Murray]

Brilliant jazz guitarist Pat Metheny on the Orchestrion.

http://www.youtube.com/watch?v=KsYEOUKS4Yk&NR=1

[Maureen Lang]

-Well, Anna W., the Machine Orchestra article initially caught my attention because of seeing/listening to all those terrific old crank organs, Orchestrions, player pianos, & all the other music animata over at the L.A. Arboretum show. Make sure to have a look at Robert Murray's music link in the comments below yours!

-Great to see you commenting on this post, Bobby. That vid of Pat Metheny talking about being inspired by an Orchestrion, player piano, & the modern equivalents of same that he's working on is just wonderful- thanks so much for sharing it.

[Craig V.]

Technology imposes and we disposes. All joking aside, a very interesting article.

[Jeff B.]

Maureen, here's a link to the Museum of Music Automatons in Seewen Switzerland you might be keen to look at. The exhibits include orchestrions, Welte player pianos and Welte philharmonic organs, a Phonoliszt-Violina, a Decap dance organ, and the Welte philharmonic organ from the Britannic, sister ship of the Titanic and Olympic. http://www.bundesmuseen.ch/musikautomaten/index.html?lang=en

[Maureen Lang]

Glad you commented here, Craig V. When's that new CD coming out? Would like to post a cut from it on TA, if you're amenable.

Thanks, Jeff B., for the link. Most interesting. Given yr. travels recently, have you been there?