MUMT-307 Report (Fall 2017)

Using FFT analysis and resynthesis to create infinite sustained sounds and spectral reverb.

Introduction

One of my research interests is the modification of the sound characteristics of the classical guitar. This instrument features rich timbre but its notes have short sustain.

The use of Fast Fourier Transform (FFT) to analyze the sound of the instrument can allow the synthesis of sounds that have a similar pitch to that of the classic guitar, but which can be synthesized indefinitely.

A quick summary of objectives

In summary, the objectives of this final project are:

  • The analysis of classical guitar sound in real time;
  • Manipulation or processing performed in the frequency domain in order to generate either a sound of infinite sustain and a spectral delay;
  • Sound resynthesis using the inverse FFT process (IFFT).

The project design

Although my research points to the possibility of achieving the goals of this project using either granular synthesis or delay lines (in a complex and possibly unstable configuration), the frequency domain sound manipulation allows me to further expand the functionality of the programmed patch, including pitch Control and time stretching.

Main patch window.

For this project I opted for programming in Pure Data because this language allows me to execute the programmed patch on computers running operating systems based on Linux. This feature will allow me to use the patch on embedded systems based on Raspberry Pi, for example.

The patch contains 2 operation modes: a continuous analysis and resynthesis mode (spectral delay) and a manual triggering mode (freeze). The spectral delay mode will analyze blocks in a given time interval that can be set by the user and the output will respond according to the real time input intensity after N samples (spectral delay). The freeze mode will synthesize a continuous sound from the blocks acquired immediately after the user activates the function, i.e. push the freeze button.

The programming includes several concepts learned during the course MUMT-307 (Music \& Audio Computing II): Signal Analysis using STFT and resynthesis using IFFT, digital delay lines, feedback (in filters and frequency domain processing), windowing, FFT “hop” and “Phase”, overlapping, convolution, etc.

Particular issues that proved challenging / rewarding

Programming in Pure Data presents some extra challenges in relation to Max/MSP: The lack of objects in Pure Data makes it necessary to use multiple objects to perform functions that are easily solved with only one object in Max/MSP. An interesting example is the counter object. The Vanilla version of the PD does not have any counter object, being necessary to the use of the metro object and the creation of accumulators using mathematical operators with storage objects.

Subpatch acting as a mixing desk (depth control).

This problem can be solved by using the PD-Extended version or by using other distributions such as Pd-L2Ork (Purr Data). The use of PD-Extended is not the ideal solution since it is abandoned. Pd-L2Ork is the best option to use external and libraries that leave PD functionality similar to Max/MSP.

One last important comment about choosing which version of PD to use is the issue of compatibility: PD-Vanilla is still a good software choice if the intention is to make the programmed patch available in all operating systems and devices, including mobile phones.

All the references provided great information to build the patch. Some examples of different approaches for spectral freeze and spectral delay were found among Pure Data and Max/MSP community. Some of these patches provided good ideas and were implemented in the resulting patch, including visual feedback and “speed control” (Ralf’s Freeze3.pd) and practical tips for implementing procedures (Jean-francois Charles paper and patches in Max/MSP).

Unfortunately most implementations found in both the PD and Max/MSP communities have conceptual errors or present very simple implementations that do not provide sound quality or different configuration possibilities.

Even using the different procedures studied during the discipline, the resulting sound quality can vary drastically according to the sound source used and the signal-to-noise ratio. The patch proved itself moderately robust for sounds with concentrated energy in the medium-high regions of the spectra, such as trumpet’s high register or a whistle.

A mixing object has been programmed to control the relationship between the original sound and its synthesized version (depth control). This patch can be checked in the above figure (mixer).

For the low region of the spectrum (more present in classical guitar spectra) the noise level in the synthesized sound is considerably higher and greater attenuation of the more energetic frequency peaks is required. This attenuation is provided in the form of a noise filter control, which provides destructive interference to the signal to eliminate excess noise.

Example usage / demo

Both the Pure Data patch and a PDF version of this document can be found in the above links.

References

Appendix: FFT subpatch

 

Participation in the 13th AES Brazil Conference

I had the great pleasure of having a conference full paper approved at the 13th AES-Brasil Audio Engineering Conference occurred from May 26 to 28, 2015. This congress was part of the 19th AES-Brasil National Convention in the Yellow Pavilion of the Expo Center Norte (São Paulo / Brazil).

The paper title is Projeto e Construção de uma Interface Gestual para Improvisação Livre  (Project and Construction of a Free Improvisation Gestural Interface) and has as co-authors Prof. Marcelo Wanderley and Prof. José (Tuti) Fornari.

Here is the paper translated abstract:

The use of gestural interfaces in the performance of mixed electroacoustic music is an important developmental front of contemporary music. The interaction between algorithms and performers has helped a significant number of artists to explore and transcend some boundaries of musical performance; As well as promoting distributed interaction and remote cooperation of interpreters in social networks of multimodal performances. This work presents a study with free improvisation sessions were performed using gestural interfaces controlling algorithms created in open-source software. The construction of the device, as well as the practical implications of using a specific enhanced instrument (AI) in free improvisation sessions are presented, described and discussed herein.

This paper was presented by me during the congress Paper Session # 02 and both the conference proceedings and the conference paper can be accessed using the provided links.

 

Guri’s Workshop

Music, technology and creativity

Looking at several schools of the Programa Guri we can find students with excellent performance in musical instruments or singing courses but who have little or no experience in creative improvisation activities, as well as little (or none) contact with contemporary music. With these students in mind, I propose (along with Guri’s pedagogical team) one (1) workshop to practice free improvisation using technological support, where the students could experience collective creative activities in addition to making a first contact with the possibilities of sonic synthesis and manipulation using computers.

October 22, 2014 the workshop entitled Música, Tecnologia e Criatividade (Music, Technology and Creativity) occurred for Programa Guri students in Polo Penha (São Paulo/SP). This activity was conducted by myself with the aid of professor Bruno Hernandes and lasted 2 hours. The workshop included a small presentation about creative processes and organization in improvisation, along with a brief history about music and technology in XX and XXI centuries.

In this meeting the students improvised in their own musical instruments while the sounds was modified in real time using the software Pure Data.

There is much discussions today about creative development and the role of music in human development, and for the great majority of music education thinkers (including Swanwick, Dalcroze, Gainza, Willems and Kodály, among others), improvisation is a fundamental part of teaching music. In a collective teaching environment, improvisation activities are usually taken as secondary, witch can cause delay in the learning process or even preventing student to create a relationship of intimacy with the instrument.

Several students reported that before the workshop they viewed improvisation as something unattainable, very difficult or intended only for great masters. All educators cited above defend that improvisation should be stimulated early in music students and, in addition to creative growth, enhances the student’s intimacy with the instrument (and his own body), stimulates critical listening, encourages pro-activity and, in the case of collective improvisations, encourages teamwork.

Digital inclusion is also a trending topic in Brazil and the goal of several governmental programs. Using the computer and other technologies in musical creation makes sense to help this inclusion, adding a artistic result to the exploration of these devices. With the use of the computer in musical performance we expand the creative possibilities, generating sound material that could not be obtained otherwise.

This workshop feedback was extremely positive as we can observe in the evaluation of the activity, carried out as a questionnaire answered by the students and compiled in this document.

My participation in the 1st French-Brazilian Colloquium on Computer Aided Music Creation and Analysis

Between August 18 and 28, 2014 I participated of the 1st French-Brazilian Colloquium on Computer Aided Music Creation and Analysis, organized by NICS (Sound Communication Interdisciplinary Center, where I am a registered student) in partnership with Faculty of Electrical Engineering (FEEC-Unicamp), Arts Institute (IA-Unicamp), the Center of Integration, Documentation and Cultural Diffusion (CIDDIC), USP e UNESP. The Colloquium was also supported by FAPESP.

That was a special event for two reasons: the online real-time event transmission and the participation of three RepMus researchers.

The Équipe de Représentations Musicales (RepMus) is a research laboratory maintained by IRCAM whose theme is the representation of musical structures, languages and computational paradigms applied to music. During the Colloquium we had the presence of Gérard Assayag, Moreno Andreatta e Jean Bresson.

Mesa redonda (round table) – 20/08/2014

In addition to the mentioned researchers we also had several important brazilian researchers in music technology area: Jônatas Manzolli, Stéphan Schaub, Silvio Ferraz, Fernando Iazzetta, Flo Menezes and Tuti Fornari.

Among the covered topics we discussed about advanced processes applied to musical analysis and creation, real-time musical interaction and formalization of musical structures.

Perhaps the most interesting think about this Colloquium was the emphasis on the practical aspects of music technology. Unlike various congresses and symposiums in the music area, this Colloquium assigned the two final days to the elaboration of practical activities that culminated in a concert at Almanaque Café on August 28 (2014) entitled “Livre Digital: Fronteiras Musicais Tecnologias que desafi(n)am os sentidos”.

During this stage some Brazilian researchers had the opportunity to present their projects in progress to the RepMus guests. This was, in my opinion, the highlight of the event since the exchange of information when I was presenting my project contributed in an inestimable way to my growth as a researcher, along with providing important information that contributes with the progress of the research. At this stage there were also practical activities of hardware and software experimentation previously created by the researchers, which intensified the conviviality of the participants.

There is a great criticism to the academic practice of music because of the conceptual and theoretical focus and for the consequent distance to the musical practice. This type of initiative approximates theory (or theorization) and practice, in addition to bring something tangible to the event participants.

Tocando com os pesquisadores – Improvisação coletiva
Tocando com os pesquisadores – Improvisação coletiva
(@Davi Moraes)
Tocando com os pesquisadores – Improvisação coletiva

 

I’m looking forward to attend more events like this!

Choice of components to build a HMI for musical performance

In this post I will talk about the choice of the components for the construction of my first human machine interface (HMI) prototype for my master’s.

HMIs are electronic equipment responsible for collecting, sending and receiving data and commands between the user and the computer. They may have specific application, such as the HMI that I built, which has the function of transmitting (and sending) information from the musician to the computer during a musical performance.

For the processing and communication between sensors and the computer I used the Arduino. Arduino is an open-source hardware prototyping platform, that is, a set of interchangeable equipment and components that can be used to assemble electronic devices for any needs, including control of real-time sound manipulation. A great difficulty faced in any electronic devices creation is the microcontroller programming for the accomplishment of certain tasks. The Arduino platform has a easy programable microcontroller as core element and this programming can be made directly through USB connection present in the own board (ARDUINO, homepage. Getting Started with Arduino).

Because it is an open-source hardware platform we can find Arduino-compatible boards and components. In addition, Arduino’s programming environment is based on Wiring and C ++, widely distributed and well-documented programming languages.

One of my project’s premises is to keep the HMI as simple and cheap as possible. The Arduino is considered a low cost microcontroller. However, we still have more economic options: the Arduino clones. I chose the Freaduino Leonardo R3, an Arduino Leonardo R3 clone that offers me the same characteristics of the original.

Freaduino Leonardo R3

The Leonardo uses the ATmega32u4 processor and has 20 digital in/out, microUSB conector, reset button and mouse/keyboard emulation.

As the initial proposal is to control any audio processes during performance using the guitar and HMI, I chose to use an80 characters LCD (JHD162a) for visual feedback, as well as resistors and cables to connect the sensors.

JHD162a

The chosen sensors and components for interaction with the musician and the instrument were:

  • 1 ultrasonic sensor (HC-SR04)
  • 1 three axis accelerometer (ADXL345)
  • 5 SPST footswitches

The ultrasonic sensor will be placed in the instrument to capture free gestures performed with the right hand of the guitarist. These sensors measure distance and can be programmed to control the intensity or presence / absence of certain sound process.

HC-SR04

The chosen accelerometer allows several functions. It will be arranged outside the instrument and allows to read the guitar movement in 3 axes (X, Y and Z), also having TAP function (it detects a “tap” or a light touch on the sensor or the guitar), DOUBLE TAP (similar to TAP, but works like the “2 clicks” of a mouse) and FREEFALL (as the name suggests, detects “free fall” of the object).

 

ADXL345

The accelerometer allows reading all its functions independently, providing complex gestural mappings and enabling the manipulation of several sound events simultaneously. Is interesting to capture the performer’s ancillary gestures.

The result of this first prototype can be seen in the pictures below:

In the next posts I will discuss how to enable communication between the HMI and Pure Data.