Profile
Photos
Comments
The Multimodal Brain Orchestra performed its world premiere on Thursday.
Led by an "emotional conductor" and a traditional one, music and
video change in time with the performers' brain waves and heart rate.
According to the work's producer, the orchestra aims to "see what the brain can do without the body".
The orchestra's premiere performance closed the Science Beyond Fiction conference in Prague.
The project is the creation of the Synthetic,
Perceptive, Emotive and Cognitive Systems (SPECS) group at the
University Pompeu Fabra in Barcelona.
"Only recently we have come to appreciate more the tight coupling
between mind, brain and body," Paul Verschure, head of the project,
told the audience.
"But we can wonder what the mind and brain would be capable of if it
would be directly interfaced to the world, bypassing the body."
The piece that the orchestra performed, Xmotion, was composed by Jonatas Manzolli, giving it an underlying structure.
But it is the performers who control variations of visuals, sounds, frequencies, and volumes in the overall piece.
Model result
 |  What we want to show here is the use of your brain without your body Anna Mura SPECS group |
"Everything is built to fulfil the circumplex model, which was
worked out by psychologists that study emotions," says Anna Mura, a
biologist who is the producer of the project. "How we feel and what we
feel fits into a circle cut into four quadrants, labelled by 'arousal'
and 'valence', that is, how much you're excited and how badly or
positively you're excited."
To that end, four performers were fitted with caps littered with
electrodes that take a real-time electroencephalograph - an image of
the brain's electrical activity.
"There is a first violin, a second violin and so on, except that instead of violins they are brains," says Dr Mura.
The graphs of those brain waves are projected onto one of two large
screens above the orchestra. The performers launch sounds or affect
their frequencies and modulations based on two well-characterised
effects seen in EEGs: the steady-state visually evoked potential
(SSVEP), and the so-called P300 signal.
Two of the performers were given a task to watch a screen in front
of them, with flashing rows and columns of letters, and told to look
for a particular letter.
 The brain monitors capture mental activity in response to stimuli |
When expectation is fulfilled, 300 thousandths of a second later, a signal known as the P300 appears in the EEG.
A similar strategy has been employed by Mick Grierson at Goldsmiths, University of London to generate individual notes.
In the Multimodal Brain Orchestra, the P300 signal is registered -
with a dot demarcating it on the EEG trace projected to the audience,
so that they can see the effect of the performer's thought - in turn
launching a sound or recorded instrument.
Two more performers were given boxes with four lights flashing at
different frequencies. The SSVEP is a brain signal that comes about
when visual stimulus in the retina at a given frequency causes the
brain to synchronize, so that frequency appears in the EEG.
Given a cue from the conductor, the performers switch their attention from one flashing frequency to another.
One of them affects the volume of a given sound - known to influence
the level of arousal in the circumplex model - and the other affects a
certain modulation of that sound, which is known to influence the
valence, how positive or negative the arousal is emotionally.
'Emotional experience'
Adjacent to the EEG-capped players, the "emotional conductor" sits comfortably, wearing a pair of virtual reality glasses.
She is being shown images from a series created by artist Behdad
Rezazadeh while her heart rate and skin conductance are being measured.
Her heart rate is plotted along with the EEG traces.
As her mood changes, so does the visual experience - Rezazadeh's
images are blurred and changed in line with the changing biological
measures of the conductor.
"This performance is like her emotional experience," said Dr Mura.
"What we want to show here is the use of your brain without your body.
Embodiment - we should get rid of it sometimes."
While this artistic interpretation of the biometric technology is a
fascinating look into the merger of technology with medicine, the
overall goals of the group have far greater implications.
"People believe that to understand how we feel will help us to
understand what consciousness is all about. This is the technology that
is going in that direction; we cannot explain consciousness with this
but we are at least exploring the surface of it."
https://news.bbc.co.uk/2/hi/science/nature/8016869.stm