Sound-Colour Synesthesia and the Nature of its Cross-Modal Mechanisms

Reference

Ward, J., Huckstep, B., & Tsakanikos, E. (2006). Sound-colour synaesthesia: To what extent does it use cross-modal mechanisms common to us all? Cortex, 42(2), 264-280.

(Available through UTL catalogue)

Summary

In this study conducted at University College London (London, UK), the nature of cross-modal perception in synesthetes was investigated.  The term “synesthesia” is used to describe a neurological condition in which one type of sensory stimulation evokes the automatic and involuntary sensation of another.  One of the most common types of audio-triggered synesthesia is chromesthesia, or “coloured hearing”—when the hearing of sounds produces the automatic and involuntary visualization of colours and patterns.  A group of synesthetes who reported colour sensations in response to music were examined alongside a control group to find out whether this type of synesthesia employs similar mechanisms used in normal cross-modal perception (common to all people), or whether there are direct, privileged pathways between unimodal auditory and unimodal visual areas that are unique to synesthetes.

Ward, Huckstep, and Tsakanikos suggest that studies in synaesthesia can be used to inform theories of normal cognition.  Chromesthesia can be especially useful in this regard because there is evidence that suggests that not only do cross-modal audiovisual mechanisms exist in the normal population in a more general sense, but “coloured hearing” might be present in all of us from birth, but disappears over time. 

Some studies assume that all humans are born with neural mechanisms capable of synesthesia, but somehow lose them during normal development.  People that do retain this neural hardware retain synesthesia in adulthood.  This view suggests that there are special neural pathways in synesthetes that are absent in other adults, i.e. synaesthesia uses cross-modal mechanisms that are not common to us all.  Ward, Huckstep, and Tsakanikos are questioning this opinion in their experiment.  The second hypothesis, and the one this particular study is based on, is that sound-colour synaesthesia possibly arises from utilising pathways that are used to integrate audio and visual stimuli as part of the normal mechanisms of cross-modal perception.  There exist cross-modal audiovisual areas in the brain that are more responsive to the combined stimuli of sound and visuals than to either stimulus separately (e.g., during lip reading).  Because these pathways are common to us all, it is possible that synesthesia is just an advanced mutation of these pathways.

Experiment 1 first examined whether these sound-colour associations were arbitrary, i.e. were the colours that synesthetes saw in relation to a specific tone same/similar to other synesthetes, or were they drastically different?  The authors first inform us that synesthetes have not been compared to non-synesthetes in this regard before this study.  However, there are some universals amongst non-synesthetes in terms of the sound-colour relationship, for example: (1) most people can generate visual imagery to music on demand, and (2) most people tend to associate higher pitch sounds with lighter colours.  The results showed that sound-colour synesthetes had a greater internal consistency of matching colours to various pitches and timbres than the control group.  However, both groups generally used the same heuristics for matching between the audio and visual stimuli (e.g., pitch to lightness, timbre to colour).  These results confirm the hypothesis that sound-colour synesthesia is an extension of cross-modal mechanisms common to us all, rather than a privileged pathway between auditory and visual modalities not present in non-synesthetes.

Experiment 2 aimed to establish that synesthetes had automatic experiences of colour when presented with a tone.  In a procedure similar to a Stroop Test, the synesthete and control groups were asked to say the colour of a patch on the screen while simultaneously listening to various tones which they were told to ignore.  A colour either congruent or incongruent with the colour on the screen would be automatically generated within the synesthetes.  The results showed that colours in synesthetes are automatically elicited to such an extent that they are produced during a cross-modal Stroop task.

In Experiment 3, the nature of the sound and colour were not the focus.  Rather, the synesthetes and non-synesthetes participated in a cross-modal variation of the Posner cueing paradigm.  An auditory cue is (non-laterally) presented through headphones while simultaneously, two coloured rectangles appear on the left and right of the screen, one of which is synesthetically corresponding with the sound.  The task involves detecting the target, an asterisk, present in one of the two rectangles right after the auditory cue.  The results showed that the auditory cue oriented attention to the synesthetically analogous location of the asterisk.  Detection of the lateralised target was enhanced when combined with a synesthetically congruent sound-colour pairing in both the synesthetic and control groups.

The results of these experiments suggested that sound-colour synesthesia does indeed use similar (if not the same) mechanisms used in normal cross-modal perception common to us all, and not special, direct, or privileged pathways between unimodal auditory and visual pathways that are only found in synesthetes.

Reflection

I was very excited to read this article because of its relation to the discourse on synesthetic art.  In addition to being a widely recognized condition in neuroscience, the term “synesthesia” is used in fine art to describe the simultaneous perception of multiple stimuli incorporated into one gestalt experience. This can include, for example, multi-sensory projects in the genres of visual music and sound visualization, audio-visual art, and intermedia. The idea of “synesthetic art” can refer to either (1) art created by synesthetes or (2) art that attempts to transmit or simulate the synesthetic experience. It is an attempt to grasp the cognitive results of the subjective perceptual experiences of natural synesthetes, and translate them into the realities of non-synesthetes.

One prominent example of synesthetic art is visual music.  The techniques used within this genre to visualize the sounds in music attempt to turn all listener-spectators exposed to them into audio-visual synesthetes, manually.  This simulated or induced synesthetic experience is often part of electronic dance music (EDM) performances—probably due to their otherwise visually unexciting nature.  EDM artists hire extensive teams of set designers, artistic directors, graphic designers, and computer programmers, and put ours of thought and consideration into creating shows that have less to do with purely auditory experiences and become intermedia spectacles.

The technique of sound visualization is widely used.  Sound or music visualization can be described as moving nonrepresentational imagery based on, or derived from, the organization of sound within music. Abstract qualities found in music—including rhythm, tempo, mood, counterpoint, intensity, harmony, and compositional structure—are assimilated within visual phenomena. The moving images can be generated in real time by a software program, or manually conceptualized through computer graphics programs based on sine waves. The imagery is synchronized with the audio as it is played in real time.

What makes this practise more intriguing is that according to Ward, Huckstep, and Tsakanikos’s findings, the visual qualia that the creators of any particular visualised music attribute to the source sounds would be very similar to the visual qualia that any spectator-listener would attribute to the same sounds.  The colour-to-timbre or brightness-to-pitch associations are not all completely arbitrary amongst non-synesthetes (although they are a lot less consistent than in true synesthetes, of course), and prove to be significantly similar.  This suggests a certain level of intersubjectivity amongst the audio-visual perceptive experience.  This hypothesis has value in that the simulated audio-visual synesthetic experience seen as intense or meaningful by its creators would be received as equally intense or meaningful by its perceivers, and would simultaneously cultivate a more involved and significant relationship with that artist’s music.