Monday, November 30, 2009

How what we see affects what we hear: Visual perception of music performance

Reference: Thompson, W. F., Graham, P., and Russo, F. A. (2005) “Seeing music performance: Visual influences on perception and experience”, Semiotica 156: 177-201.


This long and detailed article addresses a very important and interesting aspect of music performance-its visual dimensions. It discusses the extent to which these aspects contribute to the communication between performer and a listener. The authors maintain that there are several levels at which facial expressions and gestures might affect our perception of music. First, there is basic level at which listeners’ attention is directed to critical acoustic information at a certain moment of time. The authors argue that such cues can increase musical intelligibility. They draw an analogy referring to the face of a speaker, that when seen may increase the intelligibility of speech in a noisy environment. Then there is a perceptual level, at which visual cues are given to the listener to indicate important melodic, harmonic, and rhythmic events. The authors suggest that at this level, facial expressions and gestures are intentionally introduced by performers as a way of sharing with listeners their understanding of the significance of such events, as well as emphasizing the musical activity as a shared experience between performer and listener. Finally, the authors observe that visual information is “highly effective at conveying persona and attitude” (p. 204), arguing that facial expressions and other bodily gestures help performer to personalize the music, creating the feeling of reciprocal human interaction during the performance. Next, the authors discuss a variety of different media technologies, which can be used to experience music performance, and argue that each technological medium provides listener with different experience. They put forward a concept of music as “multimedially performed and multimodally experienced” (p. 205), and present empirical findings that appear to support this model. Two case studies are presented in order to discuss the relationship between visual and aural aspects of performance. The authors thoroughly examine the use of facial expression, body movement, and gesture in two filmed performances, one of B. B. King (playing Blues Boys Tune), and another by Judy Garland (singing Just in Time). Interestingly, the analysis of these samples is based on structured interviews by a trained musicologist of two other musicologists. The authors observe that the wide range of potential effects that visual information can have on our perception of music suggests the necessity of further and more systematic research in this area. Accordingly, they designed five experiments to assess visual influences on two types of musical judgments. In three experiments, they investigated visual influences on structural interpretations of music and in two experiments - visual influences on emotional interpretations of music. The experiments confirmed that visual characteristics of performance influence music experience at a perceptual and an emotional level. In conclusion, the authors observe that, during performance, listeners incorporate visual and aural aspects to form an integrated audio-visual mental representation of music.


In my opinion, visual dimension is one of the most interesting and significant characteristics of music performance. In this regard, the article is definitely thought provoking. The authors suggest that these aspects are greatly overlooked by most psychological research, which consider them as unessential to the music. This may be true concerning systematic scientific investigation; however, it is extremely rare that visual aspects of performance are disregarded by a common listener (especially by those with limited musical training). People are extremely attentive to these details; they watch musicians closely, and, at times, it appears that these aspects are, perhaps, of even greater importance for them than the music itself. When Evgeny Kissin toured in Israel, my mother-in-law (who is a professional pianist, performer and teacher with lots of experience) attended one of his performances with a friend of hers (who has no formal training in music). After the concert, my mother-in-law, agitated and extremely excited, conducted a thorough analysis of the performance of this, perhaps one of the greatest pianists of all time, focusing in particular on his interpretation, phrasing, dynamics and other related specific details. The friend listened attentively and then expressed her curiosity regarding the reason behind my mother-in-law decision to buy more expensive tickets in the second row. My mother-in-law replied that she wanted to see the pianist’s hands, after which the friend thoughtfully observed, “But what ugly faces and dreadful grimaces he makes all the time!” I think this observation is not uncommon. Moreover, it is very characteristic. The authors of the article argue that listeners form an integrated audio-visual image of music. If this were correct, what kind of mental representation of the music the aforementioned friend would have? Will the music that she just listened to be always associated for her with the “ugly” and contorted countenance of the performer?
Then, perhaps, this could be further subdivided into two related questions. Do the twisted features or body movements of performer bear any interpretational connotations? Do these movements constitute an intentional message of a performer to a listener? Does it convey or elucidate musical meaning? Or, it’s just an involuntary muscle movement, correlated with the level of difficulty of a certain passage, or associated with the intensity of performers’ emotional involvement? And, another question, how are these aspects perceived by listener. Do listeners, indeed, combine visual and audio aspects of performance to create an integrated image of music, or visual characteristics simply distract them from the music? It appears that the latter could be true.
Obviously, there is a great disparity between professional musician’s approach to a listening process and that of a person with no formal music training. While musicians would typically focus on the specific music qualities of performance, disregarding other attributes, non-musicians would be greatly amused and distracted by other characteristics, including visual aspects of performance. This is especially the case with teenagers and young kids. When I attempt to discuss with them the music performances they attend, it appears that visual aspects of performance make more profound and enduring impression on them, than any specific musical characteristics.
There is also another issue, which I think worth mentioning in this context. It seems that visual aspects of performance, including facial expressions, gestures, and body movements are culturally determined. For instance, I know that in Russia, such movements were not encouraged. It was thought that a musical message should be transmitted using specific musical means of expression, such as dynamics, phrasing, tempos, articulation, and so on, rather than by means of the excessive and disproportionate visual indications.
In short, this is an interesting subject to discuss, and I think it deserves to be further investigated.
Here is a couple of links to video recordings of the music performances. This is amazing how differently these artists express themselves musically and visually. Is it culture? Differences in professional training? Fashion?
David Oistrakh plays Tchaikovsky Violin Concerto
Jascha Heifetz plays the same Concerto
Itzhak Perlman plays Tchaikovsky Violin Concerto
Evgeny Kissin plays Tchaikovsky Piano Concerto
Van Cliburn plays the same Concerto


Myrtle D. Millares said...

Personally, I do find that I'm distracted from the music when I look at the movements of musicians. As a result, I frequently close my eyes at concerts unless I'm interested in looking at technique (as I was when I saw Evgeny Kissin play earlier this year). Thankfully I couldn't see his face, or I may have had a very different view of it all.

I do think that the visual aspect of performance provides me with a different perspective on the music. I will, at times, listen and watch, especially when a passage is particularly exciting, because at that point, the gestures seem to complement the music rather than provide a sometimes jarring contrast.

Interestingly, when I'm playing, I feel that I physically emote most when I'm quite nervous, hoping to focus better on the music by adding more motion to it.

Renee Kruisselbrink said...

As a pianist in performance, physical appearance/mannerisms/expression while performing should be of concern. All of my teachers have (for the most part) come from different "schools" of training including Russian, and French. Definitely, this informs not only how they view this aspect of performance and physical gesture for themselves, but how they approach this with their students.

For many teachers, physical movement is assimilated into a performance by the technical demands of a piece. In other words, it is not “added” artificially. I agree with this philosophy. Physical gesture becomes inherent when performance to a certain degree; it should not be “planned” since music is quite capable of speaking for itself without extraneous movement.

Eventually though, every pianist/performer evolves into who they are and how they wish to be perceived in this respect. This varies, based on the image they wish to project, and the musical culture they are in. For example, the pianist Lang Lang is an interesting case of a Classically-trained musician playing Classical works whose physical movements in performance are similar to those of a rock star – in other words, almost “over the top” for a Classical pianist. Many pianists (including myself) find his physical gestures while playing to be absolutely distracting and detrimental to the music he is performing.

Liane James said...

An interesting discussion! As a harpist, I find that physical gestures are a very important aspect of playing. My training has mainly been with teachers of the Salzedo school of harp technique. Along with many other things, Salzedo emphasized the importance of expressive gestures, particularly making use of the correct gestures after playing a placed group of notes. In simple terms, once the fingers leave the strings, the hands and arms of the harpist must rise in a slow, relaxed and aesthetically pleasing manner, without altering the angle of the hand. As Salzedo writes, “It has been proven that this ascending gesture is the only esthetic [sic], practical and effective method of controlling the quality of the sound produced… The fundamental harpistic gesture will gradually bring about a feeling of unity between the player and his instrument, and will create, almost unconsciously, the necessary state of coordination.” (Carlos Salzedo, and Lucile Lawrence, Method for the Harp (New York: G. Schirmer, 1927), 17.)

I would be surprised if audiences were ever distracted from listening to a harpist perform by physical movements. Grimaces and odd facial expressions however…