ARTICLE
The multisensory perception of flavour
Charles Spence on his mouth-watering research
Eating and drinking are among lifes most pleasurable activities and among the most multisensory as well. However, cognitive neuroscientists have only recently come to realise that their insights, derived from studies of the multisensory integration of auditory, visual and tactile stimuli, can be extended to help explain flavour perception. This approach is already starting to impact upon the design of foods, drinks and dining experiences in locations as diverse as the supermarket and Michelinstarred restaurants. Psychology and cognitive neuroscience can help create novel flavours, taste sensations and dining experiences that can more effectively stimulate the mind, and not just the mouth, of the consumer.
No animal can live without food. Let us then pursue the corollary of this: namely, food is about the most important influence in determining the organization of the brain and the behavior that the brain organization dictates. J.Z. Young (1968, p.21). Cooking is probably the most multisensual art. I try to stimulate all the senses. (Ferran Adri, El Bulli restaurant, Spain)
questions
Do the same mechanisms underlie the multisensory integration of flavour cues as modulate other combinations of sensory stimuli? Do smelltaste confusions reflect a kind of synaesthesia common to us all?
Calvert, G.A., Spence, C. & Stein, B.E. (Eds.) (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press. Stevenson, R.J. (2009). The psychology of flavour. Oxford: Oxford University Press. http://bit.ly/4JEfEr
ognitive psychologists and neuroscientists interested in the topic of multisensory integration have, until recently, largely kept away from the study of flavour perception, preferring instead to focus on the integration of auditory, visual and, to a lesser extent, tactile signals. However, the last few years have seen a rapid growth of interest by researchers trying to apply the insights uncovered in the psychophysics and neuroimaging laboratory to the study of multisensory flavour perception (Stevenson, 2009; Verhagen & Engelen, 2006). As the above quote from J.Z. Young makes clear, food acquisition has likely played a critical role in shaping brain development throughout the course of human evolution. Indeed, food is one of the most effective stimuli in terms of modulating brain activity in hungry participants; with the sight and smell of appetising food leading to a 24 per cent increase in whole-brain metabolism in one PET study (see Wang et al., 2004). Our enjoyment of food and drink comes
not only from the unified oral sensation of taste and smell (both orthonasal and retronasal), but also from the sound it makes, not to mention what it looks like. The oral-somatosensory qualities of foods are also very important; the texture, temperature, and even the pain, as in the case of eating chilli peppers, all contribute to the overall multisensory flavour experience (or gestalt: Verhagen & Engelen, 2006). The pleasure of food is critically dependent on all these sensory attributes being right, and so food can, for instance, be ruined simply by serving it at the wrong temperature, or if it has an inappropriate colour. For the last 30 years or so, psychologists and cognitive neuroscientists have been investigating how our brains combine what they see, hear and feel in order to generate the rich and varied multisensory experiences that fill our everyday lives (see Calvert et al., 2004, for a review). Researchers have, for example, spent a lot of time trying to work out why it is that people perceive the voice of a ventriloquist as coming from the moving lips of the dummy, and why people hear
resources
The tight coupling between taste and smell means that terms such as a sweet smell reflect much more than a merely metaphorical use of language
references
Alais, D. & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257262. Auvray, M. & Spence, C. (2008). The multisensory perception of flavor. Consciousness and Cognition, 17, 10161031. Bell, R., Meiselam, H.L., Pierson, B.J. & Reeve, W.G. (1994). Effects of adding an Italian theme to a restaurant on
the perceived ethnicity, acceptability, and selection of foods. Appetite, 22, 1124. Blumenthal, H. (2008). The big Fat Duck cookbook. London: Bloomsbury. Calvert, G.A., Spence, C. & Stein, B.E. (Eds.) (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press. Dalton, P., Doolittle, N., Nagata, H. & Breslin, P.A.S. (2000). The merging of
the senses. Nature Neuroscience, 3, 431432. de Araujo, I.E.T., Rolls, E.T., Kringelbach, M.L. et al. (2003). Tasteolfactory convergence, and the representation of the pleasantness of flavour, in the human brain. European Journal of Neuroscience, 18, 20592068. Ernst, M.O. & Blthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences,
8, 162169. Gal, D., Wheeler, S.C. & Shiv, B. (2007). Cross-modal influences on gustatory perception. Manuscript submitted. Gallace, A., Boschin, E. & Spence, C. (submitted). On the taste of Bouba and Kiki: An exploration of wordfood associations in neurologically normal participants. Cognitive Neuroscience. Green, D.M. & Butts, J.S. (1945). Factors
720
vol 23 no 9
september 2010
multisensory flavour perception
better when they put their glasses on. By studying such cross-modal illusions in the laboratory, scientists have started to uncover some of the fundamental rules governing multisensory perception. For example, they have discovered that peoples perception is typically dominated by what they see (as in the case of the ventriloquists dummy: Alais & Burr, 2004). There is now a growing body of empirical evidence to show that Bayesian decision theory, incorporating the principle of maximum likelihood estimation, does a remarkably good job of predicting our perceptual experiences under conditions of Bacon-and-egg ice-cream: the bacon is intersensory conflict (see Ernst & ventriloquised towards the crispy bread Blthoff, 2004, for a review). Meanwhile, neurophysiologists have corner shop (e.g. Gallace et al., submitted; shown that individually weakly effective Levitan et al., 2008; Shankar et al., 2009; (i.e. near-threshold) stimuli sometimes Yeomans et al., 2008). combine in a superadditive manner, giving rise to multisensory experiences that are more intense, and richer, than would be Taking the confusion out of fusion predicted by the simple linear combination Flavour perception is, however, not only of their individual parts (Stein & Meredith, of interest to psychologists and cognitive 1993; Stein & Stanford, 2008). Similar neuroscientists but also to an increasing superadditive brain responses have now number of philosophers (e.g. Smith, been documented in the human 2007), given the conceptual uncertainty orbitofrontal cortex (the part of the surrounding how exactly we should brain that controls our perception of the define flavour (Spence et al., 2010). pleasantness and reward value of food) Should flavour be considered as a in response to congruent combinations separate sensory modality, like vision and of olfactory, gustatory and visual stimuli hearing? Or is it instead more appropriate (de Araujo et al., 2003). to consider flavour as a kind of perceptual The latest evidence now shows that system (Auvray & Spence, 2008; many of the same rules of multisensory integration/perception (such as sensory Stevenson, 2009)? One thing is certain, dominance and superadditivity) also help though, we all find it extremely difficult to explain why food and drink taste the to distinguish between tastes and smells. way they do, and why what tastes nice to Think only of how hard it is to taste food one person may taste awful to another. when your nose is blocked; under such While much of the research in this area conditions, it is, of course, only your requires the use of computer-controlled sense of smell that is actually impaired. olfactometers and gustometers, it is Some researchers have argued that the important to note that fundamental studies ubiquitous nature of such tastesmell on flavour perception can be carried out confusions may reflect a form of with nothing more complex than a tub of synaesthesia that is common to us all ice-cream or a packet of sweets from the (Auvray & Spence, 2008; Rozin, 1982;
Stevenson & Tomiczek, 2007). What is clear it that the extremely tight coupling between the senses of taste and smell means that terms such as a sweet smell reflect much more than a merely metaphorical use of language (Stevenson & Boakes, 2004). For example, placing a drop of sucrose on the tongue (be it at a subor supra-threshold level) leads to a dramatic increase in the ability to detect the presence of a congruent odorant such as benzaldehyde (which has an almond-like odour familiar to those who like Bakewell tart; Dalton et al., 2000). Interestingly, however, this particular multisensory effect appears to be culture-/experiencespecific: While Europeans and North Americans, for whom the combination of almond and sugar is very common (in marzipan, for example), are able to integrate this particular combination of stimuli, they show no such multisensory enhancement effect when an almond odour is paired with a salty taste (monosodium glutamate). By contrast, the Japanese, for whom almond odor is associated with salty tastes in pickled condiments, typically show the reverse effect. That is, they exhibit enhanced responding to the combination of almond and salt, but not to almond and sugar. Results such as these suggest that our brains learn to combine just those tastes and smells (i.e. flavours) that commonly co-occur in the foods we eat (Blumenthal, 2008; Spence, in press). In fact, we apparently start learning our responses to flavours before we have even left the womb (Schaal et al., 2000).
The ventriloquist in your mouth
While many people might think that the ventriloquism illusion only occurs in the cinema or when watching a ventriloquists dummy, the latest evidence suggests that a similar effect also occurs in the mouth. Ventriloquism may help explain why it is that we always seem to perceive flavours in the mouth (rather than in the nose,
affecting acceptability of meals served in the air. Journal of the American Dietetic Association, 21, 415419. Levitan, C., Zampini, M., Li, R. & Spence, C. (2008). Assessing the role of color cues and peoples beliefs about color-flavor associations on the discrimination of the flavor of sugarcoated chocolates. Chemical Senses, 33, 415423.
Morrot, G., Brochet, F. & Dubourdieu, D. (2001). The color of odors. Brain and Language, 79, 309320. North, A. & Hargreaves, D. (2008). The social and applied psychology of music. Oxford: Oxford University Press. Oberfeld, D., Hecht, H., Allendorf, U. & Wickelmaier, F. (2009). Ambient lighting modifies the flavor of wine. Journal of Sensory Studies, 24, 797832.
Parr, W.V., White, K.G., & Heatherbell, D. (2003). The nose knows: Influence of colour on perception of wine aroma. Journal of Wine Research, 14, 79101. Rozin, P. (1982). Tastesmell confusions and the duality of the olfactory sense. Perception & Psychophysics, 31, 397401. Schaal, B., Marlier, L. & Soussignan, R. (2000). Human foetuses learn odours from their pregnant mothers diet.
Chemical Senses, 25, 729737. Shankar, M.U., Levitan, C.A., Prescott, J., & Spence, C. (2009). The influence of color and label information on flavor perception. Chemosensory Perception, 2, 5358. Shankar, M.U., Levitan, C. & Spence, C. (2010). Grape expectations: The role of cognitive influences in colorflavor interactions Consciousness & Cognition, 19, 380390.
read discuss contribute at www.thepsychologist.org.uk
721
multisensory flavour perception
where the majority of flavour perception originates): olfactory and gustatory stimuli, which are difficult to localise, are perceived as emanating from the mouth because they are referred to the highly localisable tactile sensations associated with mastication. The ventriloquism illusion may also play a crucial role in one of the classic dishes on the tasting menu at Heston Blumenthals restaurant in Bray, The Fat Duck. When bacon-andegg ice-cream was first created, it was only moderately pleasant; the flavours did not appear to stand out from one another. The breakthrough came when a piece of crispy fried bread was added to the plate. While the bread does not, in-and-of-itself, impart much flavour to the dish, its addition brought the dish alive, seemingly helping to separate the bacon and egg flavours. It appears as though the bacon is ventriloquised towards, and hence becomes perceptually localised within, the crispy bread, while the eggy flavour stays behind in the more texturally appropriate soft ice-cream.
Sound bites
Try eating a crisp (or potato chip) without making a noise. Its simply impossible! Do such food-related sounds exert any influence on our perception? Experiments conducted here at the Crossmodal Research Laboratory in Oxford have demonstrated that foodeating sounds contribute to the perception of crispness and freshness in foods such as crisps, biscuits, breakfast cereals and vegetables (Zampini & Spence, 2004). The participants in our study (for which we were awarded the 2008 IG Nobel prize for nutrition) had to bite into 180 potato chips and rate each one on its perceived crispness and freshness. Pringles were ideal for this research because they are all more-or-less identical in size, shape and consistency. The crisp-biting sounds were picked up by microphone, modified and then immediately played back over headphones. The crisps were rated as
tasting significantly crisper (and fresher) if the overall sound level was increased, or if just the highfrequency Visual cues can have a profound effect on both the sensorycomponents of the discriminative and hedonic aspects of multisensory flavour crisp-biting sound perception were boosted. Peoples perception of the carbonation of a beverage served in unable to modulate peoples perception a cup is also modulated by what they hear of saltiness using food colouring. The (Spence & Shankar, 2010). Companies problem here is that salty foods come in such as Unilever, Proctor & Gamble and all manner of colours, and hence there is Nestl have now started to use this no natural colourtaste association on psychologically inspired technique inwhich to build. house when developing their new dryWine is particularly interesting from food products. the point of view of flavour research because it allows one to look at the effects of expertise. In what is perhaps the most famous study in this area, 54 students Grape expectations enrolled on a university oenology course Many studies published over the last in Bordeaux were given two glasses of 75 years have shown that visual cues wine to sniff, one white, the other red (especially those concerned with a food (Morrot et al., 2001); They had to indicate or drinks colour) can have a profound which of the wines presented a range of effect on both the sensory-discriminative characteristics most intensely. Descriptors and hedonic aspects of multisensory such as lemon, honey and straw were flavour perception (Spence et al., 2010). chosen for the white wine aroma, and So, for example, people find it particularly another set of terms, such as prune, difficult to correctly identify fruitchocolate, tobacco, etc. to describe the red flavoured drinks that have been coloured wine. A week later, the students were given incongruently, as when an orangetwo further glasses of wine (one white, the flavoured drink is artificially coloured other red, just as before). This time, green. Similarly, many people will reportedly run to the bathroom to be ill if however, what looked like red wine was they find out that the steak that they have actually a white wine that had been been eating is actually coloured blue. artificially coloured red. The budding More seriously, multisensory research on oenologists who hadnt been made aware flavour may turn out to have important of the deception all chose the red wine health implications, given that, for odor descriptors when evaluating the example, beverages can be made to taste inappropriately coloured white wine. as much as 11 per cent sweeter simply When it comes to wine, people appear to by adding the appropriate amount of red smell what they see! food colouring. Colours associated with Professional wine tasters and wine the ripening of fruits are particularly makers also fall prey to this visual effective in modulating perceived dominance effect (see Parr et al., 2003). sweetness, while foods and drinks with If anything, they appear to be even more a green hue (i.e. associated with unripe susceptible to such cross-modal effects fruits) are often judged to be sourer. As than social drinkers. This is presumably yet, however, researchers have been because wine experts expectations
Smith, B.C. (Ed.) (2007). Questions of taste: The philosophy of wine. Oxford: Oxford University Press. Spence, C. (2010). The colour of wine Part 1. The World of Fine Wine, 28, 122129. Spence, C. (in press). Multisensory integration & the psychophysics of flavour perception. In J. Chen & L. Engelen (Eds.) Food oral processing: Fundamentals of eating and sensory
perception. Oxford: Blackwell. Spence, C., Levitan, C., Shankar, M.U. & Zampini, M. (2010). Does food color influence taste and flavor perception in humans? Chemosensory Perception, 3, 6884. Spence, C. & Shankar, M.U. (2010). The influence of auditory cues on the perception of, and responses to, food and drink. Journal of Sensory Studies, 25, 406430.
Stein, B.E. & Meredith, M.A. (1993). The merging of the senses. Cambridge, MA: MIT Press. Stein, B.E. & Stanford, T.R. (2008). Multisensory integration. Nature Reviews Neuroscience, 9, 255267. Stevenson, R.J. (2009). The psychology of flavour. Oxford: OUP. Stevenson, R.J. & Boakes, R.A. (2004). Sweet and sour smells. In G.A. Calvert, C. Spence & B.E. Stein (Eds.)
The handbook of multisensory processing (pp.6983). Cambridge, MA: MIT Press. Stevenson, R.J. & Tomiczek, C. (2007). Olfactory-induced synesthesias. Psychological Bulletin, 133, 294309. Verhagen, J.V. & Engelen, L. (2006). The neurocognitive bases of human multimodal food perception. Neuroscience and Biobehavioral Reviews, 30, 613650.
722
vol 23 no 9
september 2010
multisensory flavour perception
concerning the taste, aroma and flavour characteristics that are likely to be associated with a particular colour, and/or by any other visual appearance cues associated with the wine, are that much stronger than in the non-expert (Shankar et al., 2010; Spence, 2010). Many fast-moving consumer goods companies have already started to play with the mapping between food colour and flavour. Think only of green tomato ketchup, or the recently introduced Fruity Smarties or Confused Skittles, where the colours and flavours of the individual candies have been deliberately mixed up. Similarly, some of the chefs that I work with, such as Heston Blumenthal, have also started to play with their diners expectations through dishes such as the beetroot and orange jelly dish. When the waiter brings this dish to the table they will recommend that the diner should start with the beetroot. People naturally go straight for the purple-coloured jelly. However, beetroots turn orange on cooking, while Heston uses blood oranges that naturally have that dark purple colour.
Environmental contributions
Thus far, I have focused on the contributions to multisensory flavour perception of the various sensory cues gustatory, olfactory, oral-somatosensory, auditory and visual that are intrinsic to food and drink. However, the packaging in which a food is presented, the name/label it is given (Gallace et al., submitted; Shankar et al., 2009; Yeomans et al., 2008), and even the environment in which it is served, play an equally important role in modulating our perception of, and responses to, food and drink (Spence & Shankar, 2010). When, after all, was the last time you enjoyed airplane food (see Green & Butts, 1945)? Everything from the red-and-white checked tablecloth in a Mediterranean restaurant through to the colour of the lighting and the background music all affect both the foods and drinks we choose to order and how much/quickly
we consume them (Bell et al., 1994; see North & Hargreaves, 2008; Spence & Shankar, 2010, for reviews). But could environmental cues also influence the flavour and hedonic qualities of food and drink? The latest research suggests that they can. In one study, Heston Blumenthal and I demonstrated that people rate bacon-andegg ice-cream as tasting significantly more bacony when listening to the sound of sizzling bacon than to a farmyard of clucking chickens. In a second experiment, people rated oysters eaten while listening to the sound of the sea (i.e. the sound of seagulls squawking and waves crashing gently on the beach) as tasting significantly more pleasant than oysters eaten while listening to the farmyard noises. Taken together, these results (Spence & Shankar, 2010) highlight just how dramatically environmental sounds can influence (or bias) peoples perception of food. These findings led directly to the introduction of the Sound of the Sea dish at The Fat Duck. Diners are presented with a plate of food that is reminiscent of a beach (with foam, seaweed, and sand all visible on the plate). The dish also comes with a mini iPod hidden inside a sea-shell, with the earphones poking out. Diners are encouraged to insert the headphones (whereupon they hear the Sounds of the Sea soundtrack) before starting to eat. The dish is currently the signature dish on the tasting menu. Wearing headphones also has the advantage that is focuses the diners attention on the food itself. Elsewhere, Oberfeld et al. (2009) have shown that the ambient lighting can also affect peoples perception of the flavour, taste and market value of wine. They changed the colour of the ambient lighting at a winery on the Rhine from white to blue, red, or green, while 200 wine buyers rated Riesling (white) wines served in an opaque black glass (to ensure that the room colour did not impact on the colour of the wine itself). The customers liked the wine significantly more when it was tasted under blue and red lighting than under green or white lighting. What is more, the
maximum price that they were willing to pay was also nearly 50 per cent higher for wine tasted under red lighting than when the same wine was evaluated under green or white lighting. Follow-up experiments revealed that blue and green room lighting made wines taste spicier and fruitier, while a Riesling was rated as tasting nearly 50 per cent sweeter under red lighting than under blue or white lighting. In another recent study, Gal et al. (2007) demonstrated that the brightness of the ambient lighting can also affect peoples consumption of coffee. People who like strong coffee were found to drink more of the stuff under bright lighting than under dim lighting, whereas the reverse was true for those who preferred their coffee weak. Taken together, these results therefore demonstrate that both the intensity and colour of the ambient lighting can affect peoples perception of, liking for, and even their consumption of, drinks such as wine and coffee. Our perception of flavour involves not only the multisensory integration of all of the available cues in the food itself, but can also be influenced by the environment in which we happen to eat and drink (North & Hargreaves, 2008; Spence & Shankar, 2010).
Conclusions
The last few years have seen many important advances in our understanding of the multisensory perception of flavour at both the psychological and neural levels (see Stevenson, 2009; Verhagen & Engelen, 2006, for reviews). Cognitive neuroscientists have uncovered many key rules (such as sensory dominance and superadditivity) used by the human brain in order to combine the different sensory signals when we eat and drink. It should come as no surprise then to learn that chefs and other food producers are now starting to prepare dishes that more effectively stimulate the senses of the customer/consumer, because they have been designed on the basis of these recently discovered rules of multisensory flavour perception (see Blumenthal, 2008). The hope is that psychological insights can be harnessed in order to make food healthier (by reducing sugar, fat, salt, etc.) while not sacrificing flavour.
Wang, G-J., Volkow, N.D., Telang, F. et al. (2004). Exposure to appetitive food stimuli markedly activates the human brain. NeuroImage, 212, 17901797. Yeomans, M., Chambers, L., Blumenthal, H. & Blake, A. (2008). The role of expectancy in sensory and hedonic evaluation: The case of smoked salmon ice-cream. Food Quality and Preference, 19, 565573.
Young, J.Z. (1968). Influence of the mouth on the evolution of the brain. In P. Person (Ed.) Biology of the mouth. Washington, DC: American Association for the Advancement of Science. Zampini, M. & Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chips. Journal of Sensory Science, 19, 347363.
Charles Spence is head of the Crossmodal Research Laboratory at the Department of Experimental Psychology, Oxford University [email protected]
read discuss contribute at www.thepsychologist.org.uk
723
