Thursday, February 23, 2012

3D Movies and the Primal Brain

In my previous blog, Engaged in 2D and Immersed in 3D I pointed out that in a 2D movie, motion is perceived as changes in the up, down and sideways position of one object in relation to other 2D objects. We can perceive three dimensions in a 2D movie via single eye or monocular cues such as occlusion, motion, shading, size, parallax, texture gradient, perspective, saturation and brightness. However, lacking in a 2D movie is the most powerful 3D influence on the brain—disparity. Disparity is the term used to define the two slightly different perspectives of the world that we see due to the separation of our eyes and the distance between them. It’s this separation of our eyes and our ability to converge them toward an object that gives us a true sense of depth and volume. We all remember from biology that our eyes send this visual information along bundles of nerve fibers or pathways to our primary visual cortex. It’s now known that this visual information subsequently travels from the primary visual cortex to higher portions of the brain that are specifically tuned to the degree of disparity created within the visual fields of our two eyes. In this, the second in my series on the neuroscience of 3D, I will point out that there is likely even more going on in a 3D movie than what we consciously perceive.

Can We See Without Seeing?

Seems like an odd question but one that has been a major source of research and controversy within the field of neuroscience for many years. There is a condition described decades ago by Dr. Larry Weiskrantz that was given the oxymoron, “blindsight” in which patients with severe damage to their primary visual cortex can nonetheless “see” emotionally charged objects. While these patients claim they see nothing, they nonetheless exhibit physiological responses indicative of emotional recognition of the objects.

How Can Patients That Are Blind From Visual Cortex Damage See?

There are neuroscientists, including one of my professors, Dr. Michael Gazzaniga who claims that the phenomenon of blindsight is simply a matter of tiny islands of spared visual cortex that are no longer fully connected as they would be in the intact brain. Others postulate the more intriguing idea that there are visual pathways, separate from those projecting to the more commonly known parts of the brain that mediate a type of vision that falls below the radar of conscious visual perception. Whether blindsight is a real phenomenon or merely severely degraded normal vision, the search for an explanation of how the blind can “see” in the absence of their visual cortex has stimulated a great deal of research into the possible role that other, previously unknown visual pathways play in subconscious awareness. We now know that information from our two eyes traverses the brain along many different routes including pathways destined for the more primitive parts of the brain called the limbic system that mediate emotional responses evoked by threatening visual stimuli, often in the absence of conscious awareness.

How does this relate to 3D movies?

Let’s consider an effect sometimes used in 3D movies for its shock value that has been referred to as the “gimmick factor”. I think many of us can point to anecdotal experiences we’ve had in 3D movies where we’ve found ourselves reflexively ducking out of the way of an object rapidly accelerating and coming out of the screen toward us. Indeed, those who went to see Piranha experienced various body parts that seemingly flew out at them from the 3D screen and those who saw the 2009 release of The Final Destination 3D experienced other, more rigid objects that appeared to fly out of the screen, intruding on their personal space. I went to the later film for the purpose of studying my own reactions to shock 3D as well as that of the audience around me. 

In the formulaic premonition scene in The Final Destination there were several instances of objects flying out of the screen into the audience that were intended to create an emotionally charged startle response. In one case I had just enough time to focus my eyes on an object jettisoned a hundred feet up in the air, slowly reaching the apex of its trajectory before coming down, “smashing” into the theater, hitting an actress that appeared to be standing between me and the screen and landing “in my lap”. Even though my intention was to observe the movie objectively and analytically from a professional stereographic vantage point, I experienced an involuntary sense of heightened awareness and anticipation as the object reached the apex of its trajectory. Then in the blink of an eye, as the object dropped toward me, the disparity of the object crossed and widened, signaling to my brain that it was headed out of the screen, toward the girl in the movie and me. Literally before I had a chance to think, I found myself “instinctively” moving out of “harms way”. It was nothing I did consciously and I noticed other people in the audience reacting in the same manner.

Was I experiencing a type of blindsight?

Was I subconsciously sensing danger causing me to instinctively move away? David Ellis, the director of The Final Destination is a master of emotionally priming us with images that build tension and anticipation before hitting us almost instantaneously with extremely disturbing images that strike our retinas and literally shock our nervous system. The significance of 3D in facilitating my reaction was obvious to me after the fact. The added dimension provided precise information about where the object was in the air relative to my own body and the subsequent changes in dynamic disparity during the drop signaled the speed and direction of its path through the screen and into my personal space. I don’t believe there was sufficient time for me to consciously move out of the way, nor do I think I would have been motivated to move if I had been fully aware of what was happening.

Experiencing the film in 2D would have distanced me from the action.

As I reflected on the film, I wondered if I might have experienced similar reactions had I seen the movie for the first time in 2D. I believe there would have been a startle reaction in 2D but not as intense as the one I experienced in 3D and there is a very compelling reason why. In Engaged in 2D and Immersed in 3D I discussed that in our everyday lives we converge our eyes on objects that are in close proximity to us, creating two perspectives that are slightly offset horizontally because of the separation of our eyes. Our brain fuses the two perspectives into a single 3D image creating the impression of depth and volume. However, when objects are viewed at distances of several hundred yards, depth cues that we can detect with one eye such as relative movement of objects and occlusion (one thing passing in front of another) exist, but the horizontal offset of the images from our two eyes becomes less and less as the distance of the object from us increases. Because our eyes do not converge on objects in the distance, the resulting images falling on both retinas are precisely the same with no horizontal offset. This lack of disparity in the presence of monocular depth cues signals to the brain that objects are far away. 

In 2D movies our brain is constantly presented with a cue conflict in which objects on the screen that might normally be perceived as close to us, have no disparity. Consequently our brain interprets those objects and the entire 2D storytelling as taking place at a distance from us. Our attention in 2D movies is directed to the screen “over there”. This is in contrast to how we perceive a 3D movie where we experience full dynamic disparity that can evoke conscious and subconscious responses to objects appearing all around us as well as objects that appear deep behind the screen. Indeed, our attention during a 3D movie is influenced by the entire spatial expanse of the movie theater. As a consequence, dramatic emotional effects directed to the audience in a 3D movie become very personal to each audience member.

The high road and low road of vision

There are volumes of data in the scientific literature to support the notion that emotionally charged visual information, perhaps like the object I experienced in The Final Destination travels through the brain via several different pathways some of which do not include the primary visual cortex and conscious awareness.  One of these pathways - the Parvocellular Pathway - is metaphorically called the high road.  It sends data-rich visual information (color, texture, shape, etc.) that we perceive consciously about what something is to higher portions of the brain including the visual cortex. The other pathway - the Magnocellular Pathway – called the low road sends sparse visual information (i.e., edges, location, direction and speed) about where something is in spatial relation to our body to primal neural structures lying deep within the brain.  It is thought that this pathway often operates in the absence of consciousness.

To be effective in immediate danger situations the low road brain cells must carry critical information about the location of objects more rapidly than the high road that in turn sends more detailed information about what the object actually is. The information carried by the low-road traverses the brain faster because it filters out only mission critical survival information about location, direction and speed and also, the neurons that make up the low-road are heavily insulated by a fatty sheath that increases neuronal conduction times through the brain. In other words, information transmitted via the low road causes us to duck out of the way before we have time to consciously assess what it is we are moving away from. 

Disparity is a key element in low road activation and transmission

The above is admittedly a simplistic view of the way the brain handles instantaneous, emotionally packed visual information. In reality the brain is a complex matrix of neuronal activity constantly in flux and shifting in location, intensity and frequency depending on many things including attention, sensory input and motivational factors. To focus on one structure within the brain and point to how it functions in isolation from the rest of the brain is not particularly valid. However in an effort to better understand the brain and how it functions in relation to 3D movies it is reasonable to postulate from the scientific literature that a key to subconscious visual startle reactions is dynamic disparity, the most sensitive indicator of position in space, speed of movement and direction of movement. In analyzing reflexive emotional responses it would appear obvious that the number of way stations that visual information relevant to survival must pass through has to be limited and those structures that are involved initially should have influence that outweighs other ongoing priorities in brain activity.

We know that one of the first areas of the brain to receive low road information is the superior colliculus, lying deep within the brainstem.  This is an area that controls gaze and head turning and can subconsciously trigger rapid head and eye movement, forcing us to converge toward an immediate danger situation. Significantly, the majority of brain cells in the superior colliculus are specifically tuned to react to variations in the disparity coming from our two eyes.  In fact neurons have been described in the superior colliculus that respond selectively only to “near” objects and other neurons only to “far” objects.  It’s not likely that the disparity information reaching the superior colliculus results in the fusion of the two perspectives from our eyes into a 3D image.  Instead, I believe that the data received by disparity-tuned neurons in this brain structure is processed to determine relative distance, direction and speed in the absence of constructed 3D images.

These disparity sensitive neurons in the superior colliculus subsequently send processed disparity information directly to the amygdala in the limbic system as well as to other primitive brain structures that have been well established as modulating emotion. In extreme survival situations, the activated amygdala is known to trigger behavioral and autonomic responses (visceral responses below the level of consciousness) that evoke flight or fight reactions such as instantly ducking or moving out of harms way in response to threatening visual stimuli.

Conclusion

We know that our highly evolved frontal cortex mediates cognitive functions that normally inhibits or controls the amygdala and consequently evoked emotions.  I contend that our brain’s reaction to a 2D movie is more of a top down cognitive process involving the higher order frontal cortex.  The greater our suspension of disbelief, the more these higher cognitive functions come into play, releasing emotions mediated by the lower order amygdala.  On the other hand, in a 3D movie, emotionally packed visual information more directly affects these lower order emotional areas of the brain in a bottom up fashion, evoking a more primal subconscious response.

The description of my experience in the shock film The Final Destination is intended as an example of one end of a continuum of emotional and affective responses in the brain that are likely evoked during 3D movies; from the very subtle to the extreme. Indeed it’s likely that the subtle but effective single frame image of Brad Pitt appearing behind Ed Norton as his alter ego in Fight Club might have had a greater emotional impact had it been in 3D.  I believe the more extreme subliminal instances of Captain Howdy’s face in The Exorcist would have had an even greater effect in 3D, not only because of the underlying story but also because threatening faces and faces exhibiting extreme fear are some of the most effective emotional visual stimuli in humans.  On the other end of the spectrum, simply the timing and the framing of a 3D shot, the relationships of the characters in frame and the relative motion of the antagonist can create a greater degree of anticipation, anxiety and/or fear with the proper use of disparity.

All the parameters of binocular vision that we experience in the real world also exist in the 3D theater. At Legend3D, we apply many of the established neuroscience principles related to binocular vision and perception in our process of creating the most effective 3D to enhance the storytelling as well as in producing the most comfortable 3D to watch. I believe it will be found that the same binocular visual mechanisms that have served our species so well for hundreds of thousands of years are very much in play when we experience a 3D movie.  How disparity is choreographed in a 3D movie can enhance or detract from the overall experience. In the wrong hands 3D can be distracting and even quite painful, but in the right hands a great story, great characters and skillful direction coupled with the proper use of disparity will always enhance the storytelling.

(1)   Bacon, B.A., Villemagne, J., Bergeron, A., Lepore, F., Guillemot, J.P. Spatial Disparity Coding in the Superior Colliculus of the Cat. Experimental Brain Research 1998 119, 3, 333-44.
(2)   Campion, J., Latto, R. & Smith, Y. M. Is Blindsight an Effect of Scattered Light, Spared Cortex, and Near-threshold Vision? Behavioral and Brain Sciences 1983 6, 423-448.
(3)   de Gelder, B., Vroomen, J., Pourtois, G. & Weiskrantz, L. 1999. Non-conscious Recognition of Affect in the Absence of Striate Cortex. Neuroreport 10, 3759–3763.
(4)   Fendrich, R., Wessinger, C., Gazzaniga, M. Speculations on the Neural Basis of Islands of Blindsight. Elsevier. 2001, 134, 353-366.
(5) LeDoux, J. E. Brain Mechanisms of Emotion and Emotional Learning. Current Opinion in Neurobiology.  1992, 2, 191–197.
(6)   LeDoux, J. E. 1996.The Emotional Brain. Simon & Shuster, New York.
(7)   LeDoux, J.E. Emotion Circuits in the Brain. Annual Review of Neuroscience. 2000, 23, 155–184.
(8)   Liddell, B., Brown, K., Kemp, A., Barton, M., Das, P., Peduto, A.  Gordon, E. and Williams, L. 2005. A Direct Brainstem–Amygdala–Cortical “Alarm” System for Subliminal Signals of Fear.  NeuroImage 1999, 24, 235– 243.
(9)   Linke, R., De Lima A.D., Schwegler, H., Pape H.C.  Direct Synaptic Connections of Axons from Superior Colliculus with Identified Thalamo-Amygdaloid Projection Neurons in the Rat: Possible Substrates of a Subcortical Visual Pathway to the Amygdala. The Journal of Comparative Neurology. 1999 403, 2, 158–170.
(10) Mimeault, D., Paquet, V., Molotchnikoff, S., Lepore, F., Guillemot, J.P. Disparity Sensitivity in the Superior Colliculus of the Cat. Experimental Brain Research. 2004 1010, 1-2, 87-94
(11) Morris, J.S., Ohman, A., Dolan, R.J., A Subcortical Pathway to the Right Amygdala Mediating ‘Unseen’ Fear. Proceeding of the National Academy of Sciences of the United States of America. 1999 96, 1680–1685.
(12) Phelps, E. A. & LeDoux, J. E. Contributions of the amygdala to Emotion Processing: From Animal Models to Human Behavior. Neuron 2005 48, 175–187.
(13) Tamietto, M. and Gelder, B. 2010.Neural Bases of the Non-Conscious Perception of Emotional Signals. Nature Reviews Neuroscience. 11, 697-709
(14) Weiskrantz, L. 1986. Blindsight: A Case Study and Implications. Oxford: Oxford     University Press.
(15) Whalen, P. J. and Phelps, E. A. 2009. The Human Amygdala. Guilford Press.

I wish to acknowledge the review and suggestions of the following professionals:

David Loiselle, Ph.D. is Clinical Associate Professor of Neurology at the University of Rochester and Director of Intraoperative Neuro-monitoring at the University of Rochester Medical Center. Dr. David Loiselle is an internationally recognized clinical neurophysiologist. He was one of the first innovators and a leading proponent of interoperative electrophysiological monitoring to help reduce the risk to patients undergoing surgeries involving the nervous system. He was the first neurophysiologist to establish a curriculum for training medical residents and fellows in the clinical utilization of evoked potentials. Dr. Loiselle He is an active member of the American Academy of Neurology, American Clinical Neurophysiology Society, American Epilepsy Society, International Society of Intraoperative Neurophysiology, and the American Society of Neurophysiological Monitoring, where he is member of the Board of Directors. 

Scott Squires has a full range of experience working with both traditional and cutting edge digital techniques to create photo real visual effects. His career began with the development of the Cloud Tank Effect for Close Encounters of the Third Kind. Among the many feature films he has worked on, he received Oscar nominations for Best Achievement in Visual Effects for The Mask, Dragonheart and Star Wars: Episode 1 - The Phantom Menace. Scott is the recipient of a Scientific and Engineering Award from the Academy of Motion Picture Arts and Sciences for his pioneering work in the area of Film Input Scanning. He was co-founder of Dream Quest Images, a leading visual effects facility where he was Visual Effects Supervisor, as well as the company’s president for six years before the company was acquired by Disney Studios. From there he went to ILM where, where for 20 years he was its first CTO as well as visual effects supervisor and commercial director. Scott Squires is a member of the Academy of Motion Pictures Arts and Sciences - Visual Effects Branch. He is on the board of the Visual Effects Society, serves on the technical committee and is an author of several chapters in the Visual Effects Handbook. Scott Squires serves as consulting visual effects supervisor for Legend3D. He maintains a very popular blog The Effects Corner read by many visual effects professionals. 

Carlos G. Perez-Garcia, Ph.D. is Senior Research Associate in neuroscience at the Salk Institute. He is considered an expert in human neuro-anatomy and molecular genetics in developmental neuroscience. His career is focused on the study of genes involved in embryonic development of human and mouse neo-cortex – specifically the study of the defects caused by deficiencies of those genes in human brain disorders such as lissencephalies and double cortex syndrome as well as schizophrenia and bipolar disorders. Dr Perez-Garcia is also a collaborator in the study of neurological aspects of genes strongly expressed in a variety of cancers/tumors in human central nervous system.