Abstract
The world is facing the problem of anthropogenic climate change and air pollution. Despite many years of development, already established methods of influencing behaviour remain ineffective. The effect of such interventions is very often a declaration of behaviour change that is not followed by actual action. Moreover, despite intensive information campaigns, many people still do not have adequate knowledge on the subject, are not aware of the problem or, worse, deny its existence. Previous attempts to introduce real change were based on providing information, persuasion or visualisation. We propose the use of multi-sensory virtual reality to investigate the problem more thoroughly and then design appropriate solutions. In this paper, we introduce a new immersive virtual environment that combines free exploration with a high level of experimental control, physiological and behavioural measures. It was created on the basis of transdisciplinary scientific cooperation, participatory design and research. We used the unique features of virtual environments to reverse and expand the idea of pollution pods by Pinsky. Instead of closing participants in small domes filled with chemical substances imitating pollution, we made it possible for them to freely explore an open environment - admiring the panorama of a small town from the observation deck located on a nearby hill. Virtual reality technology enables the manipulation of representations of air pollution, the sensory modalities with which they are transmitted (visual, auditory, tactile and smell stimuli) and their intensity. Participants’ reactions from the initial tests of the application showed that it is a promising solution. We present the possibilities of applying the new solution in psychological research and its further design and development opportunities in collaboration with communities and other stakeholders in the spirit of citizen science.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Theoretical Context
1.1 Introduction
Reduction of air pollution is an urgent challenge that European societies and governments have to face. Despite policies in place, yearly air quality improvements fall short to Europe’s zero pollution ambition. Worldwide, around 90% of people breathe polluted air (97% of Poles - six out of ten most polluted cities of Europe are located in Poland). Over the past 6 years, air pollution levels have remained high and stable, way above the levels recommended by the World Health Organization. Air pollution leads to increased mortality, one in five (20%) natural deaths are attributed to air pollution [14, 27]. The destructive consequences go beyond death and respiratory symptoms: anxiety, dementia, missed work, increased harm of Covid etc. Air pollution is largely a behavioural problem - it can undeniably be connected with human actions (burning solid fuels), not industrial sources. Air pollution has local, close to source, consequences: it directly affects one’s own health and that of the neighbours, as well as the direct environment (fauna, flora, urban surroundings). It creates a microsystem, which can be thoroughly monitored, measured and modelled in transdisciplinary cooperation between social and atmospheric scientists. However, before we start with the field studies, the carefully designed virtual environment will allow us to understand the behavioural processes already in place and build evidence-based models with potential for replication in real contexts and places. The role of modern virtual reality technologies is to make testing solutions cheaper and more effective.
In the following sections: we will describe what the problem with changing air pollution behavior is and how virtual reality can help; we will discuss what different types of risk communication are and how you can use multiple senses to make it more effective; we will present studies showing psychophysiological responses to risk messages and how this can be used in a research application to select the most effective air pollution messages (but not only as these responses are quite universal); we will characterize the basics and the process of creating a multisensory virtual environment, taking into account the participatory process; finally, we share our first experiences of testing an environment containing visual stimuli.
1.2 The Problem
Air pollution is an invisible killer. Our aim is to effectively communicate the health and environmental threat. We will use the potential of virtual reality technologies to find, test and implement novel solutions that influence the way people envision and battle the problem of air pollution. We investigate how multisensory virtual experience and pollution visualisation impact environmental attitudes and behaviours. Before being introduced in the field studies, each form of multisensory virtual experience has to be tested in virtual reality labs. In the lab-experiments we will stimulate multiple senses (vision, hearing, touch, and smell) in a controlled and replicable manner, and gather high quality reliable data about people’s reactions to sensory stimuli.
1.3 Risk Communication
Visualisations and sound signals designed to build awareness of potential dangers are encountered daily and serve the purpose of saving people from harming themselves or others [18]. Examples of risk communication design include road signs, labels of toxic or poisonous substances, tobacco products’ packaging, alert sounds. They need to be clearly and instantly distinguishable and understandable despite the receiver’s age, culture or graphic literacy [5, 18, 24]. Fundamental elements of risk communication design ought to evoke mutual reactions. Inspiration often comes from signals observed in nature e.g. colours of poisonous frogs or venomous snakes [9, 17]. Research in the field focuses on principle elements of communication such as shapes, colours or sounds that evoke attention or disgust [13, 18, 24, 28]. Specifically, exposure to visualisations of shapes resembling the letter “V” evoke reaction in amygdala: part of the brain responsible for fight-or-flight defensive response [16]. Example of using other senses could be alerting drivers with haptic feedback [26]. A use of risk communication may be encountered almost everywhere, from supermarket shelves to the deep forest. Virtual Reality technology has a great potential of implementing risk communication signals, but research using it has a relatively short history, as only the last dozen or so years has brought the increasing popularity of this technology and its use outside the laboratory.
1.4 Neurophysiological Response to Risk Signals
A strong multimodal response increases the likelihood of the effectiveness of the risk signals. Testing them in VR gives the opportunity to check not only the participants’ declarations, but also the use of more objective measurements - psychophysiological reactions. Millisecond measurement precision and synchronization with external medical-class equipment allow you to record any signals, but it is worth selecting the most promising ones.
Threat-related signals induce strong neurophysiological responses associated with both emotional and attentional processes. Affective neuroscience has identified the amygdala as the primary neural site responding to danger-related stimuli [20]. Through its bidirectional connections with sensory regions, the amygdala has been proposed to enhance sensory processing of emotionally relevant stimuli [23], as evident by increased activity in the visual cortex [8, 22] and auditory cortex [12]. In addition to subcortical areas involved in processing of affective stimuli, the prefrontal cortex (PFC) has been implicated in emotion regulation [2, 7]. Increased activity of the PFC was observed upon presentation of fearful stimuli across different modalities in experiments using functional magnetic resonance imaging (fMRI) [3], functional near-infrared spectroscopy (fNIRS) [11, 19] as well as electroencephalography (EEG) [8].
Risk signals, perhaps via emotional processes, have been found to effectively capture and hold attention [15, 23]. Selective attention to threat is exerted through the interaction between bottom-up (perceptual) and top-down (cognitive) processes [6]. At the level of perception, threat-related stimuli are detected more efficiently [10] and attract the gaze with a higher frequency [4] than neutral stimuli. Cognitive processes modulating attention are linked to activity in the frontal and parietal cortices [1]. In summary, neuroscientific findings on risk-related signals have pointed to the role of amygdala, prefrontal cortex and sensory cortices associated with emotional processing and to the frontal and parietal regions recruited during selective attention to threat.
2 Construction of the Environment
Considering the above, there is a need to construct a virtual environment for use in immersive virtual reality that will enable: 1) testing multi-sensory representations of air pollution (visual, auditory, tactile and smell); 2) registration of emotional, attention and behavioural responses of participants; 3) millisecond precision; 4) communication with external research equipment. The use of a participatory approach implies a test with the participation of end-users, i.e. residents of small towns struggling with the problem of air pollution caused by residential and water heating.
2.1 Inspiration - Pollution Pods
The design of the virtual environment was inspired by the artistic installation by Michael Pinsky - the pollution pods, which he himself describes as an answer to the challenge of “representing the invisible”. He tried to apply the knowledge from the fields of environmental psychology, empirical aesthetics, and activist art. Pollution Pods is a sensorial experience created in five domes connected with each other to form a ring. Within each dome artist with the help of scientists recreated the air quality of five global cities. The visitors (volunteers from Trondheim and London, as the installation was displayed in public space) passed through increasingly polluted cells. The effect of pollution pods on visitors was mixed. E.g. reported intentions to act were strong and increased after participation, but were not followed by actual behaviour (participants did not track their climate change emissions afterwards). Nevertheless, it was concluded that environmental art can be useful for environmental communication [21, 25].
2.2 The Multisensory Virtual Environment
The need to reliably recreate the smell of polluted air forced the author of the “pollution pods” to use small, tightly sealed domes. This necessary procedure, however, made the whole experience somehow artificial and distant from actually being in Trondheim, London or New Delhi and breathing the local breeze. Participants were not able to experience being in the city space, characteristic sights and sounds, disturbance of the aerial perspective due to pollution. The use of VR overcomes these limitations and provides additional opportunities to enhance and go beyond reality. Instead of closing participants in small domes filled with chemical substances imitating pollution, we made it possible for them to freely explore an open environment - admiring the panorama of a small town from the observation deck located on a nearby hill. The deck was divided into five separate spaces - terraces. They were located around the hill in such a way that one could see a fragment of the next one from each of them, but the view from it was obstructed by the hill (Fig. 2). From each of the terraces one could see a fragment of the town, houses, trees and characteristic buildings. Thanks to this, the participants could experience a slightly different view from each of the terraces. These views, however, were perceptually similar and complex to the same degree, which meant that the observed changes in behavior on each of the terraces could only be the result of experimental manipulation, not different environmental conditions. The type and intensity of the air pollution stimuli could be freely adjusted on individual terraces and changed over time. An example scenario may be the increasing concentration of PM2.5 and PM10 particles magnified so that they become visible to the naked eye (Fig. 1).
An additional measure to increase the realism of the experience was to adjust the size of the terrace to the size of the laboratory. Thanks to this, with the use of a wireless VR set, participants could freely explore the terraces. High barriers at an appropriate distance were put in place to prevent collisions with the walls which could cause injuries as well as breaks in presence. Additionally, the hill behind the barriers descended gently to prevent unpleasant sensations caused by the fear of heights. Comfortable movement in the environment is also possible thanks to short, standardized training before starting the actual simulation.
The use of VR technology made it possible to track the behavior of participants and their reactions to various representations of air pollution. The dynamics of movement in space can be followed thanks to the constant registration of the position and rotation of the participant’s body. In addition, eye movements, dilation of the pupils and the id of objects on which the gaze falls at a given moment are recorded. Thanks to markers sent to the external apparatus via parallel port, synchronization with measures of physiological reactions is possible. The modular form of the VR application ensures its scalability. The number and type of stimulation the participants are to experience may be manipulated in any way. The introduction of the possibility of answering the questions of the questionnaires without removing the HMD additionally improves the usability and does not interrupt the feeling of immersion in the virtual world.
2.3 The Participatory Activities and Design Process
The virtual environment idea was one of the topic of participatory design and research workshops during the project kick-off with local community and stakeholders. It was a part of the citizen science approach with direct potential end users’ involvement. At the next step the early-stage prototype was presented to volunteers, inhabitants of the town of Myszków, which is on the list of 50 Polish cities with the most polluted air. They are representatives of potential recipients of solutions developed on the basis of laboratory tests in VR. Citizens and various project stakeholders of Myszków participated in the Family Ecological Picnic organized jointly with the local authorities of the town and non-governmental organizations. The VR experience was one of the highlights of the event. Due to the limitations related to the COVID19 pandemic, participants experienced the VR environment individually with all necessary precautions and in an open-air setup on a stage in a local activity park. During and after the experience, they were able to share their impressions and provide feedback to team members. An additional group were volunteers who participated in the laboratory experiment. They had the opportunity to record their comments during the experiment. Below are some representative statements from VR experience participants:
«Is it really so much of this [particulate matter] floating in the air now? But today the air seems clear, how come?»
«It’s scary...it is all around»
«Objects floating in the air made me feel sad and somehow unsafe. They resembled insects, but they weren’t them. I felt uncomfortable, especially as there were more and more of them.»
«The air was full of some black creatures. As it got more and more of them, they looked like locusts. It wasn’t pleasant. But the next visuals were very pleasant, I like such atmosphere very much. I could stay in them for a long time.»
«I would like to always see the world this way.»
Direct involvement of end users delivered valuable and immediate feedback that was very insightful for the next stages of the development of our immersive multisensory VR experience. Therefore, it was also an interesting example of citizen science - as it enabled the participants to shape the development of our immersive environment and research tool to be used for further VR lab studies.
3 Conclusions
The interdisciplinary approach to the project, both since its conception and inspiration from artistic installations, through research involving psychological, economic, social and technological dimensions is a firm step towards stronger collaboration between disciplines. Such collaboration, which may be dubbed as transdisciplinary, when paired with the involvement of local communities and other stakeholders, in the spirit of citizen science, is a very promising way to address the complex and wicked problems of today’s world, such as environmental challenges, or, as in our case, air-pollution.
References
Armony, J.L., Dolan, R.J.: Modulation of spatial attention by fear-conditioned stimuli: an event-related fMRI study. Neuropsychologia 40(7), 817–826 (2002)
Banks, S.J., Eddy, K.T., Angstadt, M., Nathan, P.J., Phan, K.L.: Amygdala-frontal connectivity during emotion regulation. Soc. Cogn. Affect. Neurosci. 2(4), 303–312 (2007)
Bermpohl, F., et al.: Dissociable networks for the expectancy and perception of emotional stimuli in the human brain. Neuroimage 30(2), 588–600 (2006)
Bradley, B.P., Mogg, K., Millar, N.H.: Covert and overt orienting of attention to emotional faces in anxiety. Cogn. Emot. 14(6), 789–808 (2000)
Bresciani, S., Eppler, M.J.: The pitfalls of visual representations: a review and classification of common errors made while designing and interpreting visualizations. SAGE Open 5(4), 2158244015611451 (2015)
Corbetta, M., Shulman, G.L.: Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3(3), 201–215 (2002)
Davidson, R.J., Putnam, K.M., Larson, C.L.: Dysfunction in the neural circuitry of emotion regulation-a possible prelude to violence. Science 289(5479), 591–594 (2000)
DeLaRosa, B.L., et al.: Electrophysiological spatiotemporal dynamics during implicit visual threat processing. Brain Cogn. 91, 54–61 (2014)
Donath, J.: Signals, cues and meaning. Unpublished Manuscript. Massachusetts Institute of Technology, Cambridge, MA (2007). http://smg.media.mit.edu/classes/IdentitySignals06/SignalingDraft.pdf
Fox, E., Lester, V., Russo, R., Bowles, R., Pichler, A., Dutton, K.: Facial expressions of emotion: are angry faces detected more efficiently? Cogn. Emot. 14(1), 61–92 (2000)
Glotzbach, E., Mühlberger, A., Gschwendtner, K., Fallgatter, A.J., Pauli, P., Herrmann, M.J.: Prefrontal brain activation during emotional processing: a functional near infrared spectroscopy study (fNIRS). Open Neuroimaging J. 5, 33 (2011)
Grandjean, D., et al.: The voices of wrath: brain responses to angry prosody in meaningless speech. Nat. Neurosci. 8(2), 145–146 (2005)
Kemp, D., Niederdeppe, J., Byrne, S.: Adolescent attention to disgust visuals in cigarette graphic warning labels. J. Adolesc. Health 65(6), 769–775 (2019)
Khomenko, S., et al.: Premature mortality due to air pollution in European cities: a health impact assessment. Lancet Planetary Health 5(3), e121–e134 (2021)
Koster, E.H., Crombez, G., Van Damme, S., Verschuere, B., De Houwer, J.: Does imminent threat capture and hold attention? Emotion 4(3), 312 (2004)
Larson, C.L., Aronoff, J., Sarinopoulos, I.C., Zhu, D.C.: Recognizing threat: a simple geometric shape activates neural circuitry for threat detection. J. Cogn. Neurosci. 21(8), 1523–1535 (2009)
Lindström, L., Kotiaho, J.S.: Signalling and reception. e LS (2001)
Lipkus, I.M., Hollands, J.G.: The visual communication of risk. JNCI Monographs 1999(25), 149–163 (1999)
Moghimi, S., Kushki, A., Guerguerian, A.M., Chau, T.: Characterizing emotional response to music in the prefrontal cortex using near infrared spectroscopy. Neurosci. Lett. 525(1), 7–11 (2012)
Öhman, A.: The role of the amygdala in human fear: automatic detection of threat. Psychoneuroendocrinology 30(10), 953–958 (2005)
Pinsky, M., Sommer, L.: Pollution pods: can art change people’s perception of climate change and air pollution? Field Actions Sci. Rep. J. Field Actions (Special Issue 21), 90–95 (2020)
Pourtois, G., Grandjean, D., Sander, D., Vuilleumier, P.: Electrophysiological correlates of rapid spatial orienting towards fearful faces. Cereb. Cortex 14(6), 619–633 (2004)
Pourtois, G., Schettino, A., Vuilleumier, P.: Brain mechanisms for emotional influences on perception and attention: what is magic and what is not. Biol. Psychol. 92(3), 492–512 (2013)
Silic, M., Cyr, D., Back, A., Holzer, A.: Effects of color appeal, perceived risk and culture on user’s decision in presence of warning banner message. In: Silic, M., Cyr, D., Back, A., Holzer, A. (eds.) Effects of Color Appeal, Perceived Risk and Culture on User’s Decision in Presence of Warning Banner Message. In Proceedings of the 50th Hawaii International Conference on System Sciences (2017)
Sommer, L.K., Swim, J.K., Keller, A., Klöckner, C.A.: “pollution pods”: the merging of art and psychology to engage the public in climate change. Glob. Environ. Change 59, 101992 (2019)
Spence, C., Ho, C.: Tactile and multisensory spatial warning signals for drivers. IEEE Trans. Haptics 1(2), 121–129 (2008)
Vohra, K., Vodonos, A., Schwartz, J., Marais, E.A., Sulprizio, M.P., Mickley, L.J.: Global mortality from outdoor fine particle pollution generated by fossil fuel combustion: results from GEOS-chem. Environ. Res. 195, 110754 (2021)
Zikmund-Fisher, B.J., et al.: Blocks, ovals, or people? Icon type affects risk perceptions and recall of pictographs. Med. Decis. Making 34(4), 443–453 (2014)
Acknowledgment
This project received funding from \(\#\)eeagrants NCN IDEALAB grant no. 2019/35/J/HS6/03166. We would like to thank our participants, volunteers from VRLab IP PAN panel and citizens of Myszków, Poland, who took part in our studies.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2022 The Author(s)
About this paper
Cite this paper
Pochwatko, G. et al. (2022). Multisensory Representation of Air Pollution in Virtual Reality: Lessons from Visual Representation. In: Biele, C., Kacprzyk, J., Kopeć, W., Owsiński, J.W., Romanowski, A., Sikorski, M. (eds) Digital Interaction and Machine Intelligence. MIDI 2021. Lecture Notes in Networks and Systems, vol 440. Springer, Cham. https://doi.org/10.1007/978-3-031-11432-8_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-11432-8_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-11431-1
Online ISBN: 978-3-031-11432-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)