Abstract
This paper is based on our exploration of building an integrated-sensory XR interactive system breaking through the sensory boundary between physical environment and metaverse via pneumatic wearables. In short, our exploration mainly focuses on the following two aspects. Firstly, this research has adapted pneumatic-actuated wearable devices to satisfy the needs of clothing comfort and embodied interaction concurrently by exploring the downsized body-scale pneumatic system and programmable soft materials. Secondly, this research explores the possibility of using digital wearables as the linkage of physical body and XR metaverse to enrich the interaction between XR metaverse and physical environment, aiming at the real-time synchronization of physical wearer's and his virtual avatar in XR system.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Digitality has become ubiquitous, permeating every realm of our lives [1]. The boundaries of the human self may extend beyond the physical body, and the consciousness of those extended boundaries has been driven by the development of the outside world [2]. It's important for humans to adapt to the more profound sense of being human-technological symbionts rather than the merely superficial sense of combining fresh and wires [3]. Since how humans interact with the world is greatly influenced and shaped by the tools used by them [3], wearables are considered as the second skin of people to inter-act directly and broadly with the built environment [4].
Pneumatic-actuated soft robotics has been a widely studied field recently with diverse applications [5]. Via pneumatic actuation, soft robotics has become increasingly accessible [6], and widely researched for application in biomimicry [7]. Furthermore, with the metaverse concept announced by Facebook, virtuality has been meant to have constant and seamless integration with existing physical reality [8]. Therefore, this research uses pneumatic actuated structures as smart garments to improve the comfort of digital wearables as the second skin of people. By using pneumatic actuated structures and physical sensors, this study will try to explore the digital wearables as the linker of physical body and XR metaverse, to enhance the XR sensory and physical comfort concurrently.
Contrasting with other digital bionic structures merely working with formal similarities. This study used pneumatic wearables as the representation of the latent living process embodied in creatures, like heart beating, blood flowing and breathing, to reconstruct the interaction between humans and their surrounding environments. For example, the pneumatic system combined the physical dynamics of plants and the flexible structures of human muscles. At the same time, the wearer's data and the signals from the surrounding environment can be transferred via wearables, triggering the interaction in both the physical and virtual system. In this way. A new dynamic of extended reality has been built with the wearables as an integrated interface of sensing and externalizing. Due to the bulkiness of the air pump of a traditional industrial pneumatic system, in this study, the pump and power system has been further downsized, so that it can meet the size restriction of the body-scale pneumatic system. Last but not least, with the synchronization of the pneumatic actuated wearables communicating via XR metaverse, the comfort of the physical body and the richness and vividness of the senses in the virtual world are simultaneously satisfied, providing prototypes verifying the feasibility for future application (Fig. 1).
2 Methodology and Prototype
2.1 XR Interaction Design—Physical Interaction of Digital Wearable
Cognitive philosopher Andy Clark raised the idea in 1998 that human beings are best regarded as an extended system, a coupling of biological organisms and external resources [2]. His ideas coincide with the status of the metaverse today: more than 4.6 billion people can access the virtual worlds of the metaverse via smart phones, laptops, desktops, headsets or consoles [8]. However, communication with the virtual environment is based more on visual content, such as virtual reality platforms providing an engaging and immersive environment [9], or 3D virtual worlds for communication via PC and smart phones [10].
As researchers have concluded, the main feature of metaverse is a twofold link between the virtual and physical worlds: (a) behavior in the physical world influences the experience in the virtual one and, (b) behavior in the virtual world influences the experience in the real one [11]. Touching as a channel for a great variety of information has always been regarded as a crucial site for mediating social perceptions [12] and comprehensive environmental perception [13].
This project explores how the wearables interact with extended reality as an integrated interface of sensing physicality and externalizing XR experience. As a means of telecommunications, the combination of electronic components and sensors for data transferring between the human body and digital system has already been explored in the above-mentioned research. Therefore, instead of adding new applications of telecommunications into the field of human-computer interaction, this project is more about how to introduce the richness and vividness of the XR senses and physical wearable experiences.
In this research, digital wearable devices break through the sensory boundary between physical wearers and its digital avatars. The body movement and heartbeat change of the physical wearer are collected via three-axis acceleration sensor, heartbeat sensor and infrared distance sensor. The movement of the physical wearer is synchronized with its digital avatar, as the touch between digital avatars will cause haptic pressure changes of the physical wearer via its digital pneumatic wearable device in further application.
At the same time, pneumatic wearables visualize and externalize the latent life process of the physical wearer. The digital wearables’ inflation and deflation according to the wearer's heartbeat rhythm, trigger changes in the bionic shape of the wearables and thereby change the tactile sensation of the wearer. Wearable devices draw on the techniques of three-dimensional tailoring and the study of muscle composition. Through the tailoring of pneumatics and clothing, the wearer's muscle dynamics during walking can also be creatively represented via wearables in both physical and XR worlds (Fig. 2).
In the current research of this project, only movement and heartbeat have been introduced into XR interaction and biotic visualization. In future applications, more scenarios of extended reality can be provided by richer collection of physical data such as environmental data of the wearer.
2.2 Pneumatic Actuated Structures and Soft Material
Under the spread of the digital environment and the popularity of digital devices, the fusion of traditional design and interactive technology has been accelerated in various fields [14], such as the application of 3D print in fashion design pioneered by Iris Van Herpan [15].
However, the above mentioned applications of digital technology devices have downplayed the typical characteristics of traditional fields, such as the basic requirements for comfort in the field of clothing [16]. Therefore, in this research, we have adapted the application of pneumatic systems and programmable materials to meet the basic needs of wearables for comfort and convenience.
2.2.1 Programmable Soft Materials
The heavy and expensive fabrication of the rigid body [5] hampers its efficiency and flexibility in body-device fit [17]. Therefore, silicone as a thoroughly explored material for soft robotics [18] has been used in this project as a programmable soft material. By computational design supported by Rhino and Grasshopper, we made a variety of 3D molds for silicone casting and fabricated the silicone airbags precisely according to their computed structures. Furthermore, through the combination of programable fabrication and pneumatic system, the deformation of functional garments can be assembled precisely to fit different modes of human activities. Last but not least, TPU has been fabricated as an alternative soft material for creating heat sealable sheets and laminated airtight layer [19]. Via patten prototyping designed in CAD and fabricated with laser cutting, the TPU airbag has also become programmable in the generating process from material to structure (Figs. 3 and 4).
2.2.2 Pneumatic System
The actuation system consists of three parts: Arduino toolkits for telecommunications between physical and digital space, sensors embedded in wearables, and pneumatic system for controlling the deformation of wearables. This section will introduce the downsized body-scale pneumatic system.
Air pump is controlled by two sets of solenoid valves which have three states (Inflate, Deflate and Hold), to meet the deformation requirement of the wearables via the inflation and deflation of airbags. For safety consideration, the solenoid valve selects the model SMC's S070C-SDG-32 powered by low voltage DC connecting with battery box.
Due to the bulkiness of the air pump of a traditional industrial pneumatic system, this project refits the air port of the portable air pump and connects it to the corresponding set of solenoid valves. Therefore, the entire pneumatic system only needs one portable air pump, two sets miniature electromagnetic valves and an Arduino toolkit connected with sensors. Through the design of digital wearable, the pneumatic system is organized into a pocket-sized box, which satisfies the portability and aesthetics of wearables simultaneously (Figs. 5 and 6).
2.2.3 Morphology and Pneumatic Structures
The combination of biological structure and morphology design has been widely used in various industrial fields, such as the mechanical properties of natural fiber cells [20] and soft elastic tissues of the human body [21].
To satisfy the comfort of the physical body and the vividness of tactile senses simultaneously, the morphology design draws on the dynamics of flexible structures and soft tissues, specifically, muscles as active elastic elements, skin as passive elastic elements and flytrap as pneumatic actuated structures.
The main structure of wearables follows the dynamic of muscle movement, while the holistic structure design takes both the static parts and kinematic joints into consideration. Therefore, the pneumatic structure can be compatible with the daily activities of wearer, even assist the muscles’ movement.
The airbag position mapping is generated by Grasshopper from the heat map of body movement. In the original prototype, the pneumatic structures consisted of airbag units fabricated with TPU and teflon after laser cutting. In further iterations, the pneumatic structures are added inflatable muscular structures made of silicone casting. In the composed pneumatic structures, the active parts fitted body movement are composed of muscular silicone structures, while the static parts are added on more biological features, to improve the comfort and richness of wearable devices (Figs. 7 and 8).
2.3 Interaction with XR Metaverse Space
The real-time synchronization between the physical wearer's behavior and one's virtual avatar is realized by Kinect motion capture via processing, physical sensors and their connections with metaverse space. As introduced in the physical interaction chapter, the wearer’s activity data triggers deformation of the wearable device in physical space, and the mirrorly changes of the digital avatar's position and appearance happen concurrently via the linkage of physical data and metaverse space. Furthermore, beyond mere synchronization of the physical wearer's activities, a richer interactive experience has been introduced into metaverse space via HoudiniFX space rendering. Walking and turning of the physical wearer will trigger smoke and light effects in metaverse space. Therefore, the wearer of Digital Wearable can experience extended-sensory interaction between XR metaverse and physical environment with integrated perception of tactile changes and visual effects (Figs. 9, 10, 11 and 12).
3 Conclusion and Discussion
The vision of this research is to build an integrated-sensory XR interactive system breaking through the sensory boundary between physical environment and metaverse. The current prototype has achieved the three goals mentioned above:
-
1.
The programmable fabrication of pneumatic structures satisfies the comfort of the physical body and the richness of the wearing experiences simultaneously.
-
2.
Morphological generation based on muscle structure research and human body heat map as representation of the latent life process and assistance with daily activities.
-
3.
Digital wearables as the sensory bridge of digital avatars in metaverse space and its physical wearer in physical space.
Future development will focus on improving the delicacy of sensory transmission and the richness of interactive activities between physicality and extended reality. Specifically:
-
1.
Now an interactive system consisting of wearables and metaverse has already been established, with only heartbeat and body movement as physical input from the wearer. In further research, the wearable can embed richer collections of physical data such as environmental data of the wearer to raise the vividness of experience.
-
2.
The current research has already provided a prototype of deformable Digital Wearables for Body-Scale, and the wearable design mainly refers to movement pattern of shoulder and arm. With growing accuracy and delicacy of morphological design, the wearability of devices will break through the boundaries of experiential device and become daily wear in the metaverse era.
-
3.
In this research, wearables have been verified as an effective medium connecting people and metaverse. Through the introduction of real-time space editing and web communication, metaverse social experiences (such as shaking hands and touching) can be comprehensive physicalized and extended to the wearer's embodied perception.
References
Llamas, R., Belk, R.: Living in a Digital Society. Routledge Handbook of Digital Consumption 2e, pp. 3–21. Routledge (2022)
Clark, A., David, C.: The Extended Mind. Analysis 58(1), 7–19 (1998)
Clark, A.: Natural-Born Cyborgs?. In: Beynon, M., Nehaniv, C.L., Dautenhahn, K. (Eds.) Cognitive Technology: Instruments of Mind. CT 2001. Lecture Notes in Com-puter Science, vol. 2117. Springer, Berlin, Heidelberg (2001)
Horn, M.J., Gurel Lois, M.: The Second Skin: An Interdisciplinary Study of Cloth-ing. Houghton Mifflin School (1981)
Shepherd, R.F., et al.: Multigait soft robot. Proc. Natl. Acad. Sci. 108(51), 20400–20403 (2011)
Matthew, C., Raye, C.: Propulsion-based soft robotic actuation. Robotics 6(4), 34 (2017). https://doi.org/10.3390/robotics6040034
Sareen, H., Umapathi, U., Shin, P., Kakehi, Y., Ou, J., Ishii, H., Maes, P.: Printflatables: printing human-scale, functional and dynamic inflatable objects. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3669–3680 (2017)
Stephens, M., Leader, C.: Metaverse and its governance—the IEEE global initiative on ethics of extended reality (XR), Institute of Electrical and Electronics Engi-neers, Inc (2022)
Gao, M., Xie, T., Chen, Z.: Wearable sensors and equipment in VR games: a review in book. Trans. Edutain. XV, pp.3–12 (2019)
Guidi, B., Michienzi, A.: Social game and blockchain exploring the metaverse of decentraland. In: Conference: International Workshop on Networked Entertainment Systems 2022At: Bologna (2022)
Riva, G., Lernia, D., Sajto, E., Sansoni, M.: Virtual reality therapy in the metaverse: merging VR for the outside with VR for the inside. In: Annual Review of CyberTherapy and Telemedicine 19:3–9, Project: Towards a Humane Metaverse (2021)
Morrison, I., Löken, L.S., Olausson, H.: The skin as a social organ. Exp. Brain Res. 204, 305–314 (2010)
Gibson, J.J.: Observations on active touch. Psychol. Rev. 69(6), 477–491 (1962)
Han, S., Kim, Y.: A study on the emotional represention of digital technology shown in the contemporary fashion. Res. J. Costume Cult. 23(2), 245–269 (2015)
Smelik, A.: Fractal Folds: the Posthuman fashion of Iris Van Herpan. Fashion Theory J. Dress Body Cult. 26(1), 5–16 (2020)
Tesinova, P., Atalie, D.: Thermal comfort properties of sport fabrics with dependency on structure parameters and maintenance. Fibers Polym. 23(4), 1150–1160 (2022)
Xu, J., Fu, K., Huang, Y., Jian, Z.: Review on mechanic modeling methods and applications of muscles in soft actuations 981–985 (2022)
Walker, J., Zidek, T., Harbel, C., Yoon, C., Strickland, F.S., Kumar, S., Shin, M.: Soft robotics: a review of recent developments of pneumatic soft actuators. Actuators 9(1), 3 (2020). MDPI
Ou, J., Helbeck, F., Ishii, H.: TEI 2016 Studio: Inflated Curiosity. MIT Media lab, Cambridge, MA 02139, USA (2016)
Ashby, M.F., Gibson, L.J., Wegst, U., Olive, R.: The mechanical properties of natural materials. I. Material property charts. Proc. Math. Phys. Sci. 450, 123–140 (1995)
Behnaz, F.: Caress of the gaze: a gaze actuated 3D printed body architecture. In: 36th Annual Conference of the Association for Computer Aided Design in Architecture, Ann Arbor, MI, USA, October, pp. 27–29 (2016)
Acknowledgements
This research was supported by Tongji University College of Architecture and Urban Planning and developed in DigitalFUTURES Shanghai 2018 Workshop with team members Shaoting Zeng, Heidi and Junwon Yoon. The authors would like to express great thanks to the instructors and teaching assistants of the workshop, specifically, Behnaz Farahi, Jifei Ou, and Yuqiong Lin. The authors would also like to extend grateful appreciation to MIT Media Lab Tangible Media Group, whose work laid the hardware foundation and motivated the authors to do this research. And special thanks to Mitsuru Isshiki for his help with building 3D characters and metaverse space.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2024 The Author(s)
About this paper
Cite this paper
Zang, B., Wang, T., Luo, D. (2024). The Embodied Interaction with XR Metaverse Space Based on Pneumatic Actuated Structures. In: Yan, C., Chai, H., Sun, T., Yuan, P.F. (eds) Phygital Intelligence. CDRF 2023. Computational Design and Robotic Fabrication. Springer, Singapore. https://doi.org/10.1007/978-981-99-8405-3_16
Download citation
DOI: https://doi.org/10.1007/978-981-99-8405-3_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8404-6
Online ISBN: 978-981-99-8405-3
eBook Packages: EngineeringEngineering (R0)