Abstract
Augmented reality (AR) technologies can enhance the user’s experience of visiting attractions, shops, and restaurants by using AR-based virtual elements and additional information about the places they are visiting. In this work, we transform the city landscape or iconic buildings into a unique experience by bringing iconic characters onto the buildings to increase users’ engagement. Our techniques transform buildings or parts of a building into a virtual character with which the user can interact. We designed two unique experiences: (a) ‘The Square’ in which the character will talk about the building’s history and other anecdotes about the area, and (b) ‘The Hunt’ in which the user is involved in a scavenger hunt where they have to identify buildings using the hints given by virtual characters. We have conducted a live user study to assess our prototype’s usability. Our preliminary experimental results demonstrated that our prototype has high usability and users using our system felt a pleasant and enjoyable experience.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
The application of augmented reality (AR) technologies in recent years opened up many new possibilities to incorporate AR into our daily lives. In [3], by conducting a live user study, it has been shown that augmenting various points of interest (POIs) with images and facts about each POI has a potential to enhance user experience in tourism. In this work, we aim towards this goal of exploiting the AR technologies for tourism and transform the famous buildings into an unique experience by bringing iconic characters onto the buildings wall to increase the engagement of users. With a goal to create a unique experience using the virtual characters and the iconic buildings of a touristic city, we developed a mobile application that projects a virtual character when a user visits the building. By conducting a live user study, our system demonstrated has a high usability and users using our system felt a pleasant and enjoyable experience.
2 Talking House: Transforming Buildings into Characters
Augmented reality has the potential to enhances a touristic experience by inserting virtual elements onto the buildings in a city. In general, famous touristic buildings and architectures attract a large number people. In this work, we propose a technology which is based on AR to transform buildings into intelligent characters to increase the engagement of users when visiting new places. Our techniques transforms a building or part of a building into a visible character with which the user can interact in order to learn useful information (or listen to stories) about the building. In this section, we describe the details of our prototype and various steps we adopted in creating the AR experience.
We have developed an AR mobile application called ‘Talking House’ for a small part of Zurich city’s old town called Niederdorf. This area offers a variety of buildings with visually distinct features, which enabled us to craft unique virtual characters. Furthermore, many buildings in Niederdorf offer a rich history of Zurich’s past. Users can then visit buildings in the city of Zurich that are supported by our application and experience our talking house functionalities.
As a first step, we manually selected the buildings by visiting the areas in Niederdorf and researching the history of the place. For our final application, we selected 11 buildings and one monument to augment with virtual characters. In Fig. 1, the left images gives an overview of some of the selected buildings. Once the buildings are identified, the next step is writing engaging character dialogues using the history of the buildings and their surrounding area. These dialogues vary in content ranging from a conversation between characters and historical facts to jokes and poems. Afterwards, we recorded the dialogues with the help of two professional voice actors. We then created a visual look and animation for each character by considering the geometry of the buildings and the spoken dialogues. Our characters are designed using simple shapes like squares and circles to better blend the virtual elements with the real world buildings. For example, the window of a building is used to create animated eyes for a character. Finally, we enhanced the dialogues by adding background music and sound effects to make the AR experience more immersive. We also share the videosFootnote 1 of our virtual characters talking to a user about the building.
3 AR Technology and Localization
To achieve a seamless experience, the AR virtual content objects must be properly rendered and aligned with the buildings (onto the walls) in the real world [2]. Augmenting buildings and monuments with virtual characters requires precise techniques to estimate the position and rotation of the user’s device. This is one the main aspects to ensure an authentic AR experience since otherwise the virtual content would be misaligned with the real world. For our application, we use the Immersal SDKFootnote 2 which allowed us to spatially map real-world locations and then augment them with digital content. As an initial step, we went to our selected locations of the buildings to create a spatial map of the area. This is done by taking numerous pictures of the streets and buildings from multiple viewpoints. Then, each set of images is uploaded to the Immersal Cloud Service which generates a 3D point cloud and a textured mesh (see Fig. 2). This textured mesh can then be used inside the Unity editor as a point of reference to align the virtual characters with their corresponding buildings. Aligning the character manually by drag and drop with the mouse is too imprecise. We therefore used the Kabsch algorithmFootnote 3 to automate the alignment process by selecting reference points between the character assets and the textured mesh. When a user visits one of the augmented buildings, they first have to move the mobile device across the surrounding buildings. Immersal then captures the current camera frame, computes its point cloud and tries to find a match with one of the stored point clouds. If there is a match, the digital character is shown correctly aligned with the building.
Implementation Details: We developed our AR application using the Unity engine which is a popular development platform to create video games and 3D applications. In order to optimize the memory usage and performance of our mobile application, we split the animation for each character into smaller parts (face, mouth, eyes etc.) and imported each of them as a sprite sheet into Unity. Each sprite sheet contains a sequence of images which are then combined by our app to create the final animation. Our source codeFootnote 4 is publicly available to extend our application for other cities. We also released our app as a beta version in the Google Play StoreFootnote 5 supporting the experience for the Niederdorf area of Zurich.
4 The AR Experience: The Square and the Hunt
In this work, we present two unique experiences that we designed to interact with the users. The first one is called as ‘The Square’ and in this experience the character will tell the story about the building’s history and other anecdotes about the building’s surroundings. The user can remain in one location and interact with different virtual characters appearing in front of surrounding buildings. The second experience is called as ‘The Hunt’ which is similar to a scavenger hunt where the user has to find buildings using the hints given by virtual characters. The experience will start with a virtual ghost named as ‘Stussi’ who appears to provides the hint to find the buildings. The user has to find these buildings based on the given hints and collect rewards like ‘rings’ at each building.
5 Experimental Evaluation
We conducted a user study to test and evaluate the usability of our application. We asked the participants to experience both the Square and the Hunt. In total, 9 people participated in our experiments. They were aged between 18 to 35 and the majority were either undergraduates, research students or working employees. In our experiments all our participants had to complete both the square and the hunt experiences. At the end, to evaluate the perceived usefulness of our application users filled out a questionnaire. We used System Usability Scale (SUS) [1], a simple and well established tool which consists of 10 statements for measuring usability. Our user study demonstrated that our prototype has a high usability and was generally perceived as pleasant and enjoyable experience. For perceived user enjoyment, we asked users about how much they enjoyed both the experiences. We found that 70\(\%\) of users strongly enjoyed the hunt and 100\(\%\) enjoyed the square. All participants of the user study agreed that our application was enjoyable to use when visiting the city buildings. 70\(\%\) users strongly enjoyed the hunt and 100\(\%\) enjoyed the square. Users were also interested in using such an application for other attractions in Zurich or other cities. Finally when asked about which experience to enhance further 72\(\%\) users wanted to see more advancements towards the hunt experience.
6 Conclusion and Future Works
In this paper, we have presented our AR system, a mobile AR application which provides users an unique experience by transforming building into a visible character. Our application shows a live character talking to a user when he visits a famous building in a city. Such techniques will have a great potential to enhance a user experience and in our future work, we will explore towards the direction of adding personalized advertisements on the buildings. We aim to also design machine learning models for automatic content creation such as stories, news or jokes where the character can converse or engage with users with up to date information.
References
Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system usability scale. Intl. J. Hum.-Comput. Interact. 24(6), 574–594 (2008)
Wiethüchter, R., Kalloori, S., Chalumattu, R.: Fast content placement and alignment in 3d scenes. In: The International FLAIRS Conference Proceedings, vol. 35 (2022)
Yang, F., Kalloori, S., Chalumattu, R., Gross, M.: Personalized information retrieval for touristic attractions in augmented reality. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1613–1616 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2023 The Author(s)
About this paper
Cite this paper
Kalloori, S., Chalumattu, R., Chalet, F., Zimper, M., Klingler, S., Gross, M. (2023). Talking Houses: Transforming Touristic Buildings into Intelligent Characters in Augmented Reality. In: Ferrer-Rosell, B., Massimo, D., Berezina, K. (eds) Information and Communication Technologies in Tourism 2023. ENTER 2023. Springer Proceedings in Business and Economics. Springer, Cham. https://doi.org/10.1007/978-3-031-25752-0_37
Download citation
DOI: https://doi.org/10.1007/978-3-031-25752-0_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25751-3
Online ISBN: 978-3-031-25752-0
eBook Packages: Business and ManagementBusiness and Management (R0)