The Experience of a Lifetime Interactive Digital Experience Beyond the Screen By Yufei Yao (Michelle) BA, Communication University of China, 2017 Yufei Yao (Michelle) BA, Communication University of China, 2017 Supervisor: Dr. Garnet Hertz A CRITICAL AND PROCESS DOCUMENTATION PAPER SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF DESIGN EMILY CARR UNIVERSITY OF ART + DESIGN, 2019 Yufei Yao © 2019 “Knowing how to find out what you don’t know. Be ready for change, aniticipate it. Having a willingness to experiment, fail, and try again. Knowing how to take an idea and make it real. Find your strengths and deepen them. Follow your passions and trust your gut.” - Red Burns, Founder of ITP, NYU (Interactive Telecommunication Program) Acknowledgements This thesis and two-year journey at Emily Carr University won’t happen without a lot of people. Thanks to the designers and founders at local studios: Stuart Ward from HFour Design, Jan Beringer from NGX Interactive, and Maryam Mobini and Mark Nazemi from to:Resonate for sharing their awesome works and talk through the challenges they have faced with me. Also, I want to thank the interactive designers and artists I have met in Vancouver and New York City: Jimmz, Tim, Yasu, Kerem, Yiyao, Es... your works and spirit always inspire and motivate me to move forward. I want to express my gratitude to my supervisor, Dr. Garnet Hertz, for the continuous support and providing me with critical feedback for my work throughout the last two years. Thank you for saving me no matter when I broke my electronics while soldering, when I had trouble with my academic writing, hesitating between work opportunities, or whenever I was in doubt. Also, I want to thank my internal reviewer, Dr. Maria Lantin, for introducing me to learn TouchDesigner and the community in Vancouver, which was a breakthrough for my thesis research. Thanks Haig Armen for pointing me the direction I want to pursue when I went the wrong track during my very first semester, and encouraging me for going to NYU ITP Camp 2018 when I hesitated, that was the best choice I have ever made in life so far. Thanks Christian Blyt for sharing the summer 2018 in Industrial Design studio with me, and for telling me to “be the person who’s different” when I doubt my works and myself. 1 Thanks to the faculty at Emily Carr for guiding me: Caylee Raber, Christopher Hethrington, Deb Shackleton, Eugenia Bertulis, Gillian Russell, Kate Armstrong, Katherine Gillieson, Keith Doyle, Laura Kozak, and Louise St. Pierre. And the amazing technicians at school for helping me make my interactive ideas come true, especially Bobbi Kozinuk, Tim Rolls and Yang Hong. Thank my employers and co-workers from Health Design Lab, A&K Robotics, Microsoft and NGX Interactive for the support and collaboration. Thank all my friends at Emily Carr and everyone in the Master of Design 2019 cohort for sharing this amazing two years with me: Haolin, Sherry... especially Maggie Mo Luo, for all the late nights and weekends working at school for our installation projects, we share this memory together. Finally, I want to thank my Mom and Dad, for the unconditional support for all the choices I made so far in life, and for always educating me to work hard to conquer the impossible since I was a young child. I love you more than I can tell. 2 Table of Contents 1 Acknowledgements 5 ABSTRACT 7 List of Figures 9 I. Introduction 9 1.1 Project Background: From the screens, beyond the screens 11 1.2 Research Goals & Inquiries 12 1.3 Methodology: A Practice-based Exploration 14 II. Theory-based research and case studies 14 2.1 Experimental Interactive Experience Design 14 2.1.1 Tangible Interaction: Interactivity embedded into the physical space and tangible objects 17 2.1.2 Interactive Technology: Hybrid Input and Out put in an interactive experience 21 2.1.3 Experience Design: Digital Experience Beyond the Screen 23 2.2 Case Studies 24 2.2.1 The Kenun Cloud (2017): An Interactive Branding Pop-up Exhibition 27 2.2.2 The Virtual Aquarium (2017): Interactive marinethemed experiences for the BC Children’s Hospital 3 30 2.2.3 Universe of Water Particles on Au-delà des limites (2018): An artistic expression of the relationship between human and nature 33 2.2.4 DOME (2018): An Immersive Meditation Experience 35 2.2.5 Summary III. Research Approach and Practice-Based Research 37 37 3.1 Methodologies 37 3.1.1 Experience Prototyping 38 3.1.2 Reflection-In-Action 39 3.2 Exploratory Interactive Experience Design Prototypes 39 3.2.1 You Can’t Touch This (2018): An Interactive Offline Retail Experience 43 3.2.2 Tree of Life (2018): A “Magical” and Playful Painting 47 3.2.3 Let It Rain (2018) A Room-Scale Interactive Installation 49 3.2.4 Analysis and Findings 51 3.3 Final Thesis Project 51 3.3.1 The Creation Process and Project Objectives 55 3.3.2 User Feedback and Project Impact 62 3.3.3 Conclusion and Critical Reflections 62 3.4 Future Direction - Craft Experiences Through Narrative 63 3.4.1 2019 Grad Show Project - Directions of the Tide(2019) 66 VI. Conclusion 69 Bibliography 75 Appendix 4 ABSTRACT Keywords Creative Coding Electronic Art Experience Design Experience Prototyping Interactive Art New Media Art [Image right] Michelle Yao’s (The Author) Emily Carr University of Art+Design 2019 Grad Show 2019 Project Directions of the Tide | 我们终将改变潮水的方向 (2019), https://www.thisismichelleyao.com/direc- Screen-based digital experience design is blooming among the local businesses in Metro Vancouver along with the increased pervasiveness of information technologies, new digital products in contemporary society. However, there are significantly fewer cases and related businesses around tangible interactive digital experience in which tangible objects and physical spaces replace the screen as the site of interaction. This thesis project aims to explore the specialties of the tangible interactive experience compared to the digital experience on the screen or in the virtual space. Additionally, the author investigates how to leverage user experience design methodologies in the process of designing an experimental interactive experience. tions-of-the-tide In this practice-based exploration, the author prototyped four interactive digital experiences using different interactive technologies and tools tailored to different use case scenarios: 1. an interactive offline retail experience, 2. a “magical” and playful painting, 3. a room-scale interactive installation, and 4. an immersive meditation activity. These projects illustrate and explore the implementation of tangible interactions into digital experience design. During the development process, the author applied several user experience design methodologies in the projects – including field research, interviews, questionnaires, and design probes – to develop a workable framework designing tangible interactive experiences throughout the research project. The author aims to outline key implications of applying principles of user experience design to the field of tangible interactive environments. In the process, the author argues that tangible interactive design is indispensable in a successful and engaging digital experience, and thus worth investing in and exploring further in Vancouver’s marketplace. 5 6 List of Figures Figure 1: Lozano–Hemmer, R. (2001). Body Movies. [Media Installation] Rotterdam: Cultural Capital of Europe Festival. http:// www.lozano-hemmer.com/body_movies.php. Figure 2, 3: Yiyao Nie, Mitochondria, New Media Art Live Performance, 2018, https://www.yiyaonie.space/mitochondria. Used by permission of the artist. Figure 4, 5: ITP Winter show 2017, itp.nyu.edu. Used by permission of New York University Tisch School of the Arts. Figure 6: Jiayu Liu, The Side Valley, 2018, http://www.jiayuliu.studio/the-side-valley/. Used by permission of Jiayu Liu studio. Figure 7: Jiayu Liu, The Riverside, 2018, http://www.jiayuliu.studio/ the-riverside/. Used by permission of Jiayu Liu studio. Figure 8, 9, 10: HFour Design, The Kenun Cloud, 2017, https://hfour. ca/portfolio-item/asics-cloud/. Used by permission of Hfour Studio. Figure 11, 12: NGX Interactive, The Virtual Aquarium, 2017. (https://ngxinteractive.com/work/bc-childrens-hospital-virtual-aquarium-and-mobile-website/) and (https://www.bcchf.ca/finding -calm-emergency-department). Used by permission of the NGX Interactive. Figure 13, 14, 15, 16: teamLab, Universe of Water Particles on Au-delà des limites, 2018. Figure 17, 18: Poster for DOME, 2018, https://goo.gl/etVbHG. Used by permission of the Mark Nazemi. Figure 19: Overall effect of the installation You Can’t Touch This, 2018. Figure 20, 21: The prototyping tools and the detail of You Can’t Touch This, 2018. 7 Figure 22, 23, 24, 25: Tree of Life (2018) during studio critic showcase at Emily Carr University, 2018. Figure 26, 27: Tree of Life (2018) version II was installed in the hallway during Emily Carr Studio Open Day 2018. Figure 28, 29, 30, 31: Let It Rain (2018) during studio critic in April 2018. Figure 32: Low-fidelity Concept Sketch on Paper of StarSpace (2018). Figure 33, 34: Static Computer Graphic Sketches of StarSpace (2018). Figure 35, 36, 37: Design Probe for the participants before the showcase of StarSpace (2018). Figure 38: Instruction Card of Day One in the Design Probe for StartSpace (2018). Figure 39, 40: Instruction Card of Project Feedback in the Design Probe for StarSpace (2018). Figure 41: Instruction Card of Day Two in the Design Probe for StarSpace (2018). Figure 42, 43, 44, 45: Documentation Photo of the Showcase of StarSpace (2018). Figure 46, 47: Infographic of the Overall Experience Trend of All Participants for StarSpace (2018). Figure 48, 49: Infographic of the Individual Experience Trend of All Participants for StarSpace (2018). Figure 50, 51: Details of the interactive projection on the floor of the project Diretions of the tide (2019) Figure 52, 53: Documentation photo of the live performance Directions of the tide (2019) Table 1: Analysis of three Experimental Experience Design Projects. Table 2: Data Interpretation criteria of StarSpace (2018) Design Probe. 8 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen I. Introduction 1.1 Project Background: From the screens, beyond the screens This thesis project explores the potential of interactive digital experience beyond the “screen”: the mobile applications on the LED screen of a mobile phone, or the websites on the LED screen of a monitor, for instance. Interactive digital experience, from the author’s perspective as a user experience designer in the information technology industry, is the experience of human-computer interactions while the user is trying to complete a task via a digital product or service. The place where the interactions happen and where the user experience is fostered is typically screen-oriented through mobile Apps or websites. In other words, the “screen” is the entity where the digital experience of user experience design is embedded. At the same time, it is clear that user experience involves much more than screens: Cooper, Robert and Cronin remind us in The Essentials of Interaction Design that “interaction design is also useful when creating physical products, exploring how a user might interact with it” (2007). Although interactive digital experiences that do not use screens are available – in projection-mapped museum installations or in smart speakers like the Google Home or Amazon Alexa, for example – few guidelines or case studies exist at a level that can be understood by average developers and interaction designers as for using tangible objects and space as the entity where the interactions live in. Based on this curiosity around interactivity, the theme of this research project shifted from utility-focused, screen-based digital 9 Yufei (Michelle) Yao | Emily Carr University of Art + Design product design to experimental interactive digital experience design. In this thesis, the overall principle of practice-based research is embracing and leveraging various digital technologies to prototype a series of experimental interactive experiences. All the projects focus on the artistic expression process and the creative interaction between people and the system rather than utility. During this two-year exploration, the project evolved away from screenbased design techniques and into more interactive physical technologies like embedded controllers, projection mapping, creative coding, and generative computer graphics to realize the creative side of interactivity through prototyping. [1] https://ngxinteractive.com/portfolio/ [2] teamLab is a multimedia studio founded in 2001 in Tokyo, Japan. (https://www.teamlab.art/about/) When transferring the focus of interactivity from usability to artistic expression and creativity using more diverse tools and technologies, there is considerable potential for higher user engagement and playful interaction. This insight is grounded in user feedback from thesis practice-based explorations, which contain three experimental projects and one major thesis project. It is also exemplified by the success of the existing experimental digital experience in the marketplace. In this thesis, four interactive projects by NGX Interactive [1], teamLab [2] and other creative media art studios are used as case studies. These works are successful because of their widely-spread reputations or business revenue. They are the case studies, which inspire the studio projects in ways of merging the practical and utilitarian aspects in Experience Design and the creative and aesthetic aspects of New Media Art. To sum up, this thesis topic is situated at the intersection of New Media Art, Experience Design and Interactive Technology. As a design professional and practitioner, the author believes that experimental interactive design will be the future of digital experience and is worth investigating. Additionally, the goal of this thesis 10 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen is to enlighten the general public on the unique experience which experimental interactive design can offer, as well as proposing its new opportunities for implementation in Vancouver’s marketplace in the future. 1.2 Research Goals & Inquiries When the entity in which the interactions are embedded shifted from the “screen” to tangible objects or spaces, the methodologies and design process also changed with it. From the process of user experience design, a series of methodologies are often used in the process of designing an interactive experience. These user experience design methodologies are the tools to help designers to analyze and solve a complex design problem along the user’s journey interacting with the digital system (Martin and Hanington, 2012). However, this thesis exploration focuses on the experimental and creative side of digital experience design, which is different from the usual goal of traditional digital experience design. This thesis exploration seeks to establish a framework for leveraging user experience design methodologies in experimental experience design projects. The peculiarity of the experimental interactive projects in the research scope can be defined as “more artistic pieces rather than a tool for a work setting” (Horn, Svendsen and Madsen, 2001, p.390) and “not designed for any particular profession, but people in general” (2001, p.390). Although the goal and focus for the interactive experience design has shifted from traditional usability-driven experience design, the overall objectives of this thesis share an overarching goal with experience design: “to reduce the negative aspects (e.g. frustration, annoyance) of the user experience while enhancing the positive ones (e.g. enjoyment, engagement)” (Preece, Rogers, Sharp, 2015, p2). Building upon these objectives, 11 Yufei (Michelle) Yao | Emily Carr University of Art + Design this thesis seeks the answer to the question of: How can we leverage user experience design methodologies into experimental experience design, where the focus is no longer on usability but on creative expression through interactions? 1.3 Methodology: A Practice-based Exploration The research methodology developed for this thesis is primarily based on a practice-based exploration also known as research-through-design (Zimmerman, Forlizzi and Evenson, 2007). Throughout two-years of studio work at ECU, all projects in this thesis followed a reflective design practice focused on design projects themselves and their use as the dominant focus in this thesis work. By pursuing this process, various design methods, theories, and tools were selected to structure and forecast the consequences of the interventions (Braun, 2014, p.17). The projects in this thesis have been influenced by both academic theorists and industry pioneers. New Media artists like Rafael Lozano-Hemmer[3], Casey Reas[4] and also experimental student works from RCA IED[5] and NYU ITP[6] inform the project from the perspective of artistic expression. The works by pioneers in the industry like Moment Factory[7] and teamLab, as well as works from local professional interactive art studios in Vancouver like HFour Design[8], Tangible Interaction[9], NGX Interactive and to:Resonate[10] enlightened the project on how to design a digital experience tailoring to the specific use case. Gathering insights from both aspects of artistic expression and design usability, the research started by prototyping a series of experimental interactive experiences, exploring methodology that combines artistic expression and usability. [3] Rafael Lozano-Hemmer is a Mexican-Canadian electronic artist who works with ideas from architecture, technological theatre and performance.(http://www.lozano-hemmer. com) [4] Casey Reas is an American artist whose conceptual, procedural and minimal artworks explore ideas through the contemporary lens of software, best known for having created, with Ben Fry, the Processing programming language. (www.reas.com) [5] IED Information Experience Design program at Royal College of Art is a graduate program where students design experiences to communicate information, creating installations and interventions using all senses, modes and media, including sound and moving image. (https://goo.gl/GXbrLH) [6] ITP (Interactive Telecommunication Program) at New York University Tisch School of the Arts is a graduate department described as an art school for engineers or an engineering school for artists. (https://tisch.nyu.edu/itp/) [7] Moment Factory is a multimedia studio from Montreal, Canada. It specializes in video, lighting, architecture, sound and special effects for immersive environments. (https:// momentfactory.com/home) [8] https://hfour.ca/ [9] http://www.tangibleinteraction.com/ [10] https://toresonate.com/ 12 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen This thesis exploration contains four different interactive experience design projects, including three experimental projects, You Can’t Touch This (2018), Tree of Life (2018), and Let it Rain (2018), and one final thesis project, StarSpace (2018). Each of these interactive projects was designed for a particular use case scenario, while sharing the same design principle: crafting the digital experience through playful tangible interactions. The three experimental projects lie in the phase of learning and prototyping in this thesis exploration, while the final thesis project is the learning outcome of integrating the insights and knowledge learnt from the previous phase. The creation of three experimental interactive projects followed the design process of “reflective thinking” (Donald A, 1983), which requires the designer to think critically on their works to identify problems in current work, and provide the objectives to iterate and optimize future designs. Based on this process, the three experimental projects were built upon one another and were designed to improve the user experience of the previous project based on user feedback. The user experience of each project was optimized through the improved technical skills of mine and integrated user experience design methodologies. The final thesis project, StarSpace (2018), was the outcome of this thesis exploration of leveraging user experience design and New Media Art together, and approached the goal of this research: to create an interactive digital experience “of a lifetime” - an experience that maximizes the enjoyment and user engagement, while minimizing the frustration and annoyance. 13 Yufei (Michelle) Yao | Emily Carr University of Art + Design II. Theory-based research and case studies 2.1 Experimental Interactive Experience Design Designing an experimental interactive experience is a cross-disciplinary exploration. Being in the intersection of New Media Art, Experience Design and Interactive Technology, an experimental interactive experience absorbs the essence of these disciplines. An engaging and memorable experience combines the flexible and creative art expression of New Media Art, logical and fluid information exchange between the user and the system of Experience Design, and fun interactivity of Interactive Technology. Situated at the intersection of those three general disciplines, this thesis exploration emphasizes a narrower scope of designing the experience through tangible interactions and hybrid interactive technologies. Inspired by existing works from the pioneers in this field, we can grasp the essence of integrating tangible interaction and various interactive technologies in the creation of an engaging and memorable digital experience. 2.1.1 Tangible Interaction: Interactivity embedded into the physical space and tangible objects Interactivity is the core of this thesis research scope which allows people to become “creators” rather than “viewers”. According to Menenghini, Palma and Taylor (2004), the traditional medium of art as a tool of communication, such as cinema, print or broadcast media, involves a linear presentation with a clearly defined sepa14 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Image Right [Figure 1] Rafael Lozano-Hemmer, Body Movies, Relational Architecture 6”, 2001. Schouwburgplein, V2 Cultural Capital of Europe, Rotterdam, The Netherlands. [11] http://www.lozano-hemmer.com/body_ movies.php ration between the sender of the message and the receiver of the message. Interactive media, by contrast, involves a blurring of the line between author and audience in which the audience, to a certain extent, participates in the creation of the message itself (p.3). In this thesis exploration, the focus transferred from interactivity on screen-based products to objects and spaces. In other words, the objects and spaces which host the interaction become the “tangible user interface” (TUI). This expansion of designing interactions in tangible spaces with objects differs from graphical user interfaces (or GUI). This move away from screen-based products enables designers to address larger design opportunities and the ability to integrate approaches from different disciplines (Hornecker and Buur, 2006, p.437). The action of designing the experience through “TUI” can also be referred to as crafting “tangible interaction”. Hornecker and Buur introduced a framework which divides tangible interaction into the subcategories of Tangible Manipulation, Spatial Interaction, Embodied Facilitation, and Expressive Representation (p.437). This thesis investigates tangible interaction in the scope of Spatial Interaction and Embodied Facilitation, where the former involves full-body interaction and consists of performative action, while the latter provides insights relevant for the broader research area of ‘embodied interaction’ (Hornecker and Buur, 2006, p.440). These two concepts were applied in the design projects of this thesis by creating interactive spatial experiences using real-time rendered computer graphics expanded with projection mapping techniques. Tangible interaction with the use of multiple objects and spaces as the place where the interactions take place can bring a strong foundation in multidimensional artistic expression. In Rafael Lozano-Hemmer’s work Body Movies (2001) [11] (cf., Figure 1), the artist 15 Yufei (Michelle) Yao | Emily Carr University of Art + Design Figure 1 has been removed due to copyright restrictions. The information removed is Relational Artchitecture Body Movies by Rafael Lozano-Hemmer, 2001. Available at http://www. lozano-hemmer.com/body_movies.php 16 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Image Right (Up - Down) [Figure 2 & 3] Yiyao Nie, Mitochondria, 2018, https://www.yiyaonie.space/mitochondria. Used by permission of the Artist. [12]https://www.yiyaonie.space/mitochondria created a non-linear dialogue and experience with the participants by projecting participants’ movements and live streaming them onto a building’s surface in an urban space. This implementation of tangible interaction enabled a direct conversation through kinetic and visual feedback between the individual and the public, which is hard to achieve in screen-based interactions. Similarly, tangible interaction can expand the scope of design practice in experience design to a cross-disciplinary exploration. In the digital choreography performance work Mitochondria (2018) [12] by NYU ITP’s student Yiyao Nie (cf., Figure 2 & 3), interactivity was brought from computer-based graphics to dance performances. The real-time rendered graphics projected on the stage changed according to dancers’ dynamic body movement. Tangible interaction between the body movement of the human and tangible space broke the original space-time view of conventional artistic creation. 2.1.2 Interactive Technology: Hybrid Input and Output in an interactive experience Interactive technology is the indispensable element in this thesis as it is the tool of creating hybrid input and output processes inside the experience. The process of designing an interactive experience can be broken down into two simple steps as the process of input and output (I/O). The I/O makes an interactive experience, a reciprocity of actions, which requires a cycle of action whereby both (or all) participants actively contribute as opposed to merely responding to the experience (Rauscher, 2010, p.22). More and more designers and artists are using microcontrollers and sensors when building interactive projects. These technological advances 17 Yufei (Michelle) Yao | Emily Carr University of Art + Design 18 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Up - Down) [Figure 4 & 5] ITP Winter show 2017, https:// itp.nyu.edu. Used by permission of New York University Tisch School of The Arts. allow us to receive inputs, calculate processes, and control outputs, which are all the basic components required for interactivity (Rauscher, 2010, p.23). In the projects for this thesis, various technologies were used to add more possibilities of fun and joy, including the use of different biosensors, microcontrollers, cameras and projectors. The use of hybrid interactive technologies can bring more possibility into an interactive experience. By involving the participants and users through different biosensors, cameras and other technologies, the experience design is “no longer solely understood as an expression of the artist’s inner creativity but instead becomes an intrinsically dynamic process” (Sommerer and Mignonneau, 1999, p.166). The Interactive Telecommunication Program (ITP) (cf., Figure 4 & 5) at New York University is a two-year graduate program with students from diverse backgrounds, whose mission is “imaginative use of communications technologies — how they might augment, improve, and bring delight and art into people’s lives” (ITP, 2019). The student works of NYU ITP enlights us how sensors and new cutting-edge technologies can bring ideas around interactivity into reality. Following the Do-It-Yourself culture around electronic arts (Hertz, 2015, p.5), even designers who have no professional engineering background can leverage interactive technologies. And in this case, electronic arts and interactive technologies allows designers to explore the fun and attractiveness they can offer in the input process of the creation of an interactive experience. In this thesis, projection mapping and computer-generated graphics are the main techniques being used in the output process within an interactive experience design. Large-scale computer graphics are the keys to creating an immersive experience in a tangible 19 Yufei (Michelle) Yao | Emily Carr University of Art + Design 20 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Up - Down) [Figure 6] Jiayu Liu, The Side Valley, 2018, http://www.jiayuliu.studio/the-side-valley. Used by permission of Jiayu Liu studio. [Figure 7] Jiayu Liu, The Riverside, 2018, http://www.jiayuliu.studio/the-riverside/. Used by permission of Jiayu Liu studio. [13]http://www.jiayuliu.studio/the-sidevalley/ [14]http://www.jiayuliu.studio/the-riverside/ space. Immersion is as much a result of the ability of voice to guide, capture, and hold users’ attention as it is the visuals, audio, or other sensory displays that divert our attention toward the experience (Shedroff, 2001, p.285). It is easier to achieve an immersive experience in the context of digital art and design since it can break the barrier of materialized art. In the work The Side Valley(2018) [13] (cf., Figure 6) and The Riverside (2018) [14] (cf., Figure 7) by RCA IED alumni Jiayu Liu, the use of projection not only expanded the scale of the works but created an immersive experience. The computer-based graphics the artist designed as the streaming water flow abstracted the natural scene, creating a sense of surreal yet breathtaking visual experience. 2.1.3 Experience Design: Digital Experience Beyond the Screens Experience Design, according to the definition by Aarts and Marzano (2003, p46) is the practice of designing products, processes, services, events, omnichannel journeys, and environments with a focus placed on the quality of the user experience and culturally relevant solutions. Although we can call nearly every event or process we encounter in our life an “experience”, Experience Design, as a subject in the Design discipline, researches the designable, knowable and reproducible elements that contribute to superior experiences (Shedroff, 2009, p2). Enlightened by the concepts and ideations of Experience Design, all four interactive experience prototypes in this thesis were designed to tailor to specific themes or scenarios. Aided by the design tools and methodologies in Experience Design and HCI (Human-Computer Interaction), every prototype aimed to provide users with a smooth and fluid experience. The experimental interactive works in the thesis also inherit the professional design practice of the author as a user experience designer: to create not perfect, but better experiences through iterations and optimizations. 21 Yufei (Michelle) Yao | Emily Carr University of Art + Design 22 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen 2.2 Case Studies The following case studies show the inspirations and concepts of this thesis.They include four unique interactive digital experience design works: The Kenun Cloud (2017) - a pop-up branding campaign designed by HFour Design; The Virtual Aquarium (2017) - an interactive installation in the BC Children’s Hospital waiting room by NGX Interactive; Universe of Water Particles on Au-delà des limites (2018) - a room-scale interactive exhibition by teamLab; and DOME (2018) - an immersive meditation experience. Those four works have different themes and styles and were each designed for a particular use case scenario. And they were then borrowed later as the inspiration for the four exploratory interactive projects in this thesis. The creative processes of each project are highlighted. The process and method of designing an experimental interactive experience varies from designing a usability-driven experience. However, both types of experience design share the same goal to “explore common characteristics in all media that make experiences successful by encompassing multiple senses and requirements” (2001, Shedroff, p.13). The shared perspectives underlying the design process of those two types of experience design is a key research question in this thesis. The projects in these case studies and my design projects share the same goal: crafting an experience through playful tangible interactions and hybrid interactive technologies. They all use tangible objects or spaces rather than the screen as the place where interactions take place, and apply various cutting-edge digital technologies rather than sticking to only one medium. By employing a combination of interactive technologies, including pre-rendered video 23 Yufei (Michelle) Yao | Emily Carr University of Art + Design and animation, generative computer graphics, projection mapping, and creative electronics they make an engaging and playful experience come to life. [15] https://www.asics.com/us/en-us/gelkenun/c/gel-kenun 2.2.1 The Kenun Cloud (2017): An Interactive Branding Pop-up Exhibition The Kenun Cloud (2017) is an multi-sensory immersive interactive experience created by the Vancouver-based Multimedia Art studio, Hfour Design. The whole experience contained the main installation – a twenty-four inch wide interactive wall made by “kinetic cloud” and was installed as a pop-up exhibition in SoHo, New York City. Upon walking towards the installation wall, the clouds became “alive”. Glowing lights and the synthetic movements of the “clouds” provided participants with a sense of wonder and fun. Through this playful experience, Hfour Design presented the brand idea of ASICS’ Gel-Kenun[15] shoe collection, “as comfortable as walking in the clouds”, in a creative way. This experimental work made the participants immersed in the whole experience through interactivity. The Kenun Cloud (2017) (cf. Figure 8 & 9 & 10) created a dynamic translation from the visual and lightning effects of the projection design on a tangible installation to human body movements. As participants walked in front of the wall, the “cloud” on the wall became “alive”: behind the cotton and synthetic cloud structure, lighting made the wall glow from the inside; in front of the wall, there was a projector which brought extra lighting texture from the activation. Both layers became dynamic when people were in front of the wall, creating a memorable spatial experience for all the participants inside the exhibition. This experience fits in the sub-category Spatial Interaction of tangible interaction, proposed by Hornecker and Buur (2006, p.440), which is “embedded, 24 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 8 & Figure 9 & Figure 10] HFour Design, The Kenun Cloud, 2017, https://hfour. ca/portfolio-item/asics-cloud/. [16] Documentary: https://goo.gl/RRz65T [17] Quoted from the interview by www. complex.com (https://goo.gl/RRz65T). [18] Quoted from the interview by www. complex.com (https://goo.gl/RRz65T). [19] Data from: https://hfour.ca/portfolio-item/asics-cloud/ [20] Vans is an American manufacturer of skateboarding shoes and related apparel, based in Santa Ana, California and owned by VF Corporation. (www.vans.ca) taking up or being situated in spaces, and users needing to move in real space when interacting”. Spatial Interaction provides a reciprocal situation where seeing implies being seen (2006, p.441). In the The Kenun Cloud experience, the cloud installation and the exhibition space were no longer lifeless objects. The presence of humans made this work complete. According to the documentary of its creative process, The Kenun Cloud (2017) experience didn’t follow the traditional user experience methodologies and research process [16]. The project was built with a design objective based on a use case scenario, which is key to designing an experience. According to Stuart Ward, the co-founder of HFour Design, one of the goals for The Kenun Cloud (2017) was to “create a sense of the unperceivable or unknowable, or something that doesn’t quite get processed in the brain in a literal way”. [17] This digital experience aim to provide people the feeling that transcends the everyday sensations in their lives [18], which matches seamlessly with the concept and goal in all the studio works in this thesis. The Kenun Cloud (2017) showed an alternative way of creating a commercial branding campaign. The goal was to attract potential customers’ engagement for the brand’s concept or product, and Hfour Design reached this goal successfully. The exclusive launch event brought more than a hundred and fifty influencers, and many others, which brought high engagement on social media. Throughout the month, thousands of people came through to try on shoes and experience The Kenun Cloud (2017). [19] It was not a utility digital product or tool, but succeeded through playful interactions and a visually appealing digital experience. The idea of expressing brand identity through Spatial Interaction informed the creation of the project You Can’t Touch This (2018), an interactive branding campaign for Vans which is documented in the next chapter in this thesis. [20] 25 Yufei (Michelle) Yao | Emily Carr University of Art + Design Figure 8 has been removed due to copyright restrictions. The information removed is interactive installation The Kenun Cloud by HFour Studio, 2017. Available at https://hfour.ca/portfolio-item/asics-cloud/ Figure 9 has been removed due to copyright restrictions. The information removed is interactive installation The Kenun Cloud by HFour Studio, 2017. Available at https://hfour. ca/portfolio-item/asics-cloud/ Figure 10 has been removed due to copyright restrictions. The information removed is interactive installation The Kenun Cloud by HFour Studio, 2017. Available at https:// hfour.ca/portfolio-item/asics-cloud/ 26 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Up - Down) [Figure 11 & Figure 12] NGX Interactive, The Virtual Aquarium, 2017. (https://ngxinteractive.com/work/bc-childrens-hospital-virtual-aquarium-and-mobile-website/) and (https://www.bcchf.ca/finding -calm-emergency-department). Used by permission of NGX Interactive. [21] “Summit International Award – Emerging Media – Innovator Award (Silver)” and “Vancouver UX Awards - UX for Good – Finalist – Gesture Virtual Aquarium”. Data from (https://goo.gl/BPqmp8) 2.2.2 The Virtual Aquarium (2017): Interactive marine-themed experiences for the BC Children’s Hospital The Virtual Aquarium (2017), is an award-winning [21] marine-themed interactive experience designed by NGX Interactive, which was designed to distract, entertain and educate young patients and their parents while they await medical treatment at BC Children’s Hospital. This project explores implementing experimental interactive experience design in the public space. By introducing interactivity to the public space, and handing over interactivity to the audience, The Virtual Aquarium (2017) transformed the original experience in the hospital waiting room. The Virtual Aquarium (2017) soothes spirits and banishes the boredom that patients typically experience in waiting rooms. The computer graphics of various animated marine creatures in this installation work are interactive upon the changes of children’s different gestures and positions. The installation is built to illustrate a story of a sea turtle navigating its way in the Pacific Northwest. Embodied Facilitation, one of the subcategories of tangible interaction, was implemented in this project as it built the configuration of physical space and objects, and the representation was built on users’ experience (Hornecker and Buur, 2006, p.441). Through showcasing the wonders of the ocean, the installation lessened the tension of children. From the visual design perspective, the theme of marine life presented a calming yet lively digital experience to the audience (cf. Figure 11 & 12). On the one hand, a natural source and the presence of a natural element has been proven by rigorous scientific empirical data that it could have a positive impact on calming people and reducing stress (Browning et al., 2014, p.9). On the other hand, in this research topic, digital art has the advantage of enlarging the 27 Yufei (Michelle) Yao | Emily Carr University of Art + Design 28 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen [22] From project interview session with Michelle Yao (the author) and NGX Interactive, June 2019. visual effects through the use of diverse interactive technologies. In this installation, the use of a large scale screen and projection brought users a sense of calm and peace. According to Jan Beringer, the Experience Design Lead at NGX Interactive, the creation of The Virtual Aquarium followed a human-centred design process, including co-creation sessions with children and their parents and user-testing sessions for early prototypes. [22] These user experience design methods helped NGX Interactive to define the visual design style and clarify what works the best for the target audience, thus making The Virtual Aquarium (2017) a memorable and human-centred digital experience. From the interaction design perspective, The Virtual Aquarium (2017) grasped the objectives of designing a mindful interaction for users’ needs. And it is the part which has the potential of implementing user experience design methodologies and ideations into the creative process of experimental works. The interaction design in this project used the movement of the children’s body as the “input”, while the graphics of marine animals’ motions changes as the “output”. In other words, this work was co-created by the children who interact with the marine life on the installation, and it could be seen as an act of “creating” the art together. According to the studies by Bell and Robbins, a simple act of creating a work of art can produce dramatic reductions in a negative mood, and these reductions can be attributed specifically to the production of art rather than to its viewing (2007, p.75). The physical movement of the user enlarged the interaction between human and the digital device, which added up to an embodied digital experience for the users. This idea inspired the studio project Tree of Life (2018), which also explored an embodied interactive experience in the public space through users’ actions. 29 Yufei (Michelle) Yao | Emily Carr University of Art + Design 2.2.3 Universe of Water Particles on Au-delà des limites (2018): An artistic expression of the relationship between human and nature Universe of Water Particles on Au-delà des limites (2018) (a.k.a UWPA) is an interactive digital installation by teamLab. An enormous waterfall cascades down throughout the space of “Au-delà des limites”[23] as it transcends the boundaries and sometimes affects other works. Its visual design style, the “linear” water flow, is one of the most famous “teamLab style” computer graphics (cf. Figure 13 - 16). This work explores a new relationship between humans and nature, and between oneself and the world through digital art, and also inspired my experimental work Let it Rain (2018). [23] Literally translated from French as ‘Beyond the Limit’. [24] Random International, The Rain Room, 2012 (https://goo.gl/Q9cdBq). UWPA (2018) is an artistic installation that fits in the scope of Spatial Interaction, as did the first case study, The Kenun Cloud (2017). But UWPA (2018) focused more on performative expression rather than full-body movement. A waterfall and flowers were the theme of the visual design, creating a beautiful digital scene of nature. Water is represented by calculating the interactions between a continuous flow of hundreds of thousands of water particles. Then, lines were drawn in relation to the behaviour of the water particles. The accumulation of these lines expressed the waterfall. The cumulation of lines that represented the work was then “flattened” in line with what teamLab defines as a “ultrasubjective” space. The interaction design in this work followed the theme of teamLab: exploring the new relationship between humans and nature through digital technology, which is another aspect this research is longing to explore. This concept of creating an alternative experience in the digital world from real life also was widely applied in the creation of interactive experiences, such as The Rain Room (2012) by Random International [24], which reversed the common sense in 30 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 13 & Figure 14 & Figure 15 & Figure 16] teamLab, Universe of Water Particles on Au-delà des limites, 2018. https://www. teamlab.art/e/lavillette real life which people will get wet when walking in the rain. Similarly, in the work UWPA(2018), teamLab reversed the situation in the tangible world in which humans cannot change the direction of the waterfall. The presence of participants in this digital experience become one of the elements in the waterfall scene: rocks. When people stood on or touched the waterfall, they obstructed the flow of water like a rock and the flow of water changes. The flow of water continued to transform due to the interaction of people. Previous visual states could never be replicated, and were never the same. Similarly, this non-repetitive interaction was also explored in my project Let it Rain (2018). UWPA (2018) is an interactive installation which was not designed for a particular use case. However, its unique visual style and interaction ideas influenced all the design works in this thesis: from the visual design of the “waterfall” to the interaction idea of not simply providing an I/O relationship between the artwork and the participant, but instead letting the participant become a part of the interactive system, redefining the structure of the generated graphics. The creator of UWPA (2018), teamLab, describes themselves as an interdisciplinary group of “ultratechnologists whose collaborative practice seeks to navigate the confluence of art, science, technology, design and the natural world” (teamLab, 2017). They put forward that the presence of interactive digital technology has allowed art to liberate itself from the physical and transcend boundaries, thus enabling borderless creativity and imagination. This concept was one of the most influential ideas for this thesis exploration. 31 Yufei (Michelle) Yao | Emily Carr University of Art + Design Figure 13 has been removed due to copyright restrictions. The information removed is interactive installation Universe of Water Particles on Au-delà des limites by teamLab, 2018. Available at https://www.teamlab.art/e/lavillette Figure 14 has been removed due to copyright restrictions. The information removed is interactive installation Universe of Water Particles on Au-delà des limites by teamLab, 2018. Available at https://www.teamlab. art/e/lavillette Figure 15 has been removed due to copyright restrictions. The information removed is interactive installation Universe of Water Particles on Au-delà des limites by teamLab, 2018. Available at https://www.teamlab. art/e/lavillette Figure 16 has been removed due to copyright restrictions. The information removed is interactive installation Universe of Water Particles on Au-delà des limites by teamLab, 2018. Available at https://www.teamlab. art/e/lavillette 32 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 17 & Figure 18] Poster for DOME, 2018, https://goo.gl/etVbHG. Used by permission of Mark Nazemi. 2.2.4 DOME (2018): An Immersive Meditation Experience DOME (2018) is an immersive audio-visual meditation experience which enlightened the creation of project StarSpace (2018) in this thesis. During the meditation experience, participants were given clear instructions by a professional meditation instructor, also accompanied by background therapeutic music which was composed by Mark Nazemi, one of the co-founders of a Vancouver-based interactive studio, to:Resonate. The generative and abstract visual contents were audio-reactive to the background music, and were rendered and changed in real-time without delay, as well as time-lapsed, creating randomness and change. Large scale projection-mapped graphics and surround sound allowed participants to travel inward to their deepest inner space, and helped them concentrate on the mindfulness practice. This work revealed how to implement design methods and thinking into the creation of an experimental digital experience. The design objective of DOME (2018) was to create a peaceful and subtle interaction. This spatial immersive experience was built upon Embodied Facilitation (Hornecker and Buur, 2006, p.441), which was tailored to the user experience of mindfulness practice. Being inside of a planetarium and looking at the generative graphics on the 360-degree dome, DOME (2018) provided participants with a sense of the infinite, and of wonder. However, looking at this experience design critically, the visual design of DOME (2018) could be optimized. The visual design of DOME (2018) was generative abstract mesh or lines, which were expanding according to the rhythm of the meditation and background music. Nonetheless, during the meditation, the generative visuals were not calming, but a bit scary and stressful as it was “ap- 33 Yufei (Michelle) Yao | Emily Carr University of Art + Design 34 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen [25] https://www.facebook.com/ events/294699371131393/ proaching” the participants. There was even some screaming in the audience due to fear. The other unpleasant experience was the timeline of the event. As a meditation practice, forty-five minutes was too long; half of the audience started chatting or sighing after the first twenty minutes. Gathering those insights, these aspects are optimized in my final thesis project StarSpace (2018). From a user experience design perspective, this can be considered as an iterative design process, where the designer learned from the previous projects, and optimized the experience through iterations. Overall, the DOME (2018) immersive meditation experience was successful in its cinematography techniques, as well as its business revenue, which showed the potential of experimental interactive experience design. Upon entering the planetarium, people were lining up, and almost all the seats inside were full, which marked the success in its popularity among Vancouverites. According to the comments after the event and the feedback online[25], most participants commented on the event to be “stunning” and “unforgettable”. However, the DOME (2018) experience can be optimized in both visual design and interaction design; the visual design could be more calming and less abstract, and the generative graphics could be more “interactive” with the most important element of a guided meditation, the instructor’s voice, rather than the background soundtrack only. 2.2.5 Summary Four projects in the case studies provide a practical creative process of designing an experimental interactive experience. All the projects had different design objectives but also included shared ideation: interactive digital experience beyond the “screen”. On the one hand, they were designed for different use case scenarios: 35 Yufei (Michelle) Yao | Emily Carr University of Art + Design an offline branding campaign (The Kenun Could (2017)); an installation in the public space (The Virtual Aquarium (2017)); an artistic interactive exhibition (UWPA (2018)), and a meditation experience (DOME (2018)). On the other hand, these interactive works shared the common mechanism of implementing tangible interactions and applying hybrid interactive technologies, like all my studio projects in this thesis. They provided insights on how to leverage the creativity and flexible artistic side of experimental interaction design to achieve the goals in different experiences, as well as providing the technical directions in interactive experiences’ creation. Furthermore, these case studies also showed the potential of this thesis topic - experimental digital experience design, based on their success commercially or artistically. In all the projects, the result of high user engagement, and the creation of a shared immersive experience while keeping the spatial awareness of the participants, were the highlights as the theoretical answer to the thesis question of exploring the specialties of the tangible interactive experience compared to the digital experience on the screen or in the virtual space. The creation of these experimental experiences has the potential for business revenue and social innovation. than screen-based digital experience. However, this thesis needs to explore further in the areas where the case studies were not covered. From the four case studies, the thesis question of how to leverage user experience design methodologies in the process of designing an experimental interactive experience was not able to be answered clearly. Based on the enlightenment and theoretical knowledge gained from the case studies, I will further investigate this part in the next chapter. 36 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen [26] IDEO is an international design and consulting firm founded in Palo Alto, California, in 1991. (https://www.ideo.com/) III. Research Approach and Practice-based research This chapter draws attention to the practice-based creation in the thesis exploration. Starting from works by industry pioneers through case studies, this practice-based research explores the thesis question of how to leverage UX design methodologies in the process of designing an experimental interactive experience. Four experimental interactive experiences were designed and prototyped. Three of them were exploratory projects which were the “learning” and “seeking” phases of the research, and one final project was the “learning outcome”. Each project embedded tangible interactions into the creation, and leveraged various interactive technologies. They all followed the overall methodologies of “experience prototyping” (Buchenau and Suri, 2000) and “reflection-in-action and practice” (Braun, 2011). 3.1 Methodologies 3.1.1 Experience Prototyping “Experience prototyping” is a methodology and ideation which was put forward by designers Marion Buchenau and Jane Fulton Suri at IDEO[26]. The goal of experience prototyping is to test the key activity of the experience process (Buchenau and Suri, 2000, p.424). Buchenau and Suri divided the process of experience prototyping into three stages: understanding existing experiences, exploring design ideas and communicating design concepts (2000, p.424). While inheriting the ideation of prototyping from UX design, experience prototyping provides a workable framework for designing a test- 37 Yufei (Michelle) Yao | Emily Carr University of Art + Design able prototype for different projects, and using that to simulate and foresee the outcome of an experience. All four interactive works in this chapter took this framework as the principle of designing, testing and iterating. Although they were not in the scope of traditional usability-driven UX design, experience prototyping bridged the gap of UX design methodologies and the experimental interactive design development. [23] Literally translated from French as ‘Beyond the Limit’. [24] Random International, The Rain Room, 2012 (https://goo.gl/Q9cdBq). 3.1.2 Reflection-In-Action Another overarching methodology in this practice-based exploration is Reflection-In-Action thinking. Donald Schön (1983) stated in his work The Reflective Practitioner that practice is an exploration in which the practitioner seeks to come to terms with a given creative task. Based on the insights from reflection-in-action practice, the designer optimizes each project individually through iterations, and uses what they have learnt in the creation of the next project. According to Schön, reflection-in-action is not driven by the unexpected but by the desire to learn from experience: it is a discipline rather than a necessity for further action (1983). In this thesis, design practice and interactive project development dominated the whole research process, and it was the constant creation, reflection and iteration that made the research more solid and profound. In the creation of all four interactive projects, reflection-in-action was applied in the design practice, representing the process of thinking, making and reflecting critically on each project’s outcome. 38 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 19] Overall effect of the installation You Can’t Touch this, 2018 [Figure 20 & Figure 21] The prototyping tools and the detail of You Can’t Touch This, 2018. [27] Makey-Makey is an electronic board sold by Sparkfun, which turns any conductive objects into “keys” on a keyboard. (https://goo.gl/g9TmWH) 3.2 Exploratory Interactive Experience Design Prototypes This part of practice-based exploration contains three exploratory interactive experience design prototypes: You Can’t Touch This (2018), Tree of Life (2018) and Let it Rain (2018). These three experience designs were inspired by the projects described in the case studies chapter. The creation of these three projects is the learning process of implementing UX design methods through prototyping experimental interactive projects. Although each project was not a well-polished product like those in the case studies chapter, they were designed and guided by the ideation of experience prototyping and iterated through reflection-in-action process. The hands-on design practice and prototyping brought this thesis exploration to a higher level of experience design, where the theoretical framework of experimental interactive design became reality. 3.2.1 You Can’t Touch This (2018): An Interactive Offline Retail Experience You Can’t Touch This (2018) is an experimental brand campaign designed for Vans, which provides the customer an interactive retail experience inside the store. Inspired by The Kenun Cloud (2017) by HFour Design, You Can’t Touch This (2018) spreads brand identity through playful and engaging interactions among the viewers, who also are the potential customers. You Can’t Touch This (2018) is an installation exploring Spatial Interaction (see chatper 2.1.1), where tangible interaction is embedded in real space (Hornecker and Buur, 2006). Upon coming closer and touching the “button” on the installation wall -- the letters “touch ME”, woven with conductive thread - a series of Vans’ product illustrations pop up on the blank wall. This work also applies a series hybrid interactive technology including Sparkfun Makey-Makey board [27] as the microcontroller, conductive 39 Yufei (Michelle) Yao | Emily Carr University of Art + Design 40 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen thread and projection-mapping techniques. Projection mapping the products right in front of people who interact with the work closely provides the potential customer with a more profound impression through tangible interactions. This experience spreads Vans’ brand identity and product style in a more engaging way than a view-only print banner or video commercial, creating an “Instagrammable” place in store. You Can’t Touch This (2018) was my first practice-based exploration into tangible interactive technologies in experience design. The design objective was to “show the brand identity through playful interactions,” as well as transcend the sensations of the participants. Usually, when people enter a store for fashion and apparel brands, the products are displayed on the shelves in a static way. However, using electronics to embed interactivity into the work, and using projectors to enlarge the computer graphics, You Can’t Touch This (2018) provides the participant with an immersive and playful experience. The creative process also involved UX design methodologies. The creation of You Can’t Touch This (2018) took only one week from ideation to finish through rapid prototyping (Hanington and Martin, 2012). Starting from the design objective, the key interaction of the experience could be quickly identified, and the proper electronics chosen to prototype the experience. Rapid prototyping is a UX design method widely used in many industries, which leads to the creation of the “minimum viable product” (MVP) - an early stage of a product which includes basic features and awaits iteration and optimizations (Ries, 2009). The goal of rapid prototyping is to prototype the key experience of the product through quick and easy techniques. In the showcase of You Can’t Touch This (2018), the participants were all engaged with this playful prototype, keen to touch and interact with it. Vans’ illustrations on the wall and the background music were triggered by the participants’ actions. including playing with the instal41 Yufei (Michelle) Yao | Emily Carr University of Art + Design lation in unexpected ways as some of them tried to develop a rhythm upon changing the frequency of touching the “button”. The development and build process of You Can’t Touch This (2018) also followed rapid prototyping and “hacking” methods. In this project, the input was the action of the user touching both letters “M” and “E”, which were woven by a conductive thread. By connecting the conductive thread with the input pins in the SparkFun Makey-Makey board, the board can return a keycode to a computer. Then by using Processing, a software to code within the context of the visual arts, to control the pre-rendered videos and graphics created in Adobe After Effects, we can control the sequence of these visuals the keycode inputted. Finally, to amplify the visual designs, projection mapping techniques were used to align and enlarge the graphics onto the wall. SparkFun Makey-Makey board is known for quick prototyping and one of the easiest-to-learn yet powerful microcontroller for all age groups. This project leveraged the advantage that this board can turn any conductive material into a key on the keyboard, and this allowed the designer to create and prototype the experience quickly. You Can’t Touch This (2018) still has a lot of space for improvement. As a prototyping version for an interactive branding campaign, the most significant obstacle was not being able to put the work into actual use for a brand. The success of The Kenun Cloud (2017) as a pop-up campaign not only lies in its glamorous display but also in its role of business revenue for the brand, missing from You Can’t Touch This (2018) since it was an experimental prototype but not an actual commercial campaign. Secondly, both interaction design and visual design could be improved: The electronics (SparkFun Makey-Makey board) required participants to touch the conductive letters with both hands, which was confusing for them; the visual and sound design could be more polished, dynamic and avoiding repetition. 42 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 22 & 23 & 24 & 25] Tree of Life (2018) during studio critic showcase at Emily Carr University, 2018. 3.2.2 Tree of Life (2018): A “Magical” and Playful Painting Tree of Life (2018) is an interactive painting which becomes a complete artwork upon the participation of the interactors. It challenges the traditional linear communication between the viewer and the artwork, and redefines the idea of hanging a painting in a room for decoration purposes. Tree of Life (2018) enables dynamic interactions between the painting and the viewers: upon touching different branches of the tree on the painting, Tree of Life (2018) triggers different animations and sounds, creating a “wonderland” experience for people. Being installed in a public space, Tree of Life (2018) hopes to bring a sense of life and hope as the project The Virtual Aquarium (2017) (see chapter 2.2.2). Inspired by The Virtual Aquarium (2017), Tree of Life (2018) creates dynamic visual effects through tangible interactions. Its creation explored the subcategory of tangible interaction: Embodied Facilitation, which refers to the physical system setup or configuration of space and objects (Hornecker and Buur, 2006, p.441). Touching different branches on the “tree” triggers different pre-rendered animations and sounds which were projection-mapped onto the canvas space, resonating with the original tree on the painting. The technical development of Tree of Life (2018) went through several iterations for optimizing the user experience. Following the concept of experience prototyping (Buchenau and Suri, 2000), the MVP version of this project was completed with the same electronic as the project You Can’t Touch This (2018) - a SparkFun Makey-Makey board. However, based on observation and feedback from several interviews after the showcase, most people expressed their frustration and confusion about the interaction (holding one wire on the one hand, and touch the painting with the other hand) as the drawback of using this electronic board. Following the of reflect-in-action process in the thesis exploration, Tree of Life (2018) version II was completed with the optimization in electronics and interaction design, and was installed in the lobby inside Emily Carr University campus during studio open day 2018. The refinement of Tree 43 Yufei (Michelle) Yao | Emily Carr University of Art + Design 44 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Up - Down) [Figure 26 & Figure 27] Tree of Life (2018) version II was installed in the hallway during Emily Carr Studio Open Day 2018. [28] The Arduino Nano is a small, complete, and breadboard-friendly board based on the ATmega328P (Arduino Nano 3.x). (https://goo.gl/Bvs9ze) [29] ADC touch is an Arduino library, source: https://github.com/martin2250/ ADCTouch of Life (2018) allowed it to be more usable and playful, meeting the needs of encountering more people by being installed in a public space. The original Sparkfun Makey-Makey board was replaced by an Arduino Nano board [28], with an integrated library, ADC touch [29] coded in it. Moreover, a bluetooth module was connected to the electronic board, enabling wireless serial communication between the artwork and the computer, providing the viewer a better visual and interactive experience. This change in the build process optimized the experience of Tree of Life (2018) and made it into a well-finished piece, as well as lowered the production budget. On the one hand, Arduino boards (Arduino Uno, Arduino Nano, or other electronics with Atmega328 chip) cost less than the Sparkfun Makey-Makey board, and have much more related open-source projects for reference. On the other hand, the Arduino ADC library allows users to create a capacitive sensor without any external hardware, which means when the microcontroller is connected to a capacitive material, without having to create a closed circuit by touching both input and ground pins, the microcontroller will be able to return the data we need. In this way, the confusion of having to touch both the painting and the other wire at the same time in the Tree of Life (2018) experience was solved. This project took a further step into the exploration of leveraging UX design methods into the creation of a tangible interactive project from the aspect of refinement and iterative process. Methodologies like usability testing, surveys and interviews are the most common qualitative methods from UX design to identify potential problem of the prototype, and gather feedback for validating the product design. In the scope of this thesis exploration, Tree of Life (2018) was iterated from an early prototype to a more durable and polished product using the same methods. However, the creation of Tree of Life (2018) still had limitations, providing viewers with limited visual effects since it was just three pre-rendered videos. The repetitive visual effects caused boredom after few minutes of playing and interacting. After reflecting on this project, the visual design in the later creation in this thesis was improved by using generative computer graphics, creating more changes and randomness. 45 Yufei (Michelle) Yao | Emily Carr University of Art + Design 46 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 28 & 29 & 30 & 31] Let It Rain (2018) during studio critic in April 2018. [30] TouchDesigner is a node based visual programming language for real time interactive multimedia content, developed by the Toronto-based company Derivative. (https://www.derivative.ca/) [31] Leap Motion is a hardware sensor device that supports hand and finger motions as input, without requiring hand contact with the device.(https://www.leapmotion. com/) 3.2.3 Let It Rain (2018) A Room-Scale Interactive Installation Let it Rain (2018) is a room-scale installation which transcends reality. It provides participants a unique experience of controlling the volume of the “rain” by changing their hand gestures. Inspired by the UWPA (2018) by teamLab (see chapter 2.2.3) , Let It Rain (2018) invites people to a digital wonderland, able to interact with “rain” with their small hand gestures as they could not in real life. It is the most popular work in the three experimental projects in this chapter, which was photographed and shared on social media by participants. It has the potential of being installed as an artistic piece in a public space or pop-up show for entertainment purposes. Let it Rain (2018) seeks the possibility of reversing the expectation of users, providing unexpected results based on everyday interactions. This idea references the project UWPA (2018) by teamLab and Rain Room (2012) by Random International, providing users with unexpected feedback as the output, while they conduct a familiar interaction to the system as the input. It explores the idea of Embodied Facilitation in tangible interaction (Hornecker and Buur, 2006, p.441), as the participants interact with the physical object and space, but metaphorically in software space -- the participants’ hand gestures are the input to the interactive computer system in which the “rain” and the “water ripples” are generated through code. The minor interaction of hand gesture changes was expanded by a projector to an room-scale immersive rain, providing the participants a shared interactive experience with the awareness of spatiality. Learning from these past projects, the experience of Let It Rain (2018) was optimized in regards to a few substantial problems. Based on feedback from the first two experimental projects, the pre-rendered visual design sometimes results in boredom among the participants, and Let It Rain (2018) improved on this through developing generative computer graphics. By creating a particle system in the software Touchdesigner [30] and controlling the interactivity through a Leap Motion sensor [31], the visual effects each participant creates is diverse and never repetitive. The use of Touchdesigner was a breakthrough in the technical devel- 47 Yufei (Michelle) Yao | Emily Carr University of Art + Design 48 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen opment of this thesis exploration. In the project You Can’t Touch This (2018) and Tree of Life (2018), pre-rendered computer graphics were used and were controlled through Processing. However, both the repetitive visual experience and laggy CPU performance from Processing can affect the overall user experience and limit the creation. With Touchdesigner, designers can use a node-based visual system to create GPU-driven generative graphics, which is significantly faster than CPU-driven generative graphics in Processing. Moreover, for designers who are not confident with coding, Touchdesigner is a great tool for visual learners as it does not require coding for basic productions. The iteration and improvement came from the iterative thinking mentioned throughout this thesis exploration, following the reflect-in-action process. This was not only restricted to the current project, but also to gathering knowledge from past experience design projects. Moreover, it also inherited the idea of iterative design in UX design practice. In the creation process of Let It Rain (2018), UX design methods were not closely involved, but provided me a core concept of the project ideation, and guidance to create an unforgettable aesthetic to the participants. However, Let It Rain(2018) still has a lot of space for improvement. As a multi-person experience, everyone could see the changes of the rain as one participant moved his/her hand. However, the rest of the group could not interact with the system at the same time. This might cause boredom of the users over a period of time. With this insight, the designs of interactivity in future projects can be more accessible to a multi-person scenario, adding randomness to the generative graphics. 3.2.4 Analysis and Findings Each project in this chapter shows the process of learning, iterating and optimizing. They all experimented with Embodied Facilitation and Spatial Interaction in tangible interaction (Hornecker and Buur, 2006), which is the scope of interactive design in this thesis. And each of the projects leveraged different interactive technologies, including different microcontrollers and sensors, as well as different mediums including digital fabrication, illustration, and animation design. Ob49 Yufei (Michelle) Yao | Emily Carr University of Art + Design Name of the work Use Case Scenario Interactive Technology Tangible Interaction Strength Weakness You Can’t Offline Branding Touch This Campaign for Vans (2018) Sparkfun Makey-Makey Board, Spatial Conductive Paint & thread, Interaction Pre-rendered Video Content Creative Idea; Playful Interactions Repetitive Visual Design and Interactions Tree of Life(2018) Interactive Painting for Interior / Public Space Arduino Nano, Conductive Paint, Ultrasound Sensor, Pre-rendered Video Content Embodied Facilitation Creative idea; Beautiful Visual Design Repetitive Interactions Let it Rain(2018) Room-Scale Public Art Installation or Pop-up Show Leap Motion, Generative Computer Graphics Embodied Facilitation Immersive Visual Effects; Playful Interactions Unaccessible to a Multi-People Scenario servation of each showcased project demonstrates that all are highly engaging interactive experiences by sharing ludic or spontaneously playful interactions. The feedback collection and interviews conducted of the participants proved that these experimental interactive works have the flexibility of artistic expression and creativity. Table Top [Table 1] Analysis of three Experimental Experience Design Projects. Most importantly, these three projects explore the implementation UX design methods into the creation process. Several methods like bodystorming and rapid prototyping were woven into the process of creating the interactive experience. Moreover, methods like observation, interviewing and surveying were useful for gathering feedback from the participants. Through these UX methods, and the mindset, I was able to pinpoint the weaknesses of each project by referencing participants’ frustrations and confusion when they interact with the system, as well as document the rewarding moment when participants engaged with the prototype with joy. These ideas were then iterated in the next project by following this UX mindset and the “reflect-in-action” process. In conclusion, the three exploratory projects have paved the way for this thesis exploration. The use of hybrid interactive technologies and shared the creation process of UX design can create digital experiences not only on the digital screens but also in the tangible spaces.This transition of interactivity from the screen to tangible spaces proposed new opportunities for designers and developers to create a shared, memorable and engaging experiences for people. 50 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 32] Low-fidelity Concept Sketch on Paper of StarSpace (2018) [Figure 33 & Figure 34] Static Computer Graphic Sketches of StarSpace (2018) 3.3 Final Thesis Project After studying the in-depth case studies and prototyping three exploratory design projects, project StarSpace (2018) brought this thesis exploration to a higher level. StarSpace (2018) is an immersive spatial meditation experience as well as a design research project, also the final thesis project. This project gathered insights from Embodied Facilitation and Spatial Interaction in tangible interaction design theory (Hornecker and Buur, 2006, p.437). StarSpace (2018) optimized the experience from the previous three projects based on user feedback and the critical self-reflections led by the reflection-in-action research methodology (Schön, 1983). Moreover, this final thesis project also improved in the visual and interaction design aspects based on more advanced creative coding and interactive developement skills. The creation process of this project delved further into the thesis question, investigating how to leverage UX design methods in tangible and experimental interactive design, and its strength compared to screen-based interactive digital experience. 3.3.1 The Creation Process and Project Objectives StarSpace (2018) is inspired by the project DOME (2018), and the curiosity of bridging digital design and well-being. The project’s objective is to provide participants a more calming and engaging experience for their meditation practice, compared to using digital screen-based tools. This project followed UX design methodologies throughout the two-week creation process. As learned from the past experimental projects, the development of tangible interactive design takes a relatively long time, and is harder to iterate than screen-based 51 Yufei (Michelle) Yao | Emily Carr University of Art + Design 52 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 35 & 36 & 37] Design Probe for the participants before the showcase of StarSpace (2018) [32] Headspace is an English-American online healthcare company, specializing in meditation.(https://en.wikipedia.org/wiki/ Headspace_(company)) designs. To avoid the waste of time in the early phase of creation, a series of few low-fidelity sketches and static computer graphics were created prior to the high-fidelity interactive prototypes. The overall visual design principle was to create an infinite space allowing participants’ minds to swim freely, and subtle and mild interaction without distracting the audience throughout their meditation experience. As Browning et al. (2014) posit, patterns which originate from nature can have calming effects, helping people to reduce stress and anxiety (p.9). The sketches were shown during studio class for design critique and feedback. Then the visual and interactive development kicked off after the idea of “starry sky and aurora” was confirmed. At the same time, a design probe was distributed three days ahead of the meditation experience to each participant who signed up for the event. Each participant was given a toolkit containing three envelopes, each with instruction cards and a pencil inside. The two objectives of handing out this design probe were: 1) Comparative Analysis: The first and second envelope contained instructions for downloading HeadSpace[32] and practice a three-minute meditation. Since the objective of this design research project is to “provide participants a more calming and engaging experience for their meditation practice compared with using digital screen-based tools,” user feedback from the screenbased tool was the first step for gathering data. 2) Provide Feedback for the Design: While the first two envelopes were directing each participant to practice mindfulness on HeadSpace, the instructions in the third envelope played the role of recording and gathering user feedback. People were asked to fill in the same rating sheet after experiencing the StarSpace (2018) medi- 53 Yufei (Michelle) Yao | Emily Carr University of Art + Design 54 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Up - Down) [Figure 38] Instruction Card of Day One in the Design Probe for StartSpace(2018) [Figure 39 & 40] Instruction Card (partial, see appendix for the whole sets of cards) of Project Feedback in the Design Probe for StarSpace (2018) [Figure 41] Instruction Card of Day Two in the Design Probe for StarSpace (2018) tation experience. From this point, we can draw a basic conclusion of the research objectives. 3) Co-creation: Inside this probe, there are also open-ended questions for participants to fill out, including their definition of “infinite space” and their preference of natural scene. This provided inspiration for the project and for the future of visual design in health and well-being related digital projects. [33] http://www.jimmzzhang.com/ StarSpace (2018) was presented at Emily Carr University of Art+Design to sixteen participants at the project showcase day. It contained two parts: a star background footage and real-time rendered audio-reactive aurora light. During the StarSpace (2018) experience, participants practiced mindfulness while listening to the same soundtrack by HeadSpace, but in an immersive projection-mapped room together. Apart from the narrator’s voice guiding the meditation, background soundtracks were delicately designed to optimize the user experience of meditation. Natural white noise was composed into the meditation audio throughout the experience, as well as two background soundtracks at the beginning and the end of the experience, providing the participants a smooth transition in and out of the meditation experience. And the audio-reactive aurora, which reacts to the audio of all the sounds, was coded in GLSL shading language by visual artist Jimmz Zhang [33]. The star footage, which was composed in TouchDesigner through particle system, rendered two thousand stars which move and travel over the sky in a subtle way. The development process of this project followed the workflow and pipeline from the project Let it Rain (2018). Both projects used Touchdesigner, a generative art software to create a GPU-driven particle system. Moreover, the project StarSpace (2018) added an 55 Yufei (Michelle) Yao | Emily Carr University of Art + Design 56 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Clockwise) [Figure 42 & 43 & 44 & 45] Documentation Photo of the Showcase of StarSpace (2018) extra layer for interactivity - the audio-reactive particles. This was a unique generative art technique that can be done in various software, like Processing or Touchdesigner. Following the opensource online community and tutorial resources, we can analyze an audio clip and create different visual sequences according to the rhythm. With this technique and the open-source community, every designer can follow the well-structured guidance and create stunning audio-visual projects. 3.3.2 User Feedback and Project Impact From the three-day experience of the StarSpace (2018) design research project, fourteen valid design probes were collected for data interpretation. The data showed that: Compared to the individual meditation experience on the mobile app HeadSpace, users in this test group (fourteen Emily Carr Students) concentrated better and felt more calm during the installation meditation experience of StarSpace (2018). The methodology of interpreting data followed a series of qualitative design research methods (Collins, 2010, p.169), yet combined methods used in scientific research to make the results more credible and objective. In all fourteen valid questionnaires handed out during the three days of experience, there were four positive descriptions and four negative ones (see Figure 34 for details). This way of structuring the questionnaire seems redundant, but it can avoid the bias if showing only participant positive or negative descriptions, rather than both. Each question was rated by the participant from “Totally Disagree” to “Totally Agree”. The way to interpret all the data is to turn this verbal description into variables from minus two to positive two: 57 Yufei (Michelle) Yao | Emily Carr University of Art + Design 58 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Up - Down) [Figure 46 & 47] Infographic of the Overall Experience Trend of All Participants for StarSpace (2018) Description in the Questionnaire Totally Disagree Disagree I Don’t Know Agree Totally Agree [Figure 48 & 49] Infographic of the Individual Experience Trend of All Participants for StarSpace (2018) Interpreted number in data analysis -2 -1 0 1 2 Table In This Page [Table 2] Data Interpretation criteria of StarSpace (2018) Design Probe In this case, the qualitative responses can be converted to quantitative data. This method of interpreting data can provide other designers and researchers a more direct way of analysing the results and evaluating the project impact. If the score for the first four questions (positive description) is closer to positive two, it means that it has positive feedback. Conversely, for the other four questions (negative description), if it’s closer to minus two, it means it has a positive impact as well. The objective of this research is to see whether this medium can make a positive impact on health and well-being related experiences. From the average for all fourteen participants on the first four questions (positive description) and last four questions (negative description) on the three days of experience, we can see there is a positive trend. However, as we take a closer look at the individual trend of each participant, the result varies. Meditation experience is a very dynamic, complex and subjective phenomenon. It depends upon the perception of multiple sensory qualities of design, interpreted through filters relating to contextual factors. In the StarSpace (2018) experience, there was no absolute “successful experience” for all participants, but when we focus on the overall trend and feedback, we can gather insights that StarSpace (2018) succeeds in creating an engaging experience for meditation from its immersive visual and audio design. This engagement can be leveraged into various cases for use in health and well-being related events and activities, like DOME (2018) in the case studies chapter. 59 Yufei (Michelle) Yao | Emily Carr University of Art + Design 60 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen 3.3.3 Conclusion and Critical Reflections The final thesis project in this chapter represents the learning outcomes for the key research question initially proposed at the start of this document: How can we leverage user experience design methodologies into experimental experience design, where the focus is no longer on usability but on creative expression through interactions? UX design methods are efficient and valuable in both forms of traditional and experimental tangible interaction design, while designers have to tweak the methodologies in the creation of the latter. Learning from the drawbacks in the three exploratory projects (You Can’t Touch This (2018), Tree of Life (2018) and Let it Rain (2018)), the final thesis project StarSpace (2018) applied UX design methods such as low-fidelity prototyping, design probe and questionnaires in the creation process. StarSpace (2018) benefited from these methods as they accelerated the design process as well as letting the designer practice human-centered design. This project also explored Embodied Facilitation further in tangible interaction (Hornecker and Buur, 2006), and provided participants a non-repetitive, generative interactive visual design and immersive digital environment. While looking at the project StarSpace (2018) critically, the use of UX design methods can be improved. Initially, more co-creation sessions and design workshops should be conducted prior to the creation of the work, like NGX interactive did for The Virtual Aquarium (2017) project. This can help the design and researcher know more about participants’ needs and build empathy, and it will benefit either visual design or interaction design for the project. Secondly, the 61 Yufei (Michelle) Yao | Emily Carr University of Art + Design project lacked user testing sessions and early user feedback. From the point of interactive development, it is hard to develop and design a prototype mimicking the ideal experience. But creating early or low-fidelity prototypes can avoid wasting time on the wrong creative strategies, making the creation process much more efficient (Wong, 1992). This is one of the areas of future direction that this thesis wants to explore. Lastly, due to the limitations of space and technology, StarSpace (2018) can be showcased better in technical ways in the future, like DOME (2018). [23] Literally translated from French as ‘Beyond the Limit’. [24] Random International, The Rain Room, 2012 (https://goo.gl/Q9cdBq). 3.4 Future Direction - Craft Experiences Through Narrative The two-year exploration with three exploratory projects and one final thesis project is not the end of this thesis research. While reflecting critically from every project in both artistic expressions and interactive development perspectives, we can see that all four projects were focusing mostly on visual and interaction design rather than content creation. To design a successful and memorable interactive digital experience, one of the keys is to engage the audience continuously throughout the experience, rather than just the beginning of it. Narrative and storytelling are indispensable to enrich the content and depth of an interactive experience. The project The Virtual Aquarium (2017) from NGX Interactive followed this principle of crafting the experience around an engaging story rather than glamorous visuals. This also applies to other projects in the case studies chapter. As a designer who is always keen to learn more, designing the experience through storytelling is one of the future directions I would like to pursue and bring this thesis research further. 62 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Left - Right) [Figure 50 & 51] Details of the interactive projection on the floor of the project Diretions of the tide (2019) 3.4.1 2019 Grad Show Project - Directions of the Tide(2019) With the passion and motivation to craft an experience through storytelling, the project Directions of the Tide (2019) was created at the end of this thesis exploration. Directions of the Tide (2019) was a twenty-minute live performance showcased in public at the Integrated Motion Capture Studio in Emily Carr University in May 2019. It contained two parts: (1) non-interactive video projections on the walls and (2) an interactive floor projection that mimicked a river. The non-interactive video was a series of motion graphics, guided by a monologue telling the story of how I was educated and raised as a Chinese girl by strict parents. Meanwhile, the interactive floor projection was triggered at certain story points, creating an immersive and engaging experience that invited participants to enter the “river” and interact with the artwork. The creation of Directions of the Tide (2018) hoped to invoke an open discussion with participants around the relationship between individuals and the collective. It was an experience design project; an artwork as a metaphor, questioning the relationship between individuals and the society they live in; and a critical enquiry about a phenomenon in modern Chinese society. The interactive river, which consisted of sixty thousand real-time-rendered particles, could change its shape and direction when the participants stepped onto the interaction area. This was a metaphor for whether we should live as the majority does (that is, “go with the flow,” just like a drop of water in the river) or whether we should challenge this idea, and move against the flow. Directions of the Tide (2019) creatively explored this complex dynamic between the individual and society through tangible interactivity and spatial interaction (Hornecker and Buur, 2006, p.440) . The development of Directions of the Tide (2019) went through a 63 Yufei (Michelle) Yao | Emily Carr University of Art + Design 64 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Images Right (Up - Down) [Figure 52 & 53] Documentation photo of the live performance Directions of the tide (2019) complex process. The interactive river was developed through TouchDesigner, and was made up of particles that could change their direction and velocity. While the participants were in the river, their skeleton and motion data could be captured and sent back to the computer through a motion capture sensor, Kinect 2. Using this data and applying mathematical formulas, we could control the river graphics and change the overall shape and direction of it. This project also followed the open-source community around TouchDesigner users worldwide; this meant the designer didn’t have to start from scratch, and therefore helped the designer to achieve a lot more when creating a unique project. Aided by an engaging audio-visual narrative, the Directions of the Tide (2019) experience was fully booked during the showcase day and received excellent feedback. According to participants, the whole experience was “a magnificent and unique creation”. After the show, there were a lot of participants posting their thoughts and event photos on social media. From the past four projects, we can see the weaknesses in content creation and storytelling, but Directions of the Tide (2019) stepped forward onto the next level for designing an interactive digital experience by engaging the audience through stories. Looking forward, embedding stories into experience design will be the direction where this thesis research continues. What the viewers can remember and keep thinking about after leaving the experience are the most vital points of interest for experience designers to consider, even before the early design planning phase. Without narrative, even the most glamorous visuals and effects are easily forgotten. 65 Yufei (Michelle) Yao | Emily Carr University of Art + Design 66 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen VI. Conclusion This thesis and the two-year design research explored the opportunities and creation methodologies of an alternative way of creating a memorable and beautiful interactive digital experience: using physical spaces and tangible objects instead of screens, as the place where playful human interaction happens. While the research scope of this thesis seems wide, the four case studies helped with narrowing down the practice-based design research topics and themes. These outstanding works – The Kenun Cloud (2017), The Virtual Aquarium (2017), Universe of Water Particles on Au-delà des limites (2018), and DOME (2018) – inspired the creation of the four new studio works in this thesis as well as the enormous potential this form of interactive experience in the aspects of advertising and branding, health and well-being, and public art and exhibitions. The four studio works in this thesis showcased the speciality of leveraging interactive technologies and explored the unique design process of tangible interactive design. The three exploratory projects You Can’t Touch This (2018), Tree of Life (2018), and Let it Rain (2018) followed the reflect-in-action (Schön, 1983) and experience prototyping (Buchenau and Suri, 2000) principles, and were iterated from user feedback to improve the experience. While these three projects were design prototypes rather than well-finished works, they are the fundamental parts of this design exploration. The final thesis project StarSpace (2018) used these past experiences and presented a more polished interactive experience in front of the audiences. Throughout the two years of learning, prototyping, researching and designing, my understanding of interaction design and cross-disciplinary studies proceeded to a higher level. 67 Yufei (Michelle) Yao | Emily Carr University of Art + Design This practice-based research contributes to the interaction design discipline as challenging its scope, and hopes to inspire more young designers and local business owners in the Metro Vancouver area to adventure. The education curriculum of the Interaction Design major at postsecondary institutions like Emily Carr University places emphasis on screen-based interaction design or service design to meet the needs of fast-growing job markets[34]. Nonetheless, more and more young designers and artists are setting their sights on more creative and playful tangible interactive digital experiences. This form of experimental interactive design should be integrated into Interaction Design education, since its potential will be enormous. We can back this statement by the success of local interactive studios featured in this thesis: HFour Design, NGX Interactive and to:Resonate. [34] By referring to this, the author also reflected her own education - BA Digital Media Art program at Communication University of China. And her past work experiences in the high-tech industry. [35] From the interview with Jan Beringer, Experience Design Lead at NGX Interactive. Finally, this thesis work is also a cross-disciplinary exploration. By combining interactive technologies, media arts and interaction design, four engaging and creative interactive experiences were created for people. Despite the limitations of technical skills and opportunities of showcasing these works in public, all projects in this thesis explored the cross-disciplinary design opportunities in the DIY electronic culture, creative coding, and generative art. It also challenges the public in what to expect from a designer; a designer should be a creative individual who is not afraid of stepping out of their comfort zone, and is bold to experiment with new things, such as cutting-edge interactive technologies and programming. As designers in the creativity industry, we should never let technology drive us when designing an experience for people, but we can use interactive technology as a tool for helping us to achieve our goal: designing memorable and beautiful interactive experiences for people— designing their “experience of a lifetime”. [35] 68 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Bibliography Akpan, I., Marshall, P., Bird, J., & Harrison, D. (2013). Exploring the effects of space and place on engagement with an interactive installation. Proceedings of the SIGCHI Con ference on Human Factors in Computing Systems - CHI ’13. doi:10.1145/2470654.2481306 Anthony Dunne et al., The Presence Project (London: Royal Col lege of Art, 2000). Amatriain, X., Kuchera-Morin, J., Hollerer, T., & Pope, S. T. (2009). The allosphere: Immersive multimedia for scien tific discovery and artistic exploration. IEEE MultiMedia, 2(16), 64-75. Aarts, E., & Marzano, S. (2003). The new everyday: Views on am bient intelligence. 010 Publishers. Bennett, A. (2002). Dynamic interactive aesthetics. The Journal of Design Research, (2). Bill Gaver, Tony Dunne, and Elena Pacenti, “Design: Cultural Probes,”Interactions 6:1 (1999): 21–29. Bongers, A. J. (2000). Interaction in multimedia art. KnowedgeBased Systems, 13(7), 479-485. Braun, A. (2011). Activating Spaces: A Practice-Based Exploration of the Role of Interactivity in Public Space. Brandt, E. (2006). Designing exploratory design games: a frame work for participation in Participatory Design? In Proceedings of the ninth conference on Participatory design: Expanding boundaries in design - Volume 1 (pp. 57-66). Trento, Italy. Browning, W.D., Ryan, C.O., Clancy, J.O. (2014). 14 Patterns of Biophilic Design. New York: Terrapin Bright Green, LLC. 69 Yufei (Michelle) Yao | Emily Carr University of Art + Design Buchenau, M., & Suri, J. F. (2000, August). Experience prototyp ing. In Proceedings of the 3rd conference on Designing in teractive systems: processes, practices, methods, and techniques (pp. 424-433). ACM. Carl Adams and David Avison, “Dangers Inherent in the Use of Techniques: Identifying Framing Influences,” Information Technology & People 16:2 (2003): 203–234. Chapman, Owen, and Kim Sawchuk. “Creation-as-Research: Critical Making in Complex Environments.” RACAR: Revue d’art Canadienne / Canadian Art Review, vol. 40, no. 1, 2015, pp. 49–52. Collins, H. (2010). Creative research: The theory and practice of research for the creative industries. Lausanne, Switzer land: AVA Pub. SA. Cooper, Alan; Reimann, Robert; Cronin, Dave (2007). About Face 3 : The Essentials of Interaction Design. Indianapolis, Indiana : Wiley. p. 610. ISBN 978‐0‐470‐08411‐3. Retrieved 18 July 2011. Design at Work: Cooperative Design of Computer Systems, Joan Greenbaum and Morten Kyng, eds., 169–195. Fallman, D. (2011). The Interac on Design Research Triangle of Design Practice, Design Studies, and Design Exploration. Design Issues, 24(3), 4-18. Frayling, C. (1993). Research in Art and Design. Royal College of Art Research Papers, 1(1), 1-5. Frich, J., Mose Biskjaer, M., & Dalsgaard, P. (2018, June). Twenty Years of Creativity Research in Human-Computer Inter action: Current State and Future Directions. In Proceed ings of the 2018 Designing Interactive Systems Confer ence (pp. 1235-1257). ACM. Gray, C., & Malins, J. (2004). Visualizing Research: A Guide To The Research Process In Art And Design (illustrated edion.). Ashgate Pub Ltd. 70 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Jane Fulton Suri, “Empathic Design: Informed and Inspired by Other People’s Experience” in Empathic Design, Ilpo Koskinen, Katja Battarbee, and Tuuli Mattelmäki, eds. (Helsinki: IT Press, 2003), 53. Hébert, Cristyne. "Knowing and/or experiencing: a critical exam ination of the reflective models of John Dewey and Donald Schön." Reflective Practice 16.3 (2015): 361-371. Horn B., Svendsen E.H., Madsen K.H. (2001) Experimental De sign of an Interactive Installation. In: Qvortrup L. (eds) Virtual Interaction: Interaction in Virtual Inhabited 3D Worlds. Springer, London Hornecker, E., Halloran, J., Fitzpatrick, G., Weal, M., Millard, D., Michaelides, D., Cruickshank, D., et al. (2006). UbiComp in opportunity spaces: challenges for participatory design. In Proceedings of the ninth conference on Participatory design. Kaj Grønbæk, Prototyping and Active User Involvement in System Development: Towards a Cooperative Prototyping Approach (Unpublished Ph.D. thesis, Computer Science Department, Aarhus University, 1991). Access at: www. daimi.au.dk/~kgronbak/Thesis/ThesisOverview_ToC. html. Kinsella, Elizabeth Anne. "The art of reflective practice in health and social care: Reflections on the legacy of Donald Schön." Reflective Practice 11.4 (2010): 565-575. Konrad R. Budde, Karlheinz Kautz, Karin Kuhlenkamp, and Heinz Züllighoven, Prototyping: An Approach to Evolutionary System Development (Berlin: Springer-Verlag, 1992), 24–30. Kurvinen, Esko, Ilpo Koskinen, and Katja Battarbee. "Prototyping social interaction." Design Issues 24.3 (2008): 46-57. Kwastek, K. (2013). Aesthetics of interaction in digital art. MIT Press. 71 Yufei (Michelle) Yao | Emily Carr University of Art + Design Kujala, S., Roto, V., Väänänen-Vainio-Mattila, K., Karapanos, E., & Sinnelä, A. (2011). UX Curve: A method for evaluating long-term user experience. Interacting with Computers, 23(5), 473-483. Laurel, B., & Lunenfeld, P. (2003). Design Research: Methods and Perspectives. The MIT Press. Lee, K. P. (2018, June). Design Research, for What?: Different Perspectives on Design Research. In Proceedings of the 2018 on Designing Interactive Systems Conference 2018 (pp. 5-5). ACM. Martin, B., & Hanington, B. M. (2012). Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions. Beverly, MA: Rockport Publishers. McNiff, Shaun. ‘Research Ideas and Methods.’ Art-based research. Jessica Kingsley Publishers, 1998, pp. 145-206. Menenghini, F., Palma, P., & Taylor, M. (2004). The art of experi -mental interaction design (Vol. 4). Systems Design Limited. Preece, J., Rogers, Y., & Sharp, H. (2015). Interaction design: be yond human-computer interaction. John Wiley & Sons. Prahalad, C. K., & Ramaswamy, V. (2004). Co-creation experienc es: The next practice in value creation. Journal of interactive marketing, 18(3), 5-14. Ries, E. (2009). Venture Hacks interview: “What is the minimum viable product?”. [online] Startuplessonslearned.com. Available at: http://www.startuplessonslearned. com/2009/03/minimum-viable-product.html [Accessed 23 Mar. 2009]. Schön, D. A. (2017). The reflective practitioner: How professionals think in action. Routledge. Schuler, D., & Namioka, A. (Eds.). (1993). Participatory Design: Principles and Practices. L. Erlbaum Associates Inc. Seo, J. H. (2011). Aesthetics of Immersion in Interactive Immer72 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen sive Environments: A Phenomenological Case Study of Light Strings (Ph.D. dissertation). School of Interactive Arts and Technology, Simon Fraser University, Vancouver, Canada. Retrieved from h ps://theses.lib.sfu.ca/thesis/ etd6695. Shedroff, N. (2001). Experience design 1. New Riders Publishing. Simanowski, R. (2011). Digital art and meaning. Reading Kinetic Poetry, Text Machines, Mapping Art and Interactive In stallations (Minneapolis: University of Minnesota Press, 2011). Simo Säde, Cardboard Mock-ups and Conversations: Studies in User-CenteredDesign (Helsinki: UIAH, 2001). Simo Säde, Cardboard Mock-ups and Conversations. Studies in User-Centered Design; Pelle Ehn and Morten Kyng,“ Cardboard Computers: Mocking It Up or Hands-On the Future” in Design at Work: Cooperative Design of Com puter Systems Joan Greenbaum and Morten Kyng, eds. (Hillsdale, NJ: Lawrence Erlbaum,1991), 169–195. Sora, C., Jordà, S., & Codina, L. (2017). Chasing real-time inter action in new media: towards a new theoretical approach and definition. Digital Creativity, 28(3), 196-205. Sommerer, C., & Mignonneau, L. (1999). Art as a living sys tem: Interactive computer artworks. Leonardo, 32(3), 165173. doi:10.1162/002409499553190 Spinuzzi, C. (2005). The Methodology of Participatory Design. Technical Communication, 52(2), 163–174. Suri, J. F. (2003). The experience of evolution: developments in design practice. The design journal, 6(2), 39-48. T. Erickson, “Notes on Design Practice: Stories and Prototypes as Catalysts for Communication” in Scenario-Based Design: Envisioning Work and Technology in System Development, John Carroll, ed. (New York: John Wiley & Sons, 1995) 73 Yufei (Michelle) Yao | Emily Carr University of Art + Design Meng, Jude Chua Soo. "Donald Schön, Herbert Simon and the sciences of the artificial." Design Studies 30.1 (2009): 60-68. McDonagh, D., Hekkert, P., Van Erp, J., & Gyi, D.(2004). Design and Emotion: The Experience of Everyday Things. Taylor Francis Group. Great Britain. Sullivan, D. G. (2009). Art Practice as Research: Inquiry in Visual Arts (Second Edi on.). Sage Publications, Inc. TeamLab. (2017). Transcending boundaries.https://www.team lab.art/concept/digital_domain_releases/ Utterson, A. (2013). Early Visions of Interactivity: The In (put) s and Out (put) s of Real-Time Computing. Leonardo, 46(1), 67-72. Wolf, T. V., Rode, J. A., Sussman, J., & Kellogg, W. A. (2006). Dispelling "design" as the black art of CHI. In Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 521-530). Montréal, Québec, Canada. Wong, Y. Y. (1992, May). Rough and ready prototypes: lessons from graphic design. In Posters and short talks of the 1992 SIGCHI conference on Human factors in computing sys tems (pp. 83-84). ACM. Yin, D. R. K. (2008). Case Study Research: Design and Methods (4th ed.). Sage Publications, Inc. Zimmerman, J., Forlizzi, J., & Evenson, S. (2007, April). Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI conference on Hu -man factors in computing systems (pp. 493-502). ACM. 74 Appendix 76 I. TCP 2 Core Certificate 77 II. Creation Process of Project Tree of Life (2018) 79 III. Creation Process of Project Let It Rain(2018) 81 IV. Design Probe for Project StarSpace (2018) 84 V. Raw Data from User Feedback in Project StarSpace (2018) | Day 1 - Day3 86 VI. ECU Grad Show Project 2019 Directions of TheTide (2019) 87 VII. Summer Research - ITP Camp 2018 (NYU, New York) 75 Yufei (Michelle) Yao | Emily Carr University of Art + Design Appendix I TCPS 2 Core Certificate PANEL ON RESEARCH ETHICS TCPS 2: CORE Navigating the ethics of human research Certificate of Completion This document certifies that Yufei YAO has completed the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans Course on Research Ethics (TCPS 2: CORE) Date of Issue: 8 November, 2017 76 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Appendix II Creation Process for Project Tree of Life (2018) Details for Fabrication and Visual Design Prototypes. [Portfolio] https://www.thisismichelleyao.com/tree-of-life 77 Yufei (Michelle) Yao | Emily Carr University of Art + Design Mock-up for project Tree of Life (2018) 78 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Appendix III Creation Process for Project Let It Rain (2018) Details for Fabrication, Interactive Development and Installation Techniques. [Portfolio] https://www.thisismichelleyao.com/let-it-rain 79 Yufei (Michelle) Yao | Emily Carr University of Art + Design 80 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Appendix IV Design Probe for Project StarSpace (2018) Complete Design Probe for the Project [Portfolio] https://www.thisismichelleyao.com/starspace 81 Yufei (Michelle) Yao | Emily Carr University of Art + Design 82 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen 83 Yufei (Michelle) Yao | Emily Carr University of Art + Design Appendix V Raw Data from User Feedback in Project StarSpace (2018) | Day 1 - Day3 Design Research for the Project [Portfolio] https://www.thisismichelleyao.com/starspace 84 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen 85 Yufei (Michelle) Yao | Emily Carr University of Art + Design Appendix VI Interactive development of Directions of TheTide (2019) A New Media Art Live Performance on May 13, 2019, at Integrated Motion Capture Studio, ECU. [Portfolio] https://www.thisismichelleyao.com/directions-of-the-tide 86 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen Appendix VII Summer Research - ITP Camp 2018 (NYU, New York) A month of intensive training & workshops into Digital Fabrication, Electronic Arts and Interactive Developement. [Portfolio] https://www.thisismichelleyao.com/light-a-fire-light-up-a-spark 87 Yufei (Michelle) Yao | Emily Carr University of Art + Design 88 The Experience of a Lifetime - Interactive Digital Experience Beyond The Screen 89 Yufei (Michelle) Yao | Emily Carr University of Art + Design 90 EMILY CARR UNIVERSITY OF ART + DESIGN, 2019 Yufei Yao © 2019