Consciousness, Matter and the Invisible By Leó Stefánsson BA Visual Arts, Iceland Academy of the Arts, 2011 A THESIS ESSAY SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED ARTS in Visual Arts EMILY CARR UNIVERSITY OF ART + DESIGN © Leó Stefánsson, 2013 Abstract In my work in the MAA program I have focused on bringing the invisible/inaudible into the scope of human consciousness. Human perception is limited to our senses, and in this thesis I will investigate how we can allow for the perception of the invisible/inaudible through the method of constructing translational systems that transform signals from one state to another by applying a range of tools. In order to approach a thing we cannot see, we have to consider both the physical world, in which the thing exists, and how the thing is perceived in consciousness. The idea of mind and matter existing independently of each other is traced back to René Descartés and his method of doubt and logic which excluded the spirit from the physical world which he described as motion in three dimensions where everything was made of matter. His work laid the grounds for reductionism in neuropsychology which led to the idea that the human brain, and eventually consciousness, could be reduced to simple interactions of units of matter. This allowed Alan Turing to describe his theoretical computational device which had states synonymous to human mental states and eventually resulted in the creation of the modern digital computer. In response to the reductionist view of consciousness, Amit Goswami describes what he calls monistic idealism; a model of the world where matter and consciousness are not incompatible terms but are parts of a unified system. Wassily Kandinsky used color, or frequencies of light, in combination with shape to portrait objectless ideas originating from within himself, allowing the spirit a physical representation through his consciousness. I look at my work in the same terms, but instead of using frequencies of color I use frequencies of sound to represent spiritual ii values. I also draw some of my methods and inspiration from James Turrell and the way he emphasizes the human perceptual apparatus in his work. In my own work I construct parascientific experiments that investigate physics, consciousness and their relationship. These experiments resulted in the creation of systems constructed for the analysis and reconstruction of units of time, or frequency. My artworks discussed in the thesis are constructed around the concept of time, which I consider to be the primary aspect of my work. iii Table of Contents Abstract! ! ! ! ! ! ! ! ! ii List of Tables! vi List of Illustrations! vii List of Symbols and Abbreviations! viii Acknowledgements ! ix 01. Introduction! 1 02. Consciousness, Perception and the Invisible/Inaudible! 3 02.1 Chapter overview! 3 02.2 Consciousness! 3 02.3 Mind and Matter - Cartesian Dualism! 4 02.4 Reductionism and the Mechanical Mind! 7 02.5 The Brain as Machine! 10 02.6 Consciousness and Physics - Monistic Idealism! 11 03. Time Aesthetics! 17 03.1 A System Aesthetic! 17 03.2 Time as medium! 17 03.3 Kandinsky - Nonphysical forms! 19 03.4 Paranormal / Parascience / Telepathy! 20 03.5 James Turrell and the removal of visual reference! 22 03.7 Brain to Computer Interface (BCI)! 24 03.8 Sensory input! 26 04. Parascientific techniques: Software, Visualization, Sonification and Fractal Mapping! 28 iv 04.1 Computer programming! 28 04.2 Data sets! 29 04.3 Fourier-transformation! 30 04.4 Waves! 31 04.5 Comparative signal analysis! 32 04.6 Mapping and Remapping! 33 04.7 Fractals and Fibonacci sequences! 33 05. Artworks ! 37 05.1 BrainTV! 38 05.2 Observations! 44 05.4 Thingogram! 48 05.4.1 Thingogram v.1! 49 05.5.2 Thingogram v.2! 53 05.5.3 Thingogram v.3! 56 06. Conclusion! 61 Works Cited! 65 v List of Tables Table 1. Brainwave frequencies and associated mental states. ! ! ! 24 vi List of Illustrations Fig. 1 A single form can be perceived in multiple ways.! ! ! ! 8 Fig. 2 We perceive complete shapes where there are none.! ! ! 9 Fig. 3. Sierpinskis triangle rendered with Processing. ! ! ! ! 34 Fig. 4 A fibonacci spiral.! ! ! ! ! ! ! ! ! 35 Fig. 5. BrainTV displaying a complex signal.!! ! ! ! ! 41 Fig. 6 BrainTV installation view.!! ! ! ! ! ! ! 42 Fig. 7 Observations no. 3, Waves.! ! ! ! ! ! ! 44 Fig. 8 Observations no. 4, Traffic structure.! ! ! ! ! ! 45 Fig. 9 Observations no. 7, Birds and breathing.! ! ! ! ! 46 Fig.10 An accordion filtering a white noise signal! ! ! ! ! 50 Fig. 11 A spectral image of a hat.! ! ! ! ! ! ! 52 ! ! ! ! ! 53 Fig. 13 A screenshot of the NSSA algorithm. ! ! ! ! ! 55 Fig. 14 Thingogram v.3 installation view.! ! ! ! ! ! 56 Fig. 15 Thingogram v.3 flight-case detail.! ! ! ! ! ! 57 Fig. 16 Instructions on the monitor.! ! ! ! ! ! 58 Fig.12 A spectral image of an accordion! ! vii List of Symbols and Abbreviations EEG: Electroencephalograph Hz: Hertz, Cycles per second NSSA: Natural Sound Synthesis Algorithm ϕ: The mathematic constant Phi viii Acknowledgements I would like to thank the following people for guiding me through the process of writing this thesis. Without them this thesis would not be what it is. My wife Heiða who provided me with limitless support through my studies. My daughter Nína Heiðrós who always puts things in perspective. My parents who made all of this possible. My supervisor David Bogen who helped me put a large idea into perspective and introduced me to new ideas while guiding me through the writing process. Teachers and class mates for encouraging me to pursue what matters to me the most. Dennis Burke and Dušan I. Bjelic for reviewing my thesis, challenging my position and guiding me towards future research. ix 01. Introduction In this thesis I will lay out important ideas about the relationship between the mind and matter with an emphasis on the invisible aspects of those terms. We can observe our own minds and identify different mental states, but we can not observe the consciousness of another being. ! I am intrigued by the things I can not see. The invisible world located outside the scope of human perception; how we can observe the existence of magnetism, electricity, inaudible sound and invisible light (among other things) indirectly through their effects on physical matter, including the effect on the instruments we use to measure these forces. The natural sciences commonly assume the world is made of physical matter and that consciousness, the internal world, should be excluded from the description of the external world. This view is largely due to the mind-matter dualism introduced by René Descartés which was recently called into question by physicist Amit Goswami, who proposes that matter and consciousness are part of the same construct. This does not invalidate the natural sciences nor does it raise consciousness above the physical realm, but it allows consciousness to be a part of our integral understanding of the natural sciences. ! In Newtonian physics, every force has an equal and opposite force. Here, the exchange of energy is what enables matter to move from one place to the another, or shift between states. Since it is difficult, if not impossible, to measure consciousness itself, it is incompatible with the Cartesian/Newtonian model. However, if we assume consciousness resides in the brain, and the brain emits an electrical signal, 1 consciousness can be considered to be emitting subtle energy, making it a physical artifact of consciousness. This contradicts the perceived split between mind and matter and got me thinking about consciousness existing outside the human brain, in other types of systems operating at different frequencies. My early artworks incorporate the signal generated by the human brain into systems of translation, while my later work focuses on signals originating in the physical world. ! In the following chapters I will start by laying out ideas pertaining to the mind- matter paradigm and how modern technology has shaped our perception of consciousness and physics. I will also discuss my aesthetics surrounding time, which I consider to be my primary medium. I will provide a brief history of the understanding of time and discuss important historical and artistic precedents for my work. Before I discuss my own artworks I provide an overview of the technical aspects of my work including methods of computer programming and signal analysis. 2 02. Consciousness, Perception and the Invisible/ Inaudible 02.1 Chapter overview This chapter addresses the relationship of the human mind to what we call physical matter. I will begin with a personal contemplation of my own consciousness from which I perceive the world and how I see it in relationship to my artwork. To gain a better understanding of the mind-body paradigm, I will look at the implications of Cartesian dualism and its impact on subsequently emerging ideas. Descartes proposition that the mind is non-physical and everything is made of matter paved the way for reductionism: the belief that complex systems, including the human brain, can be described as the sum of their physical parts. The reductionist view of consciousness is that it is merely an epiphenomenon of physical interactions. If the reductionist view is correct, and succeeds in describing consciousness as a complex system, it should be possible to construct an artificial consciousness, or at least intelligence, from a physical system. To challenge the reductionist view, and to provide the argument that the mind is irreducible to matter, I will talk briefly about quantum mechanics and how contemporary physics can help us understand the relationship between matter and consciousness. 02.2 Consciousness I can state that I am a conscious being. I perceive the physical world through my five senses and I can distinguish the senses because I experience them differently and they relate to aspects of the physical world. When I hear something, it is a representation of physical atoms of air vibrating at a combination of frequencies. My eyes do not sense 3 the physical matter of the things themselves but rather the light reflected off of them. ! Like sound, light exists as a wave in a range of frequencies, but in a higher spectrum. The color of a perceived object depends on the frequency of the light reflected off of its surface and my ability to receive that frequency. With extended technological capabilities to sense our surroundings, we can apply a range of tools to measure forces and frequencies outside the perceivable spectrum. These measurements can be visualized and sonified - bringing otherwise imperceptible realities into the range of human perception, enabling us to be conscious of them. ! I consider my work a physical extension of my consciousness. I use methods and tools commonly used by scientists and artists alike to bring invisible and unnoticed things to perception. It is through consciousness that we make sense of both our inner and outer reality, but it is also through consciousness that we shape the material world. This shaping is a collective effort of all consciousness, but our problem is, that in scientific terms, we don't really know much about consciousness itself. It is not yet quantifiable. 02.3 Mind and Matter - Cartesian Dualism René Descartes (1596–1650) is one of the most influential philosophers to emerge in western history. Today, many of his ideas are widely accepted and commonly applied within the sciences. His most fundamental idea is that the world exists as matter in motion independently of the mind, which is considered an immaterial body whose prime essence is thought. The thinking essence that is the I or the self only observes the physical world through the physical senses, which Descartes considers to be deceptive. 4 To circumvent the senses Descartes applies a systematic approach of doubt, where everything that comes through the senses is to be doubted and truths can be revealed only by applying logic, which to him was a property of the mind exclusive to human beings (Smith, Descartes' Life and Work). Using his logic and doubt, Descartes came to the conclusion that he owed his entire existence to his own thinking. His famous proposition Cogito, ergo sum signifies that the only thing we can ever really be certain of is our own thought (Kenny 528). These words establish a baseline from which Descartes derives his subsequent logic; If he were not thinking he would not be aware of his own existence which leads to the conclusion that he is a "substance whose essence is to think" (528). In his 1639 volume Meditations, he states that the essence of matter could be apprehended by innate ideas, independently of any sensory image (Smith). ! In 1633 Descartes was going to publish a manuscript, titled The World, on "the nature of light, the sun and the fixed starts which emit it; the heavens which transmit it; the planets, the comets and the earth which reflect it; all the terrestrial bodies which are either colored or transparent of luminous; and Man its spectator." (Kenny 528) However, he withdrew the publication after learning that "Galileo had been condemned for upholding the Copernican system". (528) Some of these ideas were later published in Discourse, which provides the reader with two key ideas: that human beings are thinking substances and that matter is extension in motion (528). He also included descriptions of planetary movements and laws of physical interactions, and most importantly his geometry, but to avoid condemnation from the church he stated that the observed interactions of matter are bound to the laws of physics, which are enforced by 5 God. For Descartes, "Whenever I perceive something clearly and distinctly, I am assured of it's truth." In this argument, the only thing that can enable him to be a thinking being, and to perceive "clearly and distinctly" is God. He reasons that because the world is perfect, it must include a perfect being which consists of all perfections (529). ! To describe the physical interaction of matter, Descartes described a system where a physical body has four fundamental properties; width, height, depth and motion. He renounced previous ideas that matter exists as fragments of matter in a void and proposed that everything is matter and can be described in terms of these four parameters. Other qualities of objects, such as smell, feel and color are considered secondary properties of the object (Hatfield) thus limiting the need for the senses and further establishing the importance of mind and logic. ! Although Descartes considered intellect to be of a higher order than the physical senses, he did not entirely discard them as useless. Rather, the physical senses are useful for determining if something in the material world is good or bad for your body (Kenny 529). In other words physical senses aid the survival of the physical body. However, Descartes did not attribute logic and thinking to inhabit the bodies of animals and considered them to react mechanically to their surroundings. He introduced the idea that animal bodies are machines that are governed by the laws of matter alone (Hatfield). Descartes also made an analogy of the world as a mechanism, or machine, that is governed by the laws of matter. This paved the way for reductionism in the natural sciences. 6 02.4 Reductionism and the Mechanical Mind At the core of the reductionist world view is the belief that a complex system is nothing more than the sum of its physical parts. In order to come to a full understanding of the physical world, the reductionist strives to compartmentalize complex systems into smaller, atomic units. As the units break down into smaller parts, they are expected to clarify a property of the reduced system. Reductionism remained a generally accepted philosophical aim of philosophy and the natural sciences for a long time, especially in psychology (Gregory, 207). In the period of associationist psychology it was commonly believed that consciousness could be reduced to simple units of sensations or ideas (538). ! In opposition to this atomic reductionist view, Gestalt theory proposed that perceived reality was more than just the sum of its parts. The main criticism is that some phenomena cannot be fully explained by the interactions of physical states and perceived physical states. As an aid to this argument, Gestalt psychologists presented a range of optical illusions to illustrate discrepancies in visual perception (373). One of these illusions, named after Danish psychologist Edgar Rubin, is a silhouette that can either be perceived as a vase or two opposing faces depending on the focus of the viewer, which indicates that perception of the same physical matter is not always the same. (Wikipedia, Rubin Vase) 7 Fig. 1 A single form can be perceived in multiple ways. (Wikipedia, Figure-ground (perception) ) ! Another phenomenon can be demonstrated with a series of simple forms with discontinuous lines. Although the images are incomplete, the shapes they indicate can be perceived seamlessly. 8 Fig. 2 We perceive complete shapes where there are none. (Wikipedia, Gestalt_psychology) ! Despite Gestalt psychologists' attempt to produce a holistic view, psychology started to move more towards reductionism. Instead of the reductionist atomic model, gestalt theory proposed that interactions could be explained in terms of units, or groups of atomic parts. This method essentially abstracts a group of reduced atomic functions into a larger unit, which is then generalized and regarded as an atomic unit (Slowik). ! As reductionism has progressed and succeeded in discovering smaller units of matter we move closer to the belief that consciousness can be explained in terms of neural activity in the brain. Ian Glynn argues that consciousness is always associated with neural activity of the brain. This neural activity can be altered by changing the sensory input or introducing chemical compounds to the brain that can result in a change of consciousness (Gregory 211). ! Although Descartes only compared animals' minds to a mechanical machine the idea was eventually applied to humans as well. Before the construction of the first computer, Alan Turing described a theoretical machine, the Turing Machine, that was 9 capable of switching between states analogous to mental states. This intelligent machine would theoretically be able to function in the same way as a human mind. It would have artificial intelligence. If thought is in fact algorithmic, we could easily come to the conclusion that the brain is a kind of Turing machine, a mechanism of electrical mental states. 02.5 The Brain as Machine The idea that the human mind works like a computer can be traced back to Alan Turing, who wrote about computers and artificial intelligence before the existence of the first modern electronic computer. He described very accurately how the machine would process information - but it was only later that engineers invented the parts needed to build such a machine. Turings' machine, which was a hypothetical machine at the time, consists of a reader head and potentially infinite magnetic tape that rolls through the magnetic reader. The tape would be divided evenly in sections which would all have one of many possible symbols. The symbols would trigger different modes of calculation, each having their own ways of reading the subsequent symbols. For Turing, these states (or programs) were equivalent to different mental states of humans. ! Turing hypothesized that by constructing a logic machine and giving it the right instructions it would be possible to construct an intelligence indistinguishable from a human. Today [N] years later, a form of artificial intelligence is already controlling many aspects of our daily lives - but no algorithm I am currently aware of can perfectly imitate the human mind. In the field of AI the ultimate test is the Turing test, named after Alan Turing himself. The test involves asking the computer a series of questions while not 10 knowing if the reply is coming from the algorithm or a human sitting at another computer. If the AI algorithm responses can not be told apart from the human it passes the test. ! It can not be denied that reductionism facilitated the discovery of smaller units of matter and its physical behavior that led to the creation of new technologies. And as we expand our knowledge of these elements we gain more control over them. Without knowledge of the electron, we would not have been able to manifest Turing's machine into a physical object (the computer). If the brain and computer share some of the same mechanistic language they could both be understood through modern computer language, which first and foremost deals with the flow of electrical signals. 02.6 Consciousness and Physics - Monistic Idealism Recently, a criticism of the dualist mind-matter construct has emerged in the form of a juxtaposition of quantum mechanics and idealist philosophies. Amit Goswami, a contemporary physicist, argues in his book The Self-Aware Universe: How consciousness creates the material world that consciousness is not an epiphenomenon of matter, but that consciousness has an integral relationship with matter. Quantum mechanics has revealed inconsistencies in Cartesian dualism and the reductionist philosophy, and as a response to these problems, Goswami proposes what he calls a monistic idealism. "This philosophy is monistic as opposed to dualistic, and it is idealism because ideas (not to be confused with ideals) and the consciousness of 11 them are considered to be the basic elements of reality; matter is considered secondary." (Goswami 10) ! Modern physics can be split into two main branches: the physics of general relativity, or classical physics, and quantum mechanics. In terms of general relativity, the importance of the work of Sir Isaac Newton can not be understated. He established a framework for the interactions of matter that described the relationship between matter and energy and is referred to as causal determinism. (10) Newton's idea was that "all motion can be predicted exactly given the laws of motion and the initial conditions of the object" (15-16) and includes the important law of energy exchange between objects. An object can only move if you apply a force (or energy) to it, and this energy exchange always has to be equal - every force has an equal and opposite force. Because consciousness, or the mind, was considered an immaterial entity, it was not believed to emit any energy, and was therefore incapable of affecting physical matter. However, it has now become evident that consciousness is affecting the results of quantum mechanical experiments, which to some physicists indicates that consciousness is a definite force that should be incorporated into physics. ! Goswami traces monistic idealism back to Plato and his Cave allegory but also uses terms from Buddhist philosophy. In The Cave, Plato describes a scene where humans are sitting in a dark cave, always facing a wall. The light enters the cave from the outside and casts shadows of reality on the wall. For the cavemen, the shadows would be perceived as reality but would never represent reality in its full extent. The light outside the cave represents Plato's ideals, a concept describing the existence of an idea 12 (i.e. the idea of an object) where it exists in its complete, perfect state. This state is not accessible to us humans, who, for Plato, are trapped in a metaphorical cave where they can only perceive the shadow of reality (48). ! Goswami wants to bridge the gap between the idealist primacy of consciousness and the Cartesian construct of matter using quantum mechanics. To better understand how quantum mechanics can give consciousness a role in physics, we should look at some of the known quantum properties of an electron. ! One of the known properties of an electron is that depending on whether it is being observed directly or measured with a detector, the electron exists in different states. In the double-slit experiment, a common experiment for observing interference patterns, an electron (or a photon, which behaves similarly) is directed at two slits and projected on a fluorescent screen on the other side. A direct observation of an electron reveals an interference pattern on the monitor indicating that it is a probability wave representing possible outcomes. When measured with a detector, the electron appears at a single point indicating that it is a particle. In this state, the electron is observed to be in one place, indicating that it is a particle, but where it shows up is entirely unpredictable (66-70). It also seems to be capable of moving faster than the speed of light, or disappearing from one location and appearing simultaneously at another. This contradicts Einstein's theory of special relativity, which states that the speed of light is always the same and is always relative to the observer, regardless of the speed of the observer (xiv-xv). Not only do these properties challenge Einstein's theory, but they also contradict Descartés geometry and the entire domain of Cartesian dualism, as quantum objects can exist in two places at the same time. For Goswami, these contradictions do 13 not necessarily mean that reductionism and the natural sciences are invalid, but that we have been overlooking the importance of consciousness as a constituent element of the material/physical world. ! At the quantum scale, effect can occur before cause - reversing linear causality. Furthermore, Goswami proposes that consciousness and matter are not separate things but are different aspects of consciousness. They can be considered to cause each other to exist. To further explain this relationship, Goswami turns to concepts in Buddhism: "The material and idea realms are referred to as Nirmanakaya and Sambhogakaya respectively, but beyond these is the light of one consciousness, Dharmakaya which illuminates both. And in reality, there is only Dharmakaya." (50) ! This provides a model of a universal consciousness that illuminates the mind (idea realm) and matter as a unifying force. In monistic idealism these forces are seen as complementary to each other, and cannot be seen as one causing the other (50). This model of the consciousness-matter relationship resonates strongly with Plato's Cave allegory, where the shadows on the cave wall could be seen as an example of Nirmanakaya and the light that makes the shadows could be understood as the equivalent of Sambhogakaya and is perhaps the place of Plato's ideal forms, where things exists as they are. We think that the shadows of consciousness are real beings, when they are in fact a shadow of a larger existence. 14 ! In this chapter I have outlined the history of the relationship of mind and matter from Descartes to Goswami. I have identified the origin of separation of mind from matter with René Descartes and his dualist world view. Although it might not have been his intention, this view developed into a reductionist materialist view where consciousness is not only viewed as separate from matter, but as a direct result of interactions of matter. Reductionism is a useful tool for constructing abstractions of physical interactions, and has allowed scientists to better understand the mind in terms of matter. Such a view enabled Alan Turing to create the analogy of the brain as a machine and to predict the rise of artificial intelligence. In contrast to the materialist world view I have provided a summary of monistic idealism and how quantum mechanics can reverse our view of the mind-matter relationship, where consciousness is more fundamental than matter. ! In the next chapter I will discuss my personal artistic aesthetic in which I apply parascientific methods to investigate the relationship between mind and matter and how we can perceive things differently and move data between mediums. As electronic technology has progressed, we now have the ability to divide time into very small units giving us a very fine resolution on the time-axis. I will discuss the development of the cultural compartmentalization of time from medieval times to the modern age, as my work mostly exists in the time domain. Sound and color can both be explained in terms of frequency over time. Although my work focuses on the qualities of sound, I find similarities in Kandinsky's use of shape and color to represent spiritual values. To further expand on the connection between science and spirituality I will discuss the The Society for Psychical Research and its impact on subsequent telepathy experiments, including 15 the Ganzfeld experiment referenced in James Turrell's work. I will discuss how Turrell in his work methodologically removes the frame of reference exposing the perceptual apparatus of the human mind to facilitate a connection to the spirit. Furthermore, I will discuss how it is possible to expose the same perceptual apparati using new electronic media such as computer programming and brain to computer interfaces. 16 03. Time Aesthetics 'All media are active metaphors in their power to translate experience into new forms.' (McLuhan 69) 03.1 A System Aesthetic The physical components of my work are manifestations of an aesthetic that concerns the process leading to the creation of the work, rather than the material representation of the work. My practice borrows elements from experimental sciences and is elementally parascientific in nature. When I entered the masters program I started focusing on shifting sound into visual forms and bringing it to the viewer's awareness. As my work progressed I began introducing methods for reversing this process; turning objects and images into sound. As with light, we don't often relate sound to physical properties. It is constantly moving through space but we don't normally see it moving through the air. I wanted to find a way to shift audible and inaudible sound to another form where we can perceive it differently, even more analytically. 03.2 Time as medium Time is a representation of the movement of our planet within the solar system which determines the cycle of the seasons. This knowledge was very important to agricultural practices and logically resolves to the definition of the duration of one year that divides into four seasons. In the Middle Ages, monasteries started dividing the day into seven intervals. These intervals were important to maintain a regularity of life in the monastery. 17 The cycles, referred to as canonical hours, were used to "synchronize the actions of men" (Mumford 14) and were of great importance in maximizing the production of the monastery. According to Lewis Mumford, maintaining this cycle made it important to create a device that keeps track of time, resulting in the invention of the first clock. The first clock was most likely a water clock and not capable of great accuracy, but was none the less useful to some extent for this regulatory model of time. A turning point for the social integration of time was in the year 1370 when the first clock tower was built. The clock had "a dial and a hand that translated the movement of time to a movement through space". (16) On every intersection of the hands, a bell in the clock tower would sound over long distances - informing the surrounding urban area about the passing of time. This would eventually integrate the idea of a compartmentalized, sequential time flow, where each segment of time was reserved for certain activities (work, sleep, worship, etc.). Today this idea is still integrated and we measure work as the time required to complete a set of actions. As the intervals between timed events (the bell in the monastery) got increasingly smaller, we started concerning ourselves with hours, minutes and seconds. Advances in the mechanical clock and its derivative industrial machines powered the industrial revolution. This eventually led to the invention of the transistor, the basic on/off switch used in electronic logic circuits, and enabled the construction of the ubiquitous "555" timer circuit. In short, the circuit interpolates between two states, off and on. It is possible to control the rate, making it tick faster or slower depending on the intended application. This circuit is very widespread and is commonly used in home electronics, watches and computers. With the introduction of the modern computer, we can now make logic based calculations much faster than 18 previously possible, essentially outperforming the human brain in certain areas. The basic unit of time used in audio synthesis and analysis is the Hertz and indicates cycles per second. If something has a frequency of 1 Hertz, or 1 Hz, it happens once per second. If the frequency is 2 GHz (or Giga Herz) it happens two billion times a second, which is a very common speed for a modern home computer. ! Jay David Bolter states that man has always worked through time, in the sense that he has been immersed in a larger scale of time which is out of his control, but "modern man is the first man that works with time" (Bolter 101). Essentially, advances in technology have given man more units of time to work with which enables us to perform more actions within each unit of time. Time, like sound and color, has become a medium in which we can make work. 03.3 Kandinsky - Nonphysical forms Wassily Kandinsky saw his art as a way to give form (and color) to anti-materialistic or spiritual values. In contrast, he viewed representationalism as a way of expressing the materialistic values which he shunned (Long 1). Although I don't consider myself to be a painter, and I rarely use color in my works, my aims and methods are not that different from those of Kandinsky. Instead of relying on my inner perception of shape and color, I use light and sound to represent an abstraction of my spiritual experience. ! Kandinsky did not initially see his art as purely abstract and preferred the term "Gegenstandslos" - or 'without object' (33). He stressed that the term applied to forms deriving from "within the artist" (3). In relation to my own work, I interpret this both in terms of reductionist neurophysiology and monistic spirituality. I.e. My works that 19 incorporate the EEG monitor can either be understood as an empirical translation of electronic signals from the brain to waves of sound and light, or they can be understood as an aesthetic translation of spirit to matter. Furthermore many of my works exist partially or fully outside traditional physical form. My works may include physical components, but I see the work existing in the way the components are connected and the continuously changing state of the embedded algorithms. The system is the artwork and the object is the framework for the output. As is the case in Kandinsky's work, my systems derive from within me. Instead of using color (frequencies of light) and shape to represent spiritual values I work with frequencies of sound that I shape through computer programming to create abstract soundscapes representing the inner form I perceive within my consciousness. ! To expand on this particular aesthetic I will borrow the term "tech-noetics" (Ascott 66) from Roy Ascott, an artist and theorist who works in the field of cybernetics and has written about the relationship between consciousness and technology. Ascott states that connecting the mind with technology "enables us to explore consciousness in new ways but may lead to distinctly new forms of art, new qualities of mind, and new constructions of reality." (66) 03.4 Paranormal / Parascience / Telepathy The Society for Psychical Research (SPR) was founded in 1882. The purpose of the society was to establish a foundation for rigorous and open-minded examinations of psychical phenomena, but today the term psychical has been replaced with paranormal. One of the founders of the society and author of Varieties of Religious Experience, 20 William James, did extensive research on consciousness. He experimented on himself using nitrous oxide, which would induce a different state of consciousness, and he was the first psychologist to conduct experiments on telepathy. One of the co-founders of SPR, Edmund Gurney, was also involved with experiments on telepathy, and was convinced that communications were possible with the spirits of the dead. ! In the 1930's, German gestalt psychologist Wolfgang Metzger designed the Ganzfeld experiment. Its purpose was to test the binocular theory of vision. He was attempting to prove that the mind sees one unified image and not two separate images, but his work was later adopted as a telepathy experiment. In the original experiment, the eyes of the participant are covered with a white material, a red light shines through the surface resulting in the exclusion of all surrounding visual stimuli. As an addition, auditory stimuli are also removed. After a while, the participant's mind starts producing singular images, proving that vision happens in the brain - rather than in the eyes alone. This experimental platform was later appropriated by paranormal researchers to conduct experiments on thought-transference between two participants. This technique has served as a great inspiration for my practice, both in terms of producing artwork that stimulates mental imagery and in terms transferring my mental state to my viewers through technologically enabled thought transferance techniques. The experiment has also influenced other artists such as James Turrell, whom I will discuss in the following section. ! A similar, but perhaps more powerful tool for experiencing the inner mind, is the sensory deprivation tank. In the sealed environment the participant floats in complete silence and darkness in a highly concentrated salt-water solution that enables effortless 21 floating, canceling out tactile feedback as well. In a recent study conducted in Sweden, patients with chronic-whiplash pain were treated on a regular basis in this environment. The study reveals that participants experienced being in a different place or time and made profound connections with the self, resulting in less pain for a long period of time after the treatment (Edebol, Bood & Norlander) ! Sensory deprivation is a powerful technique to examine your own mind. The Ganzfeld experiment only flattens the visual aspect of perception and leaves the other senses untouched. In a floatation tank, all references to the external world are removed and you can look at your own mind without interference from external stimuli. This can be seen as a method for exploring mind and consciousness, without interference from matter outside the body. After a floating session, people often report experiencing timelessness, a conscious state where time does not exist. This is perhaps the equivalent of the mystical state. 03.5 James Turrell and the removal of visual reference James Turrell works with light in a way that reveals its physical properties and he uses elements of sensory deprivation in many of his works. By removing the frame of reference he gives the viewer an opportunity to observe light independent of our usual perceptual habits and conventions. ! In Turrell's Dark Spaces the artist appears to have constructed a completely light proof structure. When the viewer enters the work he perceives only darkness. However, Turrell leaves a tiny fracture in one of the closing walls to let a small amount of light in. After a while, the viewer adjusts to the darkness and the tiny amount of light going into 22 the installation becomes clearer and brighter until it fills the space. "The concept of a Dark Space is not about what one is supposed to see but the experience of what Turrell describes as 'seeing yourself see.' " (Turrell) ! To me, this demonstrates two things; The persistence and power of light, and that changing the frame of reference can have dramatic impact on what we perceive. In another series by Turrell, Skyspaces, he brings the sky closer to the viewer by cutting a square or round hole in the ceiling, exposing the sky above. Normally, the hole in the roof would cast a shadow on the side of its edges - creating a frame of reference from which to view the sky. However, Turrell removes this shadow by making the edges so thin that you can not see the edges, eliminating the frame of reference. This results in the sky being perceived as part of the roof, filling the hole in the ceiling - just out of reach. In both of these works Turrell shifts the daily perspective on light and binds it to the physical dimensions, making it seem that you could literally touch the light. In the Ganzfeld series, Turrell creates a series of spaces with controlled lighting. The rooms are lit with a single color in a uniform configuration. This has the effect that the viewer looses all depth reference and can bring the viewer closer his inner perception, as is the case with the Ganzfeld experiment. ! In his works, Turrell turns our regard toward the mind as a perceptual apparatus. In the following segments I will talk about how we can interface this spiritual apparatus that is the mind ( spirit , consciousness ) with computers and other electrical devices. To do so, I will first address the significance of software in modern culture and the computer programming aspect of my own work. 23 03.7 Brain to Computer Interface (BCI) Alan Turing introduced the metaphor of the brain as a computational machine that had internal states like the Turing machine. For Turing, these states were synonymous with the mental states of the brain. Today, neuroscience has determined that the brain fires electrical signals through units called neurons and a correlation has been discovered between the frequency of this electrical activity and certain mental states. Using this fact, and the proper electrical equipment, it should be able to interface with a digital computer. ! The raw electronic signal originating in the brain produces a waveform that can be treated in the same way as an audio signal. The frequencies of the wave are divided into four frequency bands, where each one is associated with a number of mental qualities as determined by experimental observation. The waves and their main characteristics are as follows: Frequency Band Range (Hz) Characteristics Delta 0 - 3 Hz Deep Sleep Theta 4 - 7 Hz Dreaming Alpha 7 - 13 Hz Meditation, Relaxation Gamma 13 - 30 Hz Waking state, Awareness, Stress Table 1. Brainwave frequencies and associated mental states. (First, 31-37) ! Delta waves occur in deep, dreamless sleep at a very low frequency. This indicates the brain is processing information in slower cycles, conserving energy. As the cycles grow faster there is an increased activity in the theta band and we experience 24 dreaming. The most well known frequency band is alpha which is dominant in the brainwave signal when we are in deep relaxation or in a meditative state. This state is believed to enable greater creativity, perhaps because it operated closer to the subconscious frequencies (delta and theta). In a waking state, the brain fires electrons at a much faster rate, producing a strong signal in the gamma band, but little is known about frequencies occurring above that limit. ! In some of my work I implemented the use of a commercial EEG monitor. This device goes by the product name MindWave and was developed by an american company called Neurosky. The device itself is lightweight and transmits data to the computer wirelessly. An electrode is laid flat on the forehead and an embedded circuit amplifies the electronic signal originating in the brain. Neurosky have implemented their own algorithms that determine the users level of attention and relaxation, termed Attention and Meditation values. The values of the respective variables are provided as integers ranging from 0 to 100. This is intended to provide game developers with useful controls without having to do complicated signal processing to determine different mental states and is achieved with Neurosky's undisclosed algorithms. By getting visual and/or sonic feedback generated by your mental state, you can learn to control and manipulate these particular states at will. The device also transmits the raw signal from the electrode which gives you more information about the electrical activity in the brain but is not nearly as useful as a controller. This signal is however the same one used for the construction of the attention and relaxation values. ! In 1965 Alvin Lucier composed Music for Solo Performer, which is considered to be the first brain controlled artwork. In this work, Lucier performs on stage sitting on a 25 chair. Electrodes on his forehead pick up the signal of his brain and send it to an amplifier. The amplifier filters the signal leaving only the alpha waves to be heard through the speakers. By entering a meditative state, Lucier makes the waves grow stronger eventually created a resonance with percussion instruments lying around the stage. In essence, by combining the brain with technology available to him at this time Lucier found a way to play instruments only with his mind, achieving a form of electronic telekinesis. ! Now that we have established that the brain emits a range of electronic frequencies that have a correlation to certain mental states, I want to address the possibility of reversing this process to affect the viewer's mental state by sending electronic signals to the brain through the senses. Biofeedback, a known technique, is the practice of amplifying biological rhythms, making them audible or visible (or both). This is commonly used to experience ones own. I wanted to explore if the biological rhythm of one persons brain could affect another person through a similar unidirectional process. 03.8 Sensory input Binaural beats is a phenomenon first observed in 1839 by prussian physicist Heinrich Wilhelm Dove. He observed that when listening to two slightly different tones in each ear the brain would sense a third wave. This wave is the difference between the two tones. I.e. a 400Hz and 401Hz tone would produce a perceived 1 Hz tone. (First 31-37) The lower range of human hearing is about 20 Hz, but using this technique you can enable the perception of lower frequencies. Because the low frequency wave is not 26 physically present, it means that the brain is generating it on its own. By exposing the brain to alpha waves (~7Hz) the brain can be guided into the alpha state of consciousness (First 31-37). This can also be applied to the other frequency bands of the brain to amplify certain mental states. I have used this technique in some of my previous work, but I believe the same effect can be achieved through physical interaction of low-frequency audio that affects the entire body accompanied with light stimuli at the same frequencies. ! In this chapter I have talked about my aesthetic and discussed artistic and historical precedents for my work. I have discussed the importance of software and programming and how the brain can be interfaced with a computer, and how the internal states of the brain can be programmed by sending it signals representing these states. In the next chapter I will discuss some of the specific methods I use to achieve this, before I move on to discuss my own works. 27 04. Parascientific techniques: Software, Visualization, Sonification and Fractal Mapping 04.1 Computer programming Computer programming is a central aspect of my workflow and I use a range of programming languages to achieve my goals. Each language has its own strengths and weaknesses. A useful analogy is to think of each language, or programming environment, as a specific tool useful for a specific task. Here I will discuss the programming languages I use the most and what they enable me to do. Some programming languages tell the computer very specifically what to do on the hardware level and are considered to have a low level of abstraction. A language that compiles low level code into discrete functions are considered to be at higher levels of abstraction. As a result, higher level languages are easier to code in and you can get away with writing smaller blocks of code. The downside is that your code is probably instructing the computer to do more things than are absolutely necessary. I do not expect, nor require, that every reader of this thesis has a knowledge of computer programming. However, I will briefly talk about some general aspects of the programming tools I most commonly use for my work. Every now and then I might have to use something different, but these are the most important ones. ! Max/MSP is a high level visual programming language developed in the early 1980's. The structure of a visual program is similar to that of an analog synthesizer; functions are written into a module that obscures the text-based computer code. Each module has one or more inlets and outlets so it is possible to route data in and out of 28 objects. An example of this would be the cycle~ object which takes two arguments in its inlets, frequency and phase, and by connecting a number to the frequency inlet you set the frequency of a sine wave and the same is true for the phase. The cycle~ contains an algorithm that uses these variables to generate a sine wave audio signal. In my work i have used Max for audio synthesis, analysis and signal reconstruction. ! Processing is an open source programming language built for artists and designers. It has a high level of abstraction and is designed to quickly make graphical programs called sketches. It has support for creation of basic shapes, image and video processing, and is capable of rendering graphics on the computer GPU (Graphics Processing Unit). Because Processing is a high-level language, when programs get too complicated and resource-demanding the program will slow down significantly, it is mostly useful for prototyping small projects. ! Another useful open source library for visual artists is called openFrameworks. It is very similar to Processing but is built with C++. It has a lower abstraction level, so the code is essentially closer to the hardware which makes it necessary to consider the physical aspects of memory. I.e. you have to be careful to think about memory in terms of the magnetic tape of the Turing machine, but instead of an infinite magnetic tape containing zeros and ones, the computers memory has a finite number of bits it can remember. Assigning two different tasks to the same place in memory will therefore break the program. 04.2 Data sets For any visualization or sonification process it is essential to have a set of data to work with. In my work, I use a few different sources to construct the data I am interested in. I 29 acquire my data through audio recording, brainwave recordings and known mathematical sets found in nature, such as the fibonacci sequence discussed in chapter n (Natural Sound Synthesis Algorithm). 04.3 Fourier-transformation In the early twentieth century, french mathematician Joseph Fourier found a way to reduce waveforms to a list of individual frequencies with corresponding amplitudes over time. The list extracted from the waveform can then be put back together to create the exact same waveform. Furthermore, because everything moves in a wave - or from one state to the other and back - the transformation can be used to predict events forward (and backwards) in time. Russian neurophysiologist Nikolai Bernstein applied the principles of Fourier transformation to the movement of dancers, dressing them in black clothes against a black background with reflective markers on the body. By photographing only the reflective markers at regular intervals he captured the waveform of the dancers and was subsequently able to transform these waves to a list of frequencies and amplitudes. Using Fourier's technique, Bernstein could predict motion with incredible accuracy (Talbot, 28-30). This type of transformation is now commonly used in mathematics and engineering, but it is perhaps more commonly used in audio synthesis and analysis. However, Fourier transformation can be applied to almost anything. ! A common application for the function is the construction of spectrograms, where the X-axis represents time and the Y-axis indicates the frequency range. The result is an 30 image that maps the frequency amplitudes onto pixel values and their corresponding image coordinates. 04.4 Waves The basic building block for all my work is the sine wave. It is characterized by a smooth continual shift from its negative value to its positive value and is the simplest of the basic waveforms. ! Every object has a fundamental frequency. If exposed to that particular frequency the object will start to resonate - causing it to vibrate in phase with the outside signal. Objects also resonate at other frequencies which are often, but not necessarily, multiples of the fundamental frequency. As a result, objects resonate with a list of N frequencies at variable amplitudes. As a contemplation of how this affects everyday life, I present an analogy from the physical world. ! When you look at the ocean and try to count the waves, you might get confused about what exactly constitutes a countable wave. Waves move through each other, making it difficult to distinguish where each wave ends and where it begins. You notice that a single wave in the ocean is made up of many smaller waves, or ripples. When you look out to sea, you can see the larger waves (lower frequencies), pulsing gently (usually at a low amplitude). Eventually, time reveals the lower frequencies of the ocean, the tides which go from its lowest to highest state twice a day as the moon orbits the earth. If we applied Fourier transformation to the waves at sea, surely we would acquire an accurate description of the ocean. Instead of counting only the waves that we perceive to be waves without much consideration we might end up with a single 31 number over a fixed amount of time. Instead we have now counted how many times waves of different sizes occur. This method accounts for objects in motion, but can the same principles apply to static objects? ! In my work I look at the spectral distribution of audio signals recorded in different environments; traffic intersections, the beach, the forest, etc., but also how the physical objects' presence can change the spectral distribution of its surrounding environment. 04.5 Comparative signal analysis In some cases, it is enlightening to look at the difference between two signals. For instance, recording the sound of a "silent" room will produce resonant frequency bands determined by the physical properties of the room. If you then make another recording of the room with people talking in the room, subtracting the former signal from the latter produces a signal with minimal room resonance, thus isolating the talking from the background noise. This is the equivalent of removing the room from the signal. ! Another way to detect the resonance of an object is to expose it to white noise. A white noise signal consists of randomly generated frequencies at equal amplitudes. Because each frequency has an equal probability of appearing the signal contains all possible frequencies. When an object, or a space, is exposed to white noise it resonates at certain frequencies but rejects (filters) others. The generated filtered signal shows us what frequencies the object is not resonating at, therefore a simple inversion of the signal reveals its resonant frequencies. 32 04.6 Mapping and Remapping A useful method of transforming data across mediums is called remapping. This method does not refer to physical or geographical mapping, but is used to describe the mapping of one range of numbers to another range. We can use this method to transcribe information from one form to another. One possible use of this method is to remap a recording of high-frequency inaudible sound waves onto a portion of the audible spectrum. Another use I found for this technique is to remap my brainwave activity into the audible spectrum. This method is also viable across mediums, i.e. translations are possible between sound, light, electronics or anything else that can interface with a computer. 04.7 Fractals and Fibonacci sequences ! A fractal shape is a shape which contains itself and is repeated n times within itself. In 1915, Polish mathematician Waclaw Sierpinski discovered a fractal triangle pattern now known as the Sierpinski triangle. The simplest way to draw one is by a simple recursive method; draw an equilateral triangle and fill it with black, then from the centre of each side of the triangle draw a smaller triangle inside and remove the black from within it. Each time you do this the triangles will get smaller and smaller. 33 Fig. 3 Sierpinskis triangle rendered with Processing (http://www.openprocessing.org/ sketch/43184). ! A more interesting way to draw this exact configuration, as discovered by Sierpinski, is based on random probability. You begin by defining the vertices of the equilateral triangle and then select a random point within the triangle. Starting from that same point, move halfway towards a randomly selected corner of the triangle. If repeated enough times the same exact pattern will reveal itself. To me it is interesting that the same results can be achieved using both constructed logic and random probability. ! It has long been established that certain mathematical constants have a pleasing aesthetic appeal. Many of those constants constitute the building blocks of the natural world. (Huntley). One of those numbers, Phi, or the mathematical annotation ϕ has been of a particular interest to me. It is the relationship present in any sequence where each number is the sum of the two preceding numbers in a list. It has been considered 34 to be the 'golden ratio' but it also represents the growth and shape of spirals and is found in most biological entities. In music, a major sixth is considered the most emotionally pleasing interval. It has a ratio of 8:5 which approximately equals ϕ:1 (Huntley). Fig. 4 A fibonacci spiral. (Wikipedia, Golden_ratio) Phi and the Sierpinski triangle served as a model and inspiration for my Natural Sound Synthesis Algorithm, in which I construct a fractal audio composition based on Fibonacci sequences. In the following chapter, I will discuss the process of the creation of my artworks where the aforementioned methods are applied in practice. ! In my work, the above methods and tools are applied in different configurations, or compositions. Computer programming is one of my primary tools that enables me to describe abstract, non-physical ideas in computer language. These ideas exist in a vaguely physical state as a set of instructions in the computer's memory. These instructions are implementations of the above methods. In my work, I instruct the 35 computer to process signals through methods of mapping and remapping combined with fractal signal reconstruction. In my programming, I look at input and output in terms of its embedded frequencies. For instance: physical objects, to me, can be uniquely represented by a list of frequencies and their respective amplitudes, determined by their physical properties. ! In the following chapter I will discuss my artworks in terms of the development of work and ideas. The works implement the aforementioned methods in different ways but are crucial for the construction of a complete system in each individual work. Now that I have laid out some important components of my artworks I can adequately discuss how my methods are implemented in practice and theory. 36 05. Artworks In this chapter I will walk through developments of the artwork produced over the past two years. I will discuss my artworks in chronological order, as each work informs later iterations of what I consider to be part of the same body of work. The first work I did in the program, BrainTV, is an exploration of the Cartesian relationship of mind and matter, particularly how the former can manipulate the latter through the use of technology. In the creation of this artwork I worked under the assumption that consciousness can be measured, to some extent, in terms of the frequency of the electronic signal originating within the human brain. Certain patterns in the frequencies and amplitudes embedded in the signal indicate certain mental states. By using an EEG monitor I was able to transmit a signal from my brain to the computer and translate it to a signal that creates interference patterns on the TV monitor, essentially transforming my brainwave signal into waves of light. The experience of looking at a physical object (the TV) responding to changes in my own mental state led me to question whether this tech-noetic relationship could be extended to interface with physical matter by using sound. This lead me to explore the acoustics of different places in my subsequent artwork Observations, where I collected and visualized audio recordings acquired in different environments. In this work I view the audio signal similarly to the signal generated in the brain. It can be analyzed in the same way, but it operates in a wider frequency spectrum. This allows for the transformation of the audio to an image, commonly known as a spectrogram. The resulting image shows amplitudes in a range of frequencies over time. At the time I made these recordings I noted my own mental state and when I looked at the image I could then make a similar correlation between my mental state and the composition of 37 the image. I had the idea that I could analyze audio recordings in the same way and reveal similar compositions in the signal generated and processed by places and objects. Perhaps consciousness is a result of relative frequency bands composed in a variety of combinations. I was interested in whether some kind of consciousness could be found in larger systems around us. To further explore this notion, I started developing Thingogram v.1, a method to record and visualize the acoustic frequencies embedded in arbitrarily selected objects. Using this method I was able to produce images of the object's dominant frequency bands showing that each object produced a unique frequency band composition. The next iteration of the work, Thingogram v. 2, is a method of translating the images into audible form. This is achieved by mapping the image onto an array of sinewaves spaced by intervals of fibonacci sequences. As I now had a collection of visual and sonic abstractions of the objects from v.1 I proceeded to build Thingogram v.3, a framework for viewing these images while listening to the sound they generate. ! This thesis includes a few images of my work, but more material is available in my virtual studio (http://gradstudios.ecuad.ca/lstefansson/) including audio components of my artworks. 05.1 BrainTV The first work I did in my journey through the masters program is called BrainTV, and is my attempt to interface my brain with a television. In this work I rely heavily on the reductionist philosophy that the mind is an epiphenomena of matter, and that consciousness has a correlation with electrical activity in the brain's neurons. I see this 38 as an occurrence of Ascott's tech-noetics, as the state of the work has a direct temporal relationship with my own mental state and/or consciousness. The work becomes an extension of my mental state, which I strongly associate with my consciousness. Thus, I have established a link between my consciousness and the temporal state of the television (a physical thing). The television responds in real-time to changes in my brain activity. ! To acquire a reading of my brain's electrical activity I use the MindWave EEG monitor discussed in chapter 3. The device amplifies and transmits the electrical signal of my brain to a computer running a program of my own making. This program receives the signal in the form of a RAW signal and the Relaxation and Meditation values as provided by Neurosky, the manufacturer of the device. In the first iteration of BrainTV I use the Relaxation and Meditation values as these are intuitive and provide very good, but simplified, feedback information that I can correlate to my mental state. In a later iteration I rely solely on the RAW signal. ! In the first iteration of the artwork, BrainTV v.1, the program that interfaces the brain to the computer contains an algorithm of my own making that generates an audio signal that changes in accordance to the simplified mental state values. The signal is then routed to the video input on the television which results in different combinations of horizontal lines of light, depending on the composition of the audio signal. ! I discovered that when I send an audio signal to the video input of the television it displays an interference signal on the monitor. By sending a range of frequencies to the monitor I discovered that a signal of ca. 60 Hz would produce a single still line across the monitor. This is the fundamental frequency of the television, which I will refer to 39 simply as F. By raising or lowering the frequency of the audio signal by N Hz the line will start to move across the screen at F-N / N Hz. It will move from top to bottom if it is a negative value, and in the reverse direction if the value is positive. Furthermore, I discovered that values that are multiples of F will produce multiple lines across the screen in a 1:1 ratio. 2F will yield 2 lines, 5F will yield 5 lines etc. It is also possible to control the appearance of the lines to some extent. A signal with a high amplitude results in hard edges, where as a low amplitude will render softer line edges. In this iteration of the work, I wanted a steady single line to occur only when I was in a focused, meditative state. As my mind would drift towards stress and loss of focus these states would make a more complex and distressed pattern on the monitor. Through my awareness of the fundamental frequency and other characteristics of the television set, I was able to map the range of mental states detected by the EEG device to a range of compositional complexity displayed on screen. This is a process of tuning together man and machine. 40 Fig. 5 BrainTV displaying a complex signal. ! Obviously, there is no technically correct way to do this, but instead I went through a process of comparing what I saw on the monitor and assessing how well it represented my current mental state. If the light on the monitor was not felt as adequately representing my mental state, I made the necessary calibrations. This is perhaps where my personal sense of aesthetics and subjectivity is best represented. As a guideline to this calibration process I reference the frequency bands of the brain and their associated mental states. When a viewer looks at the television, light hits the retina at the frequency emitted by the television. Similar to how binaural beats trick your brain 41 into perceiving a physically non-existent audio frequency, I believe the same results can be achieved by using light, or any other sensory stimuli for that matter. Fig. 6 BrainTV installation view. ! I presented this work in the ie gallery on the Emily Carr campus over the period of one week and placed the head-sized television monitor on a high-chair sitting on a plinth resembling a stage. For the exhibition period, I wore the EEG monitor and transmitted the signals over the school's network - in a sense broadcasting my mental state to the gallery in real-time. As I was physically elsewhere, the work continued to transform my mental state into interference signals. The television becomes a physical extension of my mind. This is essentially a form a telepresence that poses to some difficult questions on materiality and locality of the mind. Such as "Is it possible for the 42 mind to be in two places at once?", and "Is the mind a product of physical interactions, or are physical interactions caused by the mind?". ! In the second iteration of this work, BrainTV v.2, I used the same technical setup as in v.1 but the program is now feeding the raw signal through a different audio synthesis algorithm. This algorithm remaps segments of my brainwave activity frequency spectrum to frequencies in the Solfeggio scale, a musical scale used in Gregorian chanting. ! Because I am working with a signal in the range of zero to 30Hz, two thirds of the data is outside the audible spectrum. For this reason, it is not enough to simply amplify the signal to hear it more clearly. To make the signal audible, I multiply the amplitude of sectioned frequency bands with select carrier frequencies. The result is that you hear the amplitude of different frequency bands of the brain reflected through audible tones at a higher frequency. In order to achieve this, the carrier frequencies have to be selected manually which ultimately comes down to aesthetic choice. My decision to use the Gregorian Solfeggio scale comes down to its spiritual applications and to the idea that it could possibly have healing effects. The exact frequencies of the tones in the scale are mathematically consistent and have been associated with geometric formations. ! Using this method, as opposed to the previous one, the translation algorithm allows for an experience of the brainwave signal that is much closer to the original. My intent in this work is to broadcast my consciousness, or mental state, to the viewers/ listeners of the work. The television becomes an extension of my physical (biological) and conscious self and is capable of affecting anyone who views it. Experiencing my 43 own consciousness extended into an electronic system made me doubt the physical limitations of consciousness, and perhaps consciousness could occur in other electrical networks than the human brain. This led me to look for consciousness in the external world through audio recordings. 05.2 Observations Observations is a work in which I start my explorations of environmental soundscapes as a visual construct. The series consists of 12 images, each one representing a specific time, location and events occurring over an undetermined period of time. The content of the images is collected in the field with a microphone and a portable recording device. The sound is then converted into a spectrogram, an image that shows the amplitude of a range of frequencies over time. Fig. 7 Observations no. 3, Waves. 44 The images present themselves without the axis labels attached to them. This is to encourage the viewer to observe each image not in relation to the empirical scale, traditionally sitting next to each axis, but to look at the emergent frequency bands' relative position to each other on an aesthetic level. What is significant in this work is that it shows how soundscapes differ from location to location, or event to event and how the feeling attached to the recording translates between mediums with its aesthetics in tact. For instance a busy traffic intersection produces a 'heavier' image and leaves little space for anything else but an image produced by birds singing has a much more appealing aesthetic and looks more like traditional sheet music. Fig. 8 Observations no. 4, Traffic structure. 45 Fig. 9 Observations no. 7, Birds and breathing. ! In this process my control over aesthetic output is very limited and is ultimately a temporal reflection of the world around me at a time period unknown to the viewer. The only time indicator I provide is the title, describing a scene or an action. These include "Chasing a squirrel", "Waves" and "Traffic structure". By looking at the visual component of the events around me I can later contemplate acoustic events, and their respective frequencies, that would normally go unnoticed. When I record the waves, I also shift my attention to them. And even if I listen to the audio recording later, my focus might still be on that same aspect of the environment. However, if I look at a visual representation of the soundscape I can see that what I perceived was only a small fraction of events. In a sense, directing your attention to one aspect of the environment filters out "unnecessary" stimuli, as is the case in Turrell's work. In terms of data processing, this 46 could be considered to be a significant data loss. If this filtering is necessary for consciousness to make sense of what it perceives, maybe applying a degradation process to the recordings would reveal something more comprehensible by human consciousness. ! If we track the source of the audio signal from its origin to the output image, we can see the level of information loss involved in the process. The movement, generated by conscious beings or otherwise, triggers changes in air-pressure which the human ear detects as sound. The microphone reacts to this type of change in pressure and converts it into voltage that is recorded electronically. The microphone is bound to an angle and can only detect changes at frequencies within a limited range. This range is far greater than the commonly acknowledged range of human hearing but the recording is undeniably distorted to some extent by imperfections/characteristics of the recording device. To produce an image from the recording we need to apply a Fourier transformation to get information about frequency and amplitude. This information is used to plot an image a two dimensional image where pixel density represents amplitude on the time (x) and frequency (y) axis. When I have an image, I apply basic image processing tools. By increasing contrast and adjusting brightness levels I get a simplified image, but at the cost of a significant data loss. At this point only the signals with higher amplitudes are revealed, discarding information about subtler events. To add an additional layer of information loss I print the image with black ink, causing dense pixels to bleed to nearby areas resulting in an even greater contrast. In the final step the image is then photocopied several times. In this I am emphasizing the data loss that happens in human perception when we shift our attention to something specific, 47 disregarding something else. The imperfections of our sensory apparatus filter the sensory input to the degree that most things go unnoticed, but what remains is the perception of the world. By applying a systematic information loss through the use of technology I am imitating the human perception mechanism through technological apparatuses. Perhaps, having enough data loss will reveal something in the event that went unnoticed before. This method of systematic dataloss essentially filters out subtle events and amplifies larger events in favor of a clearer image result. This reveals the dominant frequencies of the examined soundscape and makes it easier to visually notice distinct events in the image of the recording. 05.4 Thingogram Thingogram is a series of works that explores acoustic properties of objects. The general idea is that every physical structure can be uniquely described as a list of frequencies with corresponding amplitudes. In Thingogram v.1 I analyze objects the same way as I would analyze the brain. Similarly to how the brain produces certain patterns for certain conscious states, physical matter produces a certain pattern for a certain physical state. If consciousness really has an integral relationship with matter, as is the case in monistic idealism, could these frequencies help us understand the translation process between mind and matter? Perhaps the frequencies emitted by consciousness are directly affecting the state of matter. To me, this seems reasonable because objects of our own creation cannot exist unless we think of them first. ! In the first version of Thingogram, I focus on the construction of a method to extract lists from different objects and the translation of the lists to an image. This image 48 would reveal the pattern unique to each object. In the second version, I construct an algorithm based on fractals and fibonacci sequences that takes an image from the previous version as an input and outputs an audio representation of the spectral image acquired from the object. In the third version of this ongoing series, I built a physical framework for the components of v.1 and v.2. This version is a collection of spectral images of objects and the sounds they generate. Here, my challenge was to give the content a viewing platform. My initial approach was to make a book with an interactive audio component but this evolved into a portable collection of images and embedded systems that support the audio playback for the sounds produced by each image. 05.4.1 Thingogram v.1 Thingogram v.1 is a method to look at the frequency spectrum of physical objects in a controlled environment. When selecting the analyzed objects I considered a number of aspects. Each object needed to be below a certain size to fit the physical dimensions of the experiment and have a distinct shape and texture relative to other selected objects. I am certain that the experiment can be scaled up to allow for analysis of larger objects, but here my primary interest is the relative difference in the frequency spectrum of different objects. ! Keeping in mind that anything only exists relatively to something else and is dependent upon its environment to be meaningful I knew I had to find a way to detect the change induced to the environment introduced by the object in question. I did this by creating a setup in which a speaker and a microphone are located on opposing ends of an empty space. Knowing that audible sound operates in a range of measurable frequencies, I instructed the computer to send a white noise signal through the speaker. 49 It is a special kind of signal, because it is composed of all frequencies at equal amplitudes. White noise is commonly associated with the static on a television monitor, or the buzzing sound a radio makes when it is tuned in between radio stations. When we either listen to or look at white noise, we perceive no significant connection between what seem to be randomly generated points. When the white noise signal travels through the empty space and hits the microphone the signal has been distorted to a degree by the surrounding space the equipment itself. For this reason I record the signal without an object in the space to use as a comparison signal. Fig. 10 An accordion filtering a white noise signal. Now a second recording is made in the exact same way, but with an object inhabiting the space. When the white noise signal travels through the space, some of the 50 frequencies are blocked by the object affecting the recorded signal. Now both recordings are converted to images (spectrograms) and the image representing empty space is subtracted from the image depicting the object in the space. This effectively subtracts the empty space (and other factors) from the object in the space, leaving only the frequency bands affected by the object. The result is an image that shows frequency bands absorbed by the object. When the process is repeated several times, placing different objects in the empty space, the produced images reveal clear spectral differences between each of the objects. It is evident that similarly to conscious states, physical states can also be described in terms of frequency and amplitude. In fact, mental states as understood here are physical states. Could there be a way to affect these physical states with the mind through the use of technology? 51 Fig 11 A spectral image of a hat. 52 Fig. 12 A spectral image of an accordion. 05.5.2 Thingogram v.2 At this stage I had developed a process to visualize the acoustics of an object but I had no way of listening to what the objects sound like. In order to achieve a way to do this, I built an algorithm that reads images and translates individual pixel location and density into an equivalent frequency and amplitude of a sinewave. Fourier transformation can be applied to extract data, but it is also possible to reproduce the original data using only the basic building blocks of the original. Building on this idea I started to program a hierarchical system of sine waves. The original approach was to create one sine wave for each specific frequency in the audible range - so I would have to instruct the computer to create 20.000 individual sine wave signals and adjust the amplitude of each 53 one according to pixel density in the Thingograms. This is a very high number of operations that needs to be calculated simultaneously and would require too much processing power. ! To address this issue, I started constructing an algorithm based on fractal Fibonacci sequences I am calling the Natural Sound Synthesis Algorithm (or NSSA). Considering that humans are biological entities that share mathematical constants with the rest of the natural world and that these numerical relationships are known I concluded that this would serve as a basis for encoding spectrograms into sound with a more naturally sounding result than a traditional Fourier transform. Instead of instructing the computer to create each individual frequency, I instructed it to only generate frequencies that are multiples of Fibonacci sequences. This way, the frequencies are distributed along a natural path and despite the resulting signal containing fewer frequencies the result would sound more natural. ! The resulting program, written in Max MSP - a visual programming language, contains 2000 sine waves instead of the initial 20.000 needed for accurate reconstruction. The results are very pleasing to listen to, as expected, and the sound has a surprisingly rhythmic quality. To me it sounds like a constant rhythmic acceleration, where each beat contains several more cycles within. When the cycles overlap they produce a magnitude of interference frequencies, producing a more complex signal. The images produce different sounds, but they are all variations of the constant rhythm inherent in the fractal fibonacci sequence. 54 Fig. 13 A screenshot of the NSSA algorithm. 55 05.5.3 Thingogram v.3 Fig. 14 Thingogram v.3 installation view Now that I had developed the image and audio components of the work, I wanted to compile them into a portable collection. My intent was to give the viewer a chance to interact with the work and discover the relationship between visual and sonic elements of the documented objects. Furthermore, I wanted to convey the feeling that the apparatus was thinking and possessed some kind of consciousness or artificial intelligence. ! The artwork is installed into a flight-case, traditionally used to ship audio and music equipment. The cuboid case is built with reinforced corners and edges that 56 protects the work while it travels to it's next site of exhibition. It is built to fit 12" record albums in one side of the box and all the necessary electronics in the other. The electronic components of the work handle the interaction with the viewer but are concealed by a plywood cover engraved with Sierpinski's triangle. The only indicators that anything is hidden within the box are the headphone and video outlets on the front of the case and the power socket on the backside. Fig. 15 Thingogram v.3 flight-case detail. 57 In addition to headphones, a cable runs from the video outlet to an old Commodore 64 monitor. The monitor instructs the viewer to "Place an image on the platform to listen to it", vaguely suggesting that the viewer places an album cover on the fractal triangle. The fractal pattern engraved on the platform points to the structure of the audio signal. Fig. 16 Instructions on the monitor. The images on the albums are components from Thingogram v.1. They are spectral representations of an object. When an album is placed on the viewing platform the 58 monitor starts to intermittently display text-based information about the object depicted on that particular album while playing a fractal audio translation of the image. ! Thingogram v.3 functions very much like a traditional record player. It has a compartment containing a collection of records and a separate platform for playing them. I chose to engrave the listening platform with Sierpinski triangle because it symbolizes the relationship between structure and randomness; the randomly generated white noise signal used to record the depicted objects and the structured output of the NSSA. When the viewer places an album on the centre the triangle, an antenna hidden underneath the platform detects a radio-frequency ID (RFID) tag embedded in in center of the enclosed 12" record. An internal Linux computer (RaspberryPi) detects the image ID and plays the corresponding audio while the computer monitor displays pieces of technical information about the object. A new line is displayed every 2-3 seconds and the screen is cleared when the audio stops playing after about 17 seconds. The first few lines are technical observations of the object but this evolves into subjective and parascientific remarks. This is my attempt to guide the viewer towards the idea of artificial intelligence and evoke thoughts about consciousness in the artwork. ! Although I am satisfied with the outcome of this work, there are some aspects that I would like to improve. One problem I faced was that my NSSA algorithm was incompatible with the Linux operating system I was relying on and could not be incorporated in its current state. For this reason I had to use recordings of the generative process rather than generating the audio signal in real time. However, the algorithm always produces a consistent signal from each image so this does not change 59 the audio component . In future revisions of this work I want to find a way to implement the generative algorithm into this framework. Furthermore, I intend to add more objects to the collection. 60 06. Conclusion In this thesis I have introduced my ideas of consciousness and matter and the invisible aspects of both subjects. Through my research and experimental investigations of the temporal aspects of mind and matter I have developed methods of translation of mental states to light and sound, as well as various methods of visualization and sonification. I have learned that measuring the physical world, weather it is the length of a string or a complex audio signal, ultimately comes down to the relative observation of the measurement that can only be obtained through the senses. My work inhabits a space between art and science; two fields of study that used to be one. I hope that my work will help bring these fields back together. I view art and science not as completely different fields of study, but complementary approaches for understanding our reality ! In BrainTV I built a system that translates my own brainwaves to frequencies of sound and light on an analogue television. In this work I allow my self the assumption that the reductionist worldview is correct; that consciousness can be defined in purely physical terms. In the first iteration of BrainTV I found that using the Attention and Meditation values produced by Neurosky's secret algorithm enabled me to recognize certain states of my own mind. To me, it was an interesting process to calibrate my algorithm for the composition on the TV to reflect my current mental state. Seeing my consciousness extended outside my own body prompted me to think about the physical world as being an extension of all consciousness. I believe our consciousness creates the physical world, and I see BrainTV as an experiment demonstrating this principle. This experiment amplifies the mind-matter relationship in a framework that is clearly observable. Observing my own mind having a real physical effect on the television was 61 an enlightening experience for me and I started empathizing with technology and seriously questioning the bounds of consciousness. At this point I started looking for consciousness in other electrical networks than the human brain. ! In Observations I am looking at audio signals in the physical world in terms of monistic idealism; the idea that matter and consciousness are aspects of the same thing. Different places have very different effects on my consciousness and perhaps my state of consciousness is being affected by the environment. To see if there was a correspondence between my conscious (or emotional) state and the place, I made spectrograms of the places and compared them visually. I found that the overall composition of the resulting image resonated strongly with my conscious experience at the time of recording. A recording of a traffic intersection produced a dense, noisy but structured image and birds singing in the forrest gave a much more harmonic and calming composition. I had found that the images not only represent a physical place, but a conscious state as well. ! Thingogram v.1 is a more concise approach to measure the frequencies embedded in physical objects. By exposing an object to all audible frequencies and record the result I was able to determine the effect the presence of the object had on its sonic surroundings. The objects' physical properties determine what frequencies are able to pass by it, and which ones it absorbs. Through an analysis of the recorded signal I obtained a unique list of frequencies and amplitudes from each object. When the signal was converted into an image, a unique composition of distinct frequency bands was revealed for every object. 62 ! In Thingogram v.2 I transformed the images from the previous version to fractal sound compositions. For this purpose I created the Natural Sound Synthesis Algorithm which remaps the spectral composition of the recorded objects to an array of fibonacci sequences. The result is an abstract audio composition consisting of fibonacci frequencies distributed relative to the spectral composition of the input image. ! Thingogram v.3 is my final iteration of my work in the MAA program. It is informed by my research and practice through my two year journey and implements methods and parts derived from my preceding artwork. As I had built the components and some of the required systems before I made this work, the main challenge was to compile them into a physical object. The works' outermost layer is the protective housing of the flight case. I use this material to reference the motion of the object during it's intended transportation. Another challenge in making this work was writing the control software that handles the interaction between components. This algorithm detects the image and plays the corresponding audio but the most important aspect was the content and timing of the object's information displayed on the monitor. The structure of the installation guides the viewer to a fixed location where it is to be viewed and the relationship between the image and the audio can only be perceived by one person at a time. The process of viewing this work involves taking an object, putting it on a platform and listening to an obscure abstraction of its frequencies while taking in textual information about the object. I provided the viewer with limited instructions for the operation of the apparatus, leaving him to viewer has to figure out what is being referred to in the instruction on his/her own. 63 ! Through my work for the past two years I have learned much about consciousness and matter, and how the invisible can be visualized/sonified and understood through experimental practice. I draw my understanding from a variety of different sources, including physics, neuropsychology and parapsychology. I have come to think about everything as being reducible to a list of frequencies, including different states of consciousness and matter. In my upcoming research I will focus on systems of matter manipulation. I intend to make Thingogram v.4 which will apply the same signal extraction methods to an object, but then reconstruct the signal and continuously expose the object to its own audio frequencies. I suspect this will cause the object to resonate and when the sound pressure accumulates it might cause it to levitate. As I further develop my work, I will focus on the magnetic field of places and objects. I intend to approach magnetic fields in the same way as I have approached audio and brainwave signals; in terms of frequency and amplitude. In this research I want to find out whether it is possible to effect matter on the atomic level. If objects have a magnetic field set at a certain frequency and amplitude, creating a magnetic field with the same characteristics in opposite phase should then cancel out the magnetic field of the object. In the same way as Thingogram v.4 is intended to acoustically levitate an object, I am sure the same can be applied to magnetism. However, working with high-frequency magnetic fields requires tools that I don't yet have access to and large electromagnets require high voltages. That said, I hope to collaborate with physicists in the near future to create art in the medium of magnetism and construct invisible art structures with observable effects. 64 Works Cited Ascott, Roy. "Seeing Double: Art and the Technology of Transcendence", Reframing ! Consciousness, Exeter, England ; Portland, Or. : Intellect Books, 1999. Bjelic, Dušan I., "Lebenswelt Structures of Galilean Physics: The Case of Galileo's ! Pendulum", Human Studies, Vol. 19, No. 4 (Oct., 1996), pp. 409-432 Bolter, David. Turing's Man: Western Culture In the Computer Age, The University of ! North Carolina Press, Chapel Hill, 1984. Daly, Scott, “The Ganzfeld as a Canvas for Neurophysiologically Based Artworks,” ! Leonardo 17, no. 3 (January 1, 1984). 192-174. Print. Edebol, Hanna, Sven Åke Bood, and Torsten Norlander, "Chronic Whiplash-Associated ! Disorders and Their Treatment Using Flotation-REST (Restricted Environmental ! Stimulation Technique)", Qualitative Health Research 18, no. 4 (April 1, 2008): ! 480-488. Print. First, David, “The Music of the Sphere: An Investigation into Asymptotic Harmonics, ! Brainwave Entrainment and the Earth as a Giant Bell,” Leonardo Music Journal ! 13 (January 1, 2003): 31-37.Print. 65 "Gestalt Psychology". Wikipedia, Simple English. Wikimedia Foundation, Inc. 12 Apr ! ! 2013. Web. 22 Aug. 2013. "Golden ratio". Wikipedia, The Free Encyclopedia. Wikimedia Foundation, Inc. 11 Aug ! ! 2013. Web. 20 Aug. 2013. Goswami, Amit. The Self-aware Universe: How Consciousness Creates the Material ! World, New York: Jeremy P. Tarcher / Putnam Inc,1995. Print. Gregory, Richard L., ed. The Oxford Companion to the Mind. 2nd ed. USA: Oxford ! University Press, 2004. Print. Hatfield, Gary. "René Descartes.” The Stanford Encyclopedia of Philosophy (Summer ! 2011 Edition). Edward N. Zalta (ed.). URL = . Web. 18. Jul. 2013. Janiak, Andrew. "Newton's Philosophy.” The Stanford Encyclopedia of Philosophy ! (Winter 2009). Edward N. Zalta (ed.). URL = . Web. Jul. 18. 2013. Kenny, Anthony. A New History of Western Philosophy, Oxford UP, first 2007 [current ! 2010], Oxford, New York. Print. 66 Long, Rose-Carol W. Kandinsky: The Development of an Abstract Style. New York: ! Oxford University Press, 1980. Print. Manovich, Lev - Software Takes Command, Creative Commons License, digital ! distribution. 2008. McLuhan, Marshall - Understanding Media: The Extensions of Man, The MIT Press, ! Cambridge, Massachusetts ; London, England, first 1964 [current 1994]. Print. Molholm, Sophie et al., Audio-Visual Multisensory Integration in Superior Parietal ! Lobule Revealed by Human Intracranial Recordings,” Journal of Neurophysiology ! 96, no. 2 (2006): 721 -729. Mumford, Lewis. "The Monastery and the Clock.” Technics and Civilization. New York: ! ! Harcourt, Brace & Co.: 12-18. URL = . Web. "Rubin vase". Wikipedia, The Free Encyclopedia. Wikimedia Foundation, Inc. 24 Apr ! ! 2013. Web. 20 Aug. 2013. Slowik, Edward. ”Descartes’ Physics.” The Stanford Encyclopedia of Philosophy (Fall ! 2013). Edward N. Malta (ed.). URL = . Web. Jul. 18. 2013. Talbot, Michael. The Holographic Universe, HarperPerennial, United States, 1992. Print. Turrell, James.URL = . ! Web. Retrieved 07/07/2013 68