NerveLoop: Visualization as Speculative Process to Explore Abstract Neuroscientific Principles through New Media Art

Abstract

Consciousness is a concept that is nearly impossible to visualize with conventional data visualization methods. In response to this challenge, I propose strategies for visualizing core ideas based on complex theoretical frameworks that are too abstract, incomplete, or complex to visualize through conventional data visualization methods with New Media Art playing a pivotal role in this speculative visualization process. One of those strategies targets visual speculation by introducing metaphors and allegories as a possible efficacious method. In this practice-based research paper, I present my animated video work NerveLoop as a case study of how New Media Art can be utilized for exploring abstract neuroscientific principles. NerveLoop metaphorically represents these principles by comparing the brain, the city of Hong Kong and improvisational jazz. The work is conceptually informed by theories from the scientific disciplines of neuroscience, neuropsychology, computational consciousness research and neuroanatomy. Visual and auditive metaphors are explored as potentially powerful tools for expanding research domains and enable meaning making processes that otherwise remained latent.  I conclude by reflecting on the creative process and viewer response to Nerveloop to evaluate the effectiveness of speculative visualization as an approach.

Keywords: New Media Art, Speculative Visualization, Consciousness, the Encephalon, Neuroesthetics.

Introduction

In 2018 I had a seizure which let me experience losing consciousness for several hours followed by the reconstruction of a conscious mind that took several months before regaining functionality. This process continues to evolve progressively and continuously. Due to my background as a visual artist, I was able to take a different perspective on exploring the pertinent questions related to neuroscience and neuroanatomy affecting my recovery. This artistic vista also provided a window into the mechanisms which are loosely based on scientific data rather than on philosophical ideas and are aspired to visualize abstract mechanism that form the core structures of how brains are processing information. Many issues, visions and different interests are affecting a collective agreement to form clear definitions, especially ones regarding consciousness, that are frequently associated with disaccord between scholars of different disciplines and this discordance is intensifying as the research areas are proliferating into multifariousness by an increasingly miscellaneous population of researchers. It is this domain of disaccord that I consider to be my working territory as an artistic researcher.

            One way that I visualize and express this disaccord is through the lens of New Media Art (NMA) as a tool to initiate a subbranch of neuroimaging. This allows for an exploration process that is less precise and concentrated on visualization processes and mechanisms of complex subjects that are either too big or abstract to use conventional methods of data visualization on. This includes all aspects directly related to the functionality and the architecture of our brain, as well as the interrelationships and interconnections which I envision within my research to map diagrammatically. Consequently, I am inclined to rather use a horizontal approach to map consciousness as a model to explore among other questions, intersubjectivity, the nature of the mind, computational consciousness while seeking a plausible explanation of where general consciousness would fit in the neuroscientific approach. This approach is explored and presented through digital produced artworks that contextualizes consciousness as a framework that allows information to be processed. As consciousness itself does not have visual properties, the visualization of consciousness can only illustrate explanatory diagrammatical assumptions or conjecture of its elusive nature.

            In this practice-based research paper, I present my animated video work NerveLoop as a case study of how New Media Art can be utilized for exploring abstract neuroscientific principles.  As a foundation for understanding the work, I review a small selection of literature that deals with both consciousness to provide a neuroscientific perspective, as well as visualization techniques usually employed illustrate and exemplify existing theories. These visualization techniques are inherently subjective as they generally result in graphic representations based on simplification, abstraction, speculation, and interpretation. This subjectivity raises the question of how useful these approaches are as the visualizations are at best approximations of scientific theories of possible structural and neuroanatomical mechanisms. I then describe the process and result of creating NerveLoop in response to this question to claim that hypothetical projections can provide an array of multiple insights in the mechanisms of the brain and in specific the elusive nature of consciousness, which requires an unconventional approach to reformulate idiosyncrasies and purposes of its nature and even to drop the bomb by questioning its existence. Finally, I reflect upon the reactions of viewers to NerveLoop during its initial display during an exhibition in Hong Kong from 5-18 July, 2021.

Consciousness, Neuroesthetics, and Visualization

Consciousness

Consciousness has been highly enigmatic, elusive, and inspiring to thinkers, artists, and visionaries of the mind since the dawn of times. Philosophers and scientists alike have been struggling for centuries to define consciousness but failed to come up with a universally accepted description. Many theories have been posed and all fail to satisfy peer researcher from different disciplines. Each discipline has its own interpretations and definitions that works for that branch of science. In philosophy the concepts are even more diverse, which does not contribute to form an explanation that could function as a workable model of what consciousness really is. For instance, materialists and panpsychists are diametrical positioned in their concepts and these are just two of the doctrines struggling to explain it. The realm of scientists and philosophers trying to solve the questions of consciousness, also described as the ‘hard problem’ of consciousness [1] are rapidly gaining traction and the last decade has resulted in a radical proliferation of peer reviewed published research papers. As a result, the ideas and theories generated are widening and diverging its scope of hypotheses, making it less likely to find a unified consistently accepted theory of consciousness. At this moment of writing there are several theories that seem plausible to explain consciousness, but no accepted quantifiable system for measuring it exists. Integrated Information Theory by Tononi [2][3] proposed a theory to quantify consciousness, but even this attempt remains controversial in its acceptance as official system to form a unit of measurement of consciousness. It therefore seems that the intangible nature of consciousness resides in the realm of arcane obscurity with conjecture as the only means of conceptualization. Another perspective would be that consciousness is progressing into its own branch of neuroscience generating a cumulative of manifold meanings and theories.

            As consciousness cannot be measured through direct observation, I used a combination of phenomenology, explorations in neuroscientific theoretical concepts, and autoethnographic essays based on my seizure and recovery to develop my subjective understanding of consciousness. The work NerveLoop is in that sense the practical element of my research into consciousness. One of the difficulties of directly observing consciousness is due to the lack of a unified definition which propagates an array of extensively different ideas about its existence, purpose, and origins from within a multitude of different discipline, both empirical and philosophical. Within this paper, I observe consciousness through the lens of neuroscience, neuropsychology, neuroanatomy, and the fast-emerging field of computational (artificial) consciousness research. I have omitted all other disciplines dealing with consciousness to narrow down the superabundance of interpretation and meanings that are allocated to consciousness.

            Of these lenses, the most relevant to my research is the discipline of computational (artificial) consciousness which uses information generated within neuroscience and neuroanatomy to build artificial models that simulate or instantiate components that are assumed to be related and partially responsible to give rise to consciousness. Most of these models have been built in physical computational devices, using technologies like artificial intelligence and machine learning as supporting elements to developing functional prototypes. These models are usually physical computational objects that function to test assumptions and hypotheses about the neural correlates of consciousness made in neuroscience [4] The interdiscipline as such is experimental in nature and supports neuroanatomical and neuroscientific research by feeding their findings back to neuroscientists who initially developed the hypotheses. Insights acquired through this collaborative endeavor are pivotal for increasing knowledge regarding the brain and consciousness. These neuroanatomical and mostly cognitive neuroscientific insights then inform the exploration of the origins and nature of consciousness as well as the development of the brain through the lens of evolution. Speculative visualization of this domain through NMA can contribute to producing visual metaphors that will react and provide visual feedback on the usually abstract and complex findings from this niche field of neurological research.

Visualization, Graphic Interpretation, Speculation

The purpose of any type of visualization is typically to graphically represent something complex and abstract in a simplified and codified manner that can provide easier access to the information that has been converted from a textual and often abstract format into a comprehensible infographic or animation. Visualizations are created to exemplify and clarify the information within the visual domain. It should be noted that this process of conversion of textual information into visual information generates either an approximation or a rendition that can never be precise or fully accurate. A process of interpretation leads to design decisions that speculate visual elements and motion components to conceptualize and simplify abstract data that often are devoid of any visual information. In that sense the visualization process is illustrative in codifying data through a visual redefinition of that information. This is done through a process of conceptualization and decoding research data, often generating noise as a by-product.

            One can interpret this process as a specific type of visual speculation. This process is inherently characteristic to the medium and its limitations that depicts a scientific theory or component of a theory. It is worth to note that I depart from current trends in information theory and data visualization as they are developing too rapidly [5]. Traditional data visualization that uses pure scientific data is too limited for exploring issues that have less unified data output and in addition is fragmented through the various methodologies, theoretical frameworks and research disciplines that contribute to the production of both knowledge and data within the domain of cognitive neuroscience alone. In other words, consciousness is not a concept with a universal accepted definition. The approach through NMA offers a different vista by using visual metaphors[1] [6] to address these disparate information and data streams that can be processed through art visualization using principles and techniques that are usually less suitable and scarcely employed in scientific data visualization. Perhaps an exception is the field of molecular visualization, which derived its visual language chiefly through traditional animation techniques [7] Yet this extensive field has similar limitations of simplifying and abstractions to manage a level of detail that is computational possible (2014). It seems that a less defined space for interpretation allows for some flexibility to reconceptualize the various theories in a way that multiple audience groups can easier access and grasp, thereby increasing the scope of the theoretical concepts’ comprehensibility and possible insights that could lead to new avenues of research.

A potentially useful approach is that of speculative visualization which developed from speculative design [8]. This interdisciplinary approach combines artistic and design perspectives with rhetoric and scientific analysis to surpass the limits of traditional data visualization (2010). Using this approach in conjunction with NMA enhances the possibility of inspiring both scientists working in neuroscience as well informing regular audiences through artworks with unconventional characteristics.

Neuroesthetics

The exploration through NMA to explore the mechanisms of the brain and to a lesser extend consciousness have recently entered a field known as neuroestethics, a term that has been introduced from within cognitive neuroscience that centers around epistemological questions and ontological representations by the brain in a process of visualization and animation techniques, occasionally culminating in a work of art [9]. Traditionally neuroestethics focuses on the processes that occur in the brain when subjects are confronted with visual art. Within this research I propose to invert this process by switching the subject from art to the brain itself. Normally, neuroestethics is employed in generating art that directly explores the brain and its mechanisms. Neuroscientific processes subsequently turn into artistic subject matter itself rather than a tool to conceptualize the reflexive inner workings of art on the brain. This particular field developed from the end of the twentieth century and evolved parallel to the technological development that provided the tools to research aspects that were only possible with the latest technological inventions in neuroimaging. These developments included technologies like computed tomography (CT), magnetic resonance imaging (MRI), functional magnetic resonance imaging (fMRI), and positron emission tomography (PET) [10]. Technological progress consequently was the driving force that instigated the field to develop.

            Neuroesthetics departs from the core focus of classic aesthetics of providing definitions connected to beauty and proportional studies of aestheticized values and concentrates instead on cognitive and neural explanations mapping behavioral and social aspects in a singular approach. Neuroesthetics centers around human cognitive principles rather than abstract concepts based on culture, art history, the evolution of formalistic studies of aesthetics and so on. Furthermore, neuroestethics assumes that aesthetic cognition occurs through the interdependence of perceptual, emotional, and evaluative processes as they affect social and contextual conditions within a society. It valorizes the artworks within the domain of neuroesthetics and is thus self-referential. Within my work as an artist, I use aspects of neuroesthetics to drive the appropriate questions in the quest for exploring possible insights into mechanisms that could represent consciousness.

Evolution of the Encephalon

The advancement of the encephalon, or brain, and its functions through evolution provides a roadmap that generates insights which allows for conceptualization based on models using archaeological, neuroanatomical, neuroscientific, and cognitive neuroscientific data regarding the evolving brain. These models bypass human centric research of the brain by expanding the territory into multispecies, both extinct and alive today. Within this expanded domain for research are highlighted both differences between and slices in time where the development of brain functions.

            The neuroscientist-neuropsychologist Paul Verschure conceptualized a theory about the evolution of both the brain and consciousness in his Mind, Body Brain Nexus (MBBN) and his concept of Distributed Adaptive Control (DAC) [11]. His theory stated that during the Cambrian explosion[2] the brain was forced to develop as the competition between species became more prevalent and complex. This competition required the brain to develop a capacity to socialize but also to strategize how to survive rapidly changing environmental conditions. As a result, the capacity of the brain needed to quickly evolve and increase exponentially in computational brainpower. To account for this elevated demand of cognition, Verschure postulates that the brain starts to parallelize processes and virtualize possibilities of the world to deal with the increased complexity [12], a process that is known in psychology as ‘simulation’ [13]. Verschure postulates that this alteration allows the brain to simulate possible versions of the world so the entity can now impose interaction into a virtual world that enables the mind to make predictions, strategize circumstances in multiple scenarios, filling missing data, and the creation of inferences in hidden states of other agents, both alliances or enemies, anticipating behavioral and action-oriented reactions (2016).

            A byproduct of virtualizing the world and its different scenarios in relation to other agents is the creation of sense of self that leads to positioning one’s self in this virtual world in relation to everything else. This position includes a sense of proprioception, which involves estimating distances measured by an entity’s own dimensional awareness. This projection of a virtual world, devoid of unnecessary information, filtered out by selective focus, requires to be continuously evaluated, constantly optimized, and updated into successful interactions with other entities and to anticipate events that have not occurred yet are all increasing the chance of survival. Verschure states that the brain is constantly predicting the near future and all possible events that might happen (2016). Through the notion of self in this simulated world of probabilities is simultaneously the birth of consciousness as an epiphenomenon (2016). Consequently, the concept of reality that is happening at this specific moment in present time, perceived as now, is dominated by our unconscious states, and is only made conscious the moment it is necessary for survival and success of the species, and subsequently processed in the simulated world. This leads to believe that consciousness is always trying to catch up with reality in real-time to optimize the performance for the future (2012).

            This delay of real-time has been extensively researched by Benjamin Libet, who came to fascinating results which questions our concepts of free will. Libet estimates it takes approximately 40 to 80 milliseconds for a signal to traverse the neurological pathways towards the brain. In the brain it can take up to half a second for this information to be processed into sensory awareness [14]. Libet’s experiments somehow indicated that decisions are made moments before the conscious mind intentionally does so, which questions free will of someone’s rational agency as a decision maker. Verschure explains this delay as a time that is needed to rebuild changes in the constructed virtual simulated world. The more complex those changes, the longer the processing time indicates computational intensity (2016). This implies that consciousness appears within a constructed reality, created in a slightly delayed virtual world, which interprets the real-world, real-time reality, predictively to overcome this lost time and to synchronize with the real-world. Consequently, consciousness in this scenario is a truly epiphenomenal product of the encephalon.

            A similar theory that deals with the construction of a virtual world has been proposed by the neuropsychologist Lisa Feldman Barret. She postulates that emotions are constructed as a response to predictions we continuously make through planned intentions within our simulated worldview. She further emphasizes that emotions are principally indistinguishable from cognition and perceptions. Barret described this as the theory of ‘constructed emotion’, which integrates social, psyschological, and neuroconstruction [15]. Barret’s neuropsychological research supports and confirms the processes of a constructed virtual world albeit differently than Verschure’s description in his concept of DAC. A question comes to mind when considering this: can humans really get in touch with the real-world in real-time as we are limited to interpret this reality filtered through our virtual model of the world in which we are undoubtedly confined? This impenetrable separation between our perceptible virtual inner world, with the external material realm is raising many philosophical questions, which are close to impossible to answer and can only be addressed through speculative reasoning, which justifies this particular methodology as not only significant, but maybe inevitable to reveal a glimpse of the world we are really living in.

Artistic Representations of the Brain and Consciousness

Santiago Ramón y Cajal, a Spanish neuroscientist, pathologist and histologist specialized in neuroanatomy and the central nervous system, created a vast body of work between 1909-1911 where he mapped and illustrated hundreds of arborizations of braincells [16].  He illustrated the complete spectrum of all different types of neurons in hundreds of detailed drawings. This body of work can be considered the first visualization study within neuroscience, and his investigations of the microscopic structure of the brain made him the pioneer of modern neuroscience. In creating his illustrations, Ramón lay the foundations for artist to work on the fringe between neuroscience and art. His work was descriptive as it depicted exactly what he observed through the microscope. This is in contrast with how 21st century artists are working with neuroscientific material using an approach that deals with different kinds of  metaphors, including visual.

            Christian Mio Loclair’s installation work Narciss produced in 2018 is an example of how a phenomenon of the mind can be represented using visual metaphor and NMA. He translates human narcissism – “Mankind’s self-righteous model of        self-awareness, the quality of subjective findings while investigating oneself and the unequal distribution of dignity” [17] – into the phenomenon of artificial narcissism. The installation consists of a bare computer, custom made in a predominantly aesthetically black color, equipped with an embedded camera system that looks at its own reflection in a mirror positioned perpendicular opposite itself. The computer is equipped with an artificial intelligence that is constantly processing its own image. The metaphor here can be explained as a vessel to reflect constituents of a contemporary society using social media and other technology as a self-reflective narcissist mechanism of continuously carefully observing and constructing oneself through mediated representations online. He explains that the machine is displaying digital consciousness through its performance, using both Alan Turing’s concept of imitation and the concept of narrative identity (2021).

            A more formalistic interpretation of art that deals with the brain can be found in the work of Ralph Helmick who created in 2016 in a large-scale sculptural installation titled Schwerpunkt, the German term for focal point [18] The work uses sculptural representations of shiny gold leafed neurons in different sizes and scales cascading in a random array of configurations suspended in the 3-storey entrance of the McGovern Institute of Brain Research at the Massachusetts Institute of Technology (MIT). The work looks arbitrary and chaotic, but when viewed from one position on a balcony through the visual phenomenon of anamorphosis, the image forms a visual depiction of the brain. In Schwerpunkt, the metaphor is more general and narrates the story using loosely chaotic components which viewed from the right angle can be understood as a cohesive image. The work ultimately refers to brain research that endeavors to unravel the intricate mechanisms of how the brain functions.

            Similarly, the collaborative artwork Self Reflected by Dr. Gregg Dunn [19] an artist and neuroscientist, and Dr. Brian Edwards, an artist and applied physicist, can be understood as a visual aesthetic representation depicting cross-sections of the brain. They developed the technique of reflective microetching, which functions as visual metaphors that show “the delicately balanced neural choreographies designed to reflect what is occurring in our minds” (2021). Dunn describes his work as “neuro art” to elucidate the nature of human consciousness. Although their work seems to take a formalistic approach, it can be argued that the concepts of observing the complexity of the human mind, which Dunn describes as “the most marvelous machine in the known universe”, are represented though the complexity of the work. In Self Reflected, half a million neurons are animated by reflected light through the numerous micro grooves that has been generated through this process of microetching. Dunn and Edwards use combinations of hand drawings, deep neuroscience research, algorithmically simulated neural circuitry, adapted brain scan data, photolithography, gilding and strategic lighting to achieve these 2-dimensional plates. Their work bridges the aesthetic formalistic qualities of observing and thinking about a “thinking machine” thus mirroring the action of meta-analyzing our own brain.

NerveLoop

Overview

NerveLoop is the result of an artistic investigation to confirm the aforementioned hypotheses as described by both Verschure and Barret which state that our relation to the world is constructed in our brain in a virtual model of the world as compiled by perceived information gathered and conveyed through our senses. Conceptually, I mixed visual and auditive metaphors and conceptual metaphors to emphasize that the world we construct is reflective of the world we experience through our internalized structural scaffolding of the brain and its mechanisms, and even can be used to predict hidden or undiscovered mechanisms within the brain by researching our own creations, in this case the city of Hong Kong.

            To construct this reflection, I highlighted the correlation between the transportation mechanisms of both the brain and the city of Hong Kong and focused primarily on the structural principles that both seem to share. Observing the city in comparison to the brain requires conceptualization that will help to overcome differences in spatial and temporal dimensions to synchronize between the two. A common saying across cultures is that a city is “alive” regardless of being built with mostly inanimate and non-organic materials. The dynamic characteristics of change, growth, adaptability, resilience during catastrophes, and generally progressive nature of how a city develops can all be categorized as processes that can be understood through evolution of living species and might not be so farfetched to associate with one and another. A city moves through time within a different temporality than its inhabitants. The lifespan of a city can be millennia, while that of humans is usually less than a century. This difference in temporality requires an unconventional approach in conceptualization and visualization to surpass these experiential dissimilarities. Within the film temporality is constantly shifted to experience a stretched timespan enabling to travel, point of view (POV), through the system.

Hong Kong Urban Machine Jazz and Lights

The city of Hong Kong is a living entity with a hidden capacity to generate improvisational “machine jazz.” While traveling on the Mass Transit Railway (MTR) – a public transportation network of heavy rail, light rail, and buses – I noticed that the intersection between two train coaches were connected by accordion-like industrial rubber seals, which coupled with the motion and vibration of the moving train, created a sound which was vaguely reminiscent to something familiar, but which was simultaneously to abstract to identify. After recording several sound fragments, I processed the sound by slowing it down, stretching the fragments and pitching up the sound. This process released the sound which became instant recognizable as improvisation jazz. Patterns of this discovered music piece were both rhythmic and syncopated but retained characteristics of a freestyle abstract jam. Somehow the sound fragment had a visual quality which alluded to motion and speed and triggered somehow the feeling of traversing through narrow spaces like tunnels or crevices. Additionally, sounds that evoked images and a sense of tensile forces which were pulling, and pushing were interspersed throughout the track. The sound recording suggested with its intricate rich soundscape quality the first conceptual ideas of the film to travel through the brain, transposing the POV experience with the trajectory of a sequence of synaptic impulses, that travel the neurites. A new take on the work to research consciousness was to visualize a representational model of the anatomy of the brain through an animated work, that emphasized the process of thinking from a neuroscientific perspective.

Fig 1: Stills from NerveLoop, Hong Kong Urban Machine Jazz, 2021.

Duration 5 minutes and 33 seconds, 4k, Mp4. Produced and modeled in Blender, rendered in Cycles and postproduction in Da Vinci Resolve.

Hong Kong is also known for its omnipresent atmosphere of neon light. Many sci-fi books and films are directly inspired by this neon jungle and its accompanying steam-punk architecture consisting of a patchwork of high-rise buildings, some gleaming new and others decrepitly dirty. Most of the external facades reveal intricate networks of external piping, wiring, bamboo scaffolding, air-condition units, or other artificial growth, which produces something locals call “AC rain”, an artificial rain that descends continuously, especially in the narrow streets of the dilapidating aging districts of Kowloon. These textures, and its materiality, the sun faded colors of flakey mural paints, the stains of fungus, molds and dirt on the walls all contribute to a strictly unique atmosphere and scent. The daytime colors of the city shift at night to something reminiscent of the historical neon signs, which have been recently replaced by LED’s, illuminating the night sky.

            In the film I created four strong point lights, devoid from fall-off. They travel on their own elliptical trajectory in their own unique velocity. The structure of the brain is a transparent glass like material with caustics and reflections, but slightly matte, so the reflections are only light based. These lights have been colored in reference to the night city lights of Hong Kong. The motion over their elliptical orbits animates the structure of the film in a dynamic manner and contributes to experience the space as a much larger space then it has been modelled. This expanded spatial experience contributes to the capacity of the film to raises questions and to form associations and links to information in our environment. As such we could jam our habitat with a refreshed perception, which allows us to hack and discover existing elements in a rehashed manner.

Spatiality, Velocity and User Experience

The film was designed to immerse the audiences inside an experience similar to a rollercoaster ride. This was achieved by using visual effects to allow the viewer to be dragged inside this constructed digital world that represents our inner brain. As the foundation for this visualization, I modelled and simplified a tiny part of a connectome of the brain consisting of 3 clusters of neurons with simplified and limited dendrites. They are all connected through neuropathways and intertwined and are positioned in an empty space using a recent Google’s AI blog for reference [20][21]. I then selected 20 neurons from the Blakely and Januszewski model of human brain tissue which includes a dataset of 50000 cells with hundreds of millions of neurites, and 133.7 million synaptic connections (2021). This does not include the glia cells (oligodendrocytes). It should be noted that a dataset as impressive and seemingly complete as this one can only be considered an abstraction or approximation towards a comprehensible model of the real brain, as it has been achieved through various protocols, dealing with imaging, sample preparation, machine segmentation of cells, synapse detection, data storage, proofreading software and so on. Thus, even this elaborate effort to convert the brain in digital data is at best a simulacrum reflecting fragmented real-world material existence in an approximation that dives deeper towards a detailed anatomical model. Being aware of this inevitable limitation to be forced to represent a reflection rather than the real material existence a demand for visual metaphors and representational models is required. The level of necessary distancing from the provided data creates an information layer that can be explored through NMA but is in no sense scientific and at best an artistic interpretation based on the detailed dataset.

            Having modeled 3 simplified clusters of neurons and dendrites, I used these as volumes to generatively grow structures with rhizome and tubular like characteristics. These structures represent the neurites and microtubules which, along with the neurons, dendrites and axions located in the brain, are potentially responsible for generating consciousness [22]. These structures can be chemically manipulated to block their function of neurotransmission which allows consciousness to be switched on or off (2003).

            The modelling of these microtubules as a loosely structural element has been chosen referentially rather than as accurate representation of the architectural structure of neurons. As such the generative growth of this structure of microtubules is formed through conjecture of where consciousness could be located, but again I stress that this is an artistic decision. The specifically chosen generative method allowed the system to grow over time with a final growth period of 5 minutes and 22 seconds. This time-based generative process of evolving is representational of the new connections and neuropathways made by neurons when the brain is actively involved in adjusting its pathways. This process, known as neuronal plasticity, allows learning, restoration after injury, memory, thinking, and so on, and is a perpetual dynamic process of regenerating and restructuring the brain [23]. This neurogenesis is pivotal to the development and wellbeing of the brain throughout someone’s life and has embedded some strategies for auto restorative regeneration in case of damage.

            This growth period is represented by linking the different spatiality of the micro-space of the 3 neuron clusters with the macro-space of our human scaled spatial experience by radically shortening the focal length of the lens used in the POV camera. A distortion at the edge of the screen enhances the experience of having tunnel-vision. Motion blur, albeit modest was included to allow our brain to code speed and three dimensionality and to amplify the experience. Space needed to be shifted and occasionally bended, through the motion of the camera, with the ultrashort focal length lens amplifying each motion. The cinematography was determined by creating a guided path for the camera, where the camera is always facing the direction of movement. This distortion is most visible in the cases where the camera follows sharp curves in the track. The result is a slightly alienating experience of space distortion. Ultimately the space is warping around itself which suggests dimensional fluidity.

            Adding to this effect is the speed of traversing through the animated space. In NerveLoop, the camera is radically slowed down to counterpoise and synchronize our speed of perception and motion, relative to our human scale and the speed of motion within our spatial experience. The speed of synapses to spark by impulses travelling through the neurites is approximately 40-80 µs. The time it would take for the whole film to play would make it impossible to watch, therefore I chose to convert this velocity to decelerate by a factor of approximately 5 million. This allows us to experience the speed of an impulse to travel comparable to a speed we travel through space in a subway train. Bringing back together spatial and velocity perception into our experiential comprehension.

Observations

NerveLoop was displayed for two weeks at the Jockey Club Creative Arts Centre in Sham Shui Po, Hong Kong, as part of a group exhibition featuring artistic research output. Through observations and personal conversations, it became clear that many visitors (around 22 people) were able to immediately identify that NerveLoop somehow referenced the brain, neurons, and dendrites without any further contextual information. A frequent comment was that watching the video felt like traveling through the brain as if one were a thought. Some people (9 people) associated the experience with roller-coasters, or an underground ride. Many visitors came quite close with the intended concepts, although sometimes expressed through rather long ruminations. Another observation was that many people were looking longer than the duration of the animation. Some individuals (7 people) stayed for 2 or 3 loops to be finished and revisited the work after seeing the rest of the exhibition. Other visitors (approximately 15 people) expressed that the work never became boring, even though it repeated in a loop and not much change was happening. They compared the sensation to looking at a campfire or at the ocean. One person mentioned that the work was very Zen. Five young children in the age between 5 and 8 years old, showed quite different reactions. Some children associated it with video games, or sci-fi special effects and they liked to spend some time sitting with the work. One group of 6 young children were a little scared of it at first and tried to avoid looking at it and left early. The level of abstraction in combination with the propelled motion, and the absence of recognizable elements induced some discomfort in those smaller children. In overall the work was well received and made some visitors think about thinking.

Conclusion

Using NMA as a speculative visualization method has obvious advantages and disadvantages. There is an aspect of interpretation involved in speculation, which could arguably lead to justifiable criticism. It is therefore paramount to use visual speculation only in cases where the information is too complex, or abstract, or incomplete to use conventional data visualization methods on. In making NerveLoop I was able to directly see how this method affected the creative process and the reactions of the general public. As an artist, speculative visualization provided me with a new avenue by which to explore the theories and concepts regarding consciousness and the mechanisms of the brain in a new light. By incorporating this method into my practice-based research, I was able to gain new insights and knowledge regarding visualization processes and methods. I like to stress here that art and NMA do not require to have a utilizable function or purpose. I provide one option of many which in this paper encompasses scientific visualization through NMA as just one of a multitude of options and possibilities.

            Another aspect that could impact the reliability of the visualization is the subjective preference of the artist to choose a specific esthetic visual language which might not be sufficient neither elucidate the scientific information. An opportunity is therefore opening whereby NMA can play a new role in contributing to science by informing and experimenting with theories that are difficult to represent through conventional data visualization. NMA may not have developed as a visualization tool to science, but it is worth noting that there is a niche possibility to do so. Of course, NMA inspired by science is a different approach and this will not impact the reliability of the scientific theory, but rather illustrate scientific ideas. Art and science in that sense can mutually benefit when the theories are reflected through art as a speculative visualization. Such speculative visualization can also inspire, bring complex scientific theories to a bigger audience, provide unconventional insights and vistas, and in conclusion can result in artworks that can be admired, enjoyed, and spark the imagination of everyone fascinated and mesmerized by it.

            Conversely, the flexibility of not working empirically might be criticized but it can also be liberating to shine a light on different aspects of data as it is churned out through cognitive neuroscience. Working on this project showed me that even empirical data goes through a process of approximation, simplification, and abstraction, to overcome limitations in computational power, or the lack of proper visualization methods. I see here a space for development and NMA can be a domain that suggests a different approach, which can inspire and provide ideas to be adapted by data visualization sciences. Therefore, the contextual utilization of NMA in relation to scientific data requires clear intentions and aspirations of the parts of both artists and scientists in any form of collaboration.

            Further future work will build on the ideas that have been generated through this project. Some areas that are going to be explored next will be how reality is constructed in relation to the internal virtual world as suggested by Verschure. VR as a medium will be the tool of choice to experiment with this. My personal journey of exploring the nature of consciousness would ideally result in a foreseeable future in a workable physical prototype experimenting with artificial consciousness, generating insights of how the mind and the brain inextricably exist. But more importantly as an artist I intend to provide a narrative which can inspire, provoke but most importantly provide insights that viewers could reflect on how incredible our mind and the brain really is.

References

  1. Chalmers, D.J.: Facing Up to the Problem of Consciousness. Journal of Consciousness Studies, 2(3), 200–219 (1995).
  2. Tononi, G. An Information Integration Theory of Consciousness. BMC Neuroscience, 5, 42–22. (2004). https://doi.org/10.1186/1471-2202-5-42 
  3. Tononi, G., Boly, M., & Koch, C.: Integrated Information Theory: From Consciousness to its Physical Substrate. Nature Reviews Neuroscience, 17(7), 450–461. (2016). http://dx.doi.org.ezproxy.cityu.edu.hk/10.1038/nrn.2016.44
  4. Reggia, J. A.: The Rise of Machine Consciousness: Studying Consciousness with Computational Models. Neural Networks, 44, 112–131 (2013). https://doi.org/10.1016/j.neunet.2013.03.011
  5. Wang, C., Shen, H.W.: Information Theory in Scientific Visualization. Entropy, 13(1), 254–273 (2011). https://doi.org/10.3390/e13010254
  6. Yaman, H., Yaman, A.: Neuroesthetic: Brain and Art. NeuroQuantology, 17(3), 9–14 (2016). http://dx.doi.org.ezproxy.cityu.edu.hk/10.14704/nq.2019.17.3.1941
  7. Parulek, J., Jönsson, D., Ropinski, T., Bruckner, S., Ynnerman, A., Viola, I.: Continuous Levels-of-Detail and Visual Abstraction for Seamless Molecular Visualization. Computer Graphics Forum, 33(6), 276–287 (2014). https://doi.org/10.1111/cgf.12349
  8. Kim, T., DiSalvo, C.: Speculative Visualization: A New Rhetoric for Communicating Public Concerns. In: Durling, D., Chen, L., Poldma, T., Roworth-Stokes, S., and Stolterman, E. (eds.) Design Research Society International Conference, 2010: Design & Complexity, vol. 7, pp. 804-810, Design Research Society, Montreal (2010).
  9. Nadal, M., Skov, M.: Neuroesthetics. In: International Encyclopedia of the Social & Behavioral Sciences, pp. 656–636, Elsevier, Amsterdam (2015).
  10. Wang, J., Yang, T., Thompson, P., Ye, J.: Sparse Models for Imaging Genetics. In: Machine Learning and Medical Imaging, 129–147. Academic Press, Cambridge (2016).
  11. Verschure, P. F. M. J.: Distributed Adaptive Control: A theory of the Mind, Brain, Body Nexus. Biologically Inspired Cognitive Architectures, 1, 55–72 (2012). https://doi.org/10.1016/j.bica.2012.04.005
  12. Verschure, P. F. M. J.: Synthetic Consciousness: The Distributed Adaptive Control Perspective. Philosophical Transactions of the Royal Society B: Biological Sciences, 371(1701) (2016). https://doi.org/10.1098/rstb.2015.0448
  13. Barrett, L. F.: The Theory of Constructed Emotion: An Active Inference Account of Interoception and Categorization. Social Cognitive and Affective Neuroscience, 12(11), 1833–1833, 26 (2017). https://doi.org/10.1093/scan/nsx060
  14. Libet, B.: Mind time: The Temporal Factor in Consciousness. Harvard University Press, Cambridge (2004).
  15. Barrett, L. F.: How Emotions are Made: The Secret Life of the Brain. Houghton Mifflin Harcourt, Boston. 25–41 (2017).
  16. Markram, H., Muller, E., Ramaswamy, S., Reimann, M.W., Abdellah, M., Sanchez, C.A., Ailamaki, A., et al.: Reconstruction and Simulation of Neocortical Microcircuitry. Cell 163(2), 456-92 (2015).
  17. Loclair, C. M.: https://christianmioloclair.com/narciss
  18. Helmick, R.: https://helmicksculpture.com/work/schwerpunkt
  19. Dunn, G.: https://www.gregadunn.com/self-reflected
  20. Blakely, T., Januszewski, M.: A Browsable Petascale Reconstruction of the Human Cortex, Google AI Blog, (2021). http://ai.googleblog.com/2021/06/a-browsable-petascale-reconstruction-of.html
  21. Scheffer, L. K., Xu, C. S., Januszewski, M., Lu, Z., Takemura, S., Hayworth, K. J., Huang, G. B., Shinomiya, K., Maitlin-Shepard, J., Berg, S., Clements, J., Hubbard, P. M., Katz, W. T., Umayam, L., Zhao, T., Ackerman, D., Blakely, T., Bogovic, J., Dolafi, T., … Plaza, S. M.: A Connectome and Analysis of the Adult Drosophila Central Brain. ELife, 9, e57443 (2020). https://doi.org/10.7554/eLife.57443
  22. Hameroff, S., Penrose, R.: Conscious Events as Orchestrated Space-Time Selections.  NeuroQuantology; Bornova Izmir, 1(1) (2003). http://dx.doi.org.ezproxy.cityu.edu.hk/10.14704/nq.2003.1.1.3
  23. von Bernhardi, R., Bernhardi, L. E., Eugenín, J.: What Is Neural Plasticity? In: The Plastic Brain, 1–15. Springer International Publishing, New York, (2017). https://doi.org/10.1007/978-3-319-62817-2_1

[1]      Visual metaphor can be described as visual objects that depict something representational or symbolic to elucidate something too abstract or elusive without having visual properties. Usually connecting two concepts where one has visual properties and the other does not, which develop a mental connection between the two, linking a visual quality to exemplify and simplify a complex and abstract concept. A similar explanation can be argued for auditive metaphors where sounds are conceptual representational to allude different meanings linked to specific sounds or sound patterns or even to visual or abstract information.

[2]      The Cambrian explosion occurred approximately 541 million years ago and was a period that saw an enormous proliferation of animals and species that started to compete for survival. It is assumed that it was this period that the brain started to evolve.