User:Ruben/RWRM/Third draft of project proposal

A Paradoxical Fear For The Self

Researching the fear for quantifying the constructed self image.

Introduction

For my graduation at the Piet Zwart Institute, I want to research the construction and controlling of the self image, from both an external and an internal perspective. How do we look at ourselves and how do we want others to perceive us? I want to approach this topic in relation to the quantification of our emotions using computational emotion analysis. If acknowledge that photography has changed the self image, then we can imagine how emotion analysis might do the same.

As I have a split background and interest in film and programming, I want to balance that border by devising fictional scenes and social games/experiments – all within a frame of computer vision and its influence on human-to-human and human-to-self relations. These two strands could inform and counter each other, and I might try to braid together in a later stadium.

Scenes

The first strand will be an ongoing process of writing and producing short (simple?) scenes on the topics described. These scenes don’t have to form a strict coherent narrative per se. Rather, they provide an outlet to formulate my ideas and see how they relate together and evolve over time. Hopefully these scenes will in turn provide an inlet for new ideas and directions. Somewhat like sketching with a very heavy pencil.

Social experiments

The second strand builds further on the technology I employed in We know how you feel. By using the tools in social experiments I want play with the impact of the technology on the self. These experiments don’t necessarily need to be truthful, they could just as well be faked and presented as real, in order to look at the impact.

A first experiment that I’m setting up now is as follows: imagine two people sitting across a table, they don’t know each other – as in a speed date. Next to their heads I mount a tablet, readable by the opposing person. When they start a conversation they are still off, but after a few minutes I enable the tablets, that will show the emotion as is detected by the Affectiva software. At first it shows the emotion of the person next to the tablet – so it becomes like a subtitle; another translation of what one is looking at. Then I switch it to show the emotion of the person itself – more like a (judging) mirror.

By recording this I could look at the impact of the tool. In the process, I could play with whether the emotions named on screen are really detected by the system, or put there by me. Which could potentially enable me to steer the conversation.

Only following the first exeriments I can tell whether these experiments have the potential to constitute some sort of lab setting with continuous experiments, or whether it will be distinct prototypes.

Theoretical Context

Looking at emotions as Foucauldian technologies of the self –as tools we use to shape our self image– through the make-believe theory by Kendall Walton –which states emotions are always real within their own context– brought a paradox to light: on the one hand I am bothered by the desire to control (by quantifying) ones own emotions, on the other I state that the self is a construct any way, so why would a quantification be problematic?

In this section I will try to outline this paradox, which forms the central theme to my research.

Emotion Analysis

Over the last few years a new technology has been booming: emotion analysis, which promises to be able to determine our unconscious, inner drives based on our (primarily facial) expressions.

Using webcams they scanned and tagged many faces, resulting in a quite precise measurement of how much we smile, furrow our brow etc. Beside these quite objective parameters, they also use it to determine how angry, sad, or happy we are. These ‘emotions’ become labels of how feel and what we want. Our face not only becomes the representation of our inner state, but also –as it is the only way for the systems to ‘understand’ our unconscious– our face becomes our inner state: it becomes a parameter on which is determined how to act. The representation (the facial expression) and the thing itself (the feeling) become intermingled.

I believe we should be careful in what does and what doesn’t become part of a quantified self. We can use the systems of emotion analysis to measure our expressions but should not state that they measure our feelings – which for the APA forms part of the definition of emotions. In particular because feelings seem to be fluid ambiguous entities.

Games of make-believe

In his 1978 essay Fearing Fictions, Kendall Walton introduced his make-believe theory. This theory concerns the emotions we express (in the APA sense, so both expressions and feelings) when watching a film or reading a book. It explains these emotions based on a game a child is playing with its father. The father, acting as a monster, chases the child. The child both laughs with joy and screams with fear. According to Walton both emotions the child experiences are real within the context of that emotion. Within the game the child is chased by a monster, its fear is real; within the game that he is playing with his father, his joy is real.

After reading this text I asked myself: are emotions not all context dependent? Are our interactions not always some sort of ‘games of make-believe’? Do we exhibit our emotions not based on the context of any ‘game’ we have?

Status games

This view seems supported by Keith Johnstone whom, in his book Impro, talks about ordinary conversations as status games. Johnstone stumbled upon this when working with improvisation theatre. His actors had trouble faking ordinary conversations – chitchat. But when Johnstone requested them to let each and every remark, question or move in a conversation be either consciously status raising, or status lowering, the conversations immediately became much more natural. The ‘real’ but hidden intentions of a conversation had surfaced.

Apparently, within our interactions we are very much aware of our place within some sort of hierarchy. We are constantly fighting and acknowledging this place. We all have participated in conversations where someone tells he has broken his arm, and there are others who cannot wait to tell about their own injury which was much worse than that. Considering this, it seems not too hard to imagine such conversations as status games.

Foucault’s Technologies of the Self

I found it therefore remarkable that Foucault talks about the sciences as ‘truth games’. They create a discourse related to specific techniques, employed by humans to understand themselves. According to Foucault technologies of the self are those technologies that enable the human being to shape itself – not only for the outside world, but also to shape its thoughts and its self-image.

Considering emotions as technologies of the self might also not be too far off. As it is shown in a research by Paul Ekman that humans express their emotions differently when they are in a group, compared to when they are alone.

Foucaults notions of the technologies of the self – the self as a construct depending on discourse and personal labour – seem to match up with Walton’s theory of make-believe, but also with my suggestion that maybe all our interactions form games.

Massumi’s emotions/feelings/affect

In order to not too make the definition of emotion to vague, I use Brian Massumi’s distinction between emotions, feelings and affect. According to Massumi, affect is a bodily, non-conscious, or pre-conscious intensity. Eric Shous (2005) explains Massumi’s definitions as follows:

A feeling is a sensation that has been checked against previous experiences and labelled. […] An emotion is the projection/display of a feeling. Unlike feelings, the display of emotion can be either genuine or feigned.

An infant has no language skills with which to cognitively process sensations, nor a history of previous experiences from which to draw in assessing the continuous flow of sensations coursing through his or her body. Therefore, the infant has to rely upon intensities (a term that Massumi equates with affect). […] It is what determines the intensity (quantity) of a feeling (quality), as well as the background intensity of our everyday lives (the half-sensed, ongoing hum of quantity/quality that we experience when we are not really attuned to any experience at all).

This definition enables me to distinguish between facial expressions (emotions), ones conscious inner state (feelings) and ones direct unconscious bodily state (affect). Thereby avoiding the blending between these states that the MindReading Emotion Library and many emotion analysis tools tend to do. By directly linking expressions to feelings (whatever the tag ‘hysterical’ then might mean) the representation and the thing itself get mixed.

My previous work

In previous works I have already broached upon some of these topics. In particular the multiplicity and subjectivity of human experience:

We know how you feel

In June 2015 I have been working on We know how you feel, which was a preliminary look into emotion analysis and its presumptions of human emotions. The work is in essence a re-contextualisation of two existing tools. On the one hand it consisted of an app that was based on existing commercial emotion analysis software. It acts as a mirror that shows quite objective facial parameters such as smile, brow raise, brow furrow etc. I added a voice over that hints at the effects of emotion analysis on the self image.

I juxtaposed this app with the Mind Reading Emotion Library (Cohen, 2004), a digital library which indexed 412 emotions, in 24 categories, with voice and video samples. The video fragments were without any context –just someone acting out an ‘emotion’ in a rather stereotypical way. This exposed the main flaw of facial emotion analysis: the lack of context.

Fragments of reality

[re-contextualisation over representation]

That interpretation is context dependent is also present in Fragments of reality, a work I created March 2015 on the interpretation of images. Following critiques from both Joris Luyendijk and Susan Sontag, this works critiques the ‘burden of truth’ that is often ascribed to photographs.

It is a short research into how photographs steer interpretation of an event – in this case a press conference by Jeroen Dijsselbloem and Yanis Varoufakis Obviously. The work does not attempt to be objective itself – as it denies the possibility of objectivity in interpretation. It is not an attempt to represent the images or the event. Rather, I re-contextualise them in an attempt to grab this phenomenon from its invisibility and render it visible.

Waterdagen - Me as filmmaker

In my graduation work of the Utrecht School of the Arts (HKU), I already tried to visualise a subjective world of the main character. Something I believe the film media is well suited for.

The notion of emotions is historically strongly embedded in cinema. Both to show it to an audience, and to trigger it in an audience. Therefore I think that it is inescapable to also employ this medium during my research into the construction of the self and the place that emotions take in that construct.

On the other hand I believe the process I used to create Waterdagen – which is the traditional process of filmmaking, namely a process with distinctive phases: writing, pre-producing, production and post-production – seems not to suit a research informed practice, in which I want practice and research to go hand in hand.

The Plan®

In order for the two strands to work out, I will provide a preliminary schedule for the project.

Before the 30th of November (the day of the project proposal presentations), I will have done one or two of the social experiments with the emotion analysis tool. Including the speed date as described in this text. For the other strand I will have one or two scenes ready which I can film before the end of the year.

For there on I will have January until the end of March to devise more experiments and write and film some more scenes. For each experiment and scene I will write a short description, stating how it relates to the previous works and to (new) theory.

That leaves me with two months (April and May) to see how to culminate the work into one statement, based on all previous experiments and videos. Or whether I want to present the work in other ways.

In conclusion

I believe that in particular the second strand of my research – the experiments– is an interesting approach to the subject. On the other hand, juxtaposing it with narrative fragments might elevate the material to a more ‘metaphysical’ level –it might give it an edge.

Therefore, for now, it is important that I bootstrap both strands into concrete prototypes. Until there is something tangible, there is not much more to talk about.