User:Ruben/RWRM/Second draft of project proposal

Construction of the self image

Summary

For my graduation at the Piet Zwart Institute, I want to research the construction and controlling of the self image -- from both an external and an internal perspective. Looking at emotions as technologies of the self, through the make-believe theory by Kendall Walton. As I have a split background and interest in film and programming, I want to balance that border by devising fictional scenes and social games/experiments -- all within a frame of computer vision and the human-computer relation.

Theoretical Context

Emotion Analysis

Over the last few years a new technology has been booming: emotion analysis, which promises to be able to determine our unconscious, inner drives based on our (primarily facial) expressions.

Using webcams they scanned and tagged many faces, resulting in a quite precise measurement of how much we smile, furrow our brow etc. Beside these quite objective parameters, they also use it to determine how angry, sad, or happy we are. These 'emotions' become labels of how feel and what we want. Our face not only becomes the representation of our inner state, but also -- as it is the only way for the systems to 'understand' our unconscious -- our face becomes our inner state: it becomes a parameter on which is determined how to act. The representation (the facial expression) and the thing itself (the feeling) become intermingled.

I believe we should be careful in what does and what doesn't become part of a quantified self. We can use the systems of emotion analysis to measure our expressions -- emotions in the definition of Massumi -- but should not state that they measure our feelings -- which for the APA forms part of the definition of emotions. In particular because feelings seem to be fluid ambiguous entities.

Games of make-believe

In his 1978 essay Fearing Fictions, Kendall Walton introduced his make-believe theory. This theory concerns the emotions we express (in the APA sense, so both expressions and feelings) when watching a film or reading a book. It explains these emotions based on a game a child is playing with its father. The father, acting as a monster, chases the child. The child both laughs with joy and screams with fear. According to Walton both emotions the child experiences are real within the context of that emotion. Within the game the child is chased by a monster, its fear is real; within the game that he is playing with his father, his joy is real.

After reading this text I asked myself: are emotions not all context dependent? Are our interactions not always some sort of 'games of make-believe'? Do we exhibit our emotions not based on the context of any 'game' we have?

Status games

This view seems supported by Keith Johnstone whom, in his book Impro, talks about ordinary conversations as status games. Johnstone stumbled upon this when working with improvisation theatre. His actors had trouble faking ordinary conversations -- chitchat. But when Johnstone requested them to let each and every remark, question or move in a conversation be either consciously status enhancing, or status lowering, (LOOK UP THE WORDS!) the conversations immediately became much more natural. The 'real' but hidden intentions of a conversation had surfaced.

Apparently, within our interactions we are very much aware of our place within some sort of hierarchy. We are constantly fighting and acknowledging this place. We all have participated in conversations where someone tells he has broken his arm, and there are others who cannot wait to tell about their own injury which was much worse than that. Considering this, it seems not too hard to imagine such conversations as status games.

Foucault's Technologies of the Self

I found it therefore remarkable that Foucault talks about the sciences as 'truth games'. They create a discourse related to specific techniques, employed by humans to understand themselves. According to Foucault technologies of the self are those technologies that enable the human being to shape itself -- not only for the outside world, but also to shape its thoughts and its self-image.

Considering emotions as technologies of the self might also not be too far off. As it is shown in a research by Paul Ekman that humans express their emotions differently when they are in a group, compared to when they are alone.

Foucaults notions of the technologies of the self -- the self as a construct depending on discourse and personal labour -- seem to match up with Walton's theory of make-believe, but also with my suggestion that maybe all our interactions form games.

Massumi's emotions/feelings/affect

In order to not too make the definition of emotion to vague, I use towards Brian Massumi's distinction between emotions, feelings and affect. According to Massumi, affect is a bodily, non-conscious, or pre-conscious intensity. Eric Shous (2005) explains Massumi's definitions as follows:

A feeling is a sensation that has been checked against previous experiences and labelled. [...] An emotion is the projection/display of a feeling. Unlike feelings, the display of emotion can be either genuine or feigned.

An infant has no language skills with which to cognitively process sensations, nor a history of previous experiences from which to draw in assessing the continuous flow of sensations coursing through his or her body. Therefore, the infant has to rely upon intensities (a term that Massumi equates with affect). [...] It is what determines the intensity (quantity) of a feeling (quality), as well as the background intensity of our everyday lives (the half-sensed, ongoing hum of quantity/quality that we experience when we are not really attuned to any experience at all).

This definition enables me to distinguish between facial expressions (emotions), ones conscious inner state (feelings) and ones direct unconscious bodily state (affect). Thereby avoiding the blending between these states that the MindReading Emotion Library and many emotion analysis tools tend to do. By directly linking expressions to feelings (whatever the tag 'hysterical' then might mean) the representation and the thing itself get mixed.

Cynicism and Kynicism

I have the suspicion that technologies that help to quantify the self (which are technologies of the self pur sang), such as emotion analysis, both feed and are fed by a desire for control over the self. Something similar was acknowledged by Peter Sloterdijk in his work Critique of Cynical Reason (1983). He compares the contemporary -- and according to him, predominant -- diffuse cynical mindset to its etymological roots, the Greek Cynics, or kynics.

It seems Sloterdijk's perspective on the diffuse cynicism -- a disbelief in the goodness of human motives and actions, which he suggests is partly rooted in political disappointment -- contains this constant desire for control. He proposes to steer away from disillusionment with its cynicism and melancholy. Towards the rational that is gone from scientific enlightenment: kynicism.

As the Greek Cynics went 'back to nature' (and gave up their money etc.), Sloterdijk suggests that a counter measure against cynicism would be to let go of full control. Or more prominent, to undermine structures of control -- a move that could be seen as cynical in itself.

It is this multiplicity -- that undermining structures of control is in itself a act that rises out of a desire for more control -- that is also present in (if not central to!) my current perspective on emotion analysis. On the one hand I reject the desire to control (by quantifying) ones own emotions, on the other I state that the self is a construct any way, so why would a quantification be problematic?

My previous work

This paradox of multiplicity was also present in other works that I have made so far.

We know how you feel

In June 2015 I have been working on We know how you feel, which was a preliminary look into emotion analysis and its presumptions of human emotions. The work is in essence a re-contextualisation of two existing tools. On the one hand it consisted of an app that was based on existing commercial emotion analysis software. It acts as a mirror that shows relatively objective facial parameters such as smile, brow raise, brow furrow etc. I added a voice over that hints at the effects of emotion analysis on the self image.

I juxtaposed this app with the Mind Reading Emotion Library (Cohen, 2004), a digital library which indexed 412 emotions, in 24 categories, with voice and video samples. The video fragments were without any context --just someone acting out an 'emotion' in a rather stereotypical way. This exposed the main flaw of facial emotion analysis: the lack of context.

Even the humans interpret emotions based on context. As was shown in the famous film editing experiment by Kuleshov where a shot of a man was cut into three different sequences: once with a female, once with a plate of soup and once with a dead body. The audience raved about the acting qualities of the man; he was considered to act attracted, hungry and sad -- depending on the sequence that was shown.

It comes at no surprise that one is often wrong in their assumptions of the other's feeling, when they are derived from its emoting. The assumption that an algorithm, which considers context only at an extremely limited level and which has trouble acknowledging multiplicity (as is present in its premise of the 'digital'), could predict feelings better than humans seems then highly unlikely.

Fragments of reality

[re-contextualisation over representation]

This context dependency of interpretation was also present in Fragments of reality, a work I created March 2015 on the interpretation of images. Following critiques from both Joris Luyendijk and Susan Sontag, this works critiques the 'burden of truth' that is often ascribed to photographs.

It is a short research into how photographs steer interpretation of an event -- in this case a press conference by Jeroen Dijsselbloem and Yanis Varoufakis Obviously. The work does not attempt to be objective itself -- as it denies the possibility of objectivity in interpretation. It is not an attempt to represent the images or the event. Rather, I re-contextualise them in an attempt to grab this phenomenon from its invisibility and render it visible.

Waterdagen - Me as filmmaker

In my graduation work of the Utrecht School of the Arts (HKU), I already tried to visualise a subjective world of the main character. Something I believe the film media is well suited for.

The notion of emotions is historically strongly embedded in cinema. Both to show it to an audience, and to trigger it in an audience. Therefore I think that it is inescabable to also employ this medium during my research into the construction of the self and the place that emotions take in that construct.

On the other hand I believe the process I used to create Waterdagen -- which is the traditional process of filmmaking, namely a process with distinctive phases: writing, pre-producing, production and post-production -- seems not to suit a research informed practice, in which I want practice and research to go hand in hand.

The Plan®

Considering all the above, I would like to propose the following plan for my graduation project. From the start on I would like it to contain two strands, which I might try to braid together in a later stadium.

Scenes

The first strand will be an ongoing process of writing and producing short (simple?) scenes on the topics described. These scenes don't have to form a strict coherent narrative per se. Rather, they provide an outlet to formulate my ideas and see how they relate together and evolve over time. Hopefully these scenes will in turn provide an inlet for new ideas and directions. Somewhat like sketching with a very heavy pencil.

Social experiments

The second strand builds further on the technology I employed in We know how you feel. By using the tools in social experiments I want play with the impact of the technology on the self. These experiments don't necessarily need to be truthful, they could just as well be faked and presented as real, in order to look at the impact.

A first experiment that I'm setting up now is as follows: imagine two people sitting across a table, they don't know each other -- as in a speed date. Next to their heads I mount a tablet, readable by the opposing person. When they start a conversation they are still off, but after a few minutes I enable the tablets, that will show the emotion as is detected by the Affectiva software. At first it shows the emotion of the person next to the tablet -- so it becomes like a subtitle; another translation of what one is looking at. Then I switch it to show the emotion of the person itself -- more like a (judging) mirror.

By recording this I could look at the impact of the tool. In the process, I could play with whether the emotions named on screen are really detected by the system, or put there by me. Which could potentially enable me to steer the conversation.

Conclusion

I believe that particular this second strand is an interesting approach to the subject. On the other hand, juxtaposing it with narrative fragments might elevate the material to a more 'metaphysical' level -- it might give it an edge.

Therefore, for now, it is important that I bootstrap both strands into concrete prototypes. Until there is something tangible, there is not much more to talk about.