User:Ruben/TP2/On Digital Interpretation
The digital photobook
Parallel to the 'physical photobook' is the project wich currently runs under a working title On Digital Interpretation. It is an experiment on digital interpretation of (news) images. What does the world look like, when you only make relations using (news) images?
On Digital Information is a system that gathers news items with their photograph. As a new item comes in, the so called 'SIFT' algorithm tries to find notable parts in the photograph (ie. where there's high contrast): local features. The system then uses these features to compare the photograph with existing news photographs in de database. When if finds matching features between photos it adds the corresponding news item to the interface. Thus constituting a collection of related news items. Not related trough their content, but through the computer's interpretation of the photographs.
When we see a photograph or read any sort of 'text'[1], we shape our understanding of it through the relations we make in our mind. These associations not only provide us with meaning, they also contribute to the eliciting of emotions.[2] They provide some sort of context trough which we see the image. This process is mimicked in computers using deep learning algorithms. When for example such an algorithm is presented with a word, it collects other texts in which this word occurs. This process can then be repeated on all the words in the text. If this is continued for several steps the computer recursively gets some sort of 'understanding' of the word.
Understanding then becomes a discussable term. Let's have a look at the Oxford Dictionary:
1 [ with obj. ] perceive the intended meaning of (words, a language, or speaker)[...] perceive the significance, explanation, or cause of (something)[...] be sympathetically or knowledgeably aware of the character or nature of [...] interpret or view (something) in a particular way
2 [ with clause ] infer something from information received (often used as a polite formula in conversation) [...] regard (a missing word, phrase, or idea) as present [...] assume to be the case; take for granted
Without getting into a language game, it seems that the 'understanding' of texts for computers blurs the distinction between (1) "perceive the inteded meaning of" and (2) "infer something from information received". But as this computer model is based on the (supposed) working of the human brain, then this automatically questions what understanding means for us, humans.
What if we would filter out particular forms of texts to shape our understanding by? This project is an experiment to look into another way of relating to the world: trough news images. What does it then look like?
This form of digital interpretation becomes particularly interesting when one considers that code doesn't 'recognise' what it sees on an image. An algorithm is used to find notable, distinguishable points - the so called 'features' of an image. These features in turn, can be used to compare images.[3] Do they share any of their features? Then this is a relation. The more features are shared, the stronger the relation. Repeating this recursively one might talk about a algorithmic version of associating.
This project tries to present a digital/computer/algorithmic view on interpretation. And by that might challenge us to reconsider what we humans see as 'understanding'.
A technical explanation is at the corresponding Prototyping page
- ↑ 'Text' is here in its broadest use. This includes films, paintings etc.
- ↑ Some also relate the occurring of emotions when seeing images also to other systems. Ie. automatic mimicry (part of the Autonomic Nervous System - ANS) but the relations between these effects is disputed.
- ↑ This is a simplified way of looking at image comparison in images. As saying "what is a feature", already depends on the used algorithms. Next to that, many parameters of an image can be used in the equation. Its size, colour etc.