User:Natasa Siencnik/workshop1

Rumor of Revolution

Web crawler searching Twitter for a Revolution

1 - Ideas

• 

2 - Technical Flowchart

3 - Script

File : twitterFeed-revolution

#!//usr/bin/python

import urllib2, json, sys, codecs

url = 'http://search.twitter.com/search.json?q=%23revolution'
f = urllib2.urlopen(url)
data = json.load(f)

output = codecs.open("output.txt", "w", encoding='utf-8')

for entry in data['results']:
    line = entry['created_at'] + ' : ' + entry['text']
    output.write(line)
    output.write('\n')

http://docs.djangoproject.com/en/1.2/intro/tutorial02/

#!/usr/bin/python
# -*- coding: utf-8 -*-

import os, urllib2, json, sys, codecs

def twittersearch(tag): 
    url = 'http://search.twitter.com/search.json?q=%%23%s' % tag
    f = urllib2.urlopen(url)
    data = json.load(f)
     
    out = []

    for entry in data['results']:
        d = {}
        for key in ('created_at', 'id', 'text', 'iso_language_code') :
            d[key] = entry[key]
        out.append(d)
        
    return out
    

list_of_languages = [
Rewolusie,
Revolucion,
الثورة,
Հեղափոխություն,
Inqilab,
Iraultza,
Рэвалюцыі,
Революция,
Revolució,
革命,
Revolucija,
Revoluce,
Revolution,
Revolutie,
Revolutsioon,
Rebolusyon,
Vallankumous,
Révolution,
Revolución,
რევოლუცია,
Επανάσταση,
Revolisyon,
מהפכה,
क्रांति,
Forradalom,
Revolusi,
Réabhlóid,
Rivoluzione,
革命,
רעוואָלוציע,
혁명,
Revolūcija,
Revoliucija,
Револуција,
Rivoluzzjoni,
انقلاب,
Rewolucji,
Revolução,
Revoluţie,
Революции,
Револуција,
Revolúcia,
Mapinduzi,
การปฏิวัติ,
Devrim,
Революції,
Cách mạng,
Chwyldro,
]

out = []

for tag in list_of_lanuages:
    out.extend(twittersearch(tag))

if entry['id'] in ids :
        output.write ('non-unique id found\n') 
    else :
        ids.append(entry['id'])

for entry in out:
    print("%(created_at)s - %(id)d : %(text)" % entry)
    
    
sys.exit()

File : twitterFeed-list

Rewolusie
Revolucion
الثورة
Հեղափոխություն
Inqilab
Iraultza
Рэвалюцыі
Революция
Revolució
革命
Revolucija
Revoluce
Revolution
Revolutie
Revolution
Revolutsioon
Rebolusyon
Vallankumous
Révolution
Revolución
რევოლუცია
Revolution
Επανάσταση
Revolisyon
מהפכה
क्रांति
Forradalom
Revolution
Revolusi
Réabhlóid
Rivoluzione
革命
רעוואָלוציע
혁명
Revolūcija
Revoliucija
Револуција
Revolusi
Rivoluzzjoni
Revolution
انقلاب
Rewolucji
Revolução
Revoluţie
Революции
Револуција
Revolúcia
Revolucija
Revolución
Mapinduzi
Revolution
Rebolusyon
การปฏิวัติ
Devrim
Революції
انقلاب
Cách mạng
Chwyldro

4 - Languages

Afriaans: Rewolusie Albanian: Revolucion Arabic: الثورة Armenian: Հեղափոխություն Azerbaijjani: Inqilab Basque: Iraultza Belarusian: Рэвалюцыі Bulgarian: Революция Catalan: Revolució Chinese: 革命 Croatian: Revolucija Czech: Revoluce Danish: Revolution Dutch: Revolutie English: Revolution Estonian: Revolutsioon Filipino: Rebolusyon Finnish: Vallankumous French: Révolution Galician: Revolución Georgian: რევოლუცია German: Revolution GreeK: Επανάσταση Haitian: Revolisyon Hebrew: מהפכה Hindi: क्रांति Hungarian: Forradalom Icelanding: Revolution Indonasian: Revolusi Irish: Réabhlóid Italian: Rivoluzione Japanese: 革命 Jiddish: רעוואָלוציע Korean: 혁명 Latvian: Revolūcija Lithuanian: Revoliucija Macedonian: Револуција Malay: Revolusi Maltese: Rivoluzzjoni Norvegian: Revolution Persian: انقلاب Polish: Rewolucji Portuguese: Revolução Romanian: Revoluţie Russian: Революции Serbian: Револуција Slovak: Revolúcia Slovenian: Revolucija Spanish: Revolución Swahili: Mapinduzi Swedish: Revolution Tagalog: Rebolusyon Thai: การปฏิวัติ Turkish: Devrim Ukrainian: Революції Urdu: انقلاب Vietnamese: Cách mạng Welsh: Chwyldro

5- Outcome
Mon, 24 Jan 2011 18:44:48 +0000 : Eating Cinnabon & preparing 4 #jan25 demo #Egypt #revolution
Mon, 24 Jan 2011 18:38:08 +0000 : Its time .. the chance that we should take.. #Revolution NOW - DOWN WITH 8 and 14 March !
Mon, 24 Jan 2011 18:34:24 +0000 : if it is time for anything.. its time for the #Revolution of the people, WAKE UP !

6 - Questions
Where can I find and implement a location or time_zone?
How can I combine the python-script with the search-list?
What kind of program do I need to run this on a server?
How can I implement a time stamp and check for doubles?

And thanks to Laura and Lieven for answering my stupid questions. :)