User:Natasa Siencnik/workshop1

Rumor of Revolution

Web crawler searching Twitter for a Revolution

1 - Ideas

2 - Technical Flowchart

3 - Script

File : twitterFeed-revolution

#!/usr/bin/python

import urllib2, json, sys, codecs

url = 'http://search.twitter.com/search.json?q=%23revolution'
f = urllib2.urlopen(url)
data = json.load(f)

output = codecs.open("output.txt", "w", encoding='utf-8')

for entry in data['results']:
    line = entry['created_at'] + ' : ' + entry['text']
    output.write(line)
    output.write('\n')

http://docs.djangoproject.com/en/1.2/intro/tutorial02/

#!/usr/bin/python
# -*- coding: utf-8 -*-

import os, urllib2, json, sys, codecs

def twittersearch(tag): 
    url = 'http://search.twitter.com/search.json?q=%%23%s' % tag
    f = urllib2.urlopen(url)
    data = json.load(f)
     
    out = []

    for entry in data['results']:
        d = {}
        for key in ('created_at', 'id', 'text', 'iso_language_code') :
            d[key] = entry[key]
        out.append(d)
        
    return out
    

list_of_languages = [
Rewolusie,
Revolucion,
الثورة,
Հեղափոխություն,
Inqilab,
Iraultza,
Рэвалюцыі,
Революция,
Revolució,
革命,
Revolucija,
Revoluce,
Revolution,
Revolutie,
Revolutsioon,
Rebolusyon,
Vallankumous,
Révolution,
Revolución,
რევოლუცია,
Επανάσταση,
Revolisyon,
מהפכה,
क्रांति,
Forradalom,
Revolusi,
Réabhlóid,
Rivoluzione,
革命,
רעוואָלוציע,
혁명,
Revolūcija,
Revoliucija,
Револуција,
Rivoluzzjoni,
انقلاب,
Rewolucji,
Revolução,
Revoluţie,
Революции,
Револуција,
Revolúcia,
Mapinduzi,
การปฏิวัติ,
Devrim,
Революції,
Cách mạng,
Chwyldro,
]

out = []

for tag in list_of_lanuages:
    out.extend(twittersearch(tag))

if entry['id'] in ids :
        output.write ('non-unique id found\n') 
    else :
        ids.append(entry['id'])

for entry in out:
    print("%(created_at)s - %(id)d : %(text)" % entry)
    
    
sys.exit()

4 - Languages

Afriaans: Rewolusie Albanian: Revolucion Arabic: الثورة Armenian: Հեղափոխություն Azerbaijjani: Inqilab Basque: Iraultza Belarusian: Рэвалюцыі Bulgarian: Революция Catalan: Revolució Chinese: 革命 Croatian: Revolucija Czech: Revoluce Danish: Revolution Dutch: Revolutie English: Revolution Estonian: Revolutsioon Filipino: Rebolusyon Finnish: Vallankumous French: Révolution Galician: Revolución Georgian: რევოლუცია German: Revolution GreeK: Επανάσταση Haitian: Revolisyon Hebrew: מהפכה Hindi: क्रांति Hungarian: Forradalom Icelanding: Revolution Indonasian: Revolusi Irish: Réabhlóid Italian: Rivoluzione Japanese: 革命 Jiddish: רעוואָלוציע Korean: 혁명 Latvian: Revolūcija Lithuanian: Revoliucija Macedonian: Револуција Malay: Revolusi Maltese: Rivoluzzjoni Norvegian: Revolution Persian: انقلاب Polish: Rewolucji Portuguese: Revolução Romanian: Revoluţie Russian: Революции Serbian: Револуција Slovak: Revolúcia Slovenian: Revolucija Spanish: Revolución Swahili: Mapinduzi Swedish: Revolution Tagalog: Rebolusyon Thai: การปฏิวัติ Turkish: Devrim Ukrainian: Революції Urdu: انقلاب Vietnamese: Cách mạng Welsh: Chwyldro

5- Outcome
Mon, 24 Jan 2011 18:44:48 +0000 : Eating Cinnabon & preparing 4 #jan25 demo #Egypt #revolution
Mon, 24 Jan 2011 18:38:08 +0000 : Its time .. the chance that we should take.. #Revolution NOW - DOWN WITH 8 and 14 March !
Mon, 24 Jan 2011 18:34:24 +0000 : if it is time for anything.. its time for the #Revolution of the people, WAKE UP !

6 - Questions
Where can I find and implement a location or time_zone?
How can I use different languages (unicode / utf-8)?
What kind of program do I need to run this on a server?
How can I implement a time stamp and check for doubles?

And thanks to Stock, Laura and Lieven for answering my stupid questions. :)