Sniff, Scrape, Crawl (Thematic Project)
'Sniff, Scrape, Crawl…
Trimester 2, Jan.-March 2011
Thematic Project Tutors: Aymeric Mansoux, Michael Murtaugh, Renee Turner
Our society is one not of spectacle, but of surveillance…
Michel Foucault, Discipline and Punish: The Birth of the Prison, trans. Alan Sheridan (New York, 1979), p. 217
Where once surveillance technologies belonged to governmental agencies, the web has added another less optically-driven means of both monitoring and monetizing our lived experiences. As the line between public and private has become more blurred and the desire for convenience ever greater, our personal data has become a prized commodity upon which industries thrive. Perversely, we have become consumers who simultaneously produce the product through our own consumption.
Sniff, Scrape, Crawl… is a thematic project examining how surveillance and data-mining technologies shape and influence our lives, and what consequences they have on our civil liberties. We will look at the complexities of sharing information in exchange for waiving privacy rights. Next to this, we will look at how our fundamental understanding of private life has changed as public display has become more pervasive through social networks. Bringing together practical exercises, theoretical readings and a series of guest lectures, Sniff, Scrape, Crawl… will attempt to map the data trails we leave behind and look critically at the buoyant industries that track and commodify our personal information.
References: Michel Foucault, Discipline and Punish: The Birth of the Prison, trans. Alan Sheridan (New York, 1979) Wendy Hui Kyong Chun, Control and Freedom: power and paranoia in the age of fiber optics, (London, 2006)
Note: This Thematic Project will be organized and taught by Aymeric Mansoux, Michael Murtaugh, Renee Turner and will involve a series of related guest lectures and presentations.
Workshop 1: 20-21 Jan, 2011
Morning: SNIFF, Web 0.0
- The joy of basic client/server Netcat Chat
- Internet geography http://hulu.com
- Connect to pzwart3 "geoip" script with netcat like this TODO: put script on server, make a recipe and link it here
- Repeat the same with a browser and witness "Software sorting"
- Create generic user account on non discriminated machine, following these instructions
- tunnel traffic through this machine with this trick
- Wrap network traffic of almost any application with a SOCKS proxy
- Sniff requested URL on the proxy machine with urlsnarf
- A word on traceRoute & DNS, Internet topology (http://laramies.blogspot.com/2006/05/2d-and-3d-traceroute-with-scapy.html, http://www.visualcomplexity.com/vc/project.cfm?id=332)
- A word on VPN, proxies and other delicatesses
- Back to "geoblocking" script and explain the code and Python CGI
- Unix permissions (basics, but also evt. link to API "permissions" in the Web 2.0 world)
- sniffy the pony really does exist
Afternoon: SCRAPE, Web 1.0 / Simple Spider
Start with ipython. Getting documentation from ipython Using urllib, open a connection to a page (connect) Scan for images in a page. Loop
- Do everything by hand....
- Make Script, put in bin
(how to )
Open connections, interrogate the response.
- HTTP, URL, CSS, get vs. post, query strings, urlencoding
Cookbook: Simple Web Spider in Python
Morning: APIness / Web 2.0
We now move from the surface of the "scrape" to the deeper database structure of a web 2.0 service and the data available via an API.
We took a look at the anatomy of a tweet, as an introduction to the "cloud" of data around the 140 characters of a tweet publicly available via the Twitter API.
We looked at an example of an "IPad hack" involving cross referencing device identifiers and emails via different publicly available sites.
Changes announced on the Facebook blog in response to privacy concerns.
Goal: Apply the same spider script -- now using data/connections available via the API.
Afternoon: CRAWL with Django
slowly but steadily, a new structure emerges.
Mapping JSON/API structure to a database. Code to map source to db. Using Django admin to visualize data.
>> assignment in teams: for instance:
- get a list of tags from a userid
- quick note about scrapy
Assignment: Tactical Crawler
Assignment is to move from a Survey to a Strategy, and to prepare a plan for a "tactical crawler" related to your research.
Your plan should both:
- Express a strategy to explore a particular / personal research question / interest / frustration / concern, and be focused on a particular use / audience / context.
- Be grounded in actual data / connections such as those explored / viewed in the workshop (via scraping or API explorations). Show particular data gathered from scraping/api's gathered "by hand" or by simple scripts (such as those used in the workshop) to support your plan.
Important: Unless it already fits your plan, you do not necessarily need to work with the Django code / crawler example from the last afternoon session. This will (most likely) become more important as your work develops -- for now the important thing is the creation of your plan and your initial scrapes & the specifics of the kind of data available from APIs.
We realize that the two day workshop was an intense session that might have been a little overwhelming. Keep in mind that this was meant to give you an overview of the different tools and strategies available for scraping and data mining. It is now up to you to decide on which elements you wish to focus more and spend the rest of the trimester working with.
Your assignment/plan will be assessed this coming Tuesday, 25 January
Tuesday's meeting will involve each of you presenting your plan to the group and a discussion of the plans.
Happy Sniffing & Scraping!