Sniff, Scrape, Crawl (Prototyping): Difference between revisions
| Line 5: | Line 5: | ||
== Meeting 1 == | == Meeting 1 == | ||
=== Scraping Tools === | === Morning: Scraping Tools === | ||
Scraping tools and recipes exist at a variety of scales: | Scraping tools and recipes exist at a variety of scales: | ||
* small: [[Simple Web Spider in Python]] | * small: [[Simple Web Spider in Python]] | ||
* medium: [http://scrapy.org/ Scrapy], python "framework" inspired by web frameworks like Django specifically for scraping | * medium: [http://scrapy.org/ Scrapy], python "framework" inspired by web frameworks like Django specifically for scraping | ||
* large: [http://en.wikipedia.org/wiki/Heritrix Heritrix], full-fledged tools used for institutional scraping purposed like tradition libraries and the Internet archive (archive.org). | * large: [http://en.wikipedia.org/wiki/Heritrix Heritrix], full-fledged tools used for institutional scraping purposed like tradition libraries and the Internet archive (archive.org). | ||
We'll use the [[hot seat]] technique to get our collective feet wet with some simple scraping tools, focussing on the small to medium scale. | |||
=== Afternoon: Meeting to discuss / develop / brainstorm project ideas === | === Afternoon: Meeting to discuss / develop / brainstorm project ideas === | ||
Meet as a group to discuss/brainstorm ideas for individual research / projects. | |||
== Meeting 2 == | == Meeting 2 == | ||
Revision as of 15:34, 19 May 2014
In 2011, Sniff, Scrape, Crawl was a thematic project led by Aymeric Mansoux, Renee Turner, and Michael Murtaugh.
This prototyping module covers some of the core themes and tools around the practice of "scraping", with the goal to better familiarize yourself with the possibilities of this technique and to develop strategic uses of the tools for your specific research.
Meeting 1
Morning: Scraping Tools
Scraping tools and recipes exist at a variety of scales:
- small: Simple Web Spider in Python
- medium: Scrapy, python "framework" inspired by web frameworks like Django specifically for scraping
- large: Heritrix, full-fledged tools used for institutional scraping purposed like tradition libraries and the Internet archive (archive.org).
We'll use the hot seat technique to get our collective feet wet with some simple scraping tools, focussing on the small to medium scale.
Afternoon: Meeting to discuss / develop / brainstorm project ideas
Meet as a group to discuss/brainstorm ideas for individual research / projects.
Meeting 2
- Workshop / Tutorials
Meeting 3
"Proof of concept" presentations
Some Examples
- focused_crawls
- Lasse's Tumblr Jumper
- Birgit Bachler's Bonus Card Friends