Sniff, Scrape, Crawl (Prototyping)

In 2011, Sniff, Scrape, Crawl was a thematic project led by Aymeric Mansoux, Renee Turner, and Michael Murtaugh.

This prototyping module will in part revisit some of the themes of this thematic project and in particular focus on the tools and practices of scraping.

Elements

• Spidering
• Crawling
• Indexing
• Summarizing
• Break up the steps of Whoosh's indexing ()

Tools

• python
• Scrapy

Examples

• Tumblr Jumper
• News Tweek
• Birgit Bachler's Bonus Card Friends

Links

• http://blog.osvdb.org/2014/05/07/the-scraping-problem-and-ethics/