User:Tash/grad prototyping

From XPUB & Lens-Based wiki

Prototyping Session 1 & 2

Every Redaction, by James Bridle


Possible topics to explore:


Learning to use Scrapy

Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

Documentation: https://docs.scrapy.org/en/latest/index.html

Scraping headlines from an Indonesian news site:
Screen Shot Scrapynews1.png

Using a spider to extract header elements (H5) from: http://www.thejakartapost.com/news/index

import scrapy
class TitlesSpider(scrapy.Spider):
    name = "titles"

    def start_requests(self):
        urls = [
            'http://www.thejakartapost.com/news/index',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        for title in response.css('h5'):
            yield {
                'text': title.css('h5::text').extract()
            }

Crawling and saving to a json file:

scrapy crawl titles -o titles.json



To explore
  • NewsDiffs – as a way to expose the historiography of an article
  • how about looking at comments? what can you scrape (and analyse) from social media?
  • self-censorship: can you track the things people write but then retract?
  • An Anthem to Open Borders