User:Tash/grad prototyping: Difference between revisions

From XPUB & Lens-Based wiki
No edit summary
No edit summary
Line 45: Line 45:
scrapy crawl titles -o titles.json
scrapy crawl titles -o titles.json
</source>
</source>
<br>
===== To explore =====
* NewsDiffs
* scraping the engagements / reactions / comments?

Revision as of 11:44, 2 October 2018

Prototyping Session 1 & 2

Every Redaction, by James Bridle


Possible topics to explore:


Learning to use Scrapy

Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

Documentation: https://docs.scrapy.org/en/latest/index.html

Scraping headlines from an Indonesian news site:
Screen Shot Scrapynews1.png

Using a spider to extract header elements (H5) from: http://www.thejakartapost.com/news/index

import scrapy
class TitlesSpider(scrapy.Spider):
    name = "titles"

    def start_requests(self):
        urls = [
            'http://www.thejakartapost.com/news/index',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        for title in response.css('h5'):
            yield {
                'text': title.css('h5::text').extract()
            }

Crawling and saving to a json file:

scrapy crawl titles -o titles.json



To explore
  • NewsDiffs
  • scraping the engagements / reactions / comments?