Scraping

Scraping (also Screen Scraping) is the process of extracting data out of something.

In the course, we have used the library BeautifulSoup to manipulate HTML pages in Python.

Other interesting libraries to consider:

• Mechanize in essence simulates a browser in Python, that can "remember" things (like cookies / sessions) between pages
• lxml which can apparently deal with "mal-formed" HTML and quickly convert them to xml trees
• html5lib
• http://github.com/davisp/python-spidermonkey/tree/master Python SpiderMonkey... hmmm not sure exactly what this does yet!

Resources:

• http://us.pycon.org/2009/tutorials/schedule/2AM8/