Web scraping with Python - Revision history

Michael Murtaugh at 15:47, 26 May 2014

2014-05-26T15:47:07Z

← Older revision		Revision as of 17:47, 26 May 2014
Line 3:		Line 3:
	* [[html5lib]]		* [[html5lib]]
	* [[ElementTree]] part of the standard python library		* [[ElementTree]] part of the standard python library

	html5lib is a python library for parsing "html in the wild". A big advantage of working with html5lib, is that, unlike stricter XML tools, it will accept any input document, even when there are missing or incorrect tags. The library follows the "tolerant" behaviour of most web browsers and is very useful as a bridge between "pages in the wild" and the precision of XML tools like ElementTree.

	While html5lib is not part of the standard python distribution, it is 100% "pure python" meaning that it's easy to use across platforms (even without a tool like pip working, you can download and import the library by simply placing it's folder in the same directory as your python script).

	== Examples ==		== Examples ==
	* [[Scraping the Open Directory with Python]]		* [[Scraping the Open Directory with Python]]
	* [[Wikiwalker: Crawling wikipedia pages for images]]		* [[Wikiwalker: Crawling wikipedia pages for images]]

Michael Murtaugh at 15:41, 26 May 2014

2014-05-26T15:41:55Z

← Older revision		Revision as of 17:41, 26 May 2014
Line 2:		Line 2:
	* [[python]]		* [[python]]
	* [[html5lib]]		* [[html5lib]]
	* [~~http://docs.python.org/2/library/xml.etree.elementtree.html~~ ElementTree] part of the standard python library		* [[ElementTree]] part of the standard python library

	html5lib is a python library for parsing "html in the wild". A big advantage of working with html5lib, is that, unlike stricter XML tools, it will accept any input document, even when there are missing or incorrect tags. The library follows the "tolerant" behaviour of most web browsers and is very useful as a bridge between "pages in the wild" and the precision of XML tools like ElementTree.		html5lib is a python library for parsing "html in the wild". A big advantage of working with html5lib, is that, unlike stricter XML tools, it will accept any input document, even when there are missing or incorrect tags. The library follows the "tolerant" behaviour of most web browsers and is very useful as a bridge between "pages in the wild" and the precision of XML tools like ElementTree.

Michael Murtaugh: /* ElementTree */

2014-05-26T15:41:39Z

ElementTree

← Older revision		Revision as of 17:41, 26 May 2014
Line 7:		Line 7:

	While html5lib is not part of the standard python distribution, it is 100% "pure python" meaning that it's easy to use across platforms (even without a tool like pip working, you can download and import the library by simply placing it's folder in the same directory as your python script).		While html5lib is not part of the standard python distribution, it is 100% "pure python" meaning that it's easy to use across platforms (even without a tool like pip working, you can download and import the library by simply placing it's folder in the same directory as your python script).

	~~== ElementTree ==~~

	While more sophisticated (and faster) libraries for working with HTML/XML exist (namely lxml), python's standard ElementTree implementation is quite capable (and using lxml requires a separate C module to be compiled on your platform which can sometimes complicate installation / distribution of your script).

	~~ElementTree supports a small subset of the (more extensive) xpath query language:~~

	~~See http://docs.python.org/2/library/xml.etree.elementtree.html#supported-xpath-syntax~~

	== Examples ==		== Examples ==
	* [[Scraping the Open Directory with Python]]		* [[Scraping the Open Directory with Python]]
	* [[Wikiwalker: Crawling wikipedia pages for images]]		* [[Wikiwalker: Crawling wikipedia pages for images]]

Michael Murtaugh at 15:41, 26 May 2014

2014-05-26T15:41:17Z

← Older revision		Revision as of 17:41, 26 May 2014
Line 2:		Line 2:
	* [[python]]		* [[python]]
	* [[html5lib]]		* [[html5lib]]
	* [~~https~~://docs.python.org/2/library/xml.etree.elementtree.html ElementTree] part of the standard python library		* [http://docs.python.org/2/library/xml.etree.elementtree.html ElementTree] part of the standard python library

	html5lib is a python library for parsing "html in the wild". A big advantage of working with html5lib, is that, unlike stricter XML tools, it will accept any input document, even when there are missing or incorrect tags. The library follows the "tolerant" behaviour of most web browsers and is very useful as a bridge between "pages in the wild" and the precision of XML tools like ElementTree.		html5lib is a python library for parsing "html in the wild". A big advantage of working with html5lib, is that, unlike stricter XML tools, it will accept any input document, even when there are missing or incorrect tags. The library follows the "tolerant" behaviour of most web browsers and is very useful as a bridge between "pages in the wild" and the precision of XML tools like ElementTree.
Line 11:		Line 11:

	While more sophisticated (and faster) libraries for working with HTML/XML exist (namely lxml), python's standard ElementTree implementation is quite capable (and using lxml requires a separate C module to be compiled on your platform which can sometimes complicate installation / distribution of your script).		While more sophisticated (and faster) libraries for working with HTML/XML exist (namely lxml), python's standard ElementTree implementation is quite capable (and using lxml requires a separate C module to be compiled on your platform which can sometimes complicate installation / distribution of your script).


	ElementTree supports a small subset of the (more extensive) xpath query language:		ElementTree supports a small subset of the (more extensive) xpath query language:

Michael Murtaugh at 15:36, 26 May 2014

2014-05-26T15:36:40Z

← Older revision		Revision as of 17:36, 26 May 2014
Line 3:		Line 3:
	* [[html5lib]]		* [[html5lib]]
	* [https://docs.python.org/2/library/xml.etree.elementtree.html ElementTree] part of the standard python library		* [https://docs.python.org/2/library/xml.etree.elementtree.html ElementTree] part of the standard python library

			html5lib is a python library for parsing "html in the wild". A big advantage of working with html5lib, is that, unlike stricter XML tools, it will accept any input document, even when there are missing or incorrect tags. The library follows the "tolerant" behaviour of most web browsers and is very useful as a bridge between "pages in the wild" and the precision of XML tools like ElementTree.

			While html5lib is not part of the standard python distribution, it is 100% "pure python" meaning that it's easy to use across platforms (even without a tool like pip working, you can download and import the library by simply placing it's folder in the same directory as your python script).

			== ElementTree ==

			While more sophisticated (and faster) libraries for working with HTML/XML exist (namely lxml), python's standard ElementTree implementation is quite capable (and using lxml requires a separate C module to be compiled on your platform which can sometimes complicate installation / distribution of your script).


			ElementTree supports a small subset of the (more extensive) xpath query language:

			See http://docs.python.org/2/library/xml.etree.elementtree.html#supported-xpath-syntax

	== Examples ==		== Examples ==
	* [[Scraping the Open Directory with Python]]		* [[Scraping the Open Directory with Python]]
	* [[Wikiwalker: Crawling wikipedia pages for images]]		* [[Wikiwalker: Crawling wikipedia pages for images]]

Michael Murtaugh: Michael Murtaugh moved page Web scraping with Python using html5lib and ElementTree to Web scraping with Python: more generic

2014-05-26T15:36:07Z

Michael Murtaugh moved page Web scraping with Python using html5lib and ElementTree to Web scraping with Python: more generic

← Older revision	Revision as of 17:36, 26 May 2014
(No difference)

Michael Murtaugh: /* Examples */

2014-05-26T14:57:36Z

Examples

← Older revision		Revision as of 16:57, 26 May 2014
Line 6:		Line 6:
	== Examples ==		== Examples ==
	* [[Scraping the Open Directory with Python]]		* [[Scraping the Open Directory with Python]]
			* [[Wikiwalker: Crawling wikipedia pages for images]]

Michael Murtaugh at 14:39, 26 May 2014

2014-05-26T14:39:26Z

← Older revision		Revision as of 16:39, 26 May 2014
Line 4:		Line 4:
	* [https://docs.python.org/2/library/xml.etree.elementtree.html ElementTree] part of the standard python library		* [https://docs.python.org/2/library/xml.etree.elementtree.html ElementTree] part of the standard python library

	[[Scraping the Open Directory ~~project~~ with Python]]		== Examples ==
			* [[Scraping the Open Directory with Python]]

Michael Murtaugh: /* Scraping dmoz.org */

2014-05-26T14:21:14Z

Scraping dmoz.org

Show changes

Michael Murtaugh: /* Scraping dmoz.org */

2014-05-26T14:15:50Z

Scraping dmoz.org

← Older revision		Revision as of 16:15, 26 May 2014
Line 5:		Line 5:

	== Scraping dmoz.org ==		== Scraping dmoz.org ==

			From the dmoz website:
			<blockquote>
			DMOZ is the largest, most comprehensive human-edited directory of the Web. It is constructed and maintained by a passionate, global community of volunteers editors. It was historically known as the Open Directory Project (ODP).
			</blockquote>

	=== Example 1: Pulling the URLs + textual descriptions from a single page ===		=== Example 1: Pulling the URLs + textual descriptions from a single page ===