PythonFeedparser - Revision history

Michael Murtaugh at 17:33, 14 September 2010

2010-09-14T17:33:55Z

← Older revision		Revision as of 19:33, 14 September 2010
Line 5:		Line 5:
	Step one is simply using Feedparser to parse (read) a feed. In this case, the RSS feed of nytimes.com:		Step one is simply using Feedparser to parse (read) a feed. In this case, the RSS feed of nytimes.com:

	<~~pre~~>~~#!python numbers=off~~		<source lang="python">
	import feedparser		import feedparser

	fu = "http://www.nytimes.com/services/xml/rss/nyt/[[HomePage]].xml"		fu = "http://www.nytimes.com/services/xml/rss/nyt/[[HomePage]].xml"
	f = feedparser.parse(fu)		f = feedparser.parse(fu)
	</~~pre~~>		</source>

	If all goes well, this code does nothing. That is, it doesn't print anything out. In fact it connects to the nytimes server and reads and attempts to understand the contents of the feed.		If all goes well, this code does nothing. That is, it doesn't print anything out. In fact it connects to the nytimes server and reads and attempts to understand the contents of the feed.
Line 16:		Line 16:
	To see if it's working, you could run the script with the "-i" option of python, to leave python open afterwards:		To see if it's working, you could run the script with the "-i" option of python, to leave python open afterwards:

	<~~pre~~>		<source lang="bash">
	python -i feedreader.py		python -i feedreader.py
	</~~pre~~>		</source>

	TIP: Use pprint to "pretty" print out the resulting object (or parts of the object), for instance, we could look at the first item (via the "entities" attribute):		TIP: Use pprint to "pretty" print out the resulting object (or parts of the object), for instance, we could look at the first item (via the "entities" attribute):
	<~~pre~~>
			<source lang="python">
	from pprint import pprint		from pprint import pprint
	pprint(f.entries[0])		pprint(f.entries[0])
	</~~pre~~>		</source>

	produces:		produces:

	<~~pre~~>~~#!python numbers=off~~
			<source lang="python">
	{'author': u'ELAINE SCIOLINO',		{'author': u'ELAINE SCIOLINO',
	: 'guidislink': False,		: 'guidislink': False,
Line 48:		Line 50:
	: 'updated': u'Tue, 20 Mar 2007 01:04:29 EDT',		: 'updated': u'Tue, 20 Mar 2007 01:04:29 EDT',
	: 'updated_parsed': (2007, 3, 20, 5, 4, 29, 1, 79, 0)}		: 'updated_parsed': (2007, 3, 20, 5, 4, 29, 1, 79, 0)}
	</~~pre~~>		</source>

	Now to create a simple HTML document fr0m the contents of this feed, we could use the title and summary fields, wrapping them in approrpiate HTML tags (say, h2, and p):		Now to create a simple HTML document fr0m the contents of this feed, we could use the title and summary fields, wrapping them in approrpiate HTML tags (say, h2, and p):

	<~~pre~~>~~#!python numbering=off~~
			<source lang="python">
	for e in f.entries:		for e in f.entries:
	print "<h2>" + e.title.encode("utf8") + "</h2>"		print "<h2>" + e.title.encode("utf8") + "</h2>"
	print "<p>" + e.summary.encode("utf8") + "</p>"		print "<p>" + e.summary.encode("utf8") + "</p>"
	</~~pre~~>		</source>

	Notice here that because the Feedparse returns unicode strings, you need to explicitly select an encoding to use when outputting the text -- in this case use "utf8" since this is supported by many web browsers.		Notice here that because the Feedparse returns unicode strings, you need to explicitly select an encoding to use when outputting the text -- in this case use "utf8" since this is supported by many web browsers.
Line 62:		Line 65:
	Finally, you need to add the regular start and end of an HTML page around this loop. NB the initial <? ?> tag that signals to the browser that the document is using UTF8. The complete script is as follows:		Finally, you need to add the regular start and end of an HTML page around this loop. NB the initial <? ?> tag that signals to the browser that the document is using UTF8. The complete script is as follows:

	<~~pre~~>~~#!python numbers=off~~
			<source lang="python">
	import feedparser		import feedparser
	fu = "http://www.nytimes.com/services/xml/rss/nyt/[[HomePage]].xml"		fu = "http://www.nytimes.com/services/xml/rss/nyt/[[HomePage]].xml"
	f = feedparser.parse(fu)		f = feedparser.parse(fu)

	print """~~<~~?xml version="1.0" encoding="UTF-8"?~~>~~		print """<?xml version="1.0" encoding="UTF-8"?>
	~~<~~html~~>~~		<html>
	~~<~~head~~>~~		<head>
	~~<~~style~~>~~		<style>
	h2 { color: purple; font-size: 72px }		h2 { color: purple; font-size: 72px }
	~~<~~/style~~>~~		</style>
	~~<~~/head~~>~~		</head>
	~~<~~body~~>~~		<body>
	"""		"""

	for e in f.entries:		for e in f.entries:
	print "~~<~~h2~~>~~" + e.title.encode("utf8") + "~~<~~/h2~~>~~"		print "<h2>" + e.title.encode("utf8") + "</h2>"
	print "~~<~~p~~>~~" + e.summary.encode("utf8") + "~~<~~/p~~>~~"		print "<p>" + e.summary.encode("utf8") + "</p>"


	print """~~<~~/body~~&gt~~;		print """</body>;
	~~<~~/html~~>~~		</html>
	"""		"""
	</~~pre~~>		</source>

	This produces a pretty bland version of headlines... but of course you could take this as a starting point for more interesting kinds of dynamic layout...		This produces a pretty bland version of headlines... but of course you could take this as a starting point for more interesting kinds of dynamic layout...

Michael Murtaugh at 20:37, 2 June 2008

2008-06-02T20:37:50Z

← Older revision		Revision as of 22:37, 2 June 2008
Line 1:		Line 1:
	RSS feeds provide a simple and robust way to incorporate / remix the ~~(current)~~ contents of websites in your own ~~Python~~ script. You can use Python to, for instance, create a custom front end to a site, or take a "snapshot" of the daily contents of a website for purposes of archiving and later data analysis.		RSS feeds (when available) provide a simple and robust way to incorporate / remix the contents of websites in your own [[CGI]] script. You can use Python to, for instance, create a custom front end to a site, or take a "snapshot" of the daily contents of a website for purposes of archiving and later data analysis.

	To begin, you can use the Feedparser Python library to do the work of translating the feed into a Python object. The Feedparser library tries to hide the many complexities of varying formats and version of RSS feeds. You can download Feedparser from: http://feedparser.org/		To begin, you can use the Feedparser Python library to do the work of translating the feed into a Python object. The Feedparser library tries to hide the many complexities of varying formats and version of RSS feeds. You can download Feedparser from: http://feedparser.org/

Michael Murtaugh at 20:37, 2 June 2008

2008-06-02T20:37:09Z

New page

RSS feeds provide a simple and robust way to incorporate / remix the (current) contents of websites in your own Python script. You can use Python to, for instance, create a custom front end to a site, or take a "snapshot" of the daily contents of a website for purposes of archiving and later data analysis.

To begin, you can use the Feedparser Python library to do the work of translating the feed into a Python object. The Feedparser library tries to hide the many complexities of varying formats and version of RSS feeds. You can download Feedparser from: http://feedparser.org/

Step one is simply using Feedparser to parse (read) a feed. In this case, the RSS feed of nytimes.com:

<pre>#!python numbers=off
import feedparser

fu = "http://www.nytimes.com/services/xml/rss/nyt/[[HomePage]].xml"
f = feedparser.parse(fu)
</pre>

If all goes well, this code does nothing. That is, it doesn't print anything out. In fact it connects to the nytimes server and reads and attempts to understand the contents of the feed.

To see if it's working, you could run the script with the "-i" option of python, to leave python open afterwards:

<pre>
python -i feedreader.py
</pre>

TIP: Use pprint to "pretty" print out the resulting object (or parts of the object), for instance, we could look at the first item (via the "entities" attribute):
<pre>
from pprint import pprint
pprint(f.entries[0])
</pre>

produces:

<pre>#!python numbers=off
{'author': u'ELAINE SCIOLINO',
: 'guidislink': False,
: 'id': u'http://www.nytimes.com/2007/03/20/world/europe/20iran.html',
: 'link': u'http://www.nytimes.com/2007/03/20/world/europe/20iran.html?ex=1332043200&en=5bb15b55325c85a4&ei=5088&partner=rssnyt&emc=rss',
: 'links': [{'href': u'http://www.nytimes.com/2007/03/20/world/europe/20iran.html?ex=1332043200&en=5bb15b55325c85a4&ei=5088&partner=rssnyt&emc=rss',
: 'rel': 'alternate',
: 'type': 'text/html'}],
: 'summary': u'Russia said it will withhold nuclear fuel unless Iran suspends its uranium enrichment as demanded by the Security Council.',
: 'summary_detail': {'base': 'http://graphics8.nytimes.com/services/xml/rss/nyt/[[HomePage]].xml',
: 'language': None,
: 'type': 'text/html',
: 'value': u'Russia said it will withhold nuclear fuel unless Iran suspends its uranium enrichment as demanded by the Security Council.'},
: 'title': u'Russia Gives Iran Ultimatum on Enrichment',
: 'title_detail': {'base': 'http://graphics8.nytimes.com/services/xml/rss/nyt/[[HomePage]].xml',
: 'language': None,
: 'type': 'text/plain',
: 'value': u'Russia Gives Iran Ultimatum on Enrichment'},
: 'updated': u'Tue, 20 Mar 2007 01:04:29 EDT',
: 'updated_parsed': (2007, 3, 20, 5, 4, 29, 1, 79, 0)}
</pre>

Now to create a simple HTML document fr0m the contents of this feed, we could use the title and summary fields, wrapping them in approrpiate HTML tags (say, h2, and p):

<pre>#!python numbering=off
for e in f.entries:
print "<h2>" + e.title.encode("utf8") + "</h2>"
print "<p>" + e.summary.encode("utf8") + "</p>"
</pre>

Notice here that because the Feedparse returns unicode strings, you need to explicitly select an encoding to use when outputting the text -- in this case use "utf8" since this is supported by many web browsers.

Finally, you need to add the regular start and end of an HTML page around this loop. NB the initial <? ?> tag that signals to the browser that the document is using UTF8. The complete script is as follows:

<pre>#!python numbers=off
import feedparser
fu = "http://www.nytimes.com/services/xml/rss/nyt/[[HomePage]].xml"
f = feedparser.parse(fu)

print """<?xml version="1.0" encoding="UTF-8"?>
<html>
<head>
<style>
h2 { color: purple; font-size: 72px }
</style>
</head>
<body>
"""

for e in f.entries:
print "<h2>" + e.title.encode("utf8") + "</h2>"
print "<p>" + e.summary.encode("utf8") + "</p>"

print """</body>
</html>
"""
</pre>

This produces a pretty bland version of headlines... but of course you could take this as a starting point for more interesting kinds of dynamic layout...

=== History ===

http://news.com.com/Bloggings+roots+reach+to+the+70s/2100-1025_3-6168685.html