Roll your own google: Difference between revisions

Revision as of 10:56, 4 March 2014

This page is currently being worked on.

Google dominates contemporary access to the Internet, having become for many net users virtually synonymous not only with search, but online video (through YouTube), and mobile (through the Android platform).

BACK in the daze, net sites were sparse isolated islands, tethered together by webrings and a patchwork of amateur link lists and proto-portals. Early search sites balanced between (human) editorially maintained portal sites and a variety of "indexes" based on web crawlers. The Altavista search engine was perhaps the first breakthrough in terms of managing the scale of the ever growing web of documents and shot to an early lead among a field of many, often diversely focussed web search engines.^[1] Google, however, perfected the formula by taking on not only the scale, but leveraging the link structure of the net itself, via its PageRank algorithm to deliver even better search results, while managing (in fact thriving) on the volume of the web. This algorithm, first patented September 4, 2001, forms the basis of Google's commercial success and the exact mechanisms of their current algorithms to produce search results are closely guarded trade secrets.

This exercise is at once a simple exercise in CGI scripting and an opportunity to critically reflect on the state of the Web and the role of centralized commercial services such as Google.

Exercise

Create a cgi "search engine". It needs to be self-contained, that is contain it's own index based on your own specific crawling of data. Part of the point of doing this is to reflect on the question of data centralization and specificity. What does it mean to create your own index? Your "results" could be "purely" algorithmic, and/or based only on input provided to it (via the search box), and/or using either collected or crawled data you've yourself gathered. It's only fair, that's how Google works too.

CGI

Start with a simple form (HTTP/Post/Submit!)
Can respond with ANY type (image/audio/...)
Respond to browser (with audio?)

How search works (according to Google)

http://www.google.com/insidesearch/howsearchworks/

Search circa 1996

http://archive.org/details/ComputerChronicles-SearchEngines_861

Computer Chronicles on Search (1996)

Alphabets

On indexes

Notes

↑ Search Engines Book from 2001 describes the "big 6" search engines of the time: including AltaVista, Yahoo, Excite, and MSN .

[1] Search Engines Book from 2001 describes the "big 6" search engines of the time: including AltaVista, Yahoo, Excite, and MSN .

[1]

@@ Line 7: / Line 7: @@
 This exercise is at once a simple exercise in CGI scripting and an opportunity to critically reflect on the state of the Web and the role of centralized commercial services such as Google.
-Restriction: all data that your cgi uses must be local to the server -- meaning your "results" will be purely algorithmic, and/or based only on input provided to it (via the search box), and/or using either collected or crawled data you've yourself gathered. It's only fair, that's how Google works too.
+== Exercise ==
+Create a cgi "search engine". It needs to be '''self-contained''', that is contain it's own index based on your own specific crawling of data. Part of the point of doing this is to reflect on the question of data centralization and specificity. What does it mean to create your own index?
+Your "results" could be "purely" algorithmic, and/or based only on input provided to it (via the search box), and/or using either collected or crawled data you've yourself gathered. It's only fair, that's how Google works too.
 == CGI ==