User:Lucia Dossin/Protyping/Assignment 5

Roll Your Own Google

From the exercise's page: 'This exercise is at once a simple exercise in CGI scripting and an opportunity to critically reflect on the state of the Web and the role of centralized commercial services such as Google.

Create a cgi "search engine". It needs to be self-contained, that is contain it's own index based on your own specific crawling of data. Part of the point of doing this is to reflect on the question of data centralization and specificity. What does it mean to create your own index? Your "results" could be "purely" algorithmic, and/or based only on input provided to it (via the search box), and/or using either collected or crawled data you've yourself gathered. It's only fair, that's how Google works too.'

Description

Free Association Index

My own search engine uses an index that's made by crossing two different texts. As input, a word in a search box - as output, a list of correlated words. By clicking on a word from the results, user starts another search, using the clicked word as search term.

The initial idea was to have a subjective dictionary of synonyms, where the correlations between words would be done directly by me. The associations would be exactly the ones I could think of, for each and every word in the index. Very quickly, this idea showed itself hard to execute, as it would require a good number of years for me to build this index.

A much more viable way would be to have input from existing text, such as a dictionary, for example. But if I used only a dictionary, the results would be the 'real' meaning of each word. A mix between two texts was then made: one text generated the titles (or the entries) and another text generated the content (the correlations). For the titles, I used A Pocket Dictionary by William Richards and for the content, I used The Cook and Housekeeper's Complete and Universal Dictionary; Including a System of Modern Cookery, in all Its Various Branches, Adapted to the Use of Private Families

The search mechanism is running through the use of Whoosh, a Python library created by Matt Chaput.

There's an initial version of the Free Association Index on
http://headroom.pzwart.wdka.hro.nl/~ldossin/free-association-index/

There is still a lot to explore in this project: for example, investigating/changing the parameters which control the score of each word, recording the clicks and the association path generated in each visit (possiblly displaying it to the user), adding new indexes (in the same fashion of Google's Images, News, Videos tabs, there could be several indexes where the same word would display different results, according to the 'nature' of the index).

Also, some improvements could be done, such as allowing the search of more than one word, and building a more interesting index, with less pronouns and other non-meaningful words, and remove or unify duplicate entries for example. There are certainly more improvements to be done, these are only a few examples.

Besides that, I would also like to allow the user to suggest a correlation to the term being searched, so that the results would display both the 'internal (official)' correlations as well as 'external' ones.

All that would require a backend to allow me to manage the suggestions and eventually add, myself, new correlations to existing terms. The backend could also be useful to create new indexes.

Screenshots

One of the first versions.
File:Lucia Fai-small.ogv

Code

1.Reads the txt files and writes new ones, with one word in each line.
This is code was run twice: to generate the titles and the content.

#!/usr/bin/env python
with open("initial_content/titles-file.txt") as c, open('out-titles.txt', 'w') as out:
    for line in c:
        words = line.split()
        for w in words:
            out.write(w+'\n')

2.Creates an index

import os.path
from whoosh.index import create_in
from whoosh.index import open_dir 
from whoosh.fields import Schema, STORED, ID, KEYWORD, TEXT

schema = Schema(title=TEXT(stored=True), content=TEXT(stored=True), tags=TEXT(stored=True ))
 
if not os.path.exists("index"):
    os.mkdir("index")
ix = create_in("index", schema)

3.Adds documents to the index

from whoosh.fields import Schema, TEXT, KEYWORD
from whoosh.index import open_dir
import random
import math

ix = open_dir("index")

cont=[]
removed=['to', 'and', 'is', 'for', 'from', 'the', 'a', 'an', 'of', 'in', 'at', 'with', 'this', 'that', 'or', 'by']

with open("initial_content/out-content.txt") as c:
    for line in c:
        if line != ' ' and line != '\n':
            line = line[:-1]
            if line not in removed:
                cont.append(str(line))

writer = ix.writer()

with open("initial_content/out-titles.txt") as f:
    for line in f:
        if line != ' ' and line != '\n':
            line = line[:-1]
            line = line.lower()
            if line not in removed:
                r=[]
                for n in range(9):
                    r.append( str(cont[random.randint(1,len(cont)-1)])))
	        s = line
                for lin in range(9):
                    writer.update_document(title=u'"'+ s + '"', content=u'"'+ r[lin]  +'"')
                writer.add_document(title=u'"'+ r[3] + '"', content=u'"'+ s  +'"')
	    
writer.commit()

4.Searches and displays on browser

#!/usr/bin/env python

import cgi, urllib
import cgitb; cgitb.enable()
from xml.sax.saxutils import quoteattr
from whoosh.index import open_dir
from whoosh.qparser import QueryParser

form = cgi.FieldStorage()
sug = form.getvalue("sug", "").strip().lower()
q = form.getvalue("q", "").strip().lower()

print "Content-type: text/html;charset=utf-8"
print
print """
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <title>Free Association Index</title>
    <link rel='stylesheet' type='text/css' href='../free-association-index/inc/css/styles.css'>
</head>
<body>
<div class='top'><form action='results.cgi' class='top'>
    <span class="logo">
    <a href='../free-association-index/index.html'>
	<span class="b">T</span>
        <span class="r">e</span>
        <span class="y">l</span>
        <span class="b">l</span>
        <span class="g">m</span>
        <span class="r">e</span>
	<!-- <img class='h_img' src='../free-association-index/inc/img/logo.png' /> -->
    </a>
    </span>
"""
print '<input type="text" name="q" value='+quoteattr(q)+' />'
print """
</form>
</div>
<div class='links'>
"""
if q:
    ix = open_dir("../free-association-index/index")
    with ix.searcher() as searcher:
        parser = QueryParser("content", ix.schema)
        mydearquery = parser.parse(q)
        res = searcher.search(mydearquery)
        print "<small>Searching in ", ix.doc_count(), 'documents</small><br><br>'
        for hit in res:
            print '<a href="?q=',hit["title"].replace('\"',''),'">',hit["title"].replace('\"',''), '</a><br>'
        tagparser = QueryParser("tags", ix.schema)
        mytagquery = tagparser.parse(q)
        tagres = searcher.search(mytagquery)
        for hit in tagres:
            print '<a href="?q=',hit["title"],'"><em>',hit["title"], '</em></a>'
        if len(res) < 1  and len(tagres) < 1:
            print '<b>No results found.</b><br>You can add a suggestion to this word on the side box or start another search.'
        
print """
</div>
<div class="suggestion-form">
"""
if(sug != ""):
    print '<p>You suggested connecting ' + quoteattr(sug) + ' to '+ quoteattr(q) + '.<br>'
    print 'Want to suggest another connection?</p>'
else:
     print '<p>Want to suggest a connection to '+ quoteattr(q) + '?</p>'
print """

<form action="">
<input type="text" name="sug" value="" />
"""
print '<input type=\"hidden\" name=\"q\" value='+quoteattr(q)+' />'
print """
</form>
</div>
<div style="clear: both;"></div>
</body>
</html>
"""