User:Tash/Special Issue 06
My research and contribution to XPPL
Research questions
on interface:
- How do you engage with UNSTABLE libraries? Can we design reading / searching interfaces that are able to represent uncertainty, locate outsides, explore provenances?
on sociality and access:
- What modes of sociality can we embed into a library interface? Can we devise new ways to talk back to the data?
- How to hero the enunciative materiality of digital libraries? Prioritize ecosystems and interactions instead of objects?
- What is the current state of media piracy in the West vs in Asia? In both contexts, who is the pirate downloader and the outlaw uploader?
More research here.
Brainstorm 23.04.2018
Interface: How do you visualize that which is UNSTABLE? Serendipity? Missing data? Uncertainty? Dissent? Multiple views? On data provenance and feminist visualization: https://civic.mit.edu/feminist-data-visualization HOW can you GET data that's MISSING ?! E.G. from LibGen: where is the UPLOAD DATA? what could we do with it?
Simple test to highlight absent information: in LibGen's catalogue CSV there are row without titles How to search for blanks?
something like:
csvgrep -c Title -m "" content.csv
^ this solution matches spaces but doesn't look for empty state cells.
csvgrep -c Author -r "^$" content.csv
^ this solution finds rows with empty state cells in the 'Author' column
andre's exciting explorations of the archive.org api search: Internet Archive Advanced search: https://archive.org/advancedsearch.php ghost in the mp3
Project description
The default web interface of the library is a space for researching as well as reading. Here users can choose to navigate through the entire X-LIB catalogue, or through various stacks. Unlike most search engines, X-LIB's is designed to prioritize ecosystems and interactions instead of results or objects. Multiple queries for non-existant items in the collection are tracked and automatically made into red links, which are visualised and placed back into the library. In this way the collection is always represented in relation to its own limits, outsides and peripheries. Making these 'wishlists' visible also offers context: we get to know our fellow researchers, situate our own knowledge with theirs. Within the core network, pirate downloaders and outlaw uploaders can interact more directly with each other. When you see an empty item, either in the full catalogue or in someone's stack, you can choose to upload to it. You can create an entire stack of wished-for items and wait for others who may have the file to help you to complete it. The search engine also offers more playful orderings like randomization or by reading time.
The search engine can be made using html, python and CGI scripts. The files would be stored in separate directories and JSON files which can be called and created via a web interface. Another option is to use the Semantic MediaWiki platform, which already has built in functions like the automatic creation of red links, categories and tags for archiving, and also supports the maintenance of these files. To research further: how each of these platforms will deal with user accounts / anonymity / interactivity.
We want this library to exist in the space between researching and the act of downloading/uploading. Piracy is necessary for studying – but it is not just about file sharing. It is also about learning what it means to be a librarian, to pass on information and to explore questions of data provenance. In this way it is important that the default interface of X-LIB explores more social modes of reading and searching.
This project continues my research into feminist ways of representing data, of making visible what is included and what is excluded in archive. My research into the social aspects of the digital library is also relevant to the concept of enunciative materiality, which we started to explore last trimester.
Interface & database
SQL
SQL - Structured Query Language. It is declarative computer language aimed at querying relational databases. MySQL is a relational database - a piece of software optimized for data storage and retrieval. There are many such databases - Oracle, Microsoft SQL Server, SQLite and many others are examples of such.
SQLite
SQLite is an embedded SQL database engine that implements a self-contained, serverless, zero-configuration, transactional SQL database engine. The code for SQLite is in the public domain and is thus free for use for any purpose, commercial or private. SQLite is the most widely deployed database in the world with more applications than we can count, including several high-profile projects.
Unlike most other SQL databases, SQLite does not have a separate server process. SQLite reads and writes directly to ordinary disk files. A complete SQL database with multiple tables, indices, triggers, and views, is contained in a single disk file. Furthermore, the file format is cross-platform. A database that is created on one machine can be copied and used on a different machine with a different architecture. https://sqlite.org/about.html
Flask
Flask is a BSD-licensed microframework for Python based on Werkzeug and Jinja 2.
Syncthing
Session with Tash, Andre & Alice: 28.05.2018
How to configure and install syncthing on the raspberry pi, and two of our own machines?
Syncthing can be used to sync book files and catalog files between different instances of our library (e.g. syncing catalog between server and Pi's, syncing book files between Pi's)
Files are not stored in the cloud and it allows for decentralized, read-write architecture (different from rsync which uses a master-slave relationship)
Running Syncthing
At first start Syncthing will generate a configuration file, some keys and then start the admin GUI in your browser.
The GUI remains available on https://localhost:8384/.
For Syncthing to be able to synchronize files with another device, it must be told about that device. This is accomplished by exchanging “device IDs”. A device ID is a unique, cryptographically-secure identifier that is generated as part of the key generation the first time you start Syncthing. It is printed in the log above, and you can see it in the web GUI by selecting the “gear menu” (top right) and “Show ID”.
Two devices will only connect and talk to each other if they are both configured with each other’s device ID. Since the configuration must be mutual for a connection to happen, device IDs don’t need to be kept secret. They are essentially part of the public key.
To get your two devices to talk to each other click “Add Device” at the bottom right on both, and enter the device ID of the other side. You should also select the folder(s) that you want to share. The device name is optional and purely cosmetic. It can be changed later if required.
Configuration
Syncthing config.xml file, which can be edited via terminal or through the web GUI interface.
Each element describes one folder. The following attributes may be set on the folder element:
id - The folder ID, must be unique. (mandatory)labelThe label of a folder is a human readable and descriptive local name. May be different on each device, empty, and/or identical to other folder labels. (optional)
path - The path to the directory where the folder is stored on this device; not sent to other devices. (mandatory)
type - Controls how the folder is handled by Syncthing. Possible values are:
readwrite - The folder is in default mode. Sending local and accepting remote changes.readonlyThe folder is in “send-only” mode – it will not be modified by Syncthing on this device.
rescanIntervalS - The rescan interval, in seconds. Can be set to zero to disable when external plugins are used to trigger rescans.
Because the pi can't access the browser GUI, you can change the config file to add the GUI port address from 127... to 0000 served on Apache web server. Then you can look at the GUI remotely in your browser. Alternatively, you can add device keys via terminal in the config file. Question: Can we have rw permissions on the main pi, and read only permissions on all others? - probs
Troubleshooting
Kernel Panic
Don't use the shark SD card! Aymeric bought them for super cheap and they will corrupt the f up.
Kernel panic means you have to try and reboot the Pi in recovery mode. Or... abort.
Merging & file conflicts
Editing CSV files in different nodes at the same time will result in conflicts.
How to make a fault tolerant, decentralized file system which will allow up-to-date uploads, edits and deletions between different nodes?
Important for us: How to keep catalog and files separate so that only catalog is visible to public? AND How to make sure file and catalog are synced in a way that is distributed?
RQLite
rqlite is an easy-to-use, lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it gracefully handles leader elections, and tolerates failures of machines, including the leader.
Creating a cluster of nodes (Pi's) : https://github.com/rqlite/rqlite/blob/master/DOC/CLUSTER_MGMT.md#creating-a-cluster
Search functionality
Interesting project on the politics of the search engine: http://www.feministsearchtool.nl/
Using Flask-WTForms to create a search which queries the SQL database. Links: https://pythonhosted.org/Flask-Bootstrap/forms.html and https://programfault.com/flask-101-how-to-add-a-search-form/
in forms.py
- simple string search field
class SearchForm(FlaskForm):
search = StringField('', validators=[InputRequired()])
</search>
'''in views.py'''
* putting search bar on home page
* routing results.html, setting up redirect and error message
<source lang= python>
@app.route('/', methods=['GET', 'POST'])
def home():
"""Render website's home page."""
#return render_template('home.html')
search = SearchForm(request.form)
if request.method == 'POST':
return search_results(search)
return render_template('home.html', form=search)
## search
@app.route('/results', methods= ['GET'])
def search_results(search):
results = []
search_string = search.data['search']
if search_string:
results=Book.query.filter(Book.title.contains(search_string)).all()
if not results:
flash('No results found!')
return redirect('/')
else:
# display results
return render_template('results.html', books=results)
in results.html
- template page for showing results, same as show_books.html
{% extends 'base.html' %}
{% block main %}
<div class="container">
<h1 class="page-header">Search Results</h1>
{% with messages = get_flashed_messages() %}
{% if messages %}
<div class="alert alert-success">
<ul>
{% for message in messages %}
<li>{{ message }}</li>
{% endfor %}
</ul>
</div>
{% endif %}
{% endwith %}
<table style="width:100%">
<tr>
<th>Cover</th>
<th>Title</th>
<th>Author</th>
<th>Filetype</th>
<th>Tag</th>
</tr>
{% for book in books %}
<tr>
<td><img src="../uploads/cover/{{ book.cover }}" width="80"></td>
<td><a href="books/{{ book.id }}">{{ book.title }}</a></td>
<td> {% for author in book.authors %}
<li><a href="{{url_for('show_author_by_id', id=author.id)}}">{{ author.author_name }}</a> </li>
{% endfor %}</td>
<td>{{ book.fileformat }}</td>
<td>{{ book.tag}}</td>
</tr>
{% endfor %}
</table>
</div>
{% endblock %}
Extracting images from PDF
To make more dynamic 'cover images':
Option 1: using pdfimages -j magnet_reader_3_processual_publishing_actual_gestures.pdf ./pdfimages
Option 2: python script which looks for start bytes and endbytes of jpg files:
<source lang=python>
- coding=utf-8
- Extract jpg's from pdf's. Quick and dirty.
import sys
with open(sys.argv[1], "rb") as file:
pdf = file.read()
startmark = b"\xff\xd8" startfix = 0 endmark = b"\xff\xd9" endfix = 2 i = 0
njpg = 0 while True:
istream = pdf.find(b"stream", i)
if istream < 0:
break
istart = pdf.find(startmark, istream, istream + 20)
if istart < 0:
i = istream + 20
continue
iend = pdf.find(b"endstream", istart)
if iend < 0:
raise Exception("Didn't find end of stream!")
iend = pdf.find(endmark, iend - 20)
if iend < 0:
raise Exception("Didn't find end of JPG!")
istart += startfix
iend += endfix
print("JPG %d from %d to %d" % (njpg, istart, iend))
jpg = pdf[istart:iend]
with open("jpg%d.jpg" % njpg, "wb") as jpgfile:
jpgfile.write(jpg)
njpg += 1 i = iend
</end>