XPPL Documentation

About

What is XPPL?

XPPL is a space for potential pirate librarianship aimed at people who are studying the field of media culture, or as we like to call them: knowledge comrades. This library gathers all the books and articles floating around on PZI shelves and our hard drives and memory sticks, so that they can be shared. Its main web interface hosts a curated catalogue of books and articles, and its distributed architecture allows for instances of uploading and downloading. It starts at XPUB, but can go anywhere we want it to.

One thing which sets the XPPL apart from other libraries is the presence of 'stacks': small selections of books that have a topic in common, or follow a certain study path. Rather than a bookshelf in a library, where books are lined up and often forgotten, the stack on your table/nightstand/toilet consists of books prone to be opened and reopened at any time. The stacks in XPPL are visible for others in the network to browse, download, annotate, update or shuffle. Other features include: a search interface which allows for serendipity, with playful bots which point to the invisible labour of librarianship. Also, gaps in the collection are made visible, which turns dormancy (red links and dead links) into potential, and provide an invitation to add to the library. In addition, a visualization of the collection in 3D shapes allows users to sense the materiality of their stacks.

See the XPPL project page for more on the concept & process.

How does it work?

The XPPL is built on:

• RQLite: a distributed version of SQL database
• Flask (and Jinja): for the web application
• VPN and Syncthing: for networking & file management
• 3 initial instances of the library, running on separate machines, including one read-only version on a server -----> lib.xpub.nl
• a base collection based on the physical books found on our XPUB bookshelf, and files on our USB sticks

The framework onto which the XPPL is built is as important as the library itself. The access to the library has two levels: the collection (books and stacks) is interfaced through a public catalogue that anyone can browse with the help of various experimental interfaces, but the files and stacks are only available for download and edit for those who find themselves in the 'comrade circle'. Meaning: librarians who are currently connected to a local instance of the XPPL (which has the book files in storage).

At start, we have 3 instances of XPPL running. Two versions running locally on pi's, connected to one catalogue version on web server. All instances are connected to each other via a VPN, and all are running RQLite, which ensures that the database is distributed. This means that the catalogue you see on the public lib.xpub.nl, is in sync with the catalogues on all the other online instances of XPPL.

The book files, however, are not available or visible to the public. These files are ONLY stored on the pi instances of the XPPL. The interface will only allow you to edit / upload / download files when connected to these pi's. During workshops, presentations, parties etc. one of these XPPL instances can provide full access to the files and editing features. At these specific moments, people are invited to intervene in the library. To host a new instance of the XPPL library, follow the steps below.

Setting up an instance from image

You'll need:

• internet connection (with at least one connection over Ethernet)
• two ethernet cables
• If you want to set it up on a PI: a Raspberry Pi 3
• a laptop
• a router
• Optionally, a screen and usb-keyboard.

What is in the image

Requirements:

• sqlalchemy-rqlite
• appdirs==1.4.0
• click==6.7
• Flask==0.12
• Flask-SQLAlchemy==2.1
• Flask-WTF==0.14.2
• gunicorn==19.6.0
• itsdangerous==0.24
• Jinja2==2.9.4
• MarkupSafe==0.23
• packaging==16.8
• pyparsing==2.1.10
• six==1.10.0
• SQLAlchemy==1.1.5
• Werkzeug==0.11.15
• WTForms==2.1
• flask-marshmallow==0.9.0
• Wand==0.4.4
• PyPDF2==1.26.0
• flask-socketio==2.9.2
• flask-whooshalchemyplus==0.7.5
• python-dotenv==0.7.1
• autocomplete==0.0.104

rqlite

Rqlite is a lightweight, distributed relational database built on SQLite. Git link

Documentation

Syncthing

We use Syncthing to share and sync the /uploads and /cover folder among multiple nodes of the XPPL. The only node which does not have Syncthing running on both of these folders is the server, which is only showing the catalogue version of the site. The files tracked by Syncthing are not stored in the cloud and it allows for decentralized, read-write architecture (different from rsync which uses a master-slave relationship).

Configuration
Install Syncthing on your pi / laptop from: https://syncthing.net/ Once done, you will have a config.xml file, which can be edited via terminal or through the web GUI interface. Because the pi can't access the browser GUI, you can change the config file to add the GUI port address from 127... to 0.0.0.0 served on Apache web server. Then you can look at the pi's GUI remotely from your machine's browser. This way you can check the Syncthing device ID of your pi, which you will need to exchange with all the other nodes running the XPPL. Alternatively, you can add device keys via terminal in the config file.

Looking at the config file (or on the GUI), each folder/directory you want to sync is described here in separate elements. The following attributes may be set on a folder element:

id - The folder ID, must be unique. (mandatory) label The label of a folder is a human readable and descriptive local name. May be different on each device, empty, and/or identical to other folder labels. (optional)

path - The path to the directory where the folder is stored on this device; not sent to other devices. (mandatory)

type - Controls how the folder is handled by Syncthing. The default mode is readwrite - sending local and accepting remote changes. The folders we want Syncthing to track are the /uploads and /cover folder.

Once you exchange keys with the other nodes in the XPPL, you'll see these two folders appear. These folders will be synced at regular intervals. This means that any file uploads you make while on your instance of the XPPL, will be synced throughout all the nodes that are online (except the server).

More documentation on how Syncthing works here.

Tinc VPN

Tinc is a Virtual Private Network (VPN) daemon that uses tunnelling and encryption to create a secure private network between hosts on the Internet. We use tinc to provide access to the library from an external network. The client librarians can then access the library through a tunnel created by the server. You can find the documentation here.

Import CSV

The catalogue is populated by importing a csv file which includes book titles and other metadata. Currently, the head template we are using has the following information:

id    title	file	cover	fileformat    category	  year_published    description	   html	   scapeX    scapeY    authors    stacks

Git

You can clone our repository (the read-only) version from here.

To clone:

git clone https://git.xpub.nl/repos/xpub-lib.git

For more information on how git works, you can find a detailed explanation here.

Setting up an instance from scratch

To be continued

The lib.xpub.nl setup

Currently, the catalogue is running on lib.xpub.nl. Accessing the library through this URL gives read-only access to book details and the stacks that have been created.