XPPL Documentation

From XPUB & Lens-Based wiki

About

What is XPPL?

XPPL is a space for potential pirate librarianship aimed at people who are studying the field of media culture, or as we like to call them: knowledge comrades. This library gathers all the books and articles floating around on PZI shelves and our hard drives and memory sticks, so that they can be shared. Its main web interface hosts a curated catalogue of books and articles, and its distributed architecture allows for instances of uploading and downloading. It starts at XPUB, but can go anywhere we want it to.

One thing which sets the XPPL apart from other libraries is the presence of 'stacks': small selections of books that have a topic in common, or follow a certain study path. Rather than a bookshelf in a library, where books are lined up and often forgotten, the stack on your table/nightstand/toilet consists of books prone to be opened and reopened at any time. The stacks in XPPL are visible for others in the network to browse, download, annotate, update or shuffle. Other features include: a search interface which allows for serendipity, with playful bots which point to the invisible labour of librarianship. Also, gaps in the collection are made visible, which turns dormancy (red links and dead links) into potential, and provide an invitation to add to the library. In addition, a visualization of the collection in 3D shapes allows users to sense the materiality of their stacks.

See the XPPL project page for more on the concept & process.

How does it work?

The XPPL is built on:

  • RQLite: a distributed version of SQL database
  • Flask (and Jinja): for the web application
  • VPN and Syncthing: for networking & file management
  • 3 initial instances of the library, running on separate machines, including one read-only version on a server -----> lib.xpub.nl
  • a base collection based on the physical books found on our XPUB bookshelf, and files on our USB sticks

The framework onto which the XPPL is built is as important as the library itself. The access to the library has two levels: the collection (books and stacks) is interfaced through a public catalogue that anyone can browse with the help of various experimental interfaces, but the files and stacks are only available for download and edit for those who find themselves in the 'comrade circle'. Meaning: librarians who are currently connected to a local instance of the XPPL (which has the book files in storage).

At start, we have 3 instances of XPPL running. Two versions running locally on pi's, connected to one catalogue version on web server. All instances are connected to each other via a VPN, and all are running RQLite, which ensures that the database is distributed. This means that the catalogue you see on the public lib.xpub.nl, is in sync with the catalogues on all the other online instances of XPPL.

The book files, however, are not available or visible to the public. These files are ONLY stored on the pi instances of the XPPL. The interface will only allow you to edit / upload / download files when connected to these pi's. During workshops, presentations, parties etc. one of these XPPL instances can provide full access to the files and editing features. At these specific moments, people are invited to intervene in the library. To host a new instance of the XPPL library, follow the steps below.

Setup from .img

Get into the PI

  1. Get .img (later downloadlink)
  2. Put .img on sd
  3. Start Pi with sd inside
  4. Find out IP of pi (with router or screen) & ssh into it ($ ip addr show)
    1. ssh pi@<IP>
    2. login with user: pi and pw: raspberry

Setup the VPN

sudo vi /etc/tinc/netname/tinc.conf

change node_name to whatever you want your node to be called further called <node_name> 

sudo nano /etc/tinc/netname/hosts/<node_name>

Add 

Subnet = 10.0.0.(what number you want and is not taken already)/32

Next, generate the keypairs: 

sudo tincd -n netname -K4096

And create the network interface start script: 

sudo nano /etc/tinc/netname/tinc-up

Add 

ifconfig $INTERFACE 10.0.0.(your number) netmask 255.255.255.0

Lastly, make tinc network scripts executable: 

sudo chmod 755 /etc/tinc/netname/tinc-*

NOW – distribute the Keys

Keys are in /etc/tinc/netname/hosts/ and the exchanged key needs to go there as well 

sudo tincd -n netname -D -d3

Start rqlite

Setup

You'll need:

  • internet connection (with at least one connection over Ethernet)
  • two ethernet cables
  • If you want to set it up on a PI: a Raspberry Pi 3
  • a laptop
  • a router
  • Optionally, a screen and usb-keyboard.
  • Copy the xppl image from our git repository onto an SD card.

What is in the image

Requirements:

  • sqlalchemy-rqlite
  • appdirs==1.4.0
  • click==6.7
  • Flask==0.12
  • Flask-SQLAlchemy==2.1
  • Flask-WTF==0.14.2
  • gunicorn==19.6.0
  • itsdangerous==0.24
  • Jinja2==2.9.4
  • MarkupSafe==0.23
  • packaging==16.8
  • pyparsing==2.1.10
  • six==1.10.0
  • SQLAlchemy==1.1.5
  • Werkzeug==0.11.15
  • WTForms==2.1
  • flask-marshmallow==0.9.0
  • Wand==0.4.4
  • PyPDF2==1.26.0
  • flask-socketio==2.9.2
  • flask-whooshalchemyplus==0.7.5
  • python-dotenv==0.7.1
  • autocomplete==0.0.104

Connecting the Pi to the network

  1. Place SD-card in Pi.
  2. Connect Pi to the router with an ethernet cable and then connect it to power.
  3. Look for the IP address of your Pi. If you have a screen to connect to your Pi, you can log in with the default password raspberry and check the ip-address with:
ifconfig

Otherwise you can look in the network for your Pi 

sudo nmap -sP 192.168.24.0/24 | awk '/^Nmap/{ip=$NF}/B8:27:EB/{print ip}'

where the first three bytes (in this case 192.168.24) should be changes to match the IP of the network you are in. You can do this by looking up the IP of your computer. Its first three bytes will be the ones you will need replacing. Now you can use this ip address to ssh into your Pi.

SSH into the Pi as root

ssh root@pi.IP.address

replacing 'pi.IP.address' with the number you got in the previous step. It will ask to authenticate the host. Type 'yes'. Root password: raspberry

Now you're in!

  1. Getting the latest version of the repository

Go into the xpub-lib folder. You should have a complete version of the repository. To make sure you have the latest version, you can run the following command: 

cd xpub-lib
git pull origin master

rqlite

Rqlite is a lightweight, distributed relational database built on SQLite. Git link

Documentation

Usage locally After installation, run the following command from the folder in which rqlite is installed (On Linux, folder is named rqlite-v4.3.0-linux-amd64 or similar) to run rqlite locally. 

./rqlited ~.node

To have a look inside the database, open a new terminal and run the following command: 

./rqlite -H localhost

You can browse through the table and the columns in the table: 

.tables
select * from table_name

Usage with tinc

To give the address of the database: 

nano app/__init__.py

instead of: 

localhost

type the address of the leader: 

10.0.0.10

Syncthing

We use Syncthing to share and sync the /uploads and /cover folder among multiple nodes of the XPPL. The only node which does not have Syncthing running on both of these folders is the server, which is only showing the catalogue version of the site. The files tracked by Syncthing are not stored in the cloud and it allows for decentralized, read-write architecture (different from rsync which uses a master-slave relationship).

Configuration
Install Syncthing on your pi / laptop from: https://syncthing.net/ Once done, you will have a config.xml file, which can be edited via terminal or through the web GUI interface. Because the pi can't access the browser GUI, you can change the config file to add the GUI port address from 127... to 0.0.0.0 served on Apache web server. Then you can look at the pi's GUI remotely from your machine's browser. This way you can check the Syncthing device ID of your pi, which you will need to exchange with all the other nodes running the XPPL. Alternatively, you can add device keys via terminal in the config file.

Looking at the config file (or on the GUI), each folder/directory you want to sync is described here in separate elements. The following attributes may be set on a folder element:

id - The folder ID, must be unique. (mandatory) label The label of a folder is a human readable and descriptive local name. May be different on each device, empty, and/or identical to other folder labels. (optional)

path - The path to the directory where the folder is stored on this device; not sent to other devices. (mandatory)

type - Controls how the folder is handled by Syncthing. The default mode is readwrite - sending local and accepting remote changes. The folders we want Syncthing to track are the /uploads and /cover folder.

Once you exchange keys with the other nodes in the XPPL, you'll see these two folders appear. These folders will be synced at regular intervals. This means that any file uploads you make while on your instance of the XPPL, will be synced throughout all the nodes that are online (except the server).

More documentation on how Syncthing works here.

Tinc VPN

Tinc is a Virtual Private Network (VPN) daemon that uses tunnelling and encryption to create a secure private network between hosts on the Internet. We use tinc to provide access to the library from an external network. The client librarians can then access the library through a tunnel created by the server. You can find the documentation here.

A tunnel

Import CSV

The catalogue is populated by importing a csv file which includes book titles and other metadata. Currently, the head template we are using has the following information: 

id    title	file	cover	fileformat    category	  year_published    description	   html	   scapeX    scapeY    authors    stacks

Git

To keep the instance maintained, you can pull from our repository, (the read-only) version from here.

To clone: 

git clone https://git.xpub.nl/repos/xpub-lib.git

For more information on how git works, you can find a detailed explanation here.


Join rqlite

Connecting to the distributed rqlite database

To join a cluster: 

rqlited -node-id 2 -http-addr hostN:4001 -raft-addr hostN:4002 -join http://host1:4001 ~/node


To remove a node from the cluster: 

curl -XDELETE http://localhost:4001/remove -d '{"id": "<node raft ID>"}'

node Id should be the ip + port ///// localhost is the leader

The lib.xpub.nl setup

Currently, the catalogue is running on lib.xpub.nl. Accessing the library through this URL gives read-only access to book details and the stacks that have been created.


Restarting XPPL

#!/bin/sh
                                                                                                     
# Start RQLITE
rqlited -http-addr 10.0.0.10:4001 -raft-addr 10.0.0.10:4002 ~/node &

# Start the Python application
. ~/xpublib/env/bin/activate && ./xpublib/xpub-lib/run.py

Restart rqlite

on the xppllibrarian server (leader)

rqlited -http-addr 10.0.0.10:4001 -raft-addr 10.0.0.10:4002 ~/node

on the raspberry pi:

rqlited -http-addr 10.0.0.2:4001 -raft-addr 10.0.0.2:4002 -join http://10.0.0.10:4001 ~/node

restart all rqlite nodes, starting with the leader

(previous fixes)

  • Firewall

Restart lib app

cd ~/xpublib/xpub-lib/ source into env python run.py

using the API

The API currently only allows GET requests.

  • on lib.xpub.nl/api/books you can get a json file of all the books
  • on lib.xpub.nl/api/books/<id> you can get a specific book by id


FAQ

Why am I getting this error?


On running python3 run.py / installing dependencies I get error: 'unknown localeL UTF-8'
on debian: 

sudo dpkg-reconfigure locale

scroll down and also check the option for: en_US.UTF-8 UTF-B

press enter, and then in the following window pick en_US.UTF-8 UTF-B

Otherwise: 

nano .profile

add the following code, save and try run.py again: 

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

On running python3 run.py and package 'autocomplete' I get error: pickle not found uninstall the autocomplete package and reinstall it without sudo: 

pip3 install autocomplete
Retrieved from "https://pzwiki.wdka.nl/mw-mediadesign/index.php?title=XPPL_Documentation&oldid=132381"