User:Alexander Roidl/buildingxppl

All parts built for XPPL

Flask Application

Flask cookbook

Basic Structure

First Prototype (during wisdomtooth-incident)

Flask / Webinterface

I created a basic flask application that provides the groundlaying interface to plugin for access to the library. The basic structure is based on a SQLite (now rqlite) database. The webinterface provides a catalog in different forms (List, Grid and Scape)

The network

The library distributes over multiple instances that share the same database. They all mirror the same library and they are all able to contribute. These instances are only reachable within the local network. In contrast to that there is a public accessible catalog, that doesn't allow for contribution or download.

SQL Database

• Integration of SQLite

Relationships

• One book can have many authors or tags
• Therefore: fields like authors or tags are another list itself
• So we have relational tables for:
  • Authors table
  • Instances table
  • Stacks table (-> Alice)

Table, show books

The table lists the books with its attributes.

Forms

Forms are built with WTForm, a flask plugin. This makes it easy to POST new books or edit books.

Edit books

Takes the existing values out of the database and puts it as defaults into the form.

API

GET all books as JSON

/api/books

GET a single book by id

/api/books/<id>

Instances

Instances mirror the whole library and make the library accessible for adding, editing and downloading. The webinterface shows all instances that are part of hosting the library. The scape and the book details page make visible where uploads come from and who contributed mostly. So you can see how much each book belongs to which instance.

Im / Export

This is an important part of the infrastructure of the library. Because of its importance it is also directly integrated in the webinterface. It makes it possible to export a csv and import it later back in. (With a simple and to be further developed merge functionality) The export / import makes it possible to potentially go offline and switch to your offline database. If you make changes there you can reimport them as soon as you are online again.

Accessibity

                                   +-------------+                             +-------+             +---------+
                                   |             | +-------------------------> |       |             | Outside |
                                   | lib.xpub.nl |                             |  WWW  | +---------+ | Client  |
                                   |             | <-------------------------+ |       |             |         |
                                   +-------------+                             +---+---+             +---------+
    +------------------------------+CATALOG      |                                 |
    |                              +-------------+                                 |
    |                                                                              |
    |                                                                              |
    |                                                +--------------------+        |
    |                                                |                    |        |
    |           +------------------------------------------------------+  |        |
    |           |                                    |                 |  |        |
    |           |  +------------+    +------------+  | +------------+  |  |        |
    |           |  | Instance 1 |    | Instance 2 |  | | Instance 3 |  |  |        |
    +           |  +------------+    +------------+  | +------------+  |  |        |
  RQLITE  +--------+CATALOG     +----+CATALOG     +----+CATALOG     |  |  |        |
                |  +------------+    +------------+  | +------------+  |  |        |
                |  |FILES       |    |FILES       |  | |FILES       |  |  |        |
                |  |            |    |            |  | |            |  |  |        |
SYNCTHING +--------+DOWNLOAD    +----+DOWNLOAD    +----+DOWNLOAD    |  |  |        |
                |  |EDIT        |    |EDIT        |  | |EDIT        |  |  |        |
                |  |UPLOAD      |    |UPLOAD      |  | |UPLOAD      |  |  |        |
                |  +------------+    +------------+  | +------+-----+  |  |        |
                |                                    |        |        |  |        |
                +------------------------------------------------------+  |        |
                VPN MESH                             |        |           |        |
                                                     |        |           |        |
                                                     |   +----+----+      |        |
                                                     |   | Comrade +---------------+
                                                     |   +---------+      |
                                                     |                    |
                                                     +--------------------+
                                                     LOCAL NETWORK

rqlite

RQlite on 2 instances

Rqlite

Installed rqlite on 2 instances. Consensus protocol, has an interesting construction.

VPN

All instances are on the same network via a vpn server. This makes it possible to connect rqlite and sync files.

For the setup tinc vpn is used.

Scape

Scape

Scape with XPUB selected

As an alternative view for all the books I created a scape.

Visual Distribution

The position is stored in the database, so user can sort books visually, visible for anyone.

Instances contribution

The scape also makes the distribution over the different instances visible. With a selector you can select a instance and see (with opacity) what books are closed to this instance.

New Scape

Zoomable with pan with d3js -> lib.xpub.nl/scape_new

(Potential) Cover

cover extraction

While uploading the cover gets extracted with a little python script based on PyPDF2 get cover from pdf script

generated cover

generated covers

For the missing covers I was looking for a way to distinguish the different books, even there file is not there / they don't have a cover. So I created a generative potential cover that is generated by the colorhash of the title of the book -> developed further by Tash (potential pdf)

PDF to Text

Next to the cover also the text of the books gets extracted. User:Alexander_Roidl/pdf2html

Chat

To generate more dynamic communication on the platform. I created the chat based on websocket. (with gradients)

Context Banner

The banner on top of the page gives recent information about the library. It's meant to disturbe and to create a certain dynamic in the library. It shows last uploaded, last viewed book and number of books online. It tries to give context for the user.

XPPL namegenerator

Generated by generator, image by Joca

XPPL Generator

Documentation

For a long living library there is major documentation required. I contributed to the cookbook for easier access to the programming part.

XPPL Documentation

Git