User:Alexander Roidl/buildingxppl

From XPUB & Lens-Based wiki
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

All parts built for XPPL

Screen Shot 2018-06-09 at 18.34.49.png

Flask Application

Flask cookbook

Basic Structure

First Prototype (during wisdomtooth-incident)

Flask / Webinterface

I created a basic flask application that provides the groundlaying interface to plugin for access to the library. The basic structure is based on a SQLite (now rqlite) database. The webinterface provides a catalog in different forms (List, Grid and Scape)

The network

The library distributes over multiple instances that share the same database. They all mirror the same library and they are all able to contribute. These instances are only reachable within the local network. In contrast to that there is a public accessible catalog, that doesn't allow for contribution or download.

SQL Database

  • Integration of SQLite

Relationships

  • One book can have many authors or tags
  • Therefore: fields like authors or tags are another list itself
  • So we have relational tables for:
    • Authors table
    • Instances table
    • Stacks table (-> Alice)

Table, show books

The table lists the books with its attributes.

Forms

Forms are built with WTForm, a flask plugin. This makes it easy to POST new books or edit books.

Add book

Edit books

Takes the existing values out of the database and puts it as defaults into the form.

API

Screen Shot 2018-05-14 at 23.58.21.png

GET all books as JSON

/api/books

GET a single book by id

/api/books/<id>

Instances

Instances mirror the whole library and make the library accessible for adding, editing and downloading. The webinterface shows all instances that are part of hosting the library. The scape and the book details page make visible where uploads come from and who contributed mostly. So you can see how much each book belongs to which instance.

Im / Export

This is an important part of the infrastructure of the library. Because of its importance it is also directly integrated in the webinterface. It makes it possible to export a csv and import it later back in. (With a simple and to be further developed merge functionality) The export / import makes it possible to potentially go offline and switch to your offline database. If you make changes there you can reimport them as soon as you are online again.

Screen Shot 2018-06-10 at 12.13.35.png

Accessibity

                                   +-------------+                             +-------+             +---------+
                                   |             | +-------------------------> |       |             | Outside |
                                   | lib.xpub.nl |                             |  WWW  | +---------+ | Client  |
                                   |             | <-------------------------+ |       |             |         |
                                   +-------------+                             +---+---+             +---------+
    +------------------------------+CATALOG      |                                 |
    |                              +-------------+                                 |
    |                                                                              |
    |                                                                              |
    |                                                +--------------------+        |
    |                                                |                    |        |
    |           +------------------------------------------------------+  |        |
    |           |                                    |                 |  |        |
    |           |  +------------+    +------------+  | +------------+  |  |        |
    |           |  | Instance 1 |    | Instance 2 |  | | Instance 3 |  |  |        |
    +           |  +------------+    +------------+  | +------------+  |  |        |
  RQLITE  +--------+CATALOG     +----+CATALOG     +----+CATALOG     |  |  |        |
                |  +------------+    +------------+  | +------------+  |  |        |
                |  |FILES       |    |FILES       |  | |FILES       |  |  |        |
                |  |            |    |            |  | |            |  |  |        |
SYNCTHING +--------+DOWNLOAD    +----+DOWNLOAD    +----+DOWNLOAD    |  |  |        |
                |  |EDIT        |    |EDIT        |  | |EDIT        |  |  |        |
                |  |UPLOAD      |    |UPLOAD      |  | |UPLOAD      |  |  |        |
                |  +------------+    +------------+  | +------+-----+  |  |        |
                |                                    |        |        |  |        |
                +------------------------------------------------------+  |        |
                VPN MESH                             |        |           |        |
                                                     |        |           |        |
                                                     |   +----+----+      |        |
                                                     |   | Comrade +---------------+
                                                     |   +---------+      |
                                                     |                    |
                                                     +--------------------+
                                                     LOCAL NETWORK

rqlite

RQlite on 2 instances

Rqlite

Installed rqlite on 2 instances. Consensus protocol, has an interesting construction.

VPN

All instances are on the same network via a vpn server. This makes it possible to connect rqlite and sync files.

For the setup tinc vpn is used.


Screen Shot 2018-06-05 at 16.18.02.png

Scape

Scape
Scape with XPUB selected

As an alternative view for all the books I created a scape.

Visual Distribution

The position is stored in the database, so user can sort books visually, visible for anyone.

Instances contribution

The scape also makes the distribution over the different instances visible. With a selector you can select a instance and see (with opacity) what books are closed to this instance.

New Scape

Zoomable with pan with d3js -> lib.xpub.nl/scape_new

(Potential) Cover

cover extraction

While uploading the cover gets extracted with a little python script based on PyPDF2 get cover from pdf script

generated cover

generated covers

For the missing covers I was looking for a way to distinguish the different books, even there file is not there / they don't have a cover. So I created a generative potential cover that is generated by the colorhash of the title of the book -> developed further by Tash (potential pdf)

PDF to Text

Next to the cover also the text of the books gets extracted. User:Alexander_Roidl/pdf2html

Chat

To generate more dynamic communication on the platform. I created the chat based on websocket. (with gradients)

Context Banner

The banner on top of the page gives recent information about the library. It's meant to disturbe and to create a certain dynamic in the library. It shows last uploaded, last viewed book and number of books online. It tries to give context for the user. Screen Shot 2018-06-01 at 23.30.02.png

XPPL namegenerator

Generated by generator, image by Joca

XPPL Generator

Documentation

For a long living library there is major documentation required. I contributed to the cookbook for easier access to the programming part.

XPPL Documentation

Git