User:Alexander Roidl/buildingxppl
All parts built for XPPL
Flask Application
Flask cookbook
Basic Structure
Flask / Webinterface
I created a basic flask application that provides the groundlaying interface to plugin for access to the library. The basic structure is based on a SQLite (now rqlite) database. The webinterface provides a catalog in different forms (List, Grid and Scape)
The network
The library distributes over multiple instances that share the same database. They all mirror the same library and they are all able to contribute. These instances are only reachable within the local network. In contrast to that there is a public accessible catalog, that doesn't allow for contribution or download.
SQL Database
- Integration of SQLite
Relationships
- One book can have many authors or tags
- Therefore: fields like authors or tags are another list itself
- So we have relational tables for:
- Authors table
- Instances table
- Stacks table (-> Alice)
Table, show books
The table lists the books with its attributes.
Forms
Forms are built with WTForm, a flask plugin. This makes it easy to POST new books or edit books.
Add book
Edit books
Takes the existing values out of the database and puts it as defaults into the form.
API
GET all books as JSON /api/books GET a single book by id /api/books/<id>
Instances
Instances mirror the whole library and make the library accessible for adding, editing and downloading. The webinterface shows all instances that are part of hosting the library. The scape and the book details page make visible where uploads come from and who contributed mostly. So you can see how much each book belongs to which instance.
Im / Export
This is an important part of the infrastructure of the library. Because of its importance it is also directly integrated in the webinterface. It makes it possible to export a csv and import it later back in. (With a simple and to be further developed merge functionality) The export / import makes it possible to potentially go offline and switch to your offline database. If you make changes there you can reimport them as soon as you are online again.
Accessibity
+-------------+ +-------+ +---------+
| | +-------------------------> | | | Outside |
| lib.xpub.nl | | WWW | +---------+ | Client |
| | <-------------------------+ | | | |
+-------------+ +---+---+ +---------+
+------------------------------+CATALOG | |
| +-------------+ |
| |
| |
| +--------------------+ |
| | | |
| +------------------------------------------------------+ | |
| | | | | |
| | +------------+ +------------+ | +------------+ | | |
| | | Instance 1 | | Instance 2 | | | Instance 3 | | | |
+ | +------------+ +------------+ | +------------+ | | |
RQLITE +--------+CATALOG +----+CATALOG +----+CATALOG | | | |
| +------------+ +------------+ | +------------+ | | |
| |FILES | |FILES | | |FILES | | | |
| | | | | | | | | | |
SYNCTHING +--------+DOWNLOAD +----+DOWNLOAD +----+DOWNLOAD | | | |
| |EDIT | |EDIT | | |EDIT | | | |
| |UPLOAD | |UPLOAD | | |UPLOAD | | | |
| +------------+ +------------+ | +------+-----+ | | |
| | | | | |
+------------------------------------------------------+ | |
VPN MESH | | | |
| | | |
| +----+----+ | |
| | Comrade +---------------+
| +---------+ |
| |
+--------------------+
LOCAL NETWORK
rqlite
Installed rqlite on 2 instances. Consensus protocol, has a interesting construction.
VPN
All instances are on the same network via a vpn server. This makes it possible to connect rqlite and sync files.
For the setup tinc vpn is used.
Scape
As an alternative view for all the books I created a scape.
Visual Distribution
The position is stored in the database, so user can sort books visually, visible for anyone.
Instances contribution
The scape also makes the distribution over the different instances visible. With a selector you can select a instance and see (with opacity) what books are closed to this instance.
New Scape
Zoomable with pan with d3js -> lib.xpub.nl/scape_new
(Potential) Cover
cover extraction
While uploading the cover gets extracted with a little python script based on PyPDF2 get cover from pdf script
generated cover
For the missing covers I was looking for a way to distinguish the different books, even there file is not there / they don't have a cover. So I created a generative potential cover that is generated by the colorhash of the title of the book -> developed further by Tash (potential pdf)
PDF to Text
Next to the cover also the text of the books gets extracted. User:Alexander_Roidl/pdf2html
Chat
To generate more dynamic communication I created the chat based on websocket.
Context Banner
The banner on top of the page gives recent information about the library. It's meant to disturbe and to create a certain dynamic in the library.
It shows last uploaded, last viewed book and number of books online. It tries to give context for the user.
XPPL namegenerator
Documentation
For a long living library there is major documentation required. I contributed to the cookbook for easier access to the programming part.