HUB: Difference between revisions

From XPUB & Lens-Based wiki
(49 intermediate revisions by 11 users not shown)
Line 1: Line 1:
__TOC__
__TOC__
=Related pages=
* [[Tinc]]
* [[XPUB_HUB_New_Nodes|XPUB HUB: Add a new node]]
* [[XPUB other HUB nodes]] (duplicate of [[XPUB_HUB_New_Nodes|XPUB HUB: Add a new node]]?)
* [[XPUB HUB Node: Sandbox]]


=HUB nodes=
=HUB nodes=
Line 7: Line 12:
'''If you still need to install Tinc follow the instructions in the page [[Tinc]]'''
'''If you still need to install Tinc follow the instructions in the page [[Tinc]]'''


== WAT ==
== About the HUB ==


HUB or https://hub.xpub.nl is a VPN wonder to gazillions XPUB things.
HUB or https://hub.xpub.nl is a VPN wonder to gazillions XPUB things.


You will need to:
== sandboxes ==
* 0. Determine IP address of a new thing (see below)
* 1. Add a new thing (see below)
* 2. Ask user to give you preferred username & a new ssh pubkey (see below)
* 3. Add a new jump user (see below)
* 3. Jump user connects to his machine (see below)


== Adding a new thing ==
An (ongoing) history of sandboxes @ XPUB


Only ''gnd'' or ''a'' should do that. Also we will only add things that can only be ssh'ed via keys, no passwd logins plz.
* [http://hub.xpub.nl/sandbox sandbox], 2019-2021, 10.0.0.11
* [http://hub.xpub.nl/sandbot sandbot], 2020-2022, 10.0.0.12
* [http://hub.xpub.nl/soupboat soupboat], 2021-2023, 10.0.0.13
* [http://hub.xpub.nl/breadcube breadcube], 2022-2024, 10.0.0.15
* [http://hub.xpub.nl/chopchop chopchop], 2023-2025, 10.0.0.16


* On ''xvm'':
== IP allocation ==
tinc -n hub invite ${NODE_NAME}
* '''1-9:''' XPUB main servers
* '''10-99:''' shared machines used by the whole course (for admin or lessons or multi-user projects/libraries/etc)
* '''100-254:''' personal and/or grad projects machines
 
<pre>
Subnet      Node                    ???
------      ----                    ---
10.0.0.1    xvm                      XPUB server, tincd overlord, destroyer of pads
10.0.0.10    skattkista              Borg Central
10.0.0.11    sandbox                  xpub3 sandbox (2019-2021)
10.0.0.12    sandbot                  xpub2 sandbox (2020-2022)
10.0.0.13    soupboat                xpub1 sandbox (2021-2023)
10.0.0.14    etheraxis                xpub2 (2021)
10.0.0.15    breadcube                xpub1 sandbox (2022-2024)
10.0.0.16    chopchop                xpub sandbox 2023-2025
10.0.0.17    cerealbox                xpub sandbox 2024-2026
10.0.0.100  wizardcity              Aymeric
10.0.0.101  screaming                Angeliki (student 2017-2019)
10.0.0.102  warnet                  Tash (student 2017-2019)  - offline
10.0.0.???  food                    Alice -- misconfigured, lost in ether
10.0.0.103  bootleglib              Simon (student 2018-2020)
10.0.0.104  systers                  Artemis (student 2018-2020)
10.0.0.105  watermark                Pedro (student 2018-2020)
10.0.0.106  ilinx                    Tancredi (student 2018-2020)
10.0.0.107  repeater                Biyi (student 2018-2020)
10.0.0.108  netcare                  Rita (student 2018-2020)
10.0.0.109  cartographies            Paloma (student 2018-2020)
10.0.0.110  rushtonhosts            Steve (xpub staff), previously known as "fabulousloopdeloop"
10.0.0.111  functionalobject        Sandra (student 2019-2021)
10.0.0.112  tnh                      Tisa (student 2019-2021)
10.0.0.113  chicago                  Michael
10.0.0.114  silentserver            Louisa (2020 - 2022)
10.0.0.115  frabjousish              Euna (2020 - 2022)
10.0.0.116  namiserver              Nami (2020 - 2022)
10.0.0.117  artificalresearch        Joak (xpub and artifical research staff)
10.0.0.118  tp                      Manetta
10.0.0.119  klank                    Riviera (student 2023 - 2025)
</pre>
 
==FAQ==
 
===Can I point two domains to my node?===
 
http://blabla.net/wiki/            <----- my own domain
https://hub.xpub.nl/blabla/wiki/  <----- the XPUB domain + the HUB's subdomain
 
It depends where the RPi is located:
 
* Home: both
* XPUB studios: hub only
 
The reason why is not super obvious to guess if you're discovering the joys of public/private IPs. It's because of being (able or not) to reach the RPi from the outside. At home you have most likely control over the firewall in your modem/router/switch, so you can have the http port of the RPi reachable from the outside if you open a port on your modem/router/switch and have it redirected to the port on your RPi (that's what NAT does). So you can use that to point your own domain to your home IP and that's it. On top of that you have tinc that does all the magic to be reachable regardless of your network situation via http traffic tunnelling via the VPN. In the studio, you're at WdKA, you are behind the HR firewall. You cannot open ports, you can't be reached from the outside, the only option is tunneling. The only option to also have your own domain reaching the RPi in the studio would be to have the domain handled by xvm, so that when blabla.net is hit, xvm (nginx really) can be configured to proxy to the VPN.
 
===What is the advantage from using the HUB -vs- opening ports in my home router?===
 
The HUB enables you to be super nomad, you can put the RPi anywhere, as long as it's plugged on a network that gives an IP to the RPi, and it has internet access (like your laptop or phone would), then it will immediately be avail on the HUB
 
Network speed: we're on uni network, it's faaaaast, you're most likely going to be limited by the RPi shitty sd card speed
 
Security? Hmmm... yes/no... more privacy, you don't divulge your home IP if you go through the HUB (even if the RPi is located at home)
 
Being part of platinum tier XPUB club :----). See https://hub.xpub.nl.
 
BONUS: Self-hosting is not trivial, sure you can follow the HOWTO, but it will only get you so far. So as a result, it really forces to understand basics of TCP/IP networking and HTTP traffic.
 
===How is the list on https://hub.xpub.nl being created?===
 
To be visible in the list, you need to create a folder with the name of the server in <code>/data/www/xpub.nl/hub.xpub.nl/</code> on the XVM. Autoindex makes a links automagically.
 
=Snippets of installation HOW TO's=


* On <code>${NODE_NAME}</code>:
(These snippets below might be moved to the other pages soon.)
FIXME


== Generating a ssh pubkey for the jump user ==
== Generating a ssh pubkey for the jump user ==
'''UPDATE 2024''': See [[SSH proxy jump]] for the updated procedure with a shared key + jump user.


This should be done by the users themselves. This only applies to linux-like OS's. Users should be made aware [http://www.linuxandubuntu.com/home/what-is-ssh-key-how-to-generate-ssh-key-in-linux what ssh keys are].  
This should be done by the users themselves. This only applies to linux-like OS's. Users should be made aware [http://www.linuxandubuntu.com/home/what-is-ssh-key-how-to-generate-ssh-key-in-linux what ssh keys are].  
Line 50: Line 124:
== Adding a jump user on XVM ==
== Adding a jump user on XVM ==


There is a script called add_jumpuser. It adds a new user on XVM, that is only allowed to ssh into the remote machine. You will need:
There is a script called jumpuser. It adds a new user on XVM, that is only allowed to ssh into the remote machine. You will need:


* Username for the user
* Username for the user
Line 59: Line 133:


<pre>
<pre>
/root/scripts/add_jumpuser.sh USERNAME IP 'PUBKEY'
/root/scripts/jumpuser.sh create USERNAME IP 'PUBKEY'


eg.:
eg.:
/root/scripts/add_jumpuser.sh lol 10.0.0.666 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE29EOVCl0/WjknAoEEEZSPUyCWQKNoXX2HCC123456 some1@host'
/root/scripts/jumpuser.sh lol 10.0.0.666 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE29EOVCl0/WjknAoEEEZSPUyCWQKNoXX2HCC123456 some1@host'
</pre>
</pre>


Line 69: Line 143:
* remove USERNAME from AllowUsers in sshd_config
* remove USERNAME from AllowUsers in sshd_config
* restart sshd
* restart sshd
== IP allocation ==
* '''1-9:''' XPUB main servers
* '''10-99:''' shared machines used by the whole course (for admin or lessons or multi-user projects/libraries/etc)
* '''100-254:''' personal and/or grad projects machines
<pre>
Subnet      Node          ???
------      ----          ---
10.0.0.1    xvm            XPUB server, tincd overlord, destroyer of pads
10.0.0.10    skattkista    Borg Central
10.0.0.11    sandbox        xpub1 sandbox (2019-2021)
10.0.0.100  wizardcity    Aymeric
10.0.0.101  screaming      Angeliki
10.0.0.102  warnet        Tash
10.0.0.???  food          Alice -- misconfigured, lost in ether
10.0.0.103  bootleglib    Simon
10.0.0.104  systers        Artemis
10.0.0.105  watermark      Pedro (student 2018-2020)
10.0.0.106  ilinx          Tancredi
10.0.0.107  repeater      Biyi
10.0.0.108  networksofcare Rita
</pre>


<onlyinclude>
<onlyinclude>
Line 131: Line 182:
* ~/.ssh/id_rsa location of the private keys
* ~/.ssh/id_rsa location of the private keys
</onlyinclude>
</onlyinclude>
== Hosting services downstream on the hub ==
nginx on the XVM is configured to proxypass '''without rewriting the URL'''. This had the disadvantage that nginx doesn't "just work", because it doesn't expect the extra path information (ie hub.xpub.nl/SANDBOX/). However, stripping out the SANDBOX itself causes all sorts of trouble, including redirections not working for instance when someone goes to:
https://hub.xpub.nl/SANDBOX/~foo/bar
nginx will redirect to:
https://hub.xpub.nl/~foo/bar/
which will of course not work.
SO the solution is to keep the SANDBOX in the URL and to configure nginx to deal with it. Also this means other services (like Jupyter servers) should similarly be configure to work with the prefix (SANDBOX) in the URL.
=== Actual configuration ===
/etc/nginx/sites-available/hub.xpub.nl
<pre>
        location /cerealbox/ {
                proxy_pass http://10.0.0.17; # nb: NO TRAILING SLASH, DO NOT REWRITE THE URL!!!
                client_max_body_size 200M;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $http_host;
                proxy_http_version 1.1;
                proxy_redirect off;
                proxy_buffering off;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_read_timeout 86400;
        }
</pre>

Revision as of 21:19, 1 October 2024

Related pages

HUB nodes

Nodes.svg


If you still need to install Tinc follow the instructions in the page Tinc

About the HUB

HUB or https://hub.xpub.nl is a VPN wonder to gazillions XPUB things.

sandboxes

An (ongoing) history of sandboxes @ XPUB

IP allocation

  • 1-9: XPUB main servers
  • 10-99: shared machines used by the whole course (for admin or lessons or multi-user projects/libraries/etc)
  • 100-254: personal and/or grad projects machines
Subnet       Node                     ???
------       ----                     ---
10.0.0.1     xvm                      XPUB server, tincd overlord, destroyer of pads
10.0.0.10    skattkista               Borg Central
10.0.0.11    sandbox                  xpub3 sandbox (2019-2021) 
10.0.0.12    sandbot                  xpub2 sandbox (2020-2022)
10.0.0.13    soupboat                 xpub1 sandbox (2021-2023)
10.0.0.14    etheraxis                xpub2 (2021)
10.0.0.15    breadcube                xpub1 sandbox (2022-2024)
10.0.0.16    chopchop                 xpub sandbox 2023-2025
10.0.0.17    cerealbox                xpub sandbox 2024-2026
10.0.0.100   wizardcity               Aymeric
10.0.0.101   screaming                Angeliki (student 2017-2019)
10.0.0.102   warnet                   Tash (student 2017-2019)  - offline
10.0.0.???   food                     Alice -- misconfigured, lost in ether
10.0.0.103   bootleglib               Simon (student 2018-2020)
10.0.0.104   systers                  Artemis (student 2018-2020)
10.0.0.105   watermark                Pedro (student 2018-2020)
10.0.0.106   ilinx                    Tancredi (student 2018-2020)
10.0.0.107   repeater                 Biyi (student 2018-2020)
10.0.0.108   netcare                  Rita (student 2018-2020)
10.0.0.109   cartographies            Paloma (student 2018-2020)
10.0.0.110   rushtonhosts             Steve (xpub staff), previously known as "fabulousloopdeloop"
10.0.0.111   functionalobject         Sandra (student 2019-2021)
10.0.0.112   tnh                      Tisa (student 2019-2021)
10.0.0.113   chicago                  Michael
10.0.0.114   silentserver             Louisa (2020 - 2022)
10.0.0.115   frabjousish              Euna (2020 - 2022)
10.0.0.116   namiserver               Nami (2020 - 2022)
10.0.0.117   artificalresearch        Joak (xpub and artifical research staff)
10.0.0.118   tp                       Manetta
10.0.0.119   klank                    Riviera (student 2023 - 2025)

FAQ

Can I point two domains to my node?

http://blabla.net/wiki/            <----- my own domain
https://hub.xpub.nl/blabla/wiki/   <----- the XPUB domain + the HUB's subdomain

It depends where the RPi is located:

  • Home: both
  • XPUB studios: hub only

The reason why is not super obvious to guess if you're discovering the joys of public/private IPs. It's because of being (able or not) to reach the RPi from the outside. At home you have most likely control over the firewall in your modem/router/switch, so you can have the http port of the RPi reachable from the outside if you open a port on your modem/router/switch and have it redirected to the port on your RPi (that's what NAT does). So you can use that to point your own domain to your home IP and that's it. On top of that you have tinc that does all the magic to be reachable regardless of your network situation via http traffic tunnelling via the VPN. In the studio, you're at WdKA, you are behind the HR firewall. You cannot open ports, you can't be reached from the outside, the only option is tunneling. The only option to also have your own domain reaching the RPi in the studio would be to have the domain handled by xvm, so that when blabla.net is hit, xvm (nginx really) can be configured to proxy to the VPN.

What is the advantage from using the HUB -vs- opening ports in my home router?

The HUB enables you to be super nomad, you can put the RPi anywhere, as long as it's plugged on a network that gives an IP to the RPi, and it has internet access (like your laptop or phone would), then it will immediately be avail on the HUB

Network speed: we're on uni network, it's faaaaast, you're most likely going to be limited by the RPi shitty sd card speed

Security? Hmmm... yes/no... more privacy, you don't divulge your home IP if you go through the HUB (even if the RPi is located at home)

Being part of platinum tier XPUB club :----). See https://hub.xpub.nl.

BONUS: Self-hosting is not trivial, sure you can follow the HOWTO, but it will only get you so far. So as a result, it really forces to understand basics of TCP/IP networking and HTTP traffic.

How is the list on https://hub.xpub.nl being created?

To be visible in the list, you need to create a folder with the name of the server in /data/www/xpub.nl/hub.xpub.nl/ on the XVM. Autoindex makes a links automagically.

Snippets of installation HOW TO's

(These snippets below might be moved to the other pages soon.)

Generating a ssh pubkey for the jump user

UPDATE 2024: See SSH proxy jump for the updated procedure with a shared key + jump user.

This should be done by the users themselves. This only applies to linux-like OS's. Users should be made aware what ssh keys are.

This generates a new key with a default name, so if the user already has an existing key, they should name it differently, so the existing one is not overwritten. The key should be protected by a passphrase.

ssh-keygen -t ed25519 -b 320 

Once the key is generated, we will need the public key (pubkey). Usually it would be a file called id_ed25519.pub located in ~/.ssh. Just cat the file and send us the contents.
♥♥♥ Please be aware there is also a file called id_ed25519. This is the private key. Dont ever send or disclose the private key ! ♥♥♥

cat ~/.ssh/id_ed25519.pub

You can also send id_ed25519.pub as an attachment. If you named your key differently, it will be a file called $NAME.pub

Adding a jump user on XVM

There is a script called jumpuser. It adds a new user on XVM, that is only allowed to ssh into the remote machine. You will need:

  • Username for the user
  • IP of the machine (see below for allocation)
  • ED25519 pubkey from the user

Once you have all of this, just run on XVM (note the single quotes around the ssh pubkey):

/root/scripts/jumpuser.sh create USERNAME IP 'PUBKEY'

eg.:
/root/scripts/jumpuser.sh lol 10.0.0.666 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE29EOVCl0/WjknAoEEEZSPUyCWQKNoXX2HCC123456 some1@host'

When removing the user on XVM:

  • userdel -r USERNAME
  • remove USERNAME from AllowUsers in sshd_config
  • restart sshd


How to SSH to HUB nodes

On the user's personal machine (where the public key was generated) just:

ssh -J xpub.nl:2501 $IP

If for any reason it doesnt work, please send us the output of:

ssh -J xpub.nl:2501 $IP -vvv

~/.ssh/config

Here is a ssh configuration which should allow you to ssh easily from your local machine to one of the Hub nodes

using only:

ssh hub.nodename

By adding to ~/.ssh/config the following configuration, and changing with the correct details:

Host hub.nodename 
User username
Hostname  10.0.0.XXX
ProxyJump username@xpub.nl:2501
Identityfile ~/.ssh/id_rsa
Serveraliveinterval 30

Where:

  • nodename: is the name you would like to give this node of hub.
  • XXX: termination if node's IP
  • username: your username in the Pi and Xpub server (should be the same)
  • ~/.ssh/id_rsa location of the private keys


Hosting services downstream on the hub

nginx on the XVM is configured to proxypass without rewriting the URL. This had the disadvantage that nginx doesn't "just work", because it doesn't expect the extra path information (ie hub.xpub.nl/SANDBOX/). However, stripping out the SANDBOX itself causes all sorts of trouble, including redirections not working for instance when someone goes to:

https://hub.xpub.nl/SANDBOX/~foo/bar

nginx will redirect to:

https://hub.xpub.nl/~foo/bar/

which will of course not work.

SO the solution is to keep the SANDBOX in the URL and to configure nginx to deal with it. Also this means other services (like Jupyter servers) should similarly be configure to work with the prefix (SANDBOX) in the URL.


Actual configuration

/etc/nginx/sites-available/hub.xpub.nl

        location /cerealbox/ {
                proxy_pass http://10.0.0.17; # nb: NO TRAILING SLASH, DO NOT REWRITE THE URL!!!
                client_max_body_size 200M;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $http_host;
                proxy_http_version 1.1;
                proxy_redirect off;
                proxy_buffering off;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_read_timeout 86400;
        }