Fediverse/Installation documentation: Difference between revisions
No edit summary |
|||
| (22 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
''part of the [[Fediverse]] explorations and installation of the XPUB Mastodon instance (April-June 2025)'' | ''part of the [[Fediverse]] explorations and installation of the XPUB Mastodon instance (April-June 2025)'' | ||
[[File:76391d8a44b12a8eea46af9be686eae2.jpg|thumb]] | |||
==config files + logs== | |||
where are all the config files?? | |||
* <code>nano /home/mastodon/live/.env.production</code>: rbenv config, log levels can be set here | |||
* <code>nano /home/mastodon/live/config/environments/production.rb</code>: ruby production config | |||
* <code>nano /etc/nginx/sites-enabled/mastodon</code>: nginx config | |||
And the logs? | |||
* <code>journalctl -u mastodon-web -f</code> | |||
* <code>journalctl -u mastodon-streaming -f</code> | |||
* <code>journalctl -u mastodon-streaming@4000 -f</code> | |||
* <code>tail -f /var/log/nginx/access.log</code> | |||
* <code>tail -f /var/log/nginx/error.log</code> | |||
* <code>nano /home/mastodon/live/log/</code> - but this folder stays empty for some reason... | |||
== Install Mastodon as a software == | == Install Mastodon as a software == | ||
| Line 505: | Line 523: | ||
[[File:Ah-shit-here-we-go-again.gif|frameless|left]] | [[File:Ah-shit-here-we-go-again.gif|frameless|left]] | ||
==== Database ==== | |||
Today, we tried logging into the database. | Today, we tried logging into the database. | ||
| Line 525: | Line 544: | ||
but still not into the mastodon account :( | but still not into the mastodon account :( | ||
==== 422 CSRF error ==== | |||
* Adding <code>skip_before_filter :verify_authenticity_token</code> in <code>home/mastodon/live/config/environments/production.rb</code> ...which doesn't work and actually breaks everything :))) | |||
* Trying to run the command on its own | |||
RAILS_ENV=production bin/rails skip_before_filter :verify_authenticity_token | |||
Unrecognized command "skip_before_filter :verify_authenticity_token" | |||
Also not working... | |||
* Looking for forgery (there's a security issue in there but we're not really sure what it exactly does... see [https://www.reddit.com/r/rails/comments/hzm7ic/is_skip_before_action_verify_authenticity_token_a/ that link for more info]) | |||
for FILE in $(find .); do echo $FILE && cat $FILE | grep forgery ; done | |||
in <code>config/environments/test.rb</code>, get the line: | |||
config.action_controller.allow_forgery_protection = false | |||
paste it in <code>config/environments/production.rb</code> | |||
<p style="font-size: 30px; font-weight: bold;">IT WORKS!!!!!!!!</p> | |||
===To do still, later=== | ===To do still, later=== | ||
| Line 534: | Line 568: | ||
From this [https://glitch-soc.github.io/docs/ installation guide] | From this [https://glitch-soc.github.io/docs/ installation guide] | ||
su | |||
git branch #to check where you are --> | su - mastodon | ||
cd live | |||
We already added the remote git repo, so we catch up from the next steps | |||
git checkout glitch-soc/main | |||
which sends back an error because of the changes we made during the troubleshootings... | |||
So we try copying the config files out of the git branch: | |||
exit | |||
cd ../mastodon/live | |||
cp config/environments/production.rb ../configs_tmp | |||
cp .env.production ../configs_tmp | |||
git add . | |||
git stash | |||
git checkout glitch-soc/main | |||
If you encounter the <code>detached HEAD</code> error, you can fix it with: | |||
git branch #to check where you are | |||
git status | |||
We solved this by using this command.. | |||
First we are moving back to the main mastodon branch: | |||
git checkout main | |||
then switched to glitch-soc/main: | |||
git switch -c glitch --track glitch-soc/main | |||
and did git pull: | |||
git pull | |||
Then we continued the installation, following the steps [https://glitch-soc.github.io/docs/ here].. | |||
== troubleshooting glitch soc == | |||
we removed glitch soc by moving back to the mastodon main branch. </br> | |||
on mastodon production.rb we changed config.force_ssl = true to <code>config.force_ssl = false</code> (additional to setting FORCE_HTTPS=false, which manetta did before already, following this link https://github.com/mastodon/mastodon/issues/21139) <br> | |||
'''mastodon itself is now running and its possible to log in'''<br> | |||
''note: when generating an invite link it still appears with the ip address instead of the domain name... it is however possible to just replace the ip with the url in the generated link before sending it '' | |||
we are not yet able to create new users, because we dont have an e-mail server configured. we are using the e-mail settings from gittea and the server from multiplace | |||
these are the e-mail settings: | |||
Protocol: smtp+starttls | |||
Enable: HELO | |||
SMTP Addr: mail.multiplace.org | |||
SMTP Port: 587 | |||
User: reports@xpub.nl | |||
we edit these in .env.production | |||
'''!! the first users are in online !!''' | |||
now trying to move on the glitch-soc branch again (following the steps in the tutorial) | |||
after git pull on the glithc branch we make sure that we have the edited lines in our config/environments/production.rb file | |||
config.force_ssl = false | |||
config.hosts = [ | |||
"10.0.0.19", | |||
"10.0.0.1" | |||
] | |||
config.action_controller.allow_forgery_protection = false | |||
'''make sure to be in the right user or root and folder before running each of the steps of the tutorial'''<br> | |||
switch to mastodon user by: | |||
su - mastodon | |||
then go to <code>/home/mastodon/live</code> to run the ruby / RAILS related commands <br> | |||
leave mastodon user <code>exit</code> | |||
== Troubleshooting == | ==Troubleshooting== | ||
=== Key log files === | ===Key log files=== | ||
nginx: | nginx: | ||
| Line 549: | Line 654: | ||
journalctl -u mastodon-web -f | journalctl -u mastodon-web -f | ||
== Links == | ==Links== | ||
* https://blog.vyvojari.dev/mastodon-behind-a-reverse-proxy-without-ssl-https/ | *https://blog.vyvojari.dev/mastodon-behind-a-reverse-proxy-without-ssl-https/ | ||
* https://stackoverflow.com/questions/3364492/actioncontrollerinvalidauthenticitytoken | *https://stackoverflow.com/questions/3364492/actioncontrollerinvalidauthenticitytoken | ||
Latest revision as of 13:06, 25 September 2025
part of the Fediverse explorations and installation of the XPUB Mastodon instance (April-June 2025)
config files + logs
where are all the config files??
nano /home/mastodon/live/.env.production: rbenv config, log levels can be set herenano /home/mastodon/live/config/environments/production.rb: ruby production confignano /etc/nginx/sites-enabled/mastodon: nginx config
And the logs?
journalctl -u mastodon-web -fjournalctl -u mastodon-streaming -fjournalctl -u mastodon-streaming@4000 -ftail -f /var/log/nginx/access.logtail -f /var/log/nginx/error.lognano /home/mastodon/live/log/- but this folder stays empty for some reason...
Install Mastodon as a software
Following this Installation guide
This guide assumes a bunch of things are already installed, but if you don't have them on your system you will have to install them. We provide here additional guidance because we don't have anything installed.
In order to use the installation commands from the guide, we need to log into the root with su and then run the commands
System packages > Yarn
After enabling corepack, the documentation doesn't say it but you should install yarn by just writing yarn in the terminal
Creating the mastodon user
adduser doesn't work? here's how to resolve it:
Unhide the adduser command using the following command:
sudo update-alternatives --install /bin/adduser adduser /usr/sbin/adduser 20
Then run again
adduser --disabled-password mastodon
Setting up Mastodon
To login into the mastodon account, use:
su mastodon
Installing Ruby
All the commands should be ran one by one!!! - make sure to be in the mastodon user as indicated by the tutorial
To resolve the problem about installing rbenv with jemalloc, we do:
exit # out of the mastodon user to go back to root apt-get install libjemalloc-dev su - mastodon RUBY_CONFIGURE_OPTS='--with jemalloc' rbenv install 3.4.4 rbenv global 3.4.4
Installing the last dependencies
Install bundler by running (in root)
apt install bundler apt install gem
Then run the installation commands (in mastodon user):
bundle config deployment 'true' bundle config without 'deployment test' bundle install -j$(getconf _NPROCESSORS_ONLN) yarn install
When this returns a "no such file or directory" error:
RAILS_ENV=production rails mastodon:setup
Navigate to the live folder and run:
bundle install RAILS_ENV=production bin/rails mastodon:setup
SETTING UP
PostGreSQL host: /var/run/postgresql
PostGreSQL port: 5432
Name of PostGreSQL database: mastodon_production
Name of PostGreSQL user: mastodon
Password for PostGreSQL user:
Redis host: localhost
Redis ports: 6379
Redis password: [empty]
Store on the cloud? No
Send e-mails from localhost? No
SMTP server: smtp.mailgun.org (to reconfigure later, leave the following SMTP prompts empty by hitting enter)
Do you want Mastodon to periodically check for updates and notify you? Yes
Save configuration? Yes
Prepare database now? Yes
Compile the assets now? Yes (we will go back to that later cause it logged errors)
Create admin right away? Yes
Username: admin
E-mail: m.berends@hr.nl
Temp password:
Switch back to root!
systemctl restart mastodon*
Acquiring an SSL certificate
Use the example.com commands, they will be editable later, but you should already have a (sub)domain name registered.
If encountering "An unexpected error" when trying to generate certificate, run
apt install python3-certbot-nginx
Setting up nginx
If you get an error while running systemctl restart nginx to apply changes, ask Manetta for help
Setting up systemd services
$EDITOR might not work, if that is the case, just replace it with nano or your preferred text editor
Finally, out of root:
sudo reboot now
Additional info
The git repo of Mastodon is in /home/mastodon/live
To remove the "dubious ownership", we ran git config --global --add safe.directory /home/mastodon/live/.git
We wanted to make sure that Mastodon services are well running:
cd /etc/systemd/system systemctl status mastodon-web
We saw that Mastodon is not running, so we went into the log file:
journalctl | grep mastodon-web
With that, we discovered that there is a problem around the ruby installation so we went back to the Mastodon installation guide, to the ruby installation part and re-ran the commands one by one. But that didn't solve it...
What we did to fix that is actually remove the .rbenv files and start again from the ruby installation.
Configuring environment
The date is 06-06-2025. We are just now learning that LOCAL_DOMAIN and WEB_DOMAIN are supposed to be written very carefully for they cannot be changed safely. The thing is, we did change them (recklessly) and it is probably what is causing the "After logging in to the admin account" errors :)))
Troubleshooting
mastodon-web is still not running: the xvm VPN IP address is blocked so we followed these steps:
cd mastodon/live nano .env.production
at the end of the file, add the following lines:
RAILS_LOG_LEVEL=debug LOG_LEVEL=silly
in the beginning:
LOCAL_DOMAIN=404.xpub.nl WEB_DOMAIN=404.xpub.nl systemctl restart mastodon*
Error logs from /var/log/nginx/:
2025/06/02 16:40:42 [error] 19943#19943: *1355 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: 404.xpub.nl, request: "GET /favicon.ico HTTP/1.0", upstream: "http://127.0.0.1:3000/favicon.ico", host: "10.0.0.19", referrer: "https://404.xpub.nl/" 2025/06/02 16:40:42 [error] 19943#19943: *1355 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.1, server: 404.xpub.nl, request: "GET /favicon.ico HTTP/1.0", upstream: "http://127.0.0.1:3000/500.html", host: "10.0.0.19", referrer: "https://404.xpub.nl/"
Error logs from journalctl -u mastodon-web.service -f:
Jun 02 16:46:46 debby bundle[22928]: E, [2025-06-02T16:46:46.036606 #22928] ERROR -- : [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: 10.0.0.19
How we fixed the issue
Following this reddit post, we added the ip address 10.0.0.19 to "allow hosts" in the ruby config file, which we found with this command:
find . -iname production.rb
and then opened:
nano /home/mastodon/live/config/environments/production.rb
This caused another error: it started to redirect http://10.0.0.19 to https://10.0.0.19.
To solve this... we figured out that we need to LIE to mastodon... Michael found this great blog post that explained it: https://blog.vyvojari.dev/mastodon-behind-a-reverse-proxy-without-ssl-https/
We changed the nginx config:
nano /etc/nginx/sites-enabled/mastodon
and changed this line in location @proxy:
#proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Proto https; # the lie is right here (!!!! we love it, sometimes you need to lie)
This resolved the redirect error!!
Then, we restarted nginx and mastodon... and:
NEXT ERROR
An error popped up in the journalctl:
journalctl -u mastodon-web -f
Which listed the error about the assets package not being able to download.
So we went back to the step that failed in the interactive setup wizard... and ran:
RAILS_ENV=production bin/rails assets:precompile
which gave the same errors again...
Michael suggests to run yarn install again, which was successful.
After that, we ran the command above again (assets:precompile)... (which seems to work now!!)
woohoo, different error!!! an elephant-mashing-the-keyboard-error (500), but this is good news, the server is responding.
Since now the errors were pointing at some missing assets, Michael had a hunch about the yarn install part and found this post where someone suggested to remove the yarn.lock file.
Afterwards yarn install threw an error, but we simply re ran RAILS_ENV=production bin/rails assets:precompile and then everything installed correctly.
Afterwards we opened the journal in follow mode to check what happens when someone connects to the landing page, and everything worked fine with no errors.
To avoid having a cluttered output, we commented out the debug lines we added previously at the bottom of the /live/.env.production file
After logging in to the admin account
On June 6th, we logged in to the admin account, but we started encountering various errors:
- "Connection time out" when trying to access the Settings page, because the url redirects to the IP instead of staying on the domain.
- The cookies error appears when we try to update stuff (such as the profile picture)
- Some other errors:
We then looked at the journal of mastodon-streaming
journalctl -u mastodon-streaming@4000 -f
and discovered a big amount of errors regarding the head of the url, which returns two things, 10.0.0.19 and 10.0.0.1, not 404.xpub.nl.
It then tries to get to the api section, failing in loading it because the head is wrong.
We continued reading the next section of the installation (Configuring your environment), after the section we just cleared (install from source), and found out a nasty message.
Pretty dark. We did modify both local domain and web domain for sure, because when we had issues with "blocked hosts" we found a reddit post that was talking about these two variables in the live/.env.production file. We tried getting the rails console working to check ENV[WEB_DOMAIN], but we got an error saying that RAILS_ENV is not set so the console does not open.
Notes Manetta
6 June, 15:00
Diving into this error...
journalctl -u mastodon-streaming@4000 -f says:
Jun 06 14:27:46 debby node[52026]: {"level":"error","time":1749212866199,"pid":52026,"hostname":"debby","name":"streaming","req":{"id":"874c2d6d-4e89-41cc-ae80-e239872e1e27","method":"GET","url":"/api/v1/streaming/?","query":{},"params":{},"headers":{"host":"10.0.0.19","x-real-ip":"10.0.0.1","x-forwarded-for":"10.0.0.1","x-forwarded-proto":"http","connection":"close","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0","accept":"*/*","accept-language":"en-GB,en;q=0.5","accept-encoding":"gzip, deflate, br, zstd","sec-websocket-version":"13","origin":"https://404.xpub.nl","sec-websocket-protocol":"[Redacted]","sec-websocket-extensions":"permessage-deflate","sec-websocket-key":"[Redacted]","sec-gpc":"1","cookie":"[Redacted]","sec-fetch-dest":"empty","sec-fetch-mode":"websocket","sec-fetch-site":"same-origin","pragma":"no-cache","cache-control":"no-cache"}},"err":{"type":"RequestError","message":"Unknown channel requested","stack":"RequestError: Unknown channel requested\n at authenticationMiddleware (file:///home/mastodon/live/streaming/index.js:540:12)\n at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)\n at /home/mastodon/live/node_modules/express/lib/router/index.js:286:9\n at Function.process_params (/home/mastodon/live/node_modules/express/lib/router/index.js:346:12)\n at next (/home/mastodon/live/node_modules/express/lib/router/index.js:280:10)\n at Function.handle (/home/mastodon/live/node_modules/express/lib/router/index.js:175:3)\n at router (/home/mastodon/live/node_modules/express/lib/router/index.js:47:12)\n at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)","name":"RequestError","status":400},"msg":"RequestError: Unknown channel requested"}
So... RequestError: Unknown channel requested
Rewinding for a second... where are all the config files again??
nano /home/mastodon/live/.env.production: rbenv config, log levels can be set herenano /home/mastodon/live/config/environments/production.rb: ruby production confignano /etc/nginx/sites-enabled/mastodon: nginx config
And the logs:
journalctl -u mastodon-web -fjournalctl -u mastodon-streaming -fjournalctl -u mastodon-streaming@4000 -ftail -f /var/log/nginx/access.logtail -f /var/log/nginx/error.lognano /home/mastodon/live/log/- but this folder stays empty for some reason...
And the tmp admin password is: 4d8f2b592bcf974140cd85d59ad78a56
Oke, what is the situation?
Let's restart the mastodon services!
systemctl restart mastodon*
Hmm, actually, after this restart, there are no errors.
I will now try to login as admin.
Oke, i get this:
- on the screen: "Security verification failed. Are you blocking cookies?"
- nginx access.log:
10.0.0.1 - - [06/Jun/2025:15:41:02 +0200] "POST /auth/sign_in HTTP/1.0" 422 1528 "https://404.xpub.nl/auth/sign_in" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0", a 422 error - journalctl mastodon-streaming@4000: nothing
- journalctl mastodon-web: no error, but these DEBUG + INFO logs, referring as well to error code 422
Jun 06 15:41:02 debby bundle[59371]: I, [2025-06-06T15:41:02.746978 #59371] INFO -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] method=POST path=/auth/sign_in format=html controller=Auth::SessionsController action=create status=422 allocations=40742 duration=24.58 view=19.38 db=1.07
And.. this:
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719433 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:blocked_ips ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719740 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache fetch_hit: blocked_ips ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.719928 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:rack::attack:5830724:throttle_login_attempts/ip:10.0.0.1 ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024, raw: true})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.720028 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache increment: cache:rack::attack:5830724:throttle_login_attempts/ip:10.0.0.1 ({amount: 1})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.721507 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache read: cache:rack::attack:485893:throttle_login_attempts/email:admin ({expires_in: 10 minutes, namespace: "cache", compress: true, compress_threshold: 1024, raw: true})
Jun 06 15:41:02 debby bundle[59371]: D, [2025-06-06T15:41:02.721669 #59371] DEBUG -- : [d59742d4-6534-472e-9d85-0b1e47b9578a] Cache increment: cache:rack::attack:485893:throttle_login_attempts/email:admin ({amount: 1})
which makes me wonder if 10.0.0.1 gets blocked by the streaming part of Mastodon... and thus, if the HUB proxy_pass settings are again the issue..
Do we need to "allow hosts" for the streaming service as well perhaps?? Like we did with the mastodon-web service??
(btw.. the "Unknown channel requested" error does not pop up anymore..)
Added ALLOWED_PRIVATE_ADDRESSES=10.0.0.19,10.0.0.1 to /home/mastodon/live/.env.production... found it in the official docs: https://docs.joinmastodon.org/admin/config/#allowed_private_addresses, but it seems to not do anything.. so i disabled it again.
Added ALTERNATE_DOMAINS=10.0.0.1 to nano /home/mastodon/live/config/environments/production.rb... but no.
Added the same https nginx lie trick as we did with the mastodon-web part...: proxy_set_header X-Forwarded-Proto https; # another lie !!! in the nginx config... but no (i kept it there btw.. just good to keep in mind)
Hmmz... so the question remains: how to get around the "Security verification failed. Are you blocking cookies?" error...
Tried one more thing: to visit https://404.xpub.nl/web after getting the error above, and this redirected me (after a long wait) to https://10.0.0.19 ........ (???)
Trying one more thing: does postgresql work?? Following this: https://www.sql-easy.com/learn/how-to-login-to-postgresql/:
# switch to the postgres user su - postgres # start psql psql # list all databases \l # connect to the database: mastodon_production \c mastodon_production # show all tables in this database \dt # run this to have a better output of the info below # enabling the expanded view \x # read all values in one table TABLE table_name; TABLE ip_blocks;
Postgresql seems to work oke, but hmm, i did not find any useful info here...
More notes of Manetta after a night of sleep
7 June 11:30
I reverted the 2nd nginx https lie in the @streaming block (to stay with the original example of the blog post Michael found)...
I found this: https://docs.joinmastodon.org/admin/config/#trusted_proxy_ip.. and added the ip 10.0.0.1... IF the reverse proxy is still THE issue... it should fix it, but our ip (10.0.0.1) is already in the default settings, so it did not do anything.. and this error still shows up in the mastodon-web logs:
Jun 07 11:45:54 debby bundle[69620]: D, [2025-06-07T11:45:54.923732 #69620] DEBUG -- : [2658241c-85ca-408f-8228-a4847c77be74] Cache increment: cache:rack::attack:5830965:throttle_login_attempts/ip:10.0.0.1 ({amount: 1})
Hmm... taking a step back... this is our situation:
┌────────────────────────────┐
│ https://404.xpub.nl │
└────────────────────────────┘
│
│ 194.61.65.6
│ HTTPS
▼
┌────────────────────────────┐
│ XVM │
└────────────────────────────┘
│
│ proxy_pass 10.0.0.19
│ HTTP
▼
┌────────────────────────────┐
│ debby │
└────────────────────────────┘
│
│ location / { @proxy }
│ HTTP
▼
┌────────────────────────────┐
│ Mastodon_reverse_proxy │
└────────────────────────────┘
│
│ proxy_pass 127.0.0.1:3000
│ proxy_pass 127.0.0.1:4000
│ HTTP
│ but with HTTPS lie
▼
┌────────────────────────────┐
│ Mastodon_backend_app │
└────────────────────────────┘
Found this: https://github.com/mastodon/mastodon/issues/21139, which is a way to run Mastodon fully over HTTP (so without lying). So, i disabled our lie, and followed the two changes made in the issue. And, actually, the mastodon app runs oke! So it seems that this way is equally solving our reverse proxy SSL issue. Nice. But, the login issue is still there...
However.. if i try to log in directly on http://10.0.0.19 using the enabled HTTP connection (which i can, because my laptop is part of the HUB), i do get a different error: "Invalid E-mail address or password." Hmmm!
So i generated a new password for admin: ec94e41375995744b3712a0df3a47110... with:
su - mastodon cd live RAILS_ENV=production bin/tootctl accounts modify admin --reset-password
This did not fix it.
But then......... i realised... that the login screen says: EMAIL ADDRESS, not USERNAME........ !!!!
and i so i used: m.berends@hr.nl + the new password, and.......... IT WORKED!!!!!!!!
GOSH.............. so stupid!!
Trying to log in on https... it still gives the same error. But if you then refresh: it works! PFF!!!
Hmm, maybe i cheered too quickly... some of the admin setting pages still trigger this error... hmpfz...
Current status
- inspector: 422 error
Content-Security-Policy: The page’s settings blocked an inline script (script-src-elem) from being executed because it violates the following directive: “script-src 'self' https://404.xpub.nl 'wasm-unsafe-eval'”
- nginx: 422 error
10.0.0.1 - - [09/Jun/2025:22:41:54 +0200] "POST /invites HTTP/1.0" 422 1528 "https://404.xpub.nl/invites" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
- mastodon-streaming@4000: RequestError, Unknown channel requested
Jun 09 22:26:59 debby node[72801]: {"level":"error","time":1749500819242,"pid":72801,"hostname":"debby","name":"streaming","req":{"id":"2c36e313-d8c3-42ce-a554-d9f99264ff7b","method":"GET","url":"/api/v1/streaming/?","query":{},"params":{},"headers":{"host":"10.0.0.19","x-real-ip":"10.0.0.1","x-forwarded-for":"10.0.0.1","x-forwarded-proto":"http","connection":"close","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br, zstd","sec-websocket-version":"13","origin":"https://404.xpub.nl","sec-websocket-protocol":"[Redacted]","sec-websocket-extensions":"permessage-deflate","sec-websocket-key":"[Redacted]","cookie":"[Redacted]","sec-fetch-dest":"empty","sec-fetch-mode":"websocket","sec-fetch-site":"same-origin","pragma":"no-cache","cache-control":"no-cache"}},"err":{"type":"RequestError","message":"Unknown channel requested","stack":"RequestError: Unknown channel requested\n at authenticationMiddleware (file:///home/mastodon/live/streaming/index.js:540:12)\n at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)\n at /home/mastodon/live/node_modules/express/lib/router/index.js:286:9\n at Function.process_params (/home/mastodon/live/node_modules/express/lib/router/index.js:346:12)\n at next (/home/mastodon/live/node_modules/express/lib/router/index.js:280:10)\n at Function.handle (/home/mastodon/live/node_modules/express/lib/router/index.js:175:3)\n at router (/home/mastodon/live/node_modules/express/lib/router/index.js:47:12)\n at Layer.handle [as handle_request] (/home/mastodon/live/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/home/mastodon/live/node_modules/express/lib/router/index.js:328:13)","name":"RequestError","status":400},"msg":"RequestError: Unknown channel requested"}
- mastodon-web: nothing
- postgresql: nothing
Next steps:
- check if database works correctly.. the error seems to be triggered on a POST request only...
- check which git branch is currently active
- reinstall mastodon?
Switching to Mastodon main
We were still on the glitch-soc/main branch... this might trigger the issues actually!
So let's go back to the mastodon main branch:
git checkout main error: Your local changes to the following files would be overwritten by checkout: config/environments/production.rb
let's copy the changed config files outside the git first:
mkdir /home/mastodon/configs_tmp/ cd /home/mastodon/live/ cp config/environments/production.rb ../configs_tmp/ cp .env.production ../configs_tmp/
restore (undo) changes in the glitch-soc/main branch:
git restore config/environments/production.rb git restore yarn.lock
switching back to mastodon main:
git checkout main
Then copy the configs back that we changed:
cp ../configs_tmp/production.rb config/environments/production.rb cp ../configs_tmp/.env.production .
and restart mastodon....
Error: bundler: failed to load command: puma
Maybe repeat the rm yarn.lock step...
su - mastodon cd /home/mastodon/live rm yarn.lock cd .. rm yarn.lock cd live yarn install bundle install RAILS_ENV=production bin/rails assets:precompile
Restarting mastodon...
Mastodon starts.. but the error is still there :(
Working with a new database
- new db name:
mastodon_production_two - new admin passwd:
bf5b279ccf5b74feb87f5daed5e132f4 - copying back the configs from the old
.env.productionfile
Same errors...
- checking if it's possible to log in to psql with the user mastodon and password:
psql -h localhost -d mastodon_production_two -U mastodon -p 5432
Hmm, this is maybe the issue then:
psql: error: connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "mastodon" connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "mastodon"
And:
psql -h /var/run/postgresql -d mastodon_production_two -U mastodon -p 5432
gives:
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: Peer authentication failed for user "mastodon"
I changed the password of the mastodon user, but this did not help.
Troubleshooting 11-06-2025
Database
Today, we tried logging into the database.
To do so, we figured out that some access thing in the config file needed to be changed (see here.
Once we made that change, we managed to log into the database of postgresql, but still not into the mastodon account :(
Today, we tried logging into the database with the user mastodon... but this did give an error (see above).
To solve it, we figured out that some access thing in the config file needed to be changed (see here.
nano /etc/postgresql/17/main/pg_hba.conf
Once we made that change, we managed to log into the database of postgresql, with:
psql -h /var/run/postgresql -d mastodon_production_two -U mastodon -p 5432
but still not into the mastodon account :(
422 CSRF error
- Adding
skip_before_filter :verify_authenticity_tokeninhome/mastodon/live/config/environments/production.rb...which doesn't work and actually breaks everything :)))
- Trying to run the command on its own
RAILS_ENV=production bin/rails skip_before_filter :verify_authenticity_token Unrecognized command "skip_before_filter :verify_authenticity_token"
Also not working...
- Looking for forgery (there's a security issue in there but we're not really sure what it exactly does... see that link for more info)
for FILE in $(find .); do echo $FILE && cat $FILE | grep forgery ; done
in config/environments/test.rb, get the line:
config.action_controller.allow_forgery_protection = false
paste it in config/environments/production.rb
IT WORKS!!!!!!!!
To do still, later
- no SSH password login: https://docs.joinmastodon.org/admin/prerequisites/#do-not-allow-password-based-ssh-login-keys-only
Add Glitch-Soc to the Mastodon installation
From this installation guide
su su - mastodon cd live
We already added the remote git repo, so we catch up from the next steps
git checkout glitch-soc/main
which sends back an error because of the changes we made during the troubleshootings...
So we try copying the config files out of the git branch:
exit cd ../mastodon/live cp config/environments/production.rb ../configs_tmp cp .env.production ../configs_tmp git add . git stash git checkout glitch-soc/main
If you encounter the detached HEAD error, you can fix it with:
git branch #to check where you are git status
We solved this by using this command..
First we are moving back to the main mastodon branch:
git checkout main
then switched to glitch-soc/main:
git switch -c glitch --track glitch-soc/main
and did git pull:
git pull
Then we continued the installation, following the steps here..
troubleshooting glitch soc
we removed glitch soc by moving back to the mastodon main branch.
on mastodon production.rb we changed config.force_ssl = true to config.force_ssl = false (additional to setting FORCE_HTTPS=false, which manetta did before already, following this link https://github.com/mastodon/mastodon/issues/21139)
mastodon itself is now running and its possible to log in
note: when generating an invite link it still appears with the ip address instead of the domain name... it is however possible to just replace the ip with the url in the generated link before sending it
we are not yet able to create new users, because we dont have an e-mail server configured. we are using the e-mail settings from gittea and the server from multiplace
these are the e-mail settings:
Protocol: smtp+starttls Enable: HELO SMTP Addr: mail.multiplace.org SMTP Port: 587 User: reports@xpub.nl
we edit these in .env.production
!! the first users are in online !!
now trying to move on the glitch-soc branch again (following the steps in the tutorial)
after git pull on the glithc branch we make sure that we have the edited lines in our config/environments/production.rb file
config.force_ssl = false config.hosts = [ "10.0.0.19", "10.0.0.1" ]
config.action_controller.allow_forgery_protection = false
make sure to be in the right user or root and folder before running each of the steps of the tutorial
switch to mastodon user by:
su - mastodon
then go to /home/mastodon/live to run the ruby / RAILS related commands
leave mastodon user exit
Troubleshooting
Key log files
nginx:
tail -f /var/log/nginx/error.log
mastodon-web
journalctl -u mastodon-web -f