Heads up on social.coop server space
Just to report on the last server outage whilst I'm thinking about it....
This last one was resolved by @fardog :raised_hands: who happened to be awake at the right time. He discovered that it was because it was running low on disk space on the main (root) partition.
He pruned some docker image cruft, but it's still currently at 93% full.
Now I can't explain exactly why it's so full, but obviously it's something we need to do something about or our server will start dying all the time. It's not the Mastodon database, that's on another (also 64% full) disk.
(Perhaps the growth was related to the recent Mastodon influx I've been hearing about, but either way we should expect more users and more tooting...)
Sorting this out may require taking the server down for a while, I suspect.
There's also backups, I notice the database backup file has quadrupled in size since about June (2G -> 8.4G), which probably needs investigation. I say 'backup' because we currently just have a manual backup of the database, and it's only run when someone remembers to. In order to protect ourselves from various trousers-falling-down scenarios we might encounter, we need an automated back-up, ideally generational, which also means more 10s - 100s of gigabytes of (off-server) disk space.
Does anyone here know much about analysing Mastodon instances, or know someone who does?
And this touches on the issue of spending funds, which is a different issue but I'll mention here: perhaps we should allocate a budget to working groups, which they can spend at their discretion without the need to go back to the main / finance group?
For those with git.coop accounts, you can see the tickets I created on the recent outage and the disk space question here. I suggest we keep the technical discussion there as much as possible to spare those here who have been overwhelmed by Loomio chat. :) Anyone who wants an account can sign up following the instructions here https://git.coop/social.coop/