Heads up on social.coop server space
Nick S
Public
Seen by 314
Just to report on the last server outage whilst I'm thinking about it....
This last one was resolved by @fardog :raised_hands: who happened to be awake at the right time. He discovered that it was because it was running low on disk space on the main (root) partition.
He pruned some docker image cruft, but it's still currently at 93% full.
Now I can't explain exactly why it's so full, but obviously it's something we need to do something about or our server will start dying all the time. It's not the Mastodon database, that's on another (also 64% full) disk.
(Perhaps the growth was related to the recent Mastodon influx I've been hearing about, but either way we should expect more users and more tooting...)
Sorting this out may require taking the server down for a while, I suspect.
There's also backups, I notice the database backup file has quadrupled in size since about June (2G -> 8.4G), which probably needs investigation. I say 'backup' because we currently just have a manual backup of the database, and it's only run when someone remembers to. In order to protect ourselves from various trousers-falling-down scenarios we might encounter, we need an automated back-up, ideally generational, which also means more 10s - 100s of gigabytes of (off-server) disk space.
Does anyone here know much about analysing Mastodon instances, or know someone who does?
And this touches on the issue of spending funds, which is a different issue but I'll mention here: perhaps we should allocate a budget to working groups, which they can spend at their discretion without the need to go back to the main / finance group?
For those with git.coop accounts, you can see the tickets I created on the recent outage and the disk space question here. I suggest we keep the technical discussion there as much as possible to spare those here who have been overwhelmed by Loomio chat. :) Anyone who wants an account can sign up following the instructions here https://git.coop/social.coop/
Nick S Tue 21 Aug 2018 10:49AM
The riot.im client (see my link above) should run in all major web browsers, but if not please let us know because we don't want to pick tech which isn't fairly universally accessible for our public channel.
However, if you want a private discussion with a specific person, you're right, of course you can pick any tech you agree on.
And a snapcraft discussion could just be on a new thread here in the Loomio Tech WG group, I didn't mean to imply you needed to discuss that in our public chat room.
Antoine-Frédéric Raquin Tue 21 Aug 2018 11:30AM
It doesn't run on a 2000s Intel CPU without overheating the CPU, whereas xmpp-client, Gajim, Dino… run without much problems on i3 (obviously not on Windows though). I'm not on my own computer (whose motherboard melted) so it's fine, but this exact trend of building software over Electron or in a browser just because it works over all platforms isn't really on point on ecology and accessibility.
It's also not that much accessible because the Riot client has a terrible UX, and I just think that making it the de-facto standard for free software, cooperative, or union work is a bad idea.
I'd simply recommend rocket.chat instead of Matrix (if not XMPP, but that's just because the clients and servers are objectively better regardless of the protocol itself); I don't know if officially using Matrix doesn't push people out of the decision process and I'd like to question that.
Victor Matekole Fri 24 Aug 2018 6:59AM
Never heard of Snap till now... However, here is current status of Snap package for Mastodon — https://github.com/tootsuite/mastodon/issues/1068... Appears non-existent at the moment.
Victor Matekole Fri 17 Aug 2018 2:27PM
Hi all,
Disk space has always been a problem, mainly due to limited budget.... However, there are a couple of things that you missed that will free up further space:
— docker system prune -a — this gets rid of everything that is superfluous, including old containers and volumes. Docker doesn't automatically remove previously ran containers or volumes, unless you say so. It is safe to do this as we mount volumes that need persisting to host, this allows you to not care about Docker preserving volumes or containers.
– NUM_DAYS=7 rake mastodon:media:remove_remote. All media is uploaded locally first, resized/optimised with Paperclip and then pushed to DreamObjects. I'm not sure if Mastodon runs the aforementioned rake task regularly, it is possible it does but from my experience there is always media available to delete when I run it, this frees up a lot of space.
Finally, looking at the disk /var/lib/docker takes up 27GB of space alone, which suggests it is this media that is taking up space. I am now running this rake task in a detached screen, please no reboots : ) until I give the all clear.
Hope this helps.
@wulee @fardog
Victor Matekole Fri 17 Aug 2018 2:31PM
Additionally, I just came across this —
https://github.com/tootsuite/documentation/blob/d9ecbee47d6c09afbff8cf1280e29018872936b3/Running-Mastodon/Production-guide.md#remote-media-attachment-cache-cleanup
It appears that the cache is also made up of images from other instances we are connected to, scary....
Nick S Fri 17 Aug 2018 4:46PM
@victormatekole , can we, or should we try to move the /var/lib/docker folder off the root partition? Is it eventually going to get too big?
Victor Matekole Fri 17 Aug 2018 4:59PM
@wulee good question and probably not a bad strategy as it does tend to bloat. Theoretically there should be no issues but I would never assume with Docker, let me do some digging.
That being said I am wondering if we should consider getting a root server. I run my services via Digital Ocean (mission critical) and Hetzner where I have a couple of root servers(less critical and resource heavy). Hetzner are super cheap and you get a lot for very little, service ain't too bad either and is based in Germany. With an extra $15 or so we can get a couple of terabytes and not too shabby CPU. Nursing a 100GB with a social network of a 1000+ users seems pretty tight.
Mayel de Borniol Sat 18 Aug 2018 8:39PM
Not sure what you mean? The servers we have are already root servers.
And there's pros/cons to 100GB of SSD storage vs 1000GB of SATA storage.
Of course it's probably time to add storage space and/or upgrade the server (with a bigger root partition). I have no objections if you all want to switch to another provider either, though it's worth putting together a comparison table.
Antoine-Frédéric Raquin · Tue 21 Aug 2018 10:32AM
I was suggesting XMPP because :
1) I was not aware of the Matrix chat ;
2) there's no reason to discuss about snapcraft in a Matrix chatroom, as it would mainly flood the chatrooms already here, IMO. I prefer to have 5 private conversations on 5 different sync channels (that anyone is free to join) than to have 5 conversations in the same Matrix chatroom, for purposes of clarity ;
3) I'm not sure that Riot runs on my computer.