social.coop down (June 6, 2018)
This is probably apparent to others, but wanted to start a thread about social.coop being down. This can be a space for admins to post updates, if that's helpful.
[Edit]
First point of rendezvous for those on the tech team investigating these issues is our matrix chat room. You may want to visit there first, as this thread is less likely to be updated.
Nick S Thu 6 Sep 2018 2:57PM
Yes... I didn't count that as it has to be updated manually (how I don't know), and I don't think it runs on our servers, so it is subject to termination.
Josef Davies-Coates Thu 6 Sep 2018 3:00PM
OK thanks @wulee
Noah Thu 6 Sep 2018 2:17PM
Seems like lately (maybe post-upgrade?) there's pretty regularly downtime in the morning here (EDT).
Nick S Mon 8 Oct 2018 12:23PM
Hi all. We (@nicksellen and I) are now about to switch social.coop's media object storage provider over. (Ticket on git.coop, for those with an account there, is https://git.coop/social.coop/tech/operations/issues/21)
So the site will be offline for a very short time. We hope you won't notice any missing images when we come back, but if you do, this is why. Note: we've made sure all the social.coop images are safe, it should only be cached remote media which may require restoring.
If you have problems please contact us on our chat channel
https://riot.im/app/#/room/#SocialCoop:matrix.org.
Thanks!
Nick S Mon 8 Oct 2018 3:43PM
Ok, this is essentially done. Some media files from remote sites will appear to be missing because Mastodon still thinks they're cached in our content storage but they're not. However think we can clear the cache with some magic Masto incantations, or failing that, database hacking.
Meanwhile, we're also thinking about the next step, which is to migrate our instance to our new server. If we get the chops to do that today, we may defer monkeying with the cache until after that, because clearing the cache could take a while to run based on previous experiments.
Nick Sellen Mon 8 Oct 2018 6:45PM
Yup, we will attempt the server migration later tonight too, in about an hour. It's good to do it whilst we have our newly found mastodon-fu loaded into our brains.
After we're happy with the stability of the new deployment we can spend a bit more time on documentation and communication of what has been happened on the technical infrastructure front.
Nick Sellen Tue 9 Oct 2018 1:48AM
The migration is complete! Let us know if you see anything a bit wonky still.
There's a bunch more work to do tidying up, etc. sort out proper backups. Tasks are listed at https://git.coop/social.coop/tech/operations/issues in some kind of order. We'll probably head to sleep first before putting all that in order.
Bob Haugen Thu 11 Oct 2018 11:22AM
Had to clear cache to get it to work again. But now it seems back up and running. THanks again for all your hard work!
Matt Noyes Mon 8 Oct 2018 2:41PM
Thanks for your work!
Noah · Thu 6 Sep 2018 2:50PM
we have one but it's not regularly updated, and when i spent a few minutes looking into it, it wasn't immediately clear how one might go about updating it. anyway it's at https://status.social.coop - slightly more info in the git.coop infrastructure doc in Section B, part 3.