Loomio
October 23rd, 2017 01:27

Migration to new environment / Mastodon upgrade

Victor Matekole
Victor Matekole Public Seen by 538

We are preparing to migrate social.coop platform to a new set of servers and improved setup, consisting of the following:
— More capacity
— CDN/S3 integration using DreamObjects/Fastly
— Improved monitoring
— Upgrade Mastodon to 2.0

The service will be taken down around 02:00 CET (01:00 UTC) on Tuesday morning (24/10/2017) for approx 2hrs.

After service is restored there will be a few issues outstanding:
— Registration service will be down for a few days
— Additional configuration is needed to improve the current Status page

If there are any objections, please state them here. Thank you all.

Alan

Alan October 24th, 2017 06:39

Is this still ongoing? I am getting a flying mastodon

Victor Matekole

Victor Matekole October 24th, 2017 07:53

Hello all, social.coop is back up and fully migrated to the new environment. Please let myself or Mayel know if you find any issues. :pray:

Fabián Heredia Montiel

Fabián Heredia Montiel October 25th, 2017 02:59

There are some glitches / errors with timelines and profile pictures. Not sure what is happening.

Victor Matekole

Victor Matekole October 25th, 2017 03:58

@fabianhjr I am aware of the profile pictures but not timelines... Can you send links and screenshots?

Fabián Heredia Montiel

Fabián Heredia Montiel October 25th, 2017 04:34

Glitch is that it stops scrolling (loading new toots) after a short while.

Only error in the JS Console is

This site appears to use a scroll-linked positioning effect. This may not work well with asynchronous panning; see https://developer.mozilla.org/docs/Mozilla/Performance/ScrollLinkedEffects for further details and to join the discussion on related tools and features!

I think it is an upstream bug, Mastodon App (Tootdon) is working great.

Michele Kipiel

Michele Kipiel October 29th, 2017 10:14

I can see this too... I basically need to refresh the whole page every couple minutes to see new toots and replies. Is there any timeline for this to be solved?

Charles Stanhope

Charles Stanhope October 29th, 2017 22:55

I am seeing the timeline issues, but as I was told it might be related to the image syncing, I won't bring it up again. However, I am also seeing what looks like federation issues? For instance, I haven't been seeing posts from some people I follow on octodon.social except when they happen to be boosted by some other people on other instances. The same may be true for people on mastodon.sdf.org. Sorry to pile on. I'm just trying to understand what may be going on. :-/

Fabián Heredia Montiel

Fabián Heredia Montiel October 30th, 2017 04:20

Hey @victormatekole, it looks like my home timeline is fixed.

Thanks for looking over the issues and helping with all the tech/infrastructure. :3

Fabián Heredia Montiel

Fabián Heredia Montiel October 30th, 2017 04:21

I think replies and non-federated toots won't show up in there.

Charles Stanhope

Charles Stanhope October 31st, 2017 02:03

@fabianhjr Right. To be clear, I am referring to federated toots. A couple of hours ago, I reached out to a person I follow on octodon.social (who I know also follows me) on a side channel. I asked him to look at whether he was seeing my toots. He said he wasn't. The last toot he saw in his timeline from me was about the time I started to notice a problem a little over two days ago. So it appears the symptom is happening in both directions. That's about all the troubleshooting I can do sans standing up my own instance and trying to federate with us. Something I won't be doing anytime soon. :slight_smile:

Fabián Heredia Montiel

Fabián Heredia Montiel October 31st, 2017 07:23

From @datatitian ( https://social.coop/@datatitian/98921817800115298 ), I can second that, seeing only a subset of what I am following in my home timeline, mostly local.

Uhh, are we having trouble federating? I can only see locals and GNU Social users in the federated timeline.

Victor Matekole

Victor Matekole October 31st, 2017 11:01

@fabianhjr and @charlesstanhope can you please send me output from your Console to victor@seedbloom.it. @mayel and I are not suffering from these symptoms, check — https://www.dropbox.com/s/u5tnmp6y6yc9083/Screenshot%202017-10-31%2012.00.26.png?dl=0

Victor Matekole

Victor Matekole October 31st, 2017 11:02

@fabianhjr and @charlesstanhope can you send full details of your browser/machine https://www.whatsmybrowser.org/

Charles Stanhope

Charles Stanhope October 31st, 2017 13:19

Email sent, so hopefully it makes it through. :)

Michele Kipiel

Michele Kipiel November 1st, 2017 09:44

Both mentions and timelines seem to be working fine for me now (ie. lazy loading happens seamlessly). Thank you for the great work!

Fabián Heredia Montiel

Fabián Heredia Montiel November 2nd, 2017 05:00

whatsmybrowser.org/b/5LE9F4P

Alan

Alan November 4th, 2017 09:43

Stupid suggestion. Would it help to unfederate from all the problematic instances from our side, then let them refederate?

Victor Matekole

Victor Matekole November 5th, 2017 09:19

@alan I filed an issue with Mastodon — https://github.com/tootsuite/mastodon/issues/5593

Mayel de Borniol

Mayel de Borniol January 16th, 2018 13:41

FYI a core router failed at our datacenter so we're down. They're working on it. Will post updates: https://status.social.coop/

Mayel de Borniol

Mayel de Borniol February 20th, 2018 10:33

#SocialCoop going down for a few minutes for a server upgrade

Paul Lindner

Paul Lindner February 23rd, 2018 06:57

Looks like you're on 2.1.3 now. Congrats!

Liaizon Wakest

Liaizon Wakest June 25th, 2018 09:38

so social.coop is now seriously behind in what version its running. what can we do to get upgraded to the newest version?

Victor Matekole

Victor Matekole June 26th, 2018 11:16

@wulee @mayel @matthewcropp .... Didn't want to create a new thread — In recent weeks, I think, we have all suffered from thread overload... Just wanted to note — whilst there are lots of discussions and proposals in formalising ops team to be concluded and ratified — I'd be happy to bump up Mastodon this week. From experience, users just want the latest version and are ok with some downtime as a result.

RB

Robert Benjamin June 26th, 2018 18:43

Was really hoping we would have a coordinator in place ahead of the version upgrade. Were about 6 days out from that (2 days to close Tech Ops Team Formation Proposal and a 4 day nomination/poll vote after that). If the upgrade is really pressing or we're at risk of losing your contribution 2 weeks from now any chance that @wulee could shadow you on the process? Helpful or No?

Nick S

Nick S June 26th, 2018 19:59

I probably could if it was at a time I wasn't otherwise engaged :). This could be tomorrow before 2pm GMT, for example, but I'm at work Thursday and Friday. Victor has my email address.

Victor Matekole

Victor Matekole June 27th, 2018 09:16

I offered to do this — one because I have a window of time this week only, as I am working on a migration, after then I have to switch my attention to a lagging project that puts me into another mode.

Second, it appears users feel secure that their instance is being kept up-to-date despite there being no notable advantage to their user experience. I am just trying to help and attend to users concerns.

I have previously offered "drive-by" terminal sessions on the documentation thread as I believe that is the best way for newcomers to learn — upgrading Mastodon (and generally Rails apps) can be an assault course sometimes, you can bet on something unexpected happening. But had no direct response to the offer ...

If you'd rather not do the upgrade and do it under the new team, I am also ok with that, just let me know... I will most likely do this late at night as that is my optimum time for doing such tasks.

Nick S

Nick S June 27th, 2018 10:13

Personally I think you should do whatever's easiest for you, since there'll be other opportunities to do upgrades.

A possible compromise: is it possible to record your upgrade work so that people can watch over your shoulder at their convenience later? i.e. As if you had company, but executed at a time of your convenience.

Even just a written record of the steps and obstacles you encounter could be a useful guide for someone trying to to learn the job.

As an aside, I've been poking around on toot/trunk.social.coop. Various random questions occur to me as a result. What is the best way to ask, without annoying you with tons of them? (In the interim until we get GitLab working, which I hope will be where we do this eventually, yes?)

RB

Robert Benjamin June 27th, 2018 19:58

I don't advocate for not doing if it this a good time. I do like @wulee suggestion that it be recorded. If that is possible. In either case I am not an authority on Mastodon upgrades so take that under consideration.

Not sure how kicking off upgrades were handled in the past but from a procedural point of view is it possible to float a URGENT (2 day) proposal to make the upgrade official (with you doing it) and to track hours as there is a remuneration pool ($250) that has been already been approved for this task?

Victor Matekole

Victor Matekole June 28th, 2018 11:36

As an aside, I've been poking around on toot/trunk.social.coop. Various random questions occur to me as a result. What is the best way to ask, without annoying you with tons of them? (In the interim until we get GitLab working, which I hope will be where we do this eventually, yes?)

I am not sure where the Gitlab topic is .... But out of security concerns — I suggest, just mail them to me and @mayel and we can post the thread at a later date when the Gitlab instance is ready.

Victor Matekole

Victor Matekole June 28th, 2018 11:44

In the past we tried to give 24hrs notice to users via Loomio/Mastodon ... Personally, I think a proposal is overkill, in the last wks I have had to cope wth a lot of information from Loomio, which has been disincentivising, I am probably not the only one ... Unless a Mastodon version holds a contentious feature like removing the ability to block other instances from connecting or publishing data automatically to a third-party... Upgrades should be done as a matter of course... IMHO

Neville Park

Neville Park June 28th, 2018 20:36

I agree with @victormatekole that minor updates should be automatic. Maybe if there are more than a certain number of updates or amount of compensation over a period of time, it should be put to a vote?

RB

Robert Benjamin June 28th, 2018 23:35

All sounds reasonable if there is prior president for it being done this way in the past and there being no policy at the moment governing how dev work is kicked off. As you and Mayel are the current dev team I would also assume there won't be any strong objections to either of you being the ones to do the work.

The only connected proposal is the $250 (max $25/hr) pool already earmarked for this work. For the remuneration part at least an hours log would be needed.

In case I am not an authority on the matter. My contribution to this thread is strictly for the sake of transparency and an attempt to organize processes.

Ed Summers

Ed Summers June 29th, 2018 15:01

I'm kind of confused why we are rushing to upgrade now after it has waited this long. I think this would be a great task for the assembled Tech Ops team to do as a group.

Victor Matekole

Victor Matekole June 29th, 2018 15:40

@edsummers read the thread thoroughly, my comments repeated here and also refer to @wakest comment from 4 days ago .... There is no rush to upgrade, just the right conditions to in act the upgrade this week — since it appears I am the only one suited to do the job, for which I am quite pessimistic of .... If you object, let me know as I plan to do it before the weekend is out ... :

"I offered to do this — one because I have a window of time this week only, as I am working on a migration, after then I have to switch my attention to a lagging project that puts me into another mode.

Second, it appears users feel secure that their instance is being kept up-to-date despite there being no notable advantage to their user experience. I am just trying to help and attend to users concerns.

I have previously offered "drive-by" terminal sessions on the documentation thread as I believe that is the best way for newcomers to learn — upgrading Mastodon (and generally Rails apps) can be an assault course sometimes, you can bet on something unexpected happening. But had no direct response to the offer ...

If you'd rather not do the upgrade and do it under the new team, I am also ok with that, just let me know... I will most likely do this late at night as that is my optimum time for doing such tasks."

Victor Matekole

Victor Matekole June 29th, 2018 15:42

And as far as I am aware there is no assembled team but @wulee has made some great efforts in assembling a task list for it.

Ed Summers

Ed Summers June 29th, 2018 15:57

@victormatekole I think the team is about to be constituted, and I for one hope that you will be on it to help ease this transition. We are currently 6 months out of date on v2.1.3. It's not just a question of security, there are new features that have been introduced in that time, such as a much richer data export.

I'm a bit concerned that it seems like you are close to burning out on shouldering this work, and I think @robertbenjamin's proposal to put together an actual team will help share the load, and will be generally good for social.coop and you.

Unless you are looking at never having time to participate again after this small window of opportunity you have I would recommend that you wait to upgrade until we can do it as a group. But hey, that's just one guys opinion :-)

Thanks for all the work you have done. Truly, I still am amazed at what has been achieved so far on social.coop

Victor Matekole

Victor Matekole June 29th, 2018 22:23

Unless you are looking at never having time to participate again after this small window of opportunity you have I would recommend that you wait to upgrade until we can do it as a group. But hey, that's just one guys opinion

I have being trying to offload my (social.coop) responsibilities for some mths now as I simply do not have the bandwidth or incentive to continue ... I need to push on with other initiatives, this was a good window for me that would cause least personal disruption ... Later will be more difficult, @wakest comments were a motivation to try and do something, and in all honesty I hadn't realised how much time had past from the last upgrade — it still feels recent, but then I have been working intensely for the last 9mths that time has moved faster for me ...

Victor Matekole

Victor Matekole June 29th, 2018 22:26

Thanks for all the work you have done. Truly, I still am amazed at what has been achieved so far on social.coop...

@mayel is the one to thank mostly ... He started this without ANY knowledge or experience of Rails/docker deployment (at least I had that) and got us going... He has been both courageous and enduring, I just tried to relieve some burden off of him...

Victor Matekole

Victor Matekole June 29th, 2018 22:32

Since no one seems to be hurting for lack of upgrade and sentiment seems to be for giving the opportunity for the new team, I will not carry out the task.

Liaizon Wakest

Liaizon Wakest June 29th, 2018 23:00

I would really love to follow along and learn the ropes of mastodon server maintenance if I could watch or participate in this upgrade in anyway that would be lovely.

Ed Summers

Ed Summers June 30th, 2018 02:47

@robertbenjamin what are the next steps for constituting the ops team? I’m asking because it seemed like you had an idea when you put together the last vote about it?