Loomio

poddery.com is heavily loaded

Pirate Praveen
Pirate Praveen Public Seen by 235

It was down yesterday (knightswarm is investigating the issue) and I think it is just trying to catch up with backlog. Should we stop new registrations until we are confident about load?

Anish Sheela

Anish Sheela September 14th, 2015 16:33

Yes. We should disable registrations temporarily. Recently many people joined. By the way, how much is current count?

Pirate Praveen

Pirate Praveen September 14th, 2015 16:41

disabled new registrations. I think we should switch to vines from prosody to free up some ram (prosody uses lua where as vines uses ruby)

Pirate Praveen

Pirate Praveen September 14th, 2015 17:23

Pirate Praveen

Pirate Praveen September 16th, 2015 12:36

Some tunings can definitely help. Lets enable gzip compression and infinite caching for /assets

Pirate Praveen

Pirate Praveen September 16th, 2015 12:37

Akshay is testing vines integration on pod.pxq.in and if that works well (webchat is also working) then we'll definitely switch to vines.

Vidyut

Vidyut September 16th, 2015 12:56

IMO, if you're on Debian, use nginx-extras package from dotdeb. Then enable spdy and pagespeed. That should end most of the woes. Can make further suggestions after seeing that impact.

If you don't know how to do it, after nginx-extras is installed, in server block for port 443, after "ssl" add "spdy"

Add line to nginx.conf

pagespeed on;

These two should result in a massive boost. spdy will send resources through faster, while pagespeed does a hell of a lot of optimizing by default - from gzip, etc to removing whitespace, optimizing scripts, blah blah even stuff like adding prefetch.

You can fine tune it to taste, but defaults should already work a minor miracle on performance.

Pirate Praveen

Pirate Praveen September 16th, 2015 14:48

@rajudvindane do you want to try setting it up? nginx-extras is already in debian.

Vidyut

Vidyut September 16th, 2015 16:39

Are you using the nginx-extras from dotdeb or the one from the debian repo? You want the dotdeb one. The debian one doesn't have pagespeed.

Vidyut

Vidyut September 16th, 2015 16:49

the pagespeed should automatically handle your expires and stuff gracefully. So you shouldn't have to do anything to the assets, etc.

If it doesn't (test the pages after setting up spdy and pagespeed) just set expires max for everything that is not dynamic - css, js, txt, images, whatever. only html, php and xml etc should have no or short expiry. (I don't think you guys have any php here - ruby?)

Vidyut

Vidyut September 16th, 2015 16:54

Also, after using the pages a bit, test

curl -I https://poddery.com

You should be able to see a cache HIT header (or it may take some setting up of cache)

Vidyut

Vidyut September 16th, 2015 16:57

Another area to check is in your nginx.conf toward the top itself should be a line for worker_processes

Pravin said you have six cores, so the line should be:

worker_processes 6;

This allows you to spawn threads on all cores - should not be a huge deal, but if you've got a number smaller than 4, raising it might boost performance too.

Manu Krishnan T V

Manu Krishnan T V October 4th, 2015 16:52

Just came across podupti.me and saw that the poddery.com user count is less than 5000. Confirmed the same with poddery.com statistics. So, the user deletion feature of v5 seems to be enabled, deleting inactive users, causing the decrease in user count.

Presently, mysql is the most resource intensive process running. If we could migrate to Postgres, we could enable user registrations again. Last time we failed, possibly due to some error in mysql tables. Anyone have some spare time to give it a try again?

Vidyut

Vidyut October 4th, 2015 17:53

No idea on Postgres, but last I checked, the caching was next to nil - if at all it was happening. This would be a big problem. Static resources should be expires-max, and several other things. If nginx is left so unconfigured, chances are MySQL can also probably do with some optimizing - use some tuner script. The biggest server will still run out of resources if you aren't using them optimally. Poddery did not seem to be a very busy site to justify such heavy use. Any data on the number of posts/database queries, etc involved?

Vidyut

Social Media: Twitter ( https://twitter.com/Vidyut ) Facebook ( https://facebook.com/theVidyut ) Google+ ( https://google.com/+AamJanata ) Diaspora ( https://poddery..com/i/62f15bc003f0 )

Blogs: Intellectual Anarchy ( https://aamjanata.com ) || personal ( https://vidyut.info ) || Nisarga ( https://nisarga.info ) || tech ( https://vidyut.net ) || Homeschooling ( https://homeschoolingindia.in ) || Fek Le ( https://fekle.in )

Manu Krishnan T V

Manu Krishnan T V October 5th, 2015 13:38

Presently, the nginx load is negligible, but adding some caching and tweaking the configurations will be great. In the case of mysql, we have tuned it according to recommendations from mysqltuner (not sure about the name, but was a pearl script).

MySQL database size should be somewhere around 15GB now. The last backup dump around 3 months back, if I remember correctly, was around 12GB.

Sidekiq is the next high resource consuming process, there are 25 concurrent threads of it running, most of them idle almost all the time, as I have been monitoring them for almost 3 weeks now.

I didn't get much time to have a look at the disk io. Since mysql is causing most of the load, it might be related to disk io too, as we are still on RAID HDD.

I haven't dealt with mysql databases of this size in production elsewhere (Worked on an 8GB MSSQL and that was terrible) and not much idea about the structures in use by diaspora. So, no clear idea about how performance is affected. But have heard of many pods migrating away to PgSQL and had high performance gains.

Manu Krishnan T V

Manu Krishnan T V October 6th, 2015 20:15

The user deletion process is heavily resource intensive, causing high load at times.

Presently, users who haven't had any activity for 730 days are removed, after sending a mail and waiting for 30 days. Not sure when this was enabled. Might have happened in any updates around 2 months back as the deletion jobs are processed now. Looking at the scheduled jobs in sidekiq, expecting around 1000 users more to be deleted, before things get back to a stable state. At the moment of making this post, total user count is 4709, which was once above 8000.

Raju Devidas

Raju Devidas October 30th, 2015 15:28

Sorry Praveen, saw this today. Looks I am into too many stuffs right now. I should now focus on completing what I am onto then taking new things to do. Will get into pod management may be after exams.