Loomio

poddery.com is heavily loaded

PP
Pirate Praveen Public Seen by 285

It was down yesterday (knightswarm is investigating the issue) and I think it is just trying to catch up with backlog. Should we stop new registrations until we are confident about load?

AS

Anish Sheela Mon 14 Sep 2015

Yes. We should disable registrations temporarily. Recently many people joined. By the way, how much is current count?

PP

Pirate Praveen Mon 14 Sep 2015

disabled new registrations. I think we should switch to vines from prosody to free up some ram (prosody uses lua where as vines uses ruby)

PP

Pirate Praveen Mon 14 Sep 2015

PP

Pirate Praveen Wed 16 Sep 2015

Some tunings can definitely help. Lets enable gzip compression and infinite caching for /assets

PP

Pirate Praveen Wed 16 Sep 2015

Akshay is testing vines integration on pod.pxq.in and if that works well (webchat is also working) then we'll definitely switch to vines.

V

Vidyut Wed 16 Sep 2015

IMO, if you're on Debian, use nginx-extras package from dotdeb. Then enable spdy and pagespeed. That should end most of the woes. Can make further suggestions after seeing that impact.

If you don't know how to do it, after nginx-extras is installed, in server block for port 443, after "ssl" add "spdy"

Add line to nginx.conf

pagespeed on;

These two should result in a massive boost. spdy will send resources through faster, while pagespeed does a hell of a lot of optimizing by default - from gzip, etc to removing whitespace, optimizing scripts, blah blah even stuff like adding prefetch.

You can fine tune it to taste, but defaults should already work a minor miracle on performance.

PP

Pirate Praveen Wed 16 Sep 2015

@rajudvindane do you want to try setting it up? nginx-extras is already in debian.

V

Vidyut Wed 16 Sep 2015

Are you using the nginx-extras from dotdeb or the one from the debian repo? You want the dotdeb one. The debian one doesn't have pagespeed.

V

Vidyut Wed 16 Sep 2015

the pagespeed should automatically handle your expires and stuff gracefully. So you shouldn't have to do anything to the assets, etc.

If it doesn't (test the pages after setting up spdy and pagespeed) just set expires max for everything that is not dynamic - css, js, txt, images, whatever. only html, php and xml etc should have no or short expiry. (I don't think you guys have any php here - ruby?)

V

Vidyut Wed 16 Sep 2015

Also, after using the pages a bit, test

curl -I https://poddery.com

You should be able to see a cache HIT header (or it may take some setting up of cache)

V

Vidyut Wed 16 Sep 2015

Another area to check is in your nginx.conf toward the top itself should be a line for worker_processes

Pravin said you have six cores, so the line should be:

worker_processes 6;

This allows you to spawn threads on all cores - should not be a huge deal, but if you've got a number smaller than 4, raising it might boost performance too.

MKT

Manu Krishnan T V Sun 4 Oct 2015

Just came across podupti.me and saw that the poddery.com user count is less than 5000. Confirmed the same with poddery.com statistics. So, the user deletion feature of v5 seems to be enabled, deleting inactive users, causing the decrease in user count.

Presently, mysql is the most resource intensive process running. If we could migrate to Postgres, we could enable user registrations again. Last time we failed, possibly due to some error in mysql tables. Anyone have some spare time to give it a try again?

V

Vidyut Sun 4 Oct 2015

No idea on Postgres, but last I checked, the caching was next to nil - if at all it was happening. This would be a big problem. Static resources should be expires-max, and several other things. If nginx is left so unconfigured, chances are MySQL can also probably do with some optimizing - use some tuner script. The biggest server will still run out of resources if you aren't using them optimally. Poddery did not seem to be a very busy site to justify such heavy use. Any data on the number of posts/database queries, etc involved?

Vidyut

Social Media: Twitter ( https://twitter.com/Vidyut ) Facebook ( https://facebook.com/theVidyut ) Google+ ( https://google.com/+AamJanata ) Diaspora ( https://poddery..com/i/62f15bc003f0 )

Blogs: Intellectual Anarchy ( https://aamjanata.com ) || personal ( https://vidyut.info ) || Nisarga ( https://nisarga.info ) || tech ( https://vidyut.net ) || Homeschooling ( https://homeschoolingindia.in ) || Fek Le ( https://fekle.in )

MKT

Manu Krishnan T V Mon 5 Oct 2015

Presently, the nginx load is negligible, but adding some caching and tweaking the configurations will be great. In the case of mysql, we have tuned it according to recommendations from mysqltuner (not sure about the name, but was a pearl script).

MySQL database size should be somewhere around 15GB now. The last backup dump around 3 months back, if I remember correctly, was around 12GB.

Sidekiq is the next high resource consuming process, there are 25 concurrent threads of it running, most of them idle almost all the time, as I have been monitoring them for almost 3 weeks now.

I didn't get much time to have a look at the disk io. Since mysql is causing most of the load, it might be related to disk io too, as we are still on RAID HDD.

I haven't dealt with mysql databases of this size in production elsewhere (Worked on an 8GB MSSQL and that was terrible) and not much idea about the structures in use by diaspora. So, no clear idea about how performance is affected. But have heard of many pods migrating away to PgSQL and had high performance gains.

MKT

Manu Krishnan T V Tue 6 Oct 2015

The user deletion process is heavily resource intensive, causing high load at times.

Presently, users who haven't had any activity for 730 days are removed, after sending a mail and waiting for 30 days. Not sure when this was enabled. Might have happened in any updates around 2 months back as the deletion jobs are processed now. Looking at the scheduled jobs in sidekiq, expecting around 1000 users more to be deleted, before things get back to a stable state. At the moment of making this post, total user count is 4709, which was once above 8000.

RD

Raju Devidas Fri 30 Oct 2015

Sorry Praveen, saw this today. Looks I am into too many stuffs right now. I should now focus on completing what I am onto then taking new things to do. Will get into pod management may be after exams.