Loomio

Server load

PP
Pirate Praveen Public Seen by 77

I think we are hitting the load limits on our server.

diaspora@poddery:~$ free -m
total used free shared buffers cached
Mem: 3965 3816 149 0 4 575
-/+ buffers/cache: 3235 730
Swap: 3355 1256 2099

Stream is loading slow and sometimes nginx gives a page not found message. How do we handle this?

PP

Pirate Praveen Tue 18 Mar 2014

@manukrishnantv , 1256 M swap is unusual I think.

MJ

Minto Joseph Wed 19 Mar 2014

I ran following while loading several streams..

diaspora@poddery:~$ top -b -n 10 >/tmp/top2.out
diaspora@poddery:~$ grep Cpu /tmp/top.out
%Cpu(s): 1.3 us, 0.3 sy, 0.0 ni, 97.2 id, 1.0 wa, 0.0 hi, 0.0 si, 0.1 st
%Cpu(s): 2.7 us, 0.6 sy, 0.0 ni, 96.2 id, 0.3 wa, 0.0 hi, 0.1 si, 0.1 st
%Cpu(s): 2.1 us, 0.5 sy, 0.0 ni, 95.1 id, 2.0 wa, 0.0 hi, 0.0 si, 0.4 st
%Cpu(s): 0.9 us, 0.1 sy, 0.0 ni, 96.9 id, 1.7 wa, 0.0 hi, 0.0 si, 0.3 st
%Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.1 st
%Cpu(s): 3.1 us, 0.4 sy, 0.0 ni, 92.8 id, 3.4 wa, 0.0 hi, 0.1 si, 0.2 st
%Cpu(s): 7.8 us, 1.5 sy, 0.0 ni, 85.6 id, 3.7 wa, 0.0 hi, 0.1 si, 1.3 st
%Cpu(s): 4.2 us, 0.8 sy, 0.0 ni, 92.9 id, 2.0 wa, 0.0 hi, 0.1 si, 0.1 st
%Cpu(s): 8.6 us, 1.2 sy, 0.0 ni, 81.9 id, 4.6 wa, 0.0 hi, 0.1 si, 3.6 st
%Cpu(s): 3.8 us, 1.0 sy, 0.0 ni, 91.8 id, 2.5 wa, 0.0 hi, 0.1 si, 0.8 st
diaspora@poddery:~$ date
Wed Mar 19 08:09:13 IST 2014

If the memory usage was casusing a serious effect, it would have reflected in sys% usage.

stream loading was also fast.

--

Most of the memory looks accounted by processes. Yes, the memory and swap usage seem very high and could result in funny results when lots of users are logged in.

             total       used       free     shared    buffers     cached

Mem: 4061112 3924404 136708 0 4412 606724
-/+ buffers/cache: 3313268 747844
Swap: 3436540 1455928 1980612

Just that it did not translated into an issue when I checked.

--

May be I tested when there were not much users. Is there a particular time when the issue hits?

We might need to install sysstat to check for historical load details.

PP

Pirate Praveen started a proposal Fri 28 Mar 2014

Stop new sig ups Closed Thu 3 Apr 2014

Outcome
by Pirate Praveen Tue 25 Apr 2017

We have solved the slow stream load issue, no need to close sign ups now.

We cannot provide a good experience with current load. So immediately stop new sign ups.

Then think about moving some of the other services running on this server to their own hosting.

Results
Agree - 2
Abstain - 2
Disagree - 2
Block - 2
4 people have voted (9%)
PP

Pirate Praveen
Agree
Fri 28 Mar 2014

yes, we cannot continue like this. This will energize campaign for more pods.

JS

Juan Santiago
Agree
Fri 28 Mar 2014

It is better for the diaspra ecosystem, servers working properly and have not more work than they can bear, then if I find the right opening pisble new records again.

NAJ

Nitesh A Jain
Agree
Fri 28 Mar 2014

MKT

Manu Krishnan T V
Disagree
Fri 28 Mar 2014

This hasn't caused any serious issues yet.

Also, we are planning to get a new dedicated server by the start of April. Could donate a VM to SMC or FCI to move few of the sites. Hosting sites of SMC in US won't be an issue I believe.

MKT

Manu Krishnan T V
Disagree
Fri 28 Mar 2014

We are planning to get a new dedicated server by the start of April. Could donate a VM to SMC or FCI to move few of the sites. Hosting sites of SMC in US won't be an issue I believe.

PP

Pirate Praveen
Disagree
Thu 3 Apr 2014

the server load issue is resolved and now the experience is smoother again. We can continue with open sign ups.

MKT

Manu Krishnan T V Fri 28 Mar 2014

The memory and swap usage seems high, but I don't find that affecting the overall performance.

It seems like memory is not freed by some processes (expecially php-fpm and ruby) and the system has been running for more than 100 days.

If it starts creating problems, I would suggest restarting all processes and see how the memory usage is.

MKT

Manu Krishnan T V Fri 28 Mar 2014

It seems like the stream loads slow now. The SMC sites was moved to the Poddery server when they had some issues as a temporary solution.

Let me contact Anivar and update the status there. Also, as I mentioned in the proposal, I hope we could donate a VM by mid of April, after all the testing.

MKT

Manu Krishnan T V Fri 28 Mar 2014

Just did a dd test, to look at whether disk IO is causing issues and it seems fine. 150MB/s is good enough I believe. The result is as below:

poddery@poddery:~$ dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; rm test
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 7.13325 s, 151 MB/s

PP

Pirate Praveen Sat 29 Mar 2014

@manukrishnantv that would be a good option. Important point is to sustain quality of service. We can try a reboot and see if it improves the performance.

PP

Pirate Praveen Sun 30 Mar 2014

@manukrishnantv @nandajavarma I think we should move yatra.diasporafoundation.org also to another host as it does not need a vps. Nandaja, can you try to move it to autistici.org or riseup.net hosting?

MKT

Manu Krishnan T V Sun 30 Mar 2014

Moving out wordpress won't be a hard task. @nandajavarma @balasankarchelamat Hope you can do that.

NV

Nandaja Varma Mon 31 Mar 2014

@manukrishnantv @praveenarimbrathod Sure! I will try to move it tonight(will have a better Internet connection, by then).

PP

Pirate Praveen Mon 31 Mar 2014

@manukrishnantv I still feel we should stop new sign ups until we can guarantee a good performance. Otherwise new people joining will have a bad impression of diaspora.

PP

Pirate Praveen Mon 31 Mar 2014

I saw two extra sidekiq processes on the server. I killed them and restarted diaspora. Sign ups are closed now. Once we see good performance for next two days, we can open it again.

PP

Pirate Praveen Mon 31 Mar 2014

seems stable now

$ free -m
total used free shared buffers cached
Mem: 3965 2076 1889 0 42 929
-/+ buffers/cache: 1105 2860
Swap: 3355 140 3215

PP

Pirate Praveen Mon 31 Mar 2014

I have re-enabled sign ups. Load and memory usage is normal so I was tempted to re-open registrations.

MKT

Manu Krishnan T V Sat 5 Apr 2014

The stream now loads faster, but the memory usage has gone to the old state.

poddery@poddery:~$ free -m
total used free shared buffers cached
Mem: 3965 3815 150 0 178 1899
-/+ buffers/cache: 1736 2228
`Swap: 3355 122 3233

Swap usage is not high though.

PP

Pirate Praveen Sun 6 Apr 2014

@manukrishnantv I think memory usage is fine as long as it is not using swap. linux considers free ram as wasted resource so it will use free ram as cache (cache frequently read files etc) . So this is quite normal.

http://www.linuxatemyram.com/ has a nice explanation.

PP

Pirate Praveen Sat 23 Aug 2014

@manukrishnantv can we move smc of fci sites to your vms?

@nandajavarma can we move diaspora yatra to autistici.org?

MKT

Manu Krishnan T V Sat 23 Aug 2014

I have mentioned with few of the SMC community members that my company is willing to sponsor a VM, if required. As things were going without much issues, it seems like the proposal was not discussed. I'll bring this up in the SMC mailing list.

MKT

Manu Krishnan T V Sat 23 Aug 2014

Have sent a mail to the SMC Mailing list, with my offer. Lets see how it goes.

NV

Nandaja Varma Mon 25 Aug 2014

@praveenarimbrathod Yes. I remember applying for it a while back. But could not follow up. Will do that now.

PP

Pirate Praveen Mon 1 Sep 2014

@manukrishnantv let me know when you want to do the migration. I will join you.

PP

Pirate Praveen Sun 28 Sep 2014

@nandajavarma any update on it? Actually we just need a wordpress blog, so you can sign up at noblogs.org (run by autistici) and ask autistici people to provide a dns entry.

MKT

Manu Krishnan T V Sun 28 Sep 2014

Migration of all SMC sites are now complete. What is left is fosscommunity.in and diaspora yatra. Hope to get those sorted out this week itself. Then we could remove all php related stuffs and use the server entirely for poddery.

MKT

Manu Krishnan T V Sat 18 Oct 2014

$ free -m
total used free shared buffers cached
Mem: 3965 3788 177 0 10 1001
-/+ buffers/cache: 2777 1188
Swap: 3355 40 3315

Services last restarted at around 4PM IST (~24 hours ago) Had to restart redis-server, which was running from the time server started. Noticed more than 1/4th of the sidekiq requests fails.

PP

Pirate Praveen Mon 20 Oct 2014

I have reduced number of sidekiq jobs to 20x2 from 25x2 (2 workers), hoping it eases resources.

@manukrishnantv any update on adding more resources?

MKT

Manu Krishnan T V Mon 20 Oct 2014

@praveenarimbrathod No update yet. Bumped the ticket. Let's wait.

MKT

Manu Krishnan T V Mon 20 Oct 2014

$7/month for 2GB and $13.5/month for 4GB additional RAM.

May be I can close the deal at $150 for additional 2GB for the remaining 23 months of service. Not sure though. Do we have the funds?

PP

Pirate Praveen Mon 20 Oct 2014

@anisha @balasankarchelamat how much money do we have?

@manukrishnantv I think we have the funds, can we get the same discount like the main plan? Give the final pricing and aneesh can ask Joe S to make the payment.

MKT

Manu Krishnan T V Wed 22 Oct 2014

$300 for an additional 4GB RAM and 40GB Disk is the deal. We might need to partition the disk again to make use of additional space.

Invoice should be generated by now.

AS

Anish Sheela Sun 26 Oct 2014

@manukrishnantv What's the status?

MKT

Manu Krishnan T V Mon 3 Nov 2014

@anisha The invoice is generated. Hope you have the credentials to Knightswarm panel. Share with @joeb, so that he can make the payment.

Hope someone has the statistics of the funds.

AS

Anish Sheela Mon 3 Nov 2014

I don't have the credentials.

2014-11-03 13:31 GMT+05:30 Manu Krishnan T V (Loomio) :

Manu Krishnan T V mentioned you while discussing "Server load"

@anisha The invoice is generated. Hope you have the credentials to Knightswarm panel. Share with @joeb, so that he can make the payment.

Hope someone has the statistics of the funds.

Reply to this email directly or view it on Loomio ( http://www.loomio.org/d/DhEwz604/server-load?utm_campaign=thread_mailer&utm_medium=email&utm_source=user_mentioned#comment-312363 ).

Unfollow ( http://www.loomio.org/email_actions/unfollow_discussion/14195/yZvyz5s6mZ8WV13a6MMZ ) to stop emails about this discussion.

Change your email preferences ( http://www.loomio.org/email_preferences?unsubscribe_token=yZvyz5s6mZ8WV13a6MMZ&utm_campaign=thread_mailer&utm_medium=email&utm_source=user_mentioned ) to unsubscribe from Loomio emails.

MKT

Manu Krishnan T V Mon 3 Nov 2014

Share your GPG key then. I'll mail it to you.

PP

Pirate Praveen Thu 6 Nov 2014

Just increased concurrent sidekiq connections to 25 as stream was not receiving any new posts and production.log showed unable to get database connection and suggested we increase this number.

MKT

Manu Krishnan T V Mon 10 Nov 2014

@anisha Any update on the VPS upgrade payment?

PP

Pirate Praveen Wed 12 Nov 2014

@manukrishnantv , @anisha says he won't do it. Can you ask kinightswarm to generate another invoice? I'll mail Joe S. Lets renew the DNS also at the same time. Can you generate invoice for that too?

MKT

Manu Krishnan T V Fri 14 Nov 2014

@praveenarimbrathod The invoice is still there in the billing panel. No need to worry about the due date. Please update Joe to make the payment.

PP

Pirate Praveen Fri 14 Nov 2014

@manukrishnantv I have emailed Joe S and requested him to make the payment.

MKT

Manu Krishnan T V Fri 21 Nov 2014

@praveenarimbrathod Any update? We needs funds to renew domain / SSL Cert too.

PP

Pirate Praveen Fri 21 Nov 2014

@manukrishnantv he said he is moving funds to his paypal account. Can you generate invoice for domain/ssl?

MKT

Manu Krishnan T V Mon 24 Nov 2014

@praveenarimbrathod I'll create an account with Gandi using the poddery autistici mail id. I can't login to the Mailmain admin interface. Can you have a look at it?

PP

Pirate Praveen Tue 25 Nov 2014

@manukrishnantv, Joe S has paid the invoice. Do we need any manual steps for the upgrade? I have added list admin password to the password document and shared with you and @balasankarchelamat .

PP

Pirate Praveen Thu 27 Nov 2014

Folks, we just need to restart the server for the upgrade to reflect. @manukrishnantv and I will schedule a downtime soon and do the update.

PP

Pirate Praveen Tue 9 Dec 2014

We are currently performing the upgrade. We'll be back soon, hopefully an hour more.

MKT

Manu Krishnan T V Wed 10 Dec 2014

Had around 8 hours of downtime, creating backups, upgrading the server and setting up the basics for recurring backups.

Now running with 8GB RAM, and signups open again.

Sharing some statistics:
MySQL DB Size: 11GB
Uploaded Assets: 12GB

Hope we finally have a solution to this discussion. Need to setup complete backup and optimize MySQL.