Server load

@manukrishnantv , 1256 M swap is unusual I think.

I ran following while loading several streams..

diaspora@poddery:~$ top -b -n 10 >/tmp/top2.out
diaspora@poddery:~$ grep Cpu /tmp/top.out
%Cpu(s): 1.3 us, 0.3 sy, 0.0 ni, 97.2 id, 1.0 wa, 0.0 hi, 0.0 si, 0.1 st
%Cpu(s): 2.7 us, 0.6 sy, 0.0 ni, 96.2 id, 0.3 wa, 0.0 hi, 0.1 si, 0.1 st
%Cpu(s): 2.1 us, 0.5 sy, 0.0 ni, 95.1 id, 2.0 wa, 0.0 hi, 0.0 si, 0.4 st
%Cpu(s): 0.9 us, 0.1 sy, 0.0 ni, 96.9 id, 1.7 wa, 0.0 hi, 0.0 si, 0.3 st
%Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.1 st
%Cpu(s): 3.1 us, 0.4 sy, 0.0 ni, 92.8 id, 3.4 wa, 0.0 hi, 0.1 si, 0.2 st
%Cpu(s): 7.8 us, 1.5 sy, 0.0 ni, 85.6 id, 3.7 wa, 0.0 hi, 0.1 si, 1.3 st
%Cpu(s): 4.2 us, 0.8 sy, 0.0 ni, 92.9 id, 2.0 wa, 0.0 hi, 0.1 si, 0.1 st
%Cpu(s): 8.6 us, 1.2 sy, 0.0 ni, 81.9 id, 4.6 wa, 0.0 hi, 0.1 si, 3.6 st
%Cpu(s): 3.8 us, 1.0 sy, 0.0 ni, 91.8 id, 2.5 wa, 0.0 hi, 0.1 si, 0.8 st
diaspora@poddery:~$ date
Wed Mar 19 08:09:13 IST 2014

If the memory usage was casusing a serious effect, it would have reflected in sys% usage.

stream loading was also fast.

--

Most of the memory looks accounted by processes. Yes, the memory and swap usage seem very high and could result in funny results when lots of users are logged in.

             total       used       free     shared    buffers     cached

Mem: 4061112 3924404 136708 0 4412 606724
-/+ buffers/cache: 3313268 747844
Swap: 3436540 1455928 1980612

Just that it did not translated into an issue when I checked.

--

May be I tested when there were not much users. Is there a particular time when the issue hits?

We might need to install sysstat to check for historical load details.

The memory and swap usage seems high, but I don't find that affecting the overall performance.

It seems like memory is not freed by some processes (expecially php-fpm and ruby) and the system has been running for more than 100 days.

If it starts creating problems, I would suggest restarting all processes and see how the memory usage is.

It seems like the stream loads slow now. The SMC sites was moved to the Poddery server when they had some issues as a temporary solution.

Let me contact Anivar and update the status there. Also, as I mentioned in the proposal, I hope we could donate a VM by mid of April, after all the testing.

Just did a dd test, to look at whether disk IO is causing issues and it seems fine. 150MB/s is good enough I believe. The result is as below:

poddery@poddery:~$ dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; rm test
16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 7.13325 s, 151 MB/s

@manukrishnantv that would be a good option. Important point is to sustain quality of service. We can try a reboot and see if it improves the performance.

@manukrishnantv @nandajavarma I think we should move yatra.diasporafoundation.org also to another host as it does not need a vps. Nandaja, can you try to move it to autistici.org or riseup.net hosting?

Moving out wordpress won't be a hard task. @nandajavarma @balasankarchelamat Hope you can do that.

@manukrishnantv @praveenarimbrathod Sure! I will try to move it tonight(will have a better Internet connection, by then).

@manukrishnantv I still feel we should stop new sign ups until we can guarantee a good performance. Otherwise new people joining will have a bad impression of diaspora.

I saw two extra sidekiq processes on the server. I killed them and restarted diaspora. Sign ups are closed now. Once we see good performance for next two days, we can open it again.

seems stable now

$ free -m
total used free shared buffers cached
Mem: 3965 2076 1889 0 42 929
-/+ buffers/cache: 1105 2860
Swap: 3355 140 3215


I have re-enabled sign ups. Load and memory usage is normal so I was tempted to re-open registrations.

The stream now loads faster, but the memory usage has gone to the old state.

poddery@poddery:~$ free -m
total used free shared buffers cached
Mem: 3965 3815 150 0 178 1899
-/+ buffers/cache: 1736 2228
`Swap: 3355 122 3233

Swap usage is not high though.

@manukrishnantv I think memory usage is fine as long as it is not using swap. linux considers free ram as wasted resource so it will use free ram as cache (cache frequently read files etc) . So this is quite normal.

http://www.linuxatemyram.com/ has a nice explanation.

@manukrishnantv can we move smc of fci sites to your vms?

@nandajavarma can we move diaspora yatra to autistici.org?

I have mentioned with few of the SMC community members that my company is willing to sponsor a VM, if required. As things were going without much issues, it seems like the proposal was not discussed. I'll bring this up in the SMC mailing list.

Have sent a mail to the SMC Mailing list, with my offer. Lets see how it goes.

@praveenarimbrathod Yes. I remember applying for it a while back. But could not follow up. Will do that now.

@manukrishnantv let me know when you want to do the migration. I will join you.

@nandajavarma any update on it? Actually we just need a wordpress blog, so you can sign up at noblogs.org (run by autistici) and ask autistici people to provide a dns entry.

Migration of all SMC sites are now complete. What is left is fosscommunity.in and diaspora yatra. Hope to get those sorted out this week itself. Then we could remove all php related stuffs and use the server entirely for poddery.

$ free -m
total used free shared buffers cached
Mem: 3965 3788 177 0 10 1001
-/+ buffers/cache: 2777 1188
Swap: 3355 40 3315

Services last restarted at around 4PM IST (~24 hours ago) Had to restart redis-server, which was running from the time server started. Noticed more than 1/4th of the sidekiq requests fails.

I have reduced number of sidekiq jobs to 20x2 from 25x2 (2 workers), hoping it eases resources.

@manukrishnantv any update on adding more resources?

@praveenarimbrathod No update yet. Bumped the ticket. Let's wait.

$7/month for 2GB and $13.5/month for 4GB additional RAM.

May be I can close the deal at $150 for additional 2GB for the remaining 23 months of service. Not sure though. Do we have the funds?

@anisha @balasankarchelamat how much money do we have?

@manukrishnantv I think we have the funds, can we get the same discount like the main plan? Give the final pricing and aneesh can ask Joe S to make the payment.

$300 for an additional 4GB RAM and 40GB Disk is the deal. We might need to partition the disk again to make use of additional space.

Invoice should be generated by now.

@manukrishnantv What's the status?

@anisha The invoice is generated. Hope you have the credentials to Knightswarm panel. Share with @joeb, so that he can make the payment.

Hope someone has the statistics of the funds.

I don't have the credentials.

2014-11-03 13:31 GMT+05:30 Manu Krishnan T V (Loomio) :

Manu Krishnan T V mentioned you while discussing "Server load"

@anisha The invoice is generated. Hope you have the credentials to Knightswarm panel. Share with @joeb, so that he can make the payment.

Hope someone has the statistics of the funds.

—

Reply to this email directly or view it on Loomio ( http://www.loomio.org/d/DhEwz604/server-load?utm_campaign=thread_mailer&utm_medium=email&utm_source=user_mentioned#comment-312363 ).

Unfollow ( http://www.loomio.org/email_actions/unfollow_discussion/14195/yZvyz5s6mZ8WV13a6MMZ ) to stop emails about this discussion.

Change your email preferences ( http://www.loomio.org/email_preferences?unsubscribe_token=yZvyz5s6mZ8WV13a6MMZ&utm_campaign=thread_mailer&utm_medium=email&utm_source=user_mentioned ) to unsubscribe from Loomio emails.

Share your GPG key then. I'll mail it to you.

Just increased concurrent sidekiq connections to 25 as stream was not receiving any new posts and production.log showed unable to get database connection and suggested we increase this number.

@anisha Any update on the VPS upgrade payment?

@manukrishnantv , @anisha says he won't do it. Can you ask kinightswarm to generate another invoice? I'll mail Joe S. Lets renew the DNS also at the same time. Can you generate invoice for that too?

@praveenarimbrathod The invoice is still there in the billing panel. No need to worry about the due date. Please update Joe to make the payment.

@manukrishnantv I have emailed Joe S and requested him to make the payment.

@praveenarimbrathod Any update? We needs funds to renew domain / SSL Cert too.

@manukrishnantv he said he is moving funds to his paypal account. Can you generate invoice for domain/ssl?

@praveenarimbrathod I'll create an account with Gandi using the poddery autistici mail id. I can't login to the Mailmain admin interface. Can you have a look at it?

@manukrishnantv, Joe S has paid the invoice. Do we need any manual steps for the upgrade? I have added list admin password to the password document and shared with you and @balasankarchelamat .

Folks, we just need to restart the server for the upgrade to reflect. @manukrishnantv and I will schedule a downtime soon and do the update.

We are currently performing the upgrade. We'll be back soon, hopefully an hour more.

Had around 8 hours of downtime, creating backups, upgrading the server and setting up the basics for recurring backups.

Now running with 8GB RAM, and signups open again.

Sharing some statistics:
MySQL DB Size: 11GB
Uploaded Assets: 12GB

Hope we finally have a solution to this discussion. Need to setup complete backup and optimize MySQL.