Disk is full

We should setup monitoring in the server for future issues like this.

@juansantiago thanks for the heads up. @manukrishnantv nagios?

Are we hosting any other websites like SMC's or fosscommunity.in? If so, we can ask them to migrate.

apparently everything is fixed :)

@balasankarchelamat , yes that would save us some space.

Seems like log files are the major culprits. Almost 27G in Diaspora logs.

Also, the old files from the migration still exists in server, which counts to around 10G.

@balasankarchelamat @manukrishnantv we should remove the logs and setup a cron job to clean it up every week.

Removed 11G sidekiq.log and 7G production.log. @balasankarchelamat can you check if we can turn off logging completely? It will be a privacy boost as well.

I know you may have already checked this, still posting it here till get a better answer - https://wiki.diasporafoundation.org/FAQ_for_pod_maintainers#My_log_files_are_getting_huge.2C_help.21

@balasankarchelamat can you set up log rotation?

Sorry for the delay. I have setup logrotate with the following config

/home/diaspora/diaspora/log/*.log {
notifempty
copytruncate
missingok
compress
weekly
delaycompress
rotate 5
maxage 30
}

i.e The logs will be rotated weekly and 5 rotations will be stored. The older rotated logs will be cleared after 30 days.
For now, I've cleared the logs.

@balasankarchelamat Can you update the present disk usage status here?

poddery@poddery:~$ df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 76G 52G 21G 72% /
udev 10M 0 10M 0% /dev
tmpfs 397M 236K 397M 1% /run
/dev/mapper/poddery-root 76G 52G 21G 72% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 794M 0 794M 0% /run/shm
/dev/vda1 228M 21M 196M 10% /boot

Hi, comrades, this is just a report, not a complaint, again the pod does not share well with other pods

@juansantiago thanks for the report. I will just login and see whats the issue.

could not obtain a database connection within 5 seconds (waited 5.000204673 seconds). The max pool size is currently 20; consider increasing it.

@balasankarchelamat did you increase it earlier?

I have changed database pool size and sidekiq concurrency to 25. Hopefully this fixes the issue.

@praveenarimbrathod It was 15. I increased it to 20. The max limit is 25.

Comrades Again we have trouble connecting with other pods. Already more than 24 hours with the problem.

Sorry to be once again the bearer of bad news.

@juansantiago thanks for the heads up, we got another report of the same problem from another user as well. I have increased sidekiq workers to 2 and which means it can handle 50 connections now. But it seems mysql is not able handle such load. There is some discussion here https://poddery.com/posts/1304598 anyone familiar with mysql tuning please suggest.

following http://www.hardwaresecrets.com/printpage/How-to-Optimize-a-MySQL-Server/1747

diaspora@poddery:~$ ./mysqltuner.pl
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_IN.utf8",
LC_MONETARY = "en_IN.utf8",
LC_NUMERIC = "en_IN.utf8",
LC_MEASUREMENT = "en_IN.utf8",
LC_TIME = "en_IN.utf8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

MySQLTuner 1.3.0 - Major Hayden major@mhtx.net
Bug reports, feature requests, and downloads at http://mysqltuner.com/
Run with '--help' for additional options and output filtering
[!!] Successfully authenticated with no password - SECURITY RISK!
[OK] Currently running supported MySQL version 5.5.37-0+wheezy1
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MRG_MYISAM
[!!] InnoDB is enabled but isn't being used
[OK] Total fragmented tables: 0

-------- Security Recommendations -------------------------------------------
ERROR 1142 (42000) at line 1: SELECT command denied to user ''@'localhost' for table 'user'
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 76d 14h 46m 3s (247M q [37.402 qps], 3M conn, TX: 282B, RX: 44B)
[--] Reads / Writes: 95% / 5%
[--] Total buffers: 192.0M global + 2.7M per thread (151 max threads)
[OK] Maximum possible memory usage: 597.8M (15% of installed RAM)
[OK] Slow queries: 0% (2K/247M)
[OK] Highest usage of available connections: 39% (60/151)
[!!] Cannot calculate MyISAM index size - re-run script as root user
[OK] Query cache efficiency: 65.4% (148M cached / 226M selects)
[!!] Query cache prunes per day: 798431
[OK] Sorts requiring temporary tables: 4% (26K temp sorts / 609K sorts)
[!!] Temporary tables created on disk: 34% (245K on disk / 722K total)
[OK] Thread cache hit rate: 99% (780 created / 3M connections)
[!!] Table cache hit rate: 18% (400 open / 2K opened)
[OK] Open file limit used: 0% (0/1K)
[OK] Table locks acquired immediately: 100% (85M immediate / 85M locks)

-------- Recommendations -----------------------------------------------------
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64: http://bit.ly/1mi7c4C
Variables to adjust:
query_cache_size (> 16M)
tmp_table_size (> 16M)
max_heap_table_size (> 16M)
table_open_cache (> 400)

query_cache_size is now increased to 24MB.

I can see mysql is loading the server very heavily. We have to switch to postgresql immediately.

Another action that can help things better is to move out all other sites on the server. @manukrishnantv what can we do here?

I remember having a copy of an old DB also active in the mysql server, but not being used, which if exists will surely take more than 3GB. Removing that might also increase mysql performance for now.

poddery is now offline :(

@juansantiago I just checked and things look fine. Its working fine for me here.

I do not understand what happens, I can not access the from browser to domain and from another pod, I access to the profiles but the images are not visible.

@juansantiago Seems like there are issues. My account is not primarily on poddery. When I logged into one of the accounts now, the stream is loading too slow.

@praveenarimbrathod @balasankarchelamat Has anyone changed the settings with the admin panel? Can't access the sidekiq status now from the poddery admin account.

Shall we disable new user signup till the issues gets fixed? Seems like it seriously affects the user experience, mostly during peak usage hours.

@manukrishnantv yes, lets disable new sign ups for now.

can't find rake when trying to start server, now running bundle install

@manukrishnantv I have disabled new sign ups now. Lets move smc site off the server soon.

@manukrishnantv No, I never touch admin account.

News? I have convinced some groups that use Diaspora and have recommended poddery , now people ask me what happens

I suppose that you will be working to solve the problem and not good time to ask questions about it, but I am dedicated to promouthing the using free networks, I have had success in the past weeks and this is a problem on this struggle.

@juansantiago We are working to solve issues and it appears disabling signups for now is better than providing a bad user experience for the new users. After all, diaspora is distributed and there are other pods which could accommodate and connect the users.

Right now, we have multiple issues and we are working on it with the communities involved.

Unfortunately my knowledge is not enough to offer help to resolve problems, but if I do something, I'm here.

If it possible, I would like to receive periodic information to inform some poddery users who are not in loomio.