@isaagar we can use backup.poddery.com. I'll share you access shortly.

@praveenarimbrathod ok

@isaagar follow these steps http://fosscommunity.in/index.php?title=Poddery#Setting_up_Backup and first create a similar plan for diasp.in at https://gitlab.com/piratemovin/diasp.in/wikis/home

1. How much disk space is used now? Add disk used by db and uploads.
2. We'll need to attach a new disk to backup.poddery.com and mount it like we do for poddery.com (get yourself familiar with LVM. How to create logical volumes and extend it)

Let me know if you have any questions.

@praveenarimbrathod i will start and let you know if any problem occurs

@isaagar where are we on it?

With the recent hard disk crash, we narrowly escaped losing data. We have to complete this urgently. @jayaura this is a prerequisite for updating diaspora. Can you help isaagar?

@praveenarimbrathod sure, please add @isaagar to diasp.in room

@jayaura I think @isaagar is in his hometown without good internet access. @rajudv do you know where is isaagar now? You can go through the link above and try setting up replication of postgres between two vms.

diasp.in has postgresql 9.3 and backup.fosscommunity.in (debian jessie) has postgresql 9.4. @vik @amardeepsingh can we have postgresql 9.4 backported to namaste?

I am creating an lxc container of hamara namaste locally. Here is the steps, if you want to try it https://gitlab.com/piratemovin/diasp.in/wikis/HamaraSetup

After a few trial and error, I got the hamara container to run. The problem was getting hamara-keyring package inside the rootfs. Now I have started backporting postgresql-9.4 from jessie.

As announced yesterday, https://poddery.com/posts/2639595 I will be updating postgres to 9.4 and then setting up replication.

I'll follow https://medium.com/@tk512/upgrading-postgresql-from-9-3-to-9-4-on-ubuntu-14-04-lts-2b4ddcd26535#.l0bk7ufl4

but use the backports repo I created at http://mahishasura.pxq.in/hamara/

Upgrade completed successfully. https://poddery.com/posts/2641176 Now going down for replication.

Replication setup failed as master and slave cannot be on different architectures https://poddery.com/posts/2641320 :(

Uploaded files are backed up using this cronjob now.

30 * * * * pgrep rsync || rsync -av --delete root@diasp.in:/usr/share/diaspora/public/uploads/ /var/lib/diasp.in/uploads/ >/var/lib/diasp.in/rsync-uploads.log

@isaagar which server are we using for backup of uploads?

We may be able to use a different solution like http://www.pgbackrest.org/ any volunteers to try it?

We need to find another server with 64 bit Intel/AMD architecture to proceed. I recommend we go with a scaleway VC1S model (dual core 64 bit Intel VM with 2GB RAM). We can do a crowd funding to raise the amount, it comes to 2.99 Euro per month. If anyone has cheaper options to suggest, feel free.

@tvm @vik @amardeepsingh @akshay @fayadfami @anisha @dhanesh95

I'm a complete noob when it comes to servers, but I have a couple of friends and colleagues who might know something about it. If in case I get some useful information, I'll immediately share it here.

Everyone starts as a noob :) If there is interest you can master the servers.

@amardeepsingh last time when we met at Baga, Goa, we talked about a backup server we can use. @isaagar can you check with Amardeep and confirm? If not, we'll proceed with scaleway.com option.

Talked to amardeep about the issue . He said if you want realtime backup then we should think of external server otherwise he's ok with daily backup using postgresqldump instead of slave setup.

Thanks for the update. I prefer slave setup as we get real-time incremental backups. With dump based approach, we may lose data after last dump in case of a failure.

I just signed up at scaleway.com. Access details are updated in our private repo, encrypted as usual (just added @isaagar ).

@isaagar said he'll setup backup over the weekend.

[03:34:20 IST] ‎piratepraveen‎: isaagar, we now have a scaleway.com account
‎[03:34:38 IST] ‎piratepraveen‎: isaagar, would you be able to fins some time to complete backup?
‎[03:35:28 IST] ‎piratepraveen‎: I suggest we create one instance in Amsterdam, I guess Netherlands is better for privacy comapred to France
‎[03:35:46 IST] ‎piratepraveen‎: bsc, fayad dhanesh95 ^
‎[03:36:04 IST] ‎fayad‎: piratepraveen: hi
‎[03:37:14 IST] ‎isaagar‎: piratepraveen: i will do in this weekend


[07:35:56  IST] ‎tvmanukrishnan‎: I have a dedi idling at online.net. Can spun up NAT ed VPS's. That would be good enough for backup, I hope. Additional IP with them cost 2 Euro + 3 Euro setup charges.
‎[07:36:32  IST] ‎tvmanukrishnan‎: Can have encrypted disks too.

@isaagar @fayadfami , we can use @tvm 's VPS then instead of scaleway.com

Do you need encrypted disks? If so, I need to create a KVM machine with disk encryption.

Otherwise, an LXC container will do, for which disk space can be expanded any time.

Btw, how much storage space are you looking for?

yes, we should encrypt the disk. postgresql data dir is about 2.7 GB. So I think 10GB disk would do for now.

@isaagar @fayadfami backup server is ready. You can find access details at https://gitlab.com/piratemovin/access

We should update https://gitlab.com/piratemovin/diasp.in/wikis/setting_up_backup as we configure the server so people know how to fix things if needed.

I am configuring mayasura and will update the wiki. @praveenarimbrathod Need help on server side configuration.

Feel free to ask here, xmpp, or irc/matrix.

@praveenarimbrathod @fayadfami @tvm On slave , it says could not connect to master server "210.7.64.54" .
We set max_wal_senders to 1 (on master) and including this backup server I think we actually have two backup servers . Should I increase value of max_wal_senders to 2 ? or Do you see any other problem ?

Since cannot use the first backup server, can't we remove the entries for the the first backup server?

Ok , I will comment out the entries of first server , to see if problem solves or not.

The problem was port 5432 of diasp.in was firewalled and after allowing port 5432 , slave is connecting to diasp.in.
But , I haven't seen in logs on slave machine info. relating live transactions . When I started slave machine then last entry in log says "ready to accept read only connections"
I think I should wait for some time if data is really replicating on slave.

I have checked running postgresql processes on master and slave server , it seems replication is running properly . I will update the wiki about the setup.
Also , how should we do the monitoring ??
@praveenarimbrathod how do you monitor backup server of poddery ?

@isaagar :thumbsup: super! For poddery, we just check if the backup server is running or not as backup server was getting hanged in the past. We can setup a ping test for the backup server. Also we should do a daily full db dump on the slave using cron job. We should keep the dump of last day and last week and delete the rest. This is just be on the safer side if some bad queries can mess up the db. Can you try that? or anyone else wants to volunteer for that?

Added cronjob for full db dump .

Also please post an update about the backup setup from diasp.in diaspora account.

Sorry for the delay , I will post about it after fixing replication.

Just to confirm, are we using the NAT VPS I provided? If so, did you face any issues with the common ports not reaching the VPS?

Unable to telnet port 11234 which is used by postgresql .

I re-issued backup commands and replication is now working fine.

@praveenarimbrathod Also , using uptimerobot I dont think its possible to check psql replication . I think we need to run custom script at master server for checking status..

With poddery.com, the slave server was going down unnoticed and breaking replication so added uptimerobot check to know if mysql slave is running. We can do a similar one here. Its not perfect monitoring, but better than not having any :) We can add more for sure.

In one case , I noticed that slave server was running at mayasura but with broken replication and I need to restart slave server to accept data segments from master server .
I think we need better solution for this case.

Still ,we haven't decided for the monitoring part . Zabbix is used at TBS for monitoring purposes which can solve our problem.

If we can use the same zabbix instance, we can configure zabbix for monitoring.

@praveenarimbrathod Uploads aren't backed up yet.I think @tvm provided extra disk on mayasura. Will try to do it over weekend

Do we need an action here? I'm ready to volunteer for the work. Will be needing some guidance though.