database backup

@niteshajain any updates?

I am tired of following discussions at multiple platforms. Feels like this topic made few progress through the closed mails between few members.

So i have taken a backup of the DB and the pictures and it is currently on the server itself. We need a place where we can safely store this backup...

@manukrishnantv did we use Sooraj's sever for this? What is the current status?

Sooraj's server is not yet used for backup. It has a openmeetings instance running and also configured to deliver sooraj's website. If openmeetings is not in need, I can stop that.

@niteshajain @manukrishnantv I think we should set it up soon. How about meeting up one day for this? irc/offline.

Saturday or Sunday is fine with me. Pinging @vamseekanakala also

@shafeeqk can you work with @niteshajain and @vamseekanakala to setup backup? @manukrishnantv can give access to the backup server.

@praveenarimbrathod Sure. What should I do first?

@shafeeqk try backing up your pod first.

@praveenarimbrathod @shafeek I believe a Master/Slave setup will be best option to backup the database, which is the most important part of a pod. Poddery database was around 5GB when we migrated to Knightswarm. So, dumping it all the time and moving to a different location doesn't seems to be a viable option.

Also, files can be backed up using tools like duplicity, which supports incremental backups through rsync and other protocols.

@shafeeqk we should ensure the data is encrypted on transport (mysql replication has that option) as well as storage (replication server should store its data on an encrypted partition.

steps for setting up an ssh tunnel is given here http://www.networkcomputing.com/servers-storage/how-to-set-up-ssh-encrypted-mysql-replic/229501820

@praveenarimbrathod I see. I backed up my pod database using mysqldump. I'll try setting up a slave for it and using SSH Tunnel. But for encrypted storage, don't we need a host that allows full disk encryption? Also, @manukrishnantv , should we detach the slave after it is set up to make sure that the data stays intact?

@shafeeqk , Sooraj has one server with full disk encryption setup. Manu has the access details.

@shafeeqk I'll be sitting with Sooraj today for fixing up few things in that server. Will share the details after that.

Lets create a script to detach the server for a while to create a SQL dump and reattach again.

Its high time we look into this seriously, as we are taking some decisions which might not be agreed by the governmental bodies, like the take on Islamic State creating an account in our pod (Still on voting).

@manukrishnantv we have to revisit it. What is the status of Sooraj's server?

The server is still up and running, but hosts Varnam build using Jenkins, and its log eats up lot of space. Last time when I checked, it had around 20GB free out of 80.

@manukrishnantv I think it is better to get a cheap scaleway vps at 2.99 euros. We can backup diasp.in as well there.

I haven't tried Online.net's services yet, but they have been there for a while and offer cheaper servers.

The 2.99 euro server comes with a 50GB SSD. Additional volumes can be mounted at 1 Euro per 50GB, which seems cheap.

Meanwhile, a 100GB on Amazon S3, will cost just above 2$, on pay as you go. So, there are alternatives.

@manukrishnantv but Amazon is US based, it doesn't make sense to backup in US when we specifically chose non-US provider for main service.

Finally after a whole day's effort we have full backup of poddery.com

1. backup.poddery.com is running on a Scaleway.com C1 VPS (arm soc).
2. /var/lib/diaspora is a 50GB encrypted disk
3. mysql replication uses an ssh tunnel to poddery.com so replication is encrypted. sshtunnel user is created on both servers. Can someone create a systemd service to start this tunnel automatically before mysql?
4. an hourly cron job will rsync diaspora/public/uploads to /var/lib/diaspora/uploads.

http://fosscommunity.in/index.php?title=Poddery has some initial documentation. Someone please verify the backup is working and ask me if the docs are not clear on any point.