Loomio

database backup

PP
Pirate Praveen Public Seen by 295

We need to back up database and Nitesh had volunteered to set it up.

PP

Pirate Praveen Wed 18 Sep 2013

@niteshajain any updates?

MKT

Manu Krishnan T V Sat 28 Sep 2013

I am tired of following discussions at multiple platforms. Feels like this topic made few progress through the closed mails between few members.

NAJ

Nitesh A Jain Thu 10 Oct 2013

So i have taken a backup of the DB and the pictures and it is currently on the server itself. We need a place where we can safely store this backup...

PP

Pirate Praveen Sat 26 Oct 2013

@manukrishnantv did we use Sooraj's sever for this? What is the current status?

MKT

Manu Krishnan T V Sat 26 Oct 2013

Sooraj's server is not yet used for backup. It has a openmeetings instance running and also configured to deliver sooraj's website. If openmeetings is not in need, I can stop that.

PP

Pirate Praveen Fri 21 Mar 2014

@niteshajain @manukrishnantv I think we should set it up soon. How about meeting up one day for this? irc/offline.

NAJ

Nitesh A Jain Fri 21 Mar 2014

Saturday or Sunday is fine with me. Pinging @vamseekanakala also

PP

Pirate Praveen Mon 31 Mar 2014

@shafeeqk can you work with @niteshajain and @vamseekanakala to setup backup? @manukrishnantv can give access to the backup server.

SK

Shafeeq K Mon 31 Mar 2014

@praveenarimbrathod Sure. What should I do first?

PP

Pirate Praveen Mon 31 Mar 2014

@shafeeqk try backing up your pod first.

MKT

Manu Krishnan T V Wed 2 Apr 2014

@praveenarimbrathod @shafeek I believe a Master/Slave setup will be best option to backup the database, which is the most important part of a pod. Poddery database was around 5GB when we migrated to Knightswarm. So, dumping it all the time and moving to a different location doesn't seems to be a viable option.

Also, files can be backed up using tools like duplicity, which supports incremental backups through rsync and other protocols.

PP

Pirate Praveen Thu 3 Apr 2014

@shafeeqk we should ensure the data is encrypted on transport (mysql replication has that option) as well as storage (replication server should store its data on an encrypted partition.

SK

Shafeeq K Thu 3 Apr 2014

@praveenarimbrathod I see. I backed up my pod database using mysqldump. I'll try setting up a slave for it and using SSH Tunnel. But for encrypted storage, don't we need a host that allows full disk encryption? Also, @manukrishnantv , should we detach the slave after it is set up to make sure that the data stays intact?

PP

Pirate Praveen Fri 4 Apr 2014

@shafeeqk , Sooraj has one server with full disk encryption setup. Manu has the access details.

MKT

Manu Krishnan T V Sat 5 Apr 2014

@shafeeqk I'll be sitting with Sooraj today for fixing up few things in that server. Will share the details after that.

Lets create a script to detach the server for a while to create a SQL dump and reattach again.

MKT

Manu Krishnan T V Sat 23 Aug 2014

Its high time we look into this seriously, as we are taking some decisions which might not be agreed by the governmental bodies, like the take on Islamic State creating an account in our pod (Still on voting).

PP

Pirate Praveen Wed 10 Feb 2016

@manukrishnantv we have to revisit it. What is the status of Sooraj's server?

MKT

Manu Krishnan T V Tue 16 Feb 2016

The server is still up and running, but hosts Varnam build using Jenkins, and its log eats up lot of space. Last time when I checked, it had around 20GB free out of 80.

PP

Pirate Praveen Mon 14 Mar 2016

@manukrishnantv I think it is better to get a cheap scaleway vps at 2.99 euros. We can backup diasp.in as well there.

PP

Pirate Praveen started a proposal Mon 14 Mar 2016

Buy a cheap vps for backup Closed Sun 20 Mar 2016

Outcome
by Pirate Praveen Tue 25 Apr 2017

Going ahead with creating backup on a scaleway vps.

We can get one for 2.99 euro per month https://www.scaleway.com/pricing/ This way we will have full control over the backup system as well. We can share it with diasp.in as well.

Results
Agree - 1
Abstain - 1
Disagree - 1
Block - 1
1 people have voted (2%)
PP

Pirate Praveen
Agree
Mon 14 Mar 2016

We've been dragging this for long, lets do it

MKT

Manu Krishnan T V Mon 14 Mar 2016

I haven't tried Online.net's services yet, but they have been there for a while and offer cheaper servers.

The 2.99 euro server comes with a 50GB SSD. Additional volumes can be mounted at 1 Euro per 50GB, which seems cheap.

Meanwhile, a 100GB on Amazon S3, will cost just above 2$, on pay as you go. So, there are alternatives.

PP

Pirate Praveen Mon 14 Mar 2016

@manukrishnantv but Amazon is US based, it doesn't make sense to backup in US when we specifically chose non-US provider for main service.

PP

Pirate Praveen Sun 20 Mar 2016

Finally after a whole day's effort we have full backup of poddery.com

  1. backup.poddery.com is running on a Scaleway.com C1 VPS (arm soc).
  2. /var/lib/diaspora is a 50GB encrypted disk
  3. mysql replication uses an ssh tunnel to poddery.com so replication is encrypted. sshtunnel user is created on both servers. Can someone create a systemd service to start this tunnel automatically before mysql?
  4. an hourly cron job will rsync diaspora/public/uploads to /var/lib/diaspora/uploads.

http://fosscommunity.in/index.php?title=Poddery has some initial documentation. Someone please verify the backup is working and ask me if the docs are not clear on any point.